{ "id": 175721, "title": "254th Place Solution", "url": "/competitions/siim-isic-melanoma-classification/writeups/don-t-trust-our-lb-254th-place-solution", "author_name": "", "post_date": "2021-07-08T12:56:18.607Z", "votes": 18, "comment_count": 5, "views": 0, "content": "

Thanks to the kaggle and the organizers for hosting such an interesting competition. And I like to thank my teammates, brother @udaykamal, and @tahsin for the insightful discussion that we've made along with this journey. And last but not least, We're greatly thankful to the MVP of this competition, @cdeotte for all the contribution that he's shared from the very beginning of this competition.

\n

Our experiment is based on @cdeotte 's strong baseline starter and we extended it by integrating various types of the modeling approach.

\n

Brief Summary

\n\n
- `E-Net B2` (2x) \n- `E-Net B3` (2x)\n- `E-Net B4` (3x) \n- `E-Net B5` (2x)\n- `E-Net B6` (5x) \n- `E-Net B7` (6x)\n
\n\n
- Global Average Pooling (GAP)\n- Global Max Pooling (GMP)\n- Attention Weighted Net (AWN)\n- Generalized Mean Pooling (GeM)\n- Global Average Attention Mechanism (GAAM, [ours])\n
\n

Ok, here below are the full details. (TL,DR)

\n
\n

E-Net 2

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldsCVLB
a-ENet 2GAP202'20+'1851250.9040.9345
b-ENet 2GAP1234'20+'1851250.9070.9346
\n

Next, based on the CV score, we took a simple average of them.

\n
ENet 2 = (a-ENet 2 + b-ENet 2)/2\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 3

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldsCVLB
a-ENet 3GAAM101'20+'1851250.8900.9462
b-ENet 3GAP2020'20+'1851250.9120.9435
\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 3 = (a-ENet 3*1 + b-ENet 3*2)/3\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 4

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldsCVLB
a-ENet 4GAP + AWN786'20+'1851250.91200.9334
b-ENet 4GAP786'20+'1851250.92500.9454
c-ENet 4M999'20+'18+'19+Upsampled76810.93200.9326
\n

Here, M = [GAP + GeM+AWN + GMP]

\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 4 = (a-ENet 4*1 + b-ENet 4*2 + c-ENet 4*2)/5\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 5

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDatImgFoldCVLB
a-ENet 5GAP + AWN101'20+'1838450.9080.9442
b-ENet 5M786'20+'1851230.8840.9491
\n

Here, M = [GAP + GMP + AWN]

\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 5 = (a-ENet 5*2 +  b-ENet 5*1)/3\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 6

\n

Basically E-Net 6 was our first modeling approach. So, initially, we kept the validation set the same (seed 42) and experimented in the following way.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldCVLB
a-ENet 6GAP42'20 + '1838450.9040.9454
b-ENet 6GAP + AWN42'20+'1851250.9120.9422
c-ENet 6AWN42'20+'18-51250.9180.9431
d-ENet 6GeM42'20+'1851250.9050.9405
e-ENet 6GAAM42'20+'1851250.9250.9458
\n

Next, as the above 6 submissions came from the same validation split, we further tried to find the best weights that will maximize the OOF validation score max.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
-SimpleAvgPowerAvgRankAvgBaysianOptL-BFGS-B
CV0.93190.93120.93210.93220.9320
LB0.94870.94880.94830.94900.9491
\n

As we found BaysianOpt gave the maximum CV, so later we chose it for the final blending. I've published a notebook regarding this, showed a comparison between Bayesian Optimization and L-BFGS-B methods, Optimizing Metrics: Out-of-Fold Weights Ensemble

\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 7

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldCVLB
a-ENet 7GAP101'20+'1825650.9100.9307
b-ENet 7GAP42'20+'1825650.9100.9312
c-ENet 7GAP42'20+'1838440.9230.9389
d-ENet 7M2020'20 + '18 + '19 + Upsampled76810.9280.9479
e-ENet 7M1221'20 + '18 + '19 + Upsampled51250.9220.9442
\n

Here, M = [GAP+GeM+GMP+AWN]

\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 7 = (a-ENet 7*1 + b-ENet 7*1 + c-ENet 7*2 + d-ENet 7*2 + e-ENet 7*2)/8\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

Whole Data Training, 2020 SIIM Comp.

\n

Maybe it's not full ideal but we did four experiments, inspired by @agentauers 's awesome kernel. We didn't take any validation set, this was done before 2/3 weeks of the competition end.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgLB
ENet 3+4+5GAP101'202560.9384
ENet 3+4+5GAP42'203840.9470
ENet 3+4+5GAP786'205120.9438
ENet 3+4+5GAP202'207680.9385
\n

Next, we took a simple average of these four, which leads LB: 94.8.

\n

Final Blends

\n

We couldn't give enough time for metal features. So, we've used a public meta submission form here. And gave 0.1 to the meta-features in the final blending.

\n

The final blending took all the submission of EfficientNets B2 to 7 and Meta features and combined training on the whole data set of 2020. After the generate single submission from E-Net B2 to 7 and metal features (0.1), further, we did rank ensemble with combined submission. It leads the Public LB: 0.9554, and Private 0.9383.

\n
\n

Generalized Mean Pooling (GeM)

\n
class GeneralizedMeanPooling2D(tf.keras.layers.Layer):\n    def __init__(self, p=3, epsilon=1e-6, name='', **kwargs):\n        super(GeneralizedMeanPooling2D, self).__init__(name, **kwargs)\n        self.init_p = p\n        self.epsilon = epsilon\n\n    def build(self, input_shape):\n        if isinstance(input_shape, list) or len(input_shape) != 4:\n            raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n        self.build_shape = input_shape\n        self.p = self.add_weight(\n                  name='p',\n                  shape=[1,],\n                  initializer=tf.keras.initializers.Constant(value=self.init_p),\n                  regularizer=None,\n                  trainable=True,\n                  dtype=tf.float32\n                  )\n        self.built=True\n\n    def call(self, inputs):\n        input_shape = inputs.get_shape()\n        if isinstance(inputs, list) or len(input_shape) != 4:\n            raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n        return (tf.reduce_mean(tf.abs(inputs**self.p), \n                               axis=[1,2], keepdims=False) + self.epsilon)**(1.0/self.p)\n
\n

Attention Weighted Network (AWN)

\n

From here. Thanks to @wowfattie

\n
class AttentionWeightedAverage2D(Layer):\n\n    def __init__(self, **kwargs):\n        self.init = initializers.get('uniform')\n        super(AttentionWeightedAverage2D, self).__init__(** kwargs)\n\n    def build(self, input_shape):\n        self.input_spec = [InputSpec(ndim=4)]\n        assert len(input_shape) == 4\n\n        self.W = self.add_weight(shape=(input_shape[3], 1),\n                                 name='{}_W'.format(self.name),\n                                 initializer=self.init)\n        self._trainable_weights = [self.W]\n        super(AttentionWeightedAverage2D, self).build(input_shape)\n\n    def call(self, x):\n        logits = K.dot(x, self.W)\n        x_shape = K.shape(x)\n        logits = K.reshape(logits, (x_shape[0], x_shape[1], x_shape[2]))\n        ai = K.exp(logits - K.max(logits, axis=[1,2], keepdims=True))\n        att_weights = ai / (K.sum(ai, axis=[1,2], keepdims=True) + K.epsilon())\n        weighted_input = x * K.expand_dims(att_weights)\n        result = K.sum(weighted_input, axis=[1,2])\n        return result\n\n    def get_output_shape_for(self, input_shape):\n        return self.compute_output_shape(input_shape)\n\n    def compute_output_shape(self, input_shape):\n        output_len = input_shape[3]\n        return (input_shape[0], output_len)\n
\n

Global Average Attention Mechanism

\n

The main idea of this mechanism is from K Scott Mader, currently working in Apple as ML Eng. However, you can find the vanilla implementation of the mechanism from his work (kernel). Here is one public kernel of it. And @datafan07 also used it to his pipeline, great.

\n
\n

Lastly, we like to mention some work that we've published for this competition, hope it may come helpful for future readers.

\n", "messages": [ { "id": "976939", "postDate": "08/19/2020 07:27:22", "content": "

Thanks to the kaggle and the organizers for hosting such an interesting competition. And I like to thank my teammates, brother @udaykamal, and @tahsin for the insightful discussion that we've made along with this journey. And last but not least, We're greatly thankful to the MVP of this competition, @cdeotte for all the contribution that he's shared from the very beginning of this competition.

\n

Our experiment is based on @cdeotte 's strong baseline starter and we extended it by integrating various types of the modeling approach.

\n

Brief Summary

\n\n
- `E-Net B2` (2x) \n- `E-Net B3` (2x)\n- `E-Net B4` (3x) \n- `E-Net B5` (2x)\n- `E-Net B6` (5x) \n- `E-Net B7` (6x)\n
\n\n
- Global Average Pooling (GAP)\n- Global Max Pooling (GMP)\n- Attention Weighted Net (AWN)\n- Generalized Mean Pooling (GeM)\n- Global Average Attention Mechanism (GAAM, [ours])\n
\n

Ok, here below are the full details. (TL,DR)

\n
\n

E-Net 2

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldsCVLB
a-ENet 2GAP202'20+'1851250.9040.9345
b-ENet 2GAP1234'20+'1851250.9070.9346
\n

Next, based on the CV score, we took a simple average of them.

\n
ENet 2 = (a-ENet 2 + b-ENet 2)/2\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 3

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldsCVLB
a-ENet 3GAAM101'20+'1851250.8900.9462
b-ENet 3GAP2020'20+'1851250.9120.9435
\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 3 = (a-ENet 3*1 + b-ENet 3*2)/3\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 4

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldsCVLB
a-ENet 4GAP + AWN786'20+'1851250.91200.9334
b-ENet 4GAP786'20+'1851250.92500.9454
c-ENet 4M999'20+'18+'19+Upsampled76810.93200.9326
\n

Here, M = [GAP + GeM+AWN + GMP]

\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 4 = (a-ENet 4*1 + b-ENet 4*2 + c-ENet 4*2)/5\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 5

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDatImgFoldCVLB
a-ENet 5GAP + AWN101'20+'1838450.9080.9442
b-ENet 5M786'20+'1851230.8840.9491
\n

Here, M = [GAP + GMP + AWN]

\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 5 = (a-ENet 5*2 +  b-ENet 5*1)/3\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 6

\n

Basically E-Net 6 was our first modeling approach. So, initially, we kept the validation set the same (seed 42) and experimented in the following way.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldCVLB
a-ENet 6GAP42'20 + '1838450.9040.9454
b-ENet 6GAP + AWN42'20+'1851250.9120.9422
c-ENet 6AWN42'20+'18-51250.9180.9431
d-ENet 6GeM42'20+'1851250.9050.9405
e-ENet 6GAAM42'20+'1851250.9250.9458
\n

Next, as the above 6 submissions came from the same validation split, we further tried to find the best weights that will maximize the OOF validation score max.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
-SimpleAvgPowerAvgRankAvgBaysianOptL-BFGS-B
CV0.93190.93120.93210.93220.9320
LB0.94870.94880.94830.94900.9491
\n

As we found BaysianOpt gave the maximum CV, so later we chose it for the final blending. I've published a notebook regarding this, showed a comparison between Bayesian Optimization and L-BFGS-B methods, Optimizing Metrics: Out-of-Fold Weights Ensemble

\n

The resultant prediction graph as follows:
\n\"\"

\n

E-Net 7

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgFoldCVLB
a-ENet 7GAP101'20+'1825650.9100.9307
b-ENet 7GAP42'20+'1825650.9100.9312
c-ENet 7GAP42'20+'1838440.9230.9389
d-ENet 7M2020'20 + '18 + '19 + Upsampled76810.9280.9479
e-ENet 7M1221'20 + '18 + '19 + Upsampled51250.9220.9442
\n

Here, M = [GAP+GeM+GMP+AWN]

\n

Next, based on the CV score, we took a simple weighted average of them.

\n
ENet 7 = (a-ENet 7*1 + b-ENet 7*1 + c-ENet 7*2 + d-ENet 7*2 + e-ENet 7*2)/8\n
\n

The resultant prediction graph as follows:
\n\"\"

\n

Whole Data Training, 2020 SIIM Comp.

\n

Maybe it's not full ideal but we did four experiments, inspired by @agentauers 's awesome kernel. We didn't take any validation set, this was done before 2/3 weeks of the competition end.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelTopSeedDataImgLB
ENet 3+4+5GAP101'202560.9384
ENet 3+4+5GAP42'203840.9470
ENet 3+4+5GAP786'205120.9438
ENet 3+4+5GAP202'207680.9385
\n

Next, we took a simple average of these four, which leads LB: 94.8.

\n

Final Blends

\n

We couldn't give enough time for metal features. So, we've used a public meta submission form here. And gave 0.1 to the meta-features in the final blending.

\n

The final blending took all the submission of EfficientNets B2 to 7 and Meta features and combined training on the whole data set of 2020. After the generate single submission from E-Net B2 to 7 and metal features (0.1), further, we did rank ensemble with combined submission. It leads the Public LB: 0.9554, and Private 0.9383.

\n
\n

Generalized Mean Pooling (GeM)

\n
class GeneralizedMeanPooling2D(tf.keras.layers.Layer):\n    def __init__(self, p=3, epsilon=1e-6, name='', **kwargs):\n        super(GeneralizedMeanPooling2D, self).__init__(name, **kwargs)\n        self.init_p = p\n        self.epsilon = epsilon\n\n    def build(self, input_shape):\n        if isinstance(input_shape, list) or len(input_shape) != 4:\n            raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n        self.build_shape = input_shape\n        self.p = self.add_weight(\n                  name='p',\n                  shape=[1,],\n                  initializer=tf.keras.initializers.Constant(value=self.init_p),\n                  regularizer=None,\n                  trainable=True,\n                  dtype=tf.float32\n                  )\n        self.built=True\n\n    def call(self, inputs):\n        input_shape = inputs.get_shape()\n        if isinstance(inputs, list) or len(input_shape) != 4:\n            raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n        return (tf.reduce_mean(tf.abs(inputs**self.p), \n                               axis=[1,2], keepdims=False) + self.epsilon)**(1.0/self.p)\n
\n

Attention Weighted Network (AWN)

\n

From here. Thanks to @wowfattie

\n
class AttentionWeightedAverage2D(Layer):\n\n    def __init__(self, **kwargs):\n        self.init = initializers.get('uniform')\n        super(AttentionWeightedAverage2D, self).__init__(** kwargs)\n\n    def build(self, input_shape):\n        self.input_spec = [InputSpec(ndim=4)]\n        assert len(input_shape) == 4\n\n        self.W = self.add_weight(shape=(input_shape[3], 1),\n                                 name='{}_W'.format(self.name),\n                                 initializer=self.init)\n        self._trainable_weights = [self.W]\n        super(AttentionWeightedAverage2D, self).build(input_shape)\n\n    def call(self, x):\n        logits = K.dot(x, self.W)\n        x_shape = K.shape(x)\n        logits = K.reshape(logits, (x_shape[0], x_shape[1], x_shape[2]))\n        ai = K.exp(logits - K.max(logits, axis=[1,2], keepdims=True))\n        att_weights = ai / (K.sum(ai, axis=[1,2], keepdims=True) + K.epsilon())\n        weighted_input = x * K.expand_dims(att_weights)\n        result = K.sum(weighted_input, axis=[1,2])\n        return result\n\n    def get_output_shape_for(self, input_shape):\n        return self.compute_output_shape(input_shape)\n\n    def compute_output_shape(self, input_shape):\n        output_len = input_shape[3]\n        return (input_shape[0], output_len)\n
\n

Global Average Attention Mechanism

\n

The main idea of this mechanism is from K Scott Mader, currently working in Apple as ML Eng. However, you can find the vanilla implementation of the mechanism from his work (kernel). Here is one public kernel of it. And @datafan07 also used it to his pipeline, great.

\n
\n

Lastly, we like to mention some work that we've published for this competition, hope it may come helpful for future readers.

\n", "rawMarkdown": "Thanks to the kaggle and the organizers for hosting such an interesting competition. And I like to thank my teammates, brother @udaykamal, and @tahsin for the insightful discussion that we've made along with this journey. And last but not least, We're greatly thankful to the **MVP** of this competition, @cdeotte for all the contribution that he's shared from the very beginning of this competition. \n\nOur experiment is based on @cdeotte 's strong [baseline starter](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) and we extended it by integrating various types of the modeling approach.\n\n# Brief Summary\n- All the Base-Models are `EfficientNets`, and here **E-Net** for short. Total of 20 models only.\n\n```\n- `E-Net B2` (2x) \n- `E-Net B3` (2x)\n- `E-Net B4` (3x) \n- `E-Net B5` (2x)\n- `E-Net B6` (5x) \n- `E-Net B7` (6x)\n```\n\n- We have used various types of the top model including \n```\n- Global Average Pooling (GAP)\n- Global Max Pooling (GMP)\n- Attention Weighted Net (AWN)\n- Generalized Mean Pooling (GeM)\n- Global Average Attention Mechanism (GAAM, [ours])\n```\n\nOk, here below are the full details. (**TL,DR**)\n\n---\n\n# E-Net 2\n\n|Model| Top | Seed | Data | Img |Folds | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 2|GAP|202|'20+'18|512|5 |0.904|0.9345|\n|b-ENet 2|GAP|1234|'20+'18|512|5 |0.907|0.9346|\n\nNext, based on the **CV** score, we took a simple average of them. \n\n```\nENet 2 = (a-ENet 2 + b-ENet 2)/2\n```\n\nThe resultant prediction graph as follows: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Ff107b28e41ce558fa67b26e0abe56730%2F2.png?generation=1597815112995545&alt=media)\n\n\n# E-Net 3\n\n|Model| Top | Seed | Data | Img |Folds | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 3|GAAM|101|'20+'18|512|5 |0.890|0.9462|\n|b-ENet 3|GAP|2020|'20+'18|512|5 |0.912|0.9435|\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 3 = (a-ENet 3*1 + b-ENet 3*2)/3\n```\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2F4e767d1caa113e72e38aafcce541f403%2F3.png?generation=1597815707202196&alt=media)\n\n# E-Net 4\n\n|Model| Top | Seed | Data | Img |Folds | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 4|GAP + AWN |786|'20+'18|512|5 |0.9120|0.9334|\n|b-ENet 4|GAP|786|'20+'18|512|5 |0.9250|0.9454|\n|c-ENet 4|M|999|'20+'18+'19+Upsampled |768|1 |0.9320|0.9326|\n\nHere, M = **[GAP + GeM+AWN + GMP]**\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 4 = (a-ENet 4*1 + b-ENet 4*2 + c-ENet 4*2)/5\n```\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Fb70062f56a279740e23d527f38f31831%2F4.png?generation=1597816112525326&alt=media)\n\n# E-Net 5\n\n|Model| Top | Seed | Dat | Img |Fold | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 5|GAP + AWN|101|'20+'18|384|5 |0.908|0.9442|\n|b-ENet 5|M|786|'20+'18|512|3 |0.884|0.9491|\n\nHere, M = **[GAP + GMP + AWN]**\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 5 = (a-ENet 5*2 + b-ENet 5*1)/3\n```\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2F5a468bba3f008360fa58b416f70019e3%2F5.png?generation=1597816460348906&alt=media)\n\n# E-Net 6\n\nBasically `E-Net 6` was our first modeling approach. So, initially, we kept the validation set the same (seed `42`) and experimented in the following way.\n\n|Model| Top | Seed | Data| Img |Fold | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 6|GAP |42|'20 + '18|384|5 |0.904|0.9454|\n|b-ENet 6|GAP + AWN|42|'20+'18|512|5 |0.912|0.9422|\n|c-ENet 6|AWN|42|'20+'18-|512|5 |0.918|0.9431|\n|d-ENet 6|GeM |42|'20+'18|512|5|0.905|0.9405|\n|e-ENet 6|GAAM|42|'20+'18|512|5 |0.925|0.9458|\n\nNext, as the above 6 submissions came from the same validation split, we further tried to find the best weights that will maximize the **OOF** validation score max. \n\n|-| SimpleAvg | PowerAvg | RankAvg |BaysianOpt | L-BFGS-B | \n|---|---|---|---|---|---|\n|CV | 0.9319 |0.9312|0.9321| 0.9322| 0.9320 | \n|LB | 0.9487 |0.9488|0.9483| 0.9490 | 0.9491 | \n\nAs we found `BaysianOpt` gave the maximum CV, so later we chose it for the final blending. I've published a notebook regarding this, showed a comparison between **Bayesian Optimization** and **L-BFGS-B** methods, [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble)\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Faf77c044bad954675dae053b8419ec80%2F6.png?generation=1597821688482478&alt=media)\n\n# E-Net 7\n\n|Model| Top | Seed | Data | Img|Fold | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 7|GAP |101|'20+'18|256|5 |0.910|0.9307|\n|b-ENet 7|GAP |42|'20+'18|256|5 |0.910|0.9312|\n|c-ENet 7|GAP|42|'20+'18|384|4 |0.923|0.9389|\n|d-ENet 7|M|2020|'20 + '18 + '19 + Upsampled|768|1 |0.928|0.9479|\n|e-ENet 7|M|1221|'20 + '18 + '19 + Upsampled|512|5|0.922|0.9442|\n\nHere, M = **[GAP+GeM+GMP+AWN]**\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 7 = (a-ENet 7*1 + b-ENet 7*1 + c-ENet 7*2 + d-ENet 7*2 + e-ENet 7*2)/8\n```\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Fb428333f2381ff3dba527e031e401985%2F7.png?generation=1597817972968820&alt=media)\n\n\n# Whole Data Training, 2020 SIIM Comp.\n\nMaybe it's not full ideal but we did four experiments, inspired by @agentauers 's awesome [kernel](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once). We didn't take any validation set, this was done before 2/3 weeks of the competition end. \n\n|Model| Top | Seed | Data | Img | LB | \n|---|---|---|---|---|---|\n|ENet 3+4+5|GAP |101|'20|256|0.9384|\n|ENet 3+4+5|GAP |42|'20|384|0.9470|\n|ENet 3+4+5|GAP |786|'20|512|0.9438|\n|ENet 3+4+5|GAP |202|'20|768|0.9385|\n\nNext, we took a simple average of these four, which leads `LB`: **94.8**.\n\n# Final Blends\n\nWe couldn't give enough time for metal features. So, we've used a public meta submission form [here](https://www.kaggle.com/datafan07/eda-modelling-of-the-external-data-inc-ensemble). And gave `0.1` to the meta-features in the final blending.\n\nThe final blending took all the submission of `EfficientNets B2 to 7` and Meta features and combined training on the whole data set of 2020. After the generate single submission from `E-Net B2 to 7` and metal features (`0.1`), further, we did rank ensemble with combined submission. It leads the Public LB: **0.9554**, and Private **0.9383**.\n\n---\n\n## Generalized Mean Pooling (GeM)\n\n```python\nclass GeneralizedMeanPooling2D(tf.keras.layers.Layer):\n def __init__(self, p=3, epsilon=1e-6, name='', **kwargs):\n super(GeneralizedMeanPooling2D, self).__init__(name, **kwargs)\n self.init_p = p\n self.epsilon = epsilon\n \n def build(self, input_shape):\n if isinstance(input_shape, list) or len(input_shape) != 4:\n raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n self.build_shape = input_shape\n self.p = self.add_weight(\n name='p',\n shape=[1,],\n initializer=tf.keras.initializers.Constant(value=self.init_p),\n regularizer=None,\n trainable=True,\n dtype=tf.float32\n )\n self.built=True\n\n def call(self, inputs):\n input_shape = inputs.get_shape()\n if isinstance(inputs, list) or len(input_shape) != 4:\n raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n return (tf.reduce_mean(tf.abs(inputs**self.p), \n axis=[1,2], keepdims=False) + self.epsilon)**(1.0/self.p)\n\n```\n\n### Attention Weighted Network (AWN) \n\nFrom [here](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77269#454482). Thanks to @wowfattie \n\n```python\nclass AttentionWeightedAverage2D(Layer):\n\n def __init__(self, **kwargs):\n self.init = initializers.get('uniform')\n super(AttentionWeightedAverage2D, self).__init__(** kwargs)\n\n def build(self, input_shape):\n self.input_spec = [InputSpec(ndim=4)]\n assert len(input_shape) == 4\n\n self.W = self.add_weight(shape=(input_shape[3], 1),\n name='{}_W'.format(self.name),\n initializer=self.init)\n self._trainable_weights = [self.W]\n super(AttentionWeightedAverage2D, self).build(input_shape)\n\n def call(self, x):\n logits = K.dot(x, self.W)\n x_shape = K.shape(x)\n logits = K.reshape(logits, (x_shape[0], x_shape[1], x_shape[2]))\n ai = K.exp(logits - K.max(logits, axis=[1,2], keepdims=True))\n att_weights = ai / (K.sum(ai, axis=[1,2], keepdims=True) + K.epsilon())\n weighted_input = x * K.expand_dims(att_weights)\n result = K.sum(weighted_input, axis=[1,2])\n return result\n\n def get_output_shape_for(self, input_shape):\n return self.compute_output_shape(input_shape)\n\n def compute_output_shape(self, input_shape):\n output_len = input_shape[3]\n return (input_shape[0], output_len)\n```\n\n### Global Average Attention Mechanism\n\nThe main idea of this mechanism is from [K Scott Mader](https://www.kaggle.com/kmader), currently working in Apple as ML Eng. However, you can find the vanilla implementation of the mechanism from his work (kernel). Here is one [public kernel](https://www.kaggle.com/hiramcho/melanoma-efficientnetb6-with-attention-mechanism) of it. And @datafan07 also used it to his pipeline, great.\n\n---\n\nLastly, we like to mention some work that we've published for this competition, hope it may come helpful for future readers.\n\n - [TF.Keras: Melanoma Classification Starter, TabNet](https://www.kaggle.com/ipythonx/tf-keras-melanoma-classification-starter-tabnet)\n- [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble)\n- [PyTorch: [Training CV] Melanoma Starter. GhostNet + TTA](https://www.kaggle.com/ipythonx/training-cv-melanoma-starter-ghostnet-tta)\n- [PyTorch: TResNet:HP-GPU Dedicated Net+Grad-Accumulation+TTA](https://www.kaggle.com/ipythonx/tresnet-hp-gpu-dedicated-net-grad-accumulation-tta)", "votes": null }, { "id": "976964", "postDate": "08/19/2020 07:48:41", "content": "

It is really helpful and well explanation.

\n

M is quite different in each section.
\nIt mean that result use some types of the top together, is it right?

\n

Thanks for sharing!
\n@ipythonx

", "rawMarkdown": "It is really helpful and well explanation.\n\n`M` is quite different in each section.\nIt mean that result use some types of the top together, is it right?\n\nThanks for sharing! \n@ipythonx", "votes": null }, { "id": "976974", "postDate": "08/19/2020 07:55:09", "content": "

Yes, you're right. -)

", "rawMarkdown": "Yes, you're right. -)", "votes": null }, { "id": "977801", "postDate": "08/19/2020 17:56:27", "content": "

Very informative and well-written. Feeling blessed for getting the opportunity to work alongside you and @udaykamal.

", "rawMarkdown": "Very informative and well-written. Feeling blessed for getting the opportunity to work alongside you and @udaykamal.", "votes": null }, { "id": "978908", "postDate": "08/20/2020 13:44:04", "content": "

@ipythonx , So informative. I have touched with few new things and terms ever before. Thanks for sharing. I will try to use the learning from your notebooks and solutions in future.

", "rawMarkdown": "ipythonx , So informative. I have touched with few new things and terms ever before. Thanks for sharing. I will try to use the learning from your notebooks and solutions in future.", "votes": null }, { "id": "979312", "postDate": "08/20/2020 18:54:03", "content": "

Congrats @ipythonx and your brothers on a great job.

", "rawMarkdown": "Congrats @ipythonx and your brothers on a great job.", "votes": null } ], "comments": [ { "id": 976964, "author_name": "piantic", "author_url": "", "post_date": "08/19/2020 07:48:41", "content": "

It is really helpful and well explanation.

\n

M is quite different in each section.
\nIt mean that result use some types of the top together, is it right?

\n

Thanks for sharing!
\n@ipythonx

", "votes": null, "replies": [ { "id": 976974, "author_name": "ipythonx", "author_url": "", "post_date": "08/19/2020 07:55:09", "content": "

Yes, you're right. -)

", "votes": null, "replies": [] } ] }, { "id": 977801, "author_name": "tahsin", "author_url": "", "post_date": "08/19/2020 17:56:27", "content": "

Very informative and well-written. Feeling blessed for getting the opportunity to work alongside you and @udaykamal.

", "votes": null, "replies": [] }, { "id": 978908, "author_name": "mdfahimreshm", "author_url": "", "post_date": "08/20/2020 13:44:04", "content": "

@ipythonx , So informative. I have touched with few new things and terms ever before. Thanks for sharing. I will try to use the learning from your notebooks and solutions in future.

", "votes": null, "replies": [] }, { "id": 979312, "author_name": "brendan45774", "author_url": "", "post_date": "08/20/2020 18:54:03", "content": "

Congrats @ipythonx and your brothers on a great job.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "976939": "Thanks to the kaggle and the organizers for hosting such an interesting competition. And I like to thank my teammates, brother @udaykamal, and @tahsin for the insightful discussion that we've made along with this journey. And last but not least, We're greatly thankful to the **MVP** of this competition, @cdeotte for all the contribution that he's shared from the very beginning of this competition. \n\nOur experiment is based on @cdeotte 's strong [baseline starter](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) and we extended it by integrating various types of the modeling approach.\n\n# Brief Summary\n- All the Base-Models are `EfficientNets`, and here **E-Net** for short. Total of 20 models only.\n\n```\n- `E-Net B2` (2x) \n- `E-Net B3` (2x)\n- `E-Net B4` (3x) \n- `E-Net B5` (2x)\n- `E-Net B6` (5x) \n- `E-Net B7` (6x)\n```\n\n- We have used various types of the top model including \n```\n- Global Average Pooling (GAP)\n- Global Max Pooling (GMP)\n- Attention Weighted Net (AWN)\n- Generalized Mean Pooling (GeM)\n- Global Average Attention Mechanism (GAAM, [ours])\n```\n\nOk, here below are the full details. (**TL,DR**)\n\n---\n\n# E-Net 2\n\n|Model| Top | Seed | Data | Img |Folds | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 2|GAP|202|'20+'18|512|5 |0.904|0.9345|\n|b-ENet 2|GAP|1234|'20+'18|512|5 |0.907|0.9346|\n\nNext, based on the **CV** score, we took a simple average of them. \n\n```\nENet 2 = (a-ENet 2 + b-ENet 2)/2\n```\n\nThe resultant prediction graph as follows: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Ff107b28e41ce558fa67b26e0abe56730%2F2.png?generation=1597815112995545&alt=media)\n\n\n# E-Net 3\n\n|Model| Top | Seed | Data | Img |Folds | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 3|GAAM|101|'20+'18|512|5 |0.890|0.9462|\n|b-ENet 3|GAP|2020|'20+'18|512|5 |0.912|0.9435|\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 3 = (a-ENet 3*1 + b-ENet 3*2)/3\n```\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2F4e767d1caa113e72e38aafcce541f403%2F3.png?generation=1597815707202196&alt=media)\n\n# E-Net 4\n\n|Model| Top | Seed | Data | Img |Folds | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 4|GAP + AWN |786|'20+'18|512|5 |0.9120|0.9334|\n|b-ENet 4|GAP|786|'20+'18|512|5 |0.9250|0.9454|\n|c-ENet 4|M|999|'20+'18+'19+Upsampled |768|1 |0.9320|0.9326|\n\nHere, M = **[GAP + GeM+AWN + GMP]**\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 4 = (a-ENet 4*1 + b-ENet 4*2 + c-ENet 4*2)/5\n```\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Fb70062f56a279740e23d527f38f31831%2F4.png?generation=1597816112525326&alt=media)\n\n# E-Net 5\n\n|Model| Top | Seed | Dat | Img |Fold | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 5|GAP + AWN|101|'20+'18|384|5 |0.908|0.9442|\n|b-ENet 5|M|786|'20+'18|512|3 |0.884|0.9491|\n\nHere, M = **[GAP + GMP + AWN]**\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 5 = (a-ENet 5*2 + b-ENet 5*1)/3\n```\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2F5a468bba3f008360fa58b416f70019e3%2F5.png?generation=1597816460348906&alt=media)\n\n# E-Net 6\n\nBasically `E-Net 6` was our first modeling approach. So, initially, we kept the validation set the same (seed `42`) and experimented in the following way.\n\n|Model| Top | Seed | Data| Img |Fold | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 6|GAP |42|'20 + '18|384|5 |0.904|0.9454|\n|b-ENet 6|GAP + AWN|42|'20+'18|512|5 |0.912|0.9422|\n|c-ENet 6|AWN|42|'20+'18-|512|5 |0.918|0.9431|\n|d-ENet 6|GeM |42|'20+'18|512|5|0.905|0.9405|\n|e-ENet 6|GAAM|42|'20+'18|512|5 |0.925|0.9458|\n\nNext, as the above 6 submissions came from the same validation split, we further tried to find the best weights that will maximize the **OOF** validation score max. \n\n|-| SimpleAvg | PowerAvg | RankAvg |BaysianOpt | L-BFGS-B | \n|---|---|---|---|---|---|\n|CV | 0.9319 |0.9312|0.9321| 0.9322| 0.9320 | \n|LB | 0.9487 |0.9488|0.9483| 0.9490 | 0.9491 | \n\nAs we found `BaysianOpt` gave the maximum CV, so later we chose it for the final blending. I've published a notebook regarding this, showed a comparison between **Bayesian Optimization** and **L-BFGS-B** methods, [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble)\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Faf77c044bad954675dae053b8419ec80%2F6.png?generation=1597821688482478&alt=media)\n\n# E-Net 7\n\n|Model| Top | Seed | Data | Img|Fold | CV | LB | \n|---|---|---|---|---|---|---|---|\n|a-ENet 7|GAP |101|'20+'18|256|5 |0.910|0.9307|\n|b-ENet 7|GAP |42|'20+'18|256|5 |0.910|0.9312|\n|c-ENet 7|GAP|42|'20+'18|384|4 |0.923|0.9389|\n|d-ENet 7|M|2020|'20 + '18 + '19 + Upsampled|768|1 |0.928|0.9479|\n|e-ENet 7|M|1221|'20 + '18 + '19 + Upsampled|512|5|0.922|0.9442|\n\nHere, M = **[GAP+GeM+GMP+AWN]**\n\nNext, based on the **CV** score, we took a simple weighted average of them. \n\n```\nENet 7 = (a-ENet 7*1 + b-ENet 7*1 + c-ENet 7*2 + d-ENet 7*2 + e-ENet 7*2)/8\n```\n\nThe resultant prediction graph as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Fb428333f2381ff3dba527e031e401985%2F7.png?generation=1597817972968820&alt=media)\n\n\n# Whole Data Training, 2020 SIIM Comp.\n\nMaybe it's not full ideal but we did four experiments, inspired by @agentauers 's awesome [kernel](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once). We didn't take any validation set, this was done before 2/3 weeks of the competition end. \n\n|Model| Top | Seed | Data | Img | LB | \n|---|---|---|---|---|---|\n|ENet 3+4+5|GAP |101|'20|256|0.9384|\n|ENet 3+4+5|GAP |42|'20|384|0.9470|\n|ENet 3+4+5|GAP |786|'20|512|0.9438|\n|ENet 3+4+5|GAP |202|'20|768|0.9385|\n\nNext, we took a simple average of these four, which leads `LB`: **94.8**.\n\n# Final Blends\n\nWe couldn't give enough time for metal features. So, we've used a public meta submission form [here](https://www.kaggle.com/datafan07/eda-modelling-of-the-external-data-inc-ensemble). And gave `0.1` to the meta-features in the final blending.\n\nThe final blending took all the submission of `EfficientNets B2 to 7` and Meta features and combined training on the whole data set of 2020. After the generate single submission from `E-Net B2 to 7` and metal features (`0.1`), further, we did rank ensemble with combined submission. It leads the Public LB: **0.9554**, and Private **0.9383**.\n\n---\n\n## Generalized Mean Pooling (GeM)\n\n```python\nclass GeneralizedMeanPooling2D(tf.keras.layers.Layer):\n def __init__(self, p=3, epsilon=1e-6, name='', **kwargs):\n super(GeneralizedMeanPooling2D, self).__init__(name, **kwargs)\n self.init_p = p\n self.epsilon = epsilon\n \n def build(self, input_shape):\n if isinstance(input_shape, list) or len(input_shape) != 4:\n raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n self.build_shape = input_shape\n self.p = self.add_weight(\n name='p',\n shape=[1,],\n initializer=tf.keras.initializers.Constant(value=self.init_p),\n regularizer=None,\n trainable=True,\n dtype=tf.float32\n )\n self.built=True\n\n def call(self, inputs):\n input_shape = inputs.get_shape()\n if isinstance(inputs, list) or len(input_shape) != 4:\n raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n return (tf.reduce_mean(tf.abs(inputs**self.p), \n axis=[1,2], keepdims=False) + self.epsilon)**(1.0/self.p)\n\n```\n\n### Attention Weighted Network (AWN) \n\nFrom [here](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77269#454482). Thanks to @wowfattie \n\n```python\nclass AttentionWeightedAverage2D(Layer):\n\n def __init__(self, **kwargs):\n self.init = initializers.get('uniform')\n super(AttentionWeightedAverage2D, self).__init__(** kwargs)\n\n def build(self, input_shape):\n self.input_spec = [InputSpec(ndim=4)]\n assert len(input_shape) == 4\n\n self.W = self.add_weight(shape=(input_shape[3], 1),\n name='{}_W'.format(self.name),\n initializer=self.init)\n self._trainable_weights = [self.W]\n super(AttentionWeightedAverage2D, self).build(input_shape)\n\n def call(self, x):\n logits = K.dot(x, self.W)\n x_shape = K.shape(x)\n logits = K.reshape(logits, (x_shape[0], x_shape[1], x_shape[2]))\n ai = K.exp(logits - K.max(logits, axis=[1,2], keepdims=True))\n att_weights = ai / (K.sum(ai, axis=[1,2], keepdims=True) + K.epsilon())\n weighted_input = x * K.expand_dims(att_weights)\n result = K.sum(weighted_input, axis=[1,2])\n return result\n\n def get_output_shape_for(self, input_shape):\n return self.compute_output_shape(input_shape)\n\n def compute_output_shape(self, input_shape):\n output_len = input_shape[3]\n return (input_shape[0], output_len)\n```\n\n### Global Average Attention Mechanism\n\nThe main idea of this mechanism is from [K Scott Mader](https://www.kaggle.com/kmader), currently working in Apple as ML Eng. However, you can find the vanilla implementation of the mechanism from his work (kernel). Here is one [public kernel](https://www.kaggle.com/hiramcho/melanoma-efficientnetb6-with-attention-mechanism) of it. And @datafan07 also used it to his pipeline, great.\n\n---\n\nLastly, we like to mention some work that we've published for this competition, hope it may come helpful for future readers.\n\n - [TF.Keras: Melanoma Classification Starter, TabNet](https://www.kaggle.com/ipythonx/tf-keras-melanoma-classification-starter-tabnet)\n- [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble)\n- [PyTorch: [Training CV] Melanoma Starter. GhostNet + TTA](https://www.kaggle.com/ipythonx/training-cv-melanoma-starter-ghostnet-tta)\n- [PyTorch: TResNet:HP-GPU Dedicated Net+Grad-Accumulation+TTA](https://www.kaggle.com/ipythonx/tresnet-hp-gpu-dedicated-net-grad-accumulation-tta)", "976964": "It is really helpful and well explanation.\n\n`M` is quite different in each section.\nIt mean that result use some types of the top together, is it right?\n\nThanks for sharing! \n@ipythonx", "976974": "Yes, you're right. -)", "977801": "Very informative and well-written. Feeling blessed for getting the opportunity to work alongside you and @udaykamal.", "978908": "ipythonx , So informative. I have touched with few new things and terms ever before. Thanks for sharing. I will try to use the learning from your notebooks and solutions in future.", "979312": "Congrats @ipythonx and your brothers on a great job." }, "source": "meta" }