Hey guys, I felt 93 positions after the shakedown, but still managed to get a bronze, was not a very good placing but still think would be nice to share my solution and experiments at my Git in case anyone is interested in taking a look.
\nMy best solution from the 3 chosen, was the one that according to my experiment had the best CV values, so trusting my CV helped me here. Basically it was a weighted average using exponential log like some people also did.
\nThey were training using one augmentation pipeline and predicted using a lighter one (with out Cutout and shear), all using the data provide by Chris
\nAll models were very simple just a regular AVG pooling and dense head, label smoothing of 0.05, and Adam optimizer, one model example:
def model_fn(input_shape=(256, 256, 3)):\n input_image = L.Input(shape=input_shape, name='input_image')\n base_model = efn.EfficientNetB4(input_shape=input_shape, \n weights=config['BASE_MODEL_WEIGHTS'], \n include_top=False)\n\n x = base_model(input_image)\n x = L.GlobalAveragePooling2D()(x)\n\n output = L.Dense(1, activation='sigmoid', kernel_initializer='zeros', name='output')(x)\n\n model = Model(inputs=input_image, outputs=output)\n\n opt = optimizers.Adam(learning_rate=config['LEARNING_RATE'])\n loss = losses.BinaryCrossentropy(label_smoothing=0.05)\n model.compile(optimizer=opt, loss=loss, metrics=['AUC'])\n\n return model\nOne thing that worked very well for me was using upsampling, but in my case, I did in a way that I did not see people doing, I used tf.data.experimental.sample_from_datasets to sample from 2 different datasets, one was the regular data (2020 + 2018 + 2017) and the other was just malignant samples (all sets), then I used the weights [0.6, 0.4] this way 40% of the data were only malignant for every batch, this made the models converge faster.
\nUsed only TPUs, both from Kaggle and Colab
\nAlso, I got better results using a cyclical cosine learning rate with warm restarts and warm-up, shown below.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2Fa0849f9196f11493360c242a33cb1edc%2Fdownload.png?generation=1597710718834670&alt=media\")
Here is a fold training history for illustration.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1182060%2Fa488650b5cde87105510c0ed5c317a90%2Fhistory_fold2.png?generation=1597710894876834&alt=media\")
Here is the link to my Github, there you will find all my models, its scores, EDAs, scripts and a page with all relevant content that I gathered during the competition.
\nAbout this competition, it was a great opportunity to experiment a little more with TPUs and TensorFlow, especially with the dataset API, I feel that a should join another computer vision competition to do some more practice.
\nI tried a lot to make BiT(Big transfer) work but had no success, I got good results with Cutout, was getting close to making MixUp work good here but had no time left.
I tweaked a lot my augmentations pipeline and this was what roughly what gave me best results:
\ndef data_augment(image):\n p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_cutout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32) \n if p_shear > .2:\n if p_shear > .6:\n image = transform_shear(image, config['HEIGHT'], shear=20.)\n else:\n image = transform_shear(image, config['HEIGHT'], shear=-20.) \n if p_rotation > .2:\n if p_rotation > .6:\n image = transform_rotation(image, config['HEIGHT'], rotation=45.)\n else:\n image = transform_rotation(image, config['HEIGHT'], rotation=-45.)\n if p_crop > .2:\n image = data_augment_crop(image)\n if p_rotate > .2:\n image = data_augment_rotate(image) \n image = data_augment_spatial(image) \n image = tf.image.random_saturation(image, 0.7, 1.3)\n image = tf.image.random_contrast(image, 0.8, 1.2)\n image = tf.image.random_brightness(image, 0.1) \n if p_cutout > .5:\n image = data_augment_cutout(image) \n return image\nI would like to thank the community for all the helpful discussion and work shared and give a special thanks to @cdeotte for his amazing contributions to us all during the competition, this was a great learning opportunity.
", "messages": [ { "id": 974479, "postDate": "2020-08-18T00:45:28.217Z", "content": "Hey guys, I felt 93 positions after the shakedown, but still managed to get a bronze, was not a very good placing but still think would be nice to share my solution and experiments at my Git in case anyone is interested in taking a look.
\nMy best solution from the 3 chosen, was the one that according to my experiment had the best CV values, so trusting my CV helped me here. Basically it was a weighted average using exponential log like some people also did.
\nThey were training using one augmentation pipeline and predicted using a lighter one (with out Cutout and shear), all using the data provide by Chris
\nAll models were very simple just a regular AVG pooling and dense head, label smoothing of 0.05, and Adam optimizer, one model example:
def model_fn(input_shape=(256, 256, 3)):\n input_image = L.Input(shape=input_shape, name='input_image')\n base_model = efn.EfficientNetB4(input_shape=input_shape, \n weights=config['BASE_MODEL_WEIGHTS'], \n include_top=False)\n\n x = base_model(input_image)\n x = L.GlobalAveragePooling2D()(x)\n\n output = L.Dense(1, activation='sigmoid', kernel_initializer='zeros', name='output')(x)\n\n model = Model(inputs=input_image, outputs=output)\n\n opt = optimizers.Adam(learning_rate=config['LEARNING_RATE'])\n loss = losses.BinaryCrossentropy(label_smoothing=0.05)\n model.compile(optimizer=opt, loss=loss, metrics=['AUC'])\n\n return model\nOne thing that worked very well for me was using upsampling, but in my case, I did in a way that I did not see people doing, I used tf.data.experimental.sample_from_datasets to sample from 2 different datasets, one was the regular data (2020 + 2018 + 2017) and the other was just malignant samples (all sets), then I used the weights [0.6, 0.4] this way 40% of the data were only malignant for every batch, this made the models converge faster.
\nUsed only TPUs, both from Kaggle and Colab
\nAlso, I got better results using a cyclical cosine learning rate with warm restarts and warm-up, shown below.
\n
Here is a fold training history for illustration.
\n
Here is the link to my Github, there you will find all my models, its scores, EDAs, scripts and a page with all relevant content that I gathered during the competition.
\nAbout this competition, it was a great opportunity to experiment a little more with TPUs and TensorFlow, especially with the dataset API, I feel that a should join another computer vision competition to do some more practice.
\nI tried a lot to make BiT(Big transfer) work but had no success, I got good results with Cutout, was getting close to making MixUp work good here but had no time left.
I tweaked a lot my augmentations pipeline and this was what roughly what gave me best results:
\ndef data_augment(image):\n p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_cutout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32) \n if p_shear > .2:\n if p_shear > .6:\n image = transform_shear(image, config['HEIGHT'], shear=20.)\n else:\n image = transform_shear(image, config['HEIGHT'], shear=-20.) \n if p_rotation > .2:\n if p_rotation > .6:\n image = transform_rotation(image, config['HEIGHT'], rotation=45.)\n else:\n image = transform_rotation(image, config['HEIGHT'], rotation=-45.)\n if p_crop > .2:\n image = data_augment_crop(image)\n if p_rotate > .2:\n image = data_augment_rotate(image) \n image = data_augment_spatial(image) \n image = tf.image.random_saturation(image, 0.7, 1.3)\n image = tf.image.random_contrast(image, 0.8, 1.2)\n image = tf.image.random_brightness(image, 0.1) \n if p_cutout > .5:\n image = data_augment_cutout(image) \n return image\nI would like to thank the community for all the helpful discussion and work shared and give a special thanks to @cdeotte for his amazing contributions to us all during the competition, this was a great learning opportunity.
", "rawMarkdown": "Hey guys, I felt 93 positions after the shakedown, but still managed to get a bronze, was not a very good placing but still think would be nice to share my solution and experiments at my [Git](https://github.com/dimitreOliveira/melanoma-classification) in case anyone is interested in taking a look.\n\nMy best solution from the 3 chosen, was the one that according to my experiment had the best CV values, so trusting my CV helped me here. Basically it was a weighted average using exponential log like some people also did.\n\n#### Models\n- 1x EfficientNet B4 384x384\n- 3x EfficientNet B4 512x512\n- 1x EfficientNet B5 512x512\n\nThey were training using one augmentation pipeline and predicted using a lighter one (with out Cutout and shear), all using the data provide by Chris\nAll models were very simple just a regular AVG pooling and dense head, label smoothing of 0.05, and Adam optimizer, one model example:\n```\ndef model_fn(input_shape=(256, 256, 3)):\n input_image = L.Input(shape=input_shape, name='input_image')\n base_model = efn.EfficientNetB4(input_shape=input_shape, \n weights=config['BASE_MODEL_WEIGHTS'], \n include_top=False)\n\n x = base_model(input_image)\n x = L.GlobalAveragePooling2D()(x)\n \n output = L.Dense(1, activation='sigmoid', kernel_initializer='zeros', name='output')(x)\n\n model = Model(inputs=input_image, outputs=output)\n \n opt = optimizers.Adam(learning_rate=config['LEARNING_RATE'])\n loss = losses.BinaryCrossentropy(label_smoothing=0.05)\n model.compile(optimizer=opt, loss=loss, metrics=['AUC'])\n\n return model\n```\n\n#### Training\nOne thing that worked very well for me was using upsampling, but in my case, I did in a way that I did not see people doing, I used `tf.data.experimental.sample_from_datasets` to sample from 2 different datasets, one was the regular data (2020 + 2018 + 2017) and the other was just malignant samples (all sets), then I used the weights `[0.6, 0.4]` this way `40%` of the data were only malignant for every batch, this made the models converge faster.\nUsed only TPUs, both from Kaggle and Colab\nAlso, I got better results using a cyclical cosine learning rate with warm restarts and warm-up, shown below.\n\n\nHere is a fold training history for illustration.\n\n\nHere is the link to [my Github](https://github.com/dimitreOliveira/melanoma-classification), there you will find all my models, [its scores](https://github.com/dimitreOliveira/melanoma-classification/tree/master/Model%20backlog), EDAs, scripts and a [page with all relevant content](https://github.com/dimitreOliveira/melanoma-classification/tree/master/Documentation) that I gathered during the competition.\n\nAbout this competition, it was a great opportunity to experiment a little more with TPUs and TensorFlow, especially with the dataset API, I feel that a should join another computer vision competition to do some more practice. \nI tried a lot to make BiT(Big transfer) work but had no success, I got good results with Cutout, was getting close to making MixUp work good here but had no time left.\n\nI tweaked a lot my augmentations pipeline and this was what roughly what gave me best results:\n```\ndef data_augment(image):\n p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_cutout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32) \n if p_shear > .2:\n if p_shear > .6:\n image = transform_shear(image, config['HEIGHT'], shear=20.)\n else:\n image = transform_shear(image, config['HEIGHT'], shear=-20.) \n if p_rotation > .2:\n if p_rotation > .6:\n image = transform_rotation(image, config['HEIGHT'], rotation=45.)\n else:\n image = transform_rotation(image, config['HEIGHT'], rotation=-45.)\n if p_crop > .2:\n image = data_augment_crop(image)\n if p_rotate > .2:\n image = data_augment_rotate(image) \n image = data_augment_spatial(image) \n image = tf.image.random_saturation(image, 0.7, 1.3)\n image = tf.image.random_contrast(image, 0.8, 1.2)\n image = tf.image.random_brightness(image, 0.1) \n if p_cutout > .5:\n image = data_augment_cutout(image) \n return image\n```\n\nI would like to thank the community for all the helpful discussion and work shared and give a special thanks to @cdeotte for his amazing contributions to us all during the competition, this was a great learning opportunity.", "votes": 21 }, { "id": 980673, "postDate": "2020-08-21T19:13:55.207Z", "content": "Thanks for the detailed explanation
", "rawMarkdown": "Thanks for the detailed explanation", "votes": 1, "replies": [ { "id": 980772, "postDate": "2020-08-21T20:58:22.127Z", "content": "You're welcome @bijeeshavs
", "rawMarkdown": "You're welcome @bijeeshavs ", "votes": 1 } ] }, { "id": 980211, "postDate": "2020-08-21T12:11:47.690Z", "content": "Grats Dimitre! Will give 'sample_from_datasets' some tries!
\nAlso like 'cyclical cosine learning rate with warm restarts and warm-up'. I tried stochastic weight averaging, but it did not convince me.
\nI had a much lower cv score (average 0.925) and got a result which is 0.001 lower than yours - somehow interesting…
\nDid you try some oof ensembling strategies like Chris recently posted?
\nDid you use metadata in your ensembles? (I am still unsure about how much benefit it brings)
Great work - thx!
", "rawMarkdown": "Grats Dimitre! Will give 'sample_from_datasets' some tries! \n\nAlso like 'cyclical cosine learning rate with warm restarts and warm-up'. I tried stochastic weight averaging, but it did not convince me.\nI had a much lower cv score (average 0.925) and got a result which is 0.001 lower than yours - somehow interesting...\nDid you try some oof ensembling strategies like Chris recently posted?\nDid you use metadata in your ensembles? (I am still unsure about how much benefit it brings)\n\nGreat work - thx!", "votes": 1, "replies": [ { "id": 980590, "postDate": "2020-08-21T17:52:20.803Z", "content": "Thanks @romanweilguny , I also was surprised by sample_from_datasets, I've been looking for a long time a simple and elegant way to do batch sampling in TF, but be aware that it uses a little more memory.
Where you got the implementation for SWA? I have tried the version from tensorflow_addons but it crashed my Kaggle and Colab env.
I have tried the ensembling from Chris using 27 of my best OOFs and got a lower CV, public and private LB from it, my ensembling code worked better, but maybe I did something wrong.
\nI used meta-data on all my submission, but just blending values like 90%image ensemble +10% meta ensemble, but got just a small improvement.
By what I saw I had good models, maybe if I had ensembled them in a more efficient way I should be somewhere close to 50, still learning how to diversify the ensemble.
", "rawMarkdown": "Thanks @romanweilguny , I also was surprised by `sample_from_datasets `, I've been looking for a long time a simple and elegant way to do batch sampling in `TF`, but be aware that it uses a little more memory.\n\nWhere you got the implementation for `SWA`? I have tried the version from `tensorflow_addons` but it crashed my Kaggle and Colab env.\n\nI have tried the ensembling from Chris using 27 of my best OOFs and got a lower CV, public and private LB from it, my ensembling code worked better, but maybe I did something wrong.\n\nI used meta-data on all my submission, but just blending values like `90%image ensemble +10% meta ensemble`, but got just a small improvement.\n\nBy what I saw I had good models, maybe if I had ensembled them in a more efficient way I should be somewhere close to 50, still learning how to diversify the ensemble.", "votes": 1 }, { "id": 980649, "postDate": "2020-08-21T18:46:32.560Z", "content": "https://github.com/simon-larsson/keras-swa
", "rawMarkdown": "https://github.com/simon-larsson/keras-swa ", "votes": 1 } ] }, { "id": 980013, "postDate": "2020-08-21T09:14:31.533Z", "content": "First of all, congrats. It's really easy to read. Great!
", "rawMarkdown": "First of all, congrats. It's really easy to read. Great! ", "votes": 1, "replies": [ { "id": 980156, "postDate": "2020-08-21T11:15:18.353Z", "content": "Thanks @utshabkumarghosh
", "rawMarkdown": "Thanks @utshabkumarghosh " } ] }, { "id": 976555, "postDate": "2020-08-19T00:20:50.330Z", "content": "Congrats Dimitre. I wasn't aware of tf.data.experimental.sample_from_datasets thanks for sharing this nice trick.
Thanks @cdeotte I think batch sampling was the one single trick that helped me the most, after it, I got very consistently CV 9.30+, I guess it could work even better if you use two datasets one with only positive samples and another with only negative samples, I also got nice results sampling from datasets from different years, like {10%: 2019, 50%: 2020+2018, 40%: malignants}
", "rawMarkdown": "Thanks @cdeotte I think batch sampling was the one single trick that helped me the most, after it, I got very consistently CV 9.30+, I guess it could work even better if you use two datasets one with only positive samples and another with only negative samples, I also got nice results sampling from datasets from different years, like {10%: 2019, 50%: 2020+2018, 40%: malignants}", "votes": 1 }, { "id": 976605, "postDate": "2020-08-19T01:20:12.580Z", "content": "Great ideas
", "rawMarkdown": "Great ideas", "votes": 1 } ] }, { "id": 976287, "postDate": "2020-08-18T19:08:51.603Z", "content": "Amazing Git documentation and models.I especially liked the Model backlog. Was it manually created?
", "rawMarkdown": "Amazing Git documentation and models.I especially liked the Model backlog. Was it manually created?", "votes": 1, "replies": [ { "id": 976358, "postDate": "2020-08-18T20:09:33.247Z", "content": "Thanks @ambarish , Yeah everything was created manually, I wish there was an automated way đŸ˜„, but as I am updating it almost daily it becomes easy to maintain, after the competition ends it works as a cool portfolio of projects and experiments.
", "rawMarkdown": "Thanks @ambarish , Yeah everything was created manually, I wish there was an automated way đŸ˜„, but as I am updating it almost daily it becomes easy to maintain, after the competition ends it works as a cool portfolio of projects and experiments.", "votes": 1 } ] }, { "id": 974812, "postDate": "2020-08-18T04:04:03.297Z", "content": "Good work man, really like your \"Model backlog\" pretty convenient.
\nYou have a lot of notebooks, 159 looks interesting, but can you tell me the last one that relevant and finished from your point of view. Thank again, great work.
Thanks @ivanvoid !, this model backlog takes some time to update during the competitions, but it is a time saver when you need com compare and evaluated your models later.
I think that the best notebook was the 136 It was my best single model on private LB with 0.9396 and public 0.9470 also had a nice CV 0.937.
Nice documentation, will check them in detail for sure!
", "rawMarkdown": "Nice documentation, will check them in detail for sure!", "votes": 2, "replies": [ { "id": 974510, "postDate": "2020-08-18T01:01:41.427Z", "content": "Nice @datafan07 , let me know if there is anything to improve.
\nCongratulations on your placing and thanks for your contributions.
Thanks @epocxy !
", "rawMarkdown": "Thanks @epocxy !" } ] } ], "comments": [ { "id": 980673, "author_name": "Bijeesha Vs", "author_url": "", "post_date": "2020-08-21T19:13:55.207000", "content": "Thanks for the detailed explanation
", "votes": 1, "replies": [ { "id": 980772, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-08-21T20:58:22.127000", "content": "You're welcome @bijeeshavs
", "votes": 1, "replies": [] } ] }, { "id": 980211, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-08-21T12:11:47.690000", "content": "Grats Dimitre! Will give 'sample_from_datasets' some tries!
\nAlso like 'cyclical cosine learning rate with warm restarts and warm-up'. I tried stochastic weight averaging, but it did not convince me.
\nI had a much lower cv score (average 0.925) and got a result which is 0.001 lower than yours - somehow interesting…
\nDid you try some oof ensembling strategies like Chris recently posted?
\nDid you use metadata in your ensembles? (I am still unsure about how much benefit it brings)
Great work - thx!
", "votes": 1, "replies": [ { "id": 980590, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-08-21T17:52:20.803000", "content": "Thanks @romanweilguny , I also was surprised by sample_from_datasets, I've been looking for a long time a simple and elegant way to do batch sampling in TF, but be aware that it uses a little more memory.
Where you got the implementation for SWA? I have tried the version from tensorflow_addons but it crashed my Kaggle and Colab env.
I have tried the ensembling from Chris using 27 of my best OOFs and got a lower CV, public and private LB from it, my ensembling code worked better, but maybe I did something wrong.
\nI used meta-data on all my submission, but just blending values like 90%image ensemble +10% meta ensemble, but got just a small improvement.
By what I saw I had good models, maybe if I had ensembled them in a more efficient way I should be somewhere close to 50, still learning how to diversify the ensemble.
", "votes": 1, "replies": [] }, { "id": 980649, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-08-21T18:46:32.560000", "content": "https://github.com/simon-larsson/keras-swa
", "votes": 1, "replies": [] } ] }, { "id": 980013, "author_name": "Utshab Kumar Ghosh", "author_url": "", "post_date": "2020-08-21T09:14:31.533000", "content": "First of all, congrats. It's really easy to read. Great!
", "votes": 1, "replies": [ { "id": 980156, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-08-21T11:15:18.353000", "content": "Thanks @utshabkumarghosh
", "votes": 0, "replies": [] } ] }, { "id": 976555, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T00:20:50.330000", "content": "Congrats Dimitre. I wasn't aware of tf.data.experimental.sample_from_datasets thanks for sharing this nice trick.
Thanks @cdeotte I think batch sampling was the one single trick that helped me the most, after it, I got very consistently CV 9.30+, I guess it could work even better if you use two datasets one with only positive samples and another with only negative samples, I also got nice results sampling from datasets from different years, like {10%: 2019, 50%: 2020+2018, 40%: malignants}
", "votes": 1, "replies": [] }, { "id": 976605, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T01:20:12.580000", "content": "Great ideas
", "votes": 1, "replies": [] } ] }, { "id": 976287, "author_name": "Ambarish Ganguly", "author_url": "", "post_date": "2020-08-18T19:08:51.603000", "content": "Amazing Git documentation and models.I especially liked the Model backlog. Was it manually created?
", "votes": 1, "replies": [ { "id": 976358, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-08-18T20:09:33.247000", "content": "Thanks @ambarish , Yeah everything was created manually, I wish there was an automated way đŸ˜„, but as I am updating it almost daily it becomes easy to maintain, after the competition ends it works as a cool portfolio of projects and experiments.
", "votes": 1, "replies": [] } ] }, { "id": 974812, "author_name": "Ivan Void", "author_url": "", "post_date": "2020-08-18T04:04:03.297000", "content": "Good work man, really like your \"Model backlog\" pretty convenient.
\nYou have a lot of notebooks, 159 looks interesting, but can you tell me the last one that relevant and finished from your point of view. Thank again, great work.
Thanks @ivanvoid !, this model backlog takes some time to update during the competitions, but it is a time saver when you need com compare and evaluated your models later.
I think that the best notebook was the 136 It was my best single model on private LB with 0.9396 and public 0.9470 also had a nice CV 0.937.
Nice documentation, will check them in detail for sure!
", "votes": 2, "replies": [ { "id": 974510, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-08-18T01:01:41.427000", "content": "Nice @datafan07 , let me know if there is anything to improve.
\nCongratulations on your placing and thanks for your contributions.
Thanks @epocxy !
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "974479": "Hey guys, I felt 93 positions after the shakedown, but still managed to get a bronze, was not a very good placing but still think would be nice to share my solution and experiments at my [Git](https://github.com/dimitreOliveira/melanoma-classification) in case anyone is interested in taking a look.\n\nMy best solution from the 3 chosen, was the one that according to my experiment had the best CV values, so trusting my CV helped me here. Basically it was a weighted average using exponential log like some people also did.\n\n#### Models\n- 1x EfficientNet B4 384x384\n- 3x EfficientNet B4 512x512\n- 1x EfficientNet B5 512x512\n\nThey were training using one augmentation pipeline and predicted using a lighter one (with out Cutout and shear), all using the data provide by Chris\nAll models were very simple just a regular AVG pooling and dense head, label smoothing of 0.05, and Adam optimizer, one model example:\n```\ndef model_fn(input_shape=(256, 256, 3)):\n input_image = L.Input(shape=input_shape, name='input_image')\n base_model = efn.EfficientNetB4(input_shape=input_shape, \n weights=config['BASE_MODEL_WEIGHTS'], \n include_top=False)\n\n x = base_model(input_image)\n x = L.GlobalAveragePooling2D()(x)\n \n output = L.Dense(1, activation='sigmoid', kernel_initializer='zeros', name='output')(x)\n\n model = Model(inputs=input_image, outputs=output)\n \n opt = optimizers.Adam(learning_rate=config['LEARNING_RATE'])\n loss = losses.BinaryCrossentropy(label_smoothing=0.05)\n model.compile(optimizer=opt, loss=loss, metrics=['AUC'])\n\n return model\n```\n\n#### Training\nOne thing that worked very well for me was using upsampling, but in my case, I did in a way that I did not see people doing, I used `tf.data.experimental.sample_from_datasets` to sample from 2 different datasets, one was the regular data (2020 + 2018 + 2017) and the other was just malignant samples (all sets), then I used the weights `[0.6, 0.4]` this way `40%` of the data were only malignant for every batch, this made the models converge faster.\nUsed only TPUs, both from Kaggle and Colab\nAlso, I got better results using a cyclical cosine learning rate with warm restarts and warm-up, shown below.\n\n\nHere is a fold training history for illustration.\n\n\nHere is the link to [my Github](https://github.com/dimitreOliveira/melanoma-classification), there you will find all my models, [its scores](https://github.com/dimitreOliveira/melanoma-classification/tree/master/Model%20backlog), EDAs, scripts and a [page with all relevant content](https://github.com/dimitreOliveira/melanoma-classification/tree/master/Documentation) that I gathered during the competition.\n\nAbout this competition, it was a great opportunity to experiment a little more with TPUs and TensorFlow, especially with the dataset API, I feel that a should join another computer vision competition to do some more practice. \nI tried a lot to make BiT(Big transfer) work but had no success, I got good results with Cutout, was getting close to making MixUp work good here but had no time left.\n\nI tweaked a lot my augmentations pipeline and this was what roughly what gave me best results:\n```\ndef data_augment(image):\n p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_cutout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32) \n if p_shear > .2:\n if p_shear > .6:\n image = transform_shear(image, config['HEIGHT'], shear=20.)\n else:\n image = transform_shear(image, config['HEIGHT'], shear=-20.) \n if p_rotation > .2:\n if p_rotation > .6:\n image = transform_rotation(image, config['HEIGHT'], rotation=45.)\n else:\n image = transform_rotation(image, config['HEIGHT'], rotation=-45.)\n if p_crop > .2:\n image = data_augment_crop(image)\n if p_rotate > .2:\n image = data_augment_rotate(image) \n image = data_augment_spatial(image) \n image = tf.image.random_saturation(image, 0.7, 1.3)\n image = tf.image.random_contrast(image, 0.8, 1.2)\n image = tf.image.random_brightness(image, 0.1) \n if p_cutout > .5:\n image = data_augment_cutout(image) \n return image\n```\n\nI would like to thank the community for all the helpful discussion and work shared and give a special thanks to @cdeotte for his amazing contributions to us all during the competition, this was a great learning opportunity.", "980673": "Thanks for the detailed explanation", "980211": "Grats Dimitre! Will give 'sample_from_datasets' some tries! \n\nAlso like 'cyclical cosine learning rate with warm restarts and warm-up'. I tried stochastic weight averaging, but it did not convince me.\nI had a much lower cv score (average 0.925) and got a result which is 0.001 lower than yours - somehow interesting...\nDid you try some oof ensembling strategies like Chris recently posted?\nDid you use metadata in your ensembles? (I am still unsure about how much benefit it brings)\n\nGreat work - thx!", "980013": "First of all, congrats. It's really easy to read. Great! ", "976555": "Congrats Dimitre. I wasn't aware of `tf.data.experimental.sample_from_datasets` thanks for sharing this nice trick.", "976287": "Amazing Git documentation and models.I especially liked the Model backlog. Was it manually created?", "974812": "Good work man, really like your \"*Model backlog*\" pretty convenient.\nYou have a lot of notebooks, `159` looks interesting, but can you tell me the last one that relevant and finished from your point of view. Thank again, great work.", "974486": "Nice documentation, will check them in detail for sure!", "975513": "" } }