I am delighted to share with you the winning submission in these notebooks,
\n\nYour links are pinned to the key versions. Although you will probably find some parts of the code redundant or not quite efficient, I didn't want to touch a comma to show the composition of the winning solution as it is here. This way, you can check both the clean inputs included in the inference notebooks and the automatic generation of the final solution.
\n\nI just want to add 'a little' to my summary of the competition below and highlight the small details that perhaps made my score stronger.
\n\nFinally the TPU Leaderboard Prizes were settled and as you can imagine I am very happy with the results. Thank you very much to everyone. Thanks to all the Kaggle team and Google-Cloud-TPU for making this competition possible and thanks to the 848 teams that participated in this challenge.
\n\nWhether it is explicitly (sharing notebooks and posting comments and discussions) or implicitly (pushing the leaderboard up with high scores) I think Kagglers are the ones that give a unique value to any challenge.
\n\nI believe that more than a few Data Scientists will come to review the material in 'Flower Classification with TPUs' to get initiated or indeed improve the use of this powerful technology in their projects. If you are one of them here I detail, without order of preference or priority, a series of authors (sorry because I surely left some behind) of kernels whose work in this competition you should review (all Tensorflow).
\n\nThanks @mgornergoogle (Martin Görner), @xiejialun (Xie29), @yihdarshieh (Yih-Dar SHIEH), @cdeotte (Chris Deotte), @calebeverett (Caleb), @romanweilguny (Roman Weilguny) but also don't forget to look at everything else if needed.
\n\nI would state that the playground objective, which was aimed at facilitating the use of TPU technology, has been more than achieved.
\n\nThere are three key kernels at the beinning of the competition,\n- ratan123/densenet201-flower-classification-with-tpus\n- xhlulu/flowers-tpu-concise-efficientnet-b7\n- wrrosa/tpu-enet-b7-densenet
\n\nThe baseline of these notebooks is enough, for with the allowed competition data, to obtain the limit in the range of 0.9700 score. I am referring to the limit because of what we have seen in practice later on.
\n\nOn how to beat the limit of that score: the use of external data. @cdeotte (Chris Deotte) in this discussions (1) Oxford Flowers - Kaggle Dataset for Everyone. (2) How to Score Over LB 0.970+ and @hengck23 (Heng CherKeng) in [placeholder] external data and how to use them made public the manner.
\n\nIn my opinion this was the exciting part, I mean the rush to include these new datasets in training and see how the performance of the models improved very fast.
\n\nI have to make a super special mention here for @kirillblinov (Kirill Blinov) and its shared external dataset tf_flower_photo_tfrec, great work. In this way, @szacho (Michal Szachniewicz) y @calebeverett (Caleb) made an interesting contribution.
\n\nBy the way, one of the external dataset did not work for me, openimage.
I am referring here to all the final polemics about whether or not to use external datasets. The host never tried to hide the composition of the images in the test set and their origin. In my particular opinion I would have let the competition run until the end by allowing any source of data.
\n\nIt was also a very hard challenge to work on these new data and in any case the final obstacle is not so much the difficulty of the objective itself but the competition between the participants to overcome a score. The later, however, as long as everyone can access any advantage as happened here.
\n\nYou can be more or less in line with the direction that was finally decided, but you don't have to lose your commitment to respect and follow the rules of the competition for this reason.
\n\nI expressed my disagreement with the final treatment of the Private Leaderboard and explained it here.
\n\nIf you have enjoyed the competition from the beginning, participating in the published notebooks and discussions I think you won't find anything new in the kernels I have shared but you will probably be interested in knowing what could be missing or left over from your solution to achieve a higher score in the Public or Private Leaderboard.
\n\nAugmentation \nI just used the files for an image size of 512x512, I did not preprocess original images. On them, Random Blockout y Horizontal Random Flip were the best choices.
Validation \nSince the initial validation dataset was very extensive and its information very useful in terms of performance, I divided the validation files into two subsets generating an additional Fold. Something like that looking for cross validation,
Fold1>>>\nVALIDATIONFILENAMES1=VALIDATIONFILENAMES[:int(len(VALIDATIONFILENAMES)/2)] \nFold2>>> VALIDATIONFILENAMES2=VALIDATIONFILENAMES[int(len(VALIDATIONFILENAMES)/2):]\nTRAININGFILENAMES1=TRAININGFILENAMES+VALIDATIONFILENAMES1 TRAININGFILENAMES2=TRAININGFILENAMES+VALIDATIONFILENAMES2\nDenseNet-201\nImagenet Weights with tf.keras.layers.GlobalAveragePooling2D(). Little or no adjustment here, I kept BATCH_SIZE=128 and I brought to 4 the LR-RAMPUP-EPOCHS and LR-SUSTAIN-EPOCHS parameters of the well-tuned 'CustomLRSchedule' from @mgornergoogle (Martin Görner) in his ' Five 'Flowers' Notebook.
EfficientNet-B7\nNoisy-Student Weights with tf.keras.layers.GlobalAveragePooling2D(). The rest is the same as Densenet-201 except for 5 the epochs number ramp-up and sustain.
Loss y Optimizer
\nShared by both models, Loss with 'sparse_categorical_crossentropy' and Optimizer with Adam(lr='CustomLRSchedule').
Model Checkpoints\nThe following callback was used to save the best weights according to the loss in validation,
mc=tf.keras.callbacks.ModelCheckpoint('name.h5',verbose=1,monitor='val_loss', \n mode='min',save_best_only=True,save_weights_only=True)\nNUM_EPOCHS number 35, 50 to DenseNet and 35, 40 to EfficientNet.Therefore with the details above I actually had 8 trained models available with which to perform the classification inference.\n1. DenseNet (4), 2xFold with best-checkpoints epochs 24-18, Fold1 and 25-24, Fold2.\n2. EfficientNet (4), 2xFold with best-checkpoints on epochs 17-16, Fold1 and 14-13, Fold2.
Test Time Augmentation\nThe strategy here was to explode the training augmentated with 2 types of prediction. One of them (1) with the original 512x512 images without any transformation and the other (2) with 100% of the images flipped using the function tf.image.flip_left_right(image) as the only augment.
Averaging Out of Folds\nI join the oofs of each bagging and model with a simple average of the logits, not probabilities. Following TTA strategy I perform the same operation with the full-flip images and let's say I get the following sets,
1 oof-non-flip-densenet\n1 oof-non-flip-efficientnet\n1 oof-full-flip-densenet\n1 oof-full-flip-efficientnet\nBest weighting to EfficientNet-B7: 0.71\nValidation F1-Score: 0.969903\nBest weighting for EfficientNet-B7: 0.65\nValidation F1-Score: 0.970014\nBest weighting for validation Non-Flip: 0.05 (any value 0.05 to 0.21)\nValidation F1-Score: 0.970367\nI am delighted to share with you the winning submission in these notebooks,
\n\nYour links are pinned to the key versions. Although you will probably find some parts of the code redundant or not quite efficient, I didn't want to touch a comma to show the composition of the winning solution as it is here. This way, you can check both the clean inputs included in the inference notebooks and the automatic generation of the final solution.
\n\nI just want to add 'a little' to my summary of the competition below and highlight the small details that perhaps made my score stronger.
\n\nFinally the TPU Leaderboard Prizes were settled and as you can imagine I am very happy with the results. Thank you very much to everyone. Thanks to all the Kaggle team and Google-Cloud-TPU for making this competition possible and thanks to the 848 teams that participated in this challenge.
\n\nWhether it is explicitly (sharing notebooks and posting comments and discussions) or implicitly (pushing the leaderboard up with high scores) I think Kagglers are the ones that give a unique value to any challenge.
\n\nI believe that more than a few Data Scientists will come to review the material in 'Flower Classification with TPUs' to get initiated or indeed improve the use of this powerful technology in their projects. If you are one of them here I detail, without order of preference or priority, a series of authors (sorry because I surely left some behind) of kernels whose work in this competition you should review (all Tensorflow).
\n\nThanks @mgornergoogle (Martin Görner), @xiejialun (Xie29), @yihdarshieh (Yih-Dar SHIEH), @cdeotte (Chris Deotte), @calebeverett (Caleb), @romanweilguny (Roman Weilguny) but also don't forget to look at everything else if needed.
\n\nI would state that the playground objective, which was aimed at facilitating the use of TPU technology, has been more than achieved.
\n\nThere are three key kernels at the beinning of the competition,\n- ratan123/densenet201-flower-classification-with-tpus\n- xhlulu/flowers-tpu-concise-efficientnet-b7\n- wrrosa/tpu-enet-b7-densenet
\n\nThe baseline of these notebooks is enough, for with the allowed competition data, to obtain the limit in the range of 0.9700 score. I am referring to the limit because of what we have seen in practice later on.
\n\nOn how to beat the limit of that score: the use of external data. @cdeotte (Chris Deotte) in this discussions (1) Oxford Flowers - Kaggle Dataset for Everyone. (2) How to Score Over LB 0.970+ and @hengck23 (Heng CherKeng) in [placeholder] external data and how to use them made public the manner.
\n\nIn my opinion this was the exciting part, I mean the rush to include these new datasets in training and see how the performance of the models improved very fast.
\n\nI have to make a super special mention here for @kirillblinov (Kirill Blinov) and its shared external dataset tf_flower_photo_tfrec, great work. In this way, @szacho (Michal Szachniewicz) y @calebeverett (Caleb) made an interesting contribution.
\n\nBy the way, one of the external dataset did not work for me, openimage.
I am referring here to all the final polemics about whether or not to use external datasets. The host never tried to hide the composition of the images in the test set and their origin. In my particular opinion I would have let the competition run until the end by allowing any source of data.
\n\nIt was also a very hard challenge to work on these new data and in any case the final obstacle is not so much the difficulty of the objective itself but the competition between the participants to overcome a score. The later, however, as long as everyone can access any advantage as happened here.
\n\nYou can be more or less in line with the direction that was finally decided, but you don't have to lose your commitment to respect and follow the rules of the competition for this reason.
\n\nI expressed my disagreement with the final treatment of the Private Leaderboard and explained it here.
\n\nIf you have enjoyed the competition from the beginning, participating in the published notebooks and discussions I think you won't find anything new in the kernels I have shared but you will probably be interested in knowing what could be missing or left over from your solution to achieve a higher score in the Public or Private Leaderboard.
\n\nAugmentation \nI just used the files for an image size of 512x512, I did not preprocess original images. On them, Random Blockout y Horizontal Random Flip were the best choices.
Validation \nSince the initial validation dataset was very extensive and its information very useful in terms of performance, I divided the validation files into two subsets generating an additional Fold. Something like that looking for cross validation,
Fold1>>>\nVALIDATIONFILENAMES1=VALIDATIONFILENAMES[:int(len(VALIDATIONFILENAMES)/2)] \nFold2>>> VALIDATIONFILENAMES2=VALIDATIONFILENAMES[int(len(VALIDATIONFILENAMES)/2):]\nTRAININGFILENAMES1=TRAININGFILENAMES+VALIDATIONFILENAMES1 TRAININGFILENAMES2=TRAININGFILENAMES+VALIDATIONFILENAMES2\nDenseNet-201\nImagenet Weights with tf.keras.layers.GlobalAveragePooling2D(). Little or no adjustment here, I kept BATCH_SIZE=128 and I brought to 4 the LR-RAMPUP-EPOCHS and LR-SUSTAIN-EPOCHS parameters of the well-tuned 'CustomLRSchedule' from @mgornergoogle (Martin Görner) in his ' Five 'Flowers' Notebook.
EfficientNet-B7\nNoisy-Student Weights with tf.keras.layers.GlobalAveragePooling2D(). The rest is the same as Densenet-201 except for 5 the epochs number ramp-up and sustain.
Loss y Optimizer
\nShared by both models, Loss with 'sparse_categorical_crossentropy' and Optimizer with Adam(lr='CustomLRSchedule').
Model Checkpoints\nThe following callback was used to save the best weights according to the loss in validation,
mc=tf.keras.callbacks.ModelCheckpoint('name.h5',verbose=1,monitor='val_loss', \n mode='min',save_best_only=True,save_weights_only=True)\nNUM_EPOCHS number 35, 50 to DenseNet and 35, 40 to EfficientNet.Therefore with the details above I actually had 8 trained models available with which to perform the classification inference.\n1. DenseNet (4), 2xFold with best-checkpoints epochs 24-18, Fold1 and 25-24, Fold2.\n2. EfficientNet (4), 2xFold with best-checkpoints on epochs 17-16, Fold1 and 14-13, Fold2.
Test Time Augmentation\nThe strategy here was to explode the training augmentated with 2 types of prediction. One of them (1) with the original 512x512 images without any transformation and the other (2) with 100% of the images flipped using the function tf.image.flip_left_right(image) as the only augment.
Averaging Out of Folds\nI join the oofs of each bagging and model with a simple average of the logits, not probabilities. Following TTA strategy I perform the same operation with the full-flip images and let's say I get the following sets,
1 oof-non-flip-densenet\n1 oof-non-flip-efficientnet\n1 oof-full-flip-densenet\n1 oof-full-flip-efficientnet\nBest weighting to EfficientNet-B7: 0.71\nValidation F1-Score: 0.969903\nBest weighting for EfficientNet-B7: 0.65\nValidation F1-Score: 0.970014\nBest weighting for validation Non-Flip: 0.05 (any value 0.05 to 0.21)\nValidation F1-Score: 0.970367\nopenimage.\n\n### The Controversial Part\nI am referring here to all the final polemics about whether or not to use external datasets. The host never tried to hide the composition of the images in the test set and their origin. In my particular opinion I would have let the competition run until the end by allowing any source of data. \n \nIt was also a very hard challenge to work on these new data and in any case the final obstacle is not so much the difficulty of the objective itself but the competition between the participants to overcome a score. The later, however, as long as everyone can access any advantage as happened here.\n \nYou can be more or less in line with the direction that was finally decided, but you don't have to lose your commitment to respect and follow the rules of the competition for this reason. \n \nI expressed my disagreement with the final treatment of the Private Leaderboard and explained it [here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/151494#84954).\n\n### Training Topics\nIf you have enjoyed the competition from the beginning, participating in the published notebooks and discussions I think you won't find anything new in the kernels I have shared but you will probably be interested in knowing what could be missing or left over from your solution to achieve a higher score in the Public or Private Leaderboard.\n \n- **Augmentation** \nI just used the files for an image size of 512x512, I did not preprocess original images. On them, Random Blockout y Horizontal Random Flip were the best choices. \n \n- **Validation** \nSince the initial validation dataset was very extensive and its information very useful in terms of performance, I divided the validation files into two subsets generating an additional Fold. Something like that looking for cross validation,\n\n\nFold1>>>\nVALIDATIONFILENAMES1=VALIDATIONFILENAMES[:int(len(VALIDATIONFILENAMES)/2)] \nFold2>>> VALIDATIONFILENAMES2=VALIDATIONFILENAMES[int(len(VALIDATIONFILENAMES)/2):]\nTRAININGFILENAMES1=TRAININGFILENAMES+VALIDATIONFILENAMES1 TRAININGFILENAMES2=TRAININGFILENAMES+VALIDATIONFILENAMES2\nImagenet Weights with tf.keras.layers.GlobalAveragePooling2D(). Little or no adjustment here, I kept BATCH_SIZE=128 and I brought to 4 the LR-RAMPUP-EPOCHS and LR-SUSTAIN-EPOCHS parameters of the well-tuned 'CustomLRSchedule' from @mgornergoogle (Martin Görner) in his ' Five 'Flowers' Notebook.\n \n- **EfficientNet-B7**\nNoisy-Student Weights with tf.keras.layers.GlobalAveragePooling2D(). The rest is the same as Densenet-201 except for 5 the epochs number ramp-up and sustain. \n \n- **Loss y Optimizer** \nShared by both models, Loss with 'sparse_categorical_crossentropy' and Optimizer with Adam(lr='CustomLRSchedule'). \n \n- **Model Checkpoints**\nThe following callback was used to save the best weights according to the loss in validation, \nmc=tf.keras.callbacks.ModelCheckpoint('name.h5',verbose=1,monitor='val_loss', \n mode='min',save_best_only=True,save_weights_only=True)\nNUM_EPOCHS number 35, 50 to DenseNet and 35, 40 to EfficientNet.\n \n### Inference Topics\nTherefore with the details above I actually had 8 trained models available with which to perform the classification inference.\n1. DenseNet (4), 2xFold with best-checkpoints epochs 24-18, Fold1 and 25-24, Fold2.\n2. EfficientNet (4), 2xFold with best-checkpoints on epochs 17-16, Fold1 and 14-13, Fold2.\n \n- **Test Time Augmentation**\nThe strategy here was to explode the training augmentated with 2 types of prediction. One of them (1) with the original 512x512 images without any transformation and the other (2) with 100% of the images flipped using the function tf.image.flip_left_right(image) as the only augment. \n \n- **Averaging Out of Folds**\nI join the oofs of each bagging and model with a simple average of the logits, not probabilities. Following TTA strategy I perform the same operation with the full-flip images and let's say I get the following sets,\n \n1 oof-non-flip-densenet\n1 oof-non-flip-efficientnet\n1 oof-full-flip-densenet\n1 oof-full-flip-efficientnet\nBest weighting to EfficientNet-B7: 0.71\nValidation F1-Score: 0.969903\nBest weighting for EfficientNet-B7: 0.65\nValidation F1-Score: 0.970014\nBest weighting for validation Non-Flip: 0.05 (any value 0.05 to 0.21)\nValidation F1-Score: 0.970367\nCongratulations! and thanks for the mention.
", "rawMarkdown": "Congratulations! and thanks for the mention.", "votes": null }, { "id": "888931", "postDate": "06/16/2020 16:51:18", "content": "Thank you for sharing a great job!
", "rawMarkdown": "Thank you for sharing a great job!", "votes": null }, { "id": "888993", "postDate": "06/16/2020 17:47:31", "content": "Thank you for sharing the solution. I like the fact that you ensemble at the logit level instead of ensembling the probabilities. (Not quite sure how you get the logits from the probabilities though.)
", "rawMarkdown": "Thank you for sharing the solution. I like the fact that you ensemble at the logit level instead of ensembling the probabilities. (Not quite sure how you get the logits from the probabilities though.)", "votes": null }, { "id": "889142", "postDate": "06/16/2020 19:47:35", "content": "Hi @mgornergoogle , thanks to you too. I used the logit and expit (inverse logit) functions of scipy.special to get these transformations. To avoid -inf and +inf values in the logit output (for subsequent manipulations) I use a simple np.nan_to_num.
About the performance of these transformations, but with a focus on TTA, @calebeverett shared a huge notebook here as a benchmark.
", "rawMarkdown": "Hi @mgornergoogle , thanks to you too. I used thelogit and expit (inverse logit) functions of scipy.special to get these transformations. To avoid -inf and +inf values in the logit output (for subsequent manipulations) I use a simple np.nan_to_num. \n \nAbout the performance of these transformations, but with a focus on TTA, @calebeverett shared a huge notebook [here](https://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures) as a benchmark.",
"votes": null
},
{
"id": "969843",
"postDate": "08/14/2020 01:54:05",
"content": "Congratulations Manuel. You achieved a fantastic score using only competition train data. Well done!
", "rawMarkdown": "Congratulations Manuel. You achieved a fantastic score using only competition train data. Well done!", "votes": null }, { "id": "970319", "postDate": "08/14/2020 11:04:10", "content": "This solution will be better with your post here… Thanks a lot!
", "rawMarkdown": "This solution will be better with your post here... Thanks a lot!", "votes": null }, { "id": "987849", "postDate": "08/27/2020 15:13:38", "content": "Thanks for sharing, I learned a lot from looking through your notebooks!
", "rawMarkdown": "Thanks for sharing, I learned a lot from looking through your notebooks!", "votes": null }, { "id": "987853", "postDate": "08/27/2020 15:16:55", "content": "Thanks @benghertner, that's great for me!
", "rawMarkdown": "Thanks @benghertner, that's great for me!", "votes": null }, { "id": "1050391", "postDate": "10/15/2020 11:22:08", "content": "thank you very much very insightful even for newbie (me)
", "rawMarkdown": "thank you very much very insightful even for newbie (me)", "votes": null } ], "comments": [ { "id": 969843, "author_name": "cdeotte", "author_url": "", "post_date": "08/14/2020 01:54:05", "content": "Congratulations Manuel. You achieved a fantastic score using only competition train data. Well done!
", "votes": null, "replies": [ { "id": 970319, "author_name": "coreacasa", "author_url": "", "post_date": "08/14/2020 11:04:10", "content": "This solution will be better with your post here… Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 987849, "author_name": "benghertner", "author_url": "", "post_date": "08/27/2020 15:13:38", "content": "Thanks for sharing, I learned a lot from looking through your notebooks!
", "votes": null, "replies": [ { "id": 987853, "author_name": "coreacasa", "author_url": "", "post_date": "08/27/2020 15:16:55", "content": "Thanks @benghertner, that's great for me!
", "votes": null, "replies": [] } ] }, { "id": 1050391, "author_name": "rizkioktafianto", "author_url": "", "post_date": "10/15/2020 11:22:08", "content": "thank you very much very insightful even for newbie (me)
", "votes": null, "replies": [] }, { "id": 888919, "author_name": "calebeverett", "author_url": "", "post_date": "06/16/2020 16:41:39", "content": "Congratulations! and thanks for the mention.
", "votes": null, "replies": [ { "id": 888931, "author_name": "coreacasa", "author_url": "", "post_date": "06/16/2020 16:51:18", "content": "Thank you for sharing a great job!
", "votes": null, "replies": [] } ] }, { "id": 888993, "author_name": "mgorner", "author_url": "", "post_date": "06/16/2020 17:47:31", "content": "Thank you for sharing the solution. I like the fact that you ensemble at the logit level instead of ensembling the probabilities. (Not quite sure how you get the logits from the probabilities though.)
", "votes": null, "replies": [ { "id": 889142, "author_name": "coreacasa", "author_url": "", "post_date": "06/16/2020 19:47:35", "content": "Hi @mgornergoogle , thanks to you too. I used the logit and expit (inverse logit) functions of scipy.special to get these transformations. To avoid -inf and +inf values in the logit output (for subsequent manipulations) I use a simple np.nan_to_num.
About the performance of these transformations, but with a focus on TTA, @calebeverett shared a huge notebook here as a benchmark.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "888451": "### Submission Notebooks \n \nI am delighted to share with you the winning submission in these notebooks,\n \n1. [Training-Densenet201-Bagging1](https://www.kaggle.com/coreacasa/flower-class-tpu-densenet-training-v2)\n2. [Training-Densenet201-Bagging2](https://www.kaggle.com/coreacasa/flower-class-tpu-densenet-training-v3)\n3. [Training-EfficientnetB7-Bagging1](https://www.kaggle.com/coreacasa/flower-class-tpu-efficientnet-training-v2)\n4. [Training-EfficientnetB7-Bagging2](https://www.kaggle.com/coreacasa/flower-class-tpu-efficientnet-training-v3)\n5. [Inference-Flower-Classification-TPU](https://www.kaggle.com/coreacasa/flower-classification-tpu-inference-v3)\n \nYour links are pinned to the key versions. Although you will probably find some parts of the code redundant or not quite efficient, I didn't want to touch a comma to show the composition of the winning solution as it is here. This way, you can check both the clean inputs included in the inference notebooks and the automatic generation of the final solution. \n \nI just want to add 'a little' to my summary of the competition below and highlight the small details that perhaps made my score stronger.\n\n### Acknowledgements\nFinally the TPU Leaderboard Prizes were settled and as you can imagine I am very happy with the results. Thank you very much to everyone. Thanks to all the Kaggle team and Google-Cloud-TPU for making this competition possible and thanks to the 848 teams that participated in this challenge.\n \nWhether it is explicitly (sharing notebooks and posting comments and discussions) or implicitly (pushing the leaderboard up with high scores) I think Kagglers are the ones that give a unique value to any challenge.\n \n### What you can't miss \nI believe that more than a few Data Scientists will come to review the material in 'Flower Classification with TPUs' to get initiated or indeed improve the use of this powerful technology in their projects. If you are one of them here I detail, without order of preference or priority, a series of authors (sorry because I surely left some behind) of kernels whose work in this competition you should review (all Tensorflow).\n \nThanks @mgornergoogle (Martin Görner), @xiejialun (Xie29), @yihdarshieh (Yih-Dar SHIEH), @cdeotte (Chris Deotte), @calebeverett (Caleb), @romanweilguny (Roman Weilguny) but also don't forget to look at everything else if needed. \n\nI would state that the playground objective, which was aimed at facilitating the use of TPU technology, has been more than achieved.\n\n### The Funny Part \nThere are three key kernels at the beinning of the competition,\n- [ratan123/densenet201-flower-classification-with-tpus](https://www.kaggle.com/ratan123/densenet201-flower-classification-with-tpus)\n- [xhlulu/flowers-tpu-concise-efficientnet-b7](https://www.kaggle.com/xhlulu/flowers-tpu-concise-efficientnet-b7)\n- [wrrosa/tpu-enet-b7-densenet](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet)\n \nThe baseline of these notebooks is enough, for with the allowed competition data, to obtain the limit in the range of 0.9700 score. I am referring to the limit because of what we have seen in practice later on.\n \nOn how to beat the limit of that score: the use of external data. @cdeotte (Chris Deotte) in this discussions (1) [Oxford Flowers - Kaggle Dataset for Everyone](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140964). (2) [How to Score Over LB 0.970+](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329) and @hengck23 (Heng CherKeng) in [[placeholder] external data and how to use them](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/140866) made public the manner. \n \nIn my opinion this was the exciting part, I mean the rush to include these new datasets in training and see how the performance of the models improved very fast.\n \nI have to make a super special mention here for @kirillblinov (Kirill Blinov) and its shared external dataset [tf\\_flower\\_photo\\_tfrec](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec), great work. In this way, @szacho (Michal Szachniewicz) y @calebeverett (Caleb) made an interesting contribution.\n \nBy the way, one of the external dataset did not work for me,openimage.\n\n### The Controversial Part\nI am referring here to all the final polemics about whether or not to use external datasets. The host never tried to hide the composition of the images in the test set and their origin. In my particular opinion I would have let the competition run until the end by allowing any source of data. \n \nIt was also a very hard challenge to work on these new data and in any case the final obstacle is not so much the difficulty of the objective itself but the competition between the participants to overcome a score. The later, however, as long as everyone can access any advantage as happened here.\n \nYou can be more or less in line with the direction that was finally decided, but you don't have to lose your commitment to respect and follow the rules of the competition for this reason. \n \nI expressed my disagreement with the final treatment of the Private Leaderboard and explained it [here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/151494#84954).\n\n### Training Topics\nIf you have enjoyed the competition from the beginning, participating in the published notebooks and discussions I think you won't find anything new in the kernels I have shared but you will probably be interested in knowing what could be missing or left over from your solution to achieve a higher score in the Public or Private Leaderboard.\n \n- **Augmentation** \nI just used the files for an image size of 512x512, I did not preprocess original images. On them, Random Blockout y Horizontal Random Flip were the best choices. \n \n- **Validation** \nSince the initial validation dataset was very extensive and its information very useful in terms of performance, I divided the validation files into two subsets generating an additional Fold. Something like that looking for cross validation,\n\n\nFold1>>>\nVALIDATIONFILENAMES1=VALIDATIONFILENAMES[:int(len(VALIDATIONFILENAMES)/2)] \nFold2>>> VALIDATIONFILENAMES2=VALIDATIONFILENAMES[int(len(VALIDATIONFILENAMES)/2):]\nTRAININGFILENAMES1=TRAININGFILENAMES+VALIDATIONFILENAMES1 TRAININGFILENAMES2=TRAININGFILENAMES+VALIDATIONFILENAMES2\nImagenet Weights with tf.keras.layers.GlobalAveragePooling2D(). Little or no adjustment here, I kept BATCH_SIZE=128 and I brought to 4 the LR-RAMPUP-EPOCHS and LR-SUSTAIN-EPOCHS parameters of the well-tuned 'CustomLRSchedule' from @mgornergoogle (Martin Görner) in his ' Five 'Flowers' Notebook.\n \n- **EfficientNet-B7**\nNoisy-Student Weights with tf.keras.layers.GlobalAveragePooling2D(). The rest is the same as Densenet-201 except for 5 the epochs number ramp-up and sustain. \n \n- **Loss y Optimizer** \nShared by both models, Loss with 'sparse_categorical_crossentropy' and Optimizer with Adam(lr='CustomLRSchedule'). \n \n- **Model Checkpoints**\nThe following callback was used to save the best weights according to the loss in validation, \nmc=tf.keras.callbacks.ModelCheckpoint('name.h5',verbose=1,monitor='val_loss', \n mode='min',save_best_only=True,save_weights_only=True)\nNUM_EPOCHS number 35, 50 to DenseNet and 35, 40 to EfficientNet.\n \n### Inference Topics\nTherefore with the details above I actually had 8 trained models available with which to perform the classification inference.\n1. DenseNet (4), 2xFold with best-checkpoints epochs 24-18, Fold1 and 25-24, Fold2.\n2. EfficientNet (4), 2xFold with best-checkpoints on epochs 17-16, Fold1 and 14-13, Fold2.\n \n- **Test Time Augmentation**\nThe strategy here was to explode the training augmentated with 2 types of prediction. One of them (1) with the original 512x512 images without any transformation and the other (2) with 100% of the images flipped using the function tf.image.flip_left_right(image) as the only augment. \n \n- **Averaging Out of Folds**\nI join the oofs of each bagging and model with a simple average of the logits, not probabilities. Following TTA strategy I perform the same operation with the full-flip images and let's say I get the following sets,\n \n1 oof-non-flip-densenet\n1 oof-non-flip-efficientnet\n1 oof-full-flip-densenet\n1 oof-full-flip-efficientnet\nBest weighting to EfficientNet-B7: 0.71\nValidation F1-Score: 0.969903\nBest weighting for EfficientNet-B7: 0.65\nValidation F1-Score: 0.970014\nBest weighting for validation Non-Flip: 0.05 (any value 0.05 to 0.21)\nValidation F1-Score: 0.970367\nlogit and expit (inverse logit) functions of scipy.special to get these transformations. To avoid -inf and +inf values in the logit output (for subsequent manipulations) I use a simple np.nan_to_num. \n \nAbout the performance of these transformations, but with a focus on TTA, @calebeverett shared a huge notebook [here](https://www.kaggle.com/calebeverett/comparison-of-tta-prediction-procedures) as a benchmark.",
"969843": "Congratulations Manuel. You achieved a fantastic score using only competition train data. Well done!",
"970319": "This solution will be better with your post here... Thanks a lot!",
"987849": "Thanks for sharing, I learned a lot from looking through your notebooks!",
"987853": "Thanks @benghertner, that's great for me!",
"1050391": "thank you very much very insightful even for newbie (me)"
},
"source": "meta"
}