Our method isn't that different from those published in the top scoring notebooks. Let me list a few things that I believe contributed to our solution placing in the top three.
\n1 Trained on both fgvc7 and fgvc8 data. Interestingly, adding fgvc7 data seemed to contribute more than its fair share. Leaving 2000 images out of fgvc8 data didn't seem to make much difference. However, leaving out fgvc7 data (with the number of images in the same ballpark) led to a big drop in the leaderboard score. This was true even after the leak was fixed, though less glaring.
\n2 Usage of soft labels. Like many others observed, the labels in the training set are very noisy. The label \"complex\" is especially problematic. It looks like \"complex\" could mean too many diseases or an unidentified disease. As we were using cross entropy as the loss function, it didn't make much sense to use hard labels that are often wrong. To soften the labels, we used this simple approach:
\ntraining_loop:\n train_for_one_epoch()\n labels = gamma*predictions + (1 - gamma)*labels\nWe considered gamma as a hyperparameter and tuned it to minimize validation error.
3 Automated hyperparameter tuning. Tuned using a new protocol being developed by Luminide.
\n4 Scaling the probabilities. We left out the \"healthy\" label while training. During inference, all the images that came out negative for all the diseases were deemed healthy. Validation showed that this method overestimated the proportion of healthy leaves. To compensate, we multiplied the logistic outputs with 1.3 before rounding them to get the predictions (this is pretty much equivalent to setting the threshold to 0.4).
\nWe used pretrained weights from @rwightman's excellent timm library. The final submission averaged predictions from resnet50 and seresnext50.
", "messages": [ { "id": "1327214", "postDate": "05/29/2021 04:35:10", "content": "Our method isn't that different from those published in the top scoring notebooks. Let me list a few things that I believe contributed to our solution placing in the top three.
\n1 Trained on both fgvc7 and fgvc8 data. Interestingly, adding fgvc7 data seemed to contribute more than its fair share. Leaving 2000 images out of fgvc8 data didn't seem to make much difference. However, leaving out fgvc7 data (with the number of images in the same ballpark) led to a big drop in the leaderboard score. This was true even after the leak was fixed, though less glaring.
\n2 Usage of soft labels. Like many others observed, the labels in the training set are very noisy. The label \"complex\" is especially problematic. It looks like \"complex\" could mean too many diseases or an unidentified disease. As we were using cross entropy as the loss function, it didn't make much sense to use hard labels that are often wrong. To soften the labels, we used this simple approach:
\ntraining_loop:\n train_for_one_epoch()\n labels = gamma*predictions + (1 - gamma)*labels\nWe considered gamma as a hyperparameter and tuned it to minimize validation error.
3 Automated hyperparameter tuning. Tuned using a new protocol being developed by Luminide.
\n4 Scaling the probabilities. We left out the \"healthy\" label while training. During inference, all the images that came out negative for all the diseases were deemed healthy. Validation showed that this method overestimated the proportion of healthy leaves. To compensate, we multiplied the logistic outputs with 1.3 before rounding them to get the predictions (this is pretty much equivalent to setting the threshold to 0.4).
\nWe used pretrained weights from @rwightman's excellent timm library. The final submission averaged predictions from resnet50 and seresnext50.
", "rawMarkdown": "Our method isn't that different from those published in the top scoring notebooks. Let me list a few things that I believe contributed to our solution placing in the top three.\n\n**1 Trained on both fgvc7 and fgvc8 data**. Interestingly, adding fgvc7 data seemed to contribute more than its fair share. Leaving 2000 images out of fgvc8 data didn't seem to make much difference. However, leaving out fgvc7 data (with the number of images in the same ballpark) led to a big drop in the leaderboard score. This was true even after [the leak](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/234332) was fixed, though less glaring.\n\n**2 Usage of soft labels**. Like many others observed, the labels in the training set are very noisy. The label \"complex\" is especially problematic. It looks like \"complex\" could mean too many diseases or an unidentified disease. As we were using cross entropy as the loss function, it didn't make much sense to use hard labels that are often wrong. To soften the labels, we used this simple approach:\n\n```\ntraining_loop:\n train_for_one_epoch()\n labels = gamma*predictions + (1 - gamma)*labels\n```\n\nWe considered `gamma` as a hyperparameter and tuned it to minimize validation error.\n\n**3 Automated hyperparameter tuning**. Tuned using a new protocol being developed by [Luminide](https://luminide.com/).\n\n**4 Scaling the probabilities**. We left out the \"healthy\" label while training. During inference, all the images that came out negative for all the diseases were deemed healthy. Validation showed that this method overestimated the proportion of healthy leaves. To compensate, we multiplied the logistic outputs with 1.3 before rounding them to get the predictions (this is pretty much equivalent to setting the threshold to 0.4).\n\nWe used pretrained weights from @rwightman's excellent [timm](https://github.com/rwightman/pytorch-image-models) library. The final submission averaged predictions from resnet50 and seresnext50.", "votes": null }, { "id": "1327294", "postDate": "05/29/2021 06:23:34", "content": "Congratulations and thanks for sharing your approach, simple yet effective!
", "rawMarkdown": "Congratulations and thanks for sharing your approach, simple yet effective!", "votes": null }, { "id": "1327318", "postDate": "05/29/2021 07:01:49", "content": "Thanks for sharing your solution!
", "rawMarkdown": "Thanks for sharing your solution!", "votes": null }, { "id": "1328459", "postDate": "05/30/2021 09:22:32", "content": "Congratulations and thanks for your sharing.
\nAnd what size images were used?
Resized to 432x432 and then cropped to 384x384.
", "rawMarkdown": "Resized to 432x432 and then cropped to 384x384.", "votes": null }, { "id": "1339093", "postDate": "06/07/2021 03:18:41", "content": "Congratulations and thanks for your solution.
\nAnd the labels of fgvc7 and fgvc8 are different. What is your strategy and how do you use the labels of fgvc7.
Oh, they aren't very different. We replaced the label \"multiple_diseases\" with \"complex\".
", "rawMarkdown": "Oh, they aren't very different. We replaced the label \"multiple_diseases\" with \"complex\".", "votes": null } ], "comments": [ { "id": 1327294, "author_name": "ashish2001", "author_url": "", "post_date": "05/29/2021 06:23:34", "content": "Congratulations and thanks for sharing your approach, simple yet effective!
", "votes": null, "replies": [] }, { "id": 1327318, "author_name": "atamazian", "author_url": "", "post_date": "05/29/2021 07:01:49", "content": "Thanks for sharing your solution!
", "votes": null, "replies": [] }, { "id": 1328459, "author_name": "alekseyeliseev", "author_url": "", "post_date": "05/30/2021 09:22:32", "content": "Congratulations and thanks for your sharing.
\nAnd what size images were used?
Resized to 432x432 and then cropped to 384x384.
", "votes": null, "replies": [] } ] }, { "id": 1339093, "author_name": "h030162", "author_url": "", "post_date": "06/07/2021 03:18:41", "content": "Congratulations and thanks for your solution.
\nAnd the labels of fgvc7 and fgvc8 are different. What is your strategy and how do you use the labels of fgvc7.
Oh, they aren't very different. We replaced the label \"multiple_diseases\" with \"complex\".
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1327214": "Our method isn't that different from those published in the top scoring notebooks. Let me list a few things that I believe contributed to our solution placing in the top three.\n\n**1 Trained on both fgvc7 and fgvc8 data**. Interestingly, adding fgvc7 data seemed to contribute more than its fair share. Leaving 2000 images out of fgvc8 data didn't seem to make much difference. However, leaving out fgvc7 data (with the number of images in the same ballpark) led to a big drop in the leaderboard score. This was true even after [the leak](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/234332) was fixed, though less glaring.\n\n**2 Usage of soft labels**. Like many others observed, the labels in the training set are very noisy. The label \"complex\" is especially problematic. It looks like \"complex\" could mean too many diseases or an unidentified disease. As we were using cross entropy as the loss function, it didn't make much sense to use hard labels that are often wrong. To soften the labels, we used this simple approach:\n\n```\ntraining_loop:\n train_for_one_epoch()\n labels = gamma*predictions + (1 - gamma)*labels\n```\n\nWe considered `gamma` as a hyperparameter and tuned it to minimize validation error.\n\n**3 Automated hyperparameter tuning**. Tuned using a new protocol being developed by [Luminide](https://luminide.com/).\n\n**4 Scaling the probabilities**. We left out the \"healthy\" label while training. During inference, all the images that came out negative for all the diseases were deemed healthy. Validation showed that this method overestimated the proportion of healthy leaves. To compensate, we multiplied the logistic outputs with 1.3 before rounding them to get the predictions (this is pretty much equivalent to setting the threshold to 0.4).\n\nWe used pretrained weights from @rwightman's excellent [timm](https://github.com/rwightman/pytorch-image-models) library. The final submission averaged predictions from resnet50 and seresnext50.", "1327294": "Congratulations and thanks for sharing your approach, simple yet effective!", "1327318": "Thanks for sharing your solution!", "1328459": "Congratulations and thanks for your sharing.\nAnd what size images were used?", "1330141": "Resized to 432x432 and then cropped to 384x384.", "1339093": "Congratulations and thanks for your solution.\nAnd the labels of fgvc7 and fgvc8 are different. What is your strategy and how do you use the labels of fgvc7.", "1340247": "Oh, they aren't very different. We replaced the label \"multiple_diseases\" with \"complex\"." }, "source": "meta" }