Thank you Makerere University AI Lab and Kaggle for hosting this competition.
\nI would like to ask you for clarification regarding on Competition requirement about the
\nexpected number of labels that the model used for prediction.
Based on data requirement/specification, our expected labels (5 labels) are:
\n\"0\":string\"Cassava Bacterial Blight (CBB)\"
\n\"1\":string\"Cassava Brown Streak Disease (CBSD)\"
\n\"2\":string\"Cassava Green Mottle (CGM)\"
\n\"3\":string\"Cassava Mosaic Disease (CMD)\"
\n\"4\": string\"Healthy\"
Given that there's a previous research work related to this competition and have research output result posted in TensorFlow Hub: https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2. This pre-trained model was trained using 6 labels, namely:
\n\"0\":string \"Cassava Bacterial Blight (CBB)\"
\n\"1\": string \"Cassava Brown Streak Disease (CBSD)\"
\n\"2\": string \"Cassava Green Mottle (CGM)\"
\n\"3\": string \"Cassava Mosaic Disease (CMD)\"
\n\"4\": string \"Healthy\"
\n\"5\": string \"Unknown\"
if one participant uses the CropNet pre-trained model (which consist of the final output of 6 labels),
\ndid not change anything (I assume it is necessary to change the final output to 5 labels that match the competition requirement), fine-tune it and use it as the final model
\nfor generating prediction. is it an acceptable solution to this competition requirement?
@juliaelliott , please reply
\nThank you.
", "messages": [ { "id": "1211182", "postDate": "02/20/2021 03:19:00", "content": "Thank you Makerere University AI Lab and Kaggle for hosting this competition.
\nI would like to ask you for clarification regarding on Competition requirement about the
\nexpected number of labels that the model used for prediction.
Based on data requirement/specification, our expected labels (5 labels) are:
\n\"0\":string\"Cassava Bacterial Blight (CBB)\"
\n\"1\":string\"Cassava Brown Streak Disease (CBSD)\"
\n\"2\":string\"Cassava Green Mottle (CGM)\"
\n\"3\":string\"Cassava Mosaic Disease (CMD)\"
\n\"4\": string\"Healthy\"
Given that there's a previous research work related to this competition and have research output result posted in TensorFlow Hub: https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2. This pre-trained model was trained using 6 labels, namely:
\n\"0\":string \"Cassava Bacterial Blight (CBB)\"
\n\"1\": string \"Cassava Brown Streak Disease (CBSD)\"
\n\"2\": string \"Cassava Green Mottle (CGM)\"
\n\"3\": string \"Cassava Mosaic Disease (CMD)\"
\n\"4\": string \"Healthy\"
\n\"5\": string \"Unknown\"
if one participant uses the CropNet pre-trained model (which consist of the final output of 6 labels),
\ndid not change anything (I assume it is necessary to change the final output to 5 labels that match the competition requirement), fine-tune it and use it as the final model
\nfor generating prediction. is it an acceptable solution to this competition requirement?
@juliaelliott , please reply
\nThank you.
", "rawMarkdown": "Thank you Makerere University AI Lab and Kaggle for hosting this competition.\n\nI would like to ask you for clarification regarding on Competition requirement about the\nexpected number of labels that the model used for prediction.\n\nBased on data requirement/specification, our expected labels (5 labels) are: \n\n\"0\":string\"Cassava Bacterial Blight (CBB)\"\n\"1\":string\"Cassava Brown Streak Disease (CBSD)\"\n\"2\":string\"Cassava Green Mottle (CGM)\"\n\"3\":string\"Cassava Mosaic Disease (CMD)\"\n\"4\": string\"Healthy\"\n\nGiven that there's a previous research work related to this competition and have research output result posted in TensorFlow Hub: https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2. This pre-trained model was trained using 6 labels, namely:\n\n\"0\":string \"Cassava Bacterial Blight (CBB)\"\n\"1\": string \"Cassava Brown Streak Disease (CBSD)\"\n\"2\": string \"Cassava Green Mottle (CGM)\"\n\"3\": string \"Cassava Mosaic Disease (CMD)\"\n\"4\": string \"Healthy\"\n\"5\": string \"Unknown\"\n\nif one participant uses the CropNet pre-trained model (which consist of the final output of 6 labels), \ndid not change anything (I assume it is necessary to change the final output to 5 labels that match the competition requirement), fine-tune it and use it as the final model\nfor generating prediction. is it an acceptable solution to this competition requirement?\n\n@juliaelliott , please reply\n\nThank you.", "votes": null }, { "id": "1211298", "postDate": "02/20/2021 05:55:57", "content": "Isn't the 2nd place solution exactly the thing you are talking about ?
", "rawMarkdown": "Isn't the 2nd place solution exactly the thing you are talking about ?", "votes": null }, { "id": "1211632", "postDate": "02/20/2021 11:41:58", "content": "Why might this be unacceptable? (if the model is really trained using only 2019 data, and there are no leaks)
\nSince the metric is accuracy, you can assign any (even non-existent) label, e.g. if an expected label is 2 then it doesn't matter what you predicted - 3 or 999 (both are incorrect). Of course, there may be bugs in the validation code, but I doubt it.
It's interesting that the approach with \"unknown\" class probably helps… and that TFHub model works pretty well out of the box on private test data (0.888 private, no fine-tuning, no TTA, …)
\nI also just tried to modify 2nd place notebook ( https://www.kaggle.com/devonstanfield/cassava-infer ). I added post-processing to re-label 6th class predictions to other classes and got similar private score - 0.9043 … 0.9045, so looks like the model almost always predicts 1,2,3,4,5.
I also tried to fine-tune the TF Hub model using the shared notebook ( https://www.kaggle.com/devonstanfield/cassava-train ), but I cannot reproduce that 0.904 score (lucky seed?). I only got 0.8980 (which is also good) on a private test and post-processing (re-label) for 6th class much greater affects the results.
\nFYI @devonstanfield
", "rawMarkdown": "Why might this be unacceptable? (if the model is really trained using only 2019 data, and there are no leaks)\nSince the metric is accuracy, you can assign any (even non-existent) label, e.g. if an expected label is 2 then it doesn't matter what you predicted - 3 or 999 (both are incorrect). Of course, there may be bugs in the validation code, but I doubt it.\n\nIt's interesting that the approach with \"unknown\" class probably helps... and that TFHub model works pretty well out of the box on private test data (0.888 private, no fine-tuning, no TTA, ...)\nI also just tried to modify 2nd place notebook ( https://www.kaggle.com/devonstanfield/cassava-infer ). I added post-processing to re-label 6th class predictions to other classes and got similar private score - 0.9043 ... 0.9045, so looks like the model almost always predicts 1,2,3,4,5.\n\nI also tried to fine-tune the TF Hub model using the shared notebook ( https://www.kaggle.com/devonstanfield/cassava-train ), but I cannot reproduce that 0.904 score (lucky seed?). I only got 0.8980 (which is also good) on a private test and post-processing (re-label) for 6th class much greater affects the results.\n\nFYI @devonstanfield", "votes": null }, { "id": "1211706", "postDate": "02/20/2021 13:16:09", "content": "Yes, this is a lucky seed … trained again and now it gives 0.9025 private score:)
\nSo it looks like if you fine-tune this model more carefully, make an ensemble, add TTA, … you can probably get 0.91+ or higher at private LB… It's strange that no one tried that model, it gives good scores even on public LB.
I assume you have tried it with plain train/valid split (as devon did).. or kfolds ?
", "rawMarkdown": "I assume you have tried it with plain train/valid split (as devon did).. or kfolds ?", "votes": null }, { "id": "1211756", "postDate": "02/20/2021 14:12:48", "content": "Yes, I tried the notebook as it is, no changes. Just to get close results.
", "rawMarkdown": "Yes, I tried the notebook as it is, no changes. Just to get close results.", "votes": null }, { "id": "1211900", "postDate": "02/20/2021 16:36:46", "content": "I wouldn't be surprised if the first place solution is just that. i.e. the TF model and multiple models ensembeled with TTA.
", "rawMarkdown": "I wouldn't be surprised if the first place solution is just that. i.e. the TF model and multiple models ensembeled with TTA.", "votes": null }, { "id": "1211906", "postDate": "02/20/2021 16:39:48", "content": "I don't see why it should not be allowed. The model was public and trained on a different dataset so anyone could have used it and there is no unfair advantage.
", "rawMarkdown": "I don't see why it should not be allowed. The model was public and trained on a different dataset so anyone could have used it and there is no unfair advantage.", "votes": null }, { "id": "1211915", "postDate": "02/20/2021 16:55:31", "content": "Maybe, will see. I hope there is no shared data (leaks) between the dataset on which the tfhub model was trained and the private dataset, because I did not find training code and convincing evidence that they used only publicly available 2019 data.
", "rawMarkdown": "Maybe, will see. I hope there is no shared data (leaks) between the dataset on which the tfhub model was trained and the private dataset, because I did not find training code and convincing evidence that they used only publicly available 2019 data.", "votes": null }, { "id": "1212297", "postDate": "02/21/2021 04:55:00", "content": "This is for information from earlier discussion posts, not sure how/if i relates to this model but if it included 2019 data or inclass competition data. Just in case these were not seen here already.
\nDuplicates across 2019 and 2020 data, and potential leaks.
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206018
Upcoming rescore [complete]
\n\"I will be rescoring existing submissions to drop a subset of the images that were previously published in an inclass competition from the scores.\"
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803
Is the private leaderboard accessible?
\n\"I tried the method that has been discussed in the discussion.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207357
\nApparently, you can see the value of private score by following these steps.\"
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410
\nhttps://www.kaggle.com/private-leaderboard-bug-dec-2020
wrt rescore - The images are still in the test set, but aren't scored per that post. Presumably relates to the private LB too.
\nwrt private LB leak - this was noted in Dec and no mention was made from hosts changing the private test set after.
", "rawMarkdown": "This is for information from earlier discussion posts, not sure how/if i relates to this model but if it included 2019 data or inclass competition data. Just in case these were not seen here already.\n\nDuplicates across 2019 and 2020 data, and potential leaks.\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206018\n\nUpcoming rescore [complete]\n\"I will be rescoring existing submissions to drop a subset of the images that were previously published in an inclass competition from the scores.\"\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803\n\nIs the private leaderboard accessible?\n\"I tried the method that has been discussed in the discussion.\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207357\nApparently, you can see the value of private score by following these steps.\"\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410\nhttps://www.kaggle.com/private-leaderboard-bug-dec-2020\n\nwrt rescore - The images are still in the test set, but aren't scored per that post. Presumably relates to the private LB too.\n\nwrt private LB leak - this was noted in Dec and no mention was made from hosts changing the private test set after.", "votes": null } ], "comments": [ { "id": 1211298, "author_name": "zarif98sjs", "author_url": "", "post_date": "02/20/2021 05:55:57", "content": "Isn't the 2nd place solution exactly the thing you are talking about ?
", "votes": null, "replies": [] }, { "id": 1211632, "author_name": "sparakhin", "author_url": "", "post_date": "02/20/2021 11:41:58", "content": "Why might this be unacceptable? (if the model is really trained using only 2019 data, and there are no leaks)
\nSince the metric is accuracy, you can assign any (even non-existent) label, e.g. if an expected label is 2 then it doesn't matter what you predicted - 3 or 999 (both are incorrect). Of course, there may be bugs in the validation code, but I doubt it.
It's interesting that the approach with \"unknown\" class probably helps… and that TFHub model works pretty well out of the box on private test data (0.888 private, no fine-tuning, no TTA, …)
\nI also just tried to modify 2nd place notebook ( https://www.kaggle.com/devonstanfield/cassava-infer ). I added post-processing to re-label 6th class predictions to other classes and got similar private score - 0.9043 … 0.9045, so looks like the model almost always predicts 1,2,3,4,5.
I also tried to fine-tune the TF Hub model using the shared notebook ( https://www.kaggle.com/devonstanfield/cassava-train ), but I cannot reproduce that 0.904 score (lucky seed?). I only got 0.8980 (which is also good) on a private test and post-processing (re-label) for 6th class much greater affects the results.
\nFYI @devonstanfield
", "votes": null, "replies": [ { "id": 1211706, "author_name": "sparakhin", "author_url": "", "post_date": "02/20/2021 13:16:09", "content": "Yes, this is a lucky seed … trained again and now it gives 0.9025 private score:)
\nSo it looks like if you fine-tune this model more carefully, make an ensemble, add TTA, … you can probably get 0.91+ or higher at private LB… It's strange that no one tried that model, it gives good scores even on public LB.
I assume you have tried it with plain train/valid split (as devon did).. or kfolds ?
", "votes": null, "replies": [] } ] }, { "id": 1211756, "author_name": "sparakhin", "author_url": "", "post_date": "02/20/2021 14:12:48", "content": "Yes, I tried the notebook as it is, no changes. Just to get close results.
", "votes": null, "replies": [] }, { "id": 1211900, "author_name": "trushk", "author_url": "", "post_date": "02/20/2021 16:36:46", "content": "I wouldn't be surprised if the first place solution is just that. i.e. the TF model and multiple models ensembeled with TTA.
", "votes": null, "replies": [] }, { "id": 1211915, "author_name": "sparakhin", "author_url": "", "post_date": "02/20/2021 16:55:31", "content": "Maybe, will see. I hope there is no shared data (leaks) between the dataset on which the tfhub model was trained and the private dataset, because I did not find training code and convincing evidence that they used only publicly available 2019 data.
", "votes": null, "replies": [] } ] }, { "id": 1211906, "author_name": "trushk", "author_url": "", "post_date": "02/20/2021 16:39:48", "content": "I don't see why it should not be allowed. The model was public and trained on a different dataset so anyone could have used it and there is no unfair advantage.
", "votes": null, "replies": [] }, { "id": 1212297, "author_name": "something4kag", "author_url": "", "post_date": "02/21/2021 04:55:00", "content": "This is for information from earlier discussion posts, not sure how/if i relates to this model but if it included 2019 data or inclass competition data. Just in case these were not seen here already.
\nDuplicates across 2019 and 2020 data, and potential leaks.
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206018
Upcoming rescore [complete]
\n\"I will be rescoring existing submissions to drop a subset of the images that were previously published in an inclass competition from the scores.\"
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803
Is the private leaderboard accessible?
\n\"I tried the method that has been discussed in the discussion.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207357
\nApparently, you can see the value of private score by following these steps.\"
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410
\nhttps://www.kaggle.com/private-leaderboard-bug-dec-2020
wrt rescore - The images are still in the test set, but aren't scored per that post. Presumably relates to the private LB too.
\nwrt private LB leak - this was noted in Dec and no mention was made from hosts changing the private test set after.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1211182": "Thank you Makerere University AI Lab and Kaggle for hosting this competition.\n\nI would like to ask you for clarification regarding on Competition requirement about the\nexpected number of labels that the model used for prediction.\n\nBased on data requirement/specification, our expected labels (5 labels) are: \n\n\"0\":string\"Cassava Bacterial Blight (CBB)\"\n\"1\":string\"Cassava Brown Streak Disease (CBSD)\"\n\"2\":string\"Cassava Green Mottle (CGM)\"\n\"3\":string\"Cassava Mosaic Disease (CMD)\"\n\"4\": string\"Healthy\"\n\nGiven that there's a previous research work related to this competition and have research output result posted in TensorFlow Hub: https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2. This pre-trained model was trained using 6 labels, namely:\n\n\"0\":string \"Cassava Bacterial Blight (CBB)\"\n\"1\": string \"Cassava Brown Streak Disease (CBSD)\"\n\"2\": string \"Cassava Green Mottle (CGM)\"\n\"3\": string \"Cassava Mosaic Disease (CMD)\"\n\"4\": string \"Healthy\"\n\"5\": string \"Unknown\"\n\nif one participant uses the CropNet pre-trained model (which consist of the final output of 6 labels), \ndid not change anything (I assume it is necessary to change the final output to 5 labels that match the competition requirement), fine-tune it and use it as the final model\nfor generating prediction. is it an acceptable solution to this competition requirement?\n\n@juliaelliott , please reply\n\nThank you.", "1211298": "Isn't the 2nd place solution exactly the thing you are talking about ?", "1211632": "Why might this be unacceptable? (if the model is really trained using only 2019 data, and there are no leaks)\nSince the metric is accuracy, you can assign any (even non-existent) label, e.g. if an expected label is 2 then it doesn't matter what you predicted - 3 or 999 (both are incorrect). Of course, there may be bugs in the validation code, but I doubt it.\n\nIt's interesting that the approach with \"unknown\" class probably helps... and that TFHub model works pretty well out of the box on private test data (0.888 private, no fine-tuning, no TTA, ...)\nI also just tried to modify 2nd place notebook ( https://www.kaggle.com/devonstanfield/cassava-infer ). I added post-processing to re-label 6th class predictions to other classes and got similar private score - 0.9043 ... 0.9045, so looks like the model almost always predicts 1,2,3,4,5.\n\nI also tried to fine-tune the TF Hub model using the shared notebook ( https://www.kaggle.com/devonstanfield/cassava-train ), but I cannot reproduce that 0.904 score (lucky seed?). I only got 0.8980 (which is also good) on a private test and post-processing (re-label) for 6th class much greater affects the results.\n\nFYI @devonstanfield", "1211706": "Yes, this is a lucky seed ... trained again and now it gives 0.9025 private score:)\nSo it looks like if you fine-tune this model more carefully, make an ensemble, add TTA, ... you can probably get 0.91+ or higher at private LB... It's strange that no one tried that model, it gives good scores even on public LB.", "1211753": "I assume you have tried it with plain train/valid split (as devon did).. or kfolds ?", "1211756": "Yes, I tried the notebook as it is, no changes. Just to get close results.", "1211900": "I wouldn't be surprised if the first place solution is just that. i.e. the TF model and multiple models ensembeled with TTA.", "1211906": "I don't see why it should not be allowed. The model was public and trained on a different dataset so anyone could have used it and there is no unfair advantage.", "1211915": "Maybe, will see. I hope there is no shared data (leaks) between the dataset on which the tfhub model was trained and the private dataset, because I did not find training code and convincing evidence that they used only publicly available 2019 data.", "1212297": "This is for information from earlier discussion posts, not sure how/if i relates to this model but if it included 2019 data or inclass competition data. Just in case these were not seen here already.\n\nDuplicates across 2019 and 2020 data, and potential leaks.\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206018\n\nUpcoming rescore [complete]\n\"I will be rescoring existing submissions to drop a subset of the images that were previously published in an inclass competition from the scores.\"\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200803\n\nIs the private leaderboard accessible?\n\"I tried the method that has been discussed in the discussion.\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207357\nApparently, you can see the value of private score by following these steps.\"\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/207410\nhttps://www.kaggle.com/private-leaderboard-bug-dec-2020\n\nwrt rescore - The images are still in the test set, but aren't scored per that post. Presumably relates to the private LB too.\n\nwrt private LB leak - this was noted in Dec and no mention was made from hosts changing the private test set after." }, "source": "meta" }