As we know Cropnet has been one of the significant factor to be in a good position in this competition. I wanted to know do we have any information in which images the cropnet model was trained?
\nIf the images that were used to train cropnet were in the private dataset, is this a rule violation?
\nCan anyone from kaggle confirm this?
\nWhat are your views?
\nThank you.
\nModel link https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2
As we know Cropnet has been one of the significant factor to be in a good position in this competition. I wanted to know do we have any information in which images the cropnet model was trained?
\nIf the images that were used to train cropnet were in the private dataset, is this a rule violation?
\nCan anyone from kaggle confirm this?
\nWhat are your views?
\nThank you.
\nModel link https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2
It looks to be trained on Cassava only as per links below. Last link shows it is trained on concatenation of iNaturalist and plants subset of ImageNet-21K.
\noh fine then,
\nbut do we know if there is any intersection of this data and data given by kaggle for train/test?
There is lot of similarity based on sample images but not sure if there is exact overlap, would need to check that.
", "rawMarkdown": "There is lot of similarity based on sample images but not sure if there is exact overlap, would need to check that.", "votes": null }, { "id": "1218003", "postDate": "02/25/2021 13:32:59", "content": "there is exact overlap?, how we check that ?can u give me some example plz .
", "rawMarkdown": "there is exact overlap?, how we check that ?can u give me some example plz .", "votes": null }, { "id": "1218047", "postDate": "02/25/2021 14:18:58", "content": "Kaggle removes data from private LB but keeps data potentially used by TF pretrained model. Give me a break !
", "rawMarkdown": "Kaggle removes data from private LB but keeps data potentially used by TF pretrained model. Give me a break !", "votes": null }, { "id": "1218088", "postDate": "02/25/2021 14:56:29", "content": "I wonder why we don't have External Data thread anymore. If we had it, teams which use it had to share it. Then everyone could use it and the competition would be more fair. Any plans on getting External Data threads back @juliaelliott ?
", "rawMarkdown": "I wonder why we don't have External Data thread anymore. If we had it, teams which use it had to share it. Then everyone could use it and the competition would be more fair. Any plans on getting External Data threads back @juliaelliott ?", "votes": null }, { "id": "1218340", "postDate": "02/25/2021 18:24:52", "content": "Their tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.
", "rawMarkdown": "Their tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.", "votes": null }, { "id": "1218536", "postDate": "02/26/2021 00:57:44", "content": "\n\n\nTheir tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.
\n
Yes, that is true. But then, cropnets should score similar to models trained in cassava2019 data, which what most competitors tried (including myself). If we simply submit models trained with cassava2019, the score is much lower than the winning solutions (1st, 2nd).
\nThis indicates a chance of data leakage.. e.g. the cassava2019 test data is in this competition's test data as well..?
", "rawMarkdown": "sohier \n>Their tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.\n\nYes, that is true. But then, cropnets **should score similar to models trained in cassava2019 data**, which what most competitors tried (including myself). If we simply submit models trained with cassava2019, the score is much lower than the winning solutions (1st, 2nd).\n\nThis indicates a chance of data leakage.. e.g. the cassava2019 test data is in this competition's test data as well..?", "votes": null }, { "id": "1218562", "postDate": "02/26/2021 01:50:23", "content": "@aerdem4 - We made the decision to remove the official version of the thread, because our hosts generally do not have the capacity to monitor these threads, and it resulted in occasions where participants were left with a false sense of security that external data was appropriate to use if a host/admin did not say otherwise. As a result, we've moved away from an official thread and instead leave it to each participant to ensure what they use is in alignment with the competition's rules or request verification through tagging admins with clarification questions. Participants are welcome to create their own thread to share external data, but we are no longer requiring so.
", "rawMarkdown": "aerdem4 - We made the decision to remove the official version of the thread, because our hosts generally do not have the capacity to monitor these threads, and it resulted in occasions where participants were left with a false sense of security that external data was appropriate to use if a host/admin did not say otherwise. As a result, we've moved away from an official thread and instead leave it to each participant to ensure what they use is in alignment with the competition's rules or request verification through tagging admins with clarification questions. Participants are welcome to create their own thread to share external data, but we are no longer requiring so.", "votes": null }, { "id": "1218652", "postDate": "02/26/2021 04:56:42", "content": "@juliaelliott could you please reply to my clarification? it was posted in the discussion forum. Thank you
", "rawMarkdown": "juliaelliott could you please reply to my clarification? it was posted in the discussion forum. Thank you", "votes": null }, { "id": "1218677", "postDate": "02/26/2021 05:27:27", "content": "@aerdem4 - think External Data thread disappeared after Deepfake Detection Challenge where the first place team was removed with much controversy and discussion.
\nHad hoped that when Datasets Feature was added to attach to Competitions this would in some way fill the role External Data thread had. But alas this feature lasted only a few weeks and was removed. Perhaps for similar reasons as noted above.
\nhttps://www.kaggle.com/notifications/open/62565384?targetUrl=%2Fproduct-feedback%2F211372%231198100
\"If the images that were used to train cropnet were in the private dataset\" does seem a fair question but possibly needed to be raised during the competition?
\nRANZCR has similar questions about duplication with CHESTX.
\nHubMAP has private test data on the NIH HuBMAP data portal so now trying to get a new private test set. External data threads did serve a purpose as a central place for people to be aware of what leaks may exist, what is allowed with external data, etc.
@projdev I don't know that I've seen the clarification you're referring to. Can you tag me in it again or reply with the link?
", "rawMarkdown": "projdev I don't know that I've seen the clarification you're referring to. Can you tag me in it again or reply with the link?", "votes": null }, { "id": "1219589", "postDate": "02/27/2021 01:11:37", "content": "@juliaelliott : https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220904
", "rawMarkdown": "juliaelliott : https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220904", "votes": null }, { "id": "1225130", "postDate": "03/03/2021 10:43:47", "content": "thank you very much!
", "rawMarkdown": "thank you very much!", "votes": null }, { "id": "1225918", "postDate": "03/04/2021 04:28:49", "content": "@juliaelliott any updates?
", "rawMarkdown": "juliaelliott any updates?", "votes": null }, { "id": "1227924", "postDate": "03/06/2021 00:01:18", "content": "@projdev I'm not clear on what clarification is needed. Everyone is welcome to use the pre-trained model you referenced. This was indicated by others on the thread, as well. Also as Sohier clarified below, the data used in that pre-trained model was from the 2019 competition, using a different dataset than this competition. Yes, the labels associated with that pre-trained model include one additional class, but this competition had been defined with only 5. The existence of that pre-trained model does not change this competition or its outcomes.
", "rawMarkdown": "projdev I'm not clear on what clarification is needed. Everyone is welcome to use the pre-trained model you referenced. This was indicated by others on the thread, as well. Also as Sohier clarified below, the data used in that pre-trained model was from the 2019 competition, using a different dataset than this competition. Yes, the labels associated with that pre-trained model include one additional class, but this competition had been defined with only 5. The existence of that pre-trained model does not change this competition or its outcomes.", "votes": null } ], "comments": [ { "id": 1217772, "author_name": "saurabhbagchi", "author_url": "", "post_date": "02/25/2021 10:02:21", "content": "It looks to be trained on Cassava only as per links below. Last link shows it is trained on concatenation of iNaturalist and plants subset of ImageNet-21K.
\noh fine then,
\nbut do we know if there is any intersection of this data and data given by kaggle for train/test?
There is lot of similarity based on sample images but not sure if there is exact overlap, would need to check that.
", "votes": null, "replies": [] }, { "id": 1218003, "author_name": "maozragab", "author_url": "", "post_date": "02/25/2021 13:32:59", "content": "there is exact overlap?, how we check that ?can u give me some example plz .
", "votes": null, "replies": [] }, { "id": 1218340, "author_name": "sohier", "author_url": "", "post_date": "02/25/2021 18:24:52", "content": "Their tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.
", "votes": null, "replies": [] }, { "id": 1218536, "author_name": "kyoshioka47", "author_url": "", "post_date": "02/26/2021 00:57:44", "content": "\n\n\nTheir tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.
\n
Yes, that is true. But then, cropnets should score similar to models trained in cassava2019 data, which what most competitors tried (including myself). If we simply submit models trained with cassava2019, the score is much lower than the winning solutions (1st, 2nd).
\nThis indicates a chance of data leakage.. e.g. the cassava2019 test data is in this competition's test data as well..?
", "votes": null, "replies": [] } ] }, { "id": 1218047, "author_name": "serigne", "author_url": "", "post_date": "02/25/2021 14:18:58", "content": "Kaggle removes data from private LB but keeps data potentially used by TF pretrained model. Give me a break !
", "votes": null, "replies": [] }, { "id": 1218088, "author_name": "aerdem4", "author_url": "", "post_date": "02/25/2021 14:56:29", "content": "I wonder why we don't have External Data thread anymore. If we had it, teams which use it had to share it. Then everyone could use it and the competition would be more fair. Any plans on getting External Data threads back @juliaelliott ?
", "votes": null, "replies": [ { "id": 1218562, "author_name": "juliaelliott", "author_url": "", "post_date": "02/26/2021 01:50:23", "content": "@aerdem4 - We made the decision to remove the official version of the thread, because our hosts generally do not have the capacity to monitor these threads, and it resulted in occasions where participants were left with a false sense of security that external data was appropriate to use if a host/admin did not say otherwise. As a result, we've moved away from an official thread and instead leave it to each participant to ensure what they use is in alignment with the competition's rules or request verification through tagging admins with clarification questions. Participants are welcome to create their own thread to share external data, but we are no longer requiring so.
", "votes": null, "replies": [] }, { "id": 1218652, "author_name": "projdev", "author_url": "", "post_date": "02/26/2021 04:56:42", "content": "@juliaelliott could you please reply to my clarification? it was posted in the discussion forum. Thank you
", "votes": null, "replies": [] }, { "id": 1218677, "author_name": "something4kag", "author_url": "", "post_date": "02/26/2021 05:27:27", "content": "@aerdem4 - think External Data thread disappeared after Deepfake Detection Challenge where the first place team was removed with much controversy and discussion.
\nHad hoped that when Datasets Feature was added to attach to Competitions this would in some way fill the role External Data thread had. But alas this feature lasted only a few weeks and was removed. Perhaps for similar reasons as noted above.
\nhttps://www.kaggle.com/notifications/open/62565384?targetUrl=%2Fproduct-feedback%2F211372%231198100
\"If the images that were used to train cropnet were in the private dataset\" does seem a fair question but possibly needed to be raised during the competition?
\nRANZCR has similar questions about duplication with CHESTX.
\nHubMAP has private test data on the NIH HuBMAP data portal so now trying to get a new private test set. External data threads did serve a purpose as a central place for people to be aware of what leaks may exist, what is allowed with external data, etc.
@projdev I don't know that I've seen the clarification you're referring to. Can you tag me in it again or reply with the link?
", "votes": null, "replies": [] }, { "id": 1219589, "author_name": "projdev", "author_url": "", "post_date": "02/27/2021 01:11:37", "content": "@juliaelliott : https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220904
", "votes": null, "replies": [] }, { "id": 1225918, "author_name": "projdev", "author_url": "", "post_date": "03/04/2021 04:28:49", "content": "@juliaelliott any updates?
", "votes": null, "replies": [] }, { "id": 1227924, "author_name": "juliaelliott", "author_url": "", "post_date": "03/06/2021 00:01:18", "content": "@projdev I'm not clear on what clarification is needed. Everyone is welcome to use the pre-trained model you referenced. This was indicated by others on the thread, as well. Also as Sohier clarified below, the data used in that pre-trained model was from the 2019 competition, using a different dataset than this competition. Yes, the labels associated with that pre-trained model include one additional class, but this competition had been defined with only 5. The existence of that pre-trained model does not change this competition or its outcomes.
", "votes": null, "replies": [] } ] }, { "id": 1225130, "author_name": "tesssssssss", "author_url": "", "post_date": "03/03/2021 10:43:47", "content": "thank you very much!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1217426": "As we know Cropnet has been one of the significant factor to be in a good position in this competition. I wanted to know do we have any information in which images the cropnet model was trained?\nIf the images that were used to train cropnet were in the private dataset, is this a rule violation?\nCan anyone from kaggle confirm this?\nWhat are your views?\nThank you.\nModel link https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2", "1217772": "It looks to be trained on Cassava only as per links below. Last link shows it is trained on concatenation of iNaturalist and plants subset of ImageNet-21K.\n\n- https://www.tensorflow.org/hub/tutorials/cropnet_cassava\n- https://aihub.cloud.google.com/u/0/p/products%2F06dbdbde-3919-45ee-abc2-2a2c15460d58\n- https://tfhub.dev/google/cropnet/feature_vector/concat/1", "1217783": "oh fine then,\nbut do we know if there is any intersection of this data and data given by kaggle for train/test?", "1217893": "There is lot of similarity based on sample images but not sure if there is exact overlap, would need to check that.", "1218003": "there is exact overlap?, how we check that ?can u give me some example plz .", "1218047": "Kaggle removes data from private LB but keeps data potentially used by TF pretrained model. Give me a break !", "1218088": "I wonder why we don't have External Data thread anymore. If we had it, teams which use it had to share it. Then everyone could use it and the competition would be more fair. Any plans on getting External Data threads back @juliaelliott ?", "1218340": "Their tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.", "1218536": "sohier \n>Their tensorflow hub page points to https://www.kaggle.com/c/cassava-disease/overview as the data source, which is the previous competition from two years ago.\n\nYes, that is true. But then, cropnets **should score similar to models trained in cassava2019 data**, which what most competitors tried (including myself). If we simply submit models trained with cassava2019, the score is much lower than the winning solutions (1st, 2nd).\n\nThis indicates a chance of data leakage.. e.g. the cassava2019 test data is in this competition's test data as well..?", "1218562": "aerdem4 - We made the decision to remove the official version of the thread, because our hosts generally do not have the capacity to monitor these threads, and it resulted in occasions where participants were left with a false sense of security that external data was appropriate to use if a host/admin did not say otherwise. As a result, we've moved away from an official thread and instead leave it to each participant to ensure what they use is in alignment with the competition's rules or request verification through tagging admins with clarification questions. Participants are welcome to create their own thread to share external data, but we are no longer requiring so.", "1218652": "juliaelliott could you please reply to my clarification? it was posted in the discussion forum. Thank you", "1218677": "aerdem4 - think External Data thread disappeared after Deepfake Detection Challenge where the first place team was removed with much controversy and discussion. \nHad hoped that when Datasets Feature was added to attach to Competitions this would in some way fill the role External Data thread had. But alas this feature lasted only a few weeks and was removed. Perhaps for similar reasons as noted above. \nhttps://www.kaggle.com/notifications/open/62565384?targetUrl=%2Fproduct-feedback%2F211372%231198100 \n\n\"If the images that were used to train cropnet were in the private dataset\" does seem a fair question but possibly needed to be raised during the competition? \n\nRANZCR has similar questions about duplication with CHESTX.\nHubMAP has private test data on the NIH HuBMAP data portal so now trying to get a new private test set. External data threads did serve a purpose as a central place for people to be aware of what leaks may exist, what is allowed with external data, etc.", "1219501": "projdev I don't know that I've seen the clarification you're referring to. Can you tag me in it again or reply with the link?", "1219589": "juliaelliott : https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220904", "1225130": "thank you very much!", "1225918": "juliaelliott any updates?", "1227924": "projdev I'm not clear on what clarification is needed. Everyone is welcome to use the pre-trained model you referenced. This was indicated by others on the thread, as well. Also as Sohier clarified below, the data used in that pre-trained model was from the 2019 competition, using a different dataset than this competition. Yes, the labels associated with that pre-trained model include one additional class, but this competition had been defined with only 5. The existence of that pre-trained model does not change this competition or its outcomes." }, "source": "meta" }