Up to now, my best single model gets a cv score around 0.92. Since cv is more reliable than lb, I'm curious about the highest cv score in your experiments. Anyone gets a score >= 0.93 ?
", "messages": [ { "id": "1310312", "postDate": "05/16/2021 15:35:01", "content": "Up to now, my best single model gets a cv score around 0.92. Since cv is more reliable than lb, I'm curious about the highest cv score in your experiments. Anyone gets a score >= 0.93 ?
", "rawMarkdown": "Up to now, my best single model gets a cv score around 0.92. Since cv is more reliable than lb, I'm curious about the highest cv score in your experiments. Anyone gets a score >= 0.93 ?", "votes": null }, { "id": "1310600", "postDate": "05/16/2021 18:46:16", "content": "Your CV Score means Macro F1-Score?
", "rawMarkdown": "Your CV Score means Macro F1-Score?", "votes": null }, { "id": "1310609", "postDate": "05/16/2021 18:55:32", "content": "Single model - do you mean single fold, or averaged multi-fold?
", "rawMarkdown": "Single model - do you mean single fold, or averaged multi-fold?", "votes": null }, { "id": "1310808", "postDate": "05/17/2021 01:55:37", "content": "5 folds effecientnet_b5_ns
", "rawMarkdown": "5 folds effecientnet_b5_ns", "votes": null }, { "id": "1310811", "postDate": "05/17/2021 01:56:41", "content": "Mean f1-score, which is averaged over f1 score of all categories.
", "rawMarkdown": "Mean f1-score, which is averaged over f1 score of all categories.", "votes": null }, { "id": "1310816", "postDate": "05/17/2021 02:01:18", "content": "I got around 0.92 for EfficientNetB7 too.
\nYou can test the model's generalization on the last year's dataset.
\nIf posible, you can share the results here.
Seems that it's hard to get a score above 0.93.
", "rawMarkdown": "Seems that it's hard to get a score above 0.93.", "votes": null }, { "id": "1310827", "postDate": "05/17/2021 02:14:15", "content": "Maybe the noisy labels is an important reason.
\nAnd I found the model cannot find an accurate boundary between 'scab' and 'rust'.
My max CV is 0.9377 for EfficientNetB5 (Tensorflow, 5 folds averaged).
", "rawMarkdown": "My max CV is 0.9377 for EfficientNetB5 (Tensorflow, 5 folds averaged).", "votes": null }, { "id": "1311267", "postDate": "05/17/2021 09:24:51", "content": "Wow, that's really impressive.
", "rawMarkdown": "Wow, that's really impressive.", "votes": null }, { "id": "1311302", "postDate": "05/17/2021 09:57:13", "content": "I'm using tfa.metrics.F1Score(len(CFG.CLASSES), average='weighted') if you're interested.
Thanks for your advice.
", "rawMarkdown": "Thanks for your advice.", "votes": null }, { "id": "1311306", "postDate": "05/17/2021 10:03:17", "content": "complex label detection is also a problem.
hi, the second place you achieve is so impressive.I also use the efficientnet but the score in cv is 0.826 and in lb is 0.819.Can i know what the resolution in your network is?
", "rawMarkdown": "hi, the second place you achieve is so impressive.I also use the efficientnet but the score in cv is 0.826 and in lb is 0.819.Can i know what the resolution in your network is?", "votes": null }, { "id": "1311348", "postDate": "05/17/2021 10:44:15", "content": "456 * 456 (Using pretrained model from timm: https://github.com/rwightman/pytorch-image-models), you can try different image size and see which one performs best. BTW, I think we should focus more on cv instead of lb.
", "rawMarkdown": "456 * 456 (Using pretrained model from timm: https://github.com/rwightman/pytorch-image-models), you can try different image size and see which one performs best. BTW, I think we should focus more on cv instead of lb.", "votes": null }, { "id": "1311356", "postDate": "05/17/2021 10:50:37", "content": "thx very much! yeah we should more focus on cv score.My question is that i have tried the 528x528 resolution in efficientnetb6(which is the recommended resolution from the tutorial), but the result is very bad. I dont know why ,because 224x224 is a really small resolution
", "rawMarkdown": "thx very much! yeah we should more focus on cv score.My question is that i have tried the 528x528 resolution in efficientnetb6(which is the recommended resolution from the tutorial), but the result is very bad. I dont know why ,because 224x224 is a really small resolution", "votes": null }, { "id": "1311359", "postDate": "05/17/2021 10:59:46", "content": "I'm using 768x768.
", "rawMarkdown": "I'm using 768x768.", "votes": null }, { "id": "1311365", "postDate": "05/17/2021 11:04:14", "content": "thanks for your reply!
", "rawMarkdown": "thanks for your reply!", "votes": null }, { "id": "1311441", "postDate": "05/17/2021 12:01:04", "content": "Sometimes, choosing model is very tricky. Some models with more parameters just may not fit this problem. You need to try multiple different models to find the best one.
", "rawMarkdown": "Sometimes, choosing model is very tricky. Some models with more parameters just may not fit this problem. You need to try multiple different models to find the best one.", "votes": null }, { "id": "1311746", "postDate": "05/17/2021 15:27:00", "content": "thx so much！！
", "rawMarkdown": "thx so much！！", "votes": null }, { "id": "1311810", "postDate": "05/17/2021 16:17:02", "content": "What's really strange is that my model ensemble with CV ~0.956 scores on LB as 0.835. Maybe it's because of low percentage of data evaluated on public LB (just 18%).
", "rawMarkdown": "What's really strange is that my model ensemble with CV ~0.956 scores on LB as 0.835. Maybe it's because of low percentage of data evaluated on public LB (just 18%).", "votes": null }, { "id": "1311863", "postDate": "05/17/2021 16:55:42", "content": "@atamazian is there any reason you are using average='weighted' instead of using average='micro' for the evaluation metric?
Well, the training dataset is imbalanced, so I thought that 'weighted' option will give more accurate score.
@ytepzhi I still don't get you, which f1 score average method did you use?
\naverage = None, macro, micro or weighted?
As per @atamazian, 'weighted' would be a better metric.
\n@brendanartley is using 'micro'
Weighted would take into consideration the class imbalance.
\nMacro would not bother about class imbalance. It's simple averaging.
\nMicro would ignore the rare samples.
Simple averaging without weighting.
", "rawMarkdown": "Simple averaging without weighting.", "votes": null }, { "id": "1312369", "postDate": "05/18/2021 02:41:35", "content": "You should use sklearn.metrics.f1_score('samples') or another function like this.
", "rawMarkdown": "You should use sklearn.metrics.f1_score('samples') or another function like this.", "votes": null }, { "id": "1312450", "postDate": "05/18/2021 04:26:02", "content": "Most of my CV score are around 0.89~0.92. I also use the efficientNet model from timm. B.T.W why did you choose the resolution of the input image to be 456 * 456?
\nMany thanks!
Hi @ytepzhi .
\nMy CV is around 0.928 to 0.93 but LB is 0.807.
\nI don't know why.
\nIs there any trick that everyone else is using like setting different thresholds for complex etc.
I think that public LB score is not representative since it uses only 19% of data for evaluation.
", "rawMarkdown": "I think that public LB score is not representative since it uses only 19% of data for evaluation.", "votes": null }, { "id": "1320225", "postDate": "05/23/2021 21:57:36", "content": "Yes. But I was just kind of confused that if i am missing something or not.
", "rawMarkdown": "Yes. But I was just kind of confused that if i am missing something or not.", "votes": null }, { "id": "1320423", "postDate": "05/24/2021 04:46:16", "content": "Choosing threshold is important for this competition, I'm currently using threshold 0.4 for all categories. If probability(label A) > 0.4, A is in the prediction.
", "rawMarkdown": "Choosing threshold is important for this competition, I'm currently using threshold 0.4 for all categories. If probability(label A) > 0.4, A is in the prediction.", "votes": null }, { "id": "1320429", "postDate": "05/24/2021 04:49:03", "content": "Exactly ! 19 % test data is so unreliable.
", "rawMarkdown": "Exactly ! 19 % test data is so unreliable.", "votes": null }, { "id": "1320995", "postDate": "05/24/2021 13:18:58", "content": "I have a question. How are you people cross-validating and experimenting at such a speed? For Efficientnetb4 using resized images, it is taking too much time. How much time is it taking you for 5 folds?
", "rawMarkdown": "I have a question. How are you people cross-validating and experimenting at such a speed? For Efficientnetb4 using resized images, it is taking too much time. How much time is it taking you for 5 folds?", "votes": null }, { "id": "1321037", "postDate": "05/24/2021 13:44:29", "content": "@ytepzhi You are using a multi-label approach, right? 0.40 was tuned across all the folds or it's test and trial? Are you using healthy class?
", "rawMarkdown": "ytepzhi You are using a multi-label approach, right? 0.40 was tuned across all the folds or it's test and trial? Are you using healthy class?", "votes": null }, { "id": "1321038", "postDate": "05/24/2021 13:44:36", "content": "TF, 5 folds, EfficientNetB5, preaugmented TFRecords takes ~ 5 hours.
", "rawMarkdown": "TF, 5 folds, EfficientNetB5, preaugmented TFRecords takes ~ 5 hours.", "votes": null }, { "id": "1321069", "postDate": "05/24/2021 14:00:11", "content": "Thanks. Is pre augmented the right approach? I mean for every epoch we should have unique augmentations. 5 hours is a good time. I think TPU is the key here to make things faster?
", "rawMarkdown": "Thanks. Is pre augmented the right approach? I mean for every epoch we should have unique augmentations. 5 hours is a good time. I think TPU is the key here to make things faster?", "votes": null }, { "id": "1321150", "postDate": "05/24/2021 14:19:31", "content": "My CV is around 0.95 but LB is stuck at 0.807
\nThreshold is 0.5
Check this notebook for more info. And yes, TPU greatly speeds up the training.
", "rawMarkdown": "Check [this notebook](https://www.kaggle.com/nickuzmenkov/pp2021-ultimate-preprocessing) for more info. And yes, TPU greatly speeds up the training.", "votes": null }, { "id": "1323032", "postDate": "05/25/2021 22:20:17", "content": "My 5-fold CV is slightly north of 0.93 with F1-macro and threshold=0.5. Corresponding LB which is based on an ensemble of these 5 models with optimized thresholds, such that each fold and class-label has its own threshold value, gave me 0.826 on public LB.
", "rawMarkdown": "My 5-fold CV is slightly north of 0.93 with F1-macro and threshold=0.5. Corresponding LB which is based on an ensemble of these 5 models with optimized thresholds, such that each fold and class-label has its own threshold value, gave me 0.826 on public LB.", "votes": null }, { "id": "1327218", "postDate": "05/29/2021 04:44:27", "content": "Did you guys used prescribed image size by keras for the respective models for efficientnet or was it different ??
", "rawMarkdown": "Did you guys used prescribed image size by keras for the respective models for efficientnet or was it different ??", "votes": null }, { "id": "1327531", "postDate": "05/29/2021 11:50:47", "content": "I used torch, the images are resized in different shapes for different models.
", "rawMarkdown": "I used torch, the images are resized in different shapes for different models.", "votes": null } ], "comments": [ { "id": 1310600, "author_name": "rainyq", "author_url": "", "post_date": "05/16/2021 18:46:16", "content": "Your CV Score means Macro F1-Score?
", "votes": null, "replies": [ { "id": 1310811, "author_name": "ytepzhi", "author_url": "", "post_date": "05/17/2021 01:56:41", "content": "Mean f1-score, which is averaged over f1 score of all categories.
", "votes": null, "replies": [] }, { "id": 1310816, "author_name": "rainyq", "author_url": "", "post_date": "05/17/2021 02:01:18", "content": "I got around 0.92 for EfficientNetB7 too.
\nYou can test the model's generalization on the last year's dataset.
\nIf posible, you can share the results here.
Seems that it's hard to get a score above 0.93.
", "votes": null, "replies": [] }, { "id": 1310827, "author_name": "rainyq", "author_url": "", "post_date": "05/17/2021 02:14:15", "content": "Maybe the noisy labels is an important reason.
\nAnd I found the model cannot find an accurate boundary between 'scab' and 'rust'.
complex label detection is also a problem.
@ytepzhi I still don't get you, which f1 score average method did you use?
\naverage = None, macro, micro or weighted?
As per @atamazian, 'weighted' would be a better metric.
\n@brendanartley is using 'micro'
Weighted would take into consideration the class imbalance.
\nMacro would not bother about class imbalance. It's simple averaging.
\nMicro would ignore the rare samples.
Simple averaging without weighting.
", "votes": null, "replies": [] }, { "id": 1312369, "author_name": "rainyq", "author_url": "", "post_date": "05/18/2021 02:41:35", "content": "You should use sklearn.metrics.f1_score('samples') or another function like this.
", "votes": null, "replies": [] } ] }, { "id": 1310609, "author_name": "atamazian", "author_url": "", "post_date": "05/16/2021 18:55:32", "content": "Single model - do you mean single fold, or averaged multi-fold?
", "votes": null, "replies": [ { "id": 1310808, "author_name": "ytepzhi", "author_url": "", "post_date": "05/17/2021 01:55:37", "content": "5 folds effecientnet_b5_ns
", "votes": null, "replies": [] }, { "id": 1311253, "author_name": "atamazian", "author_url": "", "post_date": "05/17/2021 09:12:35", "content": "My max CV is 0.9377 for EfficientNetB5 (Tensorflow, 5 folds averaged).
", "votes": null, "replies": [] }, { "id": 1311267, "author_name": "ytepzhi", "author_url": "", "post_date": "05/17/2021 09:24:51", "content": "Wow, that's really impressive.
", "votes": null, "replies": [] }, { "id": 1311302, "author_name": "atamazian", "author_url": "", "post_date": "05/17/2021 09:57:13", "content": "I'm using tfa.metrics.F1Score(len(CFG.CLASSES), average='weighted') if you're interested.
Thanks for your advice.
", "votes": null, "replies": [] }, { "id": 1311810, "author_name": "atamazian", "author_url": "", "post_date": "05/17/2021 16:17:02", "content": "What's really strange is that my model ensemble with CV ~0.956 scores on LB as 0.835. Maybe it's because of low percentage of data evaluated on public LB (just 18%).
", "votes": null, "replies": [] }, { "id": 1311863, "author_name": "brendanartley", "author_url": "", "post_date": "05/17/2021 16:55:42", "content": "@atamazian is there any reason you are using average='weighted' instead of using average='micro' for the evaluation metric?
Well, the training dataset is imbalanced, so I thought that 'weighted' option will give more accurate score.
hi, the second place you achieve is so impressive.I also use the efficientnet but the score in cv is 0.826 and in lb is 0.819.Can i know what the resolution in your network is?
", "votes": null, "replies": [ { "id": 1311348, "author_name": "ytepzhi", "author_url": "", "post_date": "05/17/2021 10:44:15", "content": "456 * 456 (Using pretrained model from timm: https://github.com/rwightman/pytorch-image-models), you can try different image size and see which one performs best. BTW, I think we should focus more on cv instead of lb.
", "votes": null, "replies": [] }, { "id": 1311356, "author_name": "kagglezzr", "author_url": "", "post_date": "05/17/2021 10:50:37", "content": "thx very much! yeah we should more focus on cv score.My question is that i have tried the 528x528 resolution in efficientnetb6(which is the recommended resolution from the tutorial), but the result is very bad. I dont know why ,because 224x224 is a really small resolution
", "votes": null, "replies": [] }, { "id": 1311359, "author_name": "atamazian", "author_url": "", "post_date": "05/17/2021 10:59:46", "content": "I'm using 768x768.
", "votes": null, "replies": [] }, { "id": 1311365, "author_name": "kagglezzr", "author_url": "", "post_date": "05/17/2021 11:04:14", "content": "thanks for your reply!
", "votes": null, "replies": [] }, { "id": 1311441, "author_name": "ytepzhi", "author_url": "", "post_date": "05/17/2021 12:01:04", "content": "Sometimes, choosing model is very tricky. Some models with more parameters just may not fit this problem. You need to try multiple different models to find the best one.
", "votes": null, "replies": [] }, { "id": 1311746, "author_name": "kagglezzr", "author_url": "", "post_date": "05/17/2021 15:27:00", "content": "thx so much！！
", "votes": null, "replies": [] } ] }, { "id": 1312450, "author_name": "crissallan", "author_url": "", "post_date": "05/18/2021 04:26:02", "content": "Most of my CV score are around 0.89~0.92. I also use the efficientNet model from timm. B.T.W why did you choose the resolution of the input image to be 456 * 456?
\nMany thanks!
Hi @ytepzhi .
\nMy CV is around 0.928 to 0.93 but LB is 0.807.
\nI don't know why.
\nIs there any trick that everyone else is using like setting different thresholds for complex etc.
I think that public LB score is not representative since it uses only 19% of data for evaluation.
", "votes": null, "replies": [] }, { "id": 1320225, "author_name": "micheomaano", "author_url": "", "post_date": "05/23/2021 21:57:36", "content": "Yes. But I was just kind of confused that if i am missing something or not.
", "votes": null, "replies": [] }, { "id": 1320423, "author_name": "ytepzhi", "author_url": "", "post_date": "05/24/2021 04:46:16", "content": "Choosing threshold is important for this competition, I'm currently using threshold 0.4 for all categories. If probability(label A) > 0.4, A is in the prediction.
", "votes": null, "replies": [] }, { "id": 1320429, "author_name": "ytepzhi", "author_url": "", "post_date": "05/24/2021 04:49:03", "content": "Exactly ! 19 % test data is so unreliable.
", "votes": null, "replies": [] }, { "id": 1321037, "author_name": "deepdreamx", "author_url": "", "post_date": "05/24/2021 13:44:29", "content": "@ytepzhi You are using a multi-label approach, right? 0.40 was tuned across all the folds or it's test and trial? Are you using healthy class?
", "votes": null, "replies": [] } ] }, { "id": 1320995, "author_name": "deepdreamx", "author_url": "", "post_date": "05/24/2021 13:18:58", "content": "I have a question. How are you people cross-validating and experimenting at such a speed? For Efficientnetb4 using resized images, it is taking too much time. How much time is it taking you for 5 folds?
", "votes": null, "replies": [ { "id": 1321038, "author_name": "atamazian", "author_url": "", "post_date": "05/24/2021 13:44:36", "content": "TF, 5 folds, EfficientNetB5, preaugmented TFRecords takes ~ 5 hours.
", "votes": null, "replies": [] }, { "id": 1321069, "author_name": "deepdreamx", "author_url": "", "post_date": "05/24/2021 14:00:11", "content": "Thanks. Is pre augmented the right approach? I mean for every epoch we should have unique augmentations. 5 hours is a good time. I think TPU is the key here to make things faster?
", "votes": null, "replies": [] }, { "id": 1321164, "author_name": "atamazian", "author_url": "", "post_date": "05/24/2021 14:28:03", "content": "Check this notebook for more info. And yes, TPU greatly speeds up the training.
", "votes": null, "replies": [] } ] }, { "id": 1321150, "author_name": "micheomaano", "author_url": "", "post_date": "05/24/2021 14:19:31", "content": "My CV is around 0.95 but LB is stuck at 0.807
\nThreshold is 0.5
My 5-fold CV is slightly north of 0.93 with F1-macro and threshold=0.5. Corresponding LB which is based on an ensemble of these 5 models with optimized thresholds, such that each fold and class-label has its own threshold value, gave me 0.826 on public LB.
", "votes": null, "replies": [] }, { "id": 1327218, "author_name": "jl18pg052", "author_url": "", "post_date": "05/29/2021 04:44:27", "content": "Did you guys used prescribed image size by keras for the respective models for efficientnet or was it different ??
", "votes": null, "replies": [ { "id": 1327531, "author_name": "ytepzhi", "author_url": "", "post_date": "05/29/2021 11:50:47", "content": "I used torch, the images are resized in different shapes for different models.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1310312": "Up to now, my best single model gets a cv score around 0.92. Since cv is more reliable than lb, I'm curious about the highest cv score in your experiments. Anyone gets a score >= 0.93 ?", "1310600": "Your CV Score means Macro F1-Score?", "1310609": "Single model - do you mean single fold, or averaged multi-fold?", "1310808": "5 folds effecientnet_b5_ns", "1310811": "Mean f1-score, which is averaged over f1 score of all categories.", "1310816": "I got around 0.92 for EfficientNetB7 too.\nYou can test the model's generalization on the last year's dataset.\nIf posible, you can share the results here.", "1310822": "Seems that it's hard to get a score above 0.93.", "1310827": "Maybe the noisy labels is an important reason.\nAnd I found the model cannot find an accurate boundary between 'scab' and 'rust'.", "1311253": "My max CV is 0.9377 for EfficientNetB5 (Tensorflow, 5 folds averaged).", "1311267": "Wow, that's really impressive.", "1311302": "I'm using `tfa.metrics.F1Score(len(CFG.CLASSES), average='weighted')` if you're interested.", "1311305": "Thanks for your advice.", "1311306": "`complex` label detection is also a problem.", "1311336": "hi, the second place you achieve is so impressive.I also use the efficientnet but the score in cv is 0.826 and in lb is 0.819.Can i know what the resolution in your network is?", "1311348": "456 * 456 (Using pretrained model from timm: https://github.com/rwightman/pytorch-image-models), you can try different image size and see which one performs best. BTW, I think we should focus more on cv instead of lb.", "1311356": "thx very much! yeah we should more focus on cv score.My question is that i have tried the 528x528 resolution in efficientnetb6(which is the recommended resolution from the tutorial), but the result is very bad. I dont know why ,because 224x224 is a really small resolution", "1311359": "I'm using 768x768.", "1311365": "thanks for your reply!", "1311441": "Sometimes, choosing model is very tricky. Some models with more parameters just may not fit this problem. You need to try multiple different models to find the best one.", "1311746": "thx so much！！", "1311810": "What's really strange is that my model ensemble with CV ~0.956 scores on LB as 0.835. Maybe it's because of low percentage of data evaluated on public LB (just 18%).", "1311863": "atamazian is there any reason you are using `average='weighted'` instead of using `average='micro'` for the evaluation metric?", "1311936": "Well, the training dataset is imbalanced, so I thought that `'weighted'` option will give more accurate score.", "1312359": "ytepzhi I still don't get you, which f1 score average method did you use? \naverage = None, macro, micro or weighted?\n\n\nAs per @atamazian, 'weighted' would be a better metric. \n@brendanartley is using 'micro'\n\nWeighted would take into consideration the class imbalance. \nMacro would not bother about class imbalance. It's simple averaging.\nMicro would ignore the rare samples.", "1312362": "Simple averaging without weighting.", "1312369": "You should use sklearn.metrics.f1_score('samples') or another function like this.", "1312450": "Most of my CV score are around 0.89~0.92. I also use the efficientNet model from timm. B.T.W why did you choose the resolution of the input image to be 456 * 456? \nMany thanks!", "1320218": "Hi @ytepzhi .\nMy CV is around 0.928 to 0.93 but LB is 0.807.\nI don't know why.\nIs there any trick that everyone else is using like setting different thresholds for complex etc.", "1320222": "I think that public LB score is not representative since it uses only 19% of data for evaluation.", "1320225": "Yes. But I was just kind of confused that if i am missing something or not.", "1320423": "Choosing threshold is important for this competition, I'm currently using threshold 0.4 for all categories. If probability(label A) > 0.4, A is in the prediction.", "1320429": "Exactly ! 19 % test data is so unreliable.", "1320995": "I have a question. How are you people cross-validating and experimenting at such a speed? For Efficientnetb4 using resized images, it is taking too much time. How much time is it taking you for 5 folds?", "1321037": "ytepzhi You are using a multi-label approach, right? 0.40 was tuned across all the folds or it's test and trial? Are you using healthy class?", "1321038": "TF, 5 folds, EfficientNetB5, preaugmented TFRecords takes ~ 5 hours.", "1321069": "Thanks. Is pre augmented the right approach? I mean for every epoch we should have unique augmentations. 5 hours is a good time. I think TPU is the key here to make things faster?", "1321150": "My CV is around 0.95 but LB is stuck at 0.807\nThreshold is 0.5", "1321164": "Check [this notebook](https://www.kaggle.com/nickuzmenkov/pp2021-ultimate-preprocessing) for more info. And yes, TPU greatly speeds up the training.", "1323032": "My 5-fold CV is slightly north of 0.93 with F1-macro and threshold=0.5. Corresponding LB which is based on an ensemble of these 5 models with optimized thresholds, such that each fold and class-label has its own threshold value, gave me 0.826 on public LB.", "1327218": "Did you guys used prescribed image size by keras for the respective models for efficientnet or was it different ??", "1327531": "I used torch, the images are resized in different shapes for different models." }, "source": "meta" }