This is 2nd medal for our team RTX 4090[ @zaber666 @artemenon @nexh98 @awsaf49] . In this last few days we were so desperate to boost our public lb that we, unfortunately underestimated our CV which cost us a silver medal. Our best model had a score public lb:0.897 and CV:0.8988 but private lb:0.899. I don't know why after so much experiment we were having a hard time getting 0.90+ in public lb whereas people were getting that margin quite easily. That's why we chose models having moderate score in both public lb and CV which got us a bronze medal. I wish we didn't underestimate our CV.
\nlesson: Don't underestimate your CV🤕
This is 2nd medal for our team RTX 4090[ @zaber666 @artemenon @nexh98 @awsaf49] . In this last few days we were so desperate to boost our public lb that we, unfortunately underestimated our CV which cost us a silver medal. Our best model had a score public lb:0.897 and CV:0.8988 but private lb:0.899. I don't know why after so much experiment we were having a hard time getting 0.90+ in public lb whereas people were getting that margin quite easily. That's why we chose models having moderate score in both public lb and CV which got us a bronze medal. I wish we didn't underestimate our CV.
\nlesson: Don't underestimate your CV🤕
yeah our best CV submission scored 0.897 on public but 0.901 on private(didn't select it)
\nprobably it could take us into gold zone :(
", "rawMarkdown": "yeah our best CV submission scored 0.897 on public but 0.901 on private**(didn't select it)**\n\nprobably it could take us into gold zone :(", "votes": null }, { "id": "1210263", "postDate": "02/19/2021 09:56:01", "content": "☹️ Vai, for a moment we thought there won't be any shake-up.
", "rawMarkdown": "☹️ Vai, for a moment we thought there won't be any shake-up.", "votes": null }, { "id": "1210273", "postDate": "02/19/2021 10:03:57", "content": "My best cv was (5 folds) 0.904, 0.901, 0.905, 0.903, 0.900 , trusting it costed me 1100 places shake down 😂, does anyone of you experienced the same phenomenon? I selected one highest LB and one highest CV, but didn't work for me…
", "rawMarkdown": "My best cv was (5 folds) 0.904, 0.901, 0.905, 0.903, 0.900 , trusting it costed me 1100 places shake down 😂, does anyone of you experienced the same phenomenon? I selected one highest LB and one highest CV, but didn't work for me...", "votes": null }, { "id": "1210275", "postDate": "02/19/2021 10:06:56", "content": "I think it might be due to validation scheme I'm really curious about your result.
", "rawMarkdown": "I think it might be due to **validation scheme** I'm really curious about your result.", "votes": null }, { "id": "1210362", "postDate": "02/19/2021 11:28:54", "content": "I trusted CV and didn’t select best LB sub. Missed price money. I don’t think we can trust anything in this competition. Luck is all we need lol.
", "rawMarkdown": "I trusted CV and didn’t select best LB sub. Missed price money. I don’t think we can trust anything in this competition. Luck is all we need lol.", "votes": null }, { "id": "1210415", "postDate": "02/19/2021 12:15:31", "content": "But First place team stayed in their initial position so I'm guessing it's not all about luck
", "rawMarkdown": "But First place team stayed in their initial position so I'm guessing it's not all about luck", "votes": null }, { "id": "1210856", "postDate": "02/19/2021 18:42:29", "content": "Hey mate, I used Stratified k-fold , with k=5, but my best models were mainly selected from the second fold, with seed 42, i.e. 0.904 and 0.905 were selected from that fold, which means I did epoch-wise blending also. The final outcome was simple averaging. Maybe i didn't pick proper loss function, i chose the Early Layer Regularization (ELR) loss. I used 5 light TTAs (horizontal filp, vertical flip, transpose, and random resized crop, while including the original image also in the tta). I didn't resize the images but i did center crops in the tta and in the validation augmentations, while in the train augmentations i used random resized crops. What am i missing here, maybe I did something wrong?
\nCheers.
Your ensemble technique my best models were mainly selected from the second fold seems a bit strange. Don't we take an ensemble of all the folds? Taking multiple models from the same fold may create some bias.
Actually yea, I also took the remaining 3 models from fold1, fold3 and fold 4, while i didn't use fold 5 since it gave me quite lower cv. Maybe that's the case? There is quite similar idea by ensembling some models from the same fold: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220752. I am also doubting this random resized crops in the TTA, but it gave me significant boost of the LB , by 0.005 …
", "rawMarkdown": "Actually yea, I also took the remaining 3 models from fold1, fold3 and fold 4, while i didn't use fold 5 since it gave me quite lower cv. Maybe that's the case? There is quite similar idea by ensembling some models from the same fold: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220752. I am also doubting this random resized crops in the TTA, but it gave me significant boost of the LB , by 0.005 ...", "votes": null } ], "comments": [ { "id": 1210260, "author_name": "mobassir", "author_url": "", "post_date": "02/19/2021 09:52:10", "content": "yeah our best CV submission scored 0.897 on public but 0.901 on private(didn't select it)
\nprobably it could take us into gold zone :(
", "votes": null, "replies": [ { "id": 1210263, "author_name": "awsaf49", "author_url": "", "post_date": "02/19/2021 09:56:01", "content": "☹️ Vai, for a moment we thought there won't be any shake-up.
", "votes": null, "replies": [] }, { "id": 1210362, "author_name": "underwearfitting", "author_url": "", "post_date": "02/19/2021 11:28:54", "content": "I trusted CV and didn’t select best LB sub. Missed price money. I don’t think we can trust anything in this competition. Luck is all we need lol.
", "votes": null, "replies": [] }, { "id": 1210415, "author_name": "awsaf49", "author_url": "", "post_date": "02/19/2021 12:15:31", "content": "But First place team stayed in their initial position so I'm guessing it's not all about luck
", "votes": null, "replies": [] } ] }, { "id": 1210273, "author_name": "marjan1111", "author_url": "", "post_date": "02/19/2021 10:03:57", "content": "My best cv was (5 folds) 0.904, 0.901, 0.905, 0.903, 0.900 , trusting it costed me 1100 places shake down 😂, does anyone of you experienced the same phenomenon? I selected one highest LB and one highest CV, but didn't work for me…
", "votes": null, "replies": [ { "id": 1210275, "author_name": "awsaf49", "author_url": "", "post_date": "02/19/2021 10:06:56", "content": "I think it might be due to validation scheme I'm really curious about your result.
", "votes": null, "replies": [] }, { "id": 1210856, "author_name": "marjan1111", "author_url": "", "post_date": "02/19/2021 18:42:29", "content": "Hey mate, I used Stratified k-fold , with k=5, but my best models were mainly selected from the second fold, with seed 42, i.e. 0.904 and 0.905 were selected from that fold, which means I did epoch-wise blending also. The final outcome was simple averaging. Maybe i didn't pick proper loss function, i chose the Early Layer Regularization (ELR) loss. I used 5 light TTAs (horizontal filp, vertical flip, transpose, and random resized crop, while including the original image also in the tta). I didn't resize the images but i did center crops in the tta and in the validation augmentations, while in the train augmentations i used random resized crops. What am i missing here, maybe I did something wrong?
\nCheers.
Your ensemble technique my best models were mainly selected from the second fold seems a bit strange. Don't we take an ensemble of all the folds? Taking multiple models from the same fold may create some bias.
Actually yea, I also took the remaining 3 models from fold1, fold3 and fold 4, while i didn't use fold 5 since it gave me quite lower cv. Maybe that's the case? There is quite similar idea by ensembling some models from the same fold: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220752. I am also doubting this random resized crops in the TTA, but it gave me significant boost of the LB , by 0.005 …
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1210257": "This is 2nd medal for our team `RTX 4090`[ @zaber666 @artemenon @nexh98 @awsaf49] . In this last few days we were so desperate to boost our **public lb** that we, unfortunately underestimated our **CV** which cost us a silver medal. Our best model had a score **public lb**:`0.897` and **CV**:`0.8988` but **private lb**:`0.899`. I don't know why after so much experiment we were having a hard time getting `0.90+` in **public lb** whereas people were getting that margin quite easily. That's why we chose models having moderate score in both **public lb** and **CV** which got us a **bronze** medal. I wish we didn't underestimate our **CV**.\n**lesson:** Don't underestimate your **CV**🤕", "1210260": "yeah our best CV submission scored 0.897 on public but 0.901 on private**(didn't select it)**\n\nprobably it could take us into gold zone :(", "1210263": "☹️ Vai, for a moment we thought there won't be any shake-up.", "1210273": "My best cv was (5 folds) 0.904, 0.901, 0.905, 0.903, 0.900 , trusting it costed me 1100 places shake down 😂, does anyone of you experienced the same phenomenon? I selected one highest LB and one highest CV, but didn't work for me...", "1210275": "I think it might be due to **validation scheme** I'm really curious about your result.", "1210362": "I trusted CV and didn’t select best LB sub. Missed price money. I don’t think we can trust anything in this competition. Luck is all we need lol.", "1210415": "But First place team stayed in their initial position so I'm guessing it's not all about luck", "1210856": "Hey mate, I used Stratified k-fold , with k=5, but my best models were mainly selected from the second fold, with seed 42, i.e. 0.904 and 0.905 were selected from that fold, which means I did epoch-wise blending also. The final outcome was simple averaging. Maybe i didn't pick proper loss function, i chose the Early Layer Regularization (ELR) loss. I used 5 light TTAs (horizontal filp, vertical flip, transpose, and random resized crop, while including the original image also in the tta). I didn't resize the images but i did center crops in the tta and in the validation augmentations, while in the train augmentations i used random resized crops. What am i missing here, maybe I did something wrong?\nCheers.", "1210864": "Your ensemble technique `my best models were mainly selected from the second fold` seems a bit strange. Don't we take an ensemble of all the folds? Taking multiple models from the same fold may create some bias.", "1210872": "Actually yea, I also took the remaining 3 models from fold1, fold3 and fold 4, while i didn't use fold 5 since it gave me quite lower cv. Maybe that's the case? There is quite similar idea by ensembling some models from the same fold: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220752. I am also doubting this random resized crops in the TTA, but it gave me significant boost of the LB , by 0.005 ..." }, "source": "meta" }