I believe teams in the .985+ zone should have similar approaches; as DatNT @moewie94 pointed out, the score is very doable using a single model (I was not a believer but am now). I am worried about persistently optimizing single-model LB when the local validation is very small because of the risk of overfitting and shakeup (Estimating a ~.001-.005 shakeup, but honestly don't know)
\n\nThere exists a consistent .007~.015 gap between CV and LB and I think that it is the key to consistently pushing past .99 other than ensembling. Assuming that this is in the large part due to unseen triplets in training data, can we somehow disentangle the three labels?
\n\nIn other words, can we explicitly prohibit the model from learning correlation between the three labels and predict based on visual clues alone? Maybe it's doing more harm than good because the correlation should be quite informative, but no pain, no gain; otherwise we cannot effectively eliminate the gap. As always, really curious what the top did other than ensembling
\n\nBeen looking around for multi-label literature, and found few practical ideas. Looking to the ever-helpful and wise kaggle opportunity for help :P
", "messages": [ { "id": "755092", "postDate": "02/24/2020 12:25:06", "content": "I believe teams in the .985+ zone should have similar approaches; as DatNT @moewie94 pointed out, the score is very doable using a single model (I was not a believer but am now). I am worried about persistently optimizing single-model LB when the local validation is very small because of the risk of overfitting and shakeup (Estimating a ~.001-.005 shakeup, but honestly don't know)
\n\nThere exists a consistent .007~.015 gap between CV and LB and I think that it is the key to consistently pushing past .99 other than ensembling. Assuming that this is in the large part due to unseen triplets in training data, can we somehow disentangle the three labels?
\n\nIn other words, can we explicitly prohibit the model from learning correlation between the three labels and predict based on visual clues alone? Maybe it's doing more harm than good because the correlation should be quite informative, but no pain, no gain; otherwise we cannot effectively eliminate the gap. As always, really curious what the top did other than ensembling
\n\nBeen looking around for multi-label literature, and found few practical ideas. Looking to the ever-helpful and wise kaggle opportunity for help :P
", "rawMarkdown": "I believe teams in the .985+ zone should have similar approaches; as DatNT @moewie94 pointed out, the score is very doable using a single model (I was not a believer but am now). I am worried about persistently optimizing single-model LB when the local validation is very small because of the risk of overfitting and shakeup (Estimating a ~.001-.005 shakeup, but honestly don't know)\n\nThere exists a consistent .007~.015 gap between CV and LB and I think that it is the key to consistently pushing past .99 other than ensembling. Assuming that this is in the large part due to unseen triplets in training data, can we somehow disentangle the three labels?\n\nIn other words, can we explicitly prohibit the model from learning correlation between the three labels and predict based on visual clues alone? Maybe it's doing more harm than good because the correlation should be quite informative, but no pain, no gain; otherwise we cannot effectively eliminate the gap. As always, really curious what the top did other than ensembling\n\nBeen looking around for multi-label literature, and found few practical ideas. Looking to the ever-helpful and wise kaggle opportunity for help :P", "votes": null }, { "id": "755097", "postDate": "02/24/2020 12:33:10", "content": "i see large gap between cv and lb,even with cv 0.9714 i can get lb score of 0.9429 only with single model. i tried both cutmix and mixup on single channel images and both did really worst job!
", "rawMarkdown": "i see large gap between cv and lb,even with cv 0.9714 i can get lb score of 0.9429 only with single model. i tried both cutmix and mixup on single channel images and both did really worst job!", "votes": null }, { "id": "755100", "postDate": "02/24/2020 12:36:19", "content": "@mobassir I had similar differences in one of my submissions. Double-check the image size and the normalization at inference time.
", "rawMarkdown": "mobassir I had similar differences in one of my submissions. Double-check the image size and the normalization at inference time.", "votes": null }, { "id": "755104", "postDate": "02/24/2020 12:42:29", "content": "@pestipeti i trained models on 128x128x1 images , in inference kernel i used 128x128 too but i haven't normalized anything because during training models i didn't normalize images,
", "rawMarkdown": "pestipeti i trained models on 128x128x1 images , in inference kernel i used 128x128 too but i haven't normalized anything because during training models i didn't normalize images,", "votes": null }, { "id": "755105", "postDate": "02/24/2020 12:45:38", "content": "@pestipeti sorry ,i normalized each image by its max val during inference
", "rawMarkdown": "pestipeti sorry ,i normalized each image by its max val during inference", "votes": null }, { "id": "755378", "postDate": "02/24/2020 18:14:06", "content": "@mobassir same here. with cv 0.989 only got lb 0.969x. Did you solve the problem?
", "rawMarkdown": "mobassir same here. with cv 0.989 only got lb 0.969x. Did you solve the problem?", "votes": null }, { "id": "755743", "postDate": "02/25/2020 04:40:16", "content": "@tonychenxyz still trying
", "rawMarkdown": "tonychenxyz still trying", "votes": null } ], "comments": [ { "id": 755097, "author_name": "mobassir", "author_url": "", "post_date": "02/24/2020 12:33:10", "content": "i see large gap between cv and lb,even with cv 0.9714 i can get lb score of 0.9429 only with single model. i tried both cutmix and mixup on single channel images and both did really worst job!
", "votes": null, "replies": [ { "id": 755100, "author_name": "pestipeti", "author_url": "", "post_date": "02/24/2020 12:36:19", "content": "@mobassir I had similar differences in one of my submissions. Double-check the image size and the normalization at inference time.
", "votes": null, "replies": [] }, { "id": 755104, "author_name": "mobassir", "author_url": "", "post_date": "02/24/2020 12:42:29", "content": "@pestipeti i trained models on 128x128x1 images , in inference kernel i used 128x128 too but i haven't normalized anything because during training models i didn't normalize images,
", "votes": null, "replies": [] }, { "id": 755105, "author_name": "mobassir", "author_url": "", "post_date": "02/24/2020 12:45:38", "content": "@pestipeti sorry ,i normalized each image by its max val during inference
", "votes": null, "replies": [] }, { "id": 755378, "author_name": "tonychenxyz", "author_url": "", "post_date": "02/24/2020 18:14:06", "content": "@mobassir same here. with cv 0.989 only got lb 0.969x. Did you solve the problem?
", "votes": null, "replies": [] }, { "id": 755743, "author_name": "mobassir", "author_url": "", "post_date": "02/25/2020 04:40:16", "content": "@tonychenxyz still trying
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "755092": "I believe teams in the .985+ zone should have similar approaches; as DatNT @moewie94 pointed out, the score is very doable using a single model (I was not a believer but am now). I am worried about persistently optimizing single-model LB when the local validation is very small because of the risk of overfitting and shakeup (Estimating a ~.001-.005 shakeup, but honestly don't know)\n\nThere exists a consistent .007~.015 gap between CV and LB and I think that it is the key to consistently pushing past .99 other than ensembling. Assuming that this is in the large part due to unseen triplets in training data, can we somehow disentangle the three labels?\n\nIn other words, can we explicitly prohibit the model from learning correlation between the three labels and predict based on visual clues alone? Maybe it's doing more harm than good because the correlation should be quite informative, but no pain, no gain; otherwise we cannot effectively eliminate the gap. As always, really curious what the top did other than ensembling\n\nBeen looking around for multi-label literature, and found few practical ideas. Looking to the ever-helpful and wise kaggle opportunity for help :P", "755097": "i see large gap between cv and lb,even with cv 0.9714 i can get lb score of 0.9429 only with single model. i tried both cutmix and mixup on single channel images and both did really worst job!", "755100": "mobassir I had similar differences in one of my submissions. Double-check the image size and the normalization at inference time.", "755104": "pestipeti i trained models on 128x128x1 images , in inference kernel i used 128x128 too but i haven't normalized anything because during training models i didn't normalize images,", "755105": "pestipeti sorry ,i normalized each image by its max val during inference", "755378": "mobassir same here. with cv 0.989 only got lb 0.969x. Did you solve the problem?", "755743": "tonychenxyz still trying" }, "source": "meta" }