{ "id": 353260, "title": "Beginner asking for help! Got ~0.7 in 5-fold local cross-validation and 0.43 in LB", "url": "/competitions/hubmap-organ-segmentation/discussion/353260", "author_name": "", "post_date": "2022-09-17T14:40:03.107309400Z", "votes": 3, "comment_count": 8, "views": 0, "content": "

Thank you so much for clicking in.
\nAs you can see in the image below, I got a score of about 0.7 on the local validation set(5-fold), but a score of 0.43 on the public leaderboard (obtained by calculating the mean of the five outputs at 200epoch).
\n\"\"
\n\"\"
\nMy setting:
\nSegformer (mit_b1) with its proposed head
\n768*768 with no data augmentation in training and testing stage
\nmodel outputs (b, 1, h, w) with BCE loss
\nI don't know how to troubleshoot my problem and improve my performance. If you can help me, I would be very grateful. If you need more information to judge, please ask me directly!🙏

", "messages": [ { "id": "1943477", "postDate": "09/17/2022 14:40:03", "content": "

Thank you so much for clicking in.
\nAs you can see in the image below, I got a score of about 0.7 on the local validation set(5-fold), but a score of 0.43 on the public leaderboard (obtained by calculating the mean of the five outputs at 200epoch).
\n\"\"
\n\"\"
\nMy setting:
\nSegformer (mit_b1) with its proposed head
\n768*768 with no data augmentation in training and testing stage
\nmodel outputs (b, 1, h, w) with BCE loss
\nI don't know how to troubleshoot my problem and improve my performance. If you can help me, I would be very grateful. If you need more information to judge, please ask me directly!🙏

", "rawMarkdown": "Thank you so much for clicking in.\nAs you can see in the image below, I got a score of about 0.7 on the local validation set(5-fold), but a score of 0.43 on the public leaderboard (obtained by calculating the mean of the five outputs at 200epoch). \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11423388%2Fb84fa24913eea94ce6902cb37dea2d3d%2F2022-09-17%2010.21.16.png?generation=1663424658081531&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11423388%2Fe733e5b0578d914ca7c17380aadd37ff%2F2022-09-17%2010.36.07.png?generation=1663425394991306&alt=media)\nMy setting:\nSegformer (mit_b1) with its proposed head\n768*768 with no data augmentation in training and testing stage\nmodel outputs (b, 1, h, w) with BCE loss\nI don't know how to troubleshoot my problem and improve my performance. If you can help me, I would be very grateful. If you need more information to judge, please ask me directly!🙏", "votes": null }, { "id": "1943501", "postDate": "09/17/2022 14:59:51", "content": "

Train set consists of 100% HPA data while test set consists of the mix of HPA and HuBMAP data. This is why your LB score is much lower than your CV. Check this post to better understand the problem in this competition. If I were you, I would first reduce the gap between CV and LB by adding augmentations. Once CV reflects LB pretty well, I would try various things such as better modeling, higher resolution, ensemble, etc in order to improve CV score, which is also an improvement of LB. This is my humble opinion.

", "rawMarkdown": "Train set consists of 100% HPA data while test set consists of the mix of HPA and HuBMAP data. This is why your LB score is much lower than your CV. Check this [post](https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/337489#1858625:~:text=first%20step%20is%20to%20think%20of%20input%20domain%20space) to better understand the problem in this competition. If I were you, I would first reduce the gap between CV and LB by adding augmentations. Once CV reflects LB pretty well, I would try various things such as better modeling, higher resolution, ensemble, etc in order to improve CV score, which is also an improvement of LB. This is my humble opinion.", "votes": null }, { "id": "1943524", "postDate": "09/17/2022 15:16:05", "content": "

Thank you very much for your very detailed answer, I will adapt my code along your lines! ❤️❤️❤️

", "rawMarkdown": "Thank you very much for your very detailed answer, I will adapt my code along your lines! ❤️❤️❤️", "votes": null }, { "id": "1944069", "postDate": "09/18/2022 03:32:41", "content": "

You could try some augmentations: random hue, saturation, contrast, brightness, flip, shear, rotate, random resized crop. These work for me.

", "rawMarkdown": "You could try some augmentations: random hue, saturation, contrast, brightness, flip, shear, rotate, random resized crop. These work for me.", "votes": null }, { "id": "1944070", "postDate": "09/18/2022 03:36:28", "content": "

Thank you very much for your reply, I think these are used for data augmentation during training, and would like to ask you what data augmentation you used in testing (seems like everyone calls it tta), thank you very much!

", "rawMarkdown": "Thank you very much for your reply, I think these are used for data augmentation during training, and would like to ask you what data augmentation you used in testing (seems like everyone calls it tta), thank you very much!", "votes": null }, { "id": "1944223", "postDate": "09/18/2022 06:43:57", "content": "

I flip horizontally and vertically in testing and average predictions.

", "rawMarkdown": "I flip horizontally and vertically in testing and average predictions.", "votes": null }, { "id": "1944454", "postDate": "09/18/2022 10:13:40", "content": "

copy, thanks very much!

", "rawMarkdown": "copy, thanks very much!", "votes": null }, { "id": "1946829", "postDate": "09/20/2022 05:59:08", "content": "

Better add external data (PAS/HE staining) to training, as HPA looks very different with Hubmap

", "rawMarkdown": "Better add external data (PAS/HE staining) to training, as HPA looks very different with Hubmap", "votes": null }, { "id": "1947080", "postDate": "09/20/2022 08:36:21", "content": "

It makes sense and I will try! Thanks very much!

", "rawMarkdown": "It makes sense and I will try! Thanks very much!", "votes": null } ], "comments": [ { "id": 1943501, "author_name": "cheulkay", "author_url": "", "post_date": "09/17/2022 14:59:51", "content": "

Train set consists of 100% HPA data while test set consists of the mix of HPA and HuBMAP data. This is why your LB score is much lower than your CV. Check this post to better understand the problem in this competition. If I were you, I would first reduce the gap between CV and LB by adding augmentations. Once CV reflects LB pretty well, I would try various things such as better modeling, higher resolution, ensemble, etc in order to improve CV score, which is also an improvement of LB. This is my humble opinion.

", "votes": null, "replies": [ { "id": 1943524, "author_name": "xihuguan", "author_url": "", "post_date": "09/17/2022 15:16:05", "content": "

Thank you very much for your very detailed answer, I will adapt my code along your lines! ❤️❤️❤️

", "votes": null, "replies": [] } ] }, { "id": 1944069, "author_name": "electro", "author_url": "", "post_date": "09/18/2022 03:32:41", "content": "

You could try some augmentations: random hue, saturation, contrast, brightness, flip, shear, rotate, random resized crop. These work for me.

", "votes": null, "replies": [ { "id": 1944070, "author_name": "xihuguan", "author_url": "", "post_date": "09/18/2022 03:36:28", "content": "

Thank you very much for your reply, I think these are used for data augmentation during training, and would like to ask you what data augmentation you used in testing (seems like everyone calls it tta), thank you very much!

", "votes": null, "replies": [] }, { "id": 1944223, "author_name": "electro", "author_url": "", "post_date": "09/18/2022 06:43:57", "content": "

I flip horizontally and vertically in testing and average predictions.

", "votes": null, "replies": [] }, { "id": 1944454, "author_name": "xihuguan", "author_url": "", "post_date": "09/18/2022 10:13:40", "content": "

copy, thanks very much!

", "votes": null, "replies": [] } ] }, { "id": 1946829, "author_name": "lililycai", "author_url": "", "post_date": "09/20/2022 05:59:08", "content": "

Better add external data (PAS/HE staining) to training, as HPA looks very different with Hubmap

", "votes": null, "replies": [ { "id": 1947080, "author_name": "xihuguan", "author_url": "", "post_date": "09/20/2022 08:36:21", "content": "

It makes sense and I will try! Thanks very much!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1943477": "Thank you so much for clicking in.\nAs you can see in the image below, I got a score of about 0.7 on the local validation set(5-fold), but a score of 0.43 on the public leaderboard (obtained by calculating the mean of the five outputs at 200epoch). \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11423388%2Fb84fa24913eea94ce6902cb37dea2d3d%2F2022-09-17%2010.21.16.png?generation=1663424658081531&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11423388%2Fe733e5b0578d914ca7c17380aadd37ff%2F2022-09-17%2010.36.07.png?generation=1663425394991306&alt=media)\nMy setting:\nSegformer (mit_b1) with its proposed head\n768*768 with no data augmentation in training and testing stage\nmodel outputs (b, 1, h, w) with BCE loss\nI don't know how to troubleshoot my problem and improve my performance. If you can help me, I would be very grateful. If you need more information to judge, please ask me directly!🙏", "1943501": "Train set consists of 100% HPA data while test set consists of the mix of HPA and HuBMAP data. This is why your LB score is much lower than your CV. Check this [post](https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/337489#1858625:~:text=first%20step%20is%20to%20think%20of%20input%20domain%20space) to better understand the problem in this competition. If I were you, I would first reduce the gap between CV and LB by adding augmentations. Once CV reflects LB pretty well, I would try various things such as better modeling, higher resolution, ensemble, etc in order to improve CV score, which is also an improvement of LB. This is my humble opinion.", "1943524": "Thank you very much for your very detailed answer, I will adapt my code along your lines! ❤️❤️❤️", "1944069": "You could try some augmentations: random hue, saturation, contrast, brightness, flip, shear, rotate, random resized crop. These work for me.", "1944070": "Thank you very much for your reply, I think these are used for data augmentation during training, and would like to ask you what data augmentation you used in testing (seems like everyone calls it tta), thank you very much!", "1944223": "I flip horizontally and vertically in testing and average predictions.", "1944454": "copy, thanks very much!", "1946829": "Better add external data (PAS/HE staining) to training, as HPA looks very different with Hubmap", "1947080": "It makes sense and I will try! Thanks very much!" }, "source": "meta" }