![\"test\"](\"https://i.ibb.co/2ZBjJ46/Screenshot-from-2022-07-15-11-48-44.png\")
I know it's a bit early but every competition needs one so here it goes :)
\nModel:- Unet+effiecentnet-b0
\nsingle fold cv:-0.79291
\nsingle fold lb (no TTA):- 0.59
I know it's a bit early but every competition needs one so here it goes :)
\nModel:- Unet+effiecentnet-b0
\nsingle fold cv:-0.79291
\nsingle fold lb (no TTA):- 0.59
information needed to split LB-public into LB-public-Hubmap and LB-public-HPA
\n\"roughly 550 test images \"
\nthere are exactly 529 test images, of which Hubmap=448, HPA=81
public test : 55% of the test data()
\n291 --> Hubmap=210 (0.7216), HPA=81 (0.2783)
private test = 45%
\n238--> Hubmap= 238
as an example public submission:
\nall = 0.56\nHPA = 0.19 (set rle='' for data_source != HPA)\nHubmap = 0.36 (set rle='' for data_source != Hubmap)\n#\nHPA x num of public / num of HPA \n= 0.19*291 / 81\n= 0.6825925925925925\nlocal CV is 0.666\n#\nHubmap x num of public / num of Hubmap \n= 0.36*291 / 210\n= 0.49885714285714283\n#\napproximate HPA and Hubmap domain intersection\n= 0.4988/0.68259 = 0.73 ... is this a stable ratio ???\nit seems that i need to further break down score into organs to fault find the weakness.
\nAlso, we do not know how finetuning threshold on HPA will affect Hubmap. need to probe this too!
You should make 2 submissions. One that only predicts mask for HPA and one that only predicts mask for HuBMAP. Then you can compute two LB scores and compare both LB scores to your CV score. My guess is that your HPA public LB score is closer to your CV score than your HuBMAP public LB score.
", "rawMarkdown": "You should make 2 submissions. One that only predicts mask for HPA and one that only predicts mask for HuBMAP. Then you can compute two LB scores and compare both LB scores to your CV score. My guess is that your HPA public LB score is closer to your CV score than your HuBMAP public LB score.", "votes": 9, "replies": [ { "id": 1835769, "postDate": "2022-06-28T04:12:17.370Z", "content": "See this post here for more info
", "rawMarkdown": "See this post [here][1] for more info\n\n[1]: https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941#1832268", "votes": 1 }, { "id": 1857888, "postDate": "2022-07-16T13:22:59.810Z", "content": "What's the point of predicting non-HuBMAP images? They won't be in the private dataset anyway.
", "rawMarkdown": "What's the point of predicting non-HuBMAP images? They won't be in the private dataset anyway.", "votes": 2 } ] }, { "id": 1877615, "postDate": "2022-07-30T20:59:12.623Z", "content": "Nice work! Did you use multi-class segmentation or single-class segmentation?
", "rawMarkdown": "Nice work! Did you use multi-class segmentation or single-class segmentation?" }, { "id": 1886841, "postDate": "2022-08-06T07:40:02.710Z", "content": "multi-class
", "rawMarkdown": "multi-class" } ] }, { "id": 1919519, "postDate": "2022-08-30T13:26:07.200Z", "content": "DeeplabV3 with EfficientNetB4
\nCV: 0.863 (out of folds, over 4 folds)
\nLB: 0.79 (minimal threshold tweaking)
Congrats @jamesphoward, what do you mean by \"minimal threshold tweaking\"?
\nis this a single fold result?
\nwhich loss function did you use?
Im really sorry for bombarding with so many questions 😅
", "rawMarkdown": "Congrats @jamesphoward, what do you mean by \"minimal threshold tweaking\"?\nis this a single fold result? \nwhich loss function did you use?\n\nIm really sorry for bombarding with so many questions 😅" }, { "id": 1919737, "postDate": "2022-08-30T16:33:40.950Z", "content": "This is 4 folds, BCE loss, and I just completely guessed thresholds and haven't optimised them (I picked 0.1 for lung, other organs I used 0.5 for HPA and 0.2 for hubmap).
", "rawMarkdown": "This is 4 folds, BCE loss, and I just completely guessed thresholds and haven't optimised them (I picked 0.1 for lung, other organs I used 0.5 for HPA and 0.2 for hubmap).", "votes": 3 }, { "id": 1919752, "postDate": "2022-08-30T16:45:18.060Z", "content": "Thank you so much for the reply, btw why are you using MSE loss, isn't semantic segmentation a pixel level classification problem, then why mean square error[regression loss], instead of BCE/log loss etc? please correct me if im wrong here
", "rawMarkdown": "Thank you so much for the reply, btw why are you using MSE loss, isn't semantic segmentation a pixel level classification problem, then why mean square error[regression loss], instead of BCE/log loss etc? please correct me if im wrong here" }, { "id": 1919756, "postDate": "2022-08-30T16:49:31.600Z", "content": "Sorry, that was a typo! I meant BCE and wasn't concentrating…
", "rawMarkdown": "Sorry, that was a typo! I meant BCE and wasn't concentrating...", "votes": 1 }, { "id": 1919762, "postDate": "2022-08-30T16:54:59.100Z", "content": "I got scared watching that lol😂. btw, no auxiliary loss?
\nwhat did you use to ensemble those 4folds?
No aux loss, although I do use 4x TTA.
\nI just summate the prediction maps and then divine them by the number of predictions (ie take a simple pixel-wise average).
really appreciate you sharing the details. all the best with the competition.
", "rawMarkdown": "really appreciate you sharing the details. all the best with the competition. " }, { "id": 1920075, "postDate": "2022-08-30T21:27:18.783Z", "content": "Are you using whole images then resize or tiled images
", "rawMarkdown": "Are you using whole images then resize or tiled images" } ] }, { "id": 1856203, "postDate": "2022-07-15T07:19:33.937Z", "content": "Unet2D 32x64x128x256x320
\n5 Fold CV - 0.748672
\nLB - 0.44 🙄 (Whyy?)
I'd strongly suggest you visualise your prediction on the single test image before submitting the notebook. You'll be surprised by how bad your model is on HuBMAP data, I suspect…
", "rawMarkdown": "I'd strongly suggest you visualise your prediction on the single test image before submitting the notebook. You'll be surprised by how bad your model is on HuBMAP data, I suspect...", "votes": 6 }, { "id": 1856233, "postDate": "2022-07-15T07:48:30.330Z", "content": "That's what I observed as well. Annotation seems all over the place
", "rawMarkdown": "That's what I observed as well. Annotation seems all over the place" }, { "id": 1856310, "postDate": "2022-07-15T08:59:00.323Z", "content": "I suffer from the same thing. I trained a very simple unet model with efficientnet b0 encoder and my oof dice score is 0.721, but my leaderboard score is 0.19 lol. These are detailed scores of my model.
\n\"out_of_fold\": {\n \"global\": {\n \"sample_count\": 351,\n \"dice_coefficient\": {\n \"mean\": 0.7210908048892115,\n \"std\": 0.3087503658863055,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.9726454699260181\n },\n \"intersection_over_union\": {\n \"mean\": 0.6343560812096684,\n \"std\": 0.29665165227558893,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.9467476333193632\n }\n },\n \"organs\": {\n \"sample_count\": {\n \"kidney\": 99,\n \"largeintestine\": 58,\n \"lung\": 48,\n \"prostate\": 93,\n \"spleen\": 53\n },\n \"dice_coefficient\": {\n \"kidney\": {\n \"mean\": 0.8977391601781795,\n \"std\": 0.06643440652731449,\n \"min\": 0.6370120607597658,\n \"max\": 0.9726454699260181\n },\n \"largeintestine\": {\n \"mean\": 0.8625815659109379,\n \"std\": 0.11012129876516484,\n \"min\": 0.15504884979521794,\n \"max\": 0.9557906152153275\n },\n \"lung\": {\n \"mean\": 0.04145343110662958,\n \"std\": 0.11725142416682333,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.6145009488394847\n },\n \"prostate\": {\n \"mean\": 0.7947290161557055,\n \"std\": 0.17756911704942144,\n \"min\": 0.20619881455300634,\n \"max\": 0.9563853043455539\n },\n \"spleen\": {\n \"mean\": 0.7225924837743438,\n \"std\": 0.1843539475344702,\n \"min\": 0.01800969580417568,\n \"max\": 0.912698279695554\n }\n },\n \"intersection_over_union\": {\n \"kidney\": {\n \"mean\": 0.820414463802119,\n \"std\": 0.10031296950970611,\n \"min\": 0.46736441498780357,\n \"max\": 0.9467476333193632\n },\n \"largeintestine\": {\n \"mean\": 0.7698824870675187,\n \"std\": 0.12350442742012406,\n \"min\": 0.08403954206806587,\n \"max\": 0.9153246744786364\n },\n \"lung\": {\n \"mean\": 0.025768105153562062,\n \"std\": 0.07890517684618706,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.44352318275512714\n },\n \"prostate\": {\n \"mean\": 0.6900000001631348,\n \"std\": 0.21192072949716861,\n \"min\": 0.11495076278942798,\n \"max\": 0.9164160952582849\n },\n \"spleen\": {\n \"mean\": 0.5920358353910417,\n \"std\": 0.18949636140025228,\n \"min\": 0.009086672020481157,\n \"max\": 0.839415833390244\n }\n }\n }\n }\n}\nThis is my model's prediction on one test sample. It looks pretty bad right now. I think I have to add some transforms because model is not generalizing at all.
\n
One thing I don't really understand is why @thedevastator 's models do SO well at the test data in comparison with many others. His training notebook doesn't do anything that exotic, but the models generalise excellently.
", "rawMarkdown": "One thing I don't really understand is why @thedevastator 's models do SO well at the test data in comparison with many others. His training notebook doesn't do anything that exotic, but the models generalise excellently.", "votes": 7 }, { "id": 1856333, "postDate": "2022-07-15T09:25:26.400Z", "content": "I suggest checking LB on only HubMap data
", "rawMarkdown": "I suggest checking LB on only HubMap data" }, { "id": 1856343, "postDate": "2022-07-15T09:30:32.377Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1856344, "postDate": "2022-07-15T09:33:47.660Z", "content": "One thing I do know is if my CV increases, my LB also increases. We don't have access to labelled Hubmap data so I think we have to rely on heavy augmentation
", "rawMarkdown": "One thing I do know is if my CV increases, my LB also increases. We don't have access to labelled Hubmap data so I think we have to rely on heavy augmentation", "votes": 1 }, { "id": 1856418, "postDate": "2022-07-15T10:31:33.173Z", "content": "@jamesphoward I think the public baseline is not reliable. I tried breaking it with different sizes, params, models etc and most of the time I was able to get nan as loss.
\nfor 0.64 score, I would say it was luck with right combinations. even if you use 512x512 images, it will fail.
@gunesevitan are you using segmentation_models_pytorch? because I had the same results using it. I will recommend try the same with b7. more better would be if you implement it by yourself.
\nI believe this competition is more data oriented than models.
I agree, but if you visualise @thedevastator 's models predictions on the public test image it makes much much better predictions (they look like FTUs rather than random blobs) than similar models I have trained with similar parameters and augmentations.
", "rawMarkdown": "I agree, but if you visualise @thedevastator 's models predictions on the public test image it makes much much better predictions (they look like FTUs rather than random blobs) than similar models I have trained with similar parameters and augmentations." }, { "id": 1856537, "postDate": "2022-07-15T12:01:15.160Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1856553, "postDate": "2022-07-15T12:20:04.900Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1861328, "postDate": "2022-07-19T01:14:04.657Z", "content": "same thing. +1. dont know why.
", "rawMarkdown": "same thing. +1. dont know why." }, { "id": 1862752, "postDate": "2022-07-20T01:31:41.133Z", "content": "Update:
\nSame model
\nInsane Augmentation in attempts to close the domain differences gap
\nCV: 0.77809
\nLB: 0.65 (0.44 Hubmap + 0.21 HPA)
\nStill pretty large gap compared to @hengck23
\n![\"\"](\"https://www.kaggleusercontent.com/kf/101271481/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..w3aDWFam2PgY7arJAMxVIw.R2pzGthO6Quigj_oEN2Ly1Iz2KVJNjYYVwgvDwHb8cV3JocO-hpf6tu4A98D05LvNNl-hea2aifzqA8CHSkNNFFjkUo8YXpoX-nuapuUVbPFKSb7MaX_o6_ZnLvpN3QnEWIHghu4oYmDquADtjdvd6j66MzNP0Bw99GMaLF9ttSVQaJK62oiOTlOTy9dahpCBPQYRC19IEKdzUiyMzOmJRKtJsbGf_23LbqI-eiK-HEFt6g0dKajUw7MXt8QFGUpPq2mdIIsgns3PTHYw8XM1Rkrr6Kp6Rfw-dCCyH82DIvGziMat299SP0CREvjcuv7rS1tIdCZ9gSlQzzbsXManj9Vi0UA_IbSuAvYrdtZ_a0XqPd1i90tLMl7OgVBFOx98u75yRGp6oLMehgzjihHMwTxcqZ8CHtZnrf-KRYEJmzKhdO9T53PA84cS9VhWiY-xUnxiEpGFIzBW7O8RNWhMAmvJl4da6tnOaP7DDfigMqZCNb7rJbS5KLSMOeDuCTOV73YaYmXRxY4SiQnoCZ_8h4pyo1ffvq9ng3cCTU0xTsDmxLtC7sfvL5cqJrBz4go5u8YKo8Z5ZRHOZfq28l8sWrARzSqvTDagJZFZmeRW3-o5WO6NMwNm3y0V6jMs15qj3rh5LKoE9XXPoxixzqgl0nrPZhFHvTJDJifGSqzLsuiopf-iYqmn-6rFEmpuIYU.tPjoJ3-0hEFkAdQNKEPyHA/__results___files/__results___12_0.png\")
hey @quandapro How did you break the LB score into 2 parts? I can only get a total score and I'm facing exactly the same issue like you. I still can't to beat the public kernel score (0.56) with my customized training.
", "rawMarkdown": "hey @quandapro How did you break the LB score into 2 parts? I can only get a total score and I'm facing exactly the same issue like you. I still can't to beat the public kernel score (0.56) with my customized training." }, { "id": 1875389, "postDate": "2022-07-29T01:38:06.690Z", "content": "@fuckvenkatraman do 2 runs:
\nHope that helps.
", "rawMarkdown": "@fuckvenkatraman do 2 runs: \n- Run normally to get full score\n- Set RLE of HPA images to empty to get Hubmap scores\n\nHope that helps. ", "votes": 1 }, { "id": 1886753, "postDate": "2022-08-06T06:09:20.337Z", "content": "I will try you advice for my work, i suppose it will help me a lot.
", "rawMarkdown": "I will try you advice for my work, i suppose it will help me a lot." }, { "id": 1886979, "postDate": "2022-08-06T10:26:10.787Z", "content": "@quandapro i am new to competition, in the default data given we have got only HPA,from where we get the HubMap
", "rawMarkdown": "@quandapro i am new to competition, in the default data given we have got only HPA,from where we get the HubMap" }, { "id": 1929488, "postDate": "2022-09-07T05:29:59.077Z", "content": "Hi @gunesevitan , I suffered the same problem as you, cv: 0.82 lb 0.36. Have you solved the issue? Could you please provide some hints on how you solved it? Thanks.
", "rawMarkdown": "Hi @gunesevitan , I suffered the same problem as you, cv: 0.82 lb 0.36. Have you solved the issue? Could you please provide some hints on how you solved it? Thanks.", "votes": 1 }, { "id": 1930261, "postDate": "2022-09-07T17:19:48.307Z", "content": "I forgot to transpose while submitting
", "rawMarkdown": "I forgot to transpose while submitting" } ] }, { "id": 1839460, "postDate": "2022-07-01T12:31:37.733Z", "content": "I think for a proper CV strategy we need HubMap data too otherwise we are simply training it on HPA data and LB is mostly HubMap. that is why our CV is high but LB is low.
\nbtw my CV is 0.84 and LB is 0.56 😄
I agree, did an quick experiment in which i used model to predict only HPA and if HubMap then predict a circle LB was 0.51
", "rawMarkdown": "I agree, did an quick experiment in which i used model to predict only HPA and if HubMap then predict a circle LB was 0.51", "votes": 1 }, { "id": 1849526, "postDate": "2022-07-09T15:50:04.590Z", "content": "Do we know the HuBMAP data will be circles? I wonder if that's unique to HPA…
", "rawMarkdown": "Do we know the HuBMAP data will be circles? I wonder if that's unique to HPA..." }, { "id": 1849846, "postDate": "2022-07-09T21:56:54.580Z", "content": "in last years challenge there have been no circular specimen in the hubmap data. In this years test data (1 hubmap image) it also does not appear to be round as well.
\nThere are 81 HPA images in the test data, which probably achieve a better score with round masks.
Efficientnet b4, unet, bce/dice (Combo loss)
\nSingle Fold CV 0.76 macro dice, containing 3x 0 dice because of lung. Median is 0.88.
\nSingle Fold LB 0.66
Single Model Efficientnet-b5:
\nMean Macro F1: 0.75, Median Macro F1 0.89
\nLB: 0.69
Did you use the tiles images to training? and what's your CV datasets? I use efficientnet b1 got CV 0.83 but LB only 0.24…ðŸ˜
", "rawMarkdown": "Did you use the tiles images to training? and what's your CV datasets? I use efficientnet b1 got CV 0.83 but LB only 0.24...ðŸ˜", "votes": 1 }, { "id": 1857914, "postDate": "2022-07-16T13:58:11.043Z", "content": "\"CV 0.83 but LB only 0.24\"
\nmost likely it is due to:
\n\"All HPA images have a pixel size of 0.4 µm. For HuBMAP imagery the pixel size is 0.5 µm for kidney, 0.2290 µm for large intestine, 0.7562 µm for lung, 0.4945 µm for spleen, and 6.263 µm for prostate.\"
\nyou can verify by making separate submission for HuBMAP and HPA
", "rawMarkdown": "\"CV 0.83 but LB only 0.24\"\n\nmost likely it is due to:\n\n\"All HPA images have a pixel size of 0.4 µm. For HuBMAP imagery the pixel size is 0.5 µm for kidney, 0.2290 µm for large intestine, 0.7562 µm for lung, 0.4945 µm for spleen, and 6.263 µm for prostate.\"\n\nyou can verify by making separate submission for HuBMAP and HPA", "votes": 6 }, { "id": 1858158, "postDate": "2022-07-16T17:52:28.880Z", "content": "Oh i dont adjust for pixel size (yet), my micro f1/dice is 0.86 are you using micro or macro f1?
\nI am currently using a downscaling of 4 with 320x320 px sized \"tiles\".
\nCV is on 10% stratified over all classes.
Have you visualized your predictions on the one (lol) public validation image?
\nAre there any tiling aretefacts?
\nDo your augmentations also account for the fact, that the test images are stained differently?
refer to https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941
\n5-fold ensmble of smp-unet-effnet7-aug3-768
\n@hengck23 No matter what I try (so far !) I can't go over 0.54 with Hubmap !
\nI am trying with cellpose, ran into memory issues and current tests are bad !
tried pixel size norm, however, the lb score still very low.
", "rawMarkdown": "tried pixel size norm, however, the lb score still very low." }, { "id": 1863058, "postDate": "2022-07-20T06:20:03.320Z", "content": "for my submission with public LB =0.77, Hubmap score is 0.55, HPA is 0.22
", "rawMarkdown": "for my submission with public LB =0.77, Hubmap score is 0.55, HPA is 0.22", "votes": 1 }, { "id": 1864686, "postDate": "2022-07-21T08:42:01.503Z", "content": "yet another strong model
\nuper-net-convnext-large (5-fold) :
\nLB=0.75
\noof CV = 0.781449
@hengck23 could you point me to the repo please.
\nin google i could find only convnext
Single fold
\nCV - 0.7624, LB - 0.62
Per class:
\nprostate - 0.8385
\nspleen - 0.6436
\nlung - 0.1976
\nkidney - 0.9224
\nlarge intestine - 0.8873
Did you train on patch or resized image?
", "rawMarkdown": "Did you train on patch or resized image? ", "votes": 1 }, { "id": 1856372, "postDate": "2022-07-15T09:58:03.567Z", "content": "I have had my best scores on LB (0.62, 0.6) using resized image. Training on patch lead to better CV, but LB score around 0.46. Looks like patching leads to overfitting for me, need to increase augmentations.
", "rawMarkdown": "I have had my best scores on LB (0.62, 0.6) using resized image. Training on patch lead to better CV, but LB score around 0.46. Looks like patching leads to overfitting for me, need to increase augmentations.", "votes": 1 } ] }, { "id": 1893321, "postDate": "2022-08-10T17:53:49.970Z", "content": "Model : efficientnetb4 + deeplab (patch tiling)
\nsingle fold cv : 0.70
\nsingle fold lb : 0.58
\nhubmap : 38
\nhpa : 21
To confirm: fold cv is the score for the training of patches or you did resize and trained and the CV is 0.79291
", "rawMarkdown": "To confirm: fold cv is the score for the training of patches or you did resize and trained and the CV is 0.79291", "replies": [ { "id": 1864898, "postDate": "2022-07-21T11:24:02.577Z", "content": "resize
", "rawMarkdown": "resize ", "votes": 2 } ] } ], "comments": [ { "id": 1838907, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-01T01:47:32.870000", "content": "information needed to split LB-public into LB-public-Hubmap and LB-public-HPA
\n\"roughly 550 test images \"
\nthere are exactly 529 test images, of which Hubmap=448, HPA=81
public test : 55% of the test data()
\n291 --> Hubmap=210 (0.7216), HPA=81 (0.2783)
private test = 45%
\n238--> Hubmap= 238
as an example public submission:
\nall = 0.56\nHPA = 0.19 (set rle='' for data_source != HPA)\nHubmap = 0.36 (set rle='' for data_source != Hubmap)\n#\nHPA x num of public / num of HPA \n= 0.19*291 / 81\n= 0.6825925925925925\nlocal CV is 0.666\n#\nHubmap x num of public / num of Hubmap \n= 0.36*291 / 210\n= 0.49885714285714283\n#\napproximate HPA and Hubmap domain intersection\n= 0.4988/0.68259 = 0.73 ... is this a stable ratio ???\nit seems that i need to further break down score into organs to fault find the weakness.
\nAlso, we do not know how finetuning threshold on HPA will affect Hubmap. need to probe this too!
You should make 2 submissions. One that only predicts mask for HPA and one that only predicts mask for HuBMAP. Then you can compute two LB scores and compare both LB scores to your CV score. My guess is that your HPA public LB score is closer to your CV score than your HuBMAP public LB score.
", "votes": 9, "replies": [ { "id": 1835769, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-06-28T04:12:17.370000", "content": "See this post here for more info
", "votes": 1, "replies": [] }, { "id": 1857888, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-07-16T13:22:59.810000", "content": "What's the point of predicting non-HuBMAP images? They won't be in the private dataset anyway.
", "votes": 2, "replies": [] } ] }, { "id": 1877615, "author_name": "DiamondH", "author_url": "", "post_date": "2022-07-30T20:59:12.623000", "content": "Nice work! Did you use multi-class segmentation or single-class segmentation?
", "votes": 0, "replies": [] }, { "id": 1886841, "author_name": "DiamondH", "author_url": "", "post_date": "2022-08-06T07:40:02.710000", "content": "multi-class
", "votes": 0, "replies": [] } ] }, { "id": 1919519, "author_name": "James Howard", "author_url": "", "post_date": "2022-08-30T13:26:07.200000", "content": "DeeplabV3 with EfficientNetB4
\nCV: 0.863 (out of folds, over 4 folds)
\nLB: 0.79 (minimal threshold tweaking)
Congrats @jamesphoward, what do you mean by \"minimal threshold tweaking\"?
\nis this a single fold result?
\nwhich loss function did you use?
Im really sorry for bombarding with so many questions 😅
", "votes": 0, "replies": [] }, { "id": 1919737, "author_name": "James Howard", "author_url": "", "post_date": "2022-08-30T16:33:40.950000", "content": "This is 4 folds, BCE loss, and I just completely guessed thresholds and haven't optimised them (I picked 0.1 for lung, other organs I used 0.5 for HPA and 0.2 for hubmap).
", "votes": 3, "replies": [] }, { "id": 1919752, "author_name": "somuSan", "author_url": "", "post_date": "2022-08-30T16:45:18.060000", "content": "Thank you so much for the reply, btw why are you using MSE loss, isn't semantic segmentation a pixel level classification problem, then why mean square error[regression loss], instead of BCE/log loss etc? please correct me if im wrong here
", "votes": 0, "replies": [] }, { "id": 1919756, "author_name": "James Howard", "author_url": "", "post_date": "2022-08-30T16:49:31.600000", "content": "Sorry, that was a typo! I meant BCE and wasn't concentrating…
", "votes": 1, "replies": [] }, { "id": 1919762, "author_name": "somuSan", "author_url": "", "post_date": "2022-08-30T16:54:59.100000", "content": "I got scared watching that lol😂. btw, no auxiliary loss?
\nwhat did you use to ensemble those 4folds?
No aux loss, although I do use 4x TTA.
\nI just summate the prediction maps and then divine them by the number of predictions (ie take a simple pixel-wise average).
really appreciate you sharing the details. all the best with the competition.
", "votes": 0, "replies": [] }, { "id": 1920075, "author_name": "Kaggle，啟動", "author_url": "", "post_date": "2022-08-30T21:27:18.783000", "content": "Are you using whole images then resize or tiled images
", "votes": 0, "replies": [] } ] }, { "id": 1856203, "author_name": "Quan", "author_url": "", "post_date": "2022-07-15T07:19:33.937000", "content": "Unet2D 32x64x128x256x320
\n5 Fold CV - 0.748672
\nLB - 0.44 🙄 (Whyy?)
I'd strongly suggest you visualise your prediction on the single test image before submitting the notebook. You'll be surprised by how bad your model is on HuBMAP data, I suspect…
", "votes": 6, "replies": [] }, { "id": 1856233, "author_name": "Quan", "author_url": "", "post_date": "2022-07-15T07:48:30.330000", "content": "That's what I observed as well. Annotation seems all over the place
", "votes": 0, "replies": [] }, { "id": 1856310, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-07-15T08:59:00.323000", "content": "I suffer from the same thing. I trained a very simple unet model with efficientnet b0 encoder and my oof dice score is 0.721, but my leaderboard score is 0.19 lol. These are detailed scores of my model.
\n\"out_of_fold\": {\n \"global\": {\n \"sample_count\": 351,\n \"dice_coefficient\": {\n \"mean\": 0.7210908048892115,\n \"std\": 0.3087503658863055,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.9726454699260181\n },\n \"intersection_over_union\": {\n \"mean\": 0.6343560812096684,\n \"std\": 0.29665165227558893,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.9467476333193632\n }\n },\n \"organs\": {\n \"sample_count\": {\n \"kidney\": 99,\n \"largeintestine\": 58,\n \"lung\": 48,\n \"prostate\": 93,\n \"spleen\": 53\n },\n \"dice_coefficient\": {\n \"kidney\": {\n \"mean\": 0.8977391601781795,\n \"std\": 0.06643440652731449,\n \"min\": 0.6370120607597658,\n \"max\": 0.9726454699260181\n },\n \"largeintestine\": {\n \"mean\": 0.8625815659109379,\n \"std\": 0.11012129876516484,\n \"min\": 0.15504884979521794,\n \"max\": 0.9557906152153275\n },\n \"lung\": {\n \"mean\": 0.04145343110662958,\n \"std\": 0.11725142416682333,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.6145009488394847\n },\n \"prostate\": {\n \"mean\": 0.7947290161557055,\n \"std\": 0.17756911704942144,\n \"min\": 0.20619881455300634,\n \"max\": 0.9563853043455539\n },\n \"spleen\": {\n \"mean\": 0.7225924837743438,\n \"std\": 0.1843539475344702,\n \"min\": 0.01800969580417568,\n \"max\": 0.912698279695554\n }\n },\n \"intersection_over_union\": {\n \"kidney\": {\n \"mean\": 0.820414463802119,\n \"std\": 0.10031296950970611,\n \"min\": 0.46736441498780357,\n \"max\": 0.9467476333193632\n },\n \"largeintestine\": {\n \"mean\": 0.7698824870675187,\n \"std\": 0.12350442742012406,\n \"min\": 0.08403954206806587,\n \"max\": 0.9153246744786364\n },\n \"lung\": {\n \"mean\": 0.025768105153562062,\n \"std\": 0.07890517684618706,\n \"min\": 1.2499843751796857e-11,\n \"max\": 0.44352318275512714\n },\n \"prostate\": {\n \"mean\": 0.6900000001631348,\n \"std\": 0.21192072949716861,\n \"min\": 0.11495076278942798,\n \"max\": 0.9164160952582849\n },\n \"spleen\": {\n \"mean\": 0.5920358353910417,\n \"std\": 0.18949636140025228,\n \"min\": 0.009086672020481157,\n \"max\": 0.839415833390244\n }\n }\n }\n }\n}\nThis is my model's prediction on one test sample. It looks pretty bad right now. I think I have to add some transforms because model is not generalizing at all.
\n
One thing I don't really understand is why @thedevastator 's models do SO well at the test data in comparison with many others. His training notebook doesn't do anything that exotic, but the models generalise excellently.
", "votes": 7, "replies": [] }, { "id": 1856333, "author_name": "shiroe", "author_url": "", "post_date": "2022-07-15T09:25:26.400000", "content": "I suggest checking LB on only HubMap data
", "votes": 0, "replies": [] }, { "id": 1856343, "author_name": "", "author_url": "", "post_date": "2022-07-15T09:30:32.377000", "content": "", "votes": 0, "replies": [] }, { "id": 1856344, "author_name": "Quan", "author_url": "", "post_date": "2022-07-15T09:33:47.660000", "content": "One thing I do know is if my CV increases, my LB also increases. We don't have access to labelled Hubmap data so I think we have to rely on heavy augmentation
", "votes": 1, "replies": [] }, { "id": 1856418, "author_name": "Muhammad Ahmed", "author_url": "", "post_date": "2022-07-15T10:31:33.173000", "content": "@jamesphoward I think the public baseline is not reliable. I tried breaking it with different sizes, params, models etc and most of the time I was able to get nan as loss.
\nfor 0.64 score, I would say it was luck with right combinations. even if you use 512x512 images, it will fail.
@gunesevitan are you using segmentation_models_pytorch? because I had the same results using it. I will recommend try the same with b7. more better would be if you implement it by yourself.
\nI believe this competition is more data oriented than models.
I agree, but if you visualise @thedevastator 's models predictions on the public test image it makes much much better predictions (they look like FTUs rather than random blobs) than similar models I have trained with similar parameters and augmentations.
", "votes": 0, "replies": [] }, { "id": 1856537, "author_name": "", "author_url": "", "post_date": "2022-07-15T12:01:15.160000", "content": "", "votes": 0, "replies": [] }, { "id": 1856553, "author_name": "", "author_url": "", "post_date": "2022-07-15T12:20:04.900000", "content": "", "votes": 0, "replies": [] }, { "id": 1861328, "author_name": "huyidao", "author_url": "", "post_date": "2022-07-19T01:14:04.657000", "content": "same thing. +1. dont know why.
", "votes": 0, "replies": [] }, { "id": 1862752, "author_name": "Quan", "author_url": "", "post_date": "2022-07-20T01:31:41.133000", "content": "Update:
\nSame model
\nInsane Augmentation in attempts to close the domain differences gap
\nCV: 0.77809
\nLB: 0.65 (0.44 Hubmap + 0.21 HPA)
\nStill pretty large gap compared to @hengck23
\n
hey @quandapro How did you break the LB score into 2 parts? I can only get a total score and I'm facing exactly the same issue like you. I still can't to beat the public kernel score (0.56) with my customized training.
", "votes": 0, "replies": [] }, { "id": 1875389, "author_name": "Quan", "author_url": "", "post_date": "2022-07-29T01:38:06.690000", "content": "@fuckvenkatraman do 2 runs:
\nHope that helps.
", "votes": 1, "replies": [] }, { "id": 1886753, "author_name": "kingjohnson", "author_url": "", "post_date": "2022-08-06T06:09:20.337000", "content": "I will try you advice for my work, i suppose it will help me a lot.
", "votes": 0, "replies": [] }, { "id": 1886979, "author_name": "Jaideep", "author_url": "", "post_date": "2022-08-06T10:26:10.787000", "content": "@quandapro i am new to competition, in the default data given we have got only HPA,from where we get the HubMap
", "votes": 0, "replies": [] }, { "id": 1929488, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-09-07T05:29:59.077000", "content": "Hi @gunesevitan , I suffered the same problem as you, cv: 0.82 lb 0.36. Have you solved the issue? Could you please provide some hints on how you solved it? Thanks.
", "votes": 1, "replies": [] }, { "id": 1930261, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-07T17:19:48.307000", "content": "I forgot to transpose while submitting
", "votes": 0, "replies": [] } ] }, { "id": 1839460, "author_name": "Muhammad Ahmed", "author_url": "", "post_date": "2022-07-01T12:31:37.733000", "content": "I think for a proper CV strategy we need HubMap data too otherwise we are simply training it on HPA data and LB is mostly HubMap. that is why our CV is high but LB is low.
\nbtw my CV is 0.84 and LB is 0.56 😄
I agree, did an quick experiment in which i used model to predict only HPA and if HubMap then predict a circle LB was 0.51
", "votes": 1, "replies": [] }, { "id": 1849526, "author_name": "James Howard", "author_url": "", "post_date": "2022-07-09T15:50:04.590000", "content": "Do we know the HuBMAP data will be circles? I wonder if that's unique to HPA…
", "votes": 0, "replies": [] }, { "id": 1849846, "author_name": "Philipp Sodmann", "author_url": "", "post_date": "2022-07-09T21:56:54.580000", "content": "in last years challenge there have been no circular specimen in the hubmap data. In this years test data (1 hubmap image) it also does not appear to be round as well.
\nThere are 81 HPA images in the test data, which probably achieve a better score with round masks.
Efficientnet b4, unet, bce/dice (Combo loss)
\nSingle Fold CV 0.76 macro dice, containing 3x 0 dice because of lung. Median is 0.88.
\nSingle Fold LB 0.66
Single Model Efficientnet-b5:
\nMean Macro F1: 0.75, Median Macro F1 0.89
\nLB: 0.69
Did you use the tiles images to training? and what's your CV datasets? I use efficientnet b1 got CV 0.83 but LB only 0.24…ðŸ˜
", "votes": 1, "replies": [] }, { "id": 1857914, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-16T13:58:11.043000", "content": "\"CV 0.83 but LB only 0.24\"
\nmost likely it is due to:
\n\"All HPA images have a pixel size of 0.4 µm. For HuBMAP imagery the pixel size is 0.5 µm for kidney, 0.2290 µm for large intestine, 0.7562 µm for lung, 0.4945 µm for spleen, and 6.263 µm for prostate.\"
\nyou can verify by making separate submission for HuBMAP and HPA
", "votes": 6, "replies": [] }, { "id": 1858158, "author_name": "Philipp Sodmann", "author_url": "", "post_date": "2022-07-16T17:52:28.880000", "content": "Oh i dont adjust for pixel size (yet), my micro f1/dice is 0.86 are you using micro or macro f1?
\nI am currently using a downscaling of 4 with 320x320 px sized \"tiles\".
\nCV is on 10% stratified over all classes.
Have you visualized your predictions on the one (lol) public validation image?
\nAre there any tiling aretefacts?
\nDo your augmentations also account for the fact, that the test images are stained differently?
refer to https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941
\n5-fold ensmble of smp-unet-effnet7-aug3-768
\n@hengck23 No matter what I try (so far !) I can't go over 0.54 with Hubmap !
\nI am trying with cellpose, ran into memory issues and current tests are bad !
tried pixel size norm, however, the lb score still very low.
", "votes": 0, "replies": [] }, { "id": 1863058, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-20T06:20:03.320000", "content": "for my submission with public LB =0.77, Hubmap score is 0.55, HPA is 0.22
", "votes": 1, "replies": [] }, { "id": 1864686, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-21T08:42:01.503000", "content": "yet another strong model
\nuper-net-convnext-large (5-fold) :
\nLB=0.75
\noof CV = 0.781449
@hengck23 could you point me to the repo please.
\nin google i could find only convnext
Single fold
\nCV - 0.7624, LB - 0.62
Per class:
\nprostate - 0.8385
\nspleen - 0.6436
\nlung - 0.1976
\nkidney - 0.9224
\nlarge intestine - 0.8873
Did you train on patch or resized image?
", "votes": 1, "replies": [] }, { "id": 1856372, "author_name": "Jebastin Nadar", "author_url": "", "post_date": "2022-07-15T09:58:03.567000", "content": "I have had my best scores on LB (0.62, 0.6) using resized image. Training on patch lead to better CV, but LB score around 0.46. Looks like patching leads to overfitting for me, need to increase augmentations.
", "votes": 1, "replies": [] } ] }, { "id": 1893321, "author_name": "kaggler", "author_url": "", "post_date": "2022-08-10T17:53:49.970000", "content": "Model : efficientnetb4 + deeplab (patch tiling)
\nsingle fold cv : 0.70
\nsingle fold lb : 0.58
\nhubmap : 38
\nhpa : 21
To confirm: fold cv is the score for the training of patches or you did resize and trained and the CV is 0.79291
", "votes": 0, "replies": [ { "id": 1864898, "author_name": "shiroe", "author_url": "", "post_date": "2022-07-21T11:24:02.577000", "content": "resize
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1835135": "I know it's a bit early but every competition needs one so here it goes :)\nModel:- Unet+effiecentnet-b0\nsingle fold cv:-0.79291 \nsingle fold lb (no TTA):- 0.59", "1838907": "information needed to split LB-public into LB-public-Hubmap and LB-public-HPA \n\n\n\"roughly 550 test images \"\nthere are exactly 529 test images, of which Hubmap=448, HPA=81\n\npublic test : 55% of the test data()\n291 --> Hubmap=210 (0.7216), HPA=81 (0.2783)\n\nprivate test = 45%\n238--> Hubmap= 238\n\n\n", "1835768": "You should make 2 submissions. One that only predicts mask for HPA and one that only predicts mask for HuBMAP. Then you can compute two LB scores and compare both LB scores to your CV score. My guess is that your HPA public LB score is closer to your CV score than your HuBMAP public LB score.", "1877615": "* fold_4 out of 5 fold\n* local: 0.81\n* lb: 0.73\nkidney : 0.94\nprostate : 0.81\nlargeintestine : 0.92\nspleen : 0.84\nlung : 0.38\n[mmsegmentation](https://www.kaggle.com/code/w3579628328/multi-class-mmsegmentation-training)", "1919519": "DeeplabV3 with EfficientNetB4\nCV: 0.863 (out of folds, over 4 folds)\nLB: 0.79 (minimal threshold tweaking)", "1856203": "Unet2D 32x64x128x256x320\n5 Fold CV - 0.748672\nLB - 0.44 🙄 (Whyy?)", "1839460": "I think for a proper CV strategy we need HubMap data too otherwise we are simply training it on HPA data and LB is mostly HubMap. that is why our CV is high but LB is low. \nbtw my CV is 0.84 and LB is 0.56 😄", "1849216": "Efficientnet b4, unet, bce/dice (Combo loss)\nSingle Fold CV 0.76 macro dice, containing 3x 0 dice because of lung. Median is 0.88.\nSingle Fold LB 0.66 ", "1838366": "Single fold\nCV - 0.7624, LB - 0.62\n\nPer class:\nprostate - 0.8385\nspleen - 0.6436\nlung - 0.1976\nkidney - 0.9224\nlarge intestine - 0.8873", "1893321": "Model : efficientnetb4 + deeplab (patch tiling)\nsingle fold cv : 0.70\nsingle fold lb : 0.58 \nhubmap : 38 \nhpa : 21", "1841013": "To confirm: fold cv is the score for the training of patches or you did resize and trained and the CV is 0.79291" } }