\"All experiment results are only made possible by the Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards. With fast and large GPU cards, you can enjoy designing models and sleep early without worrying about out-of-memory errors :) \"
\nㅤ ㅤ
\nThis is work in progress. results will be updated as and when experiments complete !!!
\nIt has two parts:
\n1) benchmaking current solutions
\n![\"https://i.ibb.co/6nf8dDD/Picture1.png\"](\"https://i.ibb.co/6nf8dDD/Picture1.png\")
\n![\"https://i.ibb.co/JHdsCw0/Selection-081.png\"](\"https://i.ibb.co/JHdsCw0/Selection-081.png\")
\nㅤ ㅤ
\n2) progress report of my method (which hopefully will work)
\nㅤ ㅤ
\nmy approach follows the NLP method:
reference paper:
\n[1] BEiT: BERT Pre-Training of Image Transformers
\nhttps://openreview.net/pdf?id=p-BhZSz59o4
\nhttps://www.bilibili.com/video/BV1PF411z7mf/
\nㅤ
\n[2] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis
\nhttps://arxiv.org/pdf/2111.14791.pdf
\"All experiment results are only made possible by the Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards. With fast and large GPU cards, you can enjoy designing models and sleep early without worrying about out-of-memory errors :) \"
\nㅤ ㅤ
\nThis is work in progress. results will be updated as and when experiments complete !!!
\nIt has two parts:
\n1) benchmaking current solutions
\n
\n
\nㅤ ㅤ
\n2) progress report of my method (which hopefully will work)
\nㅤ ㅤ
\nmy approach follows the NLP method:
reference paper:
\n[1] BEiT: BERT Pre-Training of Image Transformers
\nhttps://openreview.net/pdf?id=p-BhZSz59o4
\nhttps://www.bilibili.com/video/BV1PF411z7mf/
\nㅤ
\n[2] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis
\nhttps://arxiv.org/pdf/2111.14791.pdf
surprise!!!!
\nthe winner is 5-fold segformer-mit-b2 with CV=0.787, LB=0.78.
\nThe model size is only 99mb.
I haven't investigate the reasons, but I think it could be:
\nGood results, I'm also trying segformer without pretrained weights from imagenet, and fold0 cv=0.76. Did you use pretrained weights?
", "rawMarkdown": "Good results, I'm also trying segformer without pretrained weights from imagenet, and fold0 cv=0.76. Did you use pretrained weights?", "votes": 2 }, { "id": 1871222, "postDate": "2022-07-26T06:41:57.667Z", "content": "i use pretrained weight.
\ni just do another probe: Hubmap-LB is an amazing 0.55 (normalised to 0.762142857)
more surprise!!!
\nsingle-fold (fold-3) segformer-mit-b2 alone is already LB 0.79 (not low 0.79+ but a good 0.79+)
breakdown:
\nlocal cv HPA = 0.783965
\nnote that score on submission page is trauncated (?) and not rounded
\n![\"https://i.ibb.co/QCVy0yR/Selection-094.png\"](\"https://i.ibb.co/QCVy0yR/Selection-094.png\")
either:
\n@hengck23 I am trying with segformer, I am not sure that I am training correct, I don't have much time for two weeks to test but I am getting values such as:
\nMean_iou: 0.7336344062222001
\nMean accuracy: 0.7336344062222001
in epoch 1 ! using 256 tiles and I am not sure this is normal or not, I am using a demo starting code which is not complex !!
", "rawMarkdown": "@hengck23 I am trying with segformer, I am not sure that I am training correct, I don't have much time for two weeks to test but I am getting values such as:\nMean_iou: 0.7336344062222001\nMean accuracy: 0.7336344062222001\n\nin epoch 1 ! using 256 tiles and I am not sure this is normal or not, I am using a demo starting code which is not complex !! ", "votes": 1 }, { "id": 1873065, "postDate": "2022-07-27T12:31:51.163Z", "content": "I smell something wrong, I just modified the basic training code with this competition dataset and rewrote the evaluation process.
", "rawMarkdown": "I smell something wrong, I just modified the basic training code with this competition dataset and rewrote the evaluation process. ", "votes": 1 }, { "id": 1873232, "postDate": "2022-07-27T13:54:23.303Z", "content": "@asalhi I don't have much time for two weeks to test but I am getting values such as:\"
\nI am not sure if 256 tile is good enough. i trained with images resized from 3000 to 768.
\nthere is a couple of tricks that may be important:
please see the new added folder at \"segformer-mit-b2\", \"train-log-segformer-mit-b2\"
\nhttps://www.kaggle.com/datasets/hengck23/hubmap-discuss-00
Thanks so much for sharing your knowledge. As always very valuable.
\nOne question I have is about the license to use your model. On SegFormer's Github it says use is for non-commercial only. Wouldn't that conflict with the competition license (open source) in case of winning?
", "rawMarkdown": "Thanks so much for sharing your knowledge. As always very valuable.\n\nOne question I have is about the license to use your model. On SegFormer's Github it says use is for non-commercial only. Wouldn't that conflict with the competition license (open source) in case of winning?", "votes": 1 }, { "id": 1873474, "postDate": "2022-07-27T16:17:55.387Z", "content": "@hengck23 hi I want ask you some question, first congratulations on your good scores!
\n\"I am not sure if 256 tile is good enough. i trained with images resized from 3000 to 768.\"
\nSo you didn't use tiles? Just resizing to 768?
", "rawMarkdown": "\"I am not sure if 256 tile is good enough. i trained with images resized from 3000 to 768.\"\n\nSo you didn't use tiles? Just resizing to 768?" }, { "id": 1873772, "postDate": "2022-07-27T22:32:22.497Z", "content": "![\"Selection-090\"](\"https://i.ibb.co/YtdDtCf/Selection-090.png\")
"
},
{
"id": 1873790,
"postDate": "2022-07-27T22:51:53.393Z",
"content": "![\"Selection-093\"](\"https://i.ibb.co/vQS8cFg/Selection-093.png\")
",
"votes": 2
},
{
"id": 1874218,
"postDate": "2022-07-28T06:25:10.993Z",
"content": "thanks for sharing. May I ask:
\nwhat makes you use 'aux2_loss' alone instead of using 4 'aux[%d]_loss' outputs ?
\nno lr schduler applied?
by experiment results. it is trial and error
", "rawMarkdown": "by experiment results. it is trial and error" }, { "id": 1874381, "postDate": "2022-07-28T08:40:46.413Z", "content": "@robsonsan \"Wouldn't that conflict with the competition license (open source) in case of winning?\"
\nthanks for pointing that out. i would consider that a month later (to see if I am still a prize contender).
\nperformance is limited by data and not method. so segformer can be easily replaced. i just have to investigate the results for its good performance :)
on a side note:
\n![\"https://i.ibb.co/fSSJx0q/Selection-096.png\"](\"https://i.ibb.co/fSSJx0q/Selection-096.png\")
did another experiment:
\nswin-v1-small + segformer (single fold only , fold 3)
\nLB (0.76+)
\nbreakdown:
\nLB-Humap (0.54+)
\nLB-HPA (0.76+) - (0.54+) = (0.22+) to (0.23+)
local cv:
\nall 0.787098
\nkidney 0.951417
\nprostate 0.815064
\nlargeintestine 0.903701
\nspleen 0.852030
\nlung 0.230929
Lung segmentation is a challenge …
", "rawMarkdown": "Lung segmentation is a challenge ..." }, { "id": 1874624, "postDate": "2022-07-28T11:46:43.687Z", "content": "\"Lung segmentation\" …
\nthis one you probably have to use some tricks.
\ne.g.
\nyou can check the size of sizes of the FTU from the json files. i suspect maybe you can can use size to filter the results, etc. but I am not sure.
you can relabel the train image to include
it is mentioned \"there is at least one FTU per image\", I haven't probe the missing rate in LB set
Thanks for the tips. I'm also working on some other hypotheses to see if my models can generalize to lung tissue. For now I'm avoiding using external data
", "rawMarkdown": "Thanks for the tips. I'm also working on some other hypotheses to see if my models can generalize to lung tissue. For now I'm avoiding using external data" }, { "id": 1894011, "postDate": "2022-08-11T08:10:50.383Z", "content": "@hengck23 FYI - I think the license was changed recently
\n ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3641623%2Fbc1eca113e4a5d5db2913330d3028b71%2Fseg_lice.png?generation=1660205517627981&alt=media\")
oops, someone discovered our post
", "rawMarkdown": "oops, someone discovered our post", "votes": 1 }, { "id": 1894356, "postDate": "2022-08-11T12:43:21.980Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F437986%2F15e768421588378942b42f3512631f0d%2FiShot2022-08-11%2020.42.43.jpg?generation=1660221783814291&alt=media\")
finally
", "rawMarkdown": "\n\nfinally" }, { "id": 1911216, "postDate": "2022-08-24T01:18:54.190Z", "content": "Can we use segformer or not? I\"m not an expert in License issues.
", "rawMarkdown": "Can we use segformer or not? I\"m not an expert in License issues.", "votes": 2 }, { "id": 1911580, "postDate": "2022-08-24T07:28:48.237Z", "content": "I think we can use Segformer unless we claim to the prize.
", "rawMarkdown": "I think we can use Segformer unless we claim to the prize. ~~@AddisonHoward, can you verify this license issue?~~" }, { "id": 1912166, "postDate": "2022-08-24T14:53:48.993Z", "content": "@cheulkay Thanks for your clarification
", "rawMarkdown": "@cheulkay Thanks for your clarification" }, { "id": 1934514, "postDate": "2022-09-11T12:20:55.087Z", "content": "how do I know overffiting in training?
", "rawMarkdown": "how do I know overffiting in training?" } ] }, { "id": 1832268, "postDate": "2022-06-24T22:03:05.767Z", "content": "how many hubmap images are there in the test set:
\n![\"https://i.ibb.co/P5hNDrb/Selection-029.png\"](\"https://i.ibb.co/P5hNDrb/Selection-029.png\")
we know:
\nnum_hubmap = len(test_df[test_df.data_source=='Hubmap'])\nnum_hpa = len(test_df[test_df.data_source=='HPA'])\nnum = len(test_df)\n\nif num_hubmap > int(0.90*num):\n submit_df.to_csv('submission.csv',index=False)\nsmarter way to probe without wasting submission slot
\n# some task which you already know the notebook running time\n\nsleep (num_hubmap*10_min)\n\n#you can deduce value of num_hubmap from additional running time\n# some monotnic function that you already know\n if (num> 0) and (num<=545): threshold =-1 #lb 0.3+ (submit all filled rle)\n if (num>=546) and (num<=546): threshold =0.3 #lb 0.56\n if (num>=547) and (num<=547): threshold =0.4 #lb 0.54\n if (num>=548) and (num<=548): threshold =0.5 #lb 0.52\n if (num>=549) and (num<1000): threshold =100 #lb 0.00 (submit all empty rle)\n\n\n#you can deduce value of num from lbscore\n\"roughly 550 test images \"
\nthere are exactly 529 test images, of which Hubmap=448, HPA=81
public test : 55% of the test data()
\n291 --> Hubmap=210, HPA=81
private test = 45%
\n238--> Hubmap= 238
if you like my work and would like to learn how to apply 3d transformer for 3d ct scan classification, please follow
\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350859
i will modify many of the 2d vit used here (e.g. mix-trasnformer, PVTv2,coat) for 3d
", "rawMarkdown": "if you like my work and would like to learn how to apply 3d transformer for 3d ct scan classification, please follow\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350859\n\ni will modify many of the 2d vit used here (e.g. mix-trasnformer, PVTv2,coat) for 3d", "votes": 7, "replies": [ { "id": 1952891, "postDate": "2022-09-24T03:59:53.573Z", "content": "besides the above, here is another competition i am taking part:
\nhttps://www.kaggle.com/competitions/novozymes-enzyme-stability-prediction/discussion/354631
", "rawMarkdown": "besides the above, here is another competition i am taking part:\n\nhttps://www.kaggle.com/competitions/novozymes-enzyme-stability-prediction/discussion/354631" } ] }, { "id": 1894487, "postDate": "2022-08-11T14:29:17.767Z", "content": "Thanks for your wonderful sharing as always 😃. I have just tried your upernet with swin-tiny and got 0.79 cv, but the lb score is only 0.61. Do I miss anything important? @hengck23
", "rawMarkdown": "Thanks for your wonderful sharing as always 😃. I have just tried your upernet with swin-tiny and got 0.79 cv, but the lb score is only 0.61. Do I miss anything important? @hengck23 ", "votes": 7 }, { "id": 1869702, "postDate": "2022-07-25T01:42:12.997Z", "content": "i put code for upernet + swin transformer v2 for variable image size input at:
\nhttps://www.kaggle.com/code/hengck23/lb-0-75-variable-size-swin-transformer-v1-and-v2
this is beta release. using provided upernet+swin v1 (or v2) will get you about LB 0.75 (15 min submission time).
\nplan:
it would be faster for me if anyone can upload and share their downloaded images from HPA, etc
", "rawMarkdown": "i put code for upernet + swin transformer v2 for variable image size input at: \nhttps://www.kaggle.com/code/hengck23/lb-0-75-variable-size-swin-transformer-v1-and-v2\n\nthis is beta release. using provided upernet+swin v1 (or v2) will get you about LB 0.75 (15 min submission time).\nplan:\n- ~~ release of my training log, hyperprameters, experimental results~~ [done]\n- modification to use UN-MAE, uniform masked autoencoder, for SSL pertaining with unlabelled data\n(https://github.com/implus/UM-MAE)\n\nit would be faster for me if anyone can upload and share their downloaded images from HPA, etc", "votes": 7, "replies": [ { "id": 1870350, "postDate": "2022-07-25T13:44:39.003Z", "content": "Do you train with whole images or tiles?
", "rawMarkdown": "Do you train with whole images or tiles?\n", "votes": 2 }, { "id": 1877490, "postDate": "2022-07-30T17:32:41.863Z", "content": "👋, @hengck23 ,
\nno need to use external data. just check the code I posted
", "rawMarkdown": "no need to use external data. just check the code I posted" }, { "id": 1877513, "postDate": "2022-07-30T18:11:22.927Z", "content": "I've tried tiled images with ConvNext, getting a LB score of 0.71 by 5 fold ensemble🙉, but oof is slightly better than your experiments, I'll try whole images later!
", "rawMarkdown": "I've tried tiled images with ConvNext, getting a LB score of 0.71 by 5 fold ensemble🙉, but oof is slightly better than your experiments, I'll try whole images later!", "votes": 1 }, { "id": 1879826, "postDate": "2022-08-01T09:38:21.637Z", "content": "i managed to put everything together, SL, SSL learning, online, offline learning.
\nthis will be next software release.
the consistency feature loss in the middle ensure SSL aligned with SL learning
\n![\"https://i.ibb.co/vzsSQ1t/Selection-127.png\"](\"https://i.ibb.co/vzsSQ1t/Selection-127.png\")
\" but oof is slightly better than your experiments, I'll try whole images later!\"
\nyour oof must be the same as your lb submission (i.e. you compute only one score for the whole image after the tiles are combined)
seems that i select the wrong submission
\n![\"Selection-317\"](\"https://i.ibb.co/MSr3LY9/Selection-317.png\")
interesting:
\n\n\ninteresting:\n- i correctly predicted the highest public LB score\n- i correctly predicted the shakeup\n- i correctly dreamt of my ranking and medal (just one off the gold medal) just the day before\n",
"votes": 5,
"replies": [
{
"id": 1951254,
"postDate": "2022-09-23T00:16:19.233Z",
"content": "oh…
\nYou are 1th to my team..
\nThank you very much for this competition.
Thanks you, @hengck23 , I learned a lot from you thread，i'll never got the order without your generous share .By the way, I also chose wrong submision.I get 0.83 last two days so chose the new😂.
", "rawMarkdown": "Thanks you, @hengck23 , I learned a lot from you thread，i'll never got the order without your generous share .By the way, I also chose wrong submision.I get 0.83 last two days so chose the new😂.", "votes": 1 }, { "id": 1952653, "postDate": "2022-09-23T19:55:16.073Z", "content": "Thanks for all the sharing in the competition @hengck23 and such a strong finish.
\nI correctly dreamt of my ranking and medal (just one off the gold medal) just the day before
\nNot going to lie, this has happened to me in few of the previous competitions where I also missed solo gold by marginal ranks 🥲 It's really disheartening haha.
", "rawMarkdown": "Thanks for all the sharing in the competition @hengck23 and such a strong finish. \n\n*I correctly dreamt of my ranking and medal (just one off the gold medal) just the day before*\n\nNot going to lie, this has happened to me in few of the previous competitions where I also missed solo gold by marginal ranks 🥲 It's really disheartening haha.\n " } ] }, { "id": 1833464, "postDate": "2022-06-26T03:27:59.327Z", "content": "my first baseline at public LB 0.56
\none fold only:
\ntrain_dataset :
\nvalid_dataset :
\n![\"https://i.ibb.co/sQmXBwB/Selection-047.png\"](\"https://i.ibb.co/sQmXBwB/Selection-047.png\")
hello @hengck23,
\ndoes your model converge after 18 epochs?
\"does your model converge after 18 epochs?\"
\nyes for the above experiment. (I use little augmentation and the number of output channels in decoder is large)
update:
\nmodel = resnext50 + unet (use ASPP for center block, adjust approriate dilation for different input image size)
\ntrain image size = 512x512
\ntrain augment = random rotate, random scale, flip
\nhyperparameter = 80 epoch, batch_size 16, lookahead+RAMDA opt
\nlocal cv: 0.707 (at threshold 0.50)
train image size = 640x640
\nlocal cv: 0.715 (fold-0 at threshold 0.50) / 0.753 (fold-1)
why do you rescale it by \"512/3000\" if your model receives images of 480x480??
", "rawMarkdown": "why do you rescale it by \"512/3000\" if your model receives images of 480x480??", "votes": 1 }, { "id": 1858816, "postDate": "2022-07-17T08:21:56.507Z", "content": "By experiment results. Sometimes, but not all times, your can get better results when your inference size is larger than the train size
", "rawMarkdown": "By experiment results. Sometimes, but not all times, your can get better results when your inference size is larger than the train size" }, { "id": 1858817, "postDate": "2022-07-17T08:25:25.277Z", "content": "I confirm that, thats the first thing I tried, learned this from starfish competition 🙂
\nBut you have to change the std and mean values also for some inference pipelines
I love your threads :-) especially the \"my experiment results\" one , It's like a story of how research and experiments should be.
\nGreat work :-)
100% agree with you @asalhi , @hengck23 threads are very informative and inspirational! Thanks for sharing all this with the wider Kaggle community.
", "rawMarkdown": "100% agree with you @asalhi , @hengck23 threads are very informative and inspirational! Thanks for sharing all this with the wider Kaggle community.", "votes": 1 } ] }, { "id": 1925481, "postDate": "2022-09-04T02:19:26.513Z", "content": "this is a nice paper to read. i think some of the conclusion also applies to this kaggle competition according to my results:
\nhttps://www.arxiv-vanity.com/papers/2201.08683/
\nA Comprehensive Study of Vision Transformers on Dense Prediction Tasks
We studied different aspects of VTs and CNNs as feature extractors for object detection and semantic segmentation on challenging and real-world data. The main results and key insights derived from our experiments are as follows:
\nVTs outperform CNNs in in-distribution dataset while having lower inference speed, but less computational complexity. Hence, if the GPUs are optimized for Transformer architectures, they have the potential to become dominant in computer vision.
VTs generalize better to OOD datasets. Our loss landscape analysis shows that VTs converge to flatter minima compared to CNNs, which can explain their generalizability.
VTs are more robust to natural corruptions and adversarial attacks compared to CNNs. We believe that this could be attributed to the global receptive field as well as the dynamic nature of self-attention.
VTs are less-texture biased than CNNs, which can be attributed to their global receptive field, allowing them to focus better on global shape-based cues as opposed to local texture-based cues.
are you stuck at 0.80?
\nare you stuck at 0.81?
\nare you stuck at 0.82?
\n… adjust accordingly
\nnote:
\nlb-HPA is always in 0.21 to 0.22
\nlb-Hubmap stops at 0.60 (even at lb 0.83)
Out of interest, what do you think are the HuBMAP-only equivalents are for these values?
", "rawMarkdown": "Out of interest, what do you think are the HuBMAP-only equivalents are for these values?" }, { "id": 1921658, "postDate": "2022-09-01T01:21:44.750Z", "content": "Same with you, my lb-HuBMAP reached 0.60 when my lb is 0.82, and when I get lb 0.83, lb-HuBMAP is still 0.60😂
", "rawMarkdown": "Same with you, my lb-HuBMAP reached 0.60 when my lb is 0.82, and when I get lb 0.83, lb-HuBMAP is still 0.60😂" }, { "id": 1921674, "postDate": "2022-09-01T01:35:26.510Z", "content": "technically, it is:
\nlb = 0.830 to 0.839
\nlb-HuBMAP= (0.600 to 0.609)/0.7217 = 0.8314 to 0.8438
\nlb-HPA= (0.221 to 0.239)/0.2783 = 0.7941 to 0.8588
local cv (HPA) ~0.82
", "rawMarkdown": "technically, it is:\nlb = 0.830 to 0.839\nlb-HuBMAP= (0.600 to 0.609)/0.7217 = 0.8314 to 0.8438\nlb-HPA= (0.221 to 0.239)/0.2783 = 0.7941 to 0.8588\n\n---\nlocal cv (HPA) ~0.82" }, { "id": 1925783, "postDate": "2022-09-04T09:48:10.267Z", "content": "Does that mean when I reach 0.6 on HuBMAP, can I finally rest and pray?
", "rawMarkdown": "Does that mean when I reach 0.6 on HuBMAP, can I finally rest and pray?", "votes": 2 }, { "id": 1925860, "postDate": "2022-09-04T11:48:56.073Z", "content": "no. you have to check your ranking. it will be 0.82 everywhere after a week
", "rawMarkdown": "no. you have to check your ranking. it will be 0.82 everywhere after a week", "votes": 1 }, { "id": 1926617, "postDate": "2022-09-05T02:02:44.963Z", "content": "an example of trying different seeds
\nLB = 0.78,0.80,0.79,0.75,0.80
\nCV = 0.803159,0.764383,0.777201,0.818149,0.791553
different fold have different results (due to label noise)
\ngood public = worse private?
\nhow to prevent shake up?
Have you test the hubmap score of different folds? Kind of curious😋
", "rawMarkdown": "Have you test the hubmap score of different folds? Kind of curious😋" }, { "id": 1927375, "postDate": "2022-09-05T15:34:36.630Z", "content": "Got stuck at 0.80 for quite a long time…
", "rawMarkdown": "Got stuck at 0.80 for quite a long time...", "votes": 1 }, { "id": 1928257, "postDate": "2022-09-06T11:18:09.643Z", "content": "\"Got stuck at 0.80 for quite a long time…\"
\nchange/select fold, ensemble resolution, change threshold
", "rawMarkdown": "\"Got stuck at 0.80 for quite a long time…\"\n\nchange/select fold, ensemble resolution, change threshold\n" }, { "id": 1928603, "postDate": "2022-09-06T14:32:25.313Z", "content": "Hi Heng, I've not tried ensembling models, but tried to change the seeds & try new augments. It looks like my CV fluctuates… Is there a good way to reduce the impact of labeling noise? I'm not an expert in the medical domain, so I'm not sure all the FTUs in the HPA images are fully labeled. Maybe pseudo-labeling might help?
", "rawMarkdown": "Hi Heng, I've not tried ensembling models, but tried to change the seeds & try new augments. It looks like my CV fluctuates... Is there a good way to reduce the impact of labeling noise? I'm not an expert in the medical domain, so I'm not sure all the FTUs in the HPA images are fully labeled. Maybe pseudo-labeling might help?" }, { "id": 1930129, "postDate": "2022-09-07T15:23:40.723Z", "content": "If we are stuck at 0.78, how can we do?
", "rawMarkdown": "If we are stuck at 0.78, how can we do?" }, { "id": 1930374, "postDate": "2022-09-07T19:40:24.757Z", "content": "\"If we are stuck at 0.78, how can we do?\"
\ni assume you are talking about single model solution at 768x768 input resolution.
\nthen you are still having some issues with augmentation, TTA, training (learning rate/overfitting), ensemble single fold (SWA) or multiple fold for single model, threshold, … etc
\nyou have two choices:
\n1) you can still finetune you single model solution. but you will be cap to about 0.79 at 768x768
2) increase resolution, proceed to train multiple architecture for ensemble. but you will be cap 0.78 + 0.02
\ni would do both and (1) and (2). more exploration would means more training results and more information for algorithm debug
", "rawMarkdown": "\"If we are stuck at 0.78, how can we do?\"\n\ni assume you are talking about single model solution at 768x768 input resolution.\n\nthen you are still having some issues with augmentation, TTA, training (learning rate/overfitting), ensemble single fold (SWA) or multiple fold for single model, threshold, ... etc\n\nyou have two choices: \n1) you can still finetune you single model solution. but you will be cap to about 0.79 at 768x768\n\n2) increase resolution, proceed to train multiple architecture for ensemble. but you will be cap 0.78 + 0.02\n\ni would do both and (1) and (2). more exploration would means more training results and more information for algorithm debug\n", "votes": 1 }, { "id": 1930522, "postDate": "2022-09-08T02:04:06.060Z", "content": "Thanks a lot for your share, I will keep moving based on your tips!
", "rawMarkdown": "Thanks a lot for your share, I will keep moving based on your tips! " }, { "id": 1930529, "postDate": "2022-09-08T02:15:12.853Z", "content": "poem : The Road Not Taken by Robert Frost
\nTwo roads diverged in a wood, and I—
\nI took the one less traveled by,
\nAnd that has made all the difference.
Thank you very much for sharing your all experiments and tips
", "rawMarkdown": "Thank you very much for sharing your all experiments and tips", "votes": 1 } ] }, { "id": 1907670, "postDate": "2022-08-21T00:54:45.903Z", "content": "this is what every kaggler should know
\n\"This a question that comes up all the time in machine learning: How well can we do with the data we have? Even when we have a result, it can be hard to tell if it's a good result. What more could we do with better approaches? Or are we already close to the upper bound?\"
\nlink:
\nhttps://drivendata.co/blog/aleatoric-limit1
\nhttps://drivendata.co/blog/aleatoric-limit2
i thing kaggle can have a blog to invite kagglers to make post like this.
\ngood post will be given free TPU/GPU hours etc.
this is not the same as notebook as this focus on content rather than code
", "rawMarkdown": "this is what every kaggler should know\n\n\"This a question that comes up all the time in machine learning: How well can we do with the data we have? Even when we have a result, it can be hard to tell if it's a good result. What more could we do with better approaches? Or are we already close to the upper bound?\"\n\nlink: \nhttps://drivendata.co/blog/aleatoric-limit1 \nhttps://drivendata.co/blog/aleatoric-limit2\n\n\ni thing kaggle can have a blog to invite kagglers to make post like this.\ngood post will be given free TPU/GPU hours etc.\n\nthis is not the same as notebook as this focus on content rather than code", "votes": 5 }, { "id": 1882428, "postDate": "2022-08-03T09:15:46.607Z", "content": "look what i have found
\n![\"https://i.ibb.co/0X3dDs4/Selection-171.png\"](\"https://i.ibb.co/0X3dDs4/Selection-171.png\")
@hengck23 Hi heng, Would you mind share more information about this image, is it from hubmap (https://portal.hubmapconsortium.org/)? thank you.
", "rawMarkdown": "@hengck23 Hi heng, Would you mind share more information about this image, is it from hubmap (https://portal.hubmapconsortium.org/)? thank you.\n" } ] }, { "id": 1874042, "postDate": "2022-07-28T03:20:45.080Z", "content": "one free trick for all:
\n![\"https://i.ibb.co/6mgvGXv/Selection-095.png\"](\"https://i.ibb.co/6mgvGXv/Selection-095.png\")
i haven't tried to add 2021 kaggle train data yet (kidney and large intestine). i also expect some improvement from there
\nwe now enter phrase 2 of the competition: from 0.80 to ~0.83
\nGreat 0.8 :-)
\nMy 0.75 was achived with basline notebook and some new training with larger bone with some size modification I achived 0.7.
\nAnd a trick ( or approch) that pushed the score from 0.70 to 0.75 , a boost that I belive you didn’t yet try and I guess with your current solution it might give you a boost of 0.02 or so. Or maybe not! If segformer is doing great with HuBMap.
That is very intuitive. It would probably work for all organs.
", "rawMarkdown": "That is very intuitive. It would probably work for all organs." }, { "id": 1886216, "postDate": "2022-08-05T16:27:16.787Z", "content": "I am curious about which way you choose for training, the whole image(resize) or image tiles.
", "rawMarkdown": "I am curious about which way you choose for training, the whole image(resize) or image tiles." } ] }, { "id": 1945022, "postDate": "2022-09-18T18:53:39.057Z", "content": "@hengck23 - Your experience of coat-small (+- parallel) with 1x1 daformer outperforming the medium model with 3x3 - is this purely because you are able to train the small model at a higher resolution than medium, or do you think the smaller model is genuinely better? I ask because I believe segformer b3 as an encoder outperforms b5 on this dataset even at higher resolutions.
", "rawMarkdown": "@hengck23 - Your experience of coat-small (+- parallel) with 1x1 daformer outperforming the medium model with 3x3 - is this purely because you are able to train the small model at a higher resolution than medium, or do you think the smaller model is genuinely better? I ask because I believe segformer b3 as an encoder outperforms b5 on this dataset even at higher resolutions.", "votes": 3, "replies": [ { "id": 1946523, "postDate": "2022-09-19T21:10:50.643Z", "content": "In my experiments, at the same resoluton, CoatMedium(4) + Daformer3x3 works better than CoatSmall(5) + Daformer1x1 - this seems to be different from others' experience.
", "rawMarkdown": "In my experiments, at the same resoluton, CoatMedium(4) + Daformer3x3 works better than CoatSmall(5) + Daformer1x1 - this seems to be different from others' experience. ", "votes": 1 } ] }, { "id": 1952700, "postDate": "2022-09-23T20:42:39.293Z", "content": "with both the private and public score revealed, i discover something interesting.
\ni made a bug by loading a mitb6 encoder with a trained mit5 model at submission inference. this means that only the early layer are loaded and the unloaded weights are randomly initialized. However the performer is better then loading the correct model. at first i thought this is better \"by chance\". but now both private and public scores actually improved, meaning that improvement are not \"by chance\".
\nfor those that are doing research and writing paper, you may want to investigate on this. it seems possible just to \"perturb from train optimum (esp on just upper layers)\" and get better validation results.
\ni also note that transformer is trained with drop path and \"residual add\" in FF and attention block enforces learning to get residual=0
", "rawMarkdown": "with both the private and public score revealed, i discover something interesting.\n\ni made a bug by loading a mitb6 encoder with a trained mit5 model at submission inference. this means that only the early layer are loaded and the unloaded weights are randomly initialized. However the performer is better then loading the correct model. at first i thought this is better \"by chance\". but now both private and public scores actually improved, meaning that improvement are not \"by chance\".\n\nfor those that are doing research and writing paper, you may want to investigate on this. it seems possible just to \"perturb from train optimum (esp on just upper layers)\" and get better validation results.\n\ni also note that transformer is trained with drop path and \"residual add\" in FF and attention block enforces learning to get residual=0", "votes": 4, "replies": [ { "id": 1956303, "postDate": "2022-09-26T11:39:19.983Z", "content": "@hengck23 On Sept23 at ~1AM UTC when the \"Private leaderboard was disclosed\" vs at ~6PM UTC when the \"leaderboard was finalized\" I noticed a change in the Private LB ranks. My team's Private LB rank improved by 30 places. Why so? Were some of the teams disqualified during this time? If yes why were they disqualified ?
", "rawMarkdown": "@hengck23 On Sept23 at ~1AM UTC when the \"Private leaderboard was disclosed\" vs at ~6PM UTC when the \"leaderboard was finalized\" I noticed a change in the Private LB ranks. My team's Private LB rank improved by 30 places. Why so? Were some of the teams disqualified during this time? If yes why were they disqualified ?" }, { "id": 1956334, "postDate": "2022-09-26T11:55:58.127Z", "content": "@desaijairav I'm not hengck23 but lemme answer. 70 teams were disqualified for cheating: related thread. Probably, they used multiple accounts to submit more than five times a day, in order to either verify modifications on their programs, or to fine-tune the threshold for each organ.
\nPS you can check the very first rule of this competition here: One account per participant
", "rawMarkdown": "@desaijairav I'm not hengck23 but lemme answer. 70 teams were disqualified for cheating: [related thread](https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/354700#:~:text=anti%2Dcheating%20algorithm). Probably, they used multiple accounts to submit more than five times a day, in order to either verify modifications on their programs, or to fine-tune the threshold for each organ.\n\nPS you can check the very first rule of this competition [here](https://www.kaggle.com/competitions/hubmap-organ-segmentation/rules#:~:text=One%20account%20per%20participant): One account per participant", "votes": 1 }, { "id": 1956746, "postDate": "2022-09-26T15:27:48.753Z", "content": "Thanks a lot @cheulkay !! That was a exactly what I was looking for.
", "rawMarkdown": "Thanks a lot @cheulkay !! That was a exactly what I was looking for. " }, { "id": 1958040, "postDate": "2022-09-27T09:00:02.890Z", "content": "Another quick question- Was Segformer allowed to be used? I could see Segformer in some of the shared notebooks which are medal winners. It had \"Apache 2.0\" license initially & later changed to \"Other\".
\nIn terms of licensing any insights/links around what is allowed/not allowed? I went through the Kaggle rules & License conditions but the Technical-Legal language often becomes difficult to understand.
", "rawMarkdown": "Another quick question- Was Segformer allowed to be used? I could see Segformer in some of the shared notebooks which are medal winners. It had \"Apache 2.0\" license initially & later changed to \"Other\". \n\nIn terms of licensing any insights/links around what is allowed/not allowed? I went through the Kaggle rules & License conditions but the Technical-Legal language often becomes difficult to understand. " }, { "id": 1958644, "postDate": "2022-09-27T15:05:08.633Z", "content": "I have inquired of the question to the competition host.
\nAfter their reply, I'll share their opinion with my solution:)
my 13-th place solution: public 0.83, private: 0.81~0.82
\nhttps://www.kaggle.com/code/hengck23/lb0-83-submit/edit/run/104911483
organ_threshold = { \n 'Hubmap': { \n 'kidney' : 0.28, \n 'prostate' : 0.28, \n 'largeintestine': 0.28, \n 'spleen' : 0.28, \n 'lung' : 0.05, \n }, \n 'HPA': { \n 'kidney' : 0.50, \n 'prostate' : 0.50, \n 'largeintestine': 0.50, \n 'spleen' : 0.50, \n 'lung' : 0.10, \n }, \n}\n\nmodel = [ \n dotdict( #[1]\n is_use=1,\n image_size=1024,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4_level5, 'decoder': daformer_conv3x3, },\n checkpoint=[\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-4-swa.pth',\n\n ],\n ),\n dotdict(#[2]\n is_use = 1,\n image_size = 1536,\n module = 'model_coat_daformer',\n param={'encoder': coat_parallel_small_level5, 'decoder':daformer_conv1x1},\n checkpoint = [ \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-4-swa.pth', \n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-0-swa.pth', \n ],\n ),\n\n\n dotdict( #[3]\n is_use=1,\n image_size=1536,\n module='model_effnet_smp_unet',\n param={'encoder': tf_efficientnet_b6, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-3-swa.pth', \n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-1-swa.pth',\n ],\n ),\n dotdict(#[4]\n is_use = 1,\n image_size = 768,\n module = 'model_dualvit_daformer',\n param={'encoder': dual_vit_b, 'decoder':daformer_conv3x3},\n checkpoint = [\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-4-swa.pth',\n ],\n ), \n\n\n dotdict(#[5]\n is_use=1,\n image_size=1280,\n module='model_convnext_smp_unet',\n param={'encoder': convnext_large_384_in22ft1k, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-4-swa.pth',\n ],\n ), \n\n\n\n dotdict(\n is_use=1,\n image_size=768,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4, 'decoder': daformer_conv3x3}, \n checkpoint=[\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-0-swa.pth', #0.78\n '../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-1-swa.pth', # 0.80\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-2-swa.pth', #0.79 \n ],\n ),\nyou will get private score 0.81 to 0.82 depending on how you select the base models (e.g. remove comment #)
", "rawMarkdown": "my 13-th place solution: public 0.83, private: 0.81~0.82\nhttps://www.kaggle.com/code/hengck23/lb0-83-submit/edit/run/104911483\n```\norgan_threshold = { \n 'Hubmap': { \n 'kidney' : 0.28, \n 'prostate' : 0.28, \n 'largeintestine': 0.28, \n 'spleen' : 0.28, \n 'lung' : 0.05, \n }, \n 'HPA': { \n 'kidney' : 0.50, \n 'prostate' : 0.50, \n 'largeintestine': 0.50, \n 'spleen' : 0.50, \n 'lung' : 0.10, \n }, \n}\n\nmodel = [ \n dotdict( #[1]\n is_use=1,\n image_size=1024,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4_level5, 'decoder': daformer_conv3x3, },\n checkpoint=[\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-4-swa.pth',\n \n ],\n ),\n dotdict(#[2]\n is_use = 1,\n image_size = 1536,\n module = 'model_coat_daformer',\n param={'encoder': coat_parallel_small_level5, 'decoder':daformer_conv1x1},\n checkpoint = [ \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-4-swa.pth', \n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-0-swa.pth', \n ],\n ),\n \n \n dotdict( #[3]\n is_use=1,\n image_size=1536,\n module='model_effnet_smp_unet',\n param={'encoder': tf_efficientnet_b6, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-3-swa.pth', \n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-1-swa.pth',\n ],\n ),\n dotdict(#[4]\n is_use = 1,\n image_size = 768,\n module = 'model_dualvit_daformer',\n param={'encoder': dual_vit_b, 'decoder':daformer_conv3x3},\n checkpoint = [\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-4-swa.pth',\n ],\n ), \n\n \n dotdict(#[5]\n is_use=1,\n image_size=1280,\n module='model_convnext_smp_unet',\n param={'encoder': convnext_large_384_in22ft1k, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-4-swa.pth',\n ],\n ), \n \n \n \n dotdict(\n is_use=1,\n image_size=768,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4, 'decoder': daformer_conv3x3}, \n checkpoint=[\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-0-swa.pth', #0.78\n '../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-1-swa.pth', # 0.80\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-2-swa.pth', #0.79 \n ],\n ),\n\n```\n\nyou will get private score 0.81 to 0.82 depending on how you select the base models (e.g. remove comment #)", "votes": 4 }, { "id": 1948830, "postDate": "2022-09-21T10:34:35.917Z", "content": "Thanks for everything @hengck23. Learned a lot from this topic. Hope to you see on Mayo Clinic competition…
", "rawMarkdown": "Thanks for everything @hengck23. Learned a lot from this topic. Hope to you see on Mayo Clinic competition...", "votes": 4, "replies": [ { "id": 1949496, "postDate": "2022-09-21T17:36:28.413Z", "content": "Many thanks, @hengck23! I also learned quite a lot from this thread.
", "rawMarkdown": "Many thanks, @hengck23! I also learned quite a lot from this thread." } ] }, { "id": 1883676, "postDate": "2022-08-04T02:27:55.620Z", "content": "best backbone for segformer
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Gu_Multi-Scale_High-Resolution_Vision_Transformer_for_Semantic_Segmentation_CVPR_2022_paper.pdf
![\"https://i.ibb.co/7XpM0P3/Selection-176.png\"](\"https://i.ibb.co/7XpM0P3/Selection-176.png\")
another paper improving segformer
\n![\"https://i.ibb.co/4jmKVFK/Selection-135.png\"](\"https://i.ibb.co/4jmKVFK/Selection-135.png\")
i realize i made a mistake. i have been computing dice validation dice at train size of 3000.
\nBut there are image with pixel size less than 0.4 um, i.e. these images are \"much larger\". hence the dice for these hidden test images (largeintestine) are worse than your local results.
hence size 768 seem to be sufficient for most organ, it may be too small for largeintestine. if you are using 768 for largeintestine, you need a better enlarging mask method (rather than just bilinear upscale)
\nit is for this reason, using larger image input like 1024, 1280, 1536 improves LB scores.
\nhere are the LB scores(single fold):
\nLB 0.81
\nLB 0.80
\nLB 0.79
\n1536 already max out my GPU.
\nto go to 2048, i either need to 4x overlap 1536 tiles or single image using gradient check.
maybe i need to rewrite the code using
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Mangalam_Reversible_Vision_Transformers_CVPR_2022_paper.pdf
\"Reversible Vision Transformers achieve a reduced memory
\nfootprint of up to 15.5× at identical model complexity, parameters and accuracy,\"
What puzzles me is that large image input(1024) gives me worse performance on LB.
", "rawMarkdown": "What puzzles me is that large image input(1024) gives me worse performance on LB.", "votes": 6 }, { "id": 1907377, "postDate": "2022-08-20T17:56:40.723Z", "content": "try a few folds?
\nthe training of large images is not the same as small images.
\nfirst your transformer needs to be larger because you need larger receptive field.
this makes the transformer too powerful and you need more regularization.
\ni use more augmentation and more aux loss.
swinv2 paper (which also uses 1536) uses feature distillation as regularization.
\nif you do it correctly you will seed you local validation improves greatly for some organs,
\nnotably my local cv has 0.915 for largeintestine
Let's begin the GPU war
", "rawMarkdown": "Let's begin the GPU war" }, { "id": 1907617, "postDate": "2022-08-21T00:00:00.593Z", "content": "i wonder if anyone try CNN + large image (e.g. efficient or resnet at 1024, 1028, 1536, 2048)?
\nit is not a gpu war but a race towards larger input size.
\nusing larger gpu is only one of the many available methods.
large image for PVTv2-b4 (single fold)
\nLB 0.80
\n input = 1024
\n augmentation = more + downsize/upsize for prostate 6 um
\n PVTv2-b4 (5 level)
LB 0.79
\n input = 768
\n augmentation = more
\n PVTv2-b4 (4 level)
![\"Selection-030\"](\"https://i.ibb.co/MhDgTn2/Selection-030.png\")
"
},
{
"id": 1911181,
"postDate": "2022-08-24T00:52:46.453Z",
"content": "Hi Heng, may I ask what's \"PVTv2\" model? I've been got stuck at 0.8 for quite a while, and have no idea to improve further. Thanks!
", "rawMarkdown": "Hi Heng, may I ask what's \"PVTv2\" model? I've been got stuck at 0.8 for quite a while, and have no idea to improve further. Thanks!" }, { "id": 1911378, "postDate": "2022-08-24T04:08:10.060Z", "content": "paper: PVT v2: Improved Baselines with Pyramid Vision Transformer
\nhttps://github.com/whai362/PVT
all CVT , PVTv2, MIT (segformer's mix transformer) share almost smiliar structure
", "rawMarkdown": "paper: PVT v2: Improved Baselines with Pyramid Vision Transformer\nhttps://github.com/whai362/PVT\n\nall CVT , PVTv2, MIT (segformer's mix transformer) share almost smiliar structure" }, { "id": 1911382, "postDate": "2022-08-24T04:12:29.887Z", "content": "i find a better (?) way to add level to pyramid transformer.
\ne.g. given pretrain weights for 4 level depths = [ 3,3,18,3], you can actually do change to 5 level depths = [3,3,15,5,3].
i.e. you can shorten the third level
", "rawMarkdown": "i find a better (?) way to add level to pyramid transformer.\ne.g. given pretrain weights for 4 level depths = [ 3,3,18,3], you can actually do change to 5 level depths = [3,3,15,5,3].\n\ni.e. you can shorten the third level", "replies": [ { "id": 1914599, "postDate": "2022-08-26T08:01:52.433Z", "content": "\n\ni find a better (?) way to add level to pyramid transformer.
\n
\n e.g. given pretrain weights for 4 level depths = [ 3,3,18,3], you can actually do change to 5 level depths = [3,3,15,5,3].i.e. you can shorten the third level
\n
Hi, @hengck23 ,
\nThank you for the models sharing, it really helps me on the learning of model structure and arrangement with backbones and decoders.
I have a few questions about the above-mentioned depths (maybe the questions is so stupid because of my ignorance):
\nAccording to the change of depths(from [3,3,18,3] to [3,3,15,5,3]), the relevant 'num_stages' will also change from 4 to 5, so the 'embed_dims / num_heads / mlp_ratios / sr_ratios' also need to be adjusted, is that right?
If I used the pre-tained weight (i.e. pvt2_b4), does pre-trained weight still match the new model level?
Thank you again for the selfless sharing.
\nBest Regards
", "rawMarkdown": "> i find a better (?) way to add level to pyramid transformer.\n> e.g. given pretrain weights for 4 level depths = [ 3,3,18,3], you can actually do change to 5 level depths = [3,3,15,5,3].\n> \n> i.e. you can shorten the third level\n\nHi, @hengck23 ,\nThank you for the models sharing, it really helps me on the learning of model structure and arrangement with backbones and decoders.\n\nI have a few questions about the above-mentioned depths (maybe the questions is so stupid because of my ignorance): \n\n1. According to the change of depths(from [3,3,18,3] to [3,3,15,5,3]), the relevant 'num_stages' will also change from 4 to 5, so the 'embed_dims / num_heads / mlp_ratios / sr_ratios' also need to be adjusted, is that right?\n\n2. If I used the pre-tained weight (i.e. pvt2_b4), does pre-trained weight still match the new model level?\n\nThank you again for the selfless sharing. \n\nBest Regards\n" } ] }, { "id": 1911496, "postDate": "2022-08-24T06:14:37.277Z", "content": "Hi @hengck23, thanks for your tips. Though having some experience in designing networks, I still struggle to have a god feeling about which direction to modify an existing network. Any rule of thumb according to your work/research experience?
\nThank you!
we already reached the max accurately , which is limited by the label noise.
\nsince performance is limited by data, there is no gain in designing better network.
however,increase of image size can reduce the predicted mask up scaling error, but there is little additional detection of missed FTU i think.
\nthe next boost will be better data or better network to learned in self-supervsied:
\nWait, the provided HPA images are not fully labeled? Or you mean external data on the internet?
\n\n\nthe next boost will be better data
\n
\n I noticed you mentioned the idea of \"mixup\" or more augmentations in the thread, this also resorts to \"creating better data\" right?
As for self-supervised learning, I'm not an expert in this domain, need to do some homework to understand how it works.
\nThank you, heng!
external data.
\naccording to the papers, all pretraining using external data for transformer would lead to 1 to 2 % improvement.
\nbut is still have to goes back to the issue, how bad are our labels, have we reached the best performance?
I did some large size experiments in cnn and transformers.
\n| net | \nimage size | \ncv | \nlb | \n
|---|---|---|---|
| efficientnet_b5 + unet | \n1536 | \n0.7966 | \n0.80 | \n
| coat_parallel_small(5 level) + daformer_conv1x1 | \n1536 | \n0.8034 | \n0.77 | \n
Both of lb are 5 folds ensemble results.
\nI don't know why my transformer model has such a big gap.
i think it is over fitting or bug.
\nyou should have aux loss for all parallel block output expect the first one. use more augmentation?
cv dice is not a good indicator.
\nhow many iterations/epoch did you train?
check your cv bce loss.
\nat least 1536 cv bce should be better than 1024, which is better than 768.
better bce indicates \"better margin\"
\ncheck the train/valid bce of your unet and coat_parallel_small
\nyou can only add a few new uninitialized serial block layers. we still need pretrain weights for most of the serial block layers/parallel block layers. else i think it will overfit.
\n0.77 seems like a bug. how about try 768 first?
\nor try without swa first.
my parallel block. use is skip first = True
\nhttps://gist.github.com/hengck23/b2c2bcd8de06248add46097ec55ce578#file-my-parallel-blovk
", "rawMarkdown": "my parallel block. use is skip first = True\n\nhttps://gist.github.com/hengck23/b2c2bcd8de06248add46097ec55ce578#file-my-parallel-blovk" }, { "id": 1911673, "postDate": "2022-08-24T08:28:55.990Z", "content": "@hengck23 Thanks for your advice. I will check them.
\nI did more aux and more augmentation and training with 400 epoch.
More aux:
\n`
if self.CFG.aux:\n aux_logits = res['aux_logits']\n for aux_logit in aux_logits:\n loss += 0.2*self.criterion_aux_loss(aux_logit, masks)\n`
\nBasic aug: flip , rotate, transpose, contrast,hsv, noise, etc.
\nMore aug:
\n ElasticTransform,\n GridDistortion,\n RandomGamma,\n Cutout, etc\n
I will try more heavy augmentation.
", "rawMarkdown": "@hengck23 Thanks for your advice. I will check them.\nI did more aux and more augmentation and training with 400 epoch.\n\nMore aux:\n`\n\n if self.CFG.aux:\n aux_logits = res['aux_logits']\n for aux_logit in aux_logits:\n loss += 0.2*self.criterion_aux_loss(aux_logit, masks)\n`\n\nBasic aug: flip , rotate, transpose, contrast,hsv, noise, etc.\nMore aug: \n\n`\n ElasticTransform,\n GridDistortion,\n RandomGamma,\n Cutout, etc\n`\n\nI will try more heavy augmentation.", "votes": 3 }, { "id": 1911677, "postDate": "2022-08-24T08:32:22.637Z", "content": "what is your batch size?
\n\"I did more aux and more augmentation and training with 400 epoch.\"
\n\"I will try more heavy augmentation.\"
i suggest you train and submit a coat small parallel at 768. use this as reference to judge and guide your 1536 training. (the loss curve should be better than 768). you must get 0.78-79 for 768. 1536 will give you +0.02
\nnote
\nswap color channel and gray definitely hurts LB score for my experiemnts
total batch size = 8.
\nI will try it following your opinion.
\ni suggest you train and submit a coat small parallel at 768. use this as reference to judge and guide your 1536 training. (the loss curve should be better than 768). you must get 0.78-79 for 768. 1536 will give you +0.02\n
Really thanks for your kind answer and always learn a lot from your disscusion.
", "rawMarkdown": "total batch size = 8.\nI will try it following your opinion.\n\n`\ni suggest you train and submit a coat small parallel at 768. use this as reference to judge and guide your 1536 training. (the loss curve should be better than 768). you must get 0.78-79 for 768. 1536 will give you +0.02\n`\n\nReally thanks for your kind answer and always learn a lot from your disscusion. " }, { "id": 1912170, "postDate": "2022-08-24T14:55:42.830Z", "content": "\n\n", "rawMarkdown": "@hengck23 \n> external data.\naccording to the papers, all pretraining using external data for transformer would lead to 1 to 2 % improvement.\nUnderstood. \n\n> but is still have to goes back to the issue, how bad are our labels, have we reached the best performance?\nYou're right. Tha't a good question. " }, { "id": 1913332, "postDate": "2022-08-25T09:28:53.363Z", "content": "\nexternal data.
\n
\n according to the papers, all pretraining using external data for transformer would lead to 1 to 2 % improvement.
\n Understood.but is still have to goes back to the issue, how bad are our labels, have we reached the best performance?
\n
\n You're right. Tha't a good question.
my results for tf_efficientnet_b6 smp unet 1536 single fold (fold3)
\nLB =0.80
(i think ensemble will be 0.80~0.81)
\nmaybe b6 is better than b5 or maybe my augmentation is better
Great job!
\n\nmaybe b6 is better than b5 or maybe my augmentation is better\n
Maybe augmentation more important, I try tf_efficientnet_b7 smp unet 1536 improve cv but drop lb seriously.
\nI guess the lb limit is 0.81 with single model( single fold or ensemble n folds).
", "rawMarkdown": "@hengck23 \n\nGreat job! \n\n`\nmaybe b6 is better than b5 or maybe my augmentation is better\n`\n\nMaybe augmentation more important, I try tf_efficientnet_b7 smp unet 1536 improve cv but drop lb seriously.\n\nI guess the lb limit is 0.81 with single model( single fold or ensemble n folds).", "votes": 1 }, { "id": 1914324, "postDate": "2022-08-26T02:12:35.947Z", "content": "\"Maybe augmentation more important, \"
\ntoo much and too little augmentation is no good.it is easy to obtain CV 0.80.
\ni once made a mistake. i load rubbish ground truth mask for external data and train together with kaggle train data.
\ni can still obtain CV 0.80 for kaggle validation in that case (but of course LB score is not good)
it is important to visualize prediction on external data.
", "rawMarkdown": "\"Maybe augmentation more important, \"\n\ntoo much and too little augmentation is no good.it is easy to obtain CV 0.80.\ni once made a mistake. i load rubbish ground truth mask for external data and train together with kaggle train data.\ni can still obtain CV 0.80 for kaggle validation in that case (but of course LB score is not good)\n\nit is important to visualize prediction on external data.\n", "votes": 1 }, { "id": 1914859, "postDate": "2022-08-26T13:37:41.830Z", "content": "\n\n\n\n'embed_dims / num_heads / mlp_ratios / sr_ratios' also need to be adjusted, is that right?
\n
\n for the last level i just use previous level values.
\n the last level is not very deep, so i don't think it matters.
from another paper:
\n![\"Selection-088\"](\"https://i.ibb.co/M6P3sgb/Selection-088.png\")
\n\n\n\nIf I used the pre-tained weight (i.e. pvt2_b4), does pre-trained weight still match the new model level?
\n
\n you can only load pretrain model up to old levels. the new level are trained from scratch. hence the new level cannot be very deep because you don't have pretrain weight.
what i am doing is like extending resnet backbone from 4 layers to 5,6 layers (p6,p7) to detect large object in FPN
", "rawMarkdown": ">> 'embed_dims / num_heads / mlp_ratios / sr_ratios' also need to be adjusted, is that right?\nfor the last level i just use previous level values. \nthe last level is not very deep, so i don't think it matters.\n\nfrom another paper:\n\n\n>>If I used the pre-tained weight (i.e. pvt2_b4), does pre-trained weight still match the new model level?\nyou can only load pretrain model up to old levels. the new level are trained from scratch. hence the new level cannot be very deep because you don't have pretrain weight.\n\nwhat i am doing is like extending resnet backbone from 4 layers to 5,6 layers (p6,p7) to detect large object in FPN\n"
},
{
"id": 1914861,
"postDate": "2022-08-26T13:40:40.460Z",
"content": "since the new layer is small, one can simply use conv instead of trasnformer
", "rawMarkdown": "since the new layer is small, one can simply use conv instead of trasnformer" }, { "id": 1915442, "postDate": "2022-08-27T01:43:00.267Z", "content": "Thank you for the fast reply. The new layer advice helps me a lot.
", "rawMarkdown": "Thank you for the fast reply. The new layer advice helps me a lot. " }, { "id": 1915512, "postDate": "2022-08-27T03:07:33.960Z", "content": "you train a normal network first.
\nthen freeze (or partially freeze) the trained network and train only the new layers, etc.
\n\n\n\ntraining with 400 epoch.
\n
it seems to me that 400 is too many.
\ni use about 200 to 230 for batch size 4
@hengck23
\nThanks for your tips.
\n I also observed this phenomenon, after epochs greater than 300, the avg_loss is increasing.
The thing about larger image with efficient net or resnet is… I think it is suggested to use 224 image size with resnet18 and effnet b0 where 240 size with efficient net b1 and goes to 700 image size for efficient net b7. Though CNN would work with any size of image still we need to create much larger efficient net like b9 Or b10( if exist)
\nDon't know if @yingpengchen if efficient net with 0.80 is most probably overfit
\nCorrect me if I am thinking in wrong direction.
@bibhabasumohapatra it's possible, but currently we don't have such a trained model like b9 or b10.
\n\nDon't know if @yingpengchen if efficient net with 0.80 is most probably overfit\n
It's hard to say, but compared to the 768 lb+0.02 improvement for me.
", "rawMarkdown": "@bibhabasumohapatra it's possible, but currently we don't have such a trained model like b9 or b10.\n\n`\nDon't know if @yingpengchen if efficient net with 0.80 is most probably overfit\n`\n\nIt's hard to say, but compared to the 768 lb+0.02 improvement for me.", "votes": 1 }, { "id": 1926789, "postDate": "2022-09-05T06:04:35.247Z", "content": "Wait, you guys are training for 300-400 epochs? My models converge after 30 epochs.
", "rawMarkdown": "Wait, you guys are training for 300-400 epochs? My models converge after 30 epochs.", "votes": 4 }, { "id": 1939399, "postDate": "2022-09-14T17:05:52.030Z", "content": "@gunesevitan exactly . . . 30 to max 35 is fine almost 45 . . . my model converges. never thought of 300 epochs . .. 😯😯
", "rawMarkdown": "@gunesevitan exactly . . . 30 to max 35 is fine almost 45 . . . my model converges. never thought of 300 epochs . .. 😯😯 " }, { "id": 1944014, "postDate": "2022-09-18T01:48:05.853Z", "content": "Hi, I am a beginner. I would like to ask what exactly \"augmentation = more\" is.
\nThe gap between my cv and lb is huge (0.7 vs 0.43).
\nA kind person told me it was because I didn't use data augmentation during training and testing.
\nI would like to ask if I can improve my score by applying more data augmentation like these.
\nOr do you have any better suggestions to improve this situation? thank you very much🙏🙏
as shown in the CV-LB graph at the top, all my CV-HPA, LB-HPA,LB-Hubmap,LB-public has converged!
\nI was right that the upper limit is about 0.80 without external data.
this could be the bronze medal limit after 2 months
\nI choose a variety of encoder, decoder, optimizer. These are not chosen at random but target at the root of problems
", "rawMarkdown": "as shown in the CV-LB graph at the top, all my CV-HPA, LB-HPA,LB-Hubmap,LB-public has converged!\nI was right that the upper limit is about 0.80 without external data.\n\nthis could be the bronze medal limit after 2 months\n\n---\n\nI choose a variety of encoder, decoder, optimizer. These are not chosen at random but target at the root of problems", "votes": 3, "replies": [ { "id": 1872506, "postDate": "2022-07-27T03:32:05.373Z", "content": "Hi hengck23,
\nHow did you reach the 0.8 upper limit? Why isn't it 1? I saw your probe post further down with the guess, but I couldn't tell how you reached that conclusion. Is it some out of distribution thing? Thanks
", "rawMarkdown": "Hi hengck23,\n\nHow did you reach the 0.8 upper limit? Why isn't it 1? I saw your probe post further down with the guess, but I couldn't tell how you reached that conclusion. Is it some out of distribution thing? Thanks" }, { "id": 1872513, "postDate": "2022-07-27T03:43:19.347Z", "content": "it cannot be 1 becuase of label noise.
\na quick guide is train dice >= local validation dice >=leaderboard dice
train dice is the upper limit at the point before overfitting. you can train you model for very long iterations to find the point of overfitting (i.e. when your validation dice becomes a V shape.)
", "rawMarkdown": "it cannot be 1 becuase of label noise.\na quick guide is train dice >= local validation dice >=leaderboard dice\n\ntrain dice is the upper limit at the point before overfitting. you can train you model for very long iterations to find the point of overfitting (i.e. when your validation dice becomes a V shape.)\n\n", "votes": 2 }, { "id": 1874056, "postDate": "2022-07-28T04:04:48.503Z", "content": "Thank you for the quick, and helpful reply!
", "rawMarkdown": "Thank you for the quick, and helpful reply!" } ] }, { "id": 1867021, "postDate": "2022-07-23T01:02:51.697Z", "content": "what is the normalized size of hidden test images?
\nin order to apply transformer segmentation, i need to have a rough ideal of the max and min input size of the test image.
\n(i need to decide absolute/relative positional encoding, token patch size ….)
\nhence i do a probe.
the results: all test image are within 0.8 to 1.2 of normalized size.
\n test_df = pd.read_csv('../input/hubmap-organ-segmentation/test.csv') \n test_df.loc[:,'norm_size']= test_df.pixel_size/0.4 * test_df.img_height/3000 \n #assume img_height==img_width \n max_norm_size = test_df.norm_size.max() \n min_norm_size = test_df.norm_size.min() \nbelow two conditions are probed true:
\nif (min_norm_size>=0.8) and (min_norm_size<1.0): \nif (max_norm_size>=1.0) and (max_norm_size<1.2): \nnote we can also deuce that smallest 160x160 test images mentioned refer to prostate
\nimages with pixel size 6.2630 um. (you should do resize artifacts augmentation for this case in training)
largest 4500x4500 test image refer to large intestine with pixel size 0.229 um
", "rawMarkdown": "what is the normalized size of hidden test images?\n\nin order to apply transformer segmentation, i need to have a rough ideal of the max and min input size of the test image.\n(i need to decide absolute/relative positional encoding, token patch size ....)\nhence i do a probe.\n\nthe results: all test image are within 0.8 to 1.2 of normalized size.\n\n```\n test_df = pd.read_csv('../input/hubmap-organ-segmentation/test.csv')\t\t\t\t\t\t\n test_df.loc[:,'norm_size']= test_df.pixel_size/0.4 * test_df.img_height/3000 \t\t\t\t\t\t\n \t\t\t\t#assume img_height==img_width\t\t\n max_norm_size = test_df.norm_size.max()\t\t\t\t\t\t\n min_norm_size = test_df.norm_size.min()\t\t\t\t\t\t\n\n```\n\nbelow two conditions are probed true:\n\n``` \nif (min_norm_size>=0.8) and (min_norm_size<1.0): \nif (max_norm_size>=1.0) and (max_norm_size<1.2): \n```\n\nnote we can also deuce that smallest 160x160 test images mentioned refer to prostate\nimages with pixel size 6.2630 um. (you should do resize artifacts augmentation for this case in training)\n\nlargest 4500x4500 test image refer to large intestine with pixel size 0.229 um", "votes": 3, "replies": [ { "id": 1868097, "postDate": "2022-07-23T17:29:41.347Z", "content": "\"assume img_height==img_width\"
\nThis passed:
\nif (df['img_width']==df['img_height']).all():
All images are square
", "rawMarkdown": "\"assume img_height==img_width\" \nThis passed:\nif (df['img_width']==df['img_height']).all():\n\nAll images are square", "votes": 1 } ] }, { "id": 1864027, "postDate": "2022-07-20T17:14:41.743Z", "content": "maybe this also works for our problem: labelled kaggle train images + unlabelled download external data
\nhttps://www.youtube.com/watch?v=FTmvdARsbnI
\nMedAI Session 25: Training medical image segmentation models with less labeled data | Sarah Hooper
this is mentioned in the video. i put it into picture.
\nstage1: train normally in supervised to get an initial model (and baseline results)
\nstage2: train joint supervised and unsupervised (and see if you get improved results)
![\"https://i.ibb.co/t2mP3Ns/Selection-118.png\"](\"https://i.ibb.co/t2mP3Ns/Selection-118.png\")
there is a final stage 3 which uses pesudo labels
", "rawMarkdown": "this is mentioned in the video. i put it into picture.\nstage1: train normally in supervised to get an initial model (and baseline results)\nstage2: train joint supervised and unsupervised (and see if you get improved results)\n\n\n\nthere is a final stage 3 which uses pesudo labels", "votes": 1 } ] }, { "id": 1881977, "postDate": "2022-08-03T01:42:43.543Z", "content": "you may think that the results of Hunmap are inferior to that of HPA because of colors of the stains.
\nBut this may not be true.
You can verify this by changing the color of your validation HPA stains (e.g. HSV shift, convert to gray, stain color transfer) and compare with results of the original colored HPA images. You should see similar relative performance in your Lb-HPA and Lb-Humap score.
\nLikewise for other factors like pixel size in um, slice thickness, resolution, etc …
\nHubmap are real lab tissue slides. there are many more issues, see below.
\nthis is left as an exercise for you: \"how to prove that if these issues actually exist in the hidden test set?\"
![\"https://i.ibb.co/DgTsVxk/Selection-164.png\"](\"https://i.ibb.co/DgTsVxk/Selection-164.png\")
even if these issues does not exist in the hidden test set, it will still boost your chance in the \"Judges prizes\" if analysis and results are presented to show your robustness of your model against real artifacts.
\non a side note, when i develop commercial computer vision model for by clients, \"black box\" testing is common. We do not see the test set at all. What we know is the feedback of the results (e.g. score against other bidding vendors, competitor products). So we have devised a way to fault find the issues even if we do not see the images. e.g.
\nexternal data may not help you to detect more FTU, but they are definitely of great help to remove false positive and show your model weakness and issues
\nthe score of LB-hubmap-public and LB-hubmap-private may be closer then what your think (i.e. little shakeup). it is likely that a sample tissues has many cropped slide samples. They are then split between public and private set.
", "rawMarkdown": "you may think that the results of Hunmap are inferior to that of HPA because of colors of the stains.\nBut this may not be true.\n\nYou can verify this by changing the color of your validation HPA stains (e.g. HSV shift, convert to gray, stain color transfer) and compare with results of the original colored HPA images. You should see similar relative performance in your Lb-HPA and Lb-Humap score.\n\nLikewise for other factors like pixel size in um, slice thickness, resolution, etc ...\n\n---\n\nHubmap are real lab tissue slides. there are many more issues, see below.\nthis is left as an exercise for you: \"how to prove that if these issues actually exist in the hidden test set?\"\n\n\n\n\n---\n\neven if these issues does not exist in the hidden test set, it will still boost your chance in the \"Judges prizes\" if analysis and results are presented to show your robustness of your model against real artifacts.\n\n---\n\non a side note, when i develop commercial computer vision model for by clients, \"black box\" testing is common. We do not see the test set at all. What we know is the feedback of the results (e.g. score against other bidding vendors, competitor products). So we have devised a way to fault find the issues even if we do not see the images. e.g.\n\n- using auto-encoders, etc to measure out-of distributions from your train set\n- apply your model on large test set (including out of distributions). train a classifier to classify the type of errors. then you can map error to input image.\n\n---\n\nexternal data may not help you to detect more FTU, but they are definitely of great help to remove false positive and show your model weakness and issues\n\n---\n\nthe score of LB-hubmap-public and LB-hubmap-private may be closer then what your think (i.e. little shakeup). it is likely that a sample tissues has many cropped slide samples. They are then split between public and private set.", "votes": 4 }, { "id": 1875484, "postDate": "2022-07-29T04:07:45.280Z", "content": "beware!!!!
\nthe distribution of the organ in Hubmap is not what you think
\n![\"https://i.ibb.co/k9FBv0V/Selection-136.png\"](\"https://i.ibb.co/k9FBv0V/Selection-136.png\")
this confirms with my observation that the recent improvement in my LB score especially from 0.76 to 8.80 actually comes from prostate, lung and spleen.
\nit is also noted that segformer is strong in lung.
\nfurther, the FTU object size for spleen is larger then the rest. one have to take cares of scale. if you are using tiling, you need to check if your tile can contain the largest FTU object or not
", "rawMarkdown": "this confirms with my observation that the recent improvement in my LB score especially from 0.76 to 8.80 actually comes from prostate, lung and spleen.\n\nit is also noted that segformer is strong in lung.\n\nfurther, the FTU object size for spleen is larger then the rest. one have to take cares of scale. if you are using tiling, you need to check if your tile can contain the largest FTU object or not", "votes": 3 } ] }, { "id": 1870646, "postDate": "2022-07-25T17:30:06.703Z", "content": "i realise one very important note:
\nupdate of results LB 0.72:
\n![\"https://i.ibb.co/rmnSNH0/Selection-060.png\"](\"https://i.ibb.co/rmnSNH0/Selection-060.png\")
![\"https://i.ibb.co/wrx84gs/Selection-096.png\"](\"https://i.ibb.co/wrx84gs/Selection-096.png\")
surprise! surprise! the probe by organ is not what i would expect …
\ni wonder anyone has probed the distribution of the organ and willing to share?
\nthe domain shift is not as large as expected. this is good news.
\nnot reported in the table are training score : ~0.85
\ni estimate final leader board at the last day of competition:
\npublic LB : 0.80 (HPA=0.81, Hubmap=0.80)
\nprivate LB : >=0.79 (Hubmap>=0.79)
what is the key to improvement?
\nreceptive field size, scale?
\ndid you interpret results correctly?
the reason why we try different models is to test if their responses are similar to the same data.
\nthis reveal data characteristics … more to come : other segmentation architecture besides unet, etc
![\"https://i.ibb.co/7yVHRcz/Selection-095.png\"](\"https://i.ibb.co/7yVHRcz/Selection-095.png\")
do compare the results with hubmap paper:
\nhttps://www.biorxiv.org/content/10.1101/2021.11.09.467810v1.full.pdf
Supplementary table 5. Algorithm performance. (without domain shift)
\nKaggle reproduced (kidney) :0.93~0.97
\nTransfer learning (trained on kidney & colon, tested on colon) : ~0.88
gives you an idea of effect of domain shift (ours), annotation error, etc
", "rawMarkdown": "do compare the results with hubmap paper:\nhttps://www.biorxiv.org/content/10.1101/2021.11.09.467810v1.full.pdf\n\nSupplementary table 5. Algorithm performance. (without domain shift)\nKaggle reproduced (kidney) :0.93~0.97\nTransfer learning (trained on kidney & colon, tested on colon) : ~0.88 \n\ngives you an idea of effect of domain shift (ours), annotation error, etc" }, { "id": 1861472, "postDate": "2022-07-19T03:58:49.477Z", "content": "5-fold unet-efficientb7 (lb=0.75)
\nresults on the given public test spleen image
![\"https://i.ibb.co/hBMg88Q/Selection-101.png\"](\"https://i.ibb.co/hBMg88Q/Selection-101.png\")
ccf-research-kaggle-2021 : hubmap CL_HandE_1234_B004_bottomleft.tiff
\n(but i don't know what is the um per pixel size for the datset. can anyone advise?)
![\"https://i.ibb.co/YcfHVLD/Selection-102.png\"](\"https://i.ibb.co/YcfHVLD/Selection-102.png\")
kaggle 2021 hubmap kidney image
\n![\"https://i.ibb.co/r5Y8gbW/Selection-104.png\"](\"https://i.ibb.co/r5Y8gbW/Selection-104.png\")
how do you ensemble 5 fold seg models? you get average? what is your method?
", "rawMarkdown": "how do you ensemble 5 fold seg models? you get average? what is your method?" }, { "id": 1862544, "postDate": "2022-07-19T19:29:04.253Z", "content": "just average
", "rawMarkdown": "just average", "votes": 1 }, { "id": 1863905, "postDate": "2022-07-20T15:30:31.547Z", "content": "added uper-net-convnext-large to the ensemble
\nthe very best model
\n1) upernet-convnext-large : LB=0.75
\n2) unet-efficientb7: LB=0.75
\n3) upernet-swin-tx-small: LB=0.75
Hi @hengck23
\nAs always, it's a joy to read your analysis and discussion. May I know for the prediction you showed for Spleen what was your threshold strategy? did you learn the threshold on all five fold? or hard-coded one?
", "rawMarkdown": "Hi @hengck23 \n\nAs always, it's a joy to read your analysis and discussion. May I know for the prediction you showed for Spleen what was your threshold strategy? did you learn the threshold on all five fold? or hard-coded one?" }, { "id": 1866959, "postDate": "2022-07-22T22:16:13.407Z", "content": "learned my experiments , turns out to be about 0.5 for organ spleen.
\nensemble such that the curve is almost flat over the maximum and insensitive to threshold
What image size are you using?
", "rawMarkdown": "What image size are you using?" }, { "id": 1870275, "postDate": "2022-07-25T12:42:31.797Z", "content": "768
", "rawMarkdown": "768 " }, { "id": 1870537, "postDate": "2022-07-25T16:11:26.737Z", "content": "\n\nccf-research-kaggle-2021 : hubmap CL_HandE_1234_B004_bottomleft.tiff
\n
\n (but i don't know what is the um per pixel size for the datset. can anyone advise?)
Is that dataset an extension of HuBMAP Kidney Segmentation? Their raw dimensions are very close to upper limit of HuBMAP image dimensions in this competition. I think large intestine images could be very similar.
", "rawMarkdown": "> ccf-research-kaggle-2021 : hubmap CL_HandE_1234_B004_bottomleft.tiff\n(but i don't know what is the um per pixel size for the datset. can anyone advise?)\n\nIs that dataset an extension of HuBMAP Kidney Segmentation? Their raw dimensions are very close to upper limit of HuBMAP image dimensions in this competition. I think large intestine images could be very similar." } ] }, { "id": 1951266, "postDate": "2022-09-23T00:38:31.290Z", "content": "I've learned a lot things from your post and comments. I deeply thank you very much for sharing your experiments and results that are very insightful and helpful.
", "rawMarkdown": "I've learned a lot things from your post and comments. I deeply thank you very much for sharing your experiments and results that are very insightful and helpful.", "votes": 1 }, { "id": 1937218, "postDate": "2022-09-13T10:52:24.920Z", "content": "Hello,
\nCould anyone help me understand what aux-loss means? Is it like a supportive loss function for organ type classification? Any notebook/snippet I can refer to?
Hi, it's usually used to stabilize the training and to add more regularization. You compute the loss between intermediate outputs (from encoder stages) and your downsampled (used in the notebooks by OP of this discussion, many thanks, @hengck23) masks. Of course, you can adjust and modify everything based on what kind of constraint you want to put or emphasize more
", "rawMarkdown": "Hi, it's usually used to stabilize the training and to add more regularization. You compute the loss between intermediate outputs (from encoder stages) and your downsampled (used in the notebooks by OP of this discussion, many thanks, @hengck23) masks. Of course, you can adjust and modify everything based on what kind of constraint you want to put or emphasize more ", "votes": 1 }, { "id": 1937338, "postDate": "2022-09-13T12:30:43.887Z", "content": "it is an old method introduced by google inception model in the early days ….
\n![\"https://drek4537l1klr.cloudfront.net/ferlitsch/Figures/CH06_F03_Ferlitsch.png\"](\"https://drek4537l1klr.cloudfront.net/ferlitsch/Figures/CH06_F03_Ferlitsch.png\")
you can also google for:
\ndeep supervised, auxiliary loss, segmentation, mmseg auxiliary loss
in the old days, because batchnom were not invented, there is problem of diminishing gradients. Hence you insert auxiliary loss to back-propagates gradients from the middle.
\nfor today when the model is too strong (like transformer), there is little gradients from the top because you have almost zero loss. in this case you can make the problem more difficult (i,e, larger loss, more gradient) by asking the model to make prediction using only up the intermediate layers
\nhttps://discuss.pytorch.org/t/how-to-implement-deeply-supervised-network/6203
\n ![\"https://discuss.pytorch.org/uploads/default/original/2X/f/f3722aaedf882e62f17b316805f8170ddaada1e2.png\"](\"https://discuss.pytorch.org/uploads/default/original/2X/f/f3722aaedf882e62f17b316805f8170ddaada1e2.png\")
in either case, the results will be a smoother loss landscape and better generalization (less over fitting)
", "rawMarkdown": "it is an old method introduced by google inception model in the early days ....\n \n\nyou can also google for:\ndeep supervised, auxiliary loss, segmentation, mmseg auxiliary loss\n\n\n---\n\nin the old days, because batchnom were not invented, there is problem of diminishing gradients. Hence you insert auxiliary loss to back-propagates gradients from the middle.\n\nfor today when the model is too strong (like transformer), there is little gradients from the top because you have almost zero loss. in this case you can make the problem more difficult (i,e, larger loss, more gradient) by asking the model to make prediction using only up the intermediate layers\n\nhttps://discuss.pytorch.org/t/how-to-implement-deeply-supervised-network/6203\n  \n\n\n---\n\nin either case, the results will be a smoother loss landscape and better generalization (less over fitting)", "votes": 4 }, { "id": 1945817, "postDate": "2022-09-19T12:23:26.220Z", "content": "Thanks for providing these resources. I was able to implement it. Wanted to ask one more question.
\nCan I change the aux loss in b/w training? For example, take from some layer for first few epochs and then shift taux loss o other layer?
\nIs this logically sound?
if you are using my code, beware!!!!
\n![\"Selection-083\"](\"https://i.ibb.co/7JTJ0hw/Selection-083.png\")
the simple fusion of the different levels uses upscale of 4x, 8x, 16x, 32x
\nit is obvious that it causes the prediction to be blocky.
\n(despite the blocky head, LB is still 0.82 after ensemble)
i need a better decoder head
", "rawMarkdown": "if you are using my code, beware!!!!\n\n\nthe simple fusion of the different levels uses upscale of 4x, 8x, 16x, 32x\nit is obvious that it causes the prediction to be blocky.\n(despite the blocky head, LB is still 0.82 after ensemble)\n\ni need a better decoder head ",
"votes": 1,
"replies": [
{
"id": 1915206,
"postDate": "2022-08-26T18:36:03.570Z",
"content": "New record for 768?
\nsingle fold (fold3)
\npvt-v2v-v4 + daformer3x3 (fuse of 4 layer) / 4x upscale : LB 0.79
\npvt-v2v-v4 + smp-unet (5 layer decoder) / 1x upscale: LB 0.80
code: apply smp unet to transformer pvt-v2-b4 and remove 4x upscaling restriction
\nhttps://gist.github.com/hengck23/15e26a84e747f7ddd12cc0ea215e9301
![\"Selection-090\"](\"https://i.ibb.co/41qQyfx/Selection-090.png\")
",
"votes": 2
},
{
"id": 1915239,
"postDate": "2022-08-26T19:30:35.650Z",
"content": "i have a feeling that this structure works well. i will use it for another competition.
\nhere, transformer block is treated as a \"convolution with super-large nonlinear kernel\".
add transformer with conv will be a good regulariser
\n![\"Selection-091\"](\"https://i.ibb.co/4767Kgk/Selection-091.png\")
",
"votes": 2
},
{
"id": 1915524,
"postDate": "2022-08-27T03:39:38.053Z",
"content": "i took my previous trained encoder, freeze it and just trained a new unet decoder. training is very fast and get better results.
\nold:
\npvt-v2-b4-level5 + daformer3x3-level5 decoder @1024 : LB 0.80 (CV : 0.797)
new:
\npvt-v2-b4-level5 + conv + unet-level6 decoder @1024 : LB 0.81 (CV : 0.799)
this also means that by freezing,
\nyou can in fact combine a couple of your old encoders (e.g. swin + pvtv2) and train a new decoder on combined freezed (or partial breezed) feature.
\nyou can also combine old trained transformer with new untrained CNN encoder, etc …
\nso there is a lot of room for modeling !!!
", "rawMarkdown": "i took my previous trained encoder, freeze it and just trained a new unet decoder. training is very fast and get better results.\n\nold: \npvt-v2-b4-level5 + daformer3x3-level5 decoder @1024 : LB 0.80 (CV : 0.797) \n\nnew: \npvt-v2-b4-level5 + conv + unet-level6 decoder @1024 : LB 0.81 (CV : 0.799)\n\n---\n\nthis also means that by freezing,\n\nyou can in fact combine a couple of your old encoders (e.g. swin + pvtv2) and train a new decoder on combined freezed (or partial breezed) feature.\n\nyou can also combine old trained transformer with new untrained CNN encoder, etc ...\n\nso there is a lot of room for modeling !!!\n ", "votes": 2 }, { "id": 1916122, "postDate": "2022-08-27T16:42:30.957Z", "content": "does your models using stain normalization?
", "rawMarkdown": "does your models using stain normalization?" }, { "id": 1916225, "postDate": "2022-08-27T17:54:10.513Z", "content": "no. i use stain augmentation only
", "rawMarkdown": "no. i use stain augmentation only", "votes": 1 }, { "id": 1928606, "postDate": "2022-09-06T14:33:45.277Z", "content": "@hengck23 @kfk42kfk May I ask what's the meaning of \"stain normalization\" & “stain augmentation”?
", "rawMarkdown": "@hengck23 @kfk42kfk May I ask what's the meaning of \"stain normalization\" & “stain augmentation”?", "votes": 2 }, { "id": 1934332, "postDate": "2022-09-11T10:03:42.077Z", "content": "I think, but not necessarily correct, you may change pixel-size or tissue thickness for HPA to Hubmap. You may refer to https://github.com/Peter554/StainTools.
", "rawMarkdown": "I think, but not necessarily correct, you may change pixel-size or tissue thickness for HPA to Hubmap. You may refer to https://github.com/Peter554/StainTools." }, { "id": 1956628, "postDate": "2022-09-26T14:33:43.570Z", "content": "I guess this blocky contour is caused by MixUpSample which chooses nearest as the mode of interpolation.
", "rawMarkdown": "I guess this blocky contour is caused by MixUpSample which chooses nearest as the mode of interpolation." } ] }, { "id": 1894634, "postDate": "2022-08-11T16:15:13.340Z", "content": "one possible replacement for segformer
\n (they are a few more to come later …)
![\"https://i.ibb.co/3s93QwN/Selection-075.png\"](\"https://i.ibb.co/3s93QwN/Selection-075.png\")
\nfor model, refer to https://www.kaggle.com/code/hengck23/lb-0-78-coat-and-simple-3x3-conv-fusion
\nweights/training are not shared
PVT v2 also works the same as segformer. i gave smiliar results on local CV
\n![\"Selection-071\"](\"https://i.ibb.co/2t52NLc/Selection-071.png\")
![\"Selection-072\"](\"https://i.ibb.co/L1bDwBP/Selection-072.png\")
\n\n"
},
{
"id": 1895217,
"postDate": "2022-08-12T02:58:34.540Z",
"content": "Quite impressive work! Did you pre-train with unsupervised learning on external data?
", "rawMarkdown": "Quite impressive work! Did you pre-train with unsupervised learning on external data?" }, { "id": 1895333, "postDate": "2022-08-12T04:44:16.117Z", "content": "no, not yet
", "rawMarkdown": "no, not yet" }, { "id": 1895557, "postDate": "2022-08-12T08:07:21.617Z", "content": "list of conv postional embedding vision transformer that works with conv3x3 fusion layer as decoder
\n[1] PVT v2: Improved Baselines with Pyramid Vision Transformer
\n[2] Co-Scale Conv-Attentional Image Transformers - ICCV 2021 (oral)
\n[3] CvT: Introducing Convolutions to Vision Transformers. (needs aspp as decoder because it only has 3 level output)
\n[4] CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention - ICRL 2022
I cannot make the below to work.
\ni think they stills works but I haven't find the right hyper-parameters or they need stronger decoder or there are some bug in my code ….
i done only partial experiment for the below. They should also work.
\nother promising work that i did not try for various reason
\nCMT: Convolutional Neural Networks Meet Vision Transformers - CVPR 2022 (it uses its own framework and not pytorch)
HRViT : Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (i think the mixed-scale convolutional feedforward
\nnetworks (MixCFN) is superior. but the author did not released pretrained model)
Learning Patch-to-Cluster Attention in Vision Transformer (no source code yet)
other candidate list
\nreference: https://github.com/cmhungsteve/Awesome-Transformer-Attention#conv--transformer
", "rawMarkdown": "list of conv postional embedding vision transformer that works with conv3x3 fusion layer as decoder\n\n[1] PVT v2: Improved Baselines with Pyramid Vision Transformer \n[2] Co-Scale Conv-Attentional Image Transformers - ICCV 2021 (oral)\n[3] CvT: Introducing Convolutions to Vision Transformers. (needs aspp as decoder because it only has 3 level output)\n[4] CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention - ICRL 2022\n\n----\n\nI cannot make the below to work.\ni think they stills works but I haven't find the right hyper-parameters or they need stronger decoder or there are some bug in my code ....\n\n- UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning- ICRL 2022 \n- DaViT: Dual Attention Vision Transformers \n\n---\n\ni done only partial experiment for the below. They should also work. \n\n- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows (this is the only swin variant that works. The other swinv1 and v2 have performer drop when switch from upernet to simple conv3x3 decoder)\n\n---\n\nother promising work that i did not try for various reason\n\n- CMT: Convolutional Neural Networks Meet Vision Transformers - CVPR 2022 (it uses its own framework and not pytorch)\n\n- HRViT : Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (i think the mixed-scale convolutional feedforward\nnetworks (MixCFN) is superior. but the author did not released pretrained model)\n\n- Learning Patch-to-Cluster Attention in Vision Transformer (no source code yet)\n\n---\nother candidate list\n\n- Contextual Transformer Networks for Visual Recognition\n- Local-to-Global Self-Attention in Vision Transformers\n- SepViT: Separable Vision Transformer\n- MaxViT: Multi-Axis Vision Transformer\n- ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer\n- Vision Transformers with Hierarchical Attention\n\nreference: https://github.com/cmhungsteve/Awesome-Transformer-Attention#conv--transformer", "votes": 1 } ] }, { "id": 1892293, "postDate": "2022-08-10T02:44:40.460Z", "content": "oh, interactive segformer!
\nSegFormer: Interactive Segmentation via Transformerswith Application to 3D Knee MR Images
\npaper: https://arxiv.org/pdf/2112.11325v6.pdf
for the use in this competition, refer to
\nhttps://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/341130#1892285
![\"https://i.ibb.co/HX2JbNq/Selection-094.png\"](\"https://i.ibb.co/HX2JbNq/Selection-094.png\")
![\"https://www.cs.toronto.edu/~garyleung/hila/resources/performance_graph.png\"](\"https://www.cs.toronto.edu/~garyleung/hila/resources/performance_graph.png\")
\nhttps://www.cs.toronto.edu/~garyleung/hila/
\nWe add HILA into SegFormer and the Swin Transformer and show notable improvements in accuracy in semantic segmentation with fewer parameters and FLOPS.
i am not sure if this would be an issue:
\n![\"https://i.ibb.co/VxcJHrM/Selection-140.png\"](\"https://i.ibb.co/VxcJHrM/Selection-140.png\")
most slide you see in HPA are ideal . in actual lab, there will be quite some artifacts.
\nIt is expected Hubmap slides will have some artifacts?
![\"https://i.ibb.co/JQR2TqS/Selection-151.png\"](\"https://i.ibb.co/JQR2TqS/Selection-151.png\")
you probably need this here (since we are dealing with shifted domain private LB data) :
\npaper: WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS
\nhttps://arxiv.org/pdf/2106.01548.pdf
\"The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations.\"
\ndo read the section on test on different domain: \"Better robustness. We also evaluate the models’ robustness using ImageNet-R (Hendrycks et al., 2020) and ImageNet-C (Hendrycks & Dietterich, 2019) and find even bigger impacts of the smoothed loss landscapes.\"
\ntake away message: it is not about early stopping …. it is about flatten your loss landscape
", "rawMarkdown": "you probably need this here (since we are dealing with shifted domain private LB data) :\n\npaper: WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS\nhttps://arxiv.org/pdf/2106.01548.pdf\n\n\"The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations.\"\n\ndo read the section on test on different domain: \"Better robustness. We also evaluate the models’ robustness using ImageNet-R (Hendrycks et al., 2020) and ImageNet-C (Hendrycks & Dietterich, 2019) and find even bigger impacts of the smoothed loss landscapes.\"\n\n---\n\ntake away message: it is not about early stopping .... it is about flatten your loss landscape", "votes": 1 }, { "id": 1869703, "postDate": "2022-07-25T01:45:06.240Z", "content": "Adversial network perturbation works well for NLP transformer. It improve robustness (e.g. spelling errors) and score. There are many kaggle code for the recent kaggle competitions for the feedback, US-patent for it.
\nI wonder if it also work for histology tissue images?
\ne.g. [paper] An Adversarial Perturbation Oriented Domain Adaptation Approach for Semantic Segmentation
", "rawMarkdown": "Adversial network perturbation works well for NLP transformer. It improve robustness (e.g. spelling errors) and score. There are many kaggle code for the recent kaggle competitions for the feedback, US-patent for it.\n\nI wonder if it also work for histology tissue images?\n\ne.g. [paper] An Adversarial Perturbation Oriented Domain Adaptation Approach for Semantic Segmentation", "votes": 1 }, { "id": 1865226, "postDate": "2022-07-21T16:32:11.620Z", "content": "you can self-supervised CNN (like transformer) too!
\npaper: Corrupted Image Modeling for Self-Supervised Visual Pre-Training
\nhttps://arxiv.org/pdf/2202.03382.pdf
\"ResNet-50. We demonstrate that CIM can also pre-train a high-capacity ResNet-50 model with the fewest possible
\nmodifications from the ViT pre-training settings that can achieve compelling fine-tuning performances on ImageNet1K.\"
an interesting cvpr 2022 oral paper:
\nSource-Free Object Detection by Learning to Overlook Domain Style
Source-free object detection (SFOD) needs to adapt a detector pre-trained on a labeled source domain to a target
\ndomain, with only unlabeled training data from the target domain. …
learning threshold
\nwhat is show: for 3 fold on validation HPA dataset
\n(but what we need is similar graph on Hubmap dataset …. many probes to be made !!!)
\n![\"https://i.ibb.co/kS8D8VM/Selection-059.png\"](\"https://i.ibb.co/kS8D8VM/Selection-059.png\")
rather than cv/lb correlation, measure robustness as well.
\nif your lb dice is robust against threshold, you are probably safe ?
![\"Selection-076\"](\"https://i.ibb.co/6wmsbPN/Selection-076.png\")
\n![\"Selection-075\"](\"https://i.ibb.co/Gn10Jbd/Selection-075.png\")
\n",
"votes": 1
}
]
},
{
"id": 1831210,
"postDate": "2022-06-24T03:33:15.350Z",
"content": "![\"https://i.ibb.co/74FWk8d/Selection-013.png\"](\"https://i.ibb.co/74FWk8d/Selection-013.png\")
\n![\"https://miro.medium.com/max/1172/1*kGi3P8DtXWNpSmqBSEbvSA.png\"](\"https://miro.medium.com/max/1172/1*kGi3P8DtXWNpSmqBSEbvSA.png\")
\nexpected improvement using pretraining transformer
\nfrom: [1] BEiT: BERT Pre-Training of Image Transformers
other related papers:
\nself-learning
\ndealing with large image for WSI
\ngood transformer model
\nan interesing paper: comapre CNN (unet, deeplab3, FPN, etc) vs tranformer (transUNet, Swin-Transformer, segmenter, etc) for WSI segmentation
\nEvaluating Transformer-based Semantic Segmentation Networks for Pathological Image Segmentation
\nhttps://arxiv.org/pdf/2108.11993.pdf
I just collected 13k H&E, PAS and DAB/H stained microscopy images. I'll experiment with those self-supervised training techniques.
", "rawMarkdown": "I just collected 13k H&E, PAS and DAB/H stained microscopy images. I'll experiment with those self-supervised training techniques." }, { "id": 1869418, "postDate": "2022-07-24T18:30:08.850Z", "content": "the first step is to use your validation set (+ training set) as unlabelled image for SSL training.
\nthe pretrained SSL model is then fine-tunned on train data.
\nthe fine-tuned model is then validated on validation set.
it should work (since validation set is seen in SSL). i use this method to debug my SSL.
\nthe the SSL is OK, then I would proceed with the proper SSL training with the extra unlabelled data downloaded (and remove validation set from SSL train set)
I expect about +0.02 improvement in local CV and public LB (both Hubmap and HPA)
\nanother way to verify SSL works correctly is the use the old pesudo-label method as baseline.
", "rawMarkdown": "the first step is to use your validation set (+ training set) as unlabelled image for SSL training.\nthe pretrained SSL model is then fine-tunned on train data.\nthe fine-tuned model is then validated on validation set.\n\nit should work (since validation set is seen in SSL). i use this method to debug my SSL.\nthe the SSL is OK, then I would proceed with the proper SSL training with the extra unlabelled data downloaded (and remove validation set from SSL train set)\n\nI expect about +0.02 improvement in local CV and public LB (both Hubmap and HPA)\n\n---\n\nanother way to verify SSL works correctly is the use the old pesudo-label method as baseline.\n" } ] }, { "id": 1914582, "postDate": "2022-08-26T07:45:52.507Z", "content": "Hello folks, I am using transformer-based approach for this task using resized data (768,768). I was wondering how should I find what is the optimal Threshold for my model for each of the organs?
\n@hengck23
\nadvice would be really helpful.
\nThanks! :)
click sort by \"newest\" and scroll to the bottom of this page. this is already mentioned before
", "rawMarkdown": "click sort by \"newest\" and scroll to the bottom of this page. this is already mentioned before", "votes": 2 } ] }, { "id": 1904239, "postDate": "2022-08-18T04:19:16.057Z", "content": "in my latest LB score of 0.81 l had LB-hubmap of 0.59.
\ni suddenly want to try 2048 ….
", "rawMarkdown": "i suddenly want to try 2048 ....", "votes": 3 }, { "id": 1906613, "postDate": "2022-08-20T04:32:25.883Z", "content": "Actually from started with segformer 1024x1024 my score was 0.72 with 768x768 ade whereas 0.76 with 1024x1024.
\nlarger images performed better in my case.
performance \"will always\" be better for larger image because of resize error (upscale error) when you up scale the mask to match the size of the image.
\nif your modeling is good, you \"may\" gain additional performance from more input pixels information at higher resolution.
\nhence the correct way to do is to improve your modeling with smaller size first (e.g. 768) then slowly up scale your model.
", "rawMarkdown": "performance \"will always\" be better for larger image because of resize error (upscale error) when you up scale the mask to match the size of the image.\n\nif your modeling is good, you \"may\" gain additional performance from more input pixels information at higher resolution.\n\n\nhence the correct way to do is to improve your modeling with smaller size first (e.g. 768) then slowly up scale your model.", "votes": 3 }, { "id": 1911522, "postDate": "2022-08-24T06:36:15.187Z", "content": "Does no-external data mean hpa data only (excluding hubmap data)?
Btw, thanks for sharing your ideas and reports.
", "rawMarkdown": "Does `no-external data` mean hpa data only (excluding hubmap data)?\n\nBtw, thanks for sharing your ideas and reports. " }, { "id": 1911770, "postDate": "2022-08-24T09:51:08.037Z", "content": "no-external data means training with kaggle train data only (i.e. hpa)
", "rawMarkdown": "no-external data means training with kaggle train data only (i.e. hpa)", "votes": 1 } ] }, { "id": 1896619, "postDate": "2022-08-13T01:46:22.917Z", "content": "yet another 0.79 model.
\n![\"https://i.ibb.co/6FMLfv4/Selection-096.png\"](\"https://i.ibb.co/6FMLfv4/Selection-096.png\")
Hi Heng, I'm wondering if SWA can be applied to CoaT model. Since the PE encoding is shared, will SWA average such learned parameters once more?
", "rawMarkdown": "Hi Heng, I'm wondering if SWA can be applied to CoaT model. Since the PE encoding is shared, will SWA average such learned parameters once more? ", "votes": -1 }, { "id": 1911354, "postDate": "2022-08-24T03:37:08.287Z", "content": "@fuckvenkatraman Just skip them in SWA.
\nskip = ['num_batches_tracked','cpe','crpe']\n
\"skip = ['num_batches_tracked','cpe','crpe']\"
\nthis is no good enough when you use parallel block. \".cpe.\", etc is more correct
\nthis is additional sharing in parallel block
(swa always improve results all the times. if you don't get improvement, likely you do something wrong)
\ntry this trick
\nlet selected checkpoints be [x1,x2,x3]
\ncompute swa.
now if you set x1=x2=x3, if should see swa exactly equal to x1.
\nthen you can check abs(swa-x1) for each of the parameters. if they are not the same, they are shared somewhere.
Note 1:\n\n because of numerical error, just use 2 checkpoint, i.e. use [x1,x2] =[x1,x1] \n\n cross checked with the code of COAT \n\n pytorch save shared paramter multiple time. another way to check is load a model pth. \ndelete repeated parameters. save the new pth. now if you load the orginal or new pth, the model should produce the same result. there is yet other way to detect sharing: compare each and every parameter pair, first by shape and then by numerical values. if they are repeated, delete\nThanks for your detailed explanation! I will try it.
", "rawMarkdown": "Thanks for your detailed explanation! I will try it." }, { "id": 1911519, "postDate": "2022-08-24T06:33:40.193Z", "content": "@hengck23 Hi Heng, thanks for your explanation. If my understanding is correct, the right SWA should host checkpoint averages for unshared params, but the latest values(not averaged) for shared positional encoding params.
", "rawMarkdown": "@hengck23 Hi Heng, thanks for your explanation. If my understanding is correct, the right SWA should host checkpoint averages for unshared params, but the latest values(not averaged) for shared positional encoding params." }, { "id": 1911521, "postDate": "2022-08-24T06:34:20.737Z", "content": "Hi @yingpengchen, thanks for your tips!
", "rawMarkdown": "Hi @yingpengchen, thanks for your tips! " }, { "id": 1911528, "postDate": "2022-08-24T06:48:04.060Z", "content": "\"but the latest values(not averaged) for shared positional encoding params.\"
\nyou can delete 'key' of shared parameters.
\nbecause the network will not load these shared parameter from the pth file.
\nthey are pointed to the unshared ones.
let me give you an example
\nline.1 class net():\nline.2 self.linear1 = ...\nline.3 self.linear2 = ...\nline.4 self.linear2 = self.linear1 #this is sharing. \n\nwe need swa.pth to save only self.linear1.weight \nduring loading net.load_state_dict(swa):
\nthis also means that swa only needs to average self.linear1.weight and ignore/skip self.linear2.weight completedly
\nanother way of saying:
\nwhen self.linear1.weight is read from pth file, it is loaded and used by both self.linear1 and self.linear2.
also, if there is another self.linear2.weight is read from pth file, it is loaded and used by both self.linear1 and self.linear2.
\nif you save and read both self.linear1.weight, self.linear2.weight, self.linear1 and self.linear2 are both read twice with the last read values replacing the previous one (for both linear modules).
\nit is also for this reason that if you average self.linear2 in swa, you are average self.linear1 too because they are \"equal\", i.e. they refer to the same module.
\nhence we only need to average self.linear1 (and ignore self.linear2)
\nif you average self.linear1 once and self.linear2 once, the results is that both self.linear1 and self.linear2 are averaged twice
\ncoat code:
\nclass ParallelBlock(nn.Module):\n self.mlp2 = self.mlp3 = self.mlp4 = self.mlp5 = Mlp(in_features=dims[1], hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)\n\n##you should delete self.mlp2 = self.mlp3 = self.mlp4 and just keep self.mlp5 in swa\n\n\n\nclass FactorAtt_ConvRelPosEnc(nn.Module):\n # Shared convolutional relative position encoding.\n self.crpe = shared_crpe\n\n##you should delete self.crpe and just keep shared_crpe in swa. you will have to trace back what the key of shared_crpe is\n\n\nclass SerialBlock(nn.Module):\n\n # Conv-Attention.\n self.cpe = shared_cpe\n\n\n##you should delete self.cpe and just keep shared_cpe in swa. ....\nactually the interesting questions is how did i discover this?
\nthis is because when i first did swa results actually decreased.
\nthe next question is, how can i be so sure that swa definitely improved performance and a software bug is intermediately suspected?
\nthis is because at the end of training, the gradient are very small. so the solutions do not differs greatly. (if you check my code, i even average batch norm/layer norm statistics, include variances. for other model, i average learned pos encoding as well).
\nif linear combinations of close solutions lead to performance drop, it means there is a hill in the basin in the loss landscape.
", "rawMarkdown": "\"but the latest values(not averaged) for shared positional encoding params.\"\n\nyou can delete 'key' of shared parameters.\nbecause the network will not load these shared parameter from the pth file.\nthey are pointed to the unshared ones.\n\nlet me give you an example\n\n```\nline.1 class net():\nline.2 self.linear1 = ...\nline.3 self.linear2 = ...\nline.4 self.linear2 = self.linear1 #this is sharing. \n\nwe need swa.pth to save only self.linear1.weight \n\n```\n\nduring loading net.load_state_dict(swa):\n- self.linear1 is loaded with self.linear1.weight in swa\n- self.linear2 is not loaded, the loading may show error but just ignore\n- line.4 of code always point self.linear2 to self.linear1. so even it seems that self.linear2 is not loaded but that is not true\n\nthis also means that swa only needs to average self.linear1.weight and ignore/skip self.linear2.weight completedly \n\n---\n\nanother way of saying:\nwhen self.linear1.weight is read from pth file, it is loaded and used by both self.linear1 and self.linear2.\n\nalso, if there is another self.linear2.weight is read from pth file, it is loaded and used by both self.linear1 and self.linear2.\n\nif you save and read both self.linear1.weight, self.linear2.weight, self.linear1 and self.linear2 are both read twice with the last read values replacing the previous one (for both linear modules).\n\n---\n\nit is also for this reason that if you average self.linear2 in swa, you are average self.linear1 too because they are \"equal\", i.e. they refer to the same module.\n\nhence we only need to average self.linear1 (and ignore self.linear2)\n\nif you average self.linear1 once and self.linear2 once, the results is that both self.linear1 and self.linear2 are averaged twice\n\n---\ncoat code:\n\n```\nclass ParallelBlock(nn.Module):\n\t\tself.mlp2 = self.mlp3 = self.mlp4 = self.mlp5 = Mlp(in_features=dims[1], hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)\n\t\n##you should delete self.mlp2 = self.mlp3 = self.mlp4 and just keep self.mlp5 in swa\n\n\n\nclass FactorAtt_ConvRelPosEnc(nn.Module):\n\t\t# Shared convolutional relative position encoding.\n\t\tself.crpe = shared_crpe\n\n##you should delete self.crpe and just keep shared_crpe in swa. you will have to trace back what the key of shared_crpe is\n\n\nclass SerialBlock(nn.Module):\n\n\t\t# Conv-Attention.\n\t\tself.cpe = shared_cpe\n\n\n##you should delete self.cpe and just keep shared_cpe in swa. ....\n```\n\n\n----\n\n\nactually the interesting questions is how did i discover this?\n\nthis is because when i first did swa results actually decreased.\n\nthe next question is, how can i be so sure that swa definitely improved performance and a software bug is intermediately suspected?\n\nthis is because at the end of training, the gradient are very small. so the solutions do not differs greatly. (if you check my code, i even average batch norm/layer norm statistics, include variances. for other model, i average learned pos encoding as well).\n \nif linear combinations of close solutions lead to performance drop, it means there is a hill in the basin in the loss landscape.", "votes": 3 }, { "id": 1911543, "postDate": "2022-08-24T07:02:40.067Z", "content": "Thank you for your vivid explanation, now I can grep a better understanding of your point!
", "rawMarkdown": "Thank you for your vivid explanation, now I can grep a better understanding of your point! " }, { "id": 1925954, "postDate": "2022-09-04T12:52:45.887Z", "content": "I tried this trick for coat-parallel model, but not found all skip parameters.
", "rawMarkdown": "I tried this trick for coat-parallel model, but not found all skip parameters." }, { "id": 1934368, "postDate": "2022-09-11T10:25:13.417Z", "content": "Thank you very much for sharing the great tips, Sensei @hengck23.
\n\n\n\n\n\nPlease disregard first two questions. I completely misunderstood SWA and your tricks. By the way, there seems a weird error that I haven't figured out. Combining two pths works fine according to CV but, more than two (three or more) doesn't work at all. It shows weird mask area here
\nSWAing two of the three hits cv 0.78 while all of them hit 0.077. Can you enlighten me to solve this problem?
Here is the error_image with three pth files for SWA, while the normal_image with two pth files (just excluded one of the three pth files)
", "rawMarkdown": "Thank you very much for sharing the great tips, Sensei @hengck23. ~~Mind if I ask three questions?\nFirstly, what's the best way to decide the first input pth for SWA? From your SWA code snippet,~~ \n\n~~checkpoints = [x1.pth, x2.pth, x3.pth]\ndef do_swa(checkpoints):\n state_dict = torch.load( ~ )\n if swa is None:\n swa = state_dict\n else:\n # skip if k in skip list\n # or sum and divide~~\n\n~~For the shared parameters that are skipped, their values are 100% from the first pth file only. So the order of pth files in checkpoints matters for the shared params. For me, the best way seems to pick up the first pth file for higher cv. I'm curious about your opinion.~~\n\n~~Secondly, I tried your trick: abs(swa(x1 and x1) - x1) != 0. I'm curious if I'm good to go as long as there is no non-zero values when I did your trick. For example, I used a counting method. ~~\n\n~~The number of all the parameters in my model -> 581\nThe number of abs(swa(x1 and x1) -x1) == 0 -> 276\nThe number of abs(swa(x1 and x1) -x1) != 0 -> 305\nWith a bit of tweak skip list, \nThe number of abs(swa(x1 and x1) -x1) == 0 -> 581\nThe number of abs(swa(x1 and x1) -x1) != 0 -> 0 ~~\n\n~~I'm curious if it's good enough. I submitted two SWA versions: one increased by 0.01, and the other one unchanged. Each SWA version consists of two pths. I couldn't try more due to the limit of submission a day~~\n\nPlease disregard first two questions. I completely misunderstood SWA and your tricks. By the way, there seems a weird error that I haven't figured out. Combining two pths works fine according to CV but, more than two (three or more) doesn't work at all. It shows weird mask area [here](https://raw.githubusercontent.com/cheul0518/Competitions/main/HuBMAP_HPA/e2.png)\nSWAing two of the three hits cv 0.78 while all of them hit 0.077. Can you enlighten me to solve this problem? \n\n~~Thirdly, when I combine three or more than three pth files for SWA, a weird error happens. Mask in a picture suddenly covers the whole image area with so many dots. It maybe seems due to underflow or overflow I'm not sure. Do you have any idea why it happens and how to solve it?\nThank you for reading my comment and hopefully hear your insight.~~\n\nHere is the [error_image](https://raw.githubusercontent.com/cheul0518/Competitions/main/HuBMAP_HPA/error.png) with three pth files for SWA, while the [normal_image](https://raw.githubusercontent.com/cheul0518/Competitions/main/HuBMAP_HPA/normal.png) with two pth files (just excluded one of the three pth files)\n\n" }, { "id": 1934730, "postDate": "2022-09-11T14:52:04.807Z", "content": "I am studying swa for the first time.
\nDo I have to use sgd opt to use swa?
@sompark , you don't have to stick with SGD. Check here. Excerpt from the link: To run SWA in auto mode you just need to wrap your optimizer base_opt of choice (can be SGD, Adam, or any other torch.optim.Optimizer) with SWA(base_opt, swa_start, swa_freq, swa_lr)
thank you😆
", "rawMarkdown": "thank you😆" }, { "id": 1939342, "postDate": "2022-09-14T16:32:11.783Z", "content": "can i ask one more??….
\nI tried it and lr is always same
\nam I doing all right?…
IMHO, lr can stay if you've set base_optimizer_lr and swa_lr to the same constant number such as SGD(lr=0.1) and SWA(swa_lr=0.1), and didn't use standard decaying schedule. For example, SWA description explicitly names standard decaying schedule when they showcase lr change in the figure 2. I'm not 100% positive so hopefully someone read your comment and give you the exact, correct answer.
", "rawMarkdown": "IMHO, lr can stay if you've set base_optimizer_lr and swa_lr to the same constant number such as SGD(lr=0.1) and SWA(swa_lr=0.1), and didn't use standard decaying schedule. For example, [SWA description] (https://pytorch.org/assets/images/swa/figure2-highres.png) explicitly names standard decaying schedule when they showcase lr change in the figure 2. I'm not 100% positive so hopefully someone read your comment and give you the exact, correct answer." }, { "id": 1939729, "postDate": "2022-09-15T01:23:29.680Z", "content": "\"I tried it and lr is always same\"
\ndata science is an art not a science. there is no wrong or right answers. there is only experimental results and conclusions.
\nwhen you read a method
\n1) understand the steps (what to do)
\n2) try to understand why (why to do), but there will always be exceptions that the reasoning wrong/correct. this is because there will be assumptions in the reasoning and everything is dependent on data
\n3) do expect to verify results and how different parameters affect results (just do it)
here in this competition, if you check my code posted in the public notebook, SWA simply means average of the saved checkpoints (maybe some abuse of terms)
", "rawMarkdown": "\"I tried it and lr is always same\"\n\ndata science is an art not a science. there is no wrong or right answers. there is only experimental results and conclusions.\n\nwhen you read a method\n1) understand the steps (what to do)\n2) try to understand why (why to do), but there will always be exceptions that the reasoning wrong/correct. this is because there will be assumptions in the reasoning and everything is dependent on data\n3) do expect to verify results and how different parameters affect results (just do it)\n\n---\n\nhere in this competition, if you check my code posted in the public notebook, SWA simply means average of the saved checkpoints (maybe some abuse of terms)", "votes": 1 }, { "id": 1940119, "postDate": "2022-09-15T07:32:30.450Z", "content": "Thank you for your insightful comment @hengck23! By the way, I've been long stuck with figuring out my SWA (combining three .pths at validation/test step). May I ask for a hint other than your previous big hint: comparing (swa(x1,x1) - x1)?
", "rawMarkdown": "Thank you for your insightful comment @hengck23! By the way, I've been long stuck with figuring out my SWA (combining three .pths at validation/test step). May I ask for a hint other than your previous big hint: comparing (swa(x1,x1) - x1)?" }, { "id": 1942865, "postDate": "2022-09-17T05:41:58.630Z", "content": "@Cheul Can I ask the difference between the lb score of swa and normal?
\nI got only 0.01…
@gunparksom No my Segformer gave 0.67 at the highest a month ago. Maybe it could've given a lot better score if I either utilized 100% of mmseg mechanics or coded it manually.
", "rawMarkdown": "@gunparksom No my Segformer gave 0.67 at the highest a month ago. Maybe it could've given a lot better score if I either utilized 100% of mmseg mechanics or coded it manually." }, { "id": 1942877, "postDate": "2022-09-17T05:53:28.203Z", "content": "@gunparksom my SWA improved 0.001~ 0.002 cv for me. I don't know if it could really improve my LB score since my rank and score remained the same. For my case of SWA cv, it reduces lots of false positive cases while reducing true positive a little bit. So I think SWA seems to work but I'm not certain like LB scores still same
", "rawMarkdown": "@gunparksom my SWA improved 0.001~ 0.002 cv for me. I don't know if it could really improve my LB score since my rank and score remained the same. For my case of SWA cv, it reduces lots of false positive cases while reducing true positive a little bit. So I think SWA seems to work but I'm not certain like LB scores still same" }, { "id": 1943582, "postDate": "2022-09-17T16:10:53.370Z", "content": "wow It makes me feel better that I wasn't the only me
\nthank you
![\"https://i.ibb.co/2skbhX5/Selection-069.png\"](\"https://i.ibb.co/2skbhX5/Selection-069.png\")
i find a way to annotate spleen images easily.
\nuse gimp extract color components (CMKY yellow)
![\"https://i.ibb.co/djRfckn/Selection-064.png\"](\"https://i.ibb.co/djRfckn/Selection-064.png\")
![\"https://i.ibb.co/VSvTF1d/Selection-066.png\"](\"https://i.ibb.co/VSvTF1d/Selection-066.png\")
\ni download random images from the internet to test the robustness of the model.
\nit perform better than I thought.
it is obvious that hand-correction of the prediction on unseen images is a way to go.
\nneed to find more images without copyright issues
more stress test
\n![\"https://i.ibb.co/bLspmJQ/Selection-084.png\"](\"https://i.ibb.co/bLspmJQ/Selection-084.png\")
i just realised besides swin v1,v2, there is a \"version 3\"
\nhttps://github.com/microsoft/CSWin-Transformer
maybe you don't need so many attention
\npaper: HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling
\n![\"https://i.ibb.co/VVK2fs3/Selection-122.png\"](\"https://i.ibb.co/VVK2fs3/Selection-122.png\")
good reading
\nHOW DO VISION TRANSFORMERS WORK?
\nhttps://arxiv.org/pdf/2202.06709.pdf
\n\"MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary\"
Blurs Behave Like Ensembles: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness
\nhttps://arxiv.org/pdf/2105.12639.pdf
\"MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary\"
That could explain why attention-based encoders and convolution-based decoders in UNETR and Swin UNETR work so well.
", "rawMarkdown": "` \"MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary\"`\n\nThat could explain why attention-based encoders and convolution-based decoders in UNETR and Swin UNETR work so well." }, { "id": 1879884, "postDate": "2022-08-01T10:31:37.917Z", "content": "i am thinking to use your current pretrain transformer, you can do something like this
\n![\"https://i.ibb.co/LnbbZGY/Selection-128.png\"](\"https://i.ibb.co/LnbbZGY/Selection-128.png\")
updated list of masked pretraining method.
\nmost opensource have complete framework code from pertaining transformer backbone and finetunning/inference it for segmentation.
![\"https://i.ibb.co/XCsKpjC/Selection-116.png\"](\"https://i.ibb.co/XCsKpjC/Selection-116.png\")
\nI missed the important Beit in the table:
\nhttps://github.com/microsoft/unilm/tree/master/beit
see also
\nhttps://github.com/cmhungsteve/Awesome-Transformer-Attention
\nunder \"Training + Transformer\" for more MIM
how should you select your solution?
\nwhich organ actually affects private LB score the most? (due to different/same class distribution probability and domain difference of private data)
we usually monitor the validation dice of all organ. but does this select the most important one to private test? or are we selecting the best model for local validation data only? (is all dice correlated to organ dice)
![\"https://i.ibb.co/j84YLNm/Selection-083.png\"](\"https://i.ibb.co/j84YLNm/Selection-083.png\")
make your own spleen labels:
\nhttps://www.purposegames.com/no/game/spleen-histology-and-red-and-white-pulp
this comment has not been deleted
", "rawMarkdown": "this comment has not been deleted", "votes": 1 }, { "id": 1832019, "postDate": "2022-06-24T15:45:58.667Z", "content": "fyi:
\nVirtual histological staining of unlabelled tissue-autofluorescence images via deep learning
\nhttps://openreview.net/pdf?id=S1xANDXRKN
\nhttps://arxiv.org/pdf/1803.11293.pdf
\n![\"https://i.ibb.co/HFqKxfX/Selection-022.png\"](\"https://i.ibb.co/HFqKxfX/Selection-022.png\")
H&E to PAS conversion
\nDeep learning-based transformation of H&E stained tissues into special stains
\nhttps://github.com/kevindehaan/stain-transformation
DAB(brown)-H(blue) to H(blue)&E(pink) papers:
\n(1) Creating virtual H&E images using samples imaged on a commercial CODEX platform
\n(2) FalseColor-Python: a rapid intensity-leveling and digital-staining package for fluorescence-based slide-free digital pathology
(3) Dual Adaptive Pyramid Network for Cross-Stain Histopathology Image Segmentation
\nIn this paper, we propose a dual adaptive pyramid network (DAPNet) for histopathological gland segmentation
\nadapting from one stain domain to another (DAB-H and H&E)
\nhttps://arxiv.org/pdf/1909.11524.pdf
\n![\"Selection-034\"](\"https://i.ibb.co/GMM6fj4/Selection-034.png\")
![\"Selection-033\"](\"https://i.ibb.co/HzZc0Vm/Selection-033.png\")
Tailoring automated data augmentation to H&E-stained histopathology
\nhttps://github.com/DIAGNijmegen/pathology-he-auto-augment
\nhttps://openreview.net/pdf?id=JrBfXaoxbA2
https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_ihc_color_separation.html
\nhttps://towardsdatascience.com/stain-estimation-on-microscopy-whole-slide-images-2b5a57062268
\n\n---\n\n\nTailoring automated data augmentation to H&E-stained histopathology\nhttps://github.com/DIAGNijmegen/pathology-he-auto-augment\nhttps://openreview.net/pdf?id=JrBfXaoxbA2\n\n---\n\nhttps://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_ihc_color_separation.html\nhttps://towardsdatascience.com/stain-estimation-on-microscopy-whole-slide-images-2b5a57062268",
"votes": 2,
"replies": [
{
"id": 1881003,
"postDate": "2022-08-02T07:21:27.157Z",
"content": "\"We propose a new deep learning method to segment epithelial tissue in digitised hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard.\"
\nhttps://www.nature.com/articles/s41598-018-37257-4.pdf
", "rawMarkdown": "\"We propose a new deep learning method to segment epithelial tissue in digitised hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard.\"\n\nhttps://www.nature.com/articles/s41598-018-37257-4.pdf" } ] }, { "id": 1938689, "postDate": "2022-09-14T09:46:31.067Z", "content": "I notice that the KEY POINT is 2xA6000 GPU cards workstation xD
", "rawMarkdown": "I notice that the KEY POINT is 2xA6000 GPU cards workstation xD" }, { "id": 1872994, "postDate": "2022-07-27T11:18:27.407Z", "content": "How did you manage to calculate the LB score on HPA and Hubmap?
\nI found in the comments the probe you used to calculate how many of each type you had in the test set. But how to get the metrics on each type is out of my understanding 😅
", "rawMarkdown": "How did you manage to calculate the LB score on HPA and Hubmap?\n\nI found in the comments the probe you used to calculate how many of each type you had in the test set. But how to get the metrics on each type is out of my understanding 😅", "votes": 1, "replies": [ { "id": 1873014, "postDate": "2022-07-27T11:36:55.443Z", "content": "your submission code:
\n![\"https://i.ibb.co/6r8dYNj/Selection-023.png\"](\"https://i.ibb.co/6r8dYNj/Selection-023.png\")
#data_source =['Hubmap', 'HPA']\ndata_source =['HPA']\n\n#organ = ['kidney', 'prostate', 'largeintestine', 'spleen', 'lung']\norgan = ['prostate', ]\n\n for i,d in test_df.iterrows():\n id = d['id']\n if (d['data_source'] in data_source) and (d['organ'] in organ):\n rle = ... run you model and compute rle # this gives your prediction score\n else:\n rle = '' # this gives zero score\ne.g if I set
\ndata_source =['HPA']
\norgan = ['kidney', 'prostate', 'largeintestine', 'spleen', 'lung']
the score on the submission page is for HPA and all organs only
", "rawMarkdown": "your submission code:\n\n\n```\n#data_source =['Hubmap', 'HPA']\ndata_source =['HPA']\n\n#organ = ['kidney', 'prostate', 'largeintestine', 'spleen', 'lung']\norgan = ['prostate', ]\n\n for i,d in test_df.iterrows():\n id = d['id']\n if (d['data_source'] in data_source) and (d['organ'] in organ):\n rle = ... run you model and compute rle # this gives your prediction score\n else:\n rle = '' # this gives zero score\n```\n\ne.g if I set \ndata_source =['HPA']\norgan = ['kidney', 'prostate', 'largeintestine', 'spleen', 'lung']\n\nthe score on the submission page is for HPA and all organs only\n", "votes": 2 }, { "id": 1884114, "postDate": "2022-08-04T09:53:54.807Z", "content": "I used this exact code & submitted for different organs (both datasources). The scores I got are 0.03, 0.08, 0.05, 0.11, 0.07 (all sum up to 0.34) for lung, spleen, kidney, prostate, largeintestine respectively. When I submitted without this code(i.e. all organs & both datasets) the score was 0.37. Why this difference 0.34!=0.37? I expected the individual organs scores to sum up to 0.37 exactly. Any pointers?
", "rawMarkdown": "I used this exact code & submitted for different organs (both datasources). The scores I got are 0.03, 0.08, 0.05, 0.11, 0.07 (all sum up to 0.34) for lung, spleen, kidney, prostate, largeintestine respectively. When I submitted without this code(i.e. all organs & both datasets) the score was 0.37. Why this difference 0.34!=0.37? I expected the individual organs scores to sum up to 0.37 exactly. Any pointers?" }, { "id": 1884158, "postDate": "2022-08-04T10:25:07.423Z", "content": "@desaijairav numerical error.
\nthe score dispayed is not round(). it is floor().
e.g. actual score could be 0.12999. but only only first 2 decimals are displayed : 0.12. it is truncated and not round
", "rawMarkdown": "@desaijairav numerical error.\nthe score dispayed is not round(). it is floor().\n\ne.g. actual score could be 0.12999. but only only first 2 decimals are displayed : 0.12. it is truncated and not round", "votes": 1 } ] }, { "id": 1839900, "postDate": "2022-07-01T19:37:48.137Z", "content": "Hi @hengck23 - Wonderful thread like usual.
\nIf you do find any useful external data sources, would you mind commenting on them on this thread? I will add them to the main body of that post so that they are shared with everyone all in one place.
", "rawMarkdown": "Hi @hengck23 - Wonderful thread like usual.\n\nIf you **do** find any useful external data sources, would you mind commenting on them on [**this thread**](https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/333886)? I will add them to the main body of that post so that they are shared with everyone all in one place.", "votes": 1, "replies": [ { "id": 1858204, "postDate": "2022-07-16T18:29:24.793Z", "content": "\nexternal data is on this thread: https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332714
", "rawMarkdown": "@dschettler8845 \n\nexternal data is on this thread: https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332714" }, { "id": 1859549, "postDate": "2022-07-17T17:50:26.980Z", "content": "I wonder where did you find H&E and DAB/H stained images. hubmapconsortium only has 46 PAS stained kidney images which are probably previous competition's dataset. There is nothing new on there.
", "rawMarkdown": "I wonder where did you find H&E and DAB/H stained images. hubmapconsortium only has 46 PAS stained kidney images which are probably previous competition's dataset. There is nothing new on there.", "votes": 1 }, { "id": 1861351, "postDate": "2022-07-19T01:32:42.170Z", "content": "read my post carefully.
\nHBMxxxx are hubmap image id
\nthere is hidden colon(large intestine) dataset that has public access
![\"https://i.ibb.co/bFfZn3D/Selection-097.png\"](\"https://i.ibb.co/bFfZn3D/Selection-097.png\")
a baseline code is needed! current public ones are not good.👀
", "rawMarkdown": "a baseline code is needed! current public ones are not good.👀", "votes": -5, "replies": [ { "id": 1862820, "postDate": "2022-07-20T03:17:35.183Z", "content": "actually when i don't publish code, it means that
\nGreat job! what pretrained model did you use, and what loss function did you use.
", "rawMarkdown": "Great job! what pretrained model did you use, and what loss function did you use.", "votes": -1 }, { "id": 1951803, "postDate": "2022-09-23T08:53:58.850Z", "content": "Could you share a snippet of data parallelism code? I saw your posted codes and tried to implement it into mine but it didn't go well.
\nI used the following code to utilize two gpus:
with amp.autocast(enabled=is_amp):\n output = data_parallel(net, batch) \n probability += F.interpolate(output['probability'], size=(H, W), mode='bilinear', align_corners=False, antialias=True)\nThe program utilized the only one gpu while training. Though I had added GPU 0,1 environment and checked its counter =2. So I'm curious if I'm missing something important to do data parallelism?
", "rawMarkdown": "Could you share a snippet of data parallelism code? I saw your posted codes and tried to implement it into mine but it didn't go well. \nI used the following code to utilize two gpus:\n```\nwith amp.autocast(enabled=is_amp):\n\toutput = data_parallel(net, batch) \n\tprobability += F.interpolate(output['probability'], size=(H, W), mode='bilinear', align_corners=False, antialias=True)\n```\nThe program utilized the only one gpu while training. Though I had added GPU 0,1 environment and checked its counter =2. So I'm curious if I'm missing something important to do data parallelism?", "replies": [ { "id": 1951938, "postDate": "2022-09-23T10:48:21.650Z", "content": "from torch.nn.parallel.data_parallel import data_parallel\nSorry for not being clear on the question. Yes I certainly did import the lib. But the following error always came up:
\nRuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper__cudnn_convolution)\nI tried to find the answer by googling but they're all about cuda:0 and cpu, nothing to do with two gpus. Could you enlighten me what I did wrong to make two gpu running?
\nPS here is the code that I used to train, and it went well when I trained the model on one gpu:
\n for i, data in enumerate(tqdm.notebook.tqdm(dl_train ,total = len(dl_train))):\n imgs, masks = data\n\n imgs = imgs.half().to(DEVICE)\n masks = masks.half().to(DEVICE) \n batch = {'imgs':None, 'masks':None}\n batch['imgs'] = imgs\n batch['masks'] = masks \n\n # zero the parameter gradients\n optimizer.zero_grad() \n\n with torch.cuda.amp.autocast(enabled=True):\n # forward + loss\n outputs = data_parallel(model, batch) \n loss_bce = outputs['bce_loss'].mean()\n loss_aux1 = outputs['aux1_loss'].mean()\n loss_aux2 = outputs['aux2_loss'].mean()\n loss_aux3 = outputs['aux3_loss'].mean()\n loss_aux4 = outputs['aux4_loss'].mean()\n loss_tr = loss_bce + 0.2*loss_aux1 + 0.2*loss_aux2 + 0.2*loss_aux3 + 0.2*loss_aux4\n\n scaler.scale(loss_tr).backward() \n scaler.unscale_(optimizer)\n scaler.step(optimizer)\n scaler.update()\n scheduler.step()\nsome of the opensource model does not support multiple gpu training (because of the way they have written the init and forward function). you will have to debug network module one by one and rewrite their code
\nyou can debug your code by using a simple model like resnet34 for classification
", "rawMarkdown": "some of the opensource model does not support multiple gpu training (because of the way they have written the init and forward function). you will have to debug network module one by one and rewrite their code\n\nyou can debug your code by using a simple model like resnet34 for classification", "votes": 1 }, { "id": 1952910, "postDate": "2022-09-24T04:23:26.427Z", "content": "I see. Thank you for the direction. I guess the post work gonna be a lot more fun. Hope you have a great weekend.
", "rawMarkdown": "I see. Thank you for the direction. I guess the post work gonna be a lot more fun. Hope you have a great weekend." } ] }, { "id": 1951493, "postDate": "2022-09-23T05:21:07.223Z", "content": "Hey . . . @hengck23 Hengck . . . . THANKS VERY MUCH ✌️✌️✌️✌️ . . . . MAIDEN MEDAL IN Competition and YOUR COAT MEDIUM + DAFORMER was winner for me, .. . .. . 🔥🔥🔥🔥🔥 . . . .
", "rawMarkdown": "Hey . . . @hengck23 Hengck . . . . THANKS VERY MUCH ✌️✌️✌️✌️ . . . . MAIDEN MEDAL IN Competition and YOUR COAT MEDIUM + DAFORMER was winner for me, .. . .. . 🔥🔥🔥🔥🔥 . . . . " }, { "id": 1949163, "postDate": "2022-09-21T14:45:13.070Z", "content": "I've just realised - you are probably going to become a competitions grand master in just over 24 hours :)
", "rawMarkdown": "I've just realised - you are probably going to become a competitions grand master in just over 24 hours :)" }, { "id": 1940951, "postDate": "2022-09-15T16:55:15.010Z", "content": "Hello!
\nI'm currently working with segformer mit b2. I use the same fold for validation and all the scripts as you. But I don't know why I can't get at least 0.75 on validation. Do you have any idea what causes the problem? Thanks a lot for your work - I've learned a lot!
train_log:rate iter epoch | dice loss tp tn | loss | time
\ntrain_log:-------------------------------------------------------------------------------------
\ntrain_log:1.00e-5 00000000* 0.00 | 0.155 0.705 0.0000 0.000 | 0.000 0.000 | 0 hr 00 min
\ntrain_log:1.00e-5 00000138* 3.00 | 0.592 0.251 0.0000 0.000 | 0.372 0.265 | 0 hr 01 min
\ntrain_log:1.00e-5 00000276* 6.00 | 0.594 0.165 0.0000 0.000 | 0.203 0.173 | 0 hr 02 min
\ntrain_log:1.00e-5 00000414* 9.00 | 0.608 0.134 0.0000 0.000 | 0.154 0.126 | 0 hr 04 min
\ntrain_log:1.00e-5 00000552* 12.00 | 0.638 0.133 0.0000 0.000 | 0.130 0.115 | 0 hr 05 min
\ntrain_log:1.00e-5 00000690* 15.00 | 0.644 0.131 0.0000 0.000 | 0.124 0.113 | 0 hr 06 min
\n….
\ntrain_log:1.00e-5 00003726* 81.00 | 0.715 0.102 0.0000 0.000 | 0.071 0.075 | 0 hr 37 min
train_log:1.00e-5 00003864* 84.00 | 0.702 0.101 0.0000 0.000 | 0.053 0.059 | 0 hr 39 min
\ntrain_log:1.00e-5 00004002* 87.00 | 0.707 0.094 0.0000 0.000 | 0.054 0.059 | 0 hr 40 min
\ntrain_log:1.00e-5 00004140* 90.00 | 0.700 0.106 0.0000 0.000 | 0.056 0.062 | 0 hr 42 min
\ntrain_log:1.00e-5 00004278* 93.00 | 0.700 0.094 0.0000 0.000 | 0.056 0.061 | 0 hr 43 min
\ntrain_log:1.00e-5 00004416* 96.00 | 0.706 0.099 0.0000 0.000 | 0.054 0.059 | 0 hr 44 min
\ntrain_log:1.00e-5 00004554* 99.00 | 0.708 0.097 0.0000 0.000 | 0.055 0.061 | 0 hr 46 min
\ntrain_log:1.00e-5 00004692* 102.00 | 0.699 0.099 0.0000 0.000 | 0.056 0.061 | 0 hr 47 min
\n….
\ntrain_log:1.00e-5 00010212* 222.00 | 0.703 0.119 0.0000 0.000 | 0.037 0.045 | 1 hr 48 min
train_log:1.00e-5 00010350* 225.00 | 0.721 0.117 0.0000 0.000 | 0.035 0.043 | 1 hr 49 min
\ntrain_log:1.00e-5 00010488* 228.00 | 0.710 0.108 0.0000 0.000 | 0.039 0.047 | 1 hr 51 min
\ntrain_log:1.00e-5 00010626* 231.00 | 0.704 0.117 0.0000 0.000 | 0.039 0.046 | 1 hr 53 min
\ntrain_log:1.00e-5 00010764* 234.00 | 0.704 0.113 0.0000 0.000 | 0.039 0.048 | 1 hr 54 min
\n….
\nAnd the dice goes around 0.71…
use fold 3 for best results. if you still cannot get same results try different fold.
\ntry different model, aug to debug your pipline
lr of 1e-5 is low. 5e-5 for batch size 8 to 16
", "rawMarkdown": "lr of 1e-5 is low. 5e-5 for batch size 8 to 16", "votes": 1 }, { "id": 1941937, "postDate": "2022-09-16T10:55:37.257Z", "content": "Thank you for your reply,
\nI achieved approximately the same results as in your logs on fold3 (same fold as your) with b2. Without using SWA this single-fold model gives 0.73 on LB (I use better on valid loss (dice=0.778, loss=0.085) and better on valid dice (dice = 0.784, loss=0.103) separately - both have 0.73 LB).
\nAm I right with my models and it looks reasonable without SWA (i.e SWA may improve LB to about 0.79)? Looks pretty strange…
\nHere is my log file - https://www.kaggle.com/datasets/resourcermixer/mylogsfold3?select=log.train.txt
\nBelow plots of valid dice and loss during the training:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8349963%2F0bd8ec37ef3a8f9f45491d51a8e3f5e6%2F1.png?generation=1663325368388846&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8349963%2F2366c846cbeca058f3e9dd1c7cc251d5%2F2.png?generation=1663325383811047&alt=media\")
\n(Looks like I should take models near 200th epoch to avoid overfitting…)
\nHere is example of predicting (model with 252 epochs )
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8349963%2Fd772189e9f4fdd1a50aca9e31ad0b2e9%2Fex.png?generation=1663325550184964&alt=media\")
Have you any ideas?
", "rawMarkdown": "Thank you for your reply,\nI achieved approximately the same results as in your logs on fold3 (same fold as your) with b2. Without using SWA this single-fold model gives 0.73 on LB (I use better on valid loss (dice=0.778, loss=0.085) and better on valid dice (dice = 0.784, loss=0.103) separately - both have 0.73 LB).\nAm I right with my models and it looks reasonable without SWA (i.e SWA may improve LB to about 0.79)? Looks pretty strange... \nHere is my log file - https://www.kaggle.com/datasets/resourcermixer/mylogsfold3?select=log.train.txt\nBelow plots of valid dice and loss during the training:\n\n\n\n\n(Looks like I should take models near 200th epoch to avoid overfitting...)\nHere is example of predicting (model with 252 epochs )\n\n\nHave you any ideas?" }, { "id": 1943602, "postDate": "2022-09-17T16:28:45.890Z", "content": "Isn't fold dataset random?
\nLooking at the conversation between the two, it looks like a fixed dataset.
yes, It's fixed
", "rawMarkdown": "yes, It's fixed" } ] }, { "id": 1937356, "postDate": "2022-09-13T12:46:09.070Z", "content": "an interesting work
\n\" We additionally benchmark improvements in domain adaptation and out-of-distribution
\ndetection, and demonstrate that semi-supervised learning outperforms supervised learning in both cases.\"
Improving colonoscopy lesion classification using semi-supervised deep learning
\npaper: https://arxiv.org/pdf/2009.03162.pdf
\nhttps://durr.jhu.edu/pubs/2020-colossl/
![\"https://durr.jhu.edu/wp-content/uploads/2020/12/2020-SSL-1200x571.png\"](\"https://durr.jhu.edu/wp-content/uploads/2020/12/2020-SSL-1200x571.png\")
@hengck23, may I hear your opinion on my case? The problem is \"higher resolution trained model performed worse on Hubmap data than less res one, while better on HAP\" I have two CoaT_small models: one trained with 768, and the other with 1024. The 1024 model performed on HAP better than the 768 model (0.01~0.02 better) while its performance significantly worse on Hubmap data (0.03 worse). In order to have bigger receptive field for higher resolution, I have added a serial block (2 times repeat) and a parallel block (6 times repeat). However, it didn't go well for Hubmap data, although they have same lr, bs, and aug. I've assumed three possibilities. 1. 1024 dataset wrong, 2. need more aug, 3. change for loss function (currently for 768, bce + 0.2(aux1~3), for 1024, bce+0.2(aux1~4). Can you enlighten me if I've missed some important part in my experiments?
", "rawMarkdown": "@hengck23, may I hear your opinion on my case? The problem is \"higher resolution trained model performed worse on Hubmap data than less res one, while better on HAP\" I have two CoaT_small models: one trained with 768, and the other with 1024. The 1024 model performed on HAP better than the 768 model (0.01~0.02 better) while its performance significantly worse on Hubmap data (0.03 worse). In order to have bigger receptive field for higher resolution, I have added a serial block (2 times repeat) and a parallel block (6 times repeat). However, it didn't go well for Hubmap data, although they have same lr, bs, and aug. I've assumed three possibilities. 1. 1024 dataset wrong, 2. need more aug, 3. change for loss function (currently for 768, bce + 0.2(aux1~3), for 1024, bce+0.2(aux1~4). Can you enlighten me if I've missed some important part in my experiments? ", "replies": [ { "id": 1938756, "postDate": "2022-09-14T10:49:47.463Z", "content": "\"while its performance significantly worse on Hubmap data (0.03 worse). In order to have bigger receptive field for higher resolution, \"
\nover fitting and over detecting of smaller object
\ni can imagine if your try to draw the results on the test image (singe spleen image), the larger model detecs many more smaller blobs.
\n\n\n\n\nperformed on HPA better than the 768 model (0.01~0.02 better)
\n
\n this is better IOU loss (because of resolution)performance significantly worse on Hubmap data (0.03 worse).
\n
\n this is because of many more false positive instance
accuracy is affected by number of true/false positive/negative instance,
\nand for each instance, the IOU value
I see. Now I've got the direction to fix. Thank you very much for your advice.
", "rawMarkdown": "I see. Now I've got the direction to fix. Thank you very much for your advice." } ] }, { "id": 1930925, "postDate": "2022-09-08T10:47:44.067Z", "content": "Thank you very much for giving away your experiments and knowledge in detail. Mind if I ask your opinion? Do you think it's safe to use \"data_source, organ, and img_height/width\" in inference? I've been using those information to infer the test images but, if any of those information is null in the other test set, then my program would not complete. So I'm afraid of it. Maybe, it's not going to be a problem because inference program look through both test sets without an issue when we make an infer? Sorry for the dumb question.
", "rawMarkdown": "Thank you very much for giving away your experiments and knowledge in detail. Mind if I ask your opinion? Do you think it's safe to use \"data_source, organ, and img_height/width\" in inference? I've been using those information to infer the test images but, if any of those information is null in the other test set, then my program would not complete. So I'm afraid of it. Maybe, it's not going to be a problem because inference program look through both test sets without an issue when we make an infer? Sorry for the dumb question.", "replies": [ { "id": 1930928, "postDate": "2022-09-08T10:53:13.093Z", "content": "all \"data_source, organ, and img_height/width\" are available in hidden public and private test set.
\n\n\n\n\nif any of those information is null in the other test set, then my program would not complete.
\n
\n you you submit, you are already running for both public and private test set.
so there is no problem
", "rawMarkdown": "all \"data_source, organ, and img_height/width\" are available in hidden public and private test set.\n\n>> if any of those information is null in the other test set, then my program would not complete.\nyou you submit, you are already running for both public and private test set.\n\nso there is no problem\n", "votes": 1 }, { "id": 1931138, "postDate": "2022-09-08T13:23:00.467Z", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!" } ] }, { "id": 1930900, "postDate": "2022-09-08T09:59:41.610Z", "content": "when i train coat small(5 levels) at input 1024, i get results below ,kidney score is too low,can anyone give me some advice? i think it is the reason i only get 0.77 at lb, not 0.8(single fold3)
\nall 0.776699
\nkidney 0.882886
\nprostate 0.823085
\nlargeintestine 0.907251
\nspleen 0.854549
\nlung 0.265985
\n☹️
Verify it is not swa bug by disable swa. Just do validation for a single checkpoint first
", "rawMarkdown": "Verify it is not swa bug by disable swa. Just do validation for a single checkpoint first", "votes": 1 }, { "id": 1930911, "postDate": "2022-09-08T10:28:01.167Z", "content": "Check augmentation, hyper parameter etc correctness by smaller model. Eg just coat small parallel 4 level or coat medium without parallel or pvt v2 etc at 786. These should be at lb 0.78 or 0.79 at single fold3.
", "rawMarkdown": "Check augmentation, hyper parameter etc correctness by smaller model. Eg just coat small parallel 4 level or coat medium without parallel or pvt v2 etc at 786. These should be at lb 0.78 or 0.79 at single fold3.", "votes": 1 }, { "id": 1946640, "postDate": "2022-09-20T01:37:18.400Z", "content": "Is 0.8 lb model normalized by 0.4~ or 0.7~ ??
\nI want to try two of them but my gpu 50(4 batch 768)epoch take 12hour
Should we use aux0 loss when training?why
", "rawMarkdown": "Should we use aux0 loss when training?why", "replies": [ { "id": 1930133, "postDate": "2022-09-07T15:25:32.487Z", "content": "I suppose that the feature corresponding to aux0 loss is just downsampled by the final segmentation map, which has been contained in the final loss, so we do not need use aux0 loss.
", "rawMarkdown": "I suppose that the feature corresponding to aux0 loss is just downsampled by the final segmentation map, which has been contained in the final loss, so we do not need use aux0 loss." } ] }, { "id": 1914865, "postDate": "2022-08-26T13:43:35.550Z", "content": "Out of interest, could you post your predicted on the single test image of your best model (thresholded, or not, or both!)? I'm interested to know what the current leading solution thinks the test image should be predicted as!
", "rawMarkdown": "Out of interest, could you post your predicted on the single test image of your best model (thresholded, or not, or both!)? I'm interested to know what the current leading solution thinks the test image should be predicted as!", "replies": [ { "id": 1914901, "postDate": "2022-08-26T14:18:43.463Z", "content": "it is just the average of the below, depending on how i ensemble.
\nlarge model 1536 detects smaller objects
\ncswin always detect smaller objects.
difference models dectes differently.
\ntechnically, the smaller detections are really partial white pulp, but i am not sure if they are annotated in the hidden ground truth.
\n![\"Selection-088\"](\"https://i.ibb.co/jHqQVrF/Selection-088.png\")
",
"votes": 3
}
]
},
{
"id": 1913542,
"postDate": "2022-08-25T11:03:17.670Z",
"content": "i realize one important thing: almost all vision transformer starts with 4x4 non-overlapping block (or maybe 8x8)
\nthis initial layer causes some similarity in all transformer, and reduces diversity in ensemble?
the same hyper-parameter also make the similarity?
", "rawMarkdown": "the same hyper-parameter also make the similarity?" }, { "id": 1914034, "postDate": "2022-08-25T17:09:12.380Z", "content": "it depends. same fold from same seed, same augmentation will have similar results
", "rawMarkdown": "it depends. same fold from same seed, same augmentation will have similar results" } ] }, { "id": 1911581, "postDate": "2022-08-24T07:29:00.590Z", "content": "the race never ends …. MaxVit seems promising …
\nRoss Wightman
\n\"I've been working on a large model update over the past month+, incl several imports from research impl and timm originals of CoAtNet (https://arxiv.org/abs/2106.04803) and MaxVit (https://arxiv.org/abs/2204.01697) in a combined MaxxVit impl. Soon to be merged w/ a few weights.\"
maybe useful
\nMesa: A Memory-saving Training Framework for Transformers
48G GPU Mem .. such expensive game. 💔
", "rawMarkdown": "48G GPU Mem .. such expensive game. 💔", "replies": [ { "id": 1904773, "postDate": "2022-08-18T13:22:40.550Z", "content": "\"48G GPU Mem .. such expensive game\"
\nhaving high memory simplifes my work.
\nbut you can achieve the same performance with lower memory system, e.g. using tiling, check gradient, etc.
\nit would just take more time and efforts.
the good news is that I am sharing my results here so that you can do \"less experiments\" to verify if your code is correct or bug
", "rawMarkdown": "\"48G GPU Mem .. such expensive game\"\n\nhaving high memory simplifes my work.\nbut you can achieve the same performance with lower memory system, e.g. using tiling, check gradient, etc.\nit would just take more time and efforts.\n\nthe good news is that I am sharing my results here so that you can do \"less experiments\" to verify if your code is correct or bug", "votes": 3 }, { "id": 1906156, "postDate": "2022-08-19T16:24:53.963Z", "content": "Is that using one single GPU or a multiple setup?
", "rawMarkdown": "Is that using one single GPU or a multiple setup? " }, { "id": 1906593, "postDate": "2022-08-20T03:19:16.543Z", "content": "There is 2 GPUs in the HP Z workstation.
\nEach has 48 GB.
\nHence total memory is 96 GB.
i am in the progress of running results for [1]
\n[1] Dual Vision Transformer
\nhttps://arxiv.org/pdf/2207.04976.pdf
\n https://github.com/YehLi/ImageNetModel
i think it is very promising.
\ninitial loss curve and validation loss already surpass that of PVT v2.
\n
\nthis is another 0.79 model with input=768
I follow your work of segformer， but i did not get the same result ，the Hubmap score is 0.55 same as yours， but HPA score is only 0.21~
", "rawMarkdown": "I follow your work of segformer， but i did not get the same result ，the Hubmap score is 0.55 same as yours， but HPA score is only 0.21~", "replies": [ { "id": 1903892, "postDate": "2022-08-17T18:18:25.457Z", "content": "\"HPA score is only 0.21\"
\ni think 0.21 is the normal score.
\nyou can higher, if you \"inject random noise\" at inference at post-processing. since the method is not stable and seems to only affects HPA (?) so I will not dwell into it. I think some of the prediction are \"borderline\" because of the labeling noise.
(e.g. you can add noise to decoder, or last few block of either of the 4 layers, or addn ew blocks (with random initialisation) and observe results. i once mistaken load a mit_b2 trained weights into mit_b3 model and surprised that it still work)
\nsince segform has license issue, I suggest switching to better models.
", "rawMarkdown": "\"HPA score is only 0.21\"\n\ni think 0.21 is the normal score.\nyou can higher, if you \"inject random noise\" at inference at post-processing. since the method is not stable and seems to only affects HPA (?) so I will not dwell into it. I think some of the prediction are \"borderline\" because of the labeling noise.\n\n(e.g. you can add noise to decoder, or last few block of either of the 4 layers, or addn ew blocks (with random initialisation) and observe results. i once mistaken load a mit_b2 trained weights into mit_b3 model and surprised that it still work)\n\nsince segform has license issue, I suggest switching to better models.", "votes": 1 }, { "id": 1904159, "postDate": "2022-08-18T02:27:22.807Z", "content": "I tried segformer too. b2 can get pb 0.76, but b4 and b5 is better can achieve 0.78(b5 is better)
", "rawMarkdown": "I tried segformer too. b2 can get pb 0.76, but b4 and b5 is better can achieve 0.78(b5 is better)" }, { "id": 1904166, "postDate": "2022-08-18T02:36:58.237Z", "content": "\"(b5 is better)\"
\nthe trick is to expand your data (e.g. extreme augmentation, external data, GAN, noise, etc … or greater size) and use a strong transformer. But one has to be careful not to learn \"too much noise\".
\ni have a feeling that 0.81 may be possible with one fold, but it is a difficult target to achieve
I thank you for sharing your experiment results. They're very informative and inspirational. By the way, mind if I ask a question about the terminology \"one fold prediction.\" ? Does that mean you do 5-fold cross validation and pick one random fold out of the five folds in order to predict? Sorry about the dumb question though.
", "rawMarkdown": "I thank you for sharing your experiment results. They're very informative and inspirational. By the way, mind if I ask a question about the terminology \"one fold prediction.\" ? Does that mean you do 5-fold cross validation and pick one random fold out of the five folds in order to predict? Sorry about the dumb question though." }, { "id": 1908951, "postDate": "2022-08-22T06:53:18.780Z", "content": "@Cheul I think is select the better performing ones previously, not random choose
", "rawMarkdown": "@Cheul I think is select the better performing ones previously, not random choose", "votes": 1 }, { "id": 1908958, "postDate": "2022-08-22T07:00:44.003Z", "content": "selecting the one closest to your average of all
", "rawMarkdown": "selecting the one closest to your average of all ", "votes": 1 }, { "id": 1908991, "postDate": "2022-08-22T07:39:34.140Z", "content": "I see. Thank you!
", "rawMarkdown": "I see. Thank you!" } ] }, { "id": 1903376, "postDate": "2022-08-17T11:24:13.840Z", "content": "Are we segmenting normal glomeruli, sclerosed glomeruli or any type of glomeruli?
", "rawMarkdown": "Are we segmenting normal glomeruli, sclerosed glomeruli or any type of glomeruli?" }, { "id": 1903332, "postDate": "2022-08-17T10:15:44.777Z", "content": "i now become confused by my results:
\nking of king: single fold LB 0.80
\nhttps://www.kaggle.com/code/hengck23/lb-0-80-coat-small-parallel-at-1024
", "rawMarkdown": "i now become confused by my results:\n\nking of king: single fold LB 0.80\n\n- COAT parallel small 5 level (I modified the code to change from 4 to 5 level)\n- increase image size from 768 to 1024\n\nhttps://www.kaggle.com/code/hengck23/lb-0-80-coat-small-parallel-at-1024" }, { "id": 1903092, "postDate": "2022-08-17T05:15:24.900Z", "content": "i came to realise that Beit is the chnapion solution for Visual Domain Adaptation Challenge 2021:
\nhttps://ai.bu.edu/visda-2021/assets/pdf/Burhan_Report.pdf
\nhttps://ai.bu.edu/visda-2021/assets/pdf/Burhan_Slides.pdf
Was curious @hengck23 did you experiment with Beit…. I did not try but was thinking of semi supervised training with best model like noisy student approach
", "rawMarkdown": "Was curious @hengck23 did you experiment with Beit.... I did not try but was thinking of semi supervised training with best model like noisy student approach" }, { "id": 1930377, "postDate": "2022-09-07T19:44:32.647Z", "content": "i have been waiting for it,
\nbeit2 code is out a few days ago !
i am interested in the Visual Tokenizer (VQ-KD)
\nhttps://github.com/microsoft/unilm/tree/master/beit2
", "rawMarkdown": "i have been waiting for it, \nbeit2 code is out a few days ago !\n\ni am interested in the Visual Tokenizer (VQ-KD)\n\nhttps://github.com/microsoft/unilm/tree/master/beit2\n\n", "votes": 1 } ] }, { "id": 1897281, "postDate": "2022-08-13T16:02:57.457Z", "content": "find a paper :
\nCROSS-STAINED SEGMENTATION FROM RENAL BIOPSY IMAGES USING MULTI-LEVELADVERSARIAL LEARNING
\nhttps://arxiv.org/pdf/2002.08587.pdf
\" Experimental results on glomeruli segmentation from renal biopsy images indicate that our network is able to improve segmentation performance on target type of stained images and use unlabeled data to achieve similar accuracy to labeled data\"
", "rawMarkdown": "find a paper :\n\nCROSS-STAINED SEGMENTATION FROM RENAL BIOPSY IMAGES USING MULTI-LEVELADVERSARIAL LEARNING\nhttps://arxiv.org/pdf/2002.08587.pdf\n\n\" Experimental results on glomeruli segmentation from renal biopsy images indicate that our network is able to improve segmentation performance on target type of stained images and use unlabeled data to achieve similar accuracy to labeled data\"\n" }, { "id": 1896239, "postDate": "2022-08-12T17:22:47.380Z", "content": "which is more important, backbone or decoder?
\n
"
},
{
"id": 1894597,
"postDate": "2022-08-11T15:40:44.293Z",
"content": "hi,
\nFirst, thanks for your sharing. I am new to the kaggle competetion. I studied your shared code. I have a dumb question. What methods can I use to filter out a few relatively well-performing model weights in a fold? like the code in your run_local_cv.py.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7991823%2F66a81de04818d64831ef0d95aa27fbfe%2F_20220811234012.png?generation=1660232428007187&alt=media\")
i just select from the log file.
\nthat is good enough.
\nselect some top model that are \"well spaced\" from each other
![\"https://user-images.githubusercontent.com/14368801/37633888-89fdc05a-2bca-11e8-88aa-dd3661a44c3f.png\"](\"https://user-images.githubusercontent.com/14368801/37633888-89fdc05a-2bca-11e8-88aa-dd3661a44c3f.png\")
w1,w2,w3 are the selected model.
\nw_swa is the final model combined.
you need to imagine how the hidden test loss landscape (not the gradient descend train loss landscape) looks like so that w_swa is in the center of a flat valley at test
\nif you want to automatic, you can do a non max suppression in code:
\nfor complicated selection, refer to paper:
\nsearch for papers reference the above
\nfor loss landscape, refer to:
\nhttps://vitalab.github.io/article/2020/05/01/lossLandscape.html
", "rawMarkdown": "i just select from the log file.\nthat is good enough.\nselect some top model that are \"well spaced\" from each other\n\n\n\nw1,w2,w3 are the selected model.\nw\\_swa is the final model combined.\n\n\nyou need to imagine how the hidden test loss landscape (not the gradient descend train loss landscape) looks like so that w\\_swa is in the center of a flat valley at test\n\n\n---\n\nif you want to automatic, you can do a non max suppression in code:\n- order all orders from best score to worse\n- select the best\n- discard model that are n iterations from the selected best\n- repeat unit you have selected K best and not nearby models\n\nfor complicated selection, refer to paper:\n- https://arxiv.org/abs/2203.05482\n- SWA : https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/\n\nsearch for papers reference the above\n\n---\n\nfor loss landscape, refer to:\n\nhttps://vitalab.github.io/article/2020/05/01/lossLandscape.html", "votes": 4 } ] }, { "id": 1887778, "postDate": "2022-08-07T03:03:38.020Z", "content": "i haven't got good results for transformer decoder. another want to share their results?
", "rawMarkdown": "i haven't got good results for transformer decoder. another want to share their results?" }, { "id": 1887743, "postDate": "2022-08-07T01:43:55.243Z", "content": "maybe a good way to use stain normalisation
\n![\"https://i.ibb.co/jGzL6PD/Selection-088.png\"](\"https://i.ibb.co/jGzL6PD/Selection-088.png\")
i made some calculation ….
\neveryone will be stuck in ~0.80. then one more decimal in LB score will be reveal.
\nonce it is in 3 decimal, score per image will be revealed. magic may happen as we may be able to \"get/probe more information\" for e.g. the test spleen image.
", "rawMarkdown": "i made some calculation ....\n\neveryone will be stuck in ~0.80. then one more decimal in LB score will be reveal.\n\nonce it is in 3 decimal, score per image will be revealed. magic may happen as we may be able to \"get/probe more information\" for e.g. the test spleen image.", "replies": [ { "id": 1887746, "postDate": "2022-08-07T01:57:57.710Z", "content": "😲I've made a Hubmap score over 0.58 (normed to 0.803). However, my HPA score is not as good as your segformer's 0.23. I think getting 0.81 or higher is not so difficult.
", "rawMarkdown": "😲I've made a Hubmap score over 0.58 (normed to 0.803). However, my HPA score is not as good as your segformer's 0.23. I think getting 0.81 or higher is not so difficult." }, { "id": 1887767, "postDate": "2022-08-07T02:45:14.560Z", "content": "maybe this is the reason.
\nthe experiment compares segformers and daformer and also some of the methods mentioned in the paper (e.g. sampling rare classes, in our case it would be sampling low score images)
\n![\"https://i.ibb.co/qjJpqfy/Selection-089.png\"](\"https://i.ibb.co/qjJpqfy/Selection-089.png\")
\"I've made a Hubmap score over 0.58\"
\nfor my current LB score of 0.80, my LB-Hubmap is also 0.58.
\ni believe all kagglers with LB 0.80 have about the same results.
i think there is strong correlation for LB-Hubmap and LB-HPA.
\nBut once you reach near 0.80, dice scores starts to \"fluctuate\"
Maybe always checking 5-fold ensemble is good for reducing unstable results. I have made many 0.58 scores with different submission configurations (add/remove/replace models).
", "rawMarkdown": "Maybe always checking 5-fold ensemble is good for reducing unstable results. I have made many 0.58 scores with different submission configurations (add/remove/replace models)." } ] }, { "id": 1883056, "postDate": "2022-08-03T15:03:16.480Z", "content": "Hello~ I am quite curious about how can you get your pre-trained mit weights? Did you use mask image modeling? I noted that official implementation of segformer-mit-b2 is in mmseg, however your code is different from the official ones.
", "rawMarkdown": "Hello~ I am quite curious about how can you get your pre-trained mit weights? Did you use mask image modeling? I noted that official implementation of segformer-mit-b2 is in mmseg, however your code is different from the official ones.", "replies": [ { "id": 1883679, "postDate": "2022-08-04T02:36:34.280Z", "content": "https://github.com/NVlabs/SegFormer
\nuse imagenet pretrain
Hello, I'm curious about the way categories are handled. Since in the end we only need to submit the rle encoding for each image, we don't need to provide categories. What way your model is trained, multi-classification or binary(organ/not organ)？
", "rawMarkdown": "Hello, I'm curious about the way categories are handled. Since in the end we only need to submit the rle encoding for each image, we don't need to provide categories. What way your model is trained, multi-classification or binary(organ/not organ)？", "replies": [ { "id": 1879847, "postDate": "2022-08-01T09:49:06.380Z", "content": "you should do experiment to confirm which is better. And better still, to explain why.
\nit depends if the different target objects have similarity of not, e.g. different organ FTU have some same basic cells.
", "rawMarkdown": "you should do experiment to confirm which is better. And better still, to explain why.\n\nit depends if the different target objects have similarity of not, e.g. different organ FTU have some same basic cells." } ] }, { "id": 1879713, "postDate": "2022-08-01T08:02:20.883Z", "content": "Hey @hengck23! Thank you for sharing all the info. I wanted to ask whether you are training with full size images or tiled ones? Also, how are you dealing with different pixel dimensions of HPA and HubMAP?
", "rawMarkdown": "Hey @hengck23! Thank you for sharing all the info. I wanted to ask whether you are training with full size images or tiled ones? Also, how are you dealing with different pixel dimensions of HPA and HubMAP?" }, { "id": 1877715, "postDate": "2022-07-31T00:23:25.383Z", "content": "did anyone suceed in training from scratch without image-net pretrain?
", "rawMarkdown": "did anyone suceed in training from scratch without image-net pretrain?" }, { "id": 1877635, "postDate": "2022-07-30T21:21:56.853Z", "content": "Awesome discussion till now to the competition
", "rawMarkdown": "Awesome discussion till now to the competition " }, { "id": 1875228, "postDate": "2022-07-28T19:42:24.417Z", "content": "masked auto encoder for SSL
\n![\"https://i.ibb.co/vQpDC3j/Selection-099.png\"](\"https://i.ibb.co/vQpDC3j/Selection-099.png\")
there is actually a way to get very good results for hidden private test data.
\nIf you can do ONLINE training at submission within 9 hrs.
SSL reconstruction error is accessible at both train and test time. It is correlated to target dice loss, accessible only at train time (when we have ground truth segmentation mask)
\nNow by monitoring the reconstruction error for SSL at test time, we can estimate the dice score (e.g. and then decide to use online model or offline model prediction)
\nonline learning:
\n![\"https://i.ibb.co/k5RjpPj/Selection-100.png\"](\"https://i.ibb.co/k5RjpPj/Selection-100.png\")
if the hypothesis that SSL and SL loss are correlated is correct, we have a way to detect label noise
\nthough it may not be possible to implement for this competition, it may be a good idea for exploration for research and paper writing.
", "rawMarkdown": "there is actually a way to get very good results for hidden private test data.\nIf you can do ONLINE training at submission within 9 hrs.\n\nSSL reconstruction error is accessible at both train and test time. It is correlated to target dice loss, accessible only at train time (when we have ground truth segmentation mask)\n\nNow by monitoring the reconstruction error for SSL at test time, we can estimate the dice score (e.g. and then decide to use online model or offline model prediction)\n\n\nonline learning:\n\n\nif the hypothesis that SSL and SL loss are correlated is correct, we have a way to detect label noise\n\nthough it may not be possible to implement for this competition, it may be a good idea for exploration for research and paper writing.", "votes": 2 }, { "id": 1875809, "postDate": "2022-07-29T10:04:01.260Z", "content": "there is yet another easier way:
\nassue you trained an masked auto encoder.
\non local validation set (or external set), you reconstruction error = E1
now apply masked auto encoder on private+public test set. you can use probing to output the reconstruction error E2.
\nby analysing E1 and E2 you probably can estimate the domain shift of the two dataset. if the auto encoder has seen the test in the training set, reconstruction error would be low.
\nnot: this is not the same as observing the submission score, which only computes for the public set in public score.
\nmy method reveals both private and test set characteristics
![\"https://i.ibb.co/xjMtjD8/Selection-114.png\"](\"https://i.ibb.co/xjMtjD8/Selection-114.png\")
i believe this is a good direction to go. The downside is that:
\n1) the training batch size needs to be large and many epoches are required.
\n2) it is two stage: pretrain and then finetune to segmentation task
transformer are indeed good autoencoder
\nI just read some papers that can directly fine-tune ecoder-decoder directly using masked patch in one stage. i will try that next
", "rawMarkdown": "\n\ni believe this is a good direction to go. The downside is that:\n1) the training batch size needs to be large and many epoches are required.\n2) it is two stage: pretrain and then finetune to segmentation task\n\ntransformer are indeed good autoencoder\n\nI just read some papers that can directly fine-tune ecoder-decoder directly using masked patch in one stage. i will try that next", "votes": 1 }, { "id": 1886204, "postDate": "2022-08-05T16:23:57.310Z", "content": "Thanks for your kind share, but i still have a question that during my own experiments, i find that val dice will shake after several epochs, so is that more epochs training necessary? My experiments is based on your segformer-mit-b2 and I send fold0, 160epochs weights get 0.66 lb without any post-process. I see no improvement of validation after 110 epoch.
", "rawMarkdown": "Thanks for your kind share, but i still have a question that during my own experiments, i find that val dice will shake after several epochs, so is that more epochs training necessary? My experiments is based on your segformer-mit-b2 and I send fold0, 160epochs weights get 0.66 lb without any post-process. I see no improvement of validation after 110 epoch.", "votes": 2 }, { "id": 1886617, "postDate": "2022-08-06T03:24:11.167Z", "content": "For me, fold 0, 360 epochs, with out SWA, segformer-mit-b2 get 0.63lb, the same parameter as before. So, I suppose that SWA makes sense to get 0.7+, and I will do experiments to check my ideas.
", "rawMarkdown": "For me, fold 0, 360 epochs, with out SWA, segformer-mit-b2 get 0.63lb, the same parameter as before. So, I suppose that SWA makes sense to get 0.7+, and I will do experiments to check my ideas.", "votes": 2 }, { "id": 1886618, "postDate": "2022-08-06T03:25:41.360Z", "content": "I think simple large training epoch without SWA can easily result in overfitting .
", "rawMarkdown": "I think simple large training epoch without SWA can easily result in overfitting .", "votes": 1 }, { "id": 1924713, "postDate": "2022-09-03T10:20:38.077Z", "content": "Thank you for your shared ideas, it's helpful to study new architecture and training ideas.
\nI'm currently experimenting with MAE pretraining of CoaT, with training on kaggle data / large intestine+lung patches
\nI'm using similar fuse decoder as Daformer (so it uses Serial + Parallel blocks the same way as in segmentation), and tried pixelshuffle/conv decoder, and tried to remove BN from decoder. .
\nToken masking is ported from PVT UMMAE.
\nThese setups keep giving blurry results even after 100+ epochs, so it is still not able to memorize training patches.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1590882%2F1b7173f22a684a2603b1a7784a9fe6d1%2Findex.png?generation=1662200102037512&alt=media\")
L1 loss, bs = 32, total ~2500 patches
\nIs it underfitting? Should I train for more? Maybe there is some other advice where may be problem? Maybe some architecture issues?
\nIt's okay to skip \"unknown\" regions without any \"signal\", but it is unable to complete \"circles at FTU\" and simple shapes. (But it captures huge white regions)
\nIt's my first time working with MIM. Thanks!
", "rawMarkdown": "Thank you for your shared ideas, it's helpful to study new architecture and training ideas.\n\nI'm currently experimenting with MAE pretraining of CoaT, with training on kaggle data / large intestine+lung patches\n\nI'm using similar fuse decoder as Daformer (so it uses Serial + Parallel blocks the same way as in segmentation), and tried pixelshuffle/conv decoder, and tried to remove BN from decoder. .\n\nToken masking is ported from PVT UMMAE.\n\nThese setups keep giving blurry results even after 100+ epochs, so it is still not able to memorize training patches.\n\n\n\n\n\nL1 loss, bs = 32, total ~2500 patches\n\nIs it underfitting? Should I train for more? Maybe there is some other advice where may be problem? Maybe some architecture issues?\n\nIt's okay to skip \"unknown\" regions without any \"signal\", but it is unable to complete \"circles at FTU\" and simple shapes. (But it captures huge white regions)\n\nIt's my first time working with MIM. Thanks!" }, { "id": 1924870, "postDate": "2022-09-03T13:28:57.753Z", "content": "if you reduce the number of patches (e.g. to 100,50, 10), you will find that the network can memories the train samples.
\nif it still cannot, something wrong with your optimization or learning rate.
\n(i suggest SImMIM, which i find easier to converge)
i think MAE/SImMIM cannot produce sharp images.
\n(you can check the reconstruction examples in the paper).
this also means that given some patch, there are so many possible to reconstruct so that the mean image of this reconstruction is not sharp.
\ni suggest:
\nSimMIM is easier to work with
you can increase the num of unmasked patch (i.e. if there is more conditioning, the num possible ground truth is \"less\", i.e. more sharp reconstruction)
maybe it is ok the reconstruction is blur. you should check your segmentation results to see if there is improvement
maybe we don't predict color per pixel. how about predict block features like HOG, wavelet coefficients, etc … you can make dictionaries of the patch (like image tokenization, reconstruction image basis, etc) … we restrict the prediction space so that only sharp prediction can be made
add more conditions for patch to be predicted e.g. instead of input mask token, you can use low resolution, etc
forget about prediction appearance. use other target, e.g.
\ninput = shuffle location, predict= unshuffle location
\"(But it captures huge white regions)\"
\ntransformer are low pass filter. it is good at predicting large structure
(you can image if you zoom your image 20x, then the prediction will be good. but then you need to build a very deep pyramid and many many more train samples. you inference engine also need to be change)
", "rawMarkdown": "if you reduce the number of patches (e.g. to 100,50, 10), you will find that the network can memories the train samples.\n\nif it still cannot, something wrong with your optimization or learning rate.\n(i suggest SImMIM, which i find easier to converge)\n\n\n#---\n\n\ni think MAE/SImMIM cannot produce sharp images.\n(you can check the reconstruction examples in the paper).\n\nthis also means that given some patch, there are so many possible to reconstruct so that the mean image of this reconstruction is not sharp.\n\n\ni suggest:\n1. SimMIM is easier to work with \n\n2. you can increase the num of unmasked patch (i.e. if there is more conditioning, the num possible ground truth is \"less\", i.e. more sharp reconstruction)\n\n3. maybe it is ok the reconstruction is blur. you should check your segmentation results to see if there is improvement\n\n4. maybe we don't predict color per pixel. how about predict block features like HOG, wavelet coefficients, etc ... you can make dictionaries of the patch (like image tokenization, reconstruction image basis, etc) ... we restrict the prediction space so that only sharp prediction can be made\n\n5. add more conditions for patch to be predicted e.g. instead of input mask token, you can use low resolution, etc\n\n4. forget about prediction appearance. use other target, e.g. \ninput = shuffle location, predict= unshuffle location\n\n---\n\n\"(But it captures huge white regions)\"\ntransformer are low pass filter. it is good at predicting large structure\n\n(you can image if you zoom your image 20x, then the prediction will be good. but then you need to build a very deep pyramid and many many more train samples. you inference engine also need to be change)", "votes": 1 } ] }, { "id": 1869213, "postDate": "2022-07-24T15:22:39.403Z", "content": "masked VAE for pyramid-based Vit like Swin transformer:
\nhttps://github.com/implus/UM-MAE
update on trasnformer results:
\nconsider the performance of imagenet, segformer mit-b2 results is ok. but it doesn't scale up when I use more powerful mit-b4
\n![\"https://i.ibb.co/QJB16HX/Selection-070.png\"](\"https://i.ibb.co/QJB16HX/Selection-070.png\")
updated transformer results for upernet + swin v1 transformers
\nwith proper regularization and more augmentation you get very good results if you ran for more iterations!
\n![\"https://i.ibb.co/0DqmS7F/Selection-072.png\"](\"https://i.ibb.co/0DqmS7F/Selection-072.png\")
why not . . . 1024x1024 cityscape one? for segformer . . . And any specific reason to interpolate to \"nearest\" - I am not pretty much clear \"bilinear\" and \"nearest\" stuff?
", "rawMarkdown": "why not . . . 1024x1024 cityscape one? for segformer . . . And any specific reason to interpolate to \"nearest\" - I am not pretty much clear \"bilinear\" and \"nearest\" stuff?" }, { "id": 1867540, "postDate": "2022-07-23T10:05:40.727Z", "content": "'1024x1024 cityscape on'
\neach task has a good scale to work for accuracy and speed/menory. i find that the tissue organ are of different size and resizing 3000 to 768 is the best scale for me.
\n' specific reason to interpolate to \"nearest\"'
\nit work best for me. nearest has the largest of num of strong activation pixels when upsized, it results in better convergence for this competition.
also segformer decoder is 1x1 conv. it cannot mix nearby pixel
\nthe best solution is a learning mixing variable for
\nmix * F.interpolate(bilinear) + (1-mix)*F.interpolate(nearest)
", "rawMarkdown": "'1024x1024 cityscape on' \n\neach task has a good scale to work for accuracy and speed/menory. i find that the tissue organ are of different size and resizing 3000 to 768 is the best scale for me.\n\n' specific reason to interpolate to \"nearest\"'\nit work best for me. nearest has the largest of num of strong activation pixels when upsized, it results in better convergence for this competition. \n\nalso segformer decoder is 1x1 conv. it cannot mix nearby pixel\n\nthe best solution is a learning mixing variable for\n\nmix * F.interpolate(bilinear) + (1-mix)*F.interpolate(nearest) ", "votes": 1 }, { "id": 1867848, "postDate": "2022-07-23T14:46:43.040Z", "content": "@hengck23 it was great , I was experimenting with \"bilinear\" and \"nearest\" actually in my experiments my CV was more with \"bilinear\" that's why I sticked with it, but when I submitted with mode as \"nearest\" my Cv-LB gap was only 0.02where earlier gap was like 0.13 - 0.14.
\nthough my training baseline is WIP - - augmentations, staining, and hyperparameters///
\nwill surely look into the
\n", "rawMarkdown": "@hengck23 it was great , I was experimenting with \"bilinear\" and \"nearest\" actually in my experiments my CV was more with \"bilinear\" that's why I sticked with it, but when I submitted with mode as \"nearest\" my Cv-LB gap was only **0.02**where earlier gap was like **0.13 - 0.14**.\nthough my training baseline is WIP - - augmentations, staining, and hyperparameters///\nwill surely look into the \n> best solution is a learning mixing variable for\n- mix * F.interpolate(bilinear) + (1-mix)*F.interpolate(nearest)", "votes": 1 }, { "id": 1870184, "postDate": "2022-07-25T10:59:34.343Z", "content": "best solution is a learning mixing variable for
\n\n
\n- mix * F.interpolate(bilinear) + (1-mix)*F.interpolate(nearest)
\n
class MixUpSample(nn.Module):\n def __init__( self, scale_factor=2):\n super().__init__()\n self.mixing = nn.Parameter(torch.tensor(0.5))\n self.scale_factor = scale_factor\n\n def forward(self, x):\n x = self.mixing *F.interpolate(x, scale_factor=self.scale_factor, mode='bilinear', align_corners=False) \\\n + (1-self.mixing )*F.interpolate(x, scale_factor=self.scale_factor, mode='nearest')\n return x\nhttps://www.youtube.com/watch?v=AqscP7rc8_M great video by computerphile 👍
", "rawMarkdown": "https://www.youtube.com/watch?v=AqscP7rc8_M great video by computerphile 👍" } ] }, { "id": 1863895, "postDate": "2022-07-20T15:23:43.003Z", "content": "anyone can explin why am i getting patch token artifacts when using transformer segmentation (using segformer)?
\nit turns up this is due to F.interpolate using mode='nearest' when i upsample the mit transformer layer results.
![\"Selection-067\"](\"https://i.ibb.co/BKjDsjL/Selection-067.png\")
\n![\"Selection-066\"](\"https://i.ibb.co/RS1Yd69/Selection-066.png\")
\n"
},
{
"id": 1945984,
"postDate": "2022-09-19T14:41:52.717Z",
"rawMarkdown": "",
"isDeleted": true,
"replies": [
{
"id": 1946894,
"postDate": "2022-09-20T06:50:48.333Z",
"content": "It seems that you've tried to add one additional block while in the code you didn't add any function to control the extra block such as range(4) -> range(5).
Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!" }, { "id": 1886151, "postDate": "2022-08-05T16:06:35.417Z", "content": "Thank you so much for sharing!
", "rawMarkdown": "Thank you so much for sharing!" }, { "id": 1886002, "postDate": "2022-08-05T13:57:20.580Z", "content": "thanks for the author, very helpful
", "rawMarkdown": "thanks for the author, very helpful" } ], "comments": [ { "id": 1871062, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-26T03:58:40.490000", "content": "surprise!!!!
\nthe winner is 5-fold segformer-mit-b2 with CV=0.787, LB=0.78.
\nThe model size is only 99mb.
I haven't investigate the reasons, but I think it could be:
\nGood results, I'm also trying segformer without pretrained weights from imagenet, and fold0 cv=0.76. Did you use pretrained weights?
", "votes": 2, "replies": [] }, { "id": 1871222, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-26T06:41:57.667000", "content": "i use pretrained weight.
\ni just do another probe: Hubmap-LB is an amazing 0.55 (normalised to 0.762142857)
more surprise!!!
\nsingle-fold (fold-3) segformer-mit-b2 alone is already LB 0.79 (not low 0.79+ but a good 0.79+)
breakdown:
\nlocal cv HPA = 0.783965
\nnote that score on submission page is trauncated (?) and not rounded
\neither:
\n@hengck23 I am trying with segformer, I am not sure that I am training correct, I don't have much time for two weeks to test but I am getting values such as:
\nMean_iou: 0.7336344062222001
\nMean accuracy: 0.7336344062222001
in epoch 1 ! using 256 tiles and I am not sure this is normal or not, I am using a demo starting code which is not complex !!
", "votes": 1, "replies": [] }, { "id": 1873065, "author_name": "Ali", "author_url": "", "post_date": "2022-07-27T12:31:51.163000", "content": "I smell something wrong, I just modified the basic training code with this competition dataset and rewrote the evaluation process.
", "votes": 1, "replies": [] }, { "id": 1873232, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-27T13:54:23.303000", "content": "@asalhi I don't have much time for two weeks to test but I am getting values such as:\"
\nI am not sure if 256 tile is good enough. i trained with images resized from 3000 to 768.
\nthere is a couple of tricks that may be important:
please see the new added folder at \"segformer-mit-b2\", \"train-log-segformer-mit-b2\"
\nhttps://www.kaggle.com/datasets/hengck23/hubmap-discuss-00
Thanks so much for sharing your knowledge. As always very valuable.
\nOne question I have is about the license to use your model. On SegFormer's Github it says use is for non-commercial only. Wouldn't that conflict with the competition license (open source) in case of winning?
", "votes": 1, "replies": [] }, { "id": 1873474, "author_name": "ananzeng", "author_url": "", "post_date": "2022-07-27T16:17:55.387000", "content": "@hengck23 hi I want ask you some question, first congratulations on your good scores!
\n\"I am not sure if 256 tile is good enough. i trained with images resized from 3000 to 768.\"
\nSo you didn't use tiles? Just resizing to 768?
", "votes": 0, "replies": [] }, { "id": 1873772, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-27T22:32:22.497000", "content": "
thanks for sharing. May I ask:
\nwhat makes you use 'aux2_loss' alone instead of using 4 'aux[%d]_loss' outputs ?
\nno lr schduler applied?
by experiment results. it is trial and error
", "votes": 0, "replies": [] }, { "id": 1874381, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-28T08:40:46.413000", "content": "@robsonsan \"Wouldn't that conflict with the competition license (open source) in case of winning?\"
\nthanks for pointing that out. i would consider that a month later (to see if I am still a prize contender).
\nperformance is limited by data and not method. so segformer can be easily replaced. i just have to investigate the results for its good performance :)
on a side note:
\n
did another experiment:
\nswin-v1-small + segformer (single fold only , fold 3)
\nLB (0.76+)
\nbreakdown:
\nLB-Humap (0.54+)
\nLB-HPA (0.76+) - (0.54+) = (0.22+) to (0.23+)
local cv:
\nall 0.787098
\nkidney 0.951417
\nprostate 0.815064
\nlargeintestine 0.903701
\nspleen 0.852030
\nlung 0.230929
Lung segmentation is a challenge …
", "votes": 0, "replies": [] }, { "id": 1874624, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-28T11:46:43.687000", "content": "\"Lung segmentation\" …
\nthis one you probably have to use some tricks.
\ne.g.
\nyou can check the size of sizes of the FTU from the json files. i suspect maybe you can can use size to filter the results, etc. but I am not sure.
you can relabel the train image to include
it is mentioned \"there is at least one FTU per image\", I haven't probe the missing rate in LB set
Thanks for the tips. I'm also working on some other hypotheses to see if my models can generalize to lung tissue. For now I'm avoiding using external data
", "votes": 0, "replies": [] }, { "id": 1894011, "author_name": "_CA℟L_", "author_url": "", "post_date": "2022-08-11T08:10:50.383000", "content": "@hengck23 FYI - I think the license was changed recently
\n
oops, someone discovered our post
", "votes": 1, "replies": [] }, { "id": 1894356, "author_name": "跟着丞相割麦子", "author_url": "", "post_date": "2022-08-11T12:43:21.980000", "content": "
finally
", "votes": 0, "replies": [] }, { "id": 1911216, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-08-24T01:18:54.190000", "content": "Can we use segformer or not? I\"m not an expert in License issues.
", "votes": 2, "replies": [] }, { "id": 1911580, "author_name": "Cheul", "author_url": "", "post_date": "2022-08-24T07:28:48.237000", "content": "I think we can use Segformer unless we claim to the prize.
", "votes": 0, "replies": [] }, { "id": 1912166, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-08-24T14:53:48.993000", "content": "@cheulkay Thanks for your clarification
", "votes": 0, "replies": [] }, { "id": 1934514, "author_name": "parkdul", "author_url": "", "post_date": "2022-09-11T12:20:55.087000", "content": "how do I know overffiting in training?
", "votes": 0, "replies": [] } ] }, { "id": 1832268, "author_name": "hengck23", "author_url": "", "post_date": "2022-06-24T22:03:05.767000", "content": "how many hubmap images are there in the test set:
\n
we know:
\nnum_hubmap = len(test_df[test_df.data_source=='Hubmap'])\nnum_hpa = len(test_df[test_df.data_source=='HPA'])\nnum = len(test_df)\n\nif num_hubmap > int(0.90*num):\n submit_df.to_csv('submission.csv',index=False)\nsmarter way to probe without wasting submission slot
\n# some task which you already know the notebook running time\n\nsleep (num_hubmap*10_min)\n\n#you can deduce value of num_hubmap from additional running time\n# some monotnic function that you already know\n if (num> 0) and (num<=545): threshold =-1 #lb 0.3+ (submit all filled rle)\n if (num>=546) and (num<=546): threshold =0.3 #lb 0.56\n if (num>=547) and (num<=547): threshold =0.4 #lb 0.54\n if (num>=548) and (num<=548): threshold =0.5 #lb 0.52\n if (num>=549) and (num<1000): threshold =100 #lb 0.00 (submit all empty rle)\n\n\n#you can deduce value of num from lbscore\n\"roughly 550 test images \"
\nthere are exactly 529 test images, of which Hubmap=448, HPA=81
public test : 55% of the test data()
\n291 --> Hubmap=210, HPA=81
private test = 45%
\n238--> Hubmap= 238
if you like my work and would like to learn how to apply 3d transformer for 3d ct scan classification, please follow
\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350859
i will modify many of the 2d vit used here (e.g. mix-trasnformer, PVTv2,coat) for 3d
", "votes": 7, "replies": [ { "id": 1952891, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-24T03:59:53.573000", "content": "besides the above, here is another competition i am taking part:
\nhttps://www.kaggle.com/competitions/novozymes-enzyme-stability-prediction/discussion/354631
", "votes": 0, "replies": [] } ] }, { "id": 1894487, "author_name": "Givan", "author_url": "", "post_date": "2022-08-11T14:29:17.767000", "content": "Thanks for your wonderful sharing as always 😃. I have just tried your upernet with swin-tiny and got 0.79 cv, but the lb score is only 0.61. Do I miss anything important? @hengck23
", "votes": 7, "replies": [] }, { "id": 1869702, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-25T01:42:12.997000", "content": "i put code for upernet + swin transformer v2 for variable image size input at:
\nhttps://www.kaggle.com/code/hengck23/lb-0-75-variable-size-swin-transformer-v1-and-v2
this is beta release. using provided upernet+swin v1 (or v2) will get you about LB 0.75 (15 min submission time).
\nplan:
it would be faster for me if anyone can upload and share their downloaded images from HPA, etc
", "votes": 7, "replies": [ { "id": 1870350, "author_name": "Mail-Suesarn", "author_url": "", "post_date": "2022-07-25T13:44:39.003000", "content": "Do you train with whole images or tiles?
", "votes": 2, "replies": [] }, { "id": 1877490, "author_name": "DiamondH", "author_url": "", "post_date": "2022-07-30T17:32:41.863000", "content": "👋, @hengck23 ,
\nno need to use external data. just check the code I posted
", "votes": 0, "replies": [] }, { "id": 1877513, "author_name": "DiamondH", "author_url": "", "post_date": "2022-07-30T18:11:22.927000", "content": "I've tried tiled images with ConvNext, getting a LB score of 0.71 by 5 fold ensemble🙉, but oof is slightly better than your experiments, I'll try whole images later!
", "votes": 1, "replies": [] }, { "id": 1879826, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-01T09:38:21.637000", "content": "i managed to put everything together, SL, SSL learning, online, offline learning.
\nthis will be next software release.
the consistency feature loss in the middle ensure SSL aligned with SL learning
\n
\" but oof is slightly better than your experiments, I'll try whole images later!\"
\nyour oof must be the same as your lb submission (i.e. you compute only one score for the whole image after the tiles are combined)
seems that i select the wrong submission
\n
interesting:
\noh…
\nYou are 1th to my team..
\nThank you very much for this competition.
Thanks you, @hengck23 , I learned a lot from you thread，i'll never got the order without your generous share .By the way, I also chose wrong submision.I get 0.83 last two days so chose the new😂.
", "votes": 1, "replies": [] }, { "id": 1952653, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2022-09-23T19:55:16.073000", "content": "Thanks for all the sharing in the competition @hengck23 and such a strong finish.
\nI correctly dreamt of my ranking and medal (just one off the gold medal) just the day before
\nNot going to lie, this has happened to me in few of the previous competitions where I also missed solo gold by marginal ranks 🥲 It's really disheartening haha.
", "votes": 0, "replies": [] } ] }, { "id": 1833464, "author_name": "hengck23", "author_url": "", "post_date": "2022-06-26T03:27:59.327000", "content": "my first baseline at public LB 0.56
\none fold only:
\ntrain_dataset :
\nvalid_dataset :
\n
hello @hengck23,
\ndoes your model converge after 18 epochs?
\"does your model converge after 18 epochs?\"
\nyes for the above experiment. (I use little augmentation and the number of output channels in decoder is large)
update:
\nmodel = resnext50 + unet (use ASPP for center block, adjust approriate dilation for different input image size)
\ntrain image size = 512x512
\ntrain augment = random rotate, random scale, flip
\nhyperparameter = 80 epoch, batch_size 16, lookahead+RAMDA opt
\nlocal cv: 0.707 (at threshold 0.50)
train image size = 640x640
\nlocal cv: 0.715 (fold-0 at threshold 0.50) / 0.753 (fold-1)
why do you rescale it by \"512/3000\" if your model receives images of 480x480??
", "votes": 1, "replies": [] }, { "id": 1858816, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-17T08:21:56.507000", "content": "By experiment results. Sometimes, but not all times, your can get better results when your inference size is larger than the train size
", "votes": 0, "replies": [] }, { "id": 1858817, "author_name": "Ali", "author_url": "", "post_date": "2022-07-17T08:25:25.277000", "content": "I confirm that, thats the first thing I tried, learned this from starfish competition 🙂
\nBut you have to change the std and mean values also for some inference pipelines
I love your threads :-) especially the \"my experiment results\" one , It's like a story of how research and experiments should be.
\nGreat work :-)
100% agree with you @asalhi , @hengck23 threads are very informative and inspirational! Thanks for sharing all this with the wider Kaggle community.
", "votes": 1, "replies": [] } ] }, { "id": 1925481, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-04T02:19:26.513000", "content": "this is a nice paper to read. i think some of the conclusion also applies to this kaggle competition according to my results:
\nhttps://www.arxiv-vanity.com/papers/2201.08683/
\nA Comprehensive Study of Vision Transformers on Dense Prediction Tasks
We studied different aspects of VTs and CNNs as feature extractors for object detection and semantic segmentation on challenging and real-world data. The main results and key insights derived from our experiments are as follows:
\nVTs outperform CNNs in in-distribution dataset while having lower inference speed, but less computational complexity. Hence, if the GPUs are optimized for Transformer architectures, they have the potential to become dominant in computer vision.
VTs generalize better to OOD datasets. Our loss landscape analysis shows that VTs converge to flatter minima compared to CNNs, which can explain their generalizability.
VTs are more robust to natural corruptions and adversarial attacks compared to CNNs. We believe that this could be attributed to the global receptive field as well as the dynamic nature of self-attention.
VTs are less-texture biased than CNNs, which can be attributed to their global receptive field, allowing them to focus better on global shape-based cues as opposed to local texture-based cues.
are you stuck at 0.80?
\nare you stuck at 0.81?
\nare you stuck at 0.82?
\n… adjust accordingly
\nnote:
\nlb-HPA is always in 0.21 to 0.22
\nlb-Hubmap stops at 0.60 (even at lb 0.83)
Out of interest, what do you think are the HuBMAP-only equivalents are for these values?
", "votes": 0, "replies": [] }, { "id": 1921658, "author_name": "Rock", "author_url": "", "post_date": "2022-09-01T01:21:44.750000", "content": "Same with you, my lb-HuBMAP reached 0.60 when my lb is 0.82, and when I get lb 0.83, lb-HuBMAP is still 0.60😂
", "votes": 0, "replies": [] }, { "id": 1921674, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-01T01:35:26.510000", "content": "technically, it is:
\nlb = 0.830 to 0.839
\nlb-HuBMAP= (0.600 to 0.609)/0.7217 = 0.8314 to 0.8438
\nlb-HPA= (0.221 to 0.239)/0.2783 = 0.7941 to 0.8588
local cv (HPA) ~0.82
", "votes": 0, "replies": [] }, { "id": 1925783, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-04T09:48:10.267000", "content": "Does that mean when I reach 0.6 on HuBMAP, can I finally rest and pray?
", "votes": 2, "replies": [] }, { "id": 1925860, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-04T11:48:56.073000", "content": "no. you have to check your ranking. it will be 0.82 everywhere after a week
", "votes": 1, "replies": [] }, { "id": 1926617, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-05T02:02:44.963000", "content": "an example of trying different seeds
\nLB = 0.78,0.80,0.79,0.75,0.80
\nCV = 0.803159,0.764383,0.777201,0.818149,0.791553
different fold have different results (due to label noise)
\ngood public = worse private?
\nhow to prevent shake up?
Have you test the hubmap score of different folds? Kind of curious😋
", "votes": 0, "replies": [] }, { "id": 1927375, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-09-05T15:34:36.630000", "content": "Got stuck at 0.80 for quite a long time…
", "votes": 1, "replies": [] }, { "id": 1928257, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-06T11:18:09.643000", "content": "\"Got stuck at 0.80 for quite a long time…\"
\nchange/select fold, ensemble resolution, change threshold
", "votes": 0, "replies": [] }, { "id": 1928603, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-09-06T14:32:25.313000", "content": "Hi Heng, I've not tried ensembling models, but tried to change the seeds & try new augments. It looks like my CV fluctuates… Is there a good way to reduce the impact of labeling noise? I'm not an expert in the medical domain, so I'm not sure all the FTUs in the HPA images are fully labeled. Maybe pseudo-labeling might help?
", "votes": 0, "replies": [] }, { "id": 1930129, "author_name": "kingjohnson", "author_url": "", "post_date": "2022-09-07T15:23:40.723000", "content": "If we are stuck at 0.78, how can we do?
", "votes": 0, "replies": [] }, { "id": 1930374, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-07T19:40:24.757000", "content": "\"If we are stuck at 0.78, how can we do?\"
\ni assume you are talking about single model solution at 768x768 input resolution.
\nthen you are still having some issues with augmentation, TTA, training (learning rate/overfitting), ensemble single fold (SWA) or multiple fold for single model, threshold, … etc
\nyou have two choices:
\n1) you can still finetune you single model solution. but you will be cap to about 0.79 at 768x768
2) increase resolution, proceed to train multiple architecture for ensemble. but you will be cap 0.78 + 0.02
\ni would do both and (1) and (2). more exploration would means more training results and more information for algorithm debug
", "votes": 1, "replies": [] }, { "id": 1930522, "author_name": "kingjohnson", "author_url": "", "post_date": "2022-09-08T02:04:06.060000", "content": "Thanks a lot for your share, I will keep moving based on your tips!
", "votes": 0, "replies": [] }, { "id": 1930529, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-08T02:15:12.853000", "content": "poem : The Road Not Taken by Robert Frost
\nTwo roads diverged in a wood, and I—
\nI took the one less traveled by,
\nAnd that has made all the difference.
Thank you very much for sharing your all experiments and tips
", "votes": 1, "replies": [] } ] }, { "id": 1907670, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-21T00:54:45.903000", "content": "this is what every kaggler should know
\n\"This a question that comes up all the time in machine learning: How well can we do with the data we have? Even when we have a result, it can be hard to tell if it's a good result. What more could we do with better approaches? Or are we already close to the upper bound?\"
\nlink:
\nhttps://drivendata.co/blog/aleatoric-limit1
\nhttps://drivendata.co/blog/aleatoric-limit2
i thing kaggle can have a blog to invite kagglers to make post like this.
\ngood post will be given free TPU/GPU hours etc.
this is not the same as notebook as this focus on content rather than code
", "votes": 5, "replies": [] }, { "id": 1882428, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-03T09:15:46.607000", "content": "look what i have found
\n
@hengck23 Hi heng, Would you mind share more information about this image, is it from hubmap (https://portal.hubmapconsortium.org/)? thank you.
", "votes": 0, "replies": [] } ] }, { "id": 1874042, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-28T03:20:45.080000", "content": "one free trick for all:
\n
i haven't tried to add 2021 kaggle train data yet (kidney and large intestine). i also expect some improvement from there
\nwe now enter phrase 2 of the competition: from 0.80 to ~0.83
\nGreat 0.8 :-)
\nMy 0.75 was achived with basline notebook and some new training with larger bone with some size modification I achived 0.7.
\nAnd a trick ( or approch) that pushed the score from 0.70 to 0.75 , a boost that I belive you didn’t yet try and I guess with your current solution it might give you a boost of 0.02 or so. Or maybe not! If segformer is doing great with HuBMap.
That is very intuitive. It would probably work for all organs.
", "votes": 0, "replies": [] }, { "id": 1886216, "author_name": "kingjohnson", "author_url": "", "post_date": "2022-08-05T16:27:16.787000", "content": "I am curious about which way you choose for training, the whole image(resize) or image tiles.
", "votes": 0, "replies": [] } ] }, { "id": 1945022, "author_name": "James Howard", "author_url": "", "post_date": "2022-09-18T18:53:39.057000", "content": "@hengck23 - Your experience of coat-small (+- parallel) with 1x1 daformer outperforming the medium model with 3x3 - is this purely because you are able to train the small model at a higher resolution than medium, or do you think the smaller model is genuinely better? I ask because I believe segformer b3 as an encoder outperforms b5 on this dataset even at higher resolutions.
", "votes": 3, "replies": [ { "id": 1946523, "author_name": "James Howard", "author_url": "", "post_date": "2022-09-19T21:10:50.643000", "content": "In my experiments, at the same resoluton, CoatMedium(4) + Daformer3x3 works better than CoatSmall(5) + Daformer1x1 - this seems to be different from others' experience.
", "votes": 1, "replies": [] } ] }, { "id": 1952700, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-23T20:42:39.293000", "content": "with both the private and public score revealed, i discover something interesting.
\ni made a bug by loading a mitb6 encoder with a trained mit5 model at submission inference. this means that only the early layer are loaded and the unloaded weights are randomly initialized. However the performer is better then loading the correct model. at first i thought this is better \"by chance\". but now both private and public scores actually improved, meaning that improvement are not \"by chance\".
\nfor those that are doing research and writing paper, you may want to investigate on this. it seems possible just to \"perturb from train optimum (esp on just upper layers)\" and get better validation results.
\ni also note that transformer is trained with drop path and \"residual add\" in FF and attention block enforces learning to get residual=0
", "votes": 4, "replies": [ { "id": 1956303, "author_name": "Jairav Desai", "author_url": "", "post_date": "2022-09-26T11:39:19.983000", "content": "@hengck23 On Sept23 at ~1AM UTC when the \"Private leaderboard was disclosed\" vs at ~6PM UTC when the \"leaderboard was finalized\" I noticed a change in the Private LB ranks. My team's Private LB rank improved by 30 places. Why so? Were some of the teams disqualified during this time? If yes why were they disqualified ?
", "votes": 0, "replies": [] }, { "id": 1956334, "author_name": "Cheul", "author_url": "", "post_date": "2022-09-26T11:55:58.127000", "content": "@desaijairav I'm not hengck23 but lemme answer. 70 teams were disqualified for cheating: related thread. Probably, they used multiple accounts to submit more than five times a day, in order to either verify modifications on their programs, or to fine-tune the threshold for each organ.
\nPS you can check the very first rule of this competition here: One account per participant
", "votes": 1, "replies": [] }, { "id": 1956746, "author_name": "Jairav Desai", "author_url": "", "post_date": "2022-09-26T15:27:48.753000", "content": "Thanks a lot @cheulkay !! That was a exactly what I was looking for.
", "votes": 0, "replies": [] }, { "id": 1958040, "author_name": "Jairav Desai", "author_url": "", "post_date": "2022-09-27T09:00:02.890000", "content": "Another quick question- Was Segformer allowed to be used? I could see Segformer in some of the shared notebooks which are medal winners. It had \"Apache 2.0\" license initially & later changed to \"Other\".
\nIn terms of licensing any insights/links around what is allowed/not allowed? I went through the Kaggle rules & License conditions but the Technical-Legal language often becomes difficult to understand.
", "votes": 0, "replies": [] }, { "id": 1958644, "author_name": "opusen", "author_url": "", "post_date": "2022-09-27T15:05:08.633000", "content": "I have inquired of the question to the competition host.
\nAfter their reply, I'll share their opinion with my solution:)
my 13-th place solution: public 0.83, private: 0.81~0.82
\nhttps://www.kaggle.com/code/hengck23/lb0-83-submit/edit/run/104911483
organ_threshold = { \n 'Hubmap': { \n 'kidney' : 0.28, \n 'prostate' : 0.28, \n 'largeintestine': 0.28, \n 'spleen' : 0.28, \n 'lung' : 0.05, \n }, \n 'HPA': { \n 'kidney' : 0.50, \n 'prostate' : 0.50, \n 'largeintestine': 0.50, \n 'spleen' : 0.50, \n 'lung' : 0.10, \n }, \n}\n\nmodel = [ \n dotdict( #[1]\n is_use=1,\n image_size=1024,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4_level5, 'decoder': daformer_conv3x3, },\n checkpoint=[\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-4-swa.pth',\n\n ],\n ),\n dotdict(#[2]\n is_use = 1,\n image_size = 1536,\n module = 'model_coat_daformer',\n param={'encoder': coat_parallel_small_level5, 'decoder':daformer_conv1x1},\n checkpoint = [ \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-4-swa.pth', \n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-0-swa.pth', \n ],\n ),\n\n\n dotdict( #[3]\n is_use=1,\n image_size=1536,\n module='model_effnet_smp_unet',\n param={'encoder': tf_efficientnet_b6, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-3-swa.pth', \n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-1-swa.pth',\n ],\n ),\n dotdict(#[4]\n is_use = 1,\n image_size = 768,\n module = 'model_dualvit_daformer',\n param={'encoder': dual_vit_b, 'decoder':daformer_conv3x3},\n checkpoint = [\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-4-swa.pth',\n ],\n ), \n\n\n dotdict(#[5]\n is_use=1,\n image_size=1280,\n module='model_convnext_smp_unet',\n param={'encoder': convnext_large_384_in22ft1k, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-1-swa.pth',\n\n #'../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-4-swa.pth',\n ],\n ), \n\n\n\n dotdict(\n is_use=1,\n image_size=768,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4, 'decoder': daformer_conv3x3}, \n checkpoint=[\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-0-swa.pth', #0.78\n '../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-1-swa.pth', # 0.80\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-2-swa.pth', #0.79 \n ],\n ),\nyou will get private score 0.81 to 0.82 depending on how you select the base models (e.g. remove comment #)
", "votes": 4, "replies": [] }, { "id": 1948830, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-21T10:34:35.917000", "content": "Thanks for everything @hengck23. Learned a lot from this topic. Hope to you see on Mayo Clinic competition…
", "votes": 4, "replies": [ { "id": 1949496, "author_name": "Katalip", "author_url": "", "post_date": "2022-09-21T17:36:28.413000", "content": "Many thanks, @hengck23! I also learned quite a lot from this thread.
", "votes": 0, "replies": [] } ] }, { "id": 1883676, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-04T02:27:55.620000", "content": "best backbone for segformer
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Gu_Multi-Scale_High-Resolution_Vision_Transformer_for_Semantic_Segmentation_CVPR_2022_paper.pdf
another paper improving segformer
\n
i realize i made a mistake. i have been computing dice validation dice at train size of 3000.
\nBut there are image with pixel size less than 0.4 um, i.e. these images are \"much larger\". hence the dice for these hidden test images (largeintestine) are worse than your local results.
hence size 768 seem to be sufficient for most organ, it may be too small for largeintestine. if you are using 768 for largeintestine, you need a better enlarging mask method (rather than just bilinear upscale)
\nit is for this reason, using larger image input like 1024, 1280, 1536 improves LB scores.
\nhere are the LB scores(single fold):
\nLB 0.81
\nLB 0.80
\nLB 0.79
\n1536 already max out my GPU.
\nto go to 2048, i either need to 4x overlap 1536 tiles or single image using gradient check.
maybe i need to rewrite the code using
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Mangalam_Reversible_Vision_Transformers_CVPR_2022_paper.pdf
\"Reversible Vision Transformers achieve a reduced memory
\nfootprint of up to 15.5× at identical model complexity, parameters and accuracy,\"
What puzzles me is that large image input(1024) gives me worse performance on LB.
", "votes": 6, "replies": [] }, { "id": 1907377, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-20T17:56:40.723000", "content": "try a few folds?
\nthe training of large images is not the same as small images.
\nfirst your transformer needs to be larger because you need larger receptive field.
this makes the transformer too powerful and you need more regularization.
\ni use more augmentation and more aux loss.
swinv2 paper (which also uses 1536) uses feature distillation as regularization.
\nif you do it correctly you will seed you local validation improves greatly for some organs,
\nnotably my local cv has 0.915 for largeintestine
Let's begin the GPU war
", "votes": 0, "replies": [] }, { "id": 1907617, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-21T00:00:00.593000", "content": "i wonder if anyone try CNN + large image (e.g. efficient or resnet at 1024, 1028, 1536, 2048)?
\nit is not a gpu war but a race towards larger input size.
\nusing larger gpu is only one of the many available methods.
large image for PVTv2-b4 (single fold)
\nLB 0.80
\n input = 1024
\n augmentation = more + downsize/upsize for prostate 6 um
\n PVTv2-b4 (5 level)
LB 0.79
\n input = 768
\n augmentation = more
\n PVTv2-b4 (4 level)
Hi Heng, may I ask what's \"PVTv2\" model? I've been got stuck at 0.8 for quite a while, and have no idea to improve further. Thanks!
", "votes": 0, "replies": [] }, { "id": 1911378, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-24T04:08:10.060000", "content": "paper: PVT v2: Improved Baselines with Pyramid Vision Transformer
\nhttps://github.com/whai362/PVT
all CVT , PVTv2, MIT (segformer's mix transformer) share almost smiliar structure
", "votes": 0, "replies": [] }, { "id": 1911382, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-24T04:12:29.887000", "content": "i find a better (?) way to add level to pyramid transformer.
\ne.g. given pretrain weights for 4 level depths = [ 3,3,18,3], you can actually do change to 5 level depths = [3,3,15,5,3].
i.e. you can shorten the third level
", "votes": 0, "replies": [ { "id": 1914599, "author_name": "York G", "author_url": "", "post_date": "2022-08-26T08:01:52.433000", "content": "\n\ni find a better (?) way to add level to pyramid transformer.
\n
\n e.g. given pretrain weights for 4 level depths = [ 3,3,18,3], you can actually do change to 5 level depths = [3,3,15,5,3].i.e. you can shorten the third level
\n
Hi, @hengck23 ,
\nThank you for the models sharing, it really helps me on the learning of model structure and arrangement with backbones and decoders.
I have a few questions about the above-mentioned depths (maybe the questions is so stupid because of my ignorance):
\nAccording to the change of depths(from [3,3,18,3] to [3,3,15,5,3]), the relevant 'num_stages' will also change from 4 to 5, so the 'embed_dims / num_heads / mlp_ratios / sr_ratios' also need to be adjusted, is that right?
If I used the pre-tained weight (i.e. pvt2_b4), does pre-trained weight still match the new model level?
Thank you again for the selfless sharing.
\nBest Regards
", "votes": 0, "replies": [] } ] }, { "id": 1911496, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-08-24T06:14:37.277000", "content": "Hi @hengck23, thanks for your tips. Though having some experience in designing networks, I still struggle to have a god feeling about which direction to modify an existing network. Any rule of thumb according to your work/research experience?
\nThank you!
we already reached the max accurately , which is limited by the label noise.
\nsince performance is limited by data, there is no gain in designing better network.
however,increase of image size can reduce the predicted mask up scaling error, but there is little additional detection of missed FTU i think.
\nthe next boost will be better data or better network to learned in self-supervsied:
\nWait, the provided HPA images are not fully labeled? Or you mean external data on the internet?
\n\n\nthe next boost will be better data
\n
\n I noticed you mentioned the idea of \"mixup\" or more augmentations in the thread, this also resorts to \"creating better data\" right?
As for self-supervised learning, I'm not an expert in this domain, need to do some homework to understand how it works.
\nThank you, heng!
external data.
\naccording to the papers, all pretraining using external data for transformer would lead to 1 to 2 % improvement.
\nbut is still have to goes back to the issue, how bad are our labels, have we reached the best performance?
I did some large size experiments in cnn and transformers.
\n| net | \nimage size | \ncv | \nlb | \n
|---|---|---|---|
| efficientnet_b5 + unet | \n1536 | \n0.7966 | \n0.80 | \n
| coat_parallel_small(5 level) + daformer_conv1x1 | \n1536 | \n0.8034 | \n0.77 | \n
Both of lb are 5 folds ensemble results.
\nI don't know why my transformer model has such a big gap.
i think it is over fitting or bug.
\nyou should have aux loss for all parallel block output expect the first one. use more augmentation?
cv dice is not a good indicator.
\nhow many iterations/epoch did you train?
check your cv bce loss.
\nat least 1536 cv bce should be better than 1024, which is better than 768.
better bce indicates \"better margin\"
\ncheck the train/valid bce of your unet and coat_parallel_small
\nyou can only add a few new uninitialized serial block layers. we still need pretrain weights for most of the serial block layers/parallel block layers. else i think it will overfit.
\n0.77 seems like a bug. how about try 768 first?
\nor try without swa first.
my parallel block. use is skip first = True
\nhttps://gist.github.com/hengck23/b2c2bcd8de06248add46097ec55ce578#file-my-parallel-blovk
", "votes": 0, "replies": [] }, { "id": 1911673, "author_name": "Roc", "author_url": "", "post_date": "2022-08-24T08:28:55.990000", "content": "@hengck23 Thanks for your advice. I will check them.
\nI did more aux and more augmentation and training with 400 epoch.
More aux:
\n`
if self.CFG.aux:\n aux_logits = res['aux_logits']\n for aux_logit in aux_logits:\n loss += 0.2*self.criterion_aux_loss(aux_logit, masks)\n`
\nBasic aug: flip , rotate, transpose, contrast,hsv, noise, etc.
\nMore aug:
\n ElasticTransform,\n GridDistortion,\n RandomGamma,\n Cutout, etc\n
I will try more heavy augmentation.
", "votes": 3, "replies": [] }, { "id": 1911677, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-24T08:32:22.637000", "content": "what is your batch size?
\n\"I did more aux and more augmentation and training with 400 epoch.\"
\n\"I will try more heavy augmentation.\"
i suggest you train and submit a coat small parallel at 768. use this as reference to judge and guide your 1536 training. (the loss curve should be better than 768). you must get 0.78-79 for 768. 1536 will give you +0.02
\nnote
\nswap color channel and gray definitely hurts LB score for my experiemnts
total batch size = 8.
\nI will try it following your opinion.
\ni suggest you train and submit a coat small parallel at 768. use this as reference to judge and guide your 1536 training. (the loss curve should be better than 768). you must get 0.78-79 for 768. 1536 will give you +0.02\n
Really thanks for your kind answer and always learn a lot from your disscusion.
", "votes": 0, "replies": [] }, { "id": 1912170, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2022-08-24T14:55:42.830000", "content": "\n\n", "votes": 0, "replies": [] }, { "id": 1913332, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-25T09:28:53.363000", "content": "\nexternal data.
\n
\n according to the papers, all pretraining using external data for transformer would lead to 1 to 2 % improvement.
\n Understood.but is still have to goes back to the issue, how bad are our labels, have we reached the best performance?
\n
\n You're right. Tha't a good question.
my results for tf_efficientnet_b6 smp unet 1536 single fold (fold3)
\nLB =0.80
(i think ensemble will be 0.80~0.81)
\nmaybe b6 is better than b5 or maybe my augmentation is better
Great job!
\n\nmaybe b6 is better than b5 or maybe my augmentation is better\n
Maybe augmentation more important, I try tf_efficientnet_b7 smp unet 1536 improve cv but drop lb seriously.
\nI guess the lb limit is 0.81 with single model( single fold or ensemble n folds).
", "votes": 1, "replies": [] }, { "id": 1914324, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-26T02:12:35.947000", "content": "\"Maybe augmentation more important, \"
\ntoo much and too little augmentation is no good.it is easy to obtain CV 0.80.
\ni once made a mistake. i load rubbish ground truth mask for external data and train together with kaggle train data.
\ni can still obtain CV 0.80 for kaggle validation in that case (but of course LB score is not good)
it is important to visualize prediction on external data.
", "votes": 1, "replies": [] }, { "id": 1914859, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-26T13:37:41.830000", "content": "\n\n\n\n'embed_dims / num_heads / mlp_ratios / sr_ratios' also need to be adjusted, is that right?
\n
\n for the last level i just use previous level values.
\n the last level is not very deep, so i don't think it matters.
from another paper:
\n
\n\n\n\nIf I used the pre-tained weight (i.e. pvt2_b4), does pre-trained weight still match the new model level?
\n
\n you can only load pretrain model up to old levels. the new level are trained from scratch. hence the new level cannot be very deep because you don't have pretrain weight.
what i am doing is like extending resnet backbone from 4 layers to 5,6 layers (p6,p7) to detect large object in FPN
", "votes": 0, "replies": [] }, { "id": 1914861, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-26T13:40:40.460000", "content": "since the new layer is small, one can simply use conv instead of trasnformer
", "votes": 0, "replies": [] }, { "id": 1915442, "author_name": "York G", "author_url": "", "post_date": "2022-08-27T01:43:00.267000", "content": "Thank you for the fast reply. The new layer advice helps me a lot.
", "votes": 0, "replies": [] }, { "id": 1915512, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-27T03:07:33.960000", "content": "you train a normal network first.
\nthen freeze (or partially freeze) the trained network and train only the new layers, etc.
\n\n\n\ntraining with 400 epoch.
\n
it seems to me that 400 is too many.
\ni use about 200 to 230 for batch size 4
@hengck23
\nThanks for your tips.
\n I also observed this phenomenon, after epochs greater than 300, the avg_loss is increasing.
The thing about larger image with efficient net or resnet is… I think it is suggested to use 224 image size with resnet18 and effnet b0 where 240 size with efficient net b1 and goes to 700 image size for efficient net b7. Though CNN would work with any size of image still we need to create much larger efficient net like b9 Or b10( if exist)
\nDon't know if @yingpengchen if efficient net with 0.80 is most probably overfit
\nCorrect me if I am thinking in wrong direction.
@bibhabasumohapatra it's possible, but currently we don't have such a trained model like b9 or b10.
\n\nDon't know if @yingpengchen if efficient net with 0.80 is most probably overfit\n
It's hard to say, but compared to the 768 lb+0.02 improvement for me.
", "votes": 1, "replies": [] }, { "id": 1926789, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-05T06:04:35.247000", "content": "Wait, you guys are training for 300-400 epochs? My models converge after 30 epochs.
", "votes": 4, "replies": [] }, { "id": 1939399, "author_name": "Bibhabasu Mohapatra", "author_url": "", "post_date": "2022-09-14T17:05:52.030000", "content": "@gunesevitan exactly . . . 30 to max 35 is fine almost 45 . . . my model converges. never thought of 300 epochs . .. 😯😯
", "votes": 0, "replies": [] }, { "id": 1944014, "author_name": "TheMinimalist", "author_url": "", "post_date": "2022-09-18T01:48:05.853000", "content": "Hi, I am a beginner. I would like to ask what exactly \"augmentation = more\" is.
\nThe gap between my cv and lb is huge (0.7 vs 0.43).
\nA kind person told me it was because I didn't use data augmentation during training and testing.
\nI would like to ask if I can improve my score by applying more data augmentation like these.
\nOr do you have any better suggestions to improve this situation? thank you very much🙏🙏
as shown in the CV-LB graph at the top, all my CV-HPA, LB-HPA,LB-Hubmap,LB-public has converged!
\nI was right that the upper limit is about 0.80 without external data.
this could be the bronze medal limit after 2 months
\nI choose a variety of encoder, decoder, optimizer. These are not chosen at random but target at the root of problems
", "votes": 3, "replies": [ { "id": 1872506, "author_name": "datadote", "author_url": "", "post_date": "2022-07-27T03:32:05.373000", "content": "Hi hengck23,
\nHow did you reach the 0.8 upper limit? Why isn't it 1? I saw your probe post further down with the guess, but I couldn't tell how you reached that conclusion. Is it some out of distribution thing? Thanks
", "votes": 0, "replies": [] }, { "id": 1872513, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-27T03:43:19.347000", "content": "it cannot be 1 becuase of label noise.
\na quick guide is train dice >= local validation dice >=leaderboard dice
train dice is the upper limit at the point before overfitting. you can train you model for very long iterations to find the point of overfitting (i.e. when your validation dice becomes a V shape.)
", "votes": 2, "replies": [] }, { "id": 1874056, "author_name": "datadote", "author_url": "", "post_date": "2022-07-28T04:04:48.503000", "content": "Thank you for the quick, and helpful reply!
", "votes": 0, "replies": [] } ] }, { "id": 1867021, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-23T01:02:51.697000", "content": "what is the normalized size of hidden test images?
\nin order to apply transformer segmentation, i need to have a rough ideal of the max and min input size of the test image.
\n(i need to decide absolute/relative positional encoding, token patch size ….)
\nhence i do a probe.
the results: all test image are within 0.8 to 1.2 of normalized size.
\n test_df = pd.read_csv('../input/hubmap-organ-segmentation/test.csv') \n test_df.loc[:,'norm_size']= test_df.pixel_size/0.4 * test_df.img_height/3000 \n #assume img_height==img_width \n max_norm_size = test_df.norm_size.max() \n min_norm_size = test_df.norm_size.min() \nbelow two conditions are probed true:
\nif (min_norm_size>=0.8) and (min_norm_size<1.0): \nif (max_norm_size>=1.0) and (max_norm_size<1.2): \nnote we can also deuce that smallest 160x160 test images mentioned refer to prostate
\nimages with pixel size 6.2630 um. (you should do resize artifacts augmentation for this case in training)
largest 4500x4500 test image refer to large intestine with pixel size 0.229 um
", "votes": 3, "replies": [ { "id": 1868097, "author_name": "_CA℟L_", "author_url": "", "post_date": "2022-07-23T17:29:41.347000", "content": "\"assume img_height==img_width\"
\nThis passed:
\nif (df['img_width']==df['img_height']).all():
All images are square
", "votes": 1, "replies": [] } ] }, { "id": 1864027, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-20T17:14:41.743000", "content": "maybe this also works for our problem: labelled kaggle train images + unlabelled download external data
\nhttps://www.youtube.com/watch?v=FTmvdARsbnI
\nMedAI Session 25: Training medical image segmentation models with less labeled data | Sarah Hooper
this is mentioned in the video. i put it into picture.
\nstage1: train normally in supervised to get an initial model (and baseline results)
\nstage2: train joint supervised and unsupervised (and see if you get improved results)
there is a final stage 3 which uses pesudo labels
", "votes": 1, "replies": [] } ] }, { "id": 1881977, "author_name": "hengck23", "author_url": "", "post_date": "2022-08-03T01:42:43.543000", "content": "you may think that the results of Hunmap are inferior to that of HPA because of colors of the stains.
\nBut this may not be true.
You can verify this by changing the color of your validation HPA stains (e.g. HSV shift, convert to gray, stain color transfer) and compare with results of the original colored HPA images. You should see similar relative performance in your Lb-HPA and Lb-Humap score.
\nLikewise for other factors like pixel size in um, slice thickness, resolution, etc …
\nHubmap are real lab tissue slides. there are many more issues, see below.
\nthis is left as an exercise for you: \"how to prove that if these issues actually exist in the hidden test set?\"
even if these issues does not exist in the hidden test set, it will still boost your chance in the \"Judges prizes\" if analysis and results are presented to show your robustness of your model against real artifacts.
\non a side note, when i develop commercial computer vision model for by clients, \"black box\" testing is common. We do not see the test set at all. What we know is the feedback of the results (e.g. score against other bidding vendors, competitor products). So we have devised a way to fault find the issues even if we do not see the images. e.g.
\nexternal data may not help you to detect more FTU, but they are definitely of great help to remove false positive and show your model weakness and issues
\nthe score of LB-hubmap-public and LB-hubmap-private may be closer then what your think (i.e. little shakeup). it is likely that a sample tissues has many cropped slide samples. They are then split between public and private set.
", "votes": 4, "replies": [] }, { "id": 1875484, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-29T04:07:45.280000", "content": "beware!!!!
\nthe distribution of the organ in Hubmap is not what you think
\n
this confirms with my observation that the recent improvement in my LB score especially from 0.76 to 8.80 actually comes from prostate, lung and spleen.
\nit is also noted that segformer is strong in lung.
\nfurther, the FTU object size for spleen is larger then the rest. one have to take cares of scale. if you are using tiling, you need to check if your tile can contain the largest FTU object or not
", "votes": 3, "replies": [] } ] }, { "id": 1870646, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-25T17:30:06.703000", "content": "i realise one very important note:
\nupdate of results LB 0.72:
\n
surprise! surprise! the probe by organ is not what i would expect …
\ni wonder anyone has probed the distribution of the organ and willing to share?
\nthe domain shift is not as large as expected. this is good news.
\nnot reported in the table are training score : ~0.85
\ni estimate final leader board at the last day of competition:
\npublic LB : 0.80 (HPA=0.81, Hubmap=0.80)
\nprivate LB : >=0.79 (Hubmap>=0.79)
what is the key to improvement?
\nreceptive field size, scale?
\ndid you interpret results correctly?
the reason why we try different models is to test if their responses are similar to the same data.
\nthis reveal data characteristics … more to come : other segmentation architecture besides unet, etc
do compare the results with hubmap paper:
\nhttps://www.biorxiv.org/content/10.1101/2021.11.09.467810v1.full.pdf
Supplementary table 5. Algorithm performance. (without domain shift)
\nKaggle reproduced (kidney) :0.93~0.97
\nTransfer learning (trained on kidney & colon, tested on colon) : ~0.88
gives you an idea of effect of domain shift (ours), annotation error, etc
", "votes": 0, "replies": [] }, { "id": 1861472, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-19T03:58:49.477000", "content": "5-fold unet-efficientb7 (lb=0.75)
\nresults on the given public test spleen image
ccf-research-kaggle-2021 : hubmap CL_HandE_1234_B004_bottomleft.tiff
\n(but i don't know what is the um per pixel size for the datset. can anyone advise?)
kaggle 2021 hubmap kidney image
\n
how do you ensemble 5 fold seg models? you get average? what is your method?
", "votes": 0, "replies": [] }, { "id": 1862544, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-19T19:29:04.253000", "content": "just average
", "votes": 1, "replies": [] }, { "id": 1863905, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-20T15:30:31.547000", "content": "added uper-net-convnext-large to the ensemble
\nthe very best model
\n1) upernet-convnext-large : LB=0.75
\n2) unet-efficientb7: LB=0.75
\n3) upernet-swin-tx-small: LB=0.75
Hi @hengck23
\nAs always, it's a joy to read your analysis and discussion. May I know for the prediction you showed for Spleen what was your threshold strategy? did you learn the threshold on all five fold? or hard-coded one?
", "votes": 0, "replies": [] }, { "id": 1866959, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-22T22:16:13.407000", "content": "learned my experiments , turns out to be about 0.5 for organ spleen.
\nensemble such that the curve is almost flat over the maximum and insensitive to threshold
What image size are you using?
", "votes": 0, "replies": [] }, { "id": 1870275, "author_name": "hengck23", "author_url": "", "post_date": "2022-07-25T12:42:31.797000", "content": "768
", "votes": 0, "replies": [] }, { "id": 1870537, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-07-25T16:11:26.737000", "content": "\n\nccf-research-kaggle-2021 : hubmap CL_HandE_1234_B004_bottomleft.tiff
\n
\n (but i don't know what is the um per pixel size for the datset. can anyone advise?)
Is that dataset an extension of HuBMAP Kidney Segmentation? Their raw dimensions are very close to upper limit of HuBMAP image dimensions in this competition. I think large intestine images could be very similar.
", "votes": 0, "replies": [] } ] }, { "id": 1951266, "author_name": "Cheul", "author_url": "", "post_date": "2022-09-23T00:38:31.290000", "content": "I've learned a lot things from your post and comments. I deeply thank you very much for sharing your experiments and results that are very insightful and helpful.
", "votes": 1, "replies": [] }, { "id": 1937218, "author_name": "Saharsh Barve", "author_url": "", "post_date": "2022-09-13T10:52:24.920000", "content": "Hello,
\nCould anyone help me understand what aux-loss means? Is it like a supportive loss function for organ type classification? Any notebook/snippet I can refer to?
Hi, it's usually used to stabilize the training and to add more regularization. You compute the loss between intermediate outputs (from encoder stages) and your downsampled (used in the notebooks by OP of this discussion, many thanks, @hengck23) masks. Of course, you can adjust and modify everything based on what kind of constraint you want to put or emphasize more
", "votes": 1, "replies": [] }, { "id": 1937338, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-13T12:30:43.887000", "content": "it is an old method introduced by google inception model in the early days ….
\n
you can also google for:
\ndeep supervised, auxiliary loss, segmentation, mmseg auxiliary loss
in the old days, because batchnom were not invented, there is problem of diminishing gradients. Hence you insert auxiliary loss to back-propagates gradients from the middle.
\nfor today when the model is too strong (like transformer), there is little gradients from the top because you have almost zero loss. in this case you can make the problem more difficult (i,e, larger loss, more gradient) by asking the model to make prediction using only up the intermediate layers
\nhttps://discuss.pytorch.org/t/how-to-implement-deeply-supervised-network/6203
\n
in either case, the results will be a smoother loss landscape and better generalization (less over fitting)
", "votes": 4, "replies": [] }, { "id": 1945817, "author_name": "Saharsh Barve", "author_url": "", "post_date": "2022-09-19T12:23:26.220000", "content": "Thanks for providing these resources. I was able to implement it. Wanted to ask one more question.
\nCan I change the aux loss in b/w training? For example, take from some layer for first few epochs and then shift taux loss o other layer?
\nIs this logically sound?
\n\ninteresting:\n- i correctly predicted the highest public LB score\n- i correctly predicted the shakeup\n- i correctly dreamt of my ranking and medal (just one off the gold medal) just the day before\n",
"1833464": "my first baseline at public LB 0.56\n\none fold only:\n\ntrain_dataset : \n- len = 280\n - kidney 81 (0.289) \n - prostate 72 (0.257) \n - largeintestine 47 (0.168) \n - spleen 46 (0.164) \n - lung 34 (0.121) \n\nvalid_dataset : \n- len = 71\n - kidney 18 (0.254) \n - prostate 21 (0.296) \n - largeintestine 11 (0.155) \n - spleen 7 (0.099) \n - lung 14 (0.197) \n\n",
"1831307": "I love your threads :-) especially the \"my experiment results\" one , It's like a story of how research and experiments should be. \nGreat work :-) \n",
"1925481": "this is a nice paper to read. i think some of the conclusion also applies to this kaggle competition according to my results:\nhttps://www.arxiv-vanity.com/papers/2201.08683/\nA Comprehensive Study of Vision Transformers on Dense Prediction Tasks\n\n\n\nWe studied different aspects of VTs and CNNs as feature extractors for object detection and semantic segmentation on challenging and real-world data. The main results and key insights derived from our experiments are as follows:\n\n- VTs outperform CNNs in in-distribution dataset while having lower inference speed, but less computational complexity. Hence, if the GPUs are optimized for Transformer architectures, they have the potential to become dominant in computer vision.\n\n- VTs generalize better to OOD datasets. Our loss landscape analysis shows that VTs converge to flatter minima compared to CNNs, which can explain their generalizability.\n\n\n- VTs are more robust to natural corruptions and adversarial attacks compared to CNNs. We believe that this could be attributed to the global receptive field as well as the dynamic nature of self-attention.\n\n- VTs are less-texture biased than CNNs, which can be attributed to their global receptive field, allowing them to focus better on global shape-based cues as opposed to local texture-based cues.\n",
"1921312": "are you stuck at 0.80?\n- build better model, augmentation, etc \n\nare you stuck at 0.81?\n- build better resolution \n- some fold are better than other (i.e. some training images are better). try different random seeds, weigh more on some fold, etc ... \nsubmit for certain folds ...\nWARNING!!!! this may overfit to lb public test and lb private test\n\n\nare you stuck at 0.82?\n- maybe the best fold \"changes with score\" \n- maybe the best threshold \"changes with score\" \n\n... adjust accordingly\n\n---\n\nnote:\nlb-HPA is always in 0.21 to 0.22\nlb-Hubmap stops at 0.60 (even at lb 0.83)",
"1907670": "this is what every kaggler should know\n\n\"This a question that comes up all the time in machine learning: How well can we do with the data we have? Even when we have a result, it can be hard to tell if it's a good result. What more could we do with better approaches? Or are we already close to the upper bound?\"\n\nlink: \nhttps://drivendata.co/blog/aleatoric-limit1 \nhttps://drivendata.co/blog/aleatoric-limit2\n\n\ni thing kaggle can have a blog to invite kagglers to make post like this.\ngood post will be given free TPU/GPU hours etc.\n\nthis is not the same as notebook as this focus on content rather than code",
"1882428": "look what i have found\n",
"1874042": "one free trick for all:\n\n\n\ni haven't tried to add 2021 kaggle train data yet (kidney and large intestine). i also expect some improvement from there\n\n---\n\nwe now enter phrase 2 of the competition: from 0.80 to ~0.83\n\n- it is now the battle of external data, SSL masked pertaining, unlabelled learning ... weak/self/semi-supervised, domain adaption, ...",
"1945022": "@hengck23 - Your experience of coat-small (+- parallel) with 1x1 daformer outperforming the medium model with 3x3 - is this purely because you are able to train the small model at a higher resolution than medium, or do you think the smaller model is genuinely better? I ask because I believe segformer b3 as an encoder outperforms b5 on this dataset even at higher resolutions.",
"1952700": "with both the private and public score revealed, i discover something interesting.\n\ni made a bug by loading a mitb6 encoder with a trained mit5 model at submission inference. this means that only the early layer are loaded and the unloaded weights are randomly initialized. However the performer is better then loading the correct model. at first i thought this is better \"by chance\". but now both private and public scores actually improved, meaning that improvement are not \"by chance\".\n\nfor those that are doing research and writing paper, you may want to investigate on this. it seems possible just to \"perturb from train optimum (esp on just upper layers)\" and get better validation results.\n\ni also note that transformer is trained with drop path and \"residual add\" in FF and attention block enforces learning to get residual=0",
"1951265": "my 13-th place solution: public 0.83, private: 0.81~0.82\nhttps://www.kaggle.com/code/hengck23/lb0-83-submit/edit/run/104911483\n```\norgan_threshold = { \n 'Hubmap': { \n 'kidney' : 0.28, \n 'prostate' : 0.28, \n 'largeintestine': 0.28, \n 'spleen' : 0.28, \n 'lung' : 0.05, \n }, \n 'HPA': { \n 'kidney' : 0.50, \n 'prostate' : 0.50, \n 'largeintestine': 0.50, \n 'spleen' : 0.50, \n 'lung' : 0.10, \n }, \n}\n\nmodel = [ \n dotdict( #[1]\n is_use=1,\n image_size=1024,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4_level5, 'decoder': daformer_conv3x3, },\n checkpoint=[\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/pvt_v2_b4_level5-daformer_conv3x3-aug5b-1024-fold-4-swa.pth',\n \n ],\n ),\n dotdict(#[2]\n is_use = 1,\n image_size = 1536,\n module = 'model_coat_daformer',\n param={'encoder': coat_parallel_small_level5, 'decoder':daformer_conv1x1},\n checkpoint = [ \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-4-swa.pth', \n #'../input/hubmap-submit-06-weight0/coat_parallel_small_level5-daformer_conv1x1-aug8b-1536-fold-0-swa.pth', \n ],\n ),\n \n \n dotdict( #[3]\n is_use=1,\n image_size=1536,\n module='model_effnet_smp_unet',\n param={'encoder': tf_efficientnet_b6, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-3-swa.pth', \n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/tf_efficientnet_b6-unet-aug8b-1536-fold-1-swa.pth',\n ],\n ),\n dotdict(#[4]\n is_use = 1,\n image_size = 768,\n module = 'model_dualvit_daformer',\n param={'encoder': dual_vit_b, 'decoder':daformer_conv3x3},\n checkpoint = [\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-2-swa.pth', \n '../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/dual_vit_b-daformer_conv3x3-aug8-768-fold-4-swa.pth',\n ],\n ), \n\n \n dotdict(#[5]\n is_use=1,\n image_size=1280,\n module='model_convnext_smp_unet',\n param={'encoder': convnext_large_384_in22ft1k, 'decoder': smp_unet}, \n checkpoint=[\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-3-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-2-swa.pth',\n '../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-1-swa.pth',\n \n #'../input/hubmap-submit-06-weight0/convnext_large_384_in22ft1k-smp_unet-aug8b-1280-fold-4-swa.pth',\n ],\n ), \n \n \n \n dotdict(\n is_use=1,\n image_size=768,\n module='model_pvt_v2_daformer',\n param={'encoder': pvt_v2_b4, 'decoder': daformer_conv3x3}, \n checkpoint=[\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-0-swa.pth', #0.78\n '../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-1-swa.pth', # 0.80\n #'../input/hubmap-submit-06-weight1/pvt_v2_b4-aug8b-768-different-seed-fold-2-swa.pth', #0.79 \n ],\n ),\n\n```\n\nyou will get private score 0.81 to 0.82 depending on how you select the base models (e.g. remove comment #)",
"1948830": "Thanks for everything @hengck23. Learned a lot from this topic. Hope to you see on Mayo Clinic competition...",
"1883676": "best backbone for segformer\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Gu_Multi-Scale_High-Resolution_Vision_Transformer_for_Semantic_Segmentation_CVPR_2022_paper.pdf\n\n",
"1880720": "another paper improving segformer\n",
"1907339": "i realize i made a mistake. i have been computing dice validation dice at train size of 3000.\nBut there are image with pixel size less than 0.4 um, i.e. these images are \"much larger\". hence the dice for these hidden test images (largeintestine) are worse than your local results.\n\nhence size 768 seem to be sufficient for most organ, it may be too small for largeintestine. if you are using 768 for largeintestine, you need a better enlarging mask method (rather than just bilinear upscale)\n\nit is for this reason, using larger image input like 1024, 1280, 1536 improves LB scores.\n\nhere are the LB scores(single fold):\n\nLB 0.81\n- input = 1536\n- augmentation = more + staintool + downsize/upsize for prostate 6 um\n- COAT-parallel small (5 level)\n\nLB 0.80\n- input = 1024\n- augmentation = more \n- COAT-parallel small (5 level)\n\nLB 0.79\n- input = 768\n- augmentation = more \n- COAT-parallel small (4 level)\n\n1536 already max out my GPU. \nto go to 2048, i either need to 4x overlap 1536 tiles or single image using gradient check.\n\nmaybe i need to rewrite the code using\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Mangalam_Reversible_Vision_Transformers_CVPR_2022_paper.pdf\n\n\"Reversible Vision Transformers achieve a reduced memory\nfootprint of up to 15.5× at identical model complexity, parameters and accuracy,\" ",
"1870978": "as shown in the CV-LB graph at the top, all my CV-HPA, LB-HPA,LB-Hubmap,LB-public has converged!\nI was right that the upper limit is about 0.80 without external data.\n\nthis could be the bronze medal limit after 2 months\n\n---\n\nI choose a variety of encoder, decoder, optimizer. These are not chosen at random but target at the root of problems",
"1867021": "what is the normalized size of hidden test images?\n\nin order to apply transformer segmentation, i need to have a rough ideal of the max and min input size of the test image.\n(i need to decide absolute/relative positional encoding, token patch size ....)\nhence i do a probe.\n\nthe results: all test image are within 0.8 to 1.2 of normalized size.\n\n```\n test_df = pd.read_csv('../input/hubmap-organ-segmentation/test.csv')\t\t\t\t\t\t\n test_df.loc[:,'norm_size']= test_df.pixel_size/0.4 * test_df.img_height/3000 \t\t\t\t\t\t\n \t\t\t\t#assume img_height==img_width\t\t\n max_norm_size = test_df.norm_size.max()\t\t\t\t\t\t\n min_norm_size = test_df.norm_size.min()\t\t\t\t\t\t\n\n```\n\nbelow two conditions are probed true:\n\n``` \nif (min_norm_size>=0.8) and (min_norm_size<1.0): \nif (max_norm_size>=1.0) and (max_norm_size<1.2): \n```\n\nnote we can also deuce that smallest 160x160 test images mentioned refer to prostate\nimages with pixel size 6.2630 um. (you should do resize artifacts augmentation for this case in training)\n\nlargest 4500x4500 test image refer to large intestine with pixel size 0.229 um",
"1864027": "maybe this also works for our problem: labelled kaggle train images + unlabelled download external data\n\nhttps://www.youtube.com/watch?v=FTmvdARsbnI\nMedAI Session 25: Training medical image segmentation models with less labeled data | Sarah Hooper",
"1881977": "you may think that the results of Hunmap are inferior to that of HPA because of colors of the stains.\nBut this may not be true.\n\nYou can verify this by changing the color of your validation HPA stains (e.g. HSV shift, convert to gray, stain color transfer) and compare with results of the original colored HPA images. You should see similar relative performance in your Lb-HPA and Lb-Humap score.\n\nLikewise for other factors like pixel size in um, slice thickness, resolution, etc ...\n\n---\n\nHubmap are real lab tissue slides. there are many more issues, see below.\nthis is left as an exercise for you: \"how to prove that if these issues actually exist in the hidden test set?\"\n\n\n\n\n---\n\neven if these issues does not exist in the hidden test set, it will still boost your chance in the \"Judges prizes\" if analysis and results are presented to show your robustness of your model against real artifacts.\n\n---\n\non a side note, when i develop commercial computer vision model for by clients, \"black box\" testing is common. We do not see the test set at all. What we know is the feedback of the results (e.g. score against other bidding vendors, competitor products). So we have devised a way to fault find the issues even if we do not see the images. e.g.\n\n- using auto-encoders, etc to measure out-of distributions from your train set\n- apply your model on large test set (including out of distributions). train a classifier to classify the type of errors. then you can map error to input image.\n\n---\n\nexternal data may not help you to detect more FTU, but they are definitely of great help to remove false positive and show your model weakness and issues\n\n---\n\nthe score of LB-hubmap-public and LB-hubmap-private may be closer then what your think (i.e. little shakeup). it is likely that a sample tissues has many cropped slide samples. They are then split between public and private set.",
"1875484": "beware!!!!\n\nthe distribution of the organ in Hubmap is not what you think\n\n\n ",
"1870646": "i realise one very important note:\n- you should make post submission to last year competition using your current model to see your domain shift ",
"1858202": "update of results LB 0.72:\n",
"1951266": "I've learned a lot things from your post and comments. I deeply thank you very much for sharing your experiments and results that are very insightful and helpful.",
"1937218": "Hello,\nCould anyone help me understand what aux-loss means? Is it like a supportive loss function for organ type classification? Any notebook/snippet I can refer to?\n",
"1913583": "if you are using my code, beware!!!!\n\n\nthe simple fusion of the different levels uses upscale of 4x, 8x, 16x, 32x\nit is obvious that it causes the prediction to be blocky.\n(despite the blocky head, LB is still 0.82 after ensemble)\n\ni need a better decoder head ",
"1894634": "one possible replacement for segformer\n (they are a few more to come later ...)\n\n \nfor model, refer to https://www.kaggle.com/code/hengck23/lb-0-78-coat-and-simple-3x3-conv-fusion\nweights/training are not shared",
"1892293": "oh, interactive segformer!\nSegFormer: Interactive Segmentation via Transformerswith Application to 3D Knee MR Images\npaper: https://arxiv.org/pdf/2112.11325v6.pdf\n\nfor the use in this competition, refer to\nhttps://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/341130#1892285\n\n",
"1884297": " \nhttps://www.cs.toronto.edu/~garyleung/hila/\n\n We add HILA into SegFormer and the Swin Transformer and show notable improvements in accuracy in semantic segmentation with fewer parameters and FLOPS.",
"1881609": "i am not sure if this would be an issue:\n",
"1870257": "you probably need this here (since we are dealing with shifted domain private LB data) :\n\npaper: WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS\nhttps://arxiv.org/pdf/2106.01548.pdf\n\n\"The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations.\"\n\ndo read the section on test on different domain: \"Better robustness. We also evaluate the models’ robustness using ImageNet-R (Hendrycks et al., 2020) and ImageNet-C (Hendrycks & Dietterich, 2019) and find even bigger impacts of the smoothed loss landscapes.\"\n\n---\n\ntake away message: it is not about early stopping .... it is about flatten your loss landscape",
"1869703": "Adversial network perturbation works well for NLP transformer. It improve robustness (e.g. spelling errors) and score. There are many kaggle code for the recent kaggle competitions for the feedback, US-patent for it.\n\nI wonder if it also work for histology tissue images?\n\ne.g. [paper] An Adversarial Perturbation Oriented Domain Adaptation Approach for Semantic Segmentation",
"1865226": "you can self-supervised CNN (like transformer) too!\n\npaper: Corrupted Image Modeling for Self-Supervised Visual Pre-Training\nhttps://arxiv.org/pdf/2202.03382.pdf\n\n\"ResNet-50. We demonstrate that CIM can also pre-train a high-capacity ResNet-50 model with the fewest possible\nmodifications from the ViT pre-training settings that can achieve compelling fine-tuning performances on ImageNet1K.\"",
"1859177": "an interesting cvpr 2022 oral paper:\nSource-Free Object Detection by Learning to Overlook Domain Style\n\nSource-free object detection (SFOD) needs to adapt a detector pre-trained on a labeled source domain to a target\ndomain, with only unlabeled training data from the target domain. ...\n\n",
"1858588": "learning threshold\n\nwhat is show: for 3 fold on validation HPA dataset\n(but what we need is similar graph on Hubmap dataset .... many probes to be made !!!)\n",
"1831210": "\n\nexpected improvement using pretraining transformer\nfrom: [1] BEiT: BERT Pre-Training of Image Transformers",
"1914582": "Hello folks, I am using transformer-based approach for this task using resized data (768,768). I was wondering how should I find what is the optimal Threshold for my model for each of the organs?\n@hengck23 \nadvice would be really helpful.\nThanks! :)",
"1904239": "in my latest LB score of 0.81 l had LB-hubmap of 0.59.\n- no external data was used\n- improvement comes from ensemble with large image model (1024x1024 as input)",
"1896619": "yet another 0.79 model.\n\n\n",
"1893249": "\n\ni find a way to annotate spleen images easily.\nuse gimp extract color components (CMKY yellow)",
"1893007": "",
"1891347": "\ni download random images from the internet to test the robustness of the model.\nit perform better than I thought.\n\nit is obvious that hand-correction of the prediction on unseen images is a way to go.\nneed to find more images without copyright issues",
"1878169": "i just realised besides swin v1,v2, there is a \"version 3\"\nhttps://github.com/microsoft/CSWin-Transformer",
"1877641": "good reading\n\nHOW DO VISION TRANSFORMERS WORK?\nhttps://arxiv.org/pdf/2202.06709.pdf\n\"MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary\"\n\n\nBlurs Behave Like Ensembles: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness\nhttps://arxiv.org/pdf/2105.12639.pdf",
"1877258": "updated list of masked pretraining method.\nmost opensource have complete framework code from pertaining transformer backbone and finetunning/inference it for segmentation.\n\n\nI missed the important Beit in the table:\nhttps://github.com/microsoft/unilm/tree/master/beit\n\n\nsee also\nhttps://github.com/cmhungsteve/Awesome-Transformer-Attention\nunder \"Training + Transformer\" for more MIM",
"1871615": "how should you select your solution?\n\n- which organ actually affects private LB score the most? (due to different/same class distribution probability and domain difference of private data)\n\n- we usually monitor the validation dice of all organ. but does this select the most important one to private test? or are we selecting the best model for local validation data only? (is all dice correlated to organ dice)\n\n",
"1861404": "make your own spleen labels:\nhttps://www.purposegames.com/no/game/spleen-histology-and-red-and-white-pulp",
"1914883": "this comment has not been deleted",
"1832019": "fyi: \nVirtual histological staining of unlabelled tissue-autofluorescence images via deep learning\nhttps://openreview.net/pdf?id=S1xANDXRKN\nhttps://arxiv.org/pdf/1803.11293.pdf\n\n\n---\nH&E to PAS conversion\nDeep learning-based transformation of H&E stained tissues into special stains\nhttps://github.com/kevindehaan/stain-transformation\n\nhttps://www.researchgate.net/publication/353875566_Deep_learning-based_transformation_of_HE_stained_tissues_into_special_stains\n\n\n---\n\nDAB(brown)-H(blue) to H(blue)&E(pink) papers:\n\n(1) Creating virtual H&E images using samples imaged on a commercial CODEX platform\n(2) FalseColor-Python: a rapid intensity-leveling and digital-staining package for fluorescence-based slide-free digital pathology\n\n(3) Dual Adaptive Pyramid Network for Cross-Stain Histopathology Image Segmentation\nIn this paper, we propose a dual adaptive pyramid network (DAPNet) for histopathological gland segmentation\nadapting from one stain domain to another (DAB-H and H&E)\nhttps://arxiv.org/pdf/1909.11524.pdf\n\n\n---\n\n\nTailoring automated data augmentation to H&E-stained histopathology\nhttps://github.com/DIAGNijmegen/pathology-he-auto-augment\nhttps://openreview.net/pdf?id=JrBfXaoxbA2\n\n---\n\nhttps://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_ihc_color_separation.html\nhttps://towardsdatascience.com/stain-estimation-on-microscopy-whole-slide-images-2b5a57062268",
"1938689": "I notice that the KEY POINT is 2xA6000 GPU cards workstation xD",
"1872994": "How did you manage to calculate the LB score on HPA and Hubmap?\n\nI found in the comments the probe you used to calculate how many of each type you had in the test set. But how to get the metrics on each type is out of my understanding 😅",
"1839900": "Hi @hengck23 - Wonderful thread like usual.\n\nIf you **do** find any useful external data sources, would you mind commenting on them on [**this thread**](https://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/333886)? I will add them to the main body of that post so that they are shared with everyone all in one place.",
"1861671": "a baseline code is needed! current public ones are not good.👀",
"1873994": "Great job! what pretrained model did you use, and what loss function did you use.",
"1951803": "Could you share a snippet of data parallelism code? I saw your posted codes and tried to implement it into mine but it didn't go well. \nI used the following code to utilize two gpus:\n```\nwith amp.autocast(enabled=is_amp):\n\toutput = data_parallel(net, batch) \n\tprobability += F.interpolate(output['probability'], size=(H, W), mode='bilinear', align_corners=False, antialias=True)\n```\nThe program utilized the only one gpu while training. Though I had added GPU 0,1 environment and checked its counter =2. So I'm curious if I'm missing something important to do data parallelism?",
"1951493": "Hey . . . @hengck23 Hengck . . . . THANKS VERY MUCH ✌️✌️✌️✌️ . . . . MAIDEN MEDAL IN Competition and YOUR COAT MEDIUM + DAFORMER was winner for me, .. . .. . 🔥🔥🔥🔥🔥 . . . . ",
"1949163": "I've just realised - you are probably going to become a competitions grand master in just over 24 hours :)",
"1940951": "Hello!\nI'm currently working with segformer mit b2. I use the same fold for validation and all the scripts as you. But I don't know why I can't get at least 0.75 on validation. Do you have any idea what causes the problem? Thanks a lot for your work - I've learned a lot!\n\n\ntrain_log:rate iter epoch | dice loss tp tn | loss | time \n\ntrain_log:-------------------------------------------------------------------------------------\n\ntrain_log:1.00e-5 00000000* 0.00 | 0.155 0.705 0.0000 0.000 | 0.000 0.000 | 0 hr 00 min\n\ntrain_log:1.00e-5 00000138* 3.00 | 0.592 0.251 0.0000 0.000 | 0.372 0.265 | 0 hr 01 min\n\ntrain_log:1.00e-5 00000276* 6.00 | 0.594 0.165 0.0000 0.000 | 0.203 0.173 | 0 hr 02 min\n\ntrain_log:1.00e-5 00000414* 9.00 | 0.608 0.134 0.0000 0.000 | 0.154 0.126 | 0 hr 04 min\n\ntrain_log:1.00e-5 00000552* 12.00 | 0.638 0.133 0.0000 0.000 | 0.130 0.115 | 0 hr 05 min\n\ntrain_log:1.00e-5 00000690* 15.00 | 0.644 0.131 0.0000 0.000 | 0.124 0.113 | 0 hr 06 min\n\n....\ntrain_log:1.00e-5 00003726* 81.00 | 0.715 0.102 0.0000 0.000 | 0.071 0.075 | 0 hr 37 min\n\ntrain_log:1.00e-5 00003864* 84.00 | 0.702 0.101 0.0000 0.000 | 0.053 0.059 | 0 hr 39 min\n\ntrain_log:1.00e-5 00004002* 87.00 | 0.707 0.094 0.0000 0.000 | 0.054 0.059 | 0 hr 40 min\n\ntrain_log:1.00e-5 00004140* 90.00 | 0.700 0.106 0.0000 0.000 | 0.056 0.062 | 0 hr 42 min\n\ntrain_log:1.00e-5 00004278* 93.00 | 0.700 0.094 0.0000 0.000 | 0.056 0.061 | 0 hr 43 min\n\ntrain_log:1.00e-5 00004416* 96.00 | 0.706 0.099 0.0000 0.000 | 0.054 0.059 | 0 hr 44 min\n\ntrain_log:1.00e-5 00004554* 99.00 | 0.708 0.097 0.0000 0.000 | 0.055 0.061 | 0 hr 46 min\n\ntrain_log:1.00e-5 00004692* 102.00 | 0.699 0.099 0.0000 0.000 | 0.056 0.061 | 0 hr 47 min\n....\ntrain_log:1.00e-5 00010212* 222.00 | 0.703 0.119 0.0000 0.000 | 0.037 0.045 | 1 hr 48 min\n\ntrain_log:1.00e-5 00010350* 225.00 | 0.721 0.117 0.0000 0.000 | 0.035 0.043 | 1 hr 49 min\n\ntrain_log:1.00e-5 00010488* 228.00 | 0.710 0.108 0.0000 0.000 | 0.039 0.047 | 1 hr 51 min\n\ntrain_log:1.00e-5 00010626* 231.00 | 0.704 0.117 0.0000 0.000 | 0.039 0.046 | 1 hr 53 min\n\ntrain_log:1.00e-5 00010764* 234.00 | 0.704 0.113 0.0000 0.000 | 0.039 0.048 | 1 hr 54 min\n....\nAnd the dice goes around 0.71...",
"1937356": "an interesting work\n\n\" We additionally benchmark improvements in domain adaptation and out-of-distribution\ndetection, and demonstrate that semi-supervised learning outperforms supervised learning in both cases.\"\n\nImproving colonoscopy lesion classification using semi-supervised deep learning\npaper: https://arxiv.org/pdf/2009.03162.pdf\nhttps://durr.jhu.edu/pubs/2020-colossl/\n\n",
"1935317": "@hengck23, may I hear your opinion on my case? The problem is \"higher resolution trained model performed worse on Hubmap data than less res one, while better on HAP\" I have two CoaT_small models: one trained with 768, and the other with 1024. The 1024 model performed on HAP better than the 768 model (0.01~0.02 better) while its performance significantly worse on Hubmap data (0.03 worse). In order to have bigger receptive field for higher resolution, I have added a serial block (2 times repeat) and a parallel block (6 times repeat). However, it didn't go well for Hubmap data, although they have same lr, bs, and aug. I've assumed three possibilities. 1. 1024 dataset wrong, 2. need more aug, 3. change for loss function (currently for 768, bce + 0.2(aux1~3), for 1024, bce+0.2(aux1~4). Can you enlighten me if I've missed some important part in my experiments? ",
"1930925": "Thank you very much for giving away your experiments and knowledge in detail. Mind if I ask your opinion? Do you think it's safe to use \"data_source, organ, and img_height/width\" in inference? I've been using those information to infer the test images but, if any of those information is null in the other test set, then my program would not complete. So I'm afraid of it. Maybe, it's not going to be a problem because inference program look through both test sets without an issue when we make an infer? Sorry for the dumb question.",
"1930900": "when i train coat small(5 levels) at input 1024, i get results below ,kidney score is too low,can anyone give me some advice? i think it is the reason i only get 0.77 at lb, not 0.8(single fold3)\nall 0.776699\nkidney 0.882886\nprostate 0.823085\nlargeintestine 0.907251\nspleen 0.854549\nlung 0.265985\n☹️",
"1928953": "Should we use aux0 loss when training?why",
"1914865": "Out of interest, could you post your predicted on the single test image of your best model (thresholded, or not, or both!)? I'm interested to know what the current leading solution thinks the test image should be predicted as!",
"1913542": "i realize one important thing: almost all vision transformer starts with 4x4 non-overlapping block (or maybe 8x8)\nthis initial layer causes some similarity in all transformer, and reduces diversity in ensemble?",
"1911581": "the race never ends .... MaxVit seems promising ...\n\n\n\nRoss Wightman\n\"I've been working on a large model update over the past month+, incl several imports from research impl and timm originals of CoAtNet (https://arxiv.org/abs/2106.04803) and MaxVit (https://arxiv.org/abs/2204.01697) in a combined `MaxxVit` impl. Soon to be merged w/ a few weights.\"",
"1908381": "maybe useful\nMesa: A Memory-saving Training Framework for Transformers",
"1904622": "48G GPU Mem .. such expensive game. 💔",
"1904256": "i am in the progress of running results for [1]\n\n[1] Dual Vision Transformer\nhttps://arxiv.org/pdf/2207.04976.pdf\n https://github.com/YehLi/ImageNetModel\n\ni think it is very promising.\ninitial loss curve and validation loss already surpass that of PVT v2.\n~~\nnumerical evaluation will be coming up in the next few days. ... but you can start your experiment first~~\nthis is another 0.79 model with input=768",
"1903807": "I follow your work of segformer， but i did not get the same result ，the Hubmap score is 0.55 same as yours， but HPA score is only 0.21~",
"1903376": "Are we segmenting normal glomeruli, sclerosed glomeruli or any type of glomeruli?",
"1903332": "i now become confused by my results:\n\nking of king: single fold LB 0.80\n\n- COAT parallel small 5 level (I modified the code to change from 4 to 5 level)\n- increase image size from 768 to 1024\n\nhttps://www.kaggle.com/code/hengck23/lb-0-80-coat-small-parallel-at-1024",
"1903092": "i came to realise that Beit is the chnapion solution for Visual Domain Adaptation Challenge 2021:\n\n\nhttps://ai.bu.edu/visda-2021/assets/pdf/Burhan_Report.pdf\nhttps://ai.bu.edu/visda-2021/assets/pdf/Burhan_Slides.pdf",
"1897281": "find a paper :\n\nCROSS-STAINED SEGMENTATION FROM RENAL BIOPSY IMAGES USING MULTI-LEVELADVERSARIAL LEARNING\nhttps://arxiv.org/pdf/2002.08587.pdf\n\n\" Experimental results on glomeruli segmentation from renal biopsy images indicate that our network is able to improve segmentation performance on target type of stained images and use unlabeled data to achieve similar accuracy to labeled data\"\n",
"1896239": "which is more important, backbone or decoder?\n ",
"1894597": "hi, \nFirst, thanks for your sharing. I am new to the kaggle competetion. I studied your shared code. I have a dumb question. What methods can I use to filter out a few relatively well-performing model weights in a fold? like the code in your run_local_cv.py.\n",
"1887778": "i haven't got good results for transformer decoder. another want to share their results?",
"1887743": "maybe a good way to use stain normalisation\n",
"1887169": "i made some calculation ....\n\neveryone will be stuck in ~0.80. then one more decimal in LB score will be reveal.\n\nonce it is in 3 decimal, score per image will be revealed. magic may happen as we may be able to \"get/probe more information\" for e.g. the test spleen image.",
"1883056": "Hello~ I am quite curious about how can you get your pre-trained mit weights? Did you use mask image modeling? I noted that official implementation of segformer-mit-b2 is in mmseg, however your code is different from the official ones.",
"1879784": "Hello, I'm curious about the way categories are handled. Since in the end we only need to submit the rle encoding for each image, we don't need to provide categories. What way your model is trained, multi-classification or binary(organ/not organ)？",
"1879713": "Hey @hengck23! Thank you for sharing all the info. I wanted to ask whether you are training with full size images or tiled ones? Also, how are you dealing with different pixel dimensions of HPA and HubMAP?",
"1877715": "did anyone suceed in training from scratch without image-net pretrain?",
"1877635": "Awesome discussion till now to the competition ",
"1875228": "masked auto encoder for SSL\n\n\n\n",
"1869213": "masked VAE for pyramid-based Vit like Swin transformer:\nhttps://github.com/implus/UM-MAE",
"1864524": "update on trasnformer results:\n\nconsider the performance of imagenet, segformer mit-b2 results is ok. but it doesn't scale up when I use more powerful mit-b4\n\n",
"1863895": "anyone can explin why am i getting patch token artifacts when using transformer segmentation (using segformer)?\nit turns up this is due to F.interpolate using mode='nearest' when i upsample the mit transformer layer results.\n\n\n\n",
"1945984": "",
"1928904": "",
"1883718": "",
"1879793": "",
"1889296": "Thank you for sharing!",
"1886151": "Thank you so much for sharing!",
"1886002": "thanks for the author, very helpful"
}
}