I have a bunch of pipelines on small patches (512, 768) that give decent baseline solutions (0.75+). But all of them are about tiling original image into small pieces.
\nWhen I try to apply exact same solutions into larger image tiles or whole image rescale (1024, 1536 etc), a lot of problems arises.
\nImages hardly fits on GPU memory, and the maximum batch_size is 1-2 in such cases. But what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2 (and after that can even decrease back to 0.02).
\nThe same situation happens for UNet-based and transformer-based solutions.
Of course, something is wrong here.
\nI've tried playing with (virtual) batch_size and learning rate, but without any success. I did not notice any improvement from here, it's still not converging, and there's no intuition about what values these parameters should have.
\nCan you suggest what might be going wrong?
\nIf the model is overfitting, then for some reason the model (i.e. UNet) cannot remember even the examples from the training set. Should I add more augmentations or soften them?
\nMaybe there are some good articles/recommendations on training models on very large images?
", "messages": [ { "id": "1918148", "postDate": "08/29/2022 11:04:43", "content": "I have a bunch of pipelines on small patches (512, 768) that give decent baseline solutions (0.75+). But all of them are about tiling original image into small pieces.
\nWhen I try to apply exact same solutions into larger image tiles or whole image rescale (1024, 1536 etc), a lot of problems arises.
\nImages hardly fits on GPU memory, and the maximum batch_size is 1-2 in such cases. But what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2 (and after that can even decrease back to 0.02).
\nThe same situation happens for UNet-based and transformer-based solutions.
Of course, something is wrong here.
\nI've tried playing with (virtual) batch_size and learning rate, but without any success. I did not notice any improvement from here, it's still not converging, and there's no intuition about what values these parameters should have.
\nCan you suggest what might be going wrong?
\nIf the model is overfitting, then for some reason the model (i.e. UNet) cannot remember even the examples from the training set. Should I add more augmentations or soften them?
\nMaybe there are some good articles/recommendations on training models on very large images?
", "rawMarkdown": "I have a bunch of pipelines on small patches (512, 768) that give decent baseline solutions (0.75+). But all of them are about tiling original image into small pieces.\n\nWhen I try to apply exact same solutions into larger image tiles or whole image rescale (1024, 1536 etc), a lot of problems arises.\n\nImages hardly fits on GPU memory, and the maximum batch_size is 1-2 in such cases. But what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2 (and after that can even decrease back to 0.02).\nThe same situation happens for UNet-based and transformer-based solutions.\n\nOf course, something is wrong here.\n\nI've tried playing with (virtual) batch_size and learning rate, but without any success. I did not notice any improvement from here, it's still not converging, and there's no intuition about what values these parameters should have.\n\nCan you suggest what might be going wrong? \n\nIf the model is overfitting, then for some reason the model (i.e. UNet) cannot remember even the examples from the training set. Should I add more augmentations or soften them?\n\nMaybe there are some good articles/recommendations on training models on very large images?", "votes": null }, { "id": "1918177", "postDate": "08/29/2022 11:19:49", "content": "Some other discussions about large images
\nhttps://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941#1907369
\n\n\n\nBut what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2
\n
each model have a \"receptive field\", it determines the the \"size of the region\" a model can see.
\nwhen you enlarge the image, for the \"same region size\" the model actually see \"less\".
\nassume the receptive field for an image is NxN pixels, which is 10% x 10% of the image
\nfor an 2x enlarged image, the receptive field is still NxN pixels, but now looking at only 5%x5% of the enlarged image.
when the model \"see less\", there is too little information to made prediction.
\nlarge image needs deeper model (for larger receptive field ).
\nif you check the image net papers, etc you will see larger models uses larger images to get better results.
note:
\n\n\n\n\nImages hardly fits on GPU memory, and the maximum batch_size is 1-2 i
\n
there is a min batch size requirement to train model. too large or too small batch size doesn't work
", "rawMarkdown": ">>But what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2\n\neach model have a \"receptive field\", it determines the the \"size of the region\" a model can see.\n\nwhen you enlarge the image, for the \"same region size\" the model actually see \"less\". \n\nassume the receptive field for an image is NxN pixels, which is 10% x 10% of the image \nfor an 2x enlarged image, the receptive field is still NxN pixels, but now looking at only 5%x5% of the enlarged image.\n\nwhen the model \"see less\", there is too little information to made prediction.\n\nlarge image needs deeper model (for larger receptive field ).\nif you check the image net papers, etc you will see larger models uses larger images to get better results.\n\n\n---\n\nnote:\n- it doesn't make sense if you can infinitely get better performance by just increasing image size for a fixed model. if so, we need not design new model ... just increase image size.\n\n\n----\n>>Images hardly fits on GPU memory, and the maximum batch_size is 1-2 i\n\nthere is a min batch size requirement to train model. too large or too small batch size doesn't work", "votes": null }, { "id": "1918369", "postDate": "08/29/2022 14:42:49", "content": "@hengck23 Thank you for your detailed explanation
\nReceptive field definitely stays the same
\nWorking setup:
\nEffUNet
\nOriginal image 3000 x 3000 > scale 0.5 > tiling with size 768x768
\nInput for training: patches 768 x 768, spatial size of pixel is 0.8 um
\nReceptive field for every output mask pixel is N x N
For example, I'm replacing it with
\nEffUNet
\nOriginal image 3000 x 3000 > scale 0.5 (no tiling)
\nInput for training: images 1500 x 1500, spatial size of pixel is still 0.8 um
\nReceptive field for every output mask pixel is still N x N (because we didn't make any changes to the model)
So the model sees the same spatial region as before at every mask pixel
\nWhat worries me is that although we did not benefit from using a larger size, we should not have drastically ruined something
", "rawMarkdown": "hengck23 Thank you for your detailed explanation\n\nReceptive field definitely stays the same\n\nWorking setup:\nEffUNet\nOriginal image 3000 x 3000 > scale 0.5 > tiling with size 768x768\nInput for training: patches 768 x 768, spatial size of pixel is 0.8 um\nReceptive field for every output mask pixel is N x N\n\nFor example, I'm replacing it with\nEffUNet \nOriginal image 3000 x 3000 > scale 0.5 (no tiling)\nInput for training: images 1500 x 1500, spatial size of pixel is still 0.8 um\nReceptive field for every output mask pixel is still N x N (because we didn't make any changes to the model)\n\nSo the model sees the same spatial region as before at every mask pixel\n\nWhat worries me is that although we did not benefit from using a larger size, we should not have drastically ruined something", "votes": null } ], "comments": [ { "id": 1918177, "author_name": "celidos", "author_url": "", "post_date": "08/29/2022 11:19:49", "content": "Some other discussions about large images
\nhttps://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941#1907369
\n\n\n\nBut what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2
\n
each model have a \"receptive field\", it determines the the \"size of the region\" a model can see.
\nwhen you enlarge the image, for the \"same region size\" the model actually see \"less\".
\nassume the receptive field for an image is NxN pixels, which is 10% x 10% of the image
\nfor an 2x enlarged image, the receptive field is still NxN pixels, but now looking at only 5%x5% of the enlarged image.
when the model \"see less\", there is too little information to made prediction.
\nlarge image needs deeper model (for larger receptive field ).
\nif you check the image net papers, etc you will see larger models uses larger images to get better results.
note:
\n\n\n\n\nImages hardly fits on GPU memory, and the maximum batch_size is 1-2 i
\n
there is a min batch size requirement to train model. too large or too small batch size doesn't work
", "votes": null, "replies": [ { "id": 1918369, "author_name": "celidos", "author_url": "", "post_date": "08/29/2022 14:42:49", "content": "@hengck23 Thank you for your detailed explanation
\nReceptive field definitely stays the same
\nWorking setup:
\nEffUNet
\nOriginal image 3000 x 3000 > scale 0.5 > tiling with size 768x768
\nInput for training: patches 768 x 768, spatial size of pixel is 0.8 um
\nReceptive field for every output mask pixel is N x N
For example, I'm replacing it with
\nEffUNet
\nOriginal image 3000 x 3000 > scale 0.5 (no tiling)
\nInput for training: images 1500 x 1500, spatial size of pixel is still 0.8 um
\nReceptive field for every output mask pixel is still N x N (because we didn't make any changes to the model)
So the model sees the same spatial region as before at every mask pixel
\nWhat worries me is that although we did not benefit from using a larger size, we should not have drastically ruined something
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1918148": "I have a bunch of pipelines on small patches (512, 768) that give decent baseline solutions (0.75+). But all of them are about tiling original image into small pieces.\n\nWhen I try to apply exact same solutions into larger image tiles or whole image rescale (1024, 1536 etc), a lot of problems arises.\n\nImages hardly fits on GPU memory, and the maximum batch_size is 1-2 in such cases. But what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2 (and after that can even decrease back to 0.02).\nThe same situation happens for UNet-based and transformer-based solutions.\n\nOf course, something is wrong here.\n\nI've tried playing with (virtual) batch_size and learning rate, but without any success. I did not notice any improvement from here, it's still not converging, and there's no intuition about what values these parameters should have.\n\nCan you suggest what might be going wrong? \n\nIf the model is overfitting, then for some reason the model (i.e. UNet) cannot remember even the examples from the training set. Should I add more augmentations or soften them?\n\nMaybe there are some good articles/recommendations on training models on very large images?", "1918177": "Some other discussions about large images\nhttps://www.kaggle.com/competitions/hubmap-organ-segmentation/discussion/332941#1907369", "1918286": ">>But what's worse, the solution doesn't converge at all, and CV dice score stuck at 0.1 - 0.2\n\neach model have a \"receptive field\", it determines the the \"size of the region\" a model can see.\n\nwhen you enlarge the image, for the \"same region size\" the model actually see \"less\". \n\nassume the receptive field for an image is NxN pixels, which is 10% x 10% of the image \nfor an 2x enlarged image, the receptive field is still NxN pixels, but now looking at only 5%x5% of the enlarged image.\n\nwhen the model \"see less\", there is too little information to made prediction.\n\nlarge image needs deeper model (for larger receptive field ).\nif you check the image net papers, etc you will see larger models uses larger images to get better results.\n\n\n---\n\nnote:\n- it doesn't make sense if you can infinitely get better performance by just increasing image size for a fixed model. if so, we need not design new model ... just increase image size.\n\n\n----\n>>Images hardly fits on GPU memory, and the maximum batch_size is 1-2 i\n\nthere is a min batch size requirement to train model. too large or too small batch size doesn't work", "1918369": "hengck23 Thank you for your detailed explanation\n\nReceptive field definitely stays the same\n\nWorking setup:\nEffUNet\nOriginal image 3000 x 3000 > scale 0.5 > tiling with size 768x768\nInput for training: patches 768 x 768, spatial size of pixel is 0.8 um\nReceptive field for every output mask pixel is N x N\n\nFor example, I'm replacing it with\nEffUNet \nOriginal image 3000 x 3000 > scale 0.5 (no tiling)\nInput for training: images 1500 x 1500, spatial size of pixel is still 0.8 um\nReceptive field for every output mask pixel is still N x N (because we didn't make any changes to the model)\n\nSo the model sees the same spatial region as before at every mask pixel\n\nWhat worries me is that although we did not benefit from using a larger size, we should not have drastically ruined something" }, "source": "meta" }