3c9a28bd-084f-4d23-9793-35fffa105652 with varying scale_factor`\n
![\"\"](\"https://i.ibb.co/8z1xZ0j/eg.png\")
During the time I experimented HPACellSegmentator on the dataset, I noticed a few issues about the model/ implementation.
\nCurrently, I couldn't identify if its my implementation problem, or the problem related to the source code. I tried to document them here though. Did you encounter similar issues? If so how did you resolve it? Would like to hear your thoughts!
\nMemory kept noticeably ramping up when I ran the model on more images. I tried to remove prediction output for each run but it didnt help. (Also, the segmentation is pretty slow)
\ne.g. My memory ramped up by 10+GB (cpu mode) simply by one pass of file_id (image size = 2048x2048)) with the following code:
def get_seg_input(file_id):\n colors = ['red', 'yellow', 'blue']\n seg_inputs = map(\n lambda color: [\n str(TRAIN_DIR/f'{file_id}_{color}.png')\n ], colors)\n return list(seg_inputs)\n\ndef run(file_id):\n sample_inputs = get_seg_input(file_id)\n scale_factors = [1., 0.5, 0.25, 0.125]\n\n # experiment different scale_factor\n for idx, scale_factor in enumerate(scale_factors):\n segmentator = cellsegmentator.CellSegmentator(\n NUC_MODEL, CELL_MODEL,\n scale_factor = scale_factor,\n device = 'cpu',\n padding = False,\n multi_channel_model = True)\n # raw cell prediction\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n # raw nuclui prediction\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n # post process cell + nuclei\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n\n # clear out memory leak\n gc.collect()\n del nuclei_seg\n del cell_seg\n del nuclei_mask\n del cell_mask\n del segmentator\nscale_factor = 1.When set scale_factor = 1. and plot out the segmentation result, the background is shown to be red. (v.s. black background when scale_factor != 1).
e.g. segmentation result on 3c9a28bd-084f-4d23-9793-35fffa105652 with varying scale_factor`
\n
The original dataset is too big for me to train/ process, so I experimented the model with varying input image size (2048x2048, 768x768, 512x512, all dtype = np.uint8) and varying scale_factor. I found that the segmentation result is quite sensitive to the 2 factors, I couldnt find a (image_size, scale_factor) combo which lead to a robust & good segmentation result so far.
e.g 1. Example on 3c9a28bd-084f-4d23-9793-35fffa105652
\n![\"\"](\"https://i.ibb.co/8gpwVcX/3c9a28bd-084f-4d23-9793-35fffa105652.png\")
Segmentation result with varying scale_factor when input image size is 512x512:
\n![\"\"](\"https://i.ibb.co/vJZDMCS/3c9a28bd-084f-4d23-9793-35fffa105652-uint8-512.png\")
Segmentation result with varying scale_factor when input image size is 768x768:
\n![\"\"](\"https://i.ibb.co/dmk11Lh/3c9a28bd-084f-4d23-9793-35fffa105652-uint8-768.png\")
Segmentation result with varying scale_factor when input image size is 2048x2048:
\n![\"\"](\"https://i.ibb.co/L9bFBJz/3c9a28bd-084f-4d23-9793-35fffa105652-uint8-2048.png\")
As you see, high res image input (e.g. 2048 x 2048) tends to yield more granular segmentation result but with more false positive, and low res image input (e.g. 512 x 512) tends to yield more coarse result with some cells being mistakenly merged together.
\n(I would like to avoid using high res image, but seems like high res image sometimes yield a better result. Is there any workaround to get a decent segmentation result with low res input image?)
During the time I experimented HPACellSegmentator on the dataset, I noticed a few issues about the model/ implementation.
\nCurrently, I couldn't identify if its my implementation problem, or the problem related to the source code. I tried to document them here though. Did you encounter similar issues? If so how did you resolve it? Would like to hear your thoughts!
\nMemory kept noticeably ramping up when I ran the model on more images. I tried to remove prediction output for each run but it didnt help. (Also, the segmentation is pretty slow)
\ne.g. My memory ramped up by 10+GB (cpu mode) simply by one pass of file_id (image size = 2048x2048)) with the following code:
def get_seg_input(file_id):\n colors = ['red', 'yellow', 'blue']\n seg_inputs = map(\n lambda color: [\n str(TRAIN_DIR/f'{file_id}_{color}.png')\n ], colors)\n return list(seg_inputs)\n\ndef run(file_id):\n sample_inputs = get_seg_input(file_id)\n scale_factors = [1., 0.5, 0.25, 0.125]\n\n # experiment different scale_factor\n for idx, scale_factor in enumerate(scale_factors):\n segmentator = cellsegmentator.CellSegmentator(\n NUC_MODEL, CELL_MODEL,\n scale_factor = scale_factor,\n device = 'cpu',\n padding = False,\n multi_channel_model = True)\n # raw cell prediction\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n # raw nuclui prediction\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n # post process cell + nuclei\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n\n # clear out memory leak\n gc.collect()\n del nuclei_seg\n del cell_seg\n del nuclei_mask\n del cell_mask\n del segmentator\nscale_factor = 1.When set scale_factor = 1. and plot out the segmentation result, the background is shown to be red. (v.s. black background when scale_factor != 1).
e.g. segmentation result on 3c9a28bd-084f-4d23-9793-35fffa105652 with varying scale_factor`
\n
The original dataset is too big for me to train/ process, so I experimented the model with varying input image size (2048x2048, 768x768, 512x512, all dtype = np.uint8) and varying scale_factor. I found that the segmentation result is quite sensitive to the 2 factors, I couldnt find a (image_size, scale_factor) combo which lead to a robust & good segmentation result so far.
e.g 1. Example on 3c9a28bd-084f-4d23-9793-35fffa105652
\n
Segmentation result with varying scale_factor when input image size is 512x512:
\n
Segmentation result with varying scale_factor when input image size is 768x768:
\n
Segmentation result with varying scale_factor when input image size is 2048x2048:
\n
As you see, high res image input (e.g. 2048 x 2048) tends to yield more granular segmentation result but with more false positive, and low res image input (e.g. 512 x 512) tends to yield more coarse result with some cells being mistakenly merged together.
\n(I would like to avoid using high res image, but seems like high res image sometimes yield a better result. Is there any workaround to get a decent segmentation result with low res input image?)
Regarding the potential memory leaks:
\nI will take a look through the code to see if I can find any memory leaks in there, and see if I can do anything about it. In the meantime, I don't know much else to do than using del liberally.
I did something different than your experiment, but did see something similar with regards to memory - since my system has 264 GB of cpu ram I only noticed it in passing and did nothing to investigate, so my recollection and guess of the issue may be uncertain.
\nBut it seems that HPA-Cell_segmentation might be loading it's models and they are not getting removed with your clean - also I would do a gc.collect() after the last del in your loop. I don't do much torch stuff so don't know if there is a better way to remove torch models from memory. Will try your loop in the next day or so when I get some free time.
\nMost folks doing submissions would only instance HPA-Cell_segmentation one time which would explain how others can have successful submissions.
", "rawMarkdown": "[Alex](https://www.kaggle.com/alexlwh)\n\nI did something different than your experiment, but did see something similar with regards to memory - since my system has 264 GB of cpu ram I only noticed it in passing and did nothing to investigate, so my recollection and guess of the issue may be uncertain.\n\nBut it seems that HPA-Cell_segmentation might be loading it's models and they are not getting removed with your clean - also I would do a gc.collect() after the last del in your loop. I don't do much torch stuff so don't know if there is a better way to remove torch models from memory. Will try your loop in the next day or so when I get some free time.\n\nMost folks doing submissions would only instance HPA-Cell_segmentation one time which would explain how others can have successful submissions.", "votes": 1, "replies": [ { "id": 1201771, "postDate": "2021-02-15T16:39:35.210Z", "content": "@pcjimmmy
\nThanks for the suggestion. I will try that out.
\nI do suspect though the memory leak is more than failure to remove the model from memory. Coz from the memory footprint I noticed, the memory ramps up so much more when high res image is passed to the model, so I guess the memory leak is dependent on the image input as well.
I would like to find a solution for that because I may need HPA-Cell_segmentation in my training pipeline as well. Making it free of memory leak & performing decently on low res image would be so much better hardware-resource-wise.
", "rawMarkdown": "@pcjimmmy \nThanks for the suggestion. I will try that out.\nI do suspect though the memory leak is more than failure to remove the model from memory. Coz from the memory footprint I noticed, the memory ramps up so much more when high res image is passed to the model, so I guess the memory leak is dependent on the image input as well. \n\nI would like to find a solution for that because I may need HPA-Cell_segmentation in my training pipeline as well. Making it free of memory leak & performing decently on low res image would be so much better hardware-resource-wise. " }, { "id": 1201962, "postDate": "2021-02-15T19:16:03.093Z", "content": "I agree - removal of models is probably the issue.
\nIt does seem that HPA-Cell_Segmentation will be the best match to the private test masks and likely the best way to segment.
\nGetting that segmentation into the pipeline seeming to be a very time intensive task. The train masks that have been shared are good start but I find a lot of them that should be dropped out. So started my own build of single cell images and collecting what I hope is useful meta data as I build the files - but very painful and slow process. My first version attempt has completed about 1/2 of the images and at around 40 hours run time. My second very just started - managed to cut a little time and found two more features to save.
\nGive update if you find a solution - I am starting to play now with your code and experiment. Don't think I will be as robust as you but not sure how to best evaluate segmentation beyond the eyeball test. Since we don't have any ground truth for the segments I am wondering what to look for - doing submissions to Kaggle is too slow of a process but assume I will need to use some LB submissions.
", "rawMarkdown": "I agree - removal of models is probably the issue. \n\nIt does seem that HPA-Cell_Segmentation will be the best match to the private test masks and likely the best way to segment.\n\nGetting that segmentation into the pipeline seeming to be a very time intensive task. The train masks that have been shared are good start but I find a lot of them that should be dropped out. So started my own build of single cell images and collecting what I hope is useful meta data as I build the files - but very painful and slow process. My first version attempt has completed about 1/2 of the images and at around 40 hours run time. My second very just started - managed to cut a little time and found two more features to save.\n\nGive update if you find a solution - I am starting to play now with your code and experiment. Don't think I will be as robust as you but not sure how to best evaluate segmentation beyond the eyeball test. Since we don't have any ground truth for the segments I am wondering what to look for - doing submissions to Kaggle is too slow of a process but assume I will need to use some LB submissions.\n\n" }, { "id": 1204232, "postDate": "2021-02-16T00:27:16.487Z", "content": "Completed my first experiment - ran 10 images at 12 different scales. Formed some general impressions using meta data I created and analyzed in JMP. Not yet added code to save the masks to do the eyeball examination. There are several major issues I see in the segmentation so far.
\n1) single cells having two nuclei - pretty sure that there should be no segmented cell containing two nuclei.
\n2) cells on the border of the image too small to be evaluated - will probably need to post process to take care of this issue.
\n3) wrong number of single cells found - with no ground truth this will be fun to fix.
I had no memory leak using only a single del and gc.collect.
\nRunning on Ubuntu 20.04 with standard Anaconda install.
Started a run of 100 to see if larger sample size confirms some of the apparent trends as scale is changed. Will try 1000 images to run over night. JMP is my quickie stat tool of choice - the version I have is very old and x32 bit so my image count limit might peak at 1000 images. But we will see :)
\nWill update later - tomorrow will add saves of the masks images for eye ball exam.
", "rawMarkdown": "Completed my first experiment - ran 10 images at 12 different scales. Formed some general impressions using meta data I created and analyzed in JMP. Not yet added code to save the masks to do the eyeball examination. There are several major issues I see in the segmentation so far.\n\n1) single cells having two nuclei - pretty sure that there should be no segmented cell containing two nuclei.\n2) cells on the border of the image too small to be evaluated - will probably need to post process to take care of this issue. \n3) wrong number of single cells found - with no ground truth this will be fun to fix.\n\n**I had no memory leak using only a single del and gc.collect.**\nRunning on Ubuntu 20.04 with standard Anaconda install.\n\n\nStarted a run of 100 to see if larger sample size confirms some of the apparent trends as scale is changed. Will try 1000 images to run over night. JMP is my quickie stat tool of choice - the version I have is very old and x32 bit so my image count limit might peak at 1000 images. But we will see :)\n\nWill update later - tomorrow will add saves of the masks images for eye ball exam.\n", "votes": 2 }, { "id": 1204643, "postDate": "2021-02-16T09:23:33.587Z", "content": "Completed 1000 images at 12 different scales with padding = True - using the loop shown below - 7.5 hours run time.
\nWorked fine for the entire range of padding = True
\n- but dies a horrible death with padding = False.
Turns out the only scale values that work when padding = False are
\nscale_range = [0.0625, 0.125, 0.25, 0.5, 0.75, 1.0]
\nRestarted the full run - so back in 7 or so hours with another update.
Using padding = True will offer more resolution to get number of segments nominal, while padding = False seems limited to the 6 values shown.
\nThe error reminds me of the good old days of tensorflow 1.x when input and ouput dimensions did not match between layers.
\nmulti_channels = [True]\npadding = [True, False]\nscale_range = [0.05, 0.1, 0.15, 0.2, 0.25, 0.3,0.4,0.5,0.6,0.7,.8,0.9]\n\nrun_number = 0\n\nfor scale in scale_range:\n print(scale)\n for pad in padding:\n print (pad)\n for multi in multi_channels:\n print(multi)\n .......\nTemporary (or maybe permanent) halt to running my experiment on 1000 images as shown above.
\nSome observations of the data NOT supported by analytical methods. (ok - I eye balled the data)
\nPadding setting seems to impact speed - True is faster than False.
\nPadding = True allows for any scale value you want. = False than only 6 levels of resolution.
\nSmall scale values and very large scale values result in large segments counts or large segments. The sweet spot seems to run from 0.15 to 0.40 - changes in 0.01 increments (with padding = True) will change the average number of segments per image on what might be increasing linear slope.
\nOn both sides of 0.25 I could find a slightly better LB score (+/- 0.05 range) - but LB score probing always comes with risk :)
\nVery low and very high scale values will result in crashes when processing 1K worth of images - assume that the \"crash\" range narrows when doing 21K of training images. Most of the crashes are in the code written by me - my error trapping is marginal at best.
\nPer the original question - I never saw a memory leak BUT memory use does change at the extremes of scale values.
\nLooking at the source code it seems likely that the models they generated and the code was based on large images - 2048x2048 size images (or other large sizes) rather than smaller images.
my plan of attack - use padding = true and scale = 0.25 on 2048x2048 to get segments. After a model is as sweet as I plan to get it - use two days worth of submissions to explore 0.20 to 0.30 scale range.
", "rawMarkdown": "Temporary (or maybe permanent) halt to running my experiment on 1000 images as shown above. \n\nSome observations of the data NOT supported by analytical methods. (ok - I eye balled the data)\n\nPadding setting seems to impact speed - True is faster than False.\nPadding = True allows for any scale value you want. = False than only 6 levels of resolution.\nSmall scale values and very large scale values result in large segments counts or large segments. The sweet spot seems to run from 0.15 to 0.40 - changes in 0.01 increments (with padding = True) will change the average number of segments per image on what might be increasing linear slope.\nOn both sides of 0.25 I could find a slightly better LB score (+/- 0.05 range) - but LB score probing always comes with risk :)\nVery low and very high scale values will result in crashes when processing 1K worth of images - assume that the \"crash\" range narrows when doing 21K of training images. Most of the crashes are in the code written by me - my error trapping is marginal at best.\n**Per the original question - I never saw a memory leak BUT memory use does change at the extremes of scale values.**\nLooking at the source code it seems likely that the models they generated and the code was based on large images - 2048x2048 size images (or other large sizes) rather than smaller images.\n\nmy plan of attack - use padding = true and scale = 0.25 on 2048x2048 to get segments. After a model is as sweet as I plan to get it - use two days worth of submissions to explore 0.20 to 0.30 scale range.\n" }, { "id": 1215265, "postDate": "2021-02-23T13:49:36.333Z", "content": "@pcjimmmy
\nThanks for your detailed experiments! I was using padding = False, didnt know padding = True could enable higher resolution of scale_factor and run faster. Nice finding!
As for memory leak issue, I encountered this with the following sample code. You could try and verify:
\n# HELPER FUNCTIONS\nTRAIN_DIR = Path(ROOT/'test')\n\ndef get_seg_input(file_id):\n colors = ['red', 'yellow', 'blue']\n seg_inputs = map(\n lambda color: [\n str(TRAIN_DIR/f'{file_id}_{color}.png')\n ], colors)\n return list(seg_inputs)\n\ndef init_segmentator(*args, **kwargs):\n segmentator = cellsegmentator.CellSegmentator(*args, **kwargs)\n return segmentator\n\n# FIRST RUN\nprint('First Run')\nkwargs = {\n \"nuclei_model\": NUC_MODEL,\n \"cell_model\": CELL_MODEL,\n \"scale_factor\": 1.,\n \"device\": \"cpu\",\n \"padding\": False,\n \"multi_channel_model\": True\n}\nsegmentator = init_segmentator(**kwargs)\nfor _file_id in fns[:3]:\n print(f'Processing: {_file_id}')\n\n sample_inputs = get_seg_input(_file_id)\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n\n print(f'Completed: {_file_id}')\n\n# SECOND RUN\nprint('Second Run')\nkwargs = {\n \"nuclei_model\": NUC_MODEL,\n \"cell_model\": CELL_MODEL,\n \"scale_factor\": 1.,\n \"device\": \"cpu\",\n \"padding\": True,\n \"multi_channel_model\": True\n}\nsegmentator = init_segmentator(**kwargs)\nfor _file_id in fns[:3]:\n print(f'Processing: {_file_id}')\n\n sample_inputs = get_seg_input(_file_id)\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n\n print(f'Completed: {_file_id}')\nI tested on scale_factor = 1 because its the configuration that used most memory, compared against other scale_factor. From what I observed, the memory started to significantly ramp up at second run. But strangely, when I set the second run to have exactly the same model configuration as the first one (i.e. kwargs is the same as first run), I didnt encounter the memory ramping up.
\nTherefore, I suspected the memory leak (if exist) is coming from model runs after change in model configuration.
", "rawMarkdown": "@pcjimmmy \nThanks for your detailed experiments! I was using padding = False, didnt know padding = True could enable higher resolution of scale_factor and run faster. Nice finding!\n\nAs for memory leak issue, I encountered this with the following sample code. You could try and verify:\n\n```\n# HELPER FUNCTIONS\nTRAIN_DIR = Path(ROOT/'test')\n\ndef get_seg_input(file_id):\n colors = ['red', 'yellow', 'blue']\n seg_inputs = map(\n lambda color: [\n str(TRAIN_DIR/f'{file_id}_{color}.png')\n ], colors)\n return list(seg_inputs)\n\ndef init_segmentator(*args, **kwargs):\n segmentator = cellsegmentator.CellSegmentator(*args, **kwargs)\n return segmentator\n\n# FIRST RUN\nprint('First Run')\nkwargs = {\n \"nuclei_model\": NUC_MODEL,\n \"cell_model\": CELL_MODEL,\n \"scale_factor\": 1.,\n \"device\": \"cpu\",\n \"padding\": False,\n \"multi_channel_model\": True\n}\nsegmentator = init_segmentator(**kwargs)\nfor _file_id in fns[:3]:\n print(f'Processing: {_file_id}')\n \n sample_inputs = get_seg_input(_file_id)\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n \n print(f'Completed: {_file_id}')\n\n# SECOND RUN\nprint('Second Run')\nkwargs = {\n \"nuclei_model\": NUC_MODEL,\n \"cell_model\": CELL_MODEL,\n \"scale_factor\": 1.,\n \"device\": \"cpu\",\n \"padding\": True,\n \"multi_channel_model\": True\n}\nsegmentator = init_segmentator(**kwargs)\nfor _file_id in fns[:3]:\n print(f'Processing: {_file_id}')\n \n sample_inputs = get_seg_input(_file_id)\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n \n print(f'Completed: {_file_id}')\n\n```\n\nI tested on scale_factor = 1 because its the configuration that used most memory, compared against other scale_factor. **From what I observed, the memory started to significantly ramp up at second run.** But strangely, when I set the second run to have exactly the same model configuration as the first one (i.e. kwargs is the same as first run), I didnt encounter the memory ramping up. \n\nTherefore, I suspected the memory leak (if exist) is coming from model runs after change in model configuration." } ] }, { "id": 1204565, "postDate": "2021-02-16T08:07:33.010Z", "content": "Also noticed the differences between 1728, 2048 and 3072 outputs.
\n2048 is better, so now i'm trying to resize all the test images to 2048, segment them and finally resize to the original size.
Thanks for sharing your finding! What value of scale_factor did u find to work best?
", "rawMarkdown": "Thanks for sharing your finding! What value of scale_factor did u find to work best?" } ] }, { "id": 1232905, "postDate": "2021-03-10T03:21:22.903Z", "content": "For issue 3. I think this is probably because the remove_small_objects methods used in the label_cell function has some parameters that are independent from the size of the image, so the small objects of the downsampled images are filtered out.
", "rawMarkdown": "For issue 3. I think this is probably because the remove_small_objects methods used in the label_cell function has some parameters that are independent from the size of the image, so the small objects of the downsampled images are filtered out. " }, { "id": 1201517, "postDate": "2021-02-15T13:05:50.790Z", "content": "Hi! I just want to point out that our base segmentation model is here https://github.com/CellProfiling/HPA-Cell-Segmentation
\nYou are of course free to use whatever model you find suitable. In our pretrained model, we used full sized images and scale_factor=0.25. Maybe it will work with the model you used above instead of 1?
\nHope someone else can shred some light on memory issue in Kaggle kernel because many have succeeded in submitting full prediction for whole test set.
I am using the base segmentation model in my experiment (See point no. 1 for my sample code)
\nOn point no. 3, I did try different image size (full, 768x768, 512x512) with different scale factors, and then showed out some sample segmentation results. It seems the config you suggest did better than others, but in some cases it may have the wrong split (see the plots in point 3). Therefore, I would like to shred some light here and see if anyone has found a config even better than this.
", "rawMarkdown": "I am using the base segmentation model in my experiment (See point no. 1 for my sample code)\n\nOn point no. 3, I did try different image size (full, 768x768, 512x512) with different scale factors, and then showed out some sample segmentation results. It seems the config you suggest did better than others, but in some cases it may have the wrong split (see the plots in point 3). Therefore, I would like to shred some light here and see if anyone has found a config even better than this." }, { "id": 1208546, "postDate": "2021-02-18T10:32:07.550Z", "content": "@alexlwh I am also a bit confused about the post, the link you posted (i.e.: https://github.com/marshuang80/cell-segmentation) point to a repo that is not from us, but in the code you seems use our HPACellSegmentator. Could you please clarify this?
", "rawMarkdown": "@alexlwh I am also a bit confused about the post, the link you posted (i.e.: https://github.com/marshuang80/cell-segmentation) point to a repo that is not from us, but in the code you seems use our HPACellSegmentator. Could you please clarify this?" }, { "id": 1215193, "postDate": "2021-02-23T12:43:16.337Z", "content": "@weiouyang
\nsorry for the confusion. I am referring to HPA Segmentation. The linked I posted is a wrong one. I have updated the link
Regarding the potential memory leaks:
\nI will take a look through the code to see if I can find any memory leaks in there, and see if I can do anything about it. In the meantime, I don't know much else to do than using del liberally.
I did something different than your experiment, but did see something similar with regards to memory - since my system has 264 GB of cpu ram I only noticed it in passing and did nothing to investigate, so my recollection and guess of the issue may be uncertain.
\nBut it seems that HPA-Cell_segmentation might be loading it's models and they are not getting removed with your clean - also I would do a gc.collect() after the last del in your loop. I don't do much torch stuff so don't know if there is a better way to remove torch models from memory. Will try your loop in the next day or so when I get some free time.
\nMost folks doing submissions would only instance HPA-Cell_segmentation one time which would explain how others can have successful submissions.
", "votes": 1, "replies": [ { "id": 1201771, "author_name": "Alex Lau", "author_url": "", "post_date": "2021-02-15T16:39:35.210000", "content": "@pcjimmmy
\nThanks for the suggestion. I will try that out.
\nI do suspect though the memory leak is more than failure to remove the model from memory. Coz from the memory footprint I noticed, the memory ramps up so much more when high res image is passed to the model, so I guess the memory leak is dependent on the image input as well.
I would like to find a solution for that because I may need HPA-Cell_segmentation in my training pipeline as well. Making it free of memory leak & performing decently on low res image would be so much better hardware-resource-wise.
", "votes": 0, "replies": [] }, { "id": 1201962, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-15T19:16:03.093000", "content": "I agree - removal of models is probably the issue.
\nIt does seem that HPA-Cell_Segmentation will be the best match to the private test masks and likely the best way to segment.
\nGetting that segmentation into the pipeline seeming to be a very time intensive task. The train masks that have been shared are good start but I find a lot of them that should be dropped out. So started my own build of single cell images and collecting what I hope is useful meta data as I build the files - but very painful and slow process. My first version attempt has completed about 1/2 of the images and at around 40 hours run time. My second very just started - managed to cut a little time and found two more features to save.
\nGive update if you find a solution - I am starting to play now with your code and experiment. Don't think I will be as robust as you but not sure how to best evaluate segmentation beyond the eyeball test. Since we don't have any ground truth for the segments I am wondering what to look for - doing submissions to Kaggle is too slow of a process but assume I will need to use some LB submissions.
", "votes": 0, "replies": [] }, { "id": 1204232, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-16T00:27:16.487000", "content": "Completed my first experiment - ran 10 images at 12 different scales. Formed some general impressions using meta data I created and analyzed in JMP. Not yet added code to save the masks to do the eyeball examination. There are several major issues I see in the segmentation so far.
\n1) single cells having two nuclei - pretty sure that there should be no segmented cell containing two nuclei.
\n2) cells on the border of the image too small to be evaluated - will probably need to post process to take care of this issue.
\n3) wrong number of single cells found - with no ground truth this will be fun to fix.
I had no memory leak using only a single del and gc.collect.
\nRunning on Ubuntu 20.04 with standard Anaconda install.
Started a run of 100 to see if larger sample size confirms some of the apparent trends as scale is changed. Will try 1000 images to run over night. JMP is my quickie stat tool of choice - the version I have is very old and x32 bit so my image count limit might peak at 1000 images. But we will see :)
\nWill update later - tomorrow will add saves of the masks images for eye ball exam.
", "votes": 2, "replies": [] }, { "id": 1204643, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-16T09:23:33.587000", "content": "Completed 1000 images at 12 different scales with padding = True - using the loop shown below - 7.5 hours run time.
\nWorked fine for the entire range of padding = True
\n- but dies a horrible death with padding = False.
Turns out the only scale values that work when padding = False are
\nscale_range = [0.0625, 0.125, 0.25, 0.5, 0.75, 1.0]
\nRestarted the full run - so back in 7 or so hours with another update.
Using padding = True will offer more resolution to get number of segments nominal, while padding = False seems limited to the 6 values shown.
\nThe error reminds me of the good old days of tensorflow 1.x when input and ouput dimensions did not match between layers.
\nmulti_channels = [True]\npadding = [True, False]\nscale_range = [0.05, 0.1, 0.15, 0.2, 0.25, 0.3,0.4,0.5,0.6,0.7,.8,0.9]\n\nrun_number = 0\n\nfor scale in scale_range:\n print(scale)\n for pad in padding:\n print (pad)\n for multi in multi_channels:\n print(multi)\n .......\nTemporary (or maybe permanent) halt to running my experiment on 1000 images as shown above.
\nSome observations of the data NOT supported by analytical methods. (ok - I eye balled the data)
\nPadding setting seems to impact speed - True is faster than False.
\nPadding = True allows for any scale value you want. = False than only 6 levels of resolution.
\nSmall scale values and very large scale values result in large segments counts or large segments. The sweet spot seems to run from 0.15 to 0.40 - changes in 0.01 increments (with padding = True) will change the average number of segments per image on what might be increasing linear slope.
\nOn both sides of 0.25 I could find a slightly better LB score (+/- 0.05 range) - but LB score probing always comes with risk :)
\nVery low and very high scale values will result in crashes when processing 1K worth of images - assume that the \"crash\" range narrows when doing 21K of training images. Most of the crashes are in the code written by me - my error trapping is marginal at best.
\nPer the original question - I never saw a memory leak BUT memory use does change at the extremes of scale values.
\nLooking at the source code it seems likely that the models they generated and the code was based on large images - 2048x2048 size images (or other large sizes) rather than smaller images.
my plan of attack - use padding = true and scale = 0.25 on 2048x2048 to get segments. After a model is as sweet as I plan to get it - use two days worth of submissions to explore 0.20 to 0.30 scale range.
", "votes": 0, "replies": [] }, { "id": 1215265, "author_name": "Alex Lau", "author_url": "", "post_date": "2021-02-23T13:49:36.333000", "content": "@pcjimmmy
\nThanks for your detailed experiments! I was using padding = False, didnt know padding = True could enable higher resolution of scale_factor and run faster. Nice finding!
As for memory leak issue, I encountered this with the following sample code. You could try and verify:
\n# HELPER FUNCTIONS\nTRAIN_DIR = Path(ROOT/'test')\n\ndef get_seg_input(file_id):\n colors = ['red', 'yellow', 'blue']\n seg_inputs = map(\n lambda color: [\n str(TRAIN_DIR/f'{file_id}_{color}.png')\n ], colors)\n return list(seg_inputs)\n\ndef init_segmentator(*args, **kwargs):\n segmentator = cellsegmentator.CellSegmentator(*args, **kwargs)\n return segmentator\n\n# FIRST RUN\nprint('First Run')\nkwargs = {\n \"nuclei_model\": NUC_MODEL,\n \"cell_model\": CELL_MODEL,\n \"scale_factor\": 1.,\n \"device\": \"cpu\",\n \"padding\": False,\n \"multi_channel_model\": True\n}\nsegmentator = init_segmentator(**kwargs)\nfor _file_id in fns[:3]:\n print(f'Processing: {_file_id}')\n\n sample_inputs = get_seg_input(_file_id)\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n\n print(f'Completed: {_file_id}')\n\n# SECOND RUN\nprint('Second Run')\nkwargs = {\n \"nuclei_model\": NUC_MODEL,\n \"cell_model\": CELL_MODEL,\n \"scale_factor\": 1.,\n \"device\": \"cpu\",\n \"padding\": True,\n \"multi_channel_model\": True\n}\nsegmentator = init_segmentator(**kwargs)\nfor _file_id in fns[:3]:\n print(f'Processing: {_file_id}')\n\n sample_inputs = get_seg_input(_file_id)\n cell_seg = segmentator.pred_cells(sample_inputs)[0]\n nuclei_seg = segmentator.pred_nuclei(sample_inputs[2])[0]\n nuclei_mask, cell_mask = label_cell(nuclei_seg, cell_seg)\n\n print(f'Completed: {_file_id}')\nI tested on scale_factor = 1 because its the configuration that used most memory, compared against other scale_factor. From what I observed, the memory started to significantly ramp up at second run. But strangely, when I set the second run to have exactly the same model configuration as the first one (i.e. kwargs is the same as first run), I didnt encounter the memory ramping up.
\nTherefore, I suspected the memory leak (if exist) is coming from model runs after change in model configuration.
", "votes": 0, "replies": [] } ] }, { "id": 1204565, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-02-16T08:07:33.010000", "content": "Also noticed the differences between 1728, 2048 and 3072 outputs.
\n2048 is better, so now i'm trying to resize all the test images to 2048, segment them and finally resize to the original size.
Thanks for sharing your finding! What value of scale_factor did u find to work best?
", "votes": 0, "replies": [] } ] }, { "id": 1232905, "author_name": "zhen", "author_url": "", "post_date": "2021-03-10T03:21:22.903000", "content": "For issue 3. I think this is probably because the remove_small_objects methods used in the label_cell function has some parameters that are independent from the size of the image, so the small objects of the downsampled images are filtered out.
", "votes": 0, "replies": [] }, { "id": 1201517, "author_name": "Trang Le", "author_url": "", "post_date": "2021-02-15T13:05:50.790000", "content": "Hi! I just want to point out that our base segmentation model is here https://github.com/CellProfiling/HPA-Cell-Segmentation
\nYou are of course free to use whatever model you find suitable. In our pretrained model, we used full sized images and scale_factor=0.25. Maybe it will work with the model you used above instead of 1?
\nHope someone else can shred some light on memory issue in Kaggle kernel because many have succeeded in submitting full prediction for whole test set.
I am using the base segmentation model in my experiment (See point no. 1 for my sample code)
\nOn point no. 3, I did try different image size (full, 768x768, 512x512) with different scale factors, and then showed out some sample segmentation results. It seems the config you suggest did better than others, but in some cases it may have the wrong split (see the plots in point 3). Therefore, I would like to shred some light here and see if anyone has found a config even better than this.
", "votes": 0, "replies": [] }, { "id": 1208546, "author_name": "Wei Ouyang", "author_url": "", "post_date": "2021-02-18T10:32:07.550000", "content": "@alexlwh I am also a bit confused about the post, the link you posted (i.e.: https://github.com/marshuang80/cell-segmentation) point to a repo that is not from us, but in the code you seems use our HPACellSegmentator. Could you please clarify this?
", "votes": 0, "replies": [] }, { "id": 1215193, "author_name": "Alex Lau", "author_url": "", "post_date": "2021-02-23T12:43:16.337000", "content": "@weiouyang
\nsorry for the confusion. I am referring to HPA Segmentation. The linked I posted is a wrong one. I have updated the link