{ "id": 222094, "title": "Puzzle-CAM", "url": "/competitions/hpa-single-cell-image-classification/discussion/222094", "author_name": "Alexander Riedel", "post_date": "2021-02-25T09:12:37.803000", "votes": 22, "comment_count": 61, "views": 0, "content": "

Hello!
\nIs anyone out there who's also working on a Puzzle-CAM approach (https://github.com/OFRIN/PuzzleCAM) and wants to share their findings, problems, etc. ?

\n

I'm currently training a ResNest50 with that method, using the standard parameters for training and explore the produced CAMs before going on training an AffinityNet and a segmentation model.

\n

In the end I will use the approach of @phalanx, using the element product of HPA Segmentor predictions and the CAMs (resp. output of the trained segmentation model).

\n

I guess phalanx only trained a model using Puzzle-CAMS to get CAMS without going the extra way of training a segmentation model.

\n

To show you what it's look so far, some nice activations :)

\n

\"\"
\n\"\"

", "messages": [ { "id": 1217725, "postDate": "2021-02-25T09:12:37.803Z", "content": "

Hello!
\nIs anyone out there who's also working on a Puzzle-CAM approach (https://github.com/OFRIN/PuzzleCAM) and wants to share their findings, problems, etc. ?

\n

I'm currently training a ResNest50 with that method, using the standard parameters for training and explore the produced CAMs before going on training an AffinityNet and a segmentation model.

\n

In the end I will use the approach of @phalanx, using the element product of HPA Segmentor predictions and the CAMs (resp. output of the trained segmentation model).

\n

I guess phalanx only trained a model using Puzzle-CAMS to get CAMS without going the extra way of training a segmentation model.

\n

To show you what it's look so far, some nice activations :)

\n

\"\"
\n\"\"

", "rawMarkdown": "Hello!\nIs anyone out there who's also working on a Puzzle-CAM approach (https://github.com/OFRIN/PuzzleCAM) and wants to share their findings, problems, etc. ?\n\nI'm currently training a ResNest50 with that method, using the standard parameters for training and explore the produced CAMs before going on training an AffinityNet and a segmentation model. \n\nIn the end I will use the approach of @phalanx, using the element product of HPA Segmentor predictions and the CAMs (resp. output of the trained segmentation model).\n\nI guess phalanx only trained a model using Puzzle-CAMS to get CAMS without going the extra way of training a segmentation model. \n\nTo show you what it's look so far, some nice activations :)\n\n![](https://i.ibb.co/rZbtVGM/Unbenannt.jpg)\n![](https://i.ibb.co/MZB98bF/Unbenann1.jpg)\n\n", "votes": 22 }, { "id": 1237865, "postDate": "2021-03-14T13:37:38.407Z", "content": "

I've upgraded my inference code, now instead of argmax on CAMs, I'm doing the following:

\n
CAMs = get_hr_cams(rgb_image, th=0.1)[...,1:]\n\nfor class_id in range(len(classes)):\n    for cell_idx in range(1, cell_mask.max()+1):\n        result = CAMs[...,class_id]*(nuclei_mask==cell_idx)\n        num_pixels = np.count_nonzero(result)\n        if num_pixels:\n            prob = result.sum()/num_pixels\n
\n

This way you don't need to standardize or sigmoid anything. You always get prob in a range [0,1]. Now it scores 0.15 on LB, which is 3 times better than with argmax before.

", "rawMarkdown": "I've upgraded my inference code, now instead of argmax on CAMs, I'm doing the following:\n\n```\nCAMs = get_hr_cams(rgb_image, th=0.1)[...,1:]\n\nfor class_id in range(len(classes)):\n for cell_idx in range(1, cell_mask.max()+1):\n result = CAMs[...,class_id]*(nuclei_mask==cell_idx)\n num_pixels = np.count_nonzero(result)\n if num_pixels:\n prob = result.sum()/num_pixels\n```\nThis way you don't need to standardize or sigmoid anything. You always get prob in a range [0,1]. Now it scores 0.15 on LB, which is 3 times better than with argmax before.", "votes": 3, "replies": [ { "id": 1237913, "postDate": "2021-03-14T13:55:06.470Z", "content": "

ok so you basically divide the \"class-pixels\" for each cell by the size of the cell? Is this it?

", "rawMarkdown": "ok so you basically divide the \"class-pixels\" for each cell by the size of the cell? Is this it? " }, { "id": 1237926, "postDate": "2021-03-14T13:58:38.553Z", "content": "

yes, I guess it's the most straightforward way to get confidence for the cell.

", "rawMarkdown": "yes, I guess it's the most straightforward way to get confidence for the cell.", "votes": 1, "replies": [ { "id": 1237940, "postDate": "2021-03-14T14:09:56.963Z", "content": "

Yes I understand. I tried to do the same but my intuition was, that for some classes, the area of activation is very small (compared to the whole cell), like the centrosome or nuclear bodies and some others. If I devide them by the full cell size, the activation will vanish maybe for the small classes.

", "rawMarkdown": "Yes I understand. I tried to do the same but my intuition was, that for some classes, the area of activation is very small (compared to the whole cell), like the centrosome or nuclear bodies and some others. If I devide them by the full cell size, the activation will vanish maybe for the small classes. " } ] }, { "id": 1238777, "postDate": "2021-03-15T09:37:49.227Z", "content": "

But I don't divide by the full cell size. I get intersection between CAM for the class and nuclei_mask corresponding to this CAM. And only from this area of intersection I calculate the confidence.

", "rawMarkdown": "But I don't divide by the full cell size. I get intersection between CAM for the class and nuclei_mask corresponding to this CAM. And only from this area of intersection I calculate the confidence.", "votes": 1 }, { "id": 1247531, "postDate": "2021-03-21T19:54:20.203Z", "content": "

I feel this is the right way as CAM was trained with avg pooling, so you basically get the avg pooling for specif nuclei in this operation

", "rawMarkdown": "I feel this is the right way as CAM was trained with avg pooling, so you basically get the avg pooling for specif nuclei in this operation" }, { "id": 1254752, "postDate": "2021-03-28T04:40:31.937Z", "content": "

@vostankovich wondering how do you find the best threshold?

", "rawMarkdown": "@vostankovich wondering how do you find the best threshold?" }, { "id": 1254989, "postDate": "2021-03-28T10:07:04.817Z", "content": "

I think that threshold didn't change anything (it applies only to background). I just took it as it was from original repo.

", "rawMarkdown": "I think that threshold didn't change anything (it applies only to background). I just took it as it was from original repo.", "votes": 1 }, { "id": 1254991, "postDate": "2021-03-28T10:10:11.227Z", "content": "

@vostankovich Are you still using Puzzle-CAM / CAMs in general for your LB Score? πŸ™‚
\nI'm going to start working on getting CAMs from Vision Transformers for now :)

", "rawMarkdown": "@vostankovich Are you still using Puzzle-CAM / CAMs in general for your LB Score? πŸ™‚\nI'm going to start working on getting CAMs from Vision Transformers for now :)", "votes": 1 }, { "id": 1260983, "postDate": "2021-04-02T15:23:58.250Z", "content": "

Oh, sorry for late reply. I gave up using CAMs approach long time ago. Don't know why but I didn't believe it could lead to high score.

", "rawMarkdown": "Oh, sorry for late reply. I gave up using CAMs approach long time ago. Don't know why but I didn't believe it could lead to high score.", "votes": 1 } ] }, { "id": 1283998, "postDate": "2021-04-25T13:24:57.487Z", "content": "

Hi, thanks a lot for sharing your experiments , it was really insightful. I have some questions -
\nDid you stick to using CAMs approach? Did it work well for you?
\nAre you using Kaggle / Colab to train these , or are you doing locally? If so , what hardware , and how long does it take you?
\nThank you!

", "rawMarkdown": "Hi, thanks a lot for sharing your experiments , it was really insightful. I have some questions - \nDid you stick to using CAMs approach? Did it work well for you?\nAre you using Kaggle / Colab to train these , or are you doing locally? If so , what hardware , and how long does it take you?\nThank you!", "votes": 2, "replies": [ { "id": 1284085, "postDate": "2021-04-25T14:44:15.567Z", "content": "

Hi,
\nyes my current score is only based on evaluating CAMs of full images, so I guess it's working alright for me. I'm using Colab for training. The important part when working with CAMs is figuring out how to transfer the activations into class probabilities…

", "rawMarkdown": "Hi,\nyes my current score is only based on evaluating CAMs of full images, so I guess it's working alright for me. I'm using Colab for training. The important part when working with CAMs is figuring out how to transfer the activations into class probabilities...", "votes": 3 }, { "id": 1284088, "postDate": "2021-04-25T14:48:38.200Z", "content": "

Thanks a lot. Really appreciate you sharing this.

", "rawMarkdown": "Thanks a lot. Really appreciate you sharing this.", "votes": 1 }, { "id": 1284236, "postDate": "2021-04-25T18:06:34.197Z", "content": "

BTW, wondering do you design your own head to improve CAM resolution?

", "rawMarkdown": "BTW, wondering do you design your own head to improve CAM resolution?" }, { "id": 1284238, "postDate": "2021-04-25T18:09:19.733Z", "content": "

I just put a global average pooling layer to get my cams, 19x19 or whatever a EffNetB4 on about 640 pixel input images produces on its last convolution is actually enough to get the CAMs

", "rawMarkdown": "I just put a global average pooling layer to get my cams, 19x19 or whatever a EffNetB4 on about 640 pixel input images produces on its last convolution is actually enough to get the CAMs" }, { "id": 1284599, "postDate": "2021-04-26T06:00:51.220Z", "content": "

Hi, another newbie question - Are you running the PuzzleCAM code in a Jupyter notebook on Colab, or accessing VSCode or something , as per the official code structure provided by the authors?

", "rawMarkdown": "Hi, another newbie question - Are you running the PuzzleCAM code in a Jupyter notebook on Colab, or accessing VSCode or something , as per the official code structure provided by the authors?", "votes": 1 }, { "id": 1284621, "postDate": "2021-04-26T06:29:30.897Z", "content": "

Im running it in colab. You can just adopt the code from the original Implementation.

\n

Actually im not even sure if Puzzle CAM is so important or if just taking CAMs from normal trained models is enough..

", "rawMarkdown": "Im running it in colab. You can just adopt the code from the original Implementation.\n\nActually im not even sure if Puzzle CAM is so important or if just taking CAMs from normal trained models is enough..", "votes": 1 }, { "id": 1284630, "postDate": "2021-04-26T06:37:36.233Z", "content": "

I see. Interesting. Thank you so much. I will try it out and let you know how it goes!

", "rawMarkdown": "I see. Interesting. Thank you so much. I will try it out and let you know how it goes!", "votes": 1 }, { "id": 1285715, "postDate": "2021-04-27T07:54:02.530Z", "content": "

Hi, I have a few more questions- Is it possible to connect elsewhere , so I can get them clarified if possible?

", "rawMarkdown": "Hi, I have a few more questions- Is it possible to connect elsewhere , so I can get them clarified if possible?", "votes": 1 }, { "id": 1285753, "postDate": "2021-04-27T08:31:05.983Z", "content": "

Hey, I don't know if thats allowed due to Kaggles rules, unless we're in a team. So just ask here and I will answer here too :)

", "rawMarkdown": "Hey, I don't know if thats allowed due to Kaggles rules, unless we're in a team. So just ask here and I will answer here too :)", "votes": 2 }, { "id": 1285798, "postDate": "2021-04-27T09:11:06.770Z", "content": "

Right, my bad , nearly forgot about it.. Here's my questions -

\n
    \n
  1. How are you using the dataset for the images? The dataset classes being used wrap around the VOCDataset Class. What changes did you do to fit it with the data we have?
    2. \n
  2. In the Dataset , a mask is being fetched. What mask is supposed to be used here? Cell or Nuclei?
    3. \n
  3. Any other changes in the code on Git , that were necessary to modify it to run on our data?
    4. \n
\n

That's all for now , thanks alot :)

", "rawMarkdown": "Right, my bad , nearly forgot about it.. Here's my questions -\n1. How are you using the dataset for the images? The dataset classes being used wrap around the VOCDataset Class. What changes did you do to fit it with the data we have?\n2. In the Dataset , a mask is being fetched. What mask is supposed to be used here? Cell or Nuclei?\n3. Any other changes in the code on Git , that were necessary to modify it to run on our data?\n\nThat's all for now , thanks alot :)\n", "votes": 1 }, { "id": 1285832, "postDate": "2021-04-27T09:44:43.913Z", "content": "

You have to write a custom dataloader to feed the training process, but you can use the same training procedure as in the Puzzle-CAM Github repo. You don't need to provide that mask for training the CAM generator but in later stages if you do the whole Puzzle-CAM pipeline.

", "rawMarkdown": "You have to write a custom dataloader to feed the training process, but you can use the same training procedure as in the Puzzle-CAM Github repo. You don't need to provide that mask for training the CAM generator but in later stages if you do the whole Puzzle-CAM pipeline. " }, { "id": 1285840, "postDate": "2021-04-27T09:51:59.073Z", "content": "

I see , thank you. For now , I am only trying to implement training the classification model , and generating CAMs with puzzle. So I won't need masks for that right?
\nEdit - The datasets used to validate/ test:
\n train_dataset_for_seg = VOC_Dataset_For_Testing_CAM(args.data_dir, 'train', test_transform)\n valid_dataset_for_seg = VOC_Dataset_For_Testing_CAM(args.data_dir, 'val', test_transform)

\n

have with_mask = true. Won't I have to provide the masks here? In my custom dataset , I will just load the images with labels and pass them to a dataloader? Is that correct?

", "rawMarkdown": "I see , thank you. For now , I am only trying to implement training the classification model , and generating CAMs with puzzle. So I won't need masks for that right?\nEdit - The datasets used to validate/ test:\n `train_dataset_for_seg = VOC_Dataset_For_Testing_CAM(args.data_dir, 'train', test_transform)\n valid_dataset_for_seg = VOC_Dataset_For_Testing_CAM(args.data_dir, 'val', test_transform)`\n\nhave with_mask = true. Won't I have to provide the masks here? In my custom dataset , I will just load the images with labels and pass them to a dataloader? Is that correct?\n", "votes": 1 }, { "id": 1285898, "postDate": "2021-04-27T11:00:32.487Z", "content": "

Right, you only return the images and the labels from your dataset. For training you can almost use the standard Puzzle Github code but for evaluating, the code in the repo evaluates on masks, so you just write your own evaluation function (https://github.com/OFRIN/PuzzleCAM/blob/master/train_classification_with_puzzle.py#L241)

", "rawMarkdown": "Right, you only return the images and the labels from your dataset. For training you can almost use the standard Puzzle Github code but for evaluating, the code in the repo evaluates on masks, so you just write your own evaluation function (https://github.com/OFRIN/PuzzleCAM/blob/master/train_classification_with_puzzle.py#L241)", "votes": 1 }, { "id": 1285903, "postDate": "2021-04-27T11:06:18.147Z", "content": "

I see. Makes sense , everything is pretty much clear now. Thank you so much for the help!

", "rawMarkdown": "I see. Makes sense , everything is pretty much clear now. Thank you so much for the help!", "votes": 1 } ] }, { "id": 1260386, "postDate": "2021-04-02T04:43:20.260Z", "content": "

May you share your visual code？ These pictures are beautiful.

", "rawMarkdown": "May you share your visual code？ These pictures are beautiful.", "replies": [ { "id": 1260598, "postDate": "2021-04-02T08:57:23.497Z", "content": "

I guess you're interested in the overlaying part of CAMs and the acutal image?
\nSuppose you have a list with all the CAMs for one image (which is 19) called all_hr_cams_test and a list with all the class names all_label_names_map and the actual image image you can use this code

\n
ncolors = 256\ncolor_array = plt.get_cmap('jet')(range(ncolors))\ncolor_array[:,-1] = np.linspace(0.15,1.0,ncolors)\nmap_object = LinearSegmentedColormap.from_list(name='rainbow_alpha',colors=color_array)\nplt.register_cmap(cmap=map_object)\n\nplt.rcParams['figure.figsize'] = [200, 100]\n\nplt.figure()\nf, axarr = plt.subplots(1,all_hr_cams_test.shape[0])\n\nfor i, cam in enumerate(all_hr_cams_test): \n  axarr[i].imshow(np.array(image)[:, :, 0:3], interpolation=\"bicubic\")\n  axarr[i].set_title(str(all_label_names_map[i]))\n  axarr[i].imshow(cam.squeeze(), cmap=\"rainbow_alpha\",alpha=0.9)\n\nplt.show()\n
\n

giving you all these CAMs overlayed on the cell image \"\"

", "rawMarkdown": "I guess you're interested in the overlaying part of CAMs and the acutal image?\nSuppose you have a list with all the CAMs for one image (which is 19) called `all_hr_cams_test` and a list with all the class names `all_label_names_map` and the actual image `image` you can use this code\n\n\n```\nncolors = 256\ncolor_array = plt.get_cmap('jet')(range(ncolors))\ncolor_array[:,-1] = np.linspace(0.15,1.0,ncolors)\nmap_object = LinearSegmentedColormap.from_list(name='rainbow_alpha',colors=color_array)\nplt.register_cmap(cmap=map_object)\n\nplt.rcParams['figure.figsize'] = [200, 100]\n\nplt.figure()\nf, axarr = plt.subplots(1,all_hr_cams_test.shape[0])\n\nfor i, cam in enumerate(all_hr_cams_test): \n axarr[i].imshow(np.array(image)[:, :, 0:3], interpolation=\"bicubic\")\n axarr[i].set_title(str(all_label_names_map[i]))\n axarr[i].imshow(cam.squeeze(), cmap=\"rainbow_alpha\",alpha=0.9)\n\nplt.show()\n```\n\ngiving you all these CAMs overlayed on the cell image ![](https://s16.directupload.net/images/210402/9lawibuq.jpg)", "votes": 3 }, { "id": 1260675, "postDate": "2021-04-02T10:34:46.143Z", "content": "

Thanks! I ploted my CAM, very nice.
\n\"\"

", "rawMarkdown": "Thanks! I ploted my CAM, very nice.\n![](https://z3.ax1x.com/2021/04/02/cmQxtU.jpg)\n", "votes": 2 }, { "id": 1260683, "postDate": "2021-04-02T10:41:51.493Z", "content": "

beautiful 😊

", "rawMarkdown": "beautiful 😊" } ] }, { "id": 1254759, "postDate": "2021-03-28T05:03:56.160Z", "content": "

One naive question, I am trying to understand what is the hr_cams in the original code refer to. Could you share your insights?

", "rawMarkdown": "One naive question, I am trying to understand what is the [hr_cams](https://github.com/OFRIN/PuzzleCAM/blob/master/inference_classification.py#L170) in the original code refer to. Could you share your insights?", "replies": [ { "id": 1254908, "postDate": "2021-03-28T08:16:23.797Z", "content": "

I guess hr means high resolution here as the cams are upscaled to input image size

", "rawMarkdown": "I guess hr means high resolution here as the cams are upscaled to input image size", "votes": 1 }, { "id": 1255314, "postDate": "2021-03-28T17:11:46.093Z", "content": "

Thanks another naive question, why the author use 4 in strided_size = get_strided_size((ori_h, ori_w), 4) and 16 in strided_up_size = get_strided_up_size((ori_h, ori_w), 16) I am not sure if I fully understand what are those parameters for (especially the 4)

", "rawMarkdown": "Thanks another naive question, why the author use 4 in `strided_size = get_strided_size((ori_h, ori_w), 4)` and 16 in `strided_up_size = get_strided_up_size((ori_h, ori_w), 16)` I am not sure if I fully understand what are those parameters for (especially the 4)" }, { "id": 1255333, "postDate": "2021-03-28T17:27:00.827Z", "content": "

I think you can clearly make it up yourself if you check out the output sizes. If you input a (640, 640) image, the strided_size is (160, 160) and the strided_up_size is (640, 640) again

", "rawMarkdown": "I think you can clearly make it up yourself if you check out the output sizes. If you input a `(640, 640)` image, the `strided_size ` is `(160, 160)` and the `strided_up_size ` is `(640, 640)` again" }, { "id": 1255635, "postDate": "2021-03-29T02:22:49.500Z", "content": "

Also wondering why the original author uses multiple scales? Thx

", "rawMarkdown": "Also wondering why the original author uses multiple scales? Thx" }, { "id": 1255830, "postDate": "2021-03-29T08:40:11.753Z", "content": "

I guess it's just TTA, as well as the flipping

", "rawMarkdown": "I guess it's just TTA, as well as the flipping", "votes": 1 } ] }, { "id": 1237124, "postDate": "2021-03-13T19:21:40.327Z", "content": "

Hi! Do you get any success with PuzzleCAM model?

", "rawMarkdown": "Hi! Do you get any success with PuzzleCAM model? ", "replies": [ { "id": 1237126, "postDate": "2021-03-13T19:24:35.060Z", "content": "

yes i think so, gut my current baseline score using Puzzle-CAM. I'm now starting to work on RGBY models, increasing sizes, ensembling, TTA and so on…

", "rawMarkdown": "yes i think so, gut my current baseline score using Puzzle-CAM. I'm now starting to work on RGBY models, increasing sizes, ensembling, TTA and so on..." }, { "id": 1237129, "postDate": "2021-03-13T19:33:03.447Z", "content": "

Glad to hear that. Did you do any modifications to original code, or any other tricks?

", "rawMarkdown": "Glad to hear that. Did you do any modifications to original code, or any other tricks? ", "replies": [ { "id": 1237131, "postDate": "2021-03-13T19:42:40.670Z", "content": "

For now I didn't modify the training actually. What I did for inferencing is

\n\n

Does this make sense to you?

\n

I'm actually struggling with this process and have tried out many things to somehow get confidence values for all classes per cell, and this was the way it's working the best for now..

", "rawMarkdown": "For now I didn't modify the training actually. What I did for inferencing is\n- multiplying the generated CAMs with the segmentation masks\n- multiplying these values with the actual class probabilities the model outputs (which results in the \"weighted class activation area per cell\" as I would call it\n- standardizing these values across the classes for each cell\n- taking the sigmoid of this standardized \"weighted class activation area per cell\" to get the class probabilities per cell\n\nDoes this make sense to you?\n\nI'm actually struggling with this process and have tried out many things to somehow get confidence values for all classes per cell, and this was the way it's working the best for now.." } ] }, { "id": 1237135, "postDate": "2021-03-13T19:51:55.997Z", "content": "

The step with standardization is a little bit unclear I would say) Btw, did you train on all images, or only those with one label? And what classification loss did you choose?

", "rawMarkdown": "The step with standardization is a little bit unclear I would say) Btw, did you train on all images, or only those with one label? And what classification loss did you choose?", "replies": [ { "id": 1237140, "postDate": "2021-03-13T19:59:45.820Z", "content": "

I trained on all the images plus these images also https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg no matter how many labels the images have.

\n

I used both the standard MultiLabelSoftMarginLoss of Puzzle-CAM and Focal Loss. I used Focal Loss on ResNest101 and MultiLabelSoftMarginLoss on ResNest50, and ResNest101 is giving me better results so I don't know the influence of the loss function really.

\n

Let me explain you.
\nAfter multiplying the CAM with the Mask and the class probabilities, I get a vector like this (for each cell):
\nMASK: 17 PROB-RAW: [ 720.3165 2.181 51.9225 1.1851 2.4141 0.0694 0.4837 0.4424 \n 0.4627 2.7915 5344.8745 35.4197 0.2577 1.4984\n0.5555 0.0026 6.2667 70.3171 9.2973]

\n

So this cell is likely to be class 0 and 10. To get confidence values from this, I need to calculate a sigmoid, but using this raw input for sigmoid will not make sense. So I standardize the values to:
\nMASK: 17 PROB: [ 0.328 -0.2739 -0.2323 -0.2748 -0.2738 -0.2757 -0.2754 -0.2754 -0.2754\n -0.2734 4.2046 -0.2461 -0.2756 -0.2745 -0.2753 -0.2758 -0.2705 -0.2168\n -0.268 ]

\n

from which I can calculate a sigmoid function giving me the class confidences:
\nMASK: 17 PROB: [0.5637 0.2793 0.2964 0.279 0.2794 0.2786 0.2787 0.2787 0.2787 0.2795\n 0.9997 0.2907 0.2787 0.2791 0.2788 0.2786 0.2807 0.3029 0.2817]

\n

It's still a bit hacky but I haven't found another way to do this for now…

", "rawMarkdown": "I trained on all the images plus these images also https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg no matter how many labels the images have.\n\nI used both the standard MultiLabelSoftMarginLoss of Puzzle-CAM and Focal Loss. I used Focal Loss on ResNest101 and MultiLabelSoftMarginLoss on ResNest50, and ResNest101 is giving me better results so I don't know the influence of the loss function really.\n\nLet me explain you.\nAfter multiplying the CAM with the Mask and the class probabilities, I get a vector like this (for each cell):\n`MASK: 17 PROB-RAW: [ 720.3165 2.181 51.9225 1.1851 2.4141 0.0694 0.4837 0.4424 \n 0.4627 2.7915 5344.8745 35.4197 0.2577 1.4984\n0.5555 0.0026 6.2667 70.3171 9.2973]`\n\nSo this cell is likely to be class 0 and 10. To get confidence values from this, I need to calculate a sigmoid, but using this raw input for sigmoid will not make sense. So I standardize the values to:\n`MASK: 17 PROB: [ 0.328 -0.2739 -0.2323 -0.2748 -0.2738 -0.2757 -0.2754 -0.2754 -0.2754\n -0.2734 4.2046 -0.2461 -0.2756 -0.2745 -0.2753 -0.2758 -0.2705 -0.2168\n -0.268 ]`\n\nfrom which I can calculate a sigmoid function giving me the class confidences:\n`MASK: 17 PROB: [0.5637 0.2793 0.2964 0.279 0.2794 0.2786 0.2787 0.2787 0.2787 0.2795\n 0.9997 0.2907 0.2787 0.2791 0.2788 0.2786 0.2807 0.3029 0.2817]`\n\n\nIt's still a bit hacky but I haven't found another way to do this for now...\n\n\n", "votes": 1 } ] }, { "id": 1237143, "postDate": "2021-03-13T20:03:32.117Z", "content": "

Ah, I see, got it now) Thanks, I'm very glad PuzzleCAM works for you) How do you validate the model during training?

", "rawMarkdown": "Ah, I see, got it now) Thanks, I'm very glad PuzzleCAM works for you) How do you validate the model during training?", "replies": [ { "id": 1237146, "postDate": "2021-03-13T20:05:04.780Z", "content": "

Just using Validation Loss. For the RGBY models I'm now also calculating F1-Micro, F1-Macro and Accuracy for validation

", "rawMarkdown": "Just using Validation Loss. For the RGBY models I'm now also calculating F1-Micro, F1-Macro and Accuracy for validation" } ] }, { "id": 1237279, "postDate": "2021-03-14T01:58:14.493Z", "content": "

Thanks for sharing your results. So, you haven't trained an affinity net on top of that, just pure puzzlecam?

", "rawMarkdown": "Thanks for sharing your results. So, you haven't trained an affinity net on top of that, just pure puzzlecam?", "replies": [ { "id": 1237575, "postDate": "2021-03-14T09:17:04.980Z", "content": "

Right. I thought we might not need the segmentation part at all, as we have the HPASegmentator. But I guess I'll try to go for the complete pipeline as supposed in the Puzzle-CAM paper once I have found my best models. Also I'm not sure if Puzzle is acutally needed or if a Grad-CAM will just do same as good.

\n

A nice thing to work on would be a Puzzle Grad-CAM actually :)

", "rawMarkdown": "Right. I thought we might not need the segmentation part at all, as we have the HPASegmentator. But I guess I'll try to go for the complete pipeline as supposed in the Puzzle-CAM paper once I have found my best models. Also I'm not sure if Puzzle is acutally needed or if a Grad-CAM will just do same as good. \n\nA nice thing to work on would be a Puzzle Grad-CAM actually :)" } ] } ] }, { "id": 1222231, "postDate": "2021-03-01T16:37:57.003Z", "content": "

Hi! I've tried training PuzzleCAM using images with only 1 label. So far got ~0.050 on LB. Maybe I'm doing something wrong during training/inference stages. Could you please release training/inference notebooks if you got some success withh PuzzleCAM please?

", "rawMarkdown": "Hi! I've tried training PuzzleCAM using images with only 1 label. So far got ~0.050 on LB. Maybe I'm doing something wrong during training/inference stages. Could you please release training/inference notebooks if you got some success withh PuzzleCAM please?", "replies": [ { "id": 1222244, "postDate": "2021-03-01T16:46:53.627Z", "content": "

This is best training run so far. With 90/10 train/val split. train_mIoU represents mIoU for validation split.\"\"

", "rawMarkdown": "This is best training run so far. With 90/10 train/val split. train_mIoU represents mIoU for validation split.![](https://i.imgur.com/pkB8k89.png)" }, { "id": 1222289, "postDate": "2021-03-01T17:22:15.593Z", "content": "

Whats your pipeline like? Do you use the cams directly and take dot product with the segmentations from HPA Segmentator? Or do you use the full pipeline incl. training a segmentation model?

\n

Your losses look alright I guess, though I don't get where the ground truth for mIoU come from, also from HPA Segmentator?

\n

I evaluate on the validation losses, but haven't build a complete pipeline yet.

", "rawMarkdown": "Whats your pipeline like? Do you use the cams directly and take dot product with the segmentations from HPA Segmentator? Or do you use the full pipeline incl. training a segmentation model?\n\nYour losses look alright I guess, though I don't get where the ground truth for mIoU come from, also from HPA Segmentator?\n\nI evaluate on the validation losses, but haven't build a complete pipeline yet." }, { "id": 1222339, "postDate": "2021-03-01T17:56:51.240Z", "content": "

Pipeline is this code with minor modifications (experimenting with losses, optimizers, augmentations, etc.). During inference stage I use dot product of CAM predictions with HPASegmentator nuclei masks, like @phalanx did (at least I hope I'm doing it right). I didn't use full pipeline for now (AffinityNet and so on).

\n

Regarding ground truth. For the training stage, the GT is (image, label). For the validation the GT is (image, mask, label) where mask comes from HPASegmentator (I've took precomputed masks from here).

\n

I kind of gave up idea to use PuzzleCAM for now, because it scores poor on LB (it might be due to my misunderstanding or mistakes), but maybe I will return to it later, especially if someone clarifies how inference should be done.

", "rawMarkdown": "Pipeline is [this](https://github.com/OFRIN/PuzzleCAM/blob/master/train_classification_with_puzzle.py) code with minor modifications (experimenting with losses, optimizers, augmentations, etc.). During inference stage I use dot product of CAM predictions with HPASegmentator nuclei masks, like @phalanx did (at least I hope I'm doing it right). I didn't use full pipeline for now (AffinityNet and so on).\n\nRegarding ground truth. For the training stage, the GT is (image, label). For the validation the GT is (image, mask, label) where mask comes from HPASegmentator (I've took precomputed masks from [here](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/215773)).\n\nI kind of gave up idea to use PuzzleCAM for now, because it scores poor on LB (it might be due to my misunderstanding or mistakes), but maybe I will return to it later, especially if someone clarifies how inference should be done.", "votes": 1 } ] }, { "id": 1219695, "postDate": "2021-02-27T06:18:49.723Z", "content": "

Wondering what we should do after get the CAM and the masks from segmenter

", "rawMarkdown": "Wondering what we should do after get the CAM and the masks from segmenter", "replies": [ { "id": 1219799, "postDate": "2021-02-27T08:59:09.147Z", "content": "

Multiply the masks from segmenter with the cam of every category = class probabilities for every segmented mask

\n

..at least i hope this works out :)
\nfor now I'm trying to go the whole way with the Puzzle-CAM: training an AffinityNet and a segmentation model and see if the complete end-to-end is going to work or if I have to use the HPA Segmenter

", "rawMarkdown": "Multiply the masks from segmenter with the cam of every category = class probabilities for every segmented mask\n\n..at least i hope this works out :)\nfor now I'm trying to go the whole way with the Puzzle-CAM: training an AffinityNet and a segmentation model and see if the complete end-to-end is going to work or if I have to use the HPA Segmenter", "votes": 1 }, { "id": 1220346, "postDate": "2021-02-27T22:06:06.403Z", "content": "

I am working on the same using TensorFlow. I believe it to be useful.

", "rawMarkdown": "I am working on the same using TensorFlow. I believe it to be useful.", "votes": 1 } ] }, { "id": 1218073, "postDate": "2021-02-25T14:50:09.587Z", "content": "

Hi, I do.. bit by bit :) I use ResNeSt50 backbone, 10 epoch.

\n

image_id = '08b80082-bba8-11e8-b2ba-ac1f6b6435d0'\ncam = np.load(f'{cam_dir}/{image_id}.npy', allow_pickle=True).item()\ncell_mask = np.load(f'{cell_mask_dir}/{image_id}.npz')['arr_0']\nprint_masked_img(image_id, cell_mask, 'train')\nplt.imshow(cam['hr_cam'].mean(axis=0))

\n

\"hpa\"

\n

and CAM averaged over channels

\n

\"hpa_cam\"

", "rawMarkdown": "Hi, I do.. bit by bit :) I use ResNeSt50 backbone, 10 epoch.\n\n`image_id = '08b80082-bba8-11e8-b2ba-ac1f6b6435d0'\ncam = np.load(f'{cam_dir}/{image_id}.npy', allow_pickle=True).item()\ncell_mask = np.load(f'{cell_mask_dir}/{image_id}.npz')['arr_0']\nprint_masked_img(image_id, cell_mask, 'train')\nplt.imshow(cam['hr_cam'].mean(axis=0))`\n\n![hpa](https://ibb.co/V3Sqprs)\n\nand CAM averaged over channels\n\n![hpa_cam](https://ibb.co/nPhxrNc)\n\n", "replies": [ { "id": 1218098, "postDate": "2021-02-25T15:01:38.083Z", "content": "

OK nice! For which class or classes is this your CAM?
\nDo you use any special augmentation techniques for training? I not sure yet whether to just resize or to random crop from like 1024x1024 images.
\nAre you using all the channels or only RGB?

", "rawMarkdown": "OK nice! For which class or classes is this your CAM?\nDo you use any special augmentation techniques for training? I not sure yet whether to just resize or to random crop from like 1024x1024 images. \nAre you using all the channels or only RGB? " }, { "id": 1218123, "postDate": "2021-02-25T15:23:19.227Z", "content": "

Nothing special, just as proposed in a github for now. Extended existing augmentation with top/bottom flip. I've resized original images to 512x512, made 4 puzzle pieces. And still it takes >1 hour to train one epoch on 4 10th generation Nvidia GPUs :)
\nIt has min/max random resize, so I think eventually it comes to a different size cells while training, so I decided to resize, not to crop.
\nWhat is strange but I haven't deep dived - why mIoU metric is so low. Here are two last training epochs

\n
\n

Epoch 8: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1022/1022 [1:09:24<00:00, 4.07s/it, total loss:0.2796 class loss full/tiled:0.1269/0.1232 reconstruction loss:0.0821 mIoU curr/best:4.88%/4.90% threshold:0.10]
\n Epoch 9: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1022/1022 [1:09:25<00:00, 4.08s/it, total loss:0.2789 class loss full/tiled:0.1269/0.1229 reconstruction loss:0.0797 mIoU curr/best:4.89%/4.90% threshold:0.10]

\n
", "rawMarkdown": "Nothing special, just as proposed in a github for now. Extended existing augmentation with top/bottom flip. I've resized original images to 512x512, made 4 puzzle pieces. And still it takes >1 hour to train one epoch on 4 10th generation Nvidia GPUs :) \nIt has min/max random resize, so I think eventually it comes to a different size cells while training, so I decided to resize, not to crop. \nWhat is strange but I haven't deep dived - why mIoU metric is so low. Here are two last training epochs\n\n> Epoch 8: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1022/1022 [1:09:24<00:00, 4.07s/it, total loss:0.2796 class loss full/tiled:0.1269/0.1232 reconstruction loss:0.0821 mIoU curr/best:4.88%/4.90% threshold:0.10]\nEpoch 9: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1022/1022 [1:09:25<00:00, 4.08s/it, total loss:0.2789 class loss full/tiled:0.1269/0.1229 reconstruction loss:0.0797 mIoU curr/best:4.89%/4.90% threshold:0.10]\n\n", "replies": [ { "id": 1218296, "postDate": "2021-02-25T17:41:12.563Z", "content": "

You need a ground truth to calculate mIoU, where do you get that from?

", "rawMarkdown": "You need a ground truth to calculate mIoU, where do you get that from?" } ] }, { "id": 1218819, "postDate": "2021-02-26T08:16:11.167Z", "content": "

Everything comes from here https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg. It is not actual ground truth, but it gives feeling on how things are going on.

", "rawMarkdown": "Everything comes from here https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg. It is not actual ground truth, but it gives feeling on how things are going on." }, { "id": 1222237, "postDate": "2021-03-01T16:39:42.147Z", "content": "

I'm sorry, can't see the images you've attached. Could you share the results somehow again? Thanks!

", "rawMarkdown": "I'm sorry, can't see the images you've attached. Could you share the results somehow again? Thanks!" }, { "id": 1222295, "postDate": "2021-03-01T17:27:16.063Z", "content": "

Kaggle has complicated images addition mechanism :)
\nNothing special with my images, they are more or less the same like Alexander's.
\nLinks: https://ibb.co/V3Sqprs and https://ibb.co/nPhxrNc

", "rawMarkdown": "Kaggle has complicated images addition mechanism :) \nNothing special with my images, they are more or less the same like Alexander's. \nLinks: https://ibb.co/V3Sqprs and https://ibb.co/nPhxrNc", "votes": 1 }, { "id": 1222343, "postDate": "2021-03-01T17:58:28.177Z", "content": "

I see, thanks)

", "rawMarkdown": "I see, thanks)" } ] }, { "id": 1250257, "postDate": "2021-03-23T21:52:18.780Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1237865, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-14T13:37:38.407000", "content": "

I've upgraded my inference code, now instead of argmax on CAMs, I'm doing the following:

\n
CAMs = get_hr_cams(rgb_image, th=0.1)[...,1:]\n\nfor class_id in range(len(classes)):\n    for cell_idx in range(1, cell_mask.max()+1):\n        result = CAMs[...,class_id]*(nuclei_mask==cell_idx)\n        num_pixels = np.count_nonzero(result)\n        if num_pixels:\n            prob = result.sum()/num_pixels\n
\n

This way you don't need to standardize or sigmoid anything. You always get prob in a range [0,1]. Now it scores 0.15 on LB, which is 3 times better than with argmax before.

", "votes": 3, "replies": [ { "id": 1237913, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-14T13:55:06.470000", "content": "

ok so you basically divide the \"class-pixels\" for each cell by the size of the cell? Is this it?

", "votes": 0, "replies": [] }, { "id": 1237926, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-14T13:58:38.553000", "content": "

yes, I guess it's the most straightforward way to get confidence for the cell.

", "votes": 1, "replies": [ { "id": 1237940, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-14T14:09:56.963000", "content": "

Yes I understand. I tried to do the same but my intuition was, that for some classes, the area of activation is very small (compared to the whole cell), like the centrosome or nuclear bodies and some others. If I devide them by the full cell size, the activation will vanish maybe for the small classes.

", "votes": 0, "replies": [] } ] }, { "id": 1238777, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-15T09:37:49.227000", "content": "

But I don't divide by the full cell size. I get intersection between CAM for the class and nuclei_mask corresponding to this CAM. And only from this area of intersection I calculate the confidence.

", "votes": 1, "replies": [] }, { "id": 1247531, "author_name": "Strideradu", "author_url": "", "post_date": "2021-03-21T19:54:20.203000", "content": "

I feel this is the right way as CAM was trained with avg pooling, so you basically get the avg pooling for specif nuclei in this operation

", "votes": 0, "replies": [] }, { "id": 1254752, "author_name": "Strideradu", "author_url": "", "post_date": "2021-03-28T04:40:31.937000", "content": "

@vostankovich wondering how do you find the best threshold?

", "votes": 0, "replies": [] }, { "id": 1254989, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-28T10:07:04.817000", "content": "

I think that threshold didn't change anything (it applies only to background). I just took it as it was from original repo.

", "votes": 1, "replies": [] }, { "id": 1254991, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-28T10:10:11.227000", "content": "

@vostankovich Are you still using Puzzle-CAM / CAMs in general for your LB Score? πŸ™‚
\nI'm going to start working on getting CAMs from Vision Transformers for now :)

", "votes": 1, "replies": [] }, { "id": 1260983, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-04-02T15:23:58.250000", "content": "

Oh, sorry for late reply. I gave up using CAMs approach long time ago. Don't know why but I didn't believe it could lead to high score.

", "votes": 1, "replies": [] } ] }, { "id": 1283998, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-25T13:24:57.487000", "content": "

Hi, thanks a lot for sharing your experiments , it was really insightful. I have some questions -
\nDid you stick to using CAMs approach? Did it work well for you?
\nAre you using Kaggle / Colab to train these , or are you doing locally? If so , what hardware , and how long does it take you?
\nThank you!

", "votes": 2, "replies": [ { "id": 1284085, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-25T14:44:15.567000", "content": "

Hi,
\nyes my current score is only based on evaluating CAMs of full images, so I guess it's working alright for me. I'm using Colab for training. The important part when working with CAMs is figuring out how to transfer the activations into class probabilities…

", "votes": 3, "replies": [] }, { "id": 1284088, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-25T14:48:38.200000", "content": "

Thanks a lot. Really appreciate you sharing this.

", "votes": 1, "replies": [] }, { "id": 1284236, "author_name": "Strideradu", "author_url": "", "post_date": "2021-04-25T18:06:34.197000", "content": "

BTW, wondering do you design your own head to improve CAM resolution?

", "votes": 0, "replies": [] }, { "id": 1284238, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-25T18:09:19.733000", "content": "

I just put a global average pooling layer to get my cams, 19x19 or whatever a EffNetB4 on about 640 pixel input images produces on its last convolution is actually enough to get the CAMs

", "votes": 0, "replies": [] }, { "id": 1284599, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-26T06:00:51.220000", "content": "

Hi, another newbie question - Are you running the PuzzleCAM code in a Jupyter notebook on Colab, or accessing VSCode or something , as per the official code structure provided by the authors?

", "votes": 1, "replies": [] }, { "id": 1284621, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-26T06:29:30.897000", "content": "

Im running it in colab. You can just adopt the code from the original Implementation.

\n

Actually im not even sure if Puzzle CAM is so important or if just taking CAMs from normal trained models is enough..

", "votes": 1, "replies": [] }, { "id": 1284630, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-26T06:37:36.233000", "content": "

I see. Interesting. Thank you so much. I will try it out and let you know how it goes!

", "votes": 1, "replies": [] }, { "id": 1285715, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-27T07:54:02.530000", "content": "

Hi, I have a few more questions- Is it possible to connect elsewhere , so I can get them clarified if possible?

", "votes": 1, "replies": [] }, { "id": 1285753, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-27T08:31:05.983000", "content": "

Hey, I don't know if thats allowed due to Kaggles rules, unless we're in a team. So just ask here and I will answer here too :)

", "votes": 2, "replies": [] }, { "id": 1285798, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-27T09:11:06.770000", "content": "

Right, my bad , nearly forgot about it.. Here's my questions -

\n
    \n
  1. How are you using the dataset for the images? The dataset classes being used wrap around the VOCDataset Class. What changes did you do to fit it with the data we have?
    2. \n
  2. In the Dataset , a mask is being fetched. What mask is supposed to be used here? Cell or Nuclei?
    3. \n
  3. Any other changes in the code on Git , that were necessary to modify it to run on our data?
    4. \n
\n

That's all for now , thanks alot :)

", "votes": 1, "replies": [] }, { "id": 1285832, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-27T09:44:43.913000", "content": "

You have to write a custom dataloader to feed the training process, but you can use the same training procedure as in the Puzzle-CAM Github repo. You don't need to provide that mask for training the CAM generator but in later stages if you do the whole Puzzle-CAM pipeline.

", "votes": 0, "replies": [] }, { "id": 1285840, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-27T09:51:59.073000", "content": "

I see , thank you. For now , I am only trying to implement training the classification model , and generating CAMs with puzzle. So I won't need masks for that right?
\nEdit - The datasets used to validate/ test:
\n train_dataset_for_seg = VOC_Dataset_For_Testing_CAM(args.data_dir, 'train', test_transform)\n valid_dataset_for_seg = VOC_Dataset_For_Testing_CAM(args.data_dir, 'val', test_transform)

\n

have with_mask = true. Won't I have to provide the masks here? In my custom dataset , I will just load the images with labels and pass them to a dataloader? Is that correct?

", "votes": 1, "replies": [] }, { "id": 1285898, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-27T11:00:32.487000", "content": "

Right, you only return the images and the labels from your dataset. For training you can almost use the standard Puzzle Github code but for evaluating, the code in the repo evaluates on masks, so you just write your own evaluation function (https://github.com/OFRIN/PuzzleCAM/blob/master/train_classification_with_puzzle.py#L241)

", "votes": 1, "replies": [] }, { "id": 1285903, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-27T11:06:18.147000", "content": "

I see. Makes sense , everything is pretty much clear now. Thank you so much for the help!

", "votes": 1, "replies": [] } ] }, { "id": 1260386, "author_name": "Correlation", "author_url": "", "post_date": "2021-04-02T04:43:20.260000", "content": "

May you share your visual code？ These pictures are beautiful.

", "votes": 0, "replies": [ { "id": 1260598, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-02T08:57:23.497000", "content": "

I guess you're interested in the overlaying part of CAMs and the acutal image?
\nSuppose you have a list with all the CAMs for one image (which is 19) called all_hr_cams_test and a list with all the class names all_label_names_map and the actual image image you can use this code

\n
ncolors = 256\ncolor_array = plt.get_cmap('jet')(range(ncolors))\ncolor_array[:,-1] = np.linspace(0.15,1.0,ncolors)\nmap_object = LinearSegmentedColormap.from_list(name='rainbow_alpha',colors=color_array)\nplt.register_cmap(cmap=map_object)\n\nplt.rcParams['figure.figsize'] = [200, 100]\n\nplt.figure()\nf, axarr = plt.subplots(1,all_hr_cams_test.shape[0])\n\nfor i, cam in enumerate(all_hr_cams_test): \n  axarr[i].imshow(np.array(image)[:, :, 0:3], interpolation=\"bicubic\")\n  axarr[i].set_title(str(all_label_names_map[i]))\n  axarr[i].imshow(cam.squeeze(), cmap=\"rainbow_alpha\",alpha=0.9)\n\nplt.show()\n
\n

giving you all these CAMs overlayed on the cell image \"\"

", "votes": 3, "replies": [] }, { "id": 1260675, "author_name": "Correlation", "author_url": "", "post_date": "2021-04-02T10:34:46.143000", "content": "

Thanks! I ploted my CAM, very nice.
\n\"\"

", "votes": 2, "replies": [] }, { "id": 1260683, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-04-02T10:41:51.493000", "content": "

beautiful 😊

", "votes": 0, "replies": [] } ] }, { "id": 1254759, "author_name": "Strideradu", "author_url": "", "post_date": "2021-03-28T05:03:56.160000", "content": "

One naive question, I am trying to understand what is the hr_cams in the original code refer to. Could you share your insights?

", "votes": 0, "replies": [ { "id": 1254908, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-28T08:16:23.797000", "content": "

I guess hr means high resolution here as the cams are upscaled to input image size

", "votes": 1, "replies": [] }, { "id": 1255314, "author_name": "Strideradu", "author_url": "", "post_date": "2021-03-28T17:11:46.093000", "content": "

Thanks another naive question, why the author use 4 in strided_size = get_strided_size((ori_h, ori_w), 4) and 16 in strided_up_size = get_strided_up_size((ori_h, ori_w), 16) I am not sure if I fully understand what are those parameters for (especially the 4)

", "votes": 0, "replies": [] }, { "id": 1255333, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-28T17:27:00.827000", "content": "

I think you can clearly make it up yourself if you check out the output sizes. If you input a (640, 640) image, the strided_size is (160, 160) and the strided_up_size is (640, 640) again

", "votes": 0, "replies": [] }, { "id": 1255635, "author_name": "Strideradu", "author_url": "", "post_date": "2021-03-29T02:22:49.500000", "content": "

Also wondering why the original author uses multiple scales? Thx

", "votes": 0, "replies": [] }, { "id": 1255830, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-29T08:40:11.753000", "content": "

I guess it's just TTA, as well as the flipping

", "votes": 1, "replies": [] } ] }, { "id": 1237124, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-13T19:21:40.327000", "content": "

Hi! Do you get any success with PuzzleCAM model?

", "votes": 0, "replies": [ { "id": 1237126, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-13T19:24:35.060000", "content": "

yes i think so, gut my current baseline score using Puzzle-CAM. I'm now starting to work on RGBY models, increasing sizes, ensembling, TTA and so on…

", "votes": 0, "replies": [] }, { "id": 1237129, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-13T19:33:03.447000", "content": "

Glad to hear that. Did you do any modifications to original code, or any other tricks?

", "votes": 0, "replies": [ { "id": 1237131, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-13T19:42:40.670000", "content": "

For now I didn't modify the training actually. What I did for inferencing is

\n\n

Does this make sense to you?

\n

I'm actually struggling with this process and have tried out many things to somehow get confidence values for all classes per cell, and this was the way it's working the best for now..

", "votes": 0, "replies": [] } ] }, { "id": 1237135, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-13T19:51:55.997000", "content": "

The step with standardization is a little bit unclear I would say) Btw, did you train on all images, or only those with one label? And what classification loss did you choose?

", "votes": 0, "replies": [ { "id": 1237140, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-13T19:59:45.820000", "content": "

I trained on all the images plus these images also https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg no matter how many labels the images have.

\n

I used both the standard MultiLabelSoftMarginLoss of Puzzle-CAM and Focal Loss. I used Focal Loss on ResNest101 and MultiLabelSoftMarginLoss on ResNest50, and ResNest101 is giving me better results so I don't know the influence of the loss function really.

\n

Let me explain you.
\nAfter multiplying the CAM with the Mask and the class probabilities, I get a vector like this (for each cell):
\nMASK: 17 PROB-RAW: [ 720.3165 2.181 51.9225 1.1851 2.4141 0.0694 0.4837 0.4424 \n 0.4627 2.7915 5344.8745 35.4197 0.2577 1.4984\n0.5555 0.0026 6.2667 70.3171 9.2973]

\n

So this cell is likely to be class 0 and 10. To get confidence values from this, I need to calculate a sigmoid, but using this raw input for sigmoid will not make sense. So I standardize the values to:
\nMASK: 17 PROB: [ 0.328 -0.2739 -0.2323 -0.2748 -0.2738 -0.2757 -0.2754 -0.2754 -0.2754\n -0.2734 4.2046 -0.2461 -0.2756 -0.2745 -0.2753 -0.2758 -0.2705 -0.2168\n -0.268 ]

\n

from which I can calculate a sigmoid function giving me the class confidences:
\nMASK: 17 PROB: [0.5637 0.2793 0.2964 0.279 0.2794 0.2786 0.2787 0.2787 0.2787 0.2795\n 0.9997 0.2907 0.2787 0.2791 0.2788 0.2786 0.2807 0.3029 0.2817]

\n

It's still a bit hacky but I haven't found another way to do this for now…

", "votes": 1, "replies": [] } ] }, { "id": 1237143, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-13T20:03:32.117000", "content": "

Ah, I see, got it now) Thanks, I'm very glad PuzzleCAM works for you) How do you validate the model during training?

", "votes": 0, "replies": [ { "id": 1237146, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-13T20:05:04.780000", "content": "

Just using Validation Loss. For the RGBY models I'm now also calculating F1-Micro, F1-Macro and Accuracy for validation

", "votes": 0, "replies": [] } ] }, { "id": 1237279, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2021-03-14T01:58:14.493000", "content": "

Thanks for sharing your results. So, you haven't trained an affinity net on top of that, just pure puzzlecam?

", "votes": 0, "replies": [ { "id": 1237575, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-14T09:17:04.980000", "content": "

Right. I thought we might not need the segmentation part at all, as we have the HPASegmentator. But I guess I'll try to go for the complete pipeline as supposed in the Puzzle-CAM paper once I have found my best models. Also I'm not sure if Puzzle is acutally needed or if a Grad-CAM will just do same as good.

\n

A nice thing to work on would be a Puzzle Grad-CAM actually :)

", "votes": 0, "replies": [] } ] } ] }, { "id": 1222231, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-01T16:37:57.003000", "content": "

Hi! I've tried training PuzzleCAM using images with only 1 label. So far got ~0.050 on LB. Maybe I'm doing something wrong during training/inference stages. Could you please release training/inference notebooks if you got some success withh PuzzleCAM please?

", "votes": 0, "replies": [ { "id": 1222244, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-01T16:46:53.627000", "content": "

This is best training run so far. With 90/10 train/val split. train_mIoU represents mIoU for validation split.\"\"

", "votes": 0, "replies": [] }, { "id": 1222289, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-03-01T17:22:15.593000", "content": "

Whats your pipeline like? Do you use the cams directly and take dot product with the segmentations from HPA Segmentator? Or do you use the full pipeline incl. training a segmentation model?

\n

Your losses look alright I guess, though I don't get where the ground truth for mIoU come from, also from HPA Segmentator?

\n

I evaluate on the validation losses, but haven't build a complete pipeline yet.

", "votes": 0, "replies": [] }, { "id": 1222339, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-01T17:56:51.240000", "content": "

Pipeline is this code with minor modifications (experimenting with losses, optimizers, augmentations, etc.). During inference stage I use dot product of CAM predictions with HPASegmentator nuclei masks, like @phalanx did (at least I hope I'm doing it right). I didn't use full pipeline for now (AffinityNet and so on).

\n

Regarding ground truth. For the training stage, the GT is (image, label). For the validation the GT is (image, mask, label) where mask comes from HPASegmentator (I've took precomputed masks from here).

\n

I kind of gave up idea to use PuzzleCAM for now, because it scores poor on LB (it might be due to my misunderstanding or mistakes), but maybe I will return to it later, especially if someone clarifies how inference should be done.

", "votes": 1, "replies": [] } ] }, { "id": 1219695, "author_name": "Strideradu", "author_url": "", "post_date": "2021-02-27T06:18:49.723000", "content": "

Wondering what we should do after get the CAM and the masks from segmenter

", "votes": 0, "replies": [ { "id": 1219799, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-02-27T08:59:09.147000", "content": "

Multiply the masks from segmenter with the cam of every category = class probabilities for every segmented mask

\n

..at least i hope this works out :)
\nfor now I'm trying to go the whole way with the Puzzle-CAM: training an AffinityNet and a segmentation model and see if the complete end-to-end is going to work or if I have to use the HPA Segmenter

", "votes": 1, "replies": [] }, { "id": 1220346, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-02-27T22:06:06.403000", "content": "

I am working on the same using TensorFlow. I believe it to be useful.

", "votes": 1, "replies": [] } ] }, { "id": 1218073, "author_name": "Ubique", "author_url": "", "post_date": "2021-02-25T14:50:09.587000", "content": "

Hi, I do.. bit by bit :) I use ResNeSt50 backbone, 10 epoch.

\n

image_id = '08b80082-bba8-11e8-b2ba-ac1f6b6435d0'\ncam = np.load(f'{cam_dir}/{image_id}.npy', allow_pickle=True).item()\ncell_mask = np.load(f'{cell_mask_dir}/{image_id}.npz')['arr_0']\nprint_masked_img(image_id, cell_mask, 'train')\nplt.imshow(cam['hr_cam'].mean(axis=0))

\n

\"hpa\"

\n

and CAM averaged over channels

\n

\"hpa_cam\"

", "votes": 0, "replies": [ { "id": 1218098, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-02-25T15:01:38.083000", "content": "

OK nice! For which class or classes is this your CAM?
\nDo you use any special augmentation techniques for training? I not sure yet whether to just resize or to random crop from like 1024x1024 images.
\nAre you using all the channels or only RGB?

", "votes": 0, "replies": [] }, { "id": 1218123, "author_name": "Ubique", "author_url": "", "post_date": "2021-02-25T15:23:19.227000", "content": "

Nothing special, just as proposed in a github for now. Extended existing augmentation with top/bottom flip. I've resized original images to 512x512, made 4 puzzle pieces. And still it takes >1 hour to train one epoch on 4 10th generation Nvidia GPUs :)
\nIt has min/max random resize, so I think eventually it comes to a different size cells while training, so I decided to resize, not to crop.
\nWhat is strange but I haven't deep dived - why mIoU metric is so low. Here are two last training epochs

\n
\n

Epoch 8: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1022/1022 [1:09:24<00:00, 4.07s/it, total loss:0.2796 class loss full/tiled:0.1269/0.1232 reconstruction loss:0.0821 mIoU curr/best:4.88%/4.90% threshold:0.10]
\n Epoch 9: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1022/1022 [1:09:25<00:00, 4.08s/it, total loss:0.2789 class loss full/tiled:0.1269/0.1229 reconstruction loss:0.0797 mIoU curr/best:4.89%/4.90% threshold:0.10]

\n
", "votes": 0, "replies": [ { "id": 1218296, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-02-25T17:41:12.563000", "content": "

You need a ground truth to calculate mIoU, where do you get that from?

", "votes": 0, "replies": [] } ] }, { "id": 1218819, "author_name": "Ubique", "author_url": "", "post_date": "2021-02-26T08:16:11.167000", "content": "

Everything comes from here https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg. It is not actual ground truth, but it gives feeling on how things are going on.

", "votes": 0, "replies": [] }, { "id": 1222237, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-01T16:39:42.147000", "content": "

I'm sorry, can't see the images you've attached. Could you share the results somehow again? Thanks!

", "votes": 0, "replies": [] }, { "id": 1222295, "author_name": "Ubique", "author_url": "", "post_date": "2021-03-01T17:27:16.063000", "content": "

Kaggle has complicated images addition mechanism :)
\nNothing special with my images, they are more or less the same like Alexander's.
\nLinks: https://ibb.co/V3Sqprs and https://ibb.co/nPhxrNc

", "votes": 1, "replies": [] }, { "id": 1222343, "author_name": "Vladislav Ostankovich", "author_url": "", "post_date": "2021-03-01T17:58:28.177000", "content": "

I see, thanks)

", "votes": 0, "replies": [] } ] }, { "id": 1250257, "author_name": "", "author_url": "", "post_date": "2021-03-23T21:52:18.780000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1217725": "Hello!\nIs anyone out there who's also working on a Puzzle-CAM approach (https://github.com/OFRIN/PuzzleCAM) and wants to share their findings, problems, etc. ?\n\nI'm currently training a ResNest50 with that method, using the standard parameters for training and explore the produced CAMs before going on training an AffinityNet and a segmentation model. \n\nIn the end I will use the approach of @phalanx, using the element product of HPA Segmentor predictions and the CAMs (resp. output of the trained segmentation model).\n\nI guess phalanx only trained a model using Puzzle-CAMS to get CAMS without going the extra way of training a segmentation model. \n\nTo show you what it's look so far, some nice activations :)\n\n![](https://i.ibb.co/rZbtVGM/Unbenannt.jpg)\n![](https://i.ibb.co/MZB98bF/Unbenann1.jpg)\n\n", "1237865": "I've upgraded my inference code, now instead of argmax on CAMs, I'm doing the following:\n\n```\nCAMs = get_hr_cams(rgb_image, th=0.1)[...,1:]\n\nfor class_id in range(len(classes)):\n for cell_idx in range(1, cell_mask.max()+1):\n result = CAMs[...,class_id]*(nuclei_mask==cell_idx)\n num_pixels = np.count_nonzero(result)\n if num_pixels:\n prob = result.sum()/num_pixels\n```\nThis way you don't need to standardize or sigmoid anything. You always get prob in a range [0,1]. Now it scores 0.15 on LB, which is 3 times better than with argmax before.", "1283998": "Hi, thanks a lot for sharing your experiments , it was really insightful. I have some questions - \nDid you stick to using CAMs approach? Did it work well for you?\nAre you using Kaggle / Colab to train these , or are you doing locally? If so , what hardware , and how long does it take you?\nThank you!", "1260386": "May you share your visual code？ These pictures are beautiful.", "1254759": "One naive question, I am trying to understand what is the [hr_cams](https://github.com/OFRIN/PuzzleCAM/blob/master/inference_classification.py#L170) in the original code refer to. Could you share your insights?", "1237124": "Hi! Do you get any success with PuzzleCAM model? ", "1222231": "Hi! I've tried training PuzzleCAM using images with only 1 label. So far got ~0.050 on LB. Maybe I'm doing something wrong during training/inference stages. Could you please release training/inference notebooks if you got some success withh PuzzleCAM please?", "1219695": "Wondering what we should do after get the CAM and the masks from segmenter", "1218073": "Hi, I do.. bit by bit :) I use ResNeSt50 backbone, 10 epoch.\n\n`image_id = '08b80082-bba8-11e8-b2ba-ac1f6b6435d0'\ncam = np.load(f'{cam_dir}/{image_id}.npy', allow_pickle=True).item()\ncell_mask = np.load(f'{cell_mask_dir}/{image_id}.npz')['arr_0']\nprint_masked_img(image_id, cell_mask, 'train')\nplt.imshow(cam['hr_cam'].mean(axis=0))`\n\n![hpa](https://ibb.co/V3Sqprs)\n\nand CAM averaged over channels\n\n![hpa_cam](https://ibb.co/nPhxrNc)\n\n", "1250257": "" } }