cell_mask and nuclei_mask. cell_mask get individual cell patches.\n
![\"img\"](\"https://i.imgur.com/fU8hjlb.png\")
For someday now I was working on preparing a single label (images with only one label associated with it) cell level (individual cells are labeled) dataset to train a better image classifier. Here's a quick summary.
\ncell_mask and nuclei_mask. cell_mask get individual cell patches.Kernels used:
\n\nKaggle Dataset:
\n\nThe image below is the label distribution of the single label cell level dataset.
\n![\"img\"](\"https://i.imgur.com/S9BKxvd.png\")
Certainly, there is an insane class imbalance. To mitigate the imbalance to some extent I used HPA Pubic Data. I decided to create separate single label cell level datasets for these classes - Nuclear Membrane, Actin Filaments, Aggresome, Mitotic Spindle, and Negative.
\nKernels used:
\nKaggle Dataset:
\nI used Weights and Biases artifacts to keep my sanity intact. There was a lot of back and forth with the dataset creation process. I used artifacts to keep a track of the .csv files produced. The interactive graph view allowed me to not repeat any step or mess up anything. Here's the graph view of the resulting artifact. (Link to my artifacts page)
![\"img\"](\"https://i.imgur.com/yCD7kJm.png\")
Here's the current distribution of labels. Mitotic Spindle is really scarce and would be hard to model against.
\n![\"img\"](\"https://i.imgur.com/JmBc0vf.png\")
For someday now I was working on preparing a single label (images with only one label associated with it) cell level (individual cells are labeled) dataset to train a better image classifier. Here's a quick summary.
\ncell_mask and nuclei_mask. cell_mask get individual cell patches.Kernels used:
\n\nKaggle Dataset:
\n\nThe image below is the label distribution of the single label cell level dataset.
\n
Certainly, there is an insane class imbalance. To mitigate the imbalance to some extent I used HPA Pubic Data. I decided to create separate single label cell level datasets for these classes - Nuclear Membrane, Actin Filaments, Aggresome, Mitotic Spindle, and Negative.
\nKernels used:
\nKaggle Dataset:
\nI used Weights and Biases artifacts to keep my sanity intact. There was a lot of back and forth with the dataset creation process. I used artifacts to keep a track of the .csv files produced. The interactive graph view allowed me to not repeat any step or mess up anything. Here's the graph view of the resulting artifact. (Link to my artifacts page)
Here's the current distribution of labels. Mitotic Spindle is really scarce and would be hard to model against.
\n
Great work on this! And thank you for sharing your hard work with the rest of the community.
\nI would like to note a couple of things, just to keep in mind when using this:
\nHPACellSeg can in some instances accidentally group some cells together so you might want to double check that the single cell image image only contains a single nuclei. It should mostly be fine, especially since you excluded cells that touched the borders of the image.
In images with a single label, there can be cells that should be considered negative. It's most common of course to see that the pattern is omnipresent but could be worth double checking.
Maybe this was already sanity checked, but figured it was worth mentioning.
", "rawMarkdown": "Great work on this! And thank you for sharing your hard work with the rest of the community.\n\nI would like to note a couple of things, just to keep in mind when using this:\n\n- HPACellSeg can in some instances accidentally group some cells together so you might want to double check that the single cell image image only contains a single nuclei. It should mostly be fine, especially since you excluded cells that touched the borders of the image.\n\n- In images with a single label, there can be cells that should be considered negative. It's most common of course to see that the pattern is omnipresent but could be worth double checking.\n\nMaybe this was already sanity checked, but figured it was worth mentioning.", "votes": 5, "replies": [ { "id": 1258037, "postDate": "2021-03-31T09:40:00.527Z", "content": "Thank you, Casper. These are great pointers that you mentioned.
\nHi, I'm using the single-label-cell-level artifact csv file from this link
\nAre all the files present in the Kaggle dataset?
\nI'm getting empty tensors for quite a few files, and getting proper images for some. It maybe an issue with something else though , I am using GCS path on Kaggle to access it. When I'm loading an image , it loads as an empty tensor , however when I open it in the dataset , it appears to be fine. Probably an issue in my code. Any idea what could be wrong?
", "rawMarkdown": "Hi, I'm using the single-label-cell-level artifact csv file from this [link](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/single_label_cell_level/1552ea5a8576a5b97179/files)\n\nAre all the files present in the Kaggle dataset? \n\nI'm getting empty tensors for quite a few files, and getting proper images for some. It maybe an issue with something else though , I am using GCS path on Kaggle to access it. When I'm loading an image , it loads as an empty tensor , however when I open it in the dataset , it appears to be fine. Probably an issue in my code. Any idea what could be wrong?", "votes": 1, "replies": [ { "id": 1267340, "postDate": "2021-04-08T13:29:04.560Z", "content": "Update - It is working now ,had some leakage in my code. However , training is taking about 1.5 hrs per epoch on TPU. Maybe because it has ~220k small images?
", "rawMarkdown": "Update - It is working now ,had some leakage in my code. However , training is taking about 1.5 hrs per epoch on TPU. Maybe because it has ~220k small images?" }, { "id": 1267417, "postDate": "2021-04-08T14:01:24.967Z", "content": "Yeah, it's gonna take a lot of time and thus it would be better if you undersample classes with a lot of labels and train smaller models on different folds of the dataset.
\nThis is the reason I made available the entire dataset as it is. Since the images have kind of noisy labels it would not hurt to discard images from the majority class. Up to you to come up with strategies. Luck. :)
", "rawMarkdown": "Yeah, it's gonna take a lot of time and thus it would be better if you undersample classes with a lot of labels and train smaller models on different folds of the dataset. \n\nThis is the reason I made available the entire dataset as it is. Since the images have kind of noisy labels it would not hurt to discard images from the majority class. Up to you to come up with strategies. Luck. :)\n\n", "votes": 2 }, { "id": 1267448, "postDate": "2021-04-08T14:18:33.097Z", "content": "Right. I was thinking of making TFRecords of all the 220k files and use them, but I think I’ll try undersampling first. Thanks!
", "rawMarkdown": "Right. I was thinking of making TFRecords of all the 220k files and use them, but I think I’ll try undersampling first. Thanks!" } ] }, { "id": 1257650, "postDate": "2021-03-31T02:34:27.263Z", "content": "Wonderful work. Thank you for doing all of this and making it available to everyone. Very appreciated!!
", "rawMarkdown": "Wonderful work. Thank you for doing all of this and making it available to everyone. Very appreciated!!", "votes": 2, "replies": [ { "id": 1258030, "postDate": "2021-03-31T09:35:18.023Z", "content": "Thank you, Darien.
", "rawMarkdown": "Thank you, Darien. " } ] }, { "id": 1267323, "postDate": "2021-04-08T13:18:56.687Z", "content": "Great work! Thanks a lot! Do you provide any matching between cell and label? May be csv file?
", "rawMarkdown": "Great work! Thanks a lot! Do you provide any matching between cell and label? May be csv file?", "replies": [ { "id": 1267339, "postDate": "2021-04-08T13:28:16.270Z", "content": "I have made a dataset using the csv file for it. Use the file with 'only' in the name :)
", "rawMarkdown": "I have made a [dataset](https://www.kaggle.com/p4rallax/hpasinglelabelcellcsv) using the csv file for it. Use the file with 'only' in the name :)" } ] }, { "id": 1258390, "postDate": "2021-03-31T15:12:11.663Z", "content": "Nice work. Thanks for sharing!
\nHowever , the link : https://www.kaggle.com/ayuraj/hpa-single-label-cell-level-dataset
\ndoesn't seem to work for me. could you please check that?
Can you check it again? My bad it was private.
", "rawMarkdown": "Can you check it again? My bad it was private. " }, { "id": 1258543, "postDate": "2021-03-31T17:43:51.860Z", "content": "Working now , thanks !
", "rawMarkdown": "Working now , thanks !" } ] } ], "comments": [ { "id": 1257998, "author_name": "Casper Winsnes", "author_url": "", "post_date": "2021-03-31T09:12:21.160000", "content": "Great work on this! And thank you for sharing your hard work with the rest of the community.
\nI would like to note a couple of things, just to keep in mind when using this:
\nHPACellSeg can in some instances accidentally group some cells together so you might want to double check that the single cell image image only contains a single nuclei. It should mostly be fine, especially since you excluded cells that touched the borders of the image.
In images with a single label, there can be cells that should be considered negative. It's most common of course to see that the pattern is omnipresent but could be worth double checking.
Maybe this was already sanity checked, but figured it was worth mentioning.
", "votes": 5, "replies": [ { "id": 1258037, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-03-31T09:40:00.527000", "content": "Thank you, Casper. These are great pointers that you mentioned.
\nHi, I'm using the single-label-cell-level artifact csv file from this link
\nAre all the files present in the Kaggle dataset?
\nI'm getting empty tensors for quite a few files, and getting proper images for some. It maybe an issue with something else though , I am using GCS path on Kaggle to access it. When I'm loading an image , it loads as an empty tensor , however when I open it in the dataset , it appears to be fine. Probably an issue in my code. Any idea what could be wrong?
", "votes": 1, "replies": [ { "id": 1267340, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-08T13:29:04.560000", "content": "Update - It is working now ,had some leakage in my code. However , training is taking about 1.5 hrs per epoch on TPU. Maybe because it has ~220k small images?
", "votes": 0, "replies": [] }, { "id": 1267417, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-04-08T14:01:24.967000", "content": "Yeah, it's gonna take a lot of time and thus it would be better if you undersample classes with a lot of labels and train smaller models on different folds of the dataset.
\nThis is the reason I made available the entire dataset as it is. Since the images have kind of noisy labels it would not hurt to discard images from the majority class. Up to you to come up with strategies. Luck. :)
", "votes": 2, "replies": [] }, { "id": 1267448, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-08T14:18:33.097000", "content": "Right. I was thinking of making TFRecords of all the 220k files and use them, but I think I’ll try undersampling first. Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 1257650, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-31T02:34:27.263000", "content": "Wonderful work. Thank you for doing all of this and making it available to everyone. Very appreciated!!
", "votes": 2, "replies": [ { "id": 1258030, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-03-31T09:35:18.023000", "content": "Thank you, Darien.
", "votes": 0, "replies": [] } ] }, { "id": 1267323, "author_name": "NaStroy", "author_url": "", "post_date": "2021-04-08T13:18:56.687000", "content": "Great work! Thanks a lot! Do you provide any matching between cell and label? May be csv file?
", "votes": 0, "replies": [ { "id": 1267339, "author_name": "Satwik", "author_url": "", "post_date": "2021-04-08T13:28:16.270000", "content": "I have made a dataset using the csv file for it. Use the file with 'only' in the name :)
", "votes": 0, "replies": [] } ] }, { "id": 1258390, "author_name": "Satwik", "author_url": "", "post_date": "2021-03-31T15:12:11.663000", "content": "Nice work. Thanks for sharing!
\nHowever , the link : https://www.kaggle.com/ayuraj/hpa-single-label-cell-level-dataset
\ndoesn't seem to work for me. could you please check that?
Can you check it again? My bad it was private.
", "votes": 0, "replies": [] }, { "id": 1258543, "author_name": "Satwik", "author_url": "", "post_date": "2021-03-31T17:43:51.860000", "content": "Working now , thanks !
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1257484": "For someday now I was working on preparing a single label (images with only one label associated with it) cell level (individual cells are labeled) dataset to train a better image classifier. Here's a quick summary.\n\n# Steps\n\n### Cell Level Image Dataset Creation\n* Start with the HPA competition image and using the HPA Segmentation tool segment it to get `cell_mask` and `nuclei_mask`. \n* Using `cell_mask` get individual cell patches.\n* Using `nuclei_mask' get individual nucleus patches. The contour of the nucleus patch is computed. Using this it is determined if the nucleus is bordering the image. I am discarding the cells whose nucleus is bordering the image. **I am assuming that such a cell would not have \"complete\" protein information**. \n* **Note** that I am creating two separate RGB and protein image datasets. The DSViz table shown in the image can give you an idea of my dataset. ([W&B DSViz table link](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/single_label_split_0/34b460a9c6ea265a4e95/files/dataset.table.json))\n\n\nKernels used:\n* [Dataset Versioning for Single Label Dataset](https://www.kaggle.com/ayuraj/dataset-versioning-for-single-label-dataset)\n* [Using W&B DSViz for Cell Level Dataset Creation](https://www.kaggle.com/ayuraj/using-w-b-dsviz-for-cell-level-dataset-creation)\n\nKaggle Dataset:\n* https://www.kaggle.com/ayuraj/hpa-single-label-cell-level-dataset\n\n### Data Distribution\n\nThe image below is the label distribution of the single label cell level dataset.\n\n\n### Extra HPA Public Data\nCertainly, there is an insane class imbalance. To mitigate the imbalance to some extent I used HPA Pubic Data. I decided to create separate single label cell level datasets for these classes - Nuclear Membrane, Actin Filaments, Aggresome, Mitotic Spindle, and Negative. \n\nKernels used:\n* [Cell Level Extra Data Download](https://www.kaggle.com/ayuraj/cell-level-extra-data-download)\n* [Single Label Cell Label Dataset Creation w\\ W&B](https://www.kaggle.com/ayuraj/using-w-b-dsviz-for-cell-level-dataset-creation)\n(Apologies for not documenting them well.)\n\nKaggle Dataset:\n* [HPA: Single Label Public Mitotic Spindle](https://www.kaggle.com/ayuraj/singlelabelpublicmitoticspindle)\n* [HPA: Single Label Public ACTIN FILAMENTS](https://www.kaggle.com/ayuraj/singlelabelpublicactinfilaments)\n* [HPA: Single Label Public Negative](https://www.kaggle.com/ayuraj/singlelabelpublicnegative)\n* [HPA: Single Label Public Nuclear Membrane](https://www.kaggle.com/ayuraj/singlelabelpublicnuclearmembrane)\n* [HPA: Single Label Public aggresome](https://www.kaggle.com/ayuraj/singlelabelpublicaggresome)\n\n### Dataset Version Control and Sanity In-check\nI used Weights and Biases artifacts to keep my sanity intact. There was a lot of back and forth with the dataset creation process. I used artifacts to keep a track of the `.csv` files produced. The interactive graph view allowed me to not repeat any step or mess up anything. Here's the graph view of the resulting artifact. ([Link to my artifacts page](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/slcl_full_dataset/fadaa879407c3c7397b0/graph))\n\n\n\n### Final Data Distribution\nHere's the current distribution of labels. Mitotic Spindle is really scarce and would be hard to model against. \n\n\n# Next Steps\n* I need to create clever splits of the dataset to train with. I will share the splits and my strategy for the same once I have them ready.\n* Train classifiers and evaluate. ", "1257998": "Great work on this! And thank you for sharing your hard work with the rest of the community.\n\nI would like to note a couple of things, just to keep in mind when using this:\n\n- HPACellSeg can in some instances accidentally group some cells together so you might want to double check that the single cell image image only contains a single nuclei. It should mostly be fine, especially since you excluded cells that touched the borders of the image.\n\n- In images with a single label, there can be cells that should be considered negative. It's most common of course to see that the pattern is omnipresent but could be worth double checking.\n\nMaybe this was already sanity checked, but figured it was worth mentioning.", "1267119": "Hi, I'm using the single-label-cell-level artifact csv file from this [link](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/single_label_cell_level/1552ea5a8576a5b97179/files)\n\nAre all the files present in the Kaggle dataset? \n\nI'm getting empty tensors for quite a few files, and getting proper images for some. It maybe an issue with something else though , I am using GCS path on Kaggle to access it. When I'm loading an image , it loads as an empty tensor , however when I open it in the dataset , it appears to be fine. Probably an issue in my code. Any idea what could be wrong?", "1257650": "Wonderful work. Thank you for doing all of this and making it available to everyone. Very appreciated!!", "1267323": "Great work! Thanks a lot! Do you provide any matching between cell and label? May be csv file?", "1258390": "Nice work. Thanks for sharing! \nHowever , the link : https://www.kaggle.com/ayuraj/hpa-single-label-cell-level-dataset\ndoesn't seem to work for me. could you please check that?" } }