{ "id": 220748, "title": "Proteins in Negatives", "url": "/competitions/hpa-single-cell-image-classification/discussion/220748", "author_name": "Raman", "post_date": "2021-02-19T11:42:32.537000", "votes": 24, "comment_count": 18, "views": 0, "content": "

Hi all,

\n

I checked negative images. To my surprise, often there're apparent protein patterns in the green channel of images labeled as \"Negative\".

\n

What have the hosts said about the \"Negative\" class?

\n

The competition host prof. Emma Lundberg, @emmalumpan, has kindly pointed out that

\n
\n

The label negative can be considered a \"none-of-the-above\" type of label (it describes that there is no green staining/pattern, i.e. no other label). Hence the label negative will only be present at the image level if all cells show no other green pattern. (post)

\n

If there is no specific green pattern (noisy or no signal) it should be labeled as negative. (post)

\n
\n

The competition host Trang Le, @lnhtrang, in the shared awesome notebook, mentioned that

\n
\n

This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).

\n
\n

What is unclear about \"Negative\" images in the train set?

\n

Unfortunately, I'm confused with some Negative images, e.g.:
\n\"\"
\n\"\"
\n\"\"

\n

I might be wrong, but it seems to me that at least Cytosol contains proteins of interest in the shared images.
\nE.g., here are some reference images with the \"Cytosol\" label:
\n\"\"
\n\"\"
\n\"\"
\n\"\"

\n

Main question

\n

Could please anyone clarify, if

\n
    \n
  1. my understanding is incorrect and in the shared \"Negative\" images no proteins are present in Cytosol (not clearly at least), or
    2. \n
  2. image-level labels are noisy, the shared \"Negative\" images should have been labeled with at least the \"Cytosol\" class, or
    3. \n
  3. when in a single cell there's a clear pattern, e.g. proteins present in Cytosol, together with an additional unspecific pattern, then such cells would be labeled as \"Negative\" in the test set.
    4. \n
\n

Thanks in advance!

", "messages": [ { "id": 1210382, "postDate": "2021-02-19T11:42:32.537Z", "content": "

Hi all,

\n

I checked negative images. To my surprise, often there're apparent protein patterns in the green channel of images labeled as \"Negative\".

\n

What have the hosts said about the \"Negative\" class?

\n

The competition host prof. Emma Lundberg, @emmalumpan, has kindly pointed out that

\n
\n

The label negative can be considered a \"none-of-the-above\" type of label (it describes that there is no green staining/pattern, i.e. no other label). Hence the label negative will only be present at the image level if all cells show no other green pattern. (post)

\n

If there is no specific green pattern (noisy or no signal) it should be labeled as negative. (post)

\n
\n

The competition host Trang Le, @lnhtrang, in the shared awesome notebook, mentioned that

\n
\n

This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).

\n
\n

What is unclear about \"Negative\" images in the train set?

\n

Unfortunately, I'm confused with some Negative images, e.g.:
\n\"\"
\n\"\"
\n\"\"

\n

I might be wrong, but it seems to me that at least Cytosol contains proteins of interest in the shared images.
\nE.g., here are some reference images with the \"Cytosol\" label:
\n\"\"
\n\"\"
\n\"\"
\n\"\"

\n

Main question

\n

Could please anyone clarify, if

\n
    \n
  1. my understanding is incorrect and in the shared \"Negative\" images no proteins are present in Cytosol (not clearly at least), or
    2. \n
  2. image-level labels are noisy, the shared \"Negative\" images should have been labeled with at least the \"Cytosol\" class, or
    3. \n
  3. when in a single cell there's a clear pattern, e.g. proteins present in Cytosol, together with an additional unspecific pattern, then such cells would be labeled as \"Negative\" in the test set.
    4. \n
\n

Thanks in advance!

", "rawMarkdown": "Hi all,\n\nI checked negative images. To my surprise, often there're apparent protein patterns in the green channel of images labeled as \"Negative\".\n\n## What have the hosts said about the \"Negative\" class?\nThe competition host prof. Emma Lundberg, @emmalumpan, has kindly pointed out that\n> The label negative can be considered a \"none-of-the-above\" type of label (it describes that there is no green staining/pattern, i.e. no other label). Hence the label negative will only be present at the image level if all cells show no other green pattern. ([post](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/219211#1200188))\n\n> If there is no specific green pattern (noisy or no signal) it should be labeled as negative. ([post](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/216924#1187646))\n\nThe competition host Trang Le, @lnhtrang, in the shared [awesome notebook](https://www.kaggle.com/lnhtrang/single-cell-patterns), mentioned that\n\n> This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).\n\n## What is unclear about \"Negative\" images in the train set?\nUnfortunately, I'm confused with some Negative images, e.g.:\n![](https://i.ibb.co/DV9vRP8/neg-1.png)\n![](https://i.ibb.co/ggcgZQC/neg-2.png)\n![](https://i.ibb.co/rmRXhtg/neg-3.png)\n\nI might be wrong, but it seems to me that at least Cytosol contains proteins of interest in the shared images. \nE.g., here are some reference images with the \"Cytosol\" label:\n![](https://i.ibb.co/gVnbh0T/cyt-4.png)\n![](https://i.ibb.co/f4vTMMb/cyt-3.png)\n![](https://i.ibb.co/QbzSnzC/cyt-2.png)\n![](https://i.ibb.co/WW7cCz8/cyt-1.png)\n\n## Main question\nCould please anyone clarify, if \n1. my understanding is incorrect and in the shared \"Negative\" images no proteins are present in Cytosol (not clearly at least), or\n2. image-level labels are noisy, the shared \"Negative\" images should have been labeled with at least the \"Cytosol\" class, or\n3. when in a single cell there's a clear pattern, e.g. proteins present in Cytosol, together with an additional unspecific pattern, then such cells would be labeled as \"Negative\" in the test set.\n\nThanks in advance!", "votes": 24 }, { "id": 1213064, "postDate": "2021-02-21T20:08:27.707Z", "content": "

This is a very good and relevant question, that I hope I'll be able to clarify.

\n

1) Image-level label uncertainty The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).

\n

2) Single cell label accuracy in test set The test set consists of images where each single cell has been annotated independently. Hence the accuracy of these labels is much better, and will be correct for each cell in every image. The statement below made by Trang Le, @lnhtrang, in our notebook explaining the patterns is correct for how the test set was annotated.

\n
\n

This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).

\n
\n

3) More negative images can be found in the HPA public image data (external data), where they are annotated as \"No Staining\".

\n

4) How to recognize unspecific patterns Unspecific patterns arise from the use of antibody reagents that do not function well, and do not bind to their intended target protein, but rather to all cellular structures (sometimes excluding the nucleolus). The second example from the top, and the cells of weaker intensity in the top image, are text-book examples of this.

", "rawMarkdown": "This is a very good and relevant question, that I hope I'll be able to clarify.\n\n1) **Image-level label uncertainty** The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).\n\n2) **Single cell label accuracy in test set** The test set consists of images where each single cell has been annotated independently. Hence the accuracy of these labels is much better, and will be correct for each cell in every image. The statement below made by Trang Le, @lnhtrang, in our [notebook explaining the patterns](https://www.kaggle.com/lnhtrang/single-cell-patterns) is correct for how the test set was annotated.\n> This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).\n\n3) **More negative images** can be found in the [HPA public image data](https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg) (external data), where they are annotated as \"No Staining\".\n\n4) **How to recognize unspecific patterns** Unspecific patterns arise from the use of antibody reagents that do not function well, and do not bind to their intended target protein, but rather to all cellular structures (sometimes excluding the nucleolus). The second example from the top, and the cells of weaker intensity in the top image, are text-book examples of this.", "votes": 9, "replies": [ { "id": 1213141, "postDate": "2021-02-21T21:24:53.233Z", "content": "

Dear Prof. Lundberg,

\n

Thank you very much for the detailed explanation! Now I understand the labeling procedures and the nature of the unspecific patterns much better! And thank you for the tip on how to find more negative images!

\n

Sincerely,
\nRaman

", "rawMarkdown": "Dear Prof. Lundberg,\n\nThank you very much for the detailed explanation! Now I understand the labeling procedures and the nature of the unspecific patterns much better! And thank you for the tip on how to find more negative images!\n\nSincerely,\nRaman" }, { "id": 1213969, "postDate": "2021-02-22T13:44:27.577Z", "content": "

This is incredibly informative. Thank you for all the detail.

\n

@emmalumpan - A quick follow-up. Is it possible for us to get the ids contained within each grouping/sample? It may help in creating a tool to correct both Image and Single-Cell Labels which will help with training and improved model performance.

\n

Thanks in advance!

", "rawMarkdown": "This is incredibly informative. Thank you for all the detail.\n\n@emmalumpan - A quick follow-up. Is it possible for us to get the ids contained within each grouping/sample? It may help in creating a tool to correct both Image and Single-Cell Labels which will help with training and improved model performance.\n\nThanks in advance!", "votes": 1 }, { "id": 1252782, "postDate": "2021-03-26T03:39:13.667Z", "content": "
\n

1) Image-level label uncertainty The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).

\n
\n

What does 'sample' mean?
\nHow are the images generated from the samples?
\nAnd if the images are generated from samples, how are the samples generated?

", "rawMarkdown": "> 1) Image-level label uncertainty The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).\n\nWhat does 'sample' mean?\nHow are the images generated from the samples?\nAnd if the images are generated from samples, how are the samples generated?", "votes": 2 }, { "id": 1252797, "postDate": "2021-03-26T04:07:20.040Z", "content": "

I guess, 'the same sample' means 'the same type of cells on the dish stained by the same antibody'.
\nEven if the condition (cellline & antibody) is the same, 'green' distribution varies depending on each cell (Single Cell Variability).

", "rawMarkdown": "I guess, 'the same sample' means 'the same type of cells on the dish stained by the same antibody'.\nEven if the condition (cellline & antibody) is the same, 'green' distribution varies depending on each cell (Single Cell Variability).", "votes": 3 }, { "id": 1252907, "postDate": "2021-03-26T06:59:40.940Z", "content": "

I see. Thank you for your very clear explanation.

", "rawMarkdown": "I see. Thank you for your very clear explanation." }, { "id": 1253099, "postDate": "2021-03-26T11:16:24.153Z", "content": "

@dschettler8845 , did you ever get an answer to this question (maybe somewhere else?)

", "rawMarkdown": "@dschettler8845 , did you ever get an answer to this question (maybe somewhere else?)" }, { "id": 1253227, "postDate": "2021-03-26T13:42:22.203Z", "content": "

I did not get the answer to this question, unfortunately. Sorry for not following up!

", "rawMarkdown": "I did not get the answer to this question, unfortunately. Sorry for not following up!" } ] }, { "id": 1211377, "postDate": "2021-02-20T07:01:50.430Z", "content": "

Generally speaking, there are some structures other than the class 0 ~ 17 inside the cell.
\nI guess, nonspecific signals (and other organelles?) are classified as class 18 (Negative).
\nNonspecific signals mean that they are just lightening, not consistent with organelles due to immunocytochemistry or microscopic nature (noise).

", "rawMarkdown": "Generally speaking, there are some structures other than the class 0 ~ 17 inside the cell.\nI guess, nonspecific signals (and other organelles?) are classified as class 18 (Negative).\nNonspecific signals mean that they are just lightening, not consistent with organelles due to immunocytochemistry or microscopic nature (noise).", "votes": 1, "replies": [ { "id": 1211427, "postDate": "2021-02-20T08:05:34.557Z", "content": "

@drtausamaru , thank you for the response!

\n

I'd like to check my understanding, please. Do you mean that in the shared \"Negative\" images proteins are located in other organelles (not covered by the competition labels), and those organelles are floating almost everywhere in the cytoplasm? So in the end it looks almost the same as if the protein of interest was in cytosol directly.

", "rawMarkdown": "@drtausamaru , thank you for the response! \n\nI'd like to check my understanding, please. Do you mean that in the shared \"Negative\" images proteins are located in other organelles (not covered by the competition labels), and those organelles are floating almost everywhere in the cytoplasm? So in the end it looks almost the same as if the protein of interest was in cytosol directly.", "votes": 1 }, { "id": 1211450, "postDate": "2021-02-20T08:36:11.407Z", "content": "

As for this competition, no idea whether they are located in other organelles…sorry.
\nBut comparing images you attached above, '18' seems to be all over the cell (likely irrelevant with cell structures). This means nonspecific, I think.
\nOn the other hand, '16' seems to be more inside the cytosol. I mean proteins in the image of '16' are floating on cytosol, not situated on any other organelles.
\nAs you know, cytoplasm and cytosol are different (Cytoplasm - Organelles = Cytosol).
\nWe have to predict what kind of structures (class 0~18) green lightening proteins are located on.

", "rawMarkdown": "As for this competition, no idea whether they are located in other organelles...sorry.\nBut comparing images you attached above, '18' seems to be all over the cell (likely irrelevant with cell structures). This means nonspecific, I think.\nOn the other hand, '16' seems to be more inside the cytosol. I mean proteins in the image of '16' are floating on cytosol, not situated on any other organelles.\nAs you know, cytoplasm and cytosol are different (Cytoplasm - Organelles = Cytosol).\nWe have to predict what kind of structures (class 0~18) green lightening proteins are located on.", "votes": 1 }, { "id": 1211457, "postDate": "2021-02-20T08:47:33.120Z", "content": "

@drtausamaru ,

\n
\n

But comparing images you attached above, '18' seems to be all over the cell (likely irrelevant with cell structures). This means nonspecific, I guess.

\n
\n

Good point! Thank you!

\n

Hm… So, if cases, when proteins seem to be located everywhere, are considered to be non-specific, and therefore, they have a \"Negative\" label… then it might be worth a try to post-process predictions somehow, i.e. when for a particular cell a model predicts almost all labels with reasonable confidence, then mark the cell as non-specific/negative.

\n

Thanks for the discussion, it already helped me to understand the data better!
\nHopefully, we might get some confirmation of our guesses from the hosts, they seemed to be amazingly active in discussions so far.

", "rawMarkdown": "@drtausamaru ,\n\n> But comparing images you attached above, '18' seems to be all over the cell (likely irrelevant with cell structures). This means nonspecific, I guess.\n\nGood point! Thank you!\n\nHm... So, if cases, when proteins seem to be located everywhere, are considered to be non-specific, and therefore, they have a \"Negative\" label... then it might be worth a try to post-process predictions somehow, i.e. when for a particular cell a model predicts almost all labels with reasonable confidence, then mark the cell as non-specific/negative. \n\nThanks for the discussion, it already helped me to understand the data better! \nHopefully, we might get some confirmation of our guesses from the hosts, they seemed to be amazingly active in discussions so far. ", "votes": 1 } ] }, { "id": 1211240, "postDate": "2021-02-20T04:57:41.140Z", "content": "

Hi Raman,

\n
    \n

  1. Are you using solely visual inspection? Here, the host says that a single cell may have a label that was neglected at the image level.

    2. \n

  2. In this comment the host does say:

    3. \n
\n
\n

these images are manually labeled and there may be errors in terms of wrong labels and missing labels, but only to a minor extent

\n
\n

but I'm not sure this is relevant to the images you've shared or not.

\n

Last. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?

", "rawMarkdown": "Hi Raman,\n\n1. Are you using solely visual inspection? [Here](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/216732#1187708), the host says that a single cell may have a label that was neglected at the image level.\n\n2. In [this comment](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/220275#1210863) the host does say: \n>these images are manually labeled and there may be errors in terms of wrong labels and missing labels, but only to a minor extent \n\nbut I'm not sure this is relevant to the images you've shared or not.\n\nLast. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?", "votes": 2, "replies": [ { "id": 1211449, "postDate": "2021-02-20T08:33:50.507Z", "content": "

Hi Arka,

\n

thank you for the response!

\n
\n

Are you using solely visual inspection?

\n
\n

Yes, I do, for now.

\n
\n

the host says that a single cell may have a label that was neglected at the image level. there may be errors in terms of wrong labels and missing labels, but only to a minor extent

\n
\n

Thanks, I see. Yet, out of only 34 Negative images, there're up to 8 images that look like cytosol-pattern to me:
\n\"\"

\n

If the observed cytosol-like patterns would correspond to missed labels, then it means that out of all Negative images every 4th one has the cytosol label missing. It might be bad luck with the \"Negative\" label, and hopefully, in the general population, wrong/missing labels are not so frequent. Or, it might be different organelles with proteins that float all over the cytoplasm, as @drtausamaru pointed out. Then it might be tricky to distinguish such patterns from the proteins directly located in cytosol.

\n
\n

Last. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?

\n
\n

Thanks for the kind words! Again! :)
\nArka, I don't know the answer for the training set. I just expect training labels to be noisier compared to the test-set labels. So, when I referred to test-set labels I basically referred to the clean, ideal labelling procedure, which the test-set annotators followed.

", "rawMarkdown": "Hi Arka,\n\nthank you for the response!\n\n> Are you using solely visual inspection?\n\nYes, I do, for now.\n\n> the host says that a single cell may have a label that was neglected at the image level. <..> there may be errors in terms of wrong labels and missing labels, but only to a minor extent\n\nThanks, I see. Yet, out of only 34 Negative images, there're up to 8 images that look like cytosol-pattern to me:\n![](https://i.ibb.co/gzjTgY2/download.png)\n\nIf the observed cytosol-like patterns would correspond to missed labels, then it means that out of all Negative images every 4th one has the cytosol label missing. It might be bad luck with the \"Negative\" label, and hopefully, in the general population, wrong/missing labels are not so frequent. Or, it might be different organelles with proteins that float all over the cytoplasm, as @drtausamaru pointed out. Then it might be tricky to distinguish such patterns from the proteins directly located in cytosol. \n\n> Last. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?\n\nThanks for the kind words! Again! :)\nArka, I don't know the answer for the training set. I just expect training labels to be noisier compared to the test-set labels. So, when I referred to test-set labels I basically referred to the clean, ideal labelling procedure, which the test-set annotators followed.", "votes": 1 }, { "id": 1211619, "postDate": "2021-02-20T11:20:29.523Z", "content": "
\n

I just expect training labels to be noisier compared to the test-set labels.

\n
\n

Is that a fair assumption to make though? Here the host says

\n
\n

the test set has higher SVC than the training set, as a result, you should expect higher inconsistency between the cell-level and image-level labels in the test set, including the case where new labels are added to the cell labels

\n
\n

I'm curious to know what you make of this comment, Raman.

", "rawMarkdown": "> I just expect training labels to be noisier compared to the test-set labels.\n\nIs that a fair assumption to make though? [Here](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/220275#1211239) the host says\n>the test set has higher SVC than the training set, as a result, you should expect higher inconsistency between the cell-level and image-level labels in the test set, including the case where new labels are added to the cell labels\n\nI'm curious to know what you make of this comment, Raman." }, { "id": 1211931, "postDate": "2021-02-20T17:12:07.330Z", "content": "
\n

Is that a fair assumption to make though?

\n
\n

Arka, I can't say for sure. I read in discussions that cell segmentations produced by HPACellSegmentation were manually corrected by annotators of the test set. Moreover, the test-set annotators had to double-check each cell labels separately.
\nAnyway, we need to understand what to expect in the test.

\n
\n

the test set has higher SVC than the training set, as a result, you should expect higher inconsistency between the cell-level and image-level labels in the test set, including the case where new labels are added to the cell labels
\n I'm curious to know what you make of this comment, Raman.

\n
\n

It should mean that in the test set there're more images with different localization patterns in individual cells.
\nE.g., if a training image is labelled as \"Cytosol and Mitochondria\", then the majority of cells in this train image might have proteins of interest in both Cytosol and Mitochondria (low SVC = more cells have similar patterns). On the other hand, a high-SVC test image with the same image-level label \"Cytosol and Mitochondria\" might have just a single cell with a protein located in Cytosol, another single cell in the image would have proteins in Mitochondria, and the majority of the cells in the image would have no staining at all.

", "rawMarkdown": "> Is that a fair assumption to make though?\n\nArka, I can't say for sure. I read in discussions that cell segmentations produced by HPACellSegmentation were manually corrected by annotators of the test set. Moreover, the test-set annotators had to double-check each cell labels separately. \nAnyway, we need to understand what to expect in the test.\n\n> the test set has higher SVC than the training set, as a result, you should expect higher inconsistency between the cell-level and image-level labels in the test set, including the case where new labels are added to the cell labels\nI'm curious to know what you make of this comment, Raman.\n\nIt should mean that in the test set there're more images with different localization patterns in individual cells. \nE.g., if a training image is labelled as \"Cytosol and Mitochondria\", then the majority of cells in this train image might have proteins of interest in both Cytosol and Mitochondria (low SVC = more cells have similar patterns). On the other hand, a high-SVC test image with the same image-level label \"Cytosol and Mitochondria\" might have just a single cell with a protein located in Cytosol, another single cell in the image would have proteins in Mitochondria, and the majority of the cells in the image would have no staining at all.", "votes": 1 }, { "id": 1212251, "postDate": "2021-02-21T03:42:46.110Z", "content": "
\n

I read in discussions that cell segmentations produced by HPACellSegmentation were manually corrected by annotators of the test set. Moreover, the test-set annotators had to double-check each cell labels separately.

\n
\n

I see, hope to get more insights as everything is still not crystal clear to me. Will be following the discussions closely, thanks!

", "rawMarkdown": "> I read in discussions that cell segmentations produced by HPACellSegmentation were manually corrected by annotators of the test set. Moreover, the test-set annotators had to double-check each cell labels separately.\n\nI see, hope to get more insights as everything is still not crystal clear to me. Will be following the discussions closely, thanks!" } ] }, { "id": 1225341, "postDate": "2021-03-03T14:55:04.313Z", "content": "

Thank you very much for the detailed explanation!

", "rawMarkdown": "Thank you very much for the detailed explanation! " } ], "comments": [ { "id": 1213064, "author_name": "Emma Lundberg", "author_url": "", "post_date": "2021-02-21T20:08:27.707000", "content": "

This is a very good and relevant question, that I hope I'll be able to clarify.

\n

1) Image-level label uncertainty The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).

\n

2) Single cell label accuracy in test set The test set consists of images where each single cell has been annotated independently. Hence the accuracy of these labels is much better, and will be correct for each cell in every image. The statement below made by Trang Le, @lnhtrang, in our notebook explaining the patterns is correct for how the test set was annotated.

\n
\n

This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).

\n
\n

3) More negative images can be found in the HPA public image data (external data), where they are annotated as \"No Staining\".

\n

4) How to recognize unspecific patterns Unspecific patterns arise from the use of antibody reagents that do not function well, and do not bind to their intended target protein, but rather to all cellular structures (sometimes excluding the nucleolus). The second example from the top, and the cells of weaker intensity in the top image, are text-book examples of this.

", "votes": 9, "replies": [ { "id": 1213141, "author_name": "Raman", "author_url": "", "post_date": "2021-02-21T21:24:53.233000", "content": "

Dear Prof. Lundberg,

\n

Thank you very much for the detailed explanation! Now I understand the labeling procedures and the nature of the unspecific patterns much better! And thank you for the tip on how to find more negative images!

\n

Sincerely,
\nRaman

", "votes": 0, "replies": [] }, { "id": 1213969, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-02-22T13:44:27.577000", "content": "

This is incredibly informative. Thank you for all the detail.

\n

@emmalumpan - A quick follow-up. Is it possible for us to get the ids contained within each grouping/sample? It may help in creating a tool to correct both Image and Single-Cell Labels which will help with training and improved model performance.

\n

Thanks in advance!

", "votes": 1, "replies": [] }, { "id": 1252782, "author_name": "Kuruton", "author_url": "", "post_date": "2021-03-26T03:39:13.667000", "content": "
\n

1) Image-level label uncertainty The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).

\n
\n

What does 'sample' mean?
\nHow are the images generated from the samples?
\nAnd if the images are generated from samples, how are the samples generated?

", "votes": 2, "replies": [] }, { "id": 1252797, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-26T04:07:20.040000", "content": "

I guess, 'the same sample' means 'the same type of cells on the dish stained by the same antibody'.
\nEven if the condition (cellline & antibody) is the same, 'green' distribution varies depending on each cell (Single Cell Variability).

", "votes": 3, "replies": [] }, { "id": 1252907, "author_name": "Kuruton", "author_url": "", "post_date": "2021-03-26T06:59:40.940000", "content": "

I see. Thank you for your very clear explanation.

", "votes": 0, "replies": [] }, { "id": 1253099, "author_name": "fnands", "author_url": "", "post_date": "2021-03-26T11:16:24.153000", "content": "

@dschettler8845 , did you ever get an answer to this question (maybe somewhere else?)

", "votes": 0, "replies": [] }, { "id": 1253227, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-26T13:42:22.203000", "content": "

I did not get the answer to this question, unfortunately. Sorry for not following up!

", "votes": 0, "replies": [] } ] }, { "id": 1211377, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-02-20T07:01:50.430000", "content": "

Generally speaking, there are some structures other than the class 0 ~ 17 inside the cell.
\nI guess, nonspecific signals (and other organelles?) are classified as class 18 (Negative).
\nNonspecific signals mean that they are just lightening, not consistent with organelles due to immunocytochemistry or microscopic nature (noise).

", "votes": 1, "replies": [ { "id": 1211427, "author_name": "Raman", "author_url": "", "post_date": "2021-02-20T08:05:34.557000", "content": "

@drtausamaru , thank you for the response!

\n

I'd like to check my understanding, please. Do you mean that in the shared \"Negative\" images proteins are located in other organelles (not covered by the competition labels), and those organelles are floating almost everywhere in the cytoplasm? So in the end it looks almost the same as if the protein of interest was in cytosol directly.

", "votes": 1, "replies": [] }, { "id": 1211450, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-02-20T08:36:11.407000", "content": "

As for this competition, no idea whether they are located in other organelles…sorry.
\nBut comparing images you attached above, '18' seems to be all over the cell (likely irrelevant with cell structures). This means nonspecific, I think.
\nOn the other hand, '16' seems to be more inside the cytosol. I mean proteins in the image of '16' are floating on cytosol, not situated on any other organelles.
\nAs you know, cytoplasm and cytosol are different (Cytoplasm - Organelles = Cytosol).
\nWe have to predict what kind of structures (class 0~18) green lightening proteins are located on.

", "votes": 1, "replies": [] }, { "id": 1211457, "author_name": "Raman", "author_url": "", "post_date": "2021-02-20T08:47:33.120000", "content": "

@drtausamaru ,

\n
\n

But comparing images you attached above, '18' seems to be all over the cell (likely irrelevant with cell structures). This means nonspecific, I guess.

\n
\n

Good point! Thank you!

\n

Hm… So, if cases, when proteins seem to be located everywhere, are considered to be non-specific, and therefore, they have a \"Negative\" label… then it might be worth a try to post-process predictions somehow, i.e. when for a particular cell a model predicts almost all labels with reasonable confidence, then mark the cell as non-specific/negative.

\n

Thanks for the discussion, it already helped me to understand the data better!
\nHopefully, we might get some confirmation of our guesses from the hosts, they seemed to be amazingly active in discussions so far.

", "votes": 1, "replies": [] } ] }, { "id": 1211240, "author_name": "Arka Saha", "author_url": "", "post_date": "2021-02-20T04:57:41.140000", "content": "

Hi Raman,

\n
    \n

  1. Are you using solely visual inspection? Here, the host says that a single cell may have a label that was neglected at the image level.

    2. \n

  2. In this comment the host does say:

    3. \n
\n
\n

these images are manually labeled and there may be errors in terms of wrong labels and missing labels, but only to a minor extent

\n
\n

but I'm not sure this is relevant to the images you've shared or not.

\n

Last. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?

", "votes": 2, "replies": [ { "id": 1211449, "author_name": "Raman", "author_url": "", "post_date": "2021-02-20T08:33:50.507000", "content": "

Hi Arka,

\n

thank you for the response!

\n
\n

Are you using solely visual inspection?

\n
\n

Yes, I do, for now.

\n
\n

the host says that a single cell may have a label that was neglected at the image level. there may be errors in terms of wrong labels and missing labels, but only to a minor extent

\n
\n

Thanks, I see. Yet, out of only 34 Negative images, there're up to 8 images that look like cytosol-pattern to me:
\n\"\"

\n

If the observed cytosol-like patterns would correspond to missed labels, then it means that out of all Negative images every 4th one has the cytosol label missing. It might be bad luck with the \"Negative\" label, and hopefully, in the general population, wrong/missing labels are not so frequent. Or, it might be different organelles with proteins that float all over the cytoplasm, as @drtausamaru pointed out. Then it might be tricky to distinguish such patterns from the proteins directly located in cytosol.

\n
\n

Last. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?

\n
\n

Thanks for the kind words! Again! :)
\nArka, I don't know the answer for the training set. I just expect training labels to be noisier compared to the test-set labels. So, when I referred to test-set labels I basically referred to the clean, ideal labelling procedure, which the test-set annotators followed.

", "votes": 1, "replies": [] }, { "id": 1211619, "author_name": "Arka Saha", "author_url": "", "post_date": "2021-02-20T11:20:29.523000", "content": "
\n

I just expect training labels to be noisier compared to the test-set labels.

\n
\n

Is that a fair assumption to make though? Here the host says

\n
\n

the test set has higher SVC than the training set, as a result, you should expect higher inconsistency between the cell-level and image-level labels in the test set, including the case where new labels are added to the cell labels

\n
\n

I'm curious to know what you make of this comment, Raman.

", "votes": 0, "replies": [] }, { "id": 1211931, "author_name": "Raman", "author_url": "", "post_date": "2021-02-20T17:12:07.330000", "content": "
\n

Is that a fair assumption to make though?

\n
\n

Arka, I can't say for sure. I read in discussions that cell segmentations produced by HPACellSegmentation were manually corrected by annotators of the test set. Moreover, the test-set annotators had to double-check each cell labels separately.
\nAnyway, we need to understand what to expect in the test.

\n
\n

the test set has higher SVC than the training set, as a result, you should expect higher inconsistency between the cell-level and image-level labels in the test set, including the case where new labels are added to the cell labels
\n I'm curious to know what you make of this comment, Raman.

\n
\n

It should mean that in the test set there're more images with different localization patterns in individual cells.
\nE.g., if a training image is labelled as \"Cytosol and Mitochondria\", then the majority of cells in this train image might have proteins of interest in both Cytosol and Mitochondria (low SVC = more cells have similar patterns). On the other hand, a high-SVC test image with the same image-level label \"Cytosol and Mitochondria\" might have just a single cell with a protein located in Cytosol, another single cell in the image would have proteins in Mitochondria, and the majority of the cells in the image would have no staining at all.

", "votes": 1, "replies": [] }, { "id": 1212251, "author_name": "Arka Saha", "author_url": "", "post_date": "2021-02-21T03:42:46.110000", "content": "
\n

I read in discussions that cell segmentations produced by HPACellSegmentation were manually corrected by annotators of the test set. Moreover, the test-set annotators had to double-check each cell labels separately.

\n
\n

I see, hope to get more insights as everything is still not crystal clear to me. Will be following the discussions closely, thanks!

", "votes": 0, "replies": [] } ] }, { "id": 1225341, "author_name": "Harsha", "author_url": "", "post_date": "2021-03-03T14:55:04.313000", "content": "

Thank you very much for the detailed explanation!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1210382": "Hi all,\n\nI checked negative images. To my surprise, often there're apparent protein patterns in the green channel of images labeled as \"Negative\".\n\n## What have the hosts said about the \"Negative\" class?\nThe competition host prof. Emma Lundberg, @emmalumpan, has kindly pointed out that\n> The label negative can be considered a \"none-of-the-above\" type of label (it describes that there is no green staining/pattern, i.e. no other label). Hence the label negative will only be present at the image level if all cells show no other green pattern. ([post](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/219211#1200188))\n\n> If there is no specific green pattern (noisy or no signal) it should be labeled as negative. ([post](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/216924#1187646))\n\nThe competition host Trang Le, @lnhtrang, in the shared [awesome notebook](https://www.kaggle.com/lnhtrang/single-cell-patterns), mentioned that\n\n> This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).\n\n## What is unclear about \"Negative\" images in the train set?\nUnfortunately, I'm confused with some Negative images, e.g.:\n![](https://i.ibb.co/DV9vRP8/neg-1.png)\n![](https://i.ibb.co/ggcgZQC/neg-2.png)\n![](https://i.ibb.co/rmRXhtg/neg-3.png)\n\nI might be wrong, but it seems to me that at least Cytosol contains proteins of interest in the shared images. \nE.g., here are some reference images with the \"Cytosol\" label:\n![](https://i.ibb.co/gVnbh0T/cyt-4.png)\n![](https://i.ibb.co/f4vTMMb/cyt-3.png)\n![](https://i.ibb.co/QbzSnzC/cyt-2.png)\n![](https://i.ibb.co/WW7cCz8/cyt-1.png)\n\n## Main question\nCould please anyone clarify, if \n1. my understanding is incorrect and in the shared \"Negative\" images no proteins are present in Cytosol (not clearly at least), or\n2. image-level labels are noisy, the shared \"Negative\" images should have been labeled with at least the \"Cytosol\" class, or\n3. when in a single cell there's a clear pattern, e.g. proteins present in Cytosol, together with an additional unspecific pattern, then such cells would be labeled as \"Negative\" in the test set.\n\nThanks in advance!", "1213064": "This is a very good and relevant question, that I hope I'll be able to clarify.\n\n1) **Image-level label uncertainty** The image-level labels are what we refer to as weak or noisy. During annotation, the image-level labels are set per sample (i.e per a group of up to 6 images from the same sample). This means that the labels present in the majority of the images will be annotated. For negative samples it is not uncommon that lets say 4 images show no staining, while the remaining 2 show some unspecific staining or some granular pattern. If you compare the image-level label with the precise pattern observed in any given cell from this group of images, the label will be correct for the vast majority of cells, but perhaps not for all of them (as in your example 1 and 3 above).\n\n2) **Single cell label accuracy in test set** The test set consists of images where each single cell has been annotated independently. Hence the accuracy of these labels is much better, and will be correct for each cell in every image. The statement below made by Trang Le, @lnhtrang, in our [notebook explaining the patterns](https://www.kaggle.com/lnhtrang/single-cell-patterns) is correct for how the test set was annotated.\n> This class (Negative) includes negative stainings and unspecific patterns. This means that the cells have no green staining (negative) or have staining but no pattern can be deciphered from the staining (unspecific).\n\n3) **More negative images** can be found in the [HPA public image data](https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg) (external data), where they are annotated as \"No Staining\".\n\n4) **How to recognize unspecific patterns** Unspecific patterns arise from the use of antibody reagents that do not function well, and do not bind to their intended target protein, but rather to all cellular structures (sometimes excluding the nucleolus). The second example from the top, and the cells of weaker intensity in the top image, are text-book examples of this.", "1211377": "Generally speaking, there are some structures other than the class 0 ~ 17 inside the cell.\nI guess, nonspecific signals (and other organelles?) are classified as class 18 (Negative).\nNonspecific signals mean that they are just lightening, not consistent with organelles due to immunocytochemistry or microscopic nature (noise).", "1211240": "Hi Raman,\n\n1. Are you using solely visual inspection? [Here](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/216732#1187708), the host says that a single cell may have a label that was neglected at the image level.\n\n2. In [this comment](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/220275#1210863) the host does say: \n>these images are manually labeled and there may be errors in terms of wrong labels and missing labels, but only to a minor extent \n\nbut I'm not sure this is relevant to the images you've shared or not.\n\nLast. That's an amazing question that has also been confusing me, what is a cell with a clear pattern AND an unspecific pattern labeled as. You've asked about the test set, do you know the answer in case of the training set?", "1225341": "Thank you very much for the detailed explanation! " } }