Hi there.
\nI think this is a pretty basic question but I wanted to ask here just in case anyone else had the same inquiry (I'm not afraid to ask the simple questions 😅). I did some Googling and have my own speculations but I always appreciate the wisdom of other Kagglers… so here we go.
\nSETUP:
\nI'm training a multi-label cell classifier.
\nWe have 19 classes (18 organelle structures, and a negative label).
QUESTION:
\nDo I frame this as a 19 class multi-label classification problem or an 18 class classification problem where a lack of confidence in the 18 organelle classes indicates the presence of the negative class?
\nFOLLOW-UP 1:
\nLet's say I decide to frame it as an 18 class multi-label classification problem. Therefore, during training, if I have n negative training samples.
\nKnowing this, should I pass the ground-truth label as a vector of length 18 full of 0s for all n negative training samples?
\nFOLLOW-UP 2:
\nDuring the training of a multi-label classification model with more than a few classes, it's obvious that most of the time the majority of the outputs will be 0.
\ni.e. Let's look at 5 outputs for a 5 class model (assuming no negative classes which would only exacerbate the problem).
\nmodel_output_1 = [0.05, 0.23, 0.10, 0.80, 0.10]\nground_truth_1 = [0.00, 0.00, 0.00, 1.00, 0.00,]\n\nmodel_output_2 = [0.95, 0.02, 0.04, 0.02, 0.10]\nground_truth_2 = [1.00, 0.00, 0.00, 0.00, 0.00]\n\nmodel_output_3 = [0.02, 0.66, 0.09, 0.08, 0.13]\nground_truth_3 = [0.00, 1.00, 0.00, 0.00, 0.00,]\n\nmodel_output_4 = [0.15, 0.03, 0.08, 0.38, 0.55]\nground_truth_4 = [0.00, 0.00, 0.00, 0.00, 1.00,]\n\nmodel_output_5 = [0.22, 0.04, 0.98, 0.32, 0.12]\nground_truth_5 = [0.00, 0.00, 1.00, 0.00, 0.00,]\nWould it not be true in this case that the model will just eventually learn to predict all 0s all the time? Without a class weighting function that is?
\nAnd if that's true, is it normal practice to pass a class-wise binary, class-weighting?
\nThanks in advance!!
", "messages": [ { "id": 1211873, "postDate": "2021-02-20T16:17:00.313Z", "content": "Hi there.
\nI think this is a pretty basic question but I wanted to ask here just in case anyone else had the same inquiry (I'm not afraid to ask the simple questions 😅). I did some Googling and have my own speculations but I always appreciate the wisdom of other Kagglers… so here we go.
\nSETUP:
\nI'm training a multi-label cell classifier.
\nWe have 19 classes (18 organelle structures, and a negative label).
QUESTION:
\nDo I frame this as a 19 class multi-label classification problem or an 18 class classification problem where a lack of confidence in the 18 organelle classes indicates the presence of the negative class?
\nFOLLOW-UP 1:
\nLet's say I decide to frame it as an 18 class multi-label classification problem. Therefore, during training, if I have n negative training samples.
\nKnowing this, should I pass the ground-truth label as a vector of length 18 full of 0s for all n negative training samples?
\nFOLLOW-UP 2:
\nDuring the training of a multi-label classification model with more than a few classes, it's obvious that most of the time the majority of the outputs will be 0.
\ni.e. Let's look at 5 outputs for a 5 class model (assuming no negative classes which would only exacerbate the problem).
\nmodel_output_1 = [0.05, 0.23, 0.10, 0.80, 0.10]\nground_truth_1 = [0.00, 0.00, 0.00, 1.00, 0.00,]\n\nmodel_output_2 = [0.95, 0.02, 0.04, 0.02, 0.10]\nground_truth_2 = [1.00, 0.00, 0.00, 0.00, 0.00]\n\nmodel_output_3 = [0.02, 0.66, 0.09, 0.08, 0.13]\nground_truth_3 = [0.00, 1.00, 0.00, 0.00, 0.00,]\n\nmodel_output_4 = [0.15, 0.03, 0.08, 0.38, 0.55]\nground_truth_4 = [0.00, 0.00, 0.00, 0.00, 1.00,]\n\nmodel_output_5 = [0.22, 0.04, 0.98, 0.32, 0.12]\nground_truth_5 = [0.00, 0.00, 1.00, 0.00, 0.00,]\nWould it not be true in this case that the model will just eventually learn to predict all 0s all the time? Without a class weighting function that is?
\nAnd if that's true, is it normal practice to pass a class-wise binary, class-weighting?
\nThanks in advance!!
", "rawMarkdown": "Hi there.\n\nI think this is a pretty basic question but I wanted to ask here just in case anyone else had the same inquiry (I'm not afraid to ask the simple questions 😅). I did some Googling and have my own speculations but I always appreciate the wisdom of other Kagglers... so here we go.\n\n---\n\nIMO there are 19 classes. But we are hampered by a lack of very many ground truth images with only that class. In another post in this competition someone examined the \"negative\" images and as I recall felt even as a amateur they would have classified about half of them with one of our base 18.
\nWe do however have an huge number (once again IMO) of the negative cells in many images. I am referring to the cells falling on the edge of the main image itself. I would expect that my models should end up calling many of these negative.
\nComments from the hosts suggest that border cells are not likely to be annotated and in the hand correction of segments I assume this means no segment will be present in the private test set.
\nThere would seem to be several options for handling the edge of image cells - I currently plan to code towards them being classified as negative. But I am open for other suggestions on how to handle partial cells from edges.
", "rawMarkdown": "IMO there are 19 classes. But we are hampered by a lack of very many ground truth images with only that class. In another [post](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/220748) in this competition someone examined the \"negative\" images and as I recall felt even as a amateur they would have classified about half of them with one of our base 18.\n\nWe do however have an huge number (once again IMO) of the negative cells in many images. I am referring to the cells falling on the edge of the main image itself. I would expect that my models should end up calling many of these negative.\n\nComments from the hosts suggest that border cells are not likely to be annotated and in the hand correction of segments I assume this means no segment will be present in the private test set.\n\nThere would seem to be several options for handling the edge of image cells - I currently plan to code towards them being classified as negative. But I am open for other suggestions on how to handle partial cells from edges.", "votes": 3 }, { "id": 1212023, "postDate": "2021-02-20T19:38:03.740Z", "content": "Good questions! My initial intuitions below:
\nThanks for the comment!
\nAgreed it can be framed in either way. I have, up until now, been treating it as a 19 class problem. Now that I've created a new dataset with a plethora of Negative labels I will be pivoting to an 18 class problem (either, as you indicated, passing a vector of zeros as the label for the Negative images, or, as Ayush indicated, having a multi-stage pipeline where you first perform Binary Classification and then if the class is positive, perform multi-label classification using the 18 classes.
Agreed
I think I understand. I guess I'm just struggling with the implementation side of things. Normally, for a multi-class classification problem (not multi-label), if I have a class imbalance, I will pass a dictionary of class weights to my fit function. If I had a model with 18 outputs, where each was a sigmoid activation function, I would normally pass 18 class weighting dictionaries ({0:0.75, 1:0.25}) indicating the percentage of positives v. negatives for every class. However, when I build my network now, I have a single output layer of length 18. I can't pass individual class weight dictionaries… I think. I guess I just have to get more comfortable with multi-label class weighting in Tensorflow. Or, alternatively, I can create a custom loss function with the class weighting baked in.
I'm currently training the two models to be used in the multi-stage approach… which should be finished in 6-8 hours. Following that, I will update my inference notebook and submit and we'll get some feedback on this new technique!
", "rawMarkdown": "Thanks for the comment! \n\n---\n\n- Agreed it can be framed in either way. I have, up until now, been treating it as a 19 class problem. Now that I've created a new dataset with a plethora of **Negative** labels I will be pivoting to an 18 class problem (either, as you indicated, passing a vector of zeros as the label for the **Negative** images, or, as Ayush indicated, having a multi-stage pipeline where you first perform Binary Classification and then if the class is positive, perform multi-label classification using the 18 classes.\n\n- Agreed\n\n- I think I understand. I guess I'm just struggling with the implementation side of things. Normally, for a multi-class classification problem (not multi-label), if I have a class imbalance, I will pass a dictionary of class weights to my fit function. If I had a model with 18 outputs, where each was a sigmoid activation function, I would normally pass 18 class weighting dictionaries (**`{0:0.75, 1:0.25}`**) indicating the percentage of positives v. negatives for every class. However, when I build my network now, I have a single output layer of length 18. I can't pass individual class weight dictionaries... I think. I guess I just have to get more comfortable with multi-label class weighting in Tensorflow. Or, alternatively, I can create a custom loss function with the class weighting baked in.\n\n---\n\nI'm currently training the two models to be used in the multi-stage approach... which should be finished in 6-8 hours. Following that, I will update my inference notebook and submit and we'll get some feedback on this new technique!\n", "votes": 1 }, { "id": 1212274, "postDate": "2021-02-21T04:13:14.593Z", "content": "\n\n{0:0.75, 1:0.25}
\n
Is there no way to pass a single dictionary having the weights? Something like {0:0.5, 1:0.7, 2:1,……,18:0.8}
", "rawMarkdown": "> {0:0.75, 1:0.25}\n\nIs there no way to pass a single dictionary having the weights? Something like {0:0.5, 1:0.7, 2:1,......,18:0.8}" }, { "id": 1212384, "postDate": "2021-02-21T07:26:35.987Z", "content": "I haven't done this myself yet. I started by creating a more balanced sample of train and use it to train my models. Planning to explore a balanced sampler as well, e.g. like this one: https://github.com/issamemari/pytorch-multilabel-balanced-sampler
", "rawMarkdown": "I haven't done this myself yet. I started by creating a more balanced sample of train and use it to train my models. Planning to explore a balanced sampler as well, e.g. like this one: https://github.com/issamemari/pytorch-multilabel-balanced-sampler", "votes": 1 } ] }, { "id": 1212502, "postDate": "2021-02-21T09:29:51.773Z", "content": "Hi. What's your CV strategy?
\nI think multilabel stratified kfold is good, but also we may have to consider multilabel stratified GROUP kfold because it is said that there are 17 different cell types.
For the classification models, I'm using MultilabelStratifiedKFold:
\nhttps://github.com/trent-b/iterative-stratification. For group kfold do you mean stratifying by cell line? We don't have that available in training though, do we?
Thanks for the reply.
\nSo, clustering is one choice.
\nIf there is no tendency that the same type of cells is easy to predict, we don't have to care about it :)
I have not implemented a CV strategy yet. I plan to use a strategy similar to the one Darek noted. That being said I will most likely implement something using tf.data.
Clustering on cell-type and stratifying would be an excellent idea. Definitely, something I hadn't thought of! Thanks for suggesting it and I look forward to seeing it implemented. (I may give it a shot if I have the bandwidth 😄).
", "rawMarkdown": "I have not implemented a CV strategy yet. I plan to use a strategy similar to the one Darek noted. That being said I will most likely implement something using **`tf.data`**.\n\nClustering on cell-type and stratifying would be an excellent idea. Definitely, something I hadn't thought of! Thanks for suggesting it and I look forward to seeing it implemented. (I may give it a shot if I have the bandwidth 😄).", "votes": 2 }, { "id": 1234239, "postDate": "2021-03-11T04:23:09.877Z", "content": "How do we distinguish cell lines?
", "rawMarkdown": "How do we distinguish cell lines?", "votes": 1 }, { "id": 1235121, "postDate": "2021-03-11T20:57:25.333Z", "content": "Actually several days ago I made a simple cellline classifier using public data, but it didn't work well for me😂
", "rawMarkdown": "Actually several days ago I made a simple cellline classifier using public data, but it didn't work well for me😂", "votes": 2 } ] }, { "id": 1211887, "postDate": "2021-02-20T16:27:52.420Z", "content": "If there were a sufficient number of negative examples I would have tried to do binary classification to classify negative from positive images. Given positive images do multi-label classification. But that's not the case, unfortunately.
\nFormulating this as a 19 multi-label classification problem should be a good starting point in my opinion. The catch as you know is to associate a negative label to a cell. And most probably many cells would fall under the negative label but that's subject to further investigation.
\nHoping others have a better answer. :)
", "rawMarkdown": "If there were a sufficient number of negative examples I would have tried to do binary classification to classify negative from positive images. Given positive images do multi-label classification. But that's not the case, unfortunately. \n\nFormulating this as a 19 multi-label classification problem should be a good starting point in my opinion. The catch as you know is to associate a negative label to a cell. And most probably many cells would fall under the negative label but that's subject to further investigation.\n\nHoping others have a better answer. :)", "votes": 1, "replies": [ { "id": 1211949, "postDate": "2021-02-20T17:31:26.430Z", "content": "Thanks for the reply @ayuraj !
\nI have created a cell-level dataset following the application of some heuristics to identify negative cells. There is now a large number of negative class cells. ~3 times more than the next highest class and thousands of times more than the lower classes.
\nThe idea of prefacing my multilabel with a negative/positive classifier is a good idea. I think I may try and implement that.
\nI'd also like to see other options for solutions!
", "rawMarkdown": "Thanks for the reply @ayuraj !\n\nI have created a cell-level dataset following the application of some heuristics to identify negative cells. There is now a large number of negative class cells. ~3 times more than the next highest class and thousands of times more than the lower classes.\n\nThe idea of prefacing my multilabel with a negative/positive classifier is a good idea. I think I may try and implement that.\n\nI'd also like to see other options for solutions!", "votes": 1 }, { "id": 1212381, "postDate": "2021-02-21T07:22:30.377Z", "content": "Looking forward to see your results! I've been playing with some heuristics to identify negative classes via probing the leaderboard, but unsuccessful so far…
", "rawMarkdown": "Looking forward to see your results! I've been playing with some heuristics to identify negative classes via probing the leaderboard, but unsuccessful so far..." }, { "id": 1212473, "postDate": "2021-02-21T08:49:05.120Z", "content": "If you want more negative images you can use the HPA public images where negative images are labeled \"No staining\".
", "rawMarkdown": "If you want more negative images you can use the [HPA public images](https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg#Downloading-HPA-public-data) where negative images are labeled \"No staining\".", "votes": 4 }, { "id": 1212526, "postDate": "2021-02-21T09:54:58.930Z", "content": "That would be helpful. Thank you for the info.
", "rawMarkdown": "That would be helpful. Thank you for the info." } ] }, { "id": 1211985, "postDate": "2021-02-20T18:17:14.190Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 1212063, "postDate": "2021-02-20T20:51:24.180Z", "content": "Hi @weka511, thanks for the comment. I have seen Darek's excellent notebook. He has given me quite a lot of inspiration in this competition.
\nI agree with your points across the board and enjoy the Kuiper Belt analogy!
", "rawMarkdown": "Hi @weka511, thanks for the comment. I have seen Darek's excellent notebook. He has given me quite a lot of inspiration in this competition.\n\nI agree with your points across the board and enjoy the Kuiper Belt analogy!", "votes": 1 }, { "id": 1212168, "postDate": "2021-02-21T00:26:59.073Z", "content": "\nThere are more than 18 - the list on the HPA site shows 35. They have some imaginative names :) It could really be an unbalanced problem if we needed to indentify all 35. I agree with your guess that they are Kuiper Belt members of a cell.
\n\n", "rawMarkdown": "[Simon](https://www.kaggle.com/weka511)\n\n There are more than 18 - the list on the HPA site shows 35. They have some imaginative names :) It could really be an unbalanced problem if we needed to indentify all 35. I agree with your guess that they are Kuiper Belt members of a cell.\n\n> Protein localization data is derived from antibody-based profiling, using immunofluorescence (ICC-IF) and confocal microscopy, and classified into 35 different organelles and fine subcellular structures.> " } ] } ], "comments": [ { "id": 1212250, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-21T03:42:23.807000", "content": "Protein localization data is derived from antibody-based profiling, using immunofluorescence (ICC-IF) and confocal microscopy, and classified into 35 different organelles and fine subcellular structures.>
\n
IMO there are 19 classes. But we are hampered by a lack of very many ground truth images with only that class. In another post in this competition someone examined the \"negative\" images and as I recall felt even as a amateur they would have classified about half of them with one of our base 18.
\nWe do however have an huge number (once again IMO) of the negative cells in many images. I am referring to the cells falling on the edge of the main image itself. I would expect that my models should end up calling many of these negative.
\nComments from the hosts suggest that border cells are not likely to be annotated and in the hand correction of segments I assume this means no segment will be present in the private test set.
\nThere would seem to be several options for handling the edge of image cells - I currently plan to code towards them being classified as negative. But I am open for other suggestions on how to handle partial cells from edges.
", "votes": 3, "replies": [] }, { "id": 1212023, "author_name": "Darek KÅ‚eczek", "author_url": "", "post_date": "2021-02-20T19:38:03.740000", "content": "Good questions! My initial intuitions below:
\nThanks for the comment!
\nAgreed it can be framed in either way. I have, up until now, been treating it as a 19 class problem. Now that I've created a new dataset with a plethora of Negative labels I will be pivoting to an 18 class problem (either, as you indicated, passing a vector of zeros as the label for the Negative images, or, as Ayush indicated, having a multi-stage pipeline where you first perform Binary Classification and then if the class is positive, perform multi-label classification using the 18 classes.
Agreed
I think I understand. I guess I'm just struggling with the implementation side of things. Normally, for a multi-class classification problem (not multi-label), if I have a class imbalance, I will pass a dictionary of class weights to my fit function. If I had a model with 18 outputs, where each was a sigmoid activation function, I would normally pass 18 class weighting dictionaries ({0:0.75, 1:0.25}) indicating the percentage of positives v. negatives for every class. However, when I build my network now, I have a single output layer of length 18. I can't pass individual class weight dictionaries… I think. I guess I just have to get more comfortable with multi-label class weighting in Tensorflow. Or, alternatively, I can create a custom loss function with the class weighting baked in.
I'm currently training the two models to be used in the multi-stage approach… which should be finished in 6-8 hours. Following that, I will update my inference notebook and submit and we'll get some feedback on this new technique!
", "votes": 1, "replies": [] }, { "id": 1212274, "author_name": "Arka Saha", "author_url": "", "post_date": "2021-02-21T04:13:14.593000", "content": "\n\n{0:0.75, 1:0.25}
\n
Is there no way to pass a single dictionary having the weights? Something like {0:0.5, 1:0.7, 2:1,……,18:0.8}
", "votes": 0, "replies": [] }, { "id": 1212384, "author_name": "Darek KÅ‚eczek", "author_url": "", "post_date": "2021-02-21T07:26:35.987000", "content": "I haven't done this myself yet. I started by creating a more balanced sample of train and use it to train my models. Planning to explore a balanced sampler as well, e.g. like this one: https://github.com/issamemari/pytorch-multilabel-balanced-sampler
", "votes": 1, "replies": [] } ] }, { "id": 1212502, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-02-21T09:29:51.773000", "content": "Hi. What's your CV strategy?
\nI think multilabel stratified kfold is good, but also we may have to consider multilabel stratified GROUP kfold because it is said that there are 17 different cell types.
For the classification models, I'm using MultilabelStratifiedKFold:
\nhttps://github.com/trent-b/iterative-stratification. For group kfold do you mean stratifying by cell line? We don't have that available in training though, do we?
Thanks for the reply.
\nSo, clustering is one choice.
\nIf there is no tendency that the same type of cells is easy to predict, we don't have to care about it :)
I have not implemented a CV strategy yet. I plan to use a strategy similar to the one Darek noted. That being said I will most likely implement something using tf.data.
Clustering on cell-type and stratifying would be an excellent idea. Definitely, something I hadn't thought of! Thanks for suggesting it and I look forward to seeing it implemented. (I may give it a shot if I have the bandwidth 😄).
", "votes": 2, "replies": [] }, { "id": 1234239, "author_name": "Correlation", "author_url": "", "post_date": "2021-03-11T04:23:09.877000", "content": "How do we distinguish cell lines?
", "votes": 1, "replies": [] }, { "id": 1235121, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-11T20:57:25.333000", "content": "Actually several days ago I made a simple cellline classifier using public data, but it didn't work well for me😂
", "votes": 2, "replies": [] } ] }, { "id": 1211887, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-02-20T16:27:52.420000", "content": "If there were a sufficient number of negative examples I would have tried to do binary classification to classify negative from positive images. Given positive images do multi-label classification. But that's not the case, unfortunately.
\nFormulating this as a 19 multi-label classification problem should be a good starting point in my opinion. The catch as you know is to associate a negative label to a cell. And most probably many cells would fall under the negative label but that's subject to further investigation.
\nHoping others have a better answer. :)
", "votes": 1, "replies": [ { "id": 1211949, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-02-20T17:31:26.430000", "content": "Thanks for the reply @ayuraj !
\nI have created a cell-level dataset following the application of some heuristics to identify negative cells. There is now a large number of negative class cells. ~3 times more than the next highest class and thousands of times more than the lower classes.
\nThe idea of prefacing my multilabel with a negative/positive classifier is a good idea. I think I may try and implement that.
\nI'd also like to see other options for solutions!
", "votes": 1, "replies": [] }, { "id": 1212381, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-02-21T07:22:30.377000", "content": "Looking forward to see your results! I've been playing with some heuristics to identify negative classes via probing the leaderboard, but unsuccessful so far…
", "votes": 0, "replies": [] }, { "id": 1212473, "author_name": "Emma Lundberg", "author_url": "", "post_date": "2021-02-21T08:49:05.120000", "content": "If you want more negative images you can use the HPA public images where negative images are labeled \"No staining\".
", "votes": 4, "replies": [] }, { "id": 1212526, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-02-21T09:54:58.930000", "content": "That would be helpful. Thank you for the info.
", "votes": 0, "replies": [] } ] }, { "id": 1211985, "author_name": "", "author_url": "", "post_date": "2021-02-20T18:17:14.190000", "content": "", "votes": 2, "replies": [ { "id": 1212063, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-02-20T20:51:24.180000", "content": "Hi @weka511, thanks for the comment. I have seen Darek's excellent notebook. He has given me quite a lot of inspiration in this competition.
\nI agree with your points across the board and enjoy the Kuiper Belt analogy!
", "votes": 1, "replies": [] }, { "id": 1212168, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-21T00:26:59.073000", "content": "\nThere are more than 18 - the list on the HPA site shows 35. They have some imaginative names :) It could really be an unbalanced problem if we needed to indentify all 35. I agree with your guess that they are Kuiper Belt members of a cell.
\n\n", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1211873": "Hi there.\n\nI think this is a pretty basic question but I wanted to ask here just in case anyone else had the same inquiry (I'm not afraid to ask the simple questions 😅). I did some Googling and have my own speculations but I always appreciate the wisdom of other Kagglers... so here we go.\n\n---\n\nProtein localization data is derived from antibody-based profiling, using immunofluorescence (ICC-IF) and confocal microscopy, and classified into 35 different organelles and fine subcellular structures.>
\n