Is anyone trying separate training of four classifiers for each cell line?
\n\nBased on a biological rationale that a single siRNA treatment can cause different effects in different cell lines, I roughly tried to train four separate models for each cell line and got a comparable result (LB 0.44) as a model trained with the whole dataset at once (LB 0.46) with some tricks applied :). Of note, it converged significantly faster!
\n\nActually, I'm surprised that the single unified model trained with all the data can do better than four separate models and wondering if I could conclude that each siRNA has the same effect for all the different cell lines. I'd appreciate your biological or machine learning-based insights!
", "messages": [ { "id": "578918", "postDate": "07/18/2019 09:36:18", "content": "Is anyone trying separate training of four classifiers for each cell line?
\n\nBased on a biological rationale that a single siRNA treatment can cause different effects in different cell lines, I roughly tried to train four separate models for each cell line and got a comparable result (LB 0.44) as a model trained with the whole dataset at once (LB 0.46) with some tricks applied :). Of note, it converged significantly faster!
\n\nActually, I'm surprised that the single unified model trained with all the data can do better than four separate models and wondering if I could conclude that each siRNA has the same effect for all the different cell lines. I'd appreciate your biological or machine learning-based insights!
", "rawMarkdown": "Is anyone trying separate training of four classifiers for each cell line?\n\nBased on a biological rationale that a single siRNA treatment can cause different effects in different cell lines, I roughly tried to train four separate models for each cell line and got a comparable result (LB 0.44) as a model trained with the whole dataset at once (LB 0.46) with some tricks applied :). Of note, it converged significantly faster!\n\nActually, I'm surprised that the single unified model trained with all the data can do better than four separate models and wondering if I could conclude that each siRNA has the same effect for all the different cell lines. I'd appreciate your biological or machine learning-based insights!", "votes": null }, { "id": "578955", "postDate": "07/18/2019 10:41:49", "content": "Very good hint, thanks!
", "rawMarkdown": "Very good hint, thanks!", "votes": null }, { "id": "579112", "postDate": "07/18/2019 14:05:21", "content": "I wanted to try to train different models for different cell lines but then I thought that the number of examples per class was not enough (within each cell line). Interesting, though!
", "rawMarkdown": "I wanted to try to train different models for different cell lines but then I thought that the number of examples per class was not enough (within each cell line). Interesting, though!", "votes": null }, { "id": "579179", "postDate": "07/18/2019 15:25:48", "content": "That's a good point, especially for U2OS cell line! For that reason now I'm giving metric learning a try, hoping it works out with the scarce samples. Also, I think a clever image augmentation strategy will give some improvement for the model, how do you think about that?
", "rawMarkdown": "That's a good point, especially for U2OS cell line! For that reason now I'm giving metric learning a try, hoping it works out with the scarce samples. Also, I think a clever image augmentation strategy will give some improvement for the model, how do you think about that?", "votes": null }, { "id": "579186", "postDate": "07/18/2019 15:32:31", "content": "It seems that a number of people (top entries) are using metric learning, but quite frankly I have literally no idea what that is! I'm still trying to train a normal classifier. Yes, definitely: I still believe doing per cell line training is a good strategy, in combination with a good data augmentation strategy...
", "rawMarkdown": "It seems that a number of people (top entries) are using metric learning, but quite frankly I have literally no idea what that is! I'm still trying to train a *normal* classifier. Yes, definitely: I still believe doing per cell line training is a good strategy, in combination with a good data augmentation strategy...", "votes": null }, { "id": "579495", "postDate": "07/18/2019 22:05:32", "content": "I also still needs to learn what is metric learning. But is seems you don't need to use it to get above 0.6 LB
", "rawMarkdown": "I also still needs to learn what is metric learning. But is seems you don't need to use it to get above 0.6 LB", "votes": null }, { "id": "579662", "postDate": "07/19/2019 04:34:17", "content": "@yuval6967 Interesting, may I ask whether you're using pretrained models? Thank you.
", "rawMarkdown": "yuval6967 Interesting, may I ask whether you're using pretrained models? Thank you.", "votes": null }, { "id": "579824", "postDate": "07/19/2019 09:11:23", "content": "yes. densnet121, 6 input channels.
", "rawMarkdown": "yes. densnet121, 6 input channels.", "votes": null }, { "id": "579865", "postDate": "07/19/2019 10:29:48", "content": "Thanks for sharing! I think I should keep on trying without metric learning as much as I can, and then see if applying it shows some improvements, then :)
", "rawMarkdown": "Thanks for sharing! I think I should keep on trying without metric learning as much as I can, and then see if applying it shows some improvements, then :)", "votes": null }, { "id": "581077", "postDate": "07/21/2019 11:20:25", "content": "@yuval6967 Can you tell us for how many epochs you're training it? I'm testing DenseNet121 (PyTorch) with 6 channels, and after 3 epochs I'm still at 0.0025 accuracy on a validation set (0.8/0.2)...
", "rawMarkdown": "yuval6967 Can you tell us for how many epochs you're training it? I'm testing DenseNet121 (PyTorch) with 6 channels, and after 3 epochs I'm still at 0.0025 accuracy on a validation set (0.8/0.2)...", "votes": null }, { "id": "596307", "postDate": "08/10/2019 13:03:37", "content": "hi @apap950419, if you dont mind sharing - when you speak of tricks, do you refer to using the controls? it is fine if you are unable to disclose. Thanks! :)
", "rawMarkdown": "hi @apap950419, if you dont mind sharing - when you speak of tricks, do you refer to using the controls? it is fine if you are unable to disclose. Thanks! :)", "votes": null }, { "id": "596315", "postDate": "08/10/2019 13:13:21", "content": "Hi. I'm still wondering how we can incorporate controls to the network training. My trick was, to exploit the fact that each siRNA appears at most once for each experiment, as you may have noticed for now, too :)
", "rawMarkdown": "Hi. I'm still wondering how we can incorporate controls to the network training. My trick was, to exploit the fact that each siRNA appears at most once for each experiment, as you may have noticed for now, too :)", "votes": null }, { "id": "596319", "postDate": "08/10/2019 13:16:06", "content": "How about you, @wjshenggggg? Any ideas for using controls?
", "rawMarkdown": "How about you, @wjshenggggg? Any ideas for using controls?", "votes": null }, { "id": "596391", "postDate": "08/10/2019 15:25:43", "content": "aha i see. that's nice! i have been taking in a random control image for every image and that is not working at all. i wonder are top teams using controls. I notice that once people surpass the 0.9 score, they keep improving... :( (People in top LB keep changing, hope they can provide some hints).
", "rawMarkdown": "aha i see. that's nice! i have been taking in a random control image for every image and that is not working at all. i wonder are top teams using controls. I notice that once people surpass the 0.9 score, they keep improving... :( (People in top LB keep changing, hope they can provide some hints).", "votes": null }, { "id": "596689", "postDate": "08/11/2019 06:15:57", "content": "Haha I'm also curious how the people got >0.9 LB, but not trying to be in hurry. There are still 2 months left for this competition, and I'm pretty sure that we can get at least >0.8 or >0.85 if we keep try to improve our models :). By the way, I also tried an approach (similar as yours) as followings: take one random control image (on the same plate) along with treatment images -> separately feed the two images to CNN feature extractor -> subtracting treatment feature from control feature -> feed the 'feature difference' to fc classifier. But could not get improvement with this model.
", "rawMarkdown": "Haha I'm also curious how the people got >0.9 LB, but not trying to be in hurry. There are still 2 months left for this competition, and I'm pretty sure that we can get at least >0.8 or >0.85 if we keep try to improve our models :). By the way, I also tried an approach (similar as yours) as followings: take one random control image (on the same plate) along with treatment images -> separately feed the two images to CNN feature extractor -> subtracting treatment feature from control feature -> feed the 'feature difference' to fc classifier. But could not get improvement with this model.", "votes": null }, { "id": "596754", "postDate": "08/11/2019 08:07:07", "content": "> ... subtracting treatment feature from control feature ...
\n\nWhy did you subtract? Let the network decide what is the optimal way to combine the two.
", "rawMarkdown": "> ... subtracting treatment feature from control feature ... \n\nWhy did you subtract? Let the network decide what is the optimal way to combine the two.", "votes": null }, { "id": "596842", "postDate": "08/11/2019 11:54:09", "content": "more then 30 epochs
", "rawMarkdown": "more then 30 epochs", "votes": null }, { "id": "596846", "postDate": "08/11/2019 11:58:01", "content": "Up to >0.9 we didn't use control at all, we are started to use it only recently.
", "rawMarkdown": "Up to >0.9 we didn't use control at all, we are started to use it only recently.", "votes": null }, { "id": "596881", "postDate": "08/11/2019 12:52:09", "content": "@liorzi Yes, definitely we can let the network decide the way to use control images! (for example by just concatenating the two vectors and feeding them to the following layers) I simply tried subtracting the two vectors (features of treatment image, and features of negative control features) to explicitly inform the network that the 'differences' of features are important. Maybe I should try the concatenating way. Thanks for your comment!
", "rawMarkdown": "liorzi Yes, definitely we can let the network decide the way to use control images! (for example by just concatenating the two vectors and feeding them to the following layers) I simply tried subtracting the two vectors (features of treatment image, and features of negative control features) to explicitly inform the network that the 'differences' of features are important. Maybe I should try the concatenating way. Thanks for your comment!", "votes": null }, { "id": "596887", "postDate": "08/11/2019 12:55:32", "content": "@yuval6967 It's surprising, and thanks for the tip! I should keep on trying to improve my baseline model without controls. Nowadays I'm realizing that the normalization method has substantial effects on the performance of my models, and working on improving normalization methods. At first, I thought that plate-based normalization (i.e. normalizing pixel values with plate-mean and plate-standard deviation) should outperform simple well-by-well normalization (i.e. normalizing pixel values with mean and standard deviation of only that image) since it preserves the relative differences of pixel intensities across the wells in a plate. However today I found the latter works better... :) Also, I think that I should deal with non-uniform illumination of images which arises as an artifact of microscopes, somehow. So, could you share some of your ideas about image normalization, if you don't mind?
", "rawMarkdown": "yuval6967 It's surprising, and thanks for the tip! I should keep on trying to improve my baseline model without controls. Nowadays I'm realizing that the normalization method has substantial effects on the performance of my models, and working on improving normalization methods. At first, I thought that plate-based normalization (i.e. normalizing pixel values with plate-mean and plate-standard deviation) should outperform simple well-by-well normalization (i.e. normalizing pixel values with mean and standard deviation of only that image) since it preserves the relative differences of pixel intensities across the wells in a plate. However today I found the latter works better... :) Also, I think that I should [deal with non-uniform illumination of images](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwic7s_P8frjAhUKvZQKHa7dBqkQFjAAegQIABAC&url=http%3A%2F%2Fd1zymp9ayga15t.cloudfront.net%2Fcontent%2FExampleIlluminationCorrection_Tutorial.pdf&usg=AOvVaw1QIhtZUThEfkQ8CeJJBAE1) which arises as an artifact of microscopes, somehow. So, could you share some of your ideas about image normalization, if you don't mind?", "votes": null }, { "id": "596921", "postDate": "08/11/2019 14:06:07", "content": "Thanks yuval. I am starting to doubt myself... you are treating this as a normal image classification problem - did not apply any trick apart from the masking when do your submission and you manage 0.9. So i guess i should continue searching for an optimal learning rate since my pretrained network with no metric learning can never converge or achieve a validation accuracy of > 0.1.
\n\nhey LiorZ: Another thing is that there are ~30 control image and it is fairly slow to account for all control images if we do the concatenating way
\n\ndohlee: Thanks for sharing. But would it be better to normalize with respect to channels? Such as using the global stats provided in the starter code. I also tried normalizing by dividing 255 or subtracting 127.5 and then dividing by 255. All did not work.
", "rawMarkdown": "Thanks yuval. I am starting to doubt myself... you are treating this as a normal image classification problem - did not apply any trick apart from the masking when do your submission and you manage 0.9. So i guess i should continue searching for an optimal learning rate since my pretrained network with no metric learning can never converge or achieve a validation accuracy of > 0.1.\n\nhey LiorZ: Another thing is that there are ~30 control image and it is fairly slow to account for all control images if we do the concatenating way\n\ndohlee: Thanks for sharing. But would it be better to normalize with respect to channels? Such as using the global stats provided in the starter code. I also tried normalizing by dividing 255 or subtracting 127.5 and then dividing by 255. All did not work.", "votes": null }, { "id": "596922", "postDate": "08/11/2019 14:06:59", "content": "also, I am rooting for yuval's team to be in top 3! :)
", "rawMarkdown": "also, I am rooting for yuval's team to be in top 3! :)", "votes": null }, { "id": "596929", "postDate": "08/11/2019 14:24:04", "content": "@wjshenggggg Oh, thanks for pointing that out. I surely did per-channel normalization with pixel stats given in the data :)
", "rawMarkdown": "wjshenggggg Oh, thanks for pointing that out. I surely did per-channel normalization with pixel stats given in the data :)", "votes": null }, { "id": "597002", "postDate": "08/11/2019 16:33:07", "content": "@wjshenggggg just to make sure you are not mislead by my comments, I am not treating it as a normal image classification challenge, we just didn't use the controls until we were above 0.9, and didn't use any special cell profiling software up to now.\nThis challenge has it's own special features and we try to find how to exploit these features to improve our solution. @zaharch , my partner, found the 277 issue when he tried to learn more about the data, and we immediately used it (while alerting the organizers, as we considered it as a leak). Like in every competition, the key for top scores is finding the special angle to attack to get the best solution.
", "rawMarkdown": "wjshenggggg just to make sure you are not mislead by my comments, I am not treating it as a normal image classification challenge, we just didn't use the controls until we were above 0.9, and didn't use any special cell profiling software up to now.\nThis challenge has it's own special features and we try to find how to exploit these features to improve our solution. @zaharch , my partner, found the 277 issue when he tried to learn more about the data, and we immediately used it (while alerting the organizers, as we considered it as a leak). Like in every competition, the key for top scores is finding the special angle to attack to get the best solution.", "votes": null }, { "id": "597008", "postDate": "08/11/2019 16:39:35", "content": "@apap950419 the truth is, I didn't really think about normalization until now. Just used the pixel_stats data and calculated the average of mean ,std per plate, channel and experiment. I must look at it again, maybe I'm also missing something.
", "rawMarkdown": "apap950419 the truth is, I didn't really think about normalization until now. Just used the pixel_stats data and calculated the average of mean ,std per plate, channel and experiment. I must look at it again, maybe I'm also missing something.", "votes": null }, { "id": "597010", "postDate": "08/11/2019 16:41:20", "content": "Hi yuval, thanks for pointing out! I guess my point is we can treat it as a normal image classification task without considering controls and that would allow us to achieve somewhat 0.6 in LB without applying leak as you mentioned weeks ago. Using controls will boost the score further, is that right? Thanks, once again!
", "rawMarkdown": "Hi yuval, thanks for pointing out! I guess my point is we can treat it as a normal image classification task without considering controls and that would allow us to achieve somewhat 0.6 in LB without applying leak as you mentioned weeks ago. Using controls will boost the score further, is that right? Thanks, once again!", "votes": null }, { "id": "597052", "postDate": "08/11/2019 18:02:02", "content": "We had some improvement using control, but I believe we have only scratched the surface yet.
", "rawMarkdown": "We had some improvement using control, but I believe we have only scratched the surface yet.", "votes": null }, { "id": "608060", "postDate": "08/26/2019 09:13:33", "content": "@yuval6967 I'm trying to normalize the images as you said, taking the stats from a Pandas DataFrame. The problem is that it becomes extremely slow. Are you also storing them in a DataFrame?
", "rawMarkdown": "yuval6967 I'm trying to normalize the images as you said, taking the stats from a Pandas DataFrame. The problem is that it becomes extremely slow. Are you also storing them in a DataFrame?", "votes": null }, { "id": "608078", "postDate": "08/26/2019 09:54:02", "content": "I copied the data to a numpy array.\nDidn't see any performance issues, (and as the size of the panda's dataframe is not very big, I don't think you should see it even if you use the data directly from there)
", "rawMarkdown": "I copied the data to a numpy array.\nDidn't see any performance issues, (and as the size of the panda's dataframe is not very big, I don't think you should see it even if you use the data directly from there)", "votes": null }, { "id": "608082", "postDate": "08/26/2019 10:00:43", "content": "Thanks. So probably I'm doing something wrong.
", "rawMarkdown": "Thanks. So probably I'm doing something wrong.", "votes": null } ], "comments": [ { "id": 578955, "author_name": "narsil", "author_url": "", "post_date": "07/18/2019 10:41:49", "content": "Very good hint, thanks!
", "votes": null, "replies": [] }, { "id": 579112, "author_name": "lorenzofabbri92", "author_url": "", "post_date": "07/18/2019 14:05:21", "content": "I wanted to try to train different models for different cell lines but then I thought that the number of examples per class was not enough (within each cell line). Interesting, though!
", "votes": null, "replies": [ { "id": 579179, "author_name": "apap950419", "author_url": "", "post_date": "07/18/2019 15:25:48", "content": "That's a good point, especially for U2OS cell line! For that reason now I'm giving metric learning a try, hoping it works out with the scarce samples. Also, I think a clever image augmentation strategy will give some improvement for the model, how do you think about that?
", "votes": null, "replies": [] }, { "id": 579186, "author_name": "lorenzofabbri92", "author_url": "", "post_date": "07/18/2019 15:32:31", "content": "It seems that a number of people (top entries) are using metric learning, but quite frankly I have literally no idea what that is! I'm still trying to train a normal classifier. Yes, definitely: I still believe doing per cell line training is a good strategy, in combination with a good data augmentation strategy...
", "votes": null, "replies": [] }, { "id": 579495, "author_name": "yuval6967", "author_url": "", "post_date": "07/18/2019 22:05:32", "content": "I also still needs to learn what is metric learning. But is seems you don't need to use it to get above 0.6 LB
", "votes": null, "replies": [] }, { "id": 579662, "author_name": "lorenzofabbri92", "author_url": "", "post_date": "07/19/2019 04:34:17", "content": "@yuval6967 Interesting, may I ask whether you're using pretrained models? Thank you.
", "votes": null, "replies": [] }, { "id": 579824, "author_name": "yuval6967", "author_url": "", "post_date": "07/19/2019 09:11:23", "content": "yes. densnet121, 6 input channels.
", "votes": null, "replies": [] }, { "id": 579865, "author_name": "apap950419", "author_url": "", "post_date": "07/19/2019 10:29:48", "content": "Thanks for sharing! I think I should keep on trying without metric learning as much as I can, and then see if applying it shows some improvements, then :)
", "votes": null, "replies": [] }, { "id": 581077, "author_name": "lorenzofabbri92", "author_url": "", "post_date": "07/21/2019 11:20:25", "content": "@yuval6967 Can you tell us for how many epochs you're training it? I'm testing DenseNet121 (PyTorch) with 6 channels, and after 3 epochs I'm still at 0.0025 accuracy on a validation set (0.8/0.2)...
", "votes": null, "replies": [] }, { "id": 596842, "author_name": "yuval6967", "author_url": "", "post_date": "08/11/2019 11:54:09", "content": "more then 30 epochs
", "votes": null, "replies": [] } ] }, { "id": 596307, "author_name": "wjshenggggg", "author_url": "", "post_date": "08/10/2019 13:03:37", "content": "hi @apap950419, if you dont mind sharing - when you speak of tricks, do you refer to using the controls? it is fine if you are unable to disclose. Thanks! :)
", "votes": null, "replies": [ { "id": 596315, "author_name": "apap950419", "author_url": "", "post_date": "08/10/2019 13:13:21", "content": "Hi. I'm still wondering how we can incorporate controls to the network training. My trick was, to exploit the fact that each siRNA appears at most once for each experiment, as you may have noticed for now, too :)
", "votes": null, "replies": [] }, { "id": 596319, "author_name": "apap950419", "author_url": "", "post_date": "08/10/2019 13:16:06", "content": "How about you, @wjshenggggg? Any ideas for using controls?
", "votes": null, "replies": [] }, { "id": 596391, "author_name": "wjshenggggg", "author_url": "", "post_date": "08/10/2019 15:25:43", "content": "aha i see. that's nice! i have been taking in a random control image for every image and that is not working at all. i wonder are top teams using controls. I notice that once people surpass the 0.9 score, they keep improving... :( (People in top LB keep changing, hope they can provide some hints).
", "votes": null, "replies": [] }, { "id": 596689, "author_name": "apap950419", "author_url": "", "post_date": "08/11/2019 06:15:57", "content": "Haha I'm also curious how the people got >0.9 LB, but not trying to be in hurry. There are still 2 months left for this competition, and I'm pretty sure that we can get at least >0.8 or >0.85 if we keep try to improve our models :). By the way, I also tried an approach (similar as yours) as followings: take one random control image (on the same plate) along with treatment images -> separately feed the two images to CNN feature extractor -> subtracting treatment feature from control feature -> feed the 'feature difference' to fc classifier. But could not get improvement with this model.
", "votes": null, "replies": [] }, { "id": 596754, "author_name": "liorzi", "author_url": "", "post_date": "08/11/2019 08:07:07", "content": "> ... subtracting treatment feature from control feature ...
\n\nWhy did you subtract? Let the network decide what is the optimal way to combine the two.
", "votes": null, "replies": [] }, { "id": 596846, "author_name": "yuval6967", "author_url": "", "post_date": "08/11/2019 11:58:01", "content": "Up to >0.9 we didn't use control at all, we are started to use it only recently.
", "votes": null, "replies": [] }, { "id": 596881, "author_name": "apap950419", "author_url": "", "post_date": "08/11/2019 12:52:09", "content": "@liorzi Yes, definitely we can let the network decide the way to use control images! (for example by just concatenating the two vectors and feeding them to the following layers) I simply tried subtracting the two vectors (features of treatment image, and features of negative control features) to explicitly inform the network that the 'differences' of features are important. Maybe I should try the concatenating way. Thanks for your comment!
", "votes": null, "replies": [] }, { "id": 596887, "author_name": "apap950419", "author_url": "", "post_date": "08/11/2019 12:55:32", "content": "@yuval6967 It's surprising, and thanks for the tip! I should keep on trying to improve my baseline model without controls. Nowadays I'm realizing that the normalization method has substantial effects on the performance of my models, and working on improving normalization methods. At first, I thought that plate-based normalization (i.e. normalizing pixel values with plate-mean and plate-standard deviation) should outperform simple well-by-well normalization (i.e. normalizing pixel values with mean and standard deviation of only that image) since it preserves the relative differences of pixel intensities across the wells in a plate. However today I found the latter works better... :) Also, I think that I should deal with non-uniform illumination of images which arises as an artifact of microscopes, somehow. So, could you share some of your ideas about image normalization, if you don't mind?
", "votes": null, "replies": [] }, { "id": 596921, "author_name": "wjshenggggg", "author_url": "", "post_date": "08/11/2019 14:06:07", "content": "Thanks yuval. I am starting to doubt myself... you are treating this as a normal image classification problem - did not apply any trick apart from the masking when do your submission and you manage 0.9. So i guess i should continue searching for an optimal learning rate since my pretrained network with no metric learning can never converge or achieve a validation accuracy of > 0.1.
\n\nhey LiorZ: Another thing is that there are ~30 control image and it is fairly slow to account for all control images if we do the concatenating way
\n\ndohlee: Thanks for sharing. But would it be better to normalize with respect to channels? Such as using the global stats provided in the starter code. I also tried normalizing by dividing 255 or subtracting 127.5 and then dividing by 255. All did not work.
", "votes": null, "replies": [] }, { "id": 596922, "author_name": "wjshenggggg", "author_url": "", "post_date": "08/11/2019 14:06:59", "content": "also, I am rooting for yuval's team to be in top 3! :)
", "votes": null, "replies": [] }, { "id": 596929, "author_name": "apap950419", "author_url": "", "post_date": "08/11/2019 14:24:04", "content": "@wjshenggggg Oh, thanks for pointing that out. I surely did per-channel normalization with pixel stats given in the data :)
", "votes": null, "replies": [] }, { "id": 597002, "author_name": "yuval6967", "author_url": "", "post_date": "08/11/2019 16:33:07", "content": "@wjshenggggg just to make sure you are not mislead by my comments, I am not treating it as a normal image classification challenge, we just didn't use the controls until we were above 0.9, and didn't use any special cell profiling software up to now.\nThis challenge has it's own special features and we try to find how to exploit these features to improve our solution. @zaharch , my partner, found the 277 issue when he tried to learn more about the data, and we immediately used it (while alerting the organizers, as we considered it as a leak). Like in every competition, the key for top scores is finding the special angle to attack to get the best solution.
", "votes": null, "replies": [] }, { "id": 597008, "author_name": "yuval6967", "author_url": "", "post_date": "08/11/2019 16:39:35", "content": "@apap950419 the truth is, I didn't really think about normalization until now. Just used the pixel_stats data and calculated the average of mean ,std per plate, channel and experiment. I must look at it again, maybe I'm also missing something.
", "votes": null, "replies": [] }, { "id": 597010, "author_name": "wjshenggggg", "author_url": "", "post_date": "08/11/2019 16:41:20", "content": "Hi yuval, thanks for pointing out! I guess my point is we can treat it as a normal image classification task without considering controls and that would allow us to achieve somewhat 0.6 in LB without applying leak as you mentioned weeks ago. Using controls will boost the score further, is that right? Thanks, once again!
", "votes": null, "replies": [] }, { "id": 597052, "author_name": "yuval6967", "author_url": "", "post_date": "08/11/2019 18:02:02", "content": "We had some improvement using control, but I believe we have only scratched the surface yet.
", "votes": null, "replies": [] }, { "id": 608060, "author_name": "lorenzofabbri92", "author_url": "", "post_date": "08/26/2019 09:13:33", "content": "@yuval6967 I'm trying to normalize the images as you said, taking the stats from a Pandas DataFrame. The problem is that it becomes extremely slow. Are you also storing them in a DataFrame?
", "votes": null, "replies": [] }, { "id": 608078, "author_name": "yuval6967", "author_url": "", "post_date": "08/26/2019 09:54:02", "content": "I copied the data to a numpy array.\nDidn't see any performance issues, (and as the size of the panda's dataframe is not very big, I don't think you should see it even if you use the data directly from there)
", "votes": null, "replies": [] }, { "id": 608082, "author_name": "lorenzofabbri92", "author_url": "", "post_date": "08/26/2019 10:00:43", "content": "Thanks. So probably I'm doing something wrong.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "578918": "Is anyone trying separate training of four classifiers for each cell line?\n\nBased on a biological rationale that a single siRNA treatment can cause different effects in different cell lines, I roughly tried to train four separate models for each cell line and got a comparable result (LB 0.44) as a model trained with the whole dataset at once (LB 0.46) with some tricks applied :). Of note, it converged significantly faster!\n\nActually, I'm surprised that the single unified model trained with all the data can do better than four separate models and wondering if I could conclude that each siRNA has the same effect for all the different cell lines. I'd appreciate your biological or machine learning-based insights!", "578955": "Very good hint, thanks!", "579112": "I wanted to try to train different models for different cell lines but then I thought that the number of examples per class was not enough (within each cell line). Interesting, though!", "579179": "That's a good point, especially for U2OS cell line! For that reason now I'm giving metric learning a try, hoping it works out with the scarce samples. Also, I think a clever image augmentation strategy will give some improvement for the model, how do you think about that?", "579186": "It seems that a number of people (top entries) are using metric learning, but quite frankly I have literally no idea what that is! I'm still trying to train a *normal* classifier. Yes, definitely: I still believe doing per cell line training is a good strategy, in combination with a good data augmentation strategy...", "579495": "I also still needs to learn what is metric learning. But is seems you don't need to use it to get above 0.6 LB", "579662": "yuval6967 Interesting, may I ask whether you're using pretrained models? Thank you.", "579824": "yes. densnet121, 6 input channels.", "579865": "Thanks for sharing! I think I should keep on trying without metric learning as much as I can, and then see if applying it shows some improvements, then :)", "581077": "yuval6967 Can you tell us for how many epochs you're training it? I'm testing DenseNet121 (PyTorch) with 6 channels, and after 3 epochs I'm still at 0.0025 accuracy on a validation set (0.8/0.2)...", "596307": "hi @apap950419, if you dont mind sharing - when you speak of tricks, do you refer to using the controls? it is fine if you are unable to disclose. Thanks! :)", "596315": "Hi. I'm still wondering how we can incorporate controls to the network training. My trick was, to exploit the fact that each siRNA appears at most once for each experiment, as you may have noticed for now, too :)", "596319": "How about you, @wjshenggggg? Any ideas for using controls?", "596391": "aha i see. that's nice! i have been taking in a random control image for every image and that is not working at all. i wonder are top teams using controls. I notice that once people surpass the 0.9 score, they keep improving... :( (People in top LB keep changing, hope they can provide some hints).", "596689": "Haha I'm also curious how the people got >0.9 LB, but not trying to be in hurry. There are still 2 months left for this competition, and I'm pretty sure that we can get at least >0.8 or >0.85 if we keep try to improve our models :). By the way, I also tried an approach (similar as yours) as followings: take one random control image (on the same plate) along with treatment images -> separately feed the two images to CNN feature extractor -> subtracting treatment feature from control feature -> feed the 'feature difference' to fc classifier. But could not get improvement with this model.", "596754": "> ... subtracting treatment feature from control feature ... \n\nWhy did you subtract? Let the network decide what is the optimal way to combine the two.", "596842": "more then 30 epochs", "596846": "Up to >0.9 we didn't use control at all, we are started to use it only recently.", "596881": "liorzi Yes, definitely we can let the network decide the way to use control images! (for example by just concatenating the two vectors and feeding them to the following layers) I simply tried subtracting the two vectors (features of treatment image, and features of negative control features) to explicitly inform the network that the 'differences' of features are important. Maybe I should try the concatenating way. Thanks for your comment!", "596887": "yuval6967 It's surprising, and thanks for the tip! I should keep on trying to improve my baseline model without controls. Nowadays I'm realizing that the normalization method has substantial effects on the performance of my models, and working on improving normalization methods. At first, I thought that plate-based normalization (i.e. normalizing pixel values with plate-mean and plate-standard deviation) should outperform simple well-by-well normalization (i.e. normalizing pixel values with mean and standard deviation of only that image) since it preserves the relative differences of pixel intensities across the wells in a plate. However today I found the latter works better... :) Also, I think that I should [deal with non-uniform illumination of images](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwic7s_P8frjAhUKvZQKHa7dBqkQFjAAegQIABAC&url=http%3A%2F%2Fd1zymp9ayga15t.cloudfront.net%2Fcontent%2FExampleIlluminationCorrection_Tutorial.pdf&usg=AOvVaw1QIhtZUThEfkQ8CeJJBAE1) which arises as an artifact of microscopes, somehow. So, could you share some of your ideas about image normalization, if you don't mind?", "596921": "Thanks yuval. I am starting to doubt myself... you are treating this as a normal image classification problem - did not apply any trick apart from the masking when do your submission and you manage 0.9. So i guess i should continue searching for an optimal learning rate since my pretrained network with no metric learning can never converge or achieve a validation accuracy of > 0.1.\n\nhey LiorZ: Another thing is that there are ~30 control image and it is fairly slow to account for all control images if we do the concatenating way\n\ndohlee: Thanks for sharing. But would it be better to normalize with respect to channels? Such as using the global stats provided in the starter code. I also tried normalizing by dividing 255 or subtracting 127.5 and then dividing by 255. All did not work.", "596922": "also, I am rooting for yuval's team to be in top 3! :)", "596929": "wjshenggggg Oh, thanks for pointing that out. I surely did per-channel normalization with pixel stats given in the data :)", "597002": "wjshenggggg just to make sure you are not mislead by my comments, I am not treating it as a normal image classification challenge, we just didn't use the controls until we were above 0.9, and didn't use any special cell profiling software up to now.\nThis challenge has it's own special features and we try to find how to exploit these features to improve our solution. @zaharch , my partner, found the 277 issue when he tried to learn more about the data, and we immediately used it (while alerting the organizers, as we considered it as a leak). Like in every competition, the key for top scores is finding the special angle to attack to get the best solution.", "597008": "apap950419 the truth is, I didn't really think about normalization until now. Just used the pixel_stats data and calculated the average of mean ,std per plate, channel and experiment. I must look at it again, maybe I'm also missing something.", "597010": "Hi yuval, thanks for pointing out! I guess my point is we can treat it as a normal image classification task without considering controls and that would allow us to achieve somewhat 0.6 in LB without applying leak as you mentioned weeks ago. Using controls will boost the score further, is that right? Thanks, once again!", "597052": "We had some improvement using control, but I believe we have only scratched the surface yet.", "608060": "yuval6967 I'm trying to normalize the images as you said, taking the stats from a Pandas DataFrame. The problem is that it becomes extremely slow. Are you also storing them in a DataFrame?", "608078": "I copied the data to a numpy array.\nDidn't see any performance issues, (and as the size of the panda's dataframe is not very big, I don't think you should see it even if you use the data directly from there)", "608082": "Thanks. So probably I'm doing something wrong." }, "source": "meta" }