Hi guys! The competition is about to finish soon, and after few weeks break I found a significant change in LB.
\n\nI wonder if you can share your approach (maybe not detailed yet at this stage, but simple answer - metric learning or classification) for statistic's collection.
\n\nFrom my side: all my \"metric learning\"-based approaches were slightly better than vanilla classification.
\n\nA lot of people try to convince me, that in 2019 we should not use CrossEntropy anymore, because in every task metric learning is working better. It would be useful to check if this hypothesis is True for biology data as well:)
", "messages": [ { "id": "630378", "postDate": "09/20/2019 06:33:46", "content": "Hi guys! The competition is about to finish soon, and after few weeks break I found a significant change in LB.
\n\nI wonder if you can share your approach (maybe not detailed yet at this stage, but simple answer - metric learning or classification) for statistic's collection.
\n\nFrom my side: all my \"metric learning\"-based approaches were slightly better than vanilla classification.
\n\nA lot of people try to convince me, that in 2019 we should not use CrossEntropy anymore, because in every task metric learning is working better. It would be useful to check if this hypothesis is True for biology data as well:)
", "rawMarkdown": "Hi guys! The competition is about to finish soon, and after few weeks break I found a significant change in LB. \n\nI wonder if you can share your approach (maybe not detailed yet at this stage, but simple answer - **metric learning** or **classification**) for statistic's collection. \n\nFrom my side: all my \"metric learning\"-based approaches were slightly better than vanilla classification.\n\nA lot of people try to convince me, that **in 2019** we should not use **CrossEntropy** anymore, because in every task *metric learning is working better.* It would be useful to check if this hypothesis is True for biology data as well:)", "votes": null }, { "id": "630474", "postDate": "09/20/2019 08:52:19", "content": "We are still still doing plain vanilla classification. Metric Learning was slightly worse.
\n\nIt's amazing how \"little\" tricks in training procedure can change the score by 0.1.
", "rawMarkdown": "We are still still doing plain vanilla classification. Metric Learning was slightly worse.\n\nIt's amazing how \"little\" tricks in training procedure can change the score by 0.1.", "votes": null }, { "id": "630491", "postDate": "09/20/2019 09:17:18", "content": "@narsil thanks for sharing! Your score is really good for vanilla classification, so it looks like your tricks are better than mine:)
", "rawMarkdown": "narsil thanks for sharing! Your score is really good for vanilla classification, so it looks like your tricks are better than mine:)", "votes": null }, { "id": "630988", "postDate": "09/21/2019 05:59:39", "content": "Frankly speaking, I am quite astonished that some of those tricks do work. This whole competition I feel like from the level of 0.4LB to 0.9LB we didn't change THAT much in the model or training procedure (maybe apart from the leak reported kindly by @zaharch ). The devil is in the details. They can make a real difference.
\n\nI like this competition because the score of the top teams is higher by ~5-10pp from the rest of the pack - which is a difference that would matter in a business setting. (as compared to majority of competitions, where we are squeezing fourth decimal point of accuracy)
", "rawMarkdown": "Frankly speaking, I am quite astonished that some of those tricks do work. This whole competition I feel like from the level of 0.4LB to 0.9LB we didn't change THAT much in the model or training procedure (maybe apart from the leak reported kindly by @zaharch ). The devil is in the details. They can make a real difference. \n\nI like this competition because the score of the top teams is higher by ~5-10pp from the rest of the pack - which is a difference that would matter in a business setting. (as compared to majority of competitions, where we are squeezing fourth decimal point of accuracy)", "votes": null }, { "id": "631030", "postDate": "09/21/2019 07:50:39", "content": "@narsil, thanks for your insights. I'm also a little stuck with my current score. I jumped in only 10 days ago, still hope to try metric learning, but not sure, maybe it's better to try to find some \"tricks\" for classification. May I ask, what you mean by \"model or training procedure\"? Have you tried some special normalization techniques, or some advanced optimizers/LR schedulers? The question that I'm sure interests most of the participants, have you found the way to use controls? \nI Tried using control set for pretraining, it gave slightly better local score but slightly worse LB. Also I tried training separate models for each cell type, it gave pretty much the same result as training on whole set. I'm using quite a lot of augmentation, but no affine/distorsion. Augmentation seem to improve result, but I was really surprised how much plain normalization matters.
", "rawMarkdown": "narsil, thanks for your insights. I'm also a little stuck with my current score. I jumped in only 10 days ago, still hope to try metric learning, but not sure, maybe it's better to try to find some \"tricks\" for classification. May I ask, what you mean by \"model or training procedure\"? Have you tried some special normalization techniques, or some advanced optimizers/LR schedulers? The question that I'm sure interests most of the participants, have you found the way to use controls? \nI Tried using control set for pretraining, it gave slightly better local score but slightly worse LB. Also I tried training separate models for each cell type, it gave pretty much the same result as training on whole set. I'm using quite a lot of augmentation, but no affine/distorsion. Augmentation seem to improve result, but I was really surprised how much plain normalization matters.", "votes": null }, { "id": "631152", "postDate": "09/21/2019 13:19:37", "content": "As for me, metric learning works slightly better than cross entropy. It depends on your data processing, model etc.
", "rawMarkdown": "As for me, metric learning works slightly better than cross entropy. It depends on your data processing, model etc.", "votes": null }, { "id": "631339", "postDate": "09/21/2019 21:10:12", "content": "@analokamus cool, thanks for comment! The point is that if you are trying to solve classification as a metric learning problem it's much harder - you should not only separate classes, but to project them to so-called \"cluster\" on hyper-sphere (and get at least margin between them). And if you are good with optimizing this kind of loss function - logically you should obtain better generalization results:)
", "rawMarkdown": "analokamus cool, thanks for comment! The point is that if you are trying to solve classification as a metric learning problem it's much harder - you should not only separate classes, but to project them to so-called \"cluster\" on hyper-sphere (and get at least margin between them). And if you are good with optimizing this kind of loss function - logically you should obtain better generalization results:)", "votes": null }, { "id": "631989", "postDate": "09/23/2019 04:14:54", "content": "Would you mind sharing what details you changed boosted your score from 0.4 to 0.9? I tried several networks and tricks, but the LB was still aboult 0.3~0.4, which confused me a lot.
", "rawMarkdown": "Would you mind sharing what details you changed boosted your score from 0.4 to 0.9? I tried several networks and tricks, but the LB was still aboult 0.3~0.4, which confused me a lot.", "votes": null }, { "id": "632083", "postDate": "09/23/2019 07:56:32", "content": "@apotato, I've tried several tips written in other disucussion threads or kernels.
\n\nBelow are tips that worked for my model.\n- train longer with 2 sites\n- average 2 site predictions\n- use plate leak\n- use Hungarian aloghrithm\n- use bigger images
\n\nMore and more training seems to give better results.\nBut we don't have much time left...
", "rawMarkdown": "apotato, I've tried several tips written in other disucussion threads or kernels.\n\nBelow are tips that worked for my model.\n- train longer with 2 sites\n- average 2 site predictions\n- use plate leak\n- use Hungarian aloghrithm\n- use bigger images\n\nMore and more training seems to give better results.\nBut we don't have much time left...", "votes": null }, { "id": "633093", "postDate": "09/24/2019 12:35:26", "content": "I will share details on training procedure after the competition ends. I hope you understand :) So far we are not using controls in any way
", "rawMarkdown": "I will share details on training procedure after the competition ends. I hope you understand :) So far we are not using controls in any way", "votes": null }, { "id": "633106", "postDate": "09/24/2019 12:50:20", "content": "@narsil that's fair:)
", "rawMarkdown": "narsil that's fair:)", "votes": null } ], "comments": [ { "id": 630474, "author_name": "narsil", "author_url": "", "post_date": "09/20/2019 08:52:19", "content": "We are still still doing plain vanilla classification. Metric Learning was slightly worse.
\n\nIt's amazing how \"little\" tricks in training procedure can change the score by 0.1.
", "votes": null, "replies": [ { "id": 630491, "author_name": "alexgruzdev", "author_url": "", "post_date": "09/20/2019 09:17:18", "content": "@narsil thanks for sharing! Your score is really good for vanilla classification, so it looks like your tricks are better than mine:)
", "votes": null, "replies": [] }, { "id": 630988, "author_name": "narsil", "author_url": "", "post_date": "09/21/2019 05:59:39", "content": "Frankly speaking, I am quite astonished that some of those tricks do work. This whole competition I feel like from the level of 0.4LB to 0.9LB we didn't change THAT much in the model or training procedure (maybe apart from the leak reported kindly by @zaharch ). The devil is in the details. They can make a real difference.
\n\nI like this competition because the score of the top teams is higher by ~5-10pp from the rest of the pack - which is a difference that would matter in a business setting. (as compared to majority of competitions, where we are squeezing fourth decimal point of accuracy)
", "votes": null, "replies": [] }, { "id": 631030, "author_name": "cateek", "author_url": "", "post_date": "09/21/2019 07:50:39", "content": "@narsil, thanks for your insights. I'm also a little stuck with my current score. I jumped in only 10 days ago, still hope to try metric learning, but not sure, maybe it's better to try to find some \"tricks\" for classification. May I ask, what you mean by \"model or training procedure\"? Have you tried some special normalization techniques, or some advanced optimizers/LR schedulers? The question that I'm sure interests most of the participants, have you found the way to use controls? \nI Tried using control set for pretraining, it gave slightly better local score but slightly worse LB. Also I tried training separate models for each cell type, it gave pretty much the same result as training on whole set. I'm using quite a lot of augmentation, but no affine/distorsion. Augmentation seem to improve result, but I was really surprised how much plain normalization matters.
", "votes": null, "replies": [] }, { "id": 631989, "author_name": "apotato", "author_url": "", "post_date": "09/23/2019 04:14:54", "content": "Would you mind sharing what details you changed boosted your score from 0.4 to 0.9? I tried several networks and tricks, but the LB was still aboult 0.3~0.4, which confused me a lot.
", "votes": null, "replies": [] }, { "id": 632083, "author_name": "momi64", "author_url": "", "post_date": "09/23/2019 07:56:32", "content": "@apotato, I've tried several tips written in other disucussion threads or kernels.
\n\nBelow are tips that worked for my model.\n- train longer with 2 sites\n- average 2 site predictions\n- use plate leak\n- use Hungarian aloghrithm\n- use bigger images
\n\nMore and more training seems to give better results.\nBut we don't have much time left...
", "votes": null, "replies": [] }, { "id": 633093, "author_name": "narsil", "author_url": "", "post_date": "09/24/2019 12:35:26", "content": "I will share details on training procedure after the competition ends. I hope you understand :) So far we are not using controls in any way
", "votes": null, "replies": [] }, { "id": 633106, "author_name": "alexgruzdev", "author_url": "", "post_date": "09/24/2019 12:50:20", "content": "@narsil that's fair:)
", "votes": null, "replies": [] } ] }, { "id": 631152, "author_name": "analokamus", "author_url": "", "post_date": "09/21/2019 13:19:37", "content": "As for me, metric learning works slightly better than cross entropy. It depends on your data processing, model etc.
", "votes": null, "replies": [ { "id": 631339, "author_name": "alexgruzdev", "author_url": "", "post_date": "09/21/2019 21:10:12", "content": "@analokamus cool, thanks for comment! The point is that if you are trying to solve classification as a metric learning problem it's much harder - you should not only separate classes, but to project them to so-called \"cluster\" on hyper-sphere (and get at least margin between them). And if you are good with optimizing this kind of loss function - logically you should obtain better generalization results:)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "630378": "Hi guys! The competition is about to finish soon, and after few weeks break I found a significant change in LB. \n\nI wonder if you can share your approach (maybe not detailed yet at this stage, but simple answer - **metric learning** or **classification**) for statistic's collection. \n\nFrom my side: all my \"metric learning\"-based approaches were slightly better than vanilla classification.\n\nA lot of people try to convince me, that **in 2019** we should not use **CrossEntropy** anymore, because in every task *metric learning is working better.* It would be useful to check if this hypothesis is True for biology data as well:)", "630474": "We are still still doing plain vanilla classification. Metric Learning was slightly worse.\n\nIt's amazing how \"little\" tricks in training procedure can change the score by 0.1.", "630491": "narsil thanks for sharing! Your score is really good for vanilla classification, so it looks like your tricks are better than mine:)", "630988": "Frankly speaking, I am quite astonished that some of those tricks do work. This whole competition I feel like from the level of 0.4LB to 0.9LB we didn't change THAT much in the model or training procedure (maybe apart from the leak reported kindly by @zaharch ). The devil is in the details. They can make a real difference. \n\nI like this competition because the score of the top teams is higher by ~5-10pp from the rest of the pack - which is a difference that would matter in a business setting. (as compared to majority of competitions, where we are squeezing fourth decimal point of accuracy)", "631030": "narsil, thanks for your insights. I'm also a little stuck with my current score. I jumped in only 10 days ago, still hope to try metric learning, but not sure, maybe it's better to try to find some \"tricks\" for classification. May I ask, what you mean by \"model or training procedure\"? Have you tried some special normalization techniques, or some advanced optimizers/LR schedulers? The question that I'm sure interests most of the participants, have you found the way to use controls? \nI Tried using control set for pretraining, it gave slightly better local score but slightly worse LB. Also I tried training separate models for each cell type, it gave pretty much the same result as training on whole set. I'm using quite a lot of augmentation, but no affine/distorsion. Augmentation seem to improve result, but I was really surprised how much plain normalization matters.", "631152": "As for me, metric learning works slightly better than cross entropy. It depends on your data processing, model etc.", "631339": "analokamus cool, thanks for comment! The point is that if you are trying to solve classification as a metric learning problem it's much harder - you should not only separate classes, but to project them to so-called \"cluster\" on hyper-sphere (and get at least margin between them). And if you are good with optimizing this kind of loss function - logically you should obtain better generalization results:)", "631989": "Would you mind sharing what details you changed boosted your score from 0.4 to 0.9? I tried several networks and tricks, but the LB was still aboult 0.3~0.4, which confused me a lot.", "632083": "apotato, I've tried several tips written in other disucussion threads or kernels.\n\nBelow are tips that worked for my model.\n- train longer with 2 sites\n- average 2 site predictions\n- use plate leak\n- use Hungarian aloghrithm\n- use bigger images\n\nMore and more training seems to give better results.\nBut we don't have much time left...", "633093": "I will share details on training procedure after the competition ends. I hope you understand :) So far we are not using controls in any way", "633106": "narsil that's fair:)" }, "source": "meta" }