First of all, thanks to Kaggle and HPA team for this interesting competition! Even leak couldn’t spoil it!
\n\nOur solution is an ensemble of 14 models. Most of them were trained on 512x512 RGB with additional data.
\n\nMany thanks Jeremy Howard for great fast.ai framework! It’s nice how easy it is to patch it. Fast.ai has been changed 40 times since the competition began, so it’d have hurt if it was not open sourced.
\n\nOur models are: Se-ResNext-50 trained on 256x256, 512x512 and 768x768 sizes, InceptionV4, BN-Inception and Xception (all trained on 512x512).
\nWe didn’t have enough resources to train models on high resolution images. There were two ways to deal with it: resizing and crops, but training on crops is risky since some organelles occur only once per image (e.g. cytokinetic bridge), and it is difficult to select proper crops, so we didn’t do it.
Things that worked:\n1. Learning rate finder and cyclic learning rate (long cycles at the beginning, short cycles at the end).\n2. Differential learning rate with gradual reducing (as described here: https://blog.slavv.com/differential-learning-rates-59eff5209a4f ) helped to preserve weights from ImageNet.\n3. Focal loss with default gamma, LSEP loss ( https://arxiv.org/pdf/1704.03135.pdf ).\n4. Simple one-layer network head.\n5. Brightness augmentations, D4 and wrap transforms.\n6. Average of 32 TTAs (we used same augmentations as during training).
\n\nThings that didn’t work:\n1. Training on RGBY.\n2. Training with sample pairing ( https://arxiv.org/abs/1801.02929 ).\n3. Mixup, which probably didn’t work because green channel is too important ( https://arxiv.org/abs/1710.09412 ).\n4. Complex network head.\n5. Large architectures such as Nasnet or Senet-154 (they would have probably worked, if we had more GPUs).\n6. Training one-vs-all models and training on subsets of similar classes.\n7. Training classifier over bottleneck features of networks (we tried lots of approaches here, but unfortunately all of them proved to be worse than our models).\n8. Complex augmentations, such as green channel modifications discussed here: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/75768
\n\nExternal data was acquired from Human Protein Atlas site. After trying to find duplicates between train and externaldata we found out that labels wasn't match some times. First of all they shouldn't be taken from subcellular_location.tsv as these locations is raleted to gene/protein not to the sample. But parsing labels from xml files also wasn't quite correct. More correct labels are actually on web-site, so we took them. And it looks like they merged some rare labels that wasn't presented in our task into others, so we do the same.
\n\nOur resources: 4x1080ti. It seems that the full training cycle consumed approximately two weeks of computing time.
\n\nFindings:\n1. Yellow channel wasn’t very helpfulas many participants noticed on the forum.\n2. Labeling was probably quite noisy (we found several contradictions between HPAv18 data and data provided in this competition).\n3. Several classes are extremely similar visually. It is almost impossible to distinguish between lysosomes and endosomes, for example (no wonder: endosomes are literally reborn into lysosomes at some point of their lifecycle). So it is not surprising that models don’t perform well enough on these classes too.
\n\nOur validation is a kind of an Adversarial Validation. We arranged train by similarity to the test using an NN with simple architecture and got 8K samples as a holdout. We used this holdout to fit the thresholds for single models and check the scores. We tried our best to avoid duplicates between train and our holdout. All leaked images from HPA were added to the validation set too.
\n\nThe most challenging part of the competition is how to deal with “small” classes with only a few positive cases. We couldn’t handle it better than ensembling models with linear models and voting between those stacks.
\n\nOther classes were stacked with LightGBM model. It was stable enough on the validation set because we followed the “folds-inside-folds” scheme (similar to Strategy C from https://www.kaggle.com/general/18793 ). The gap between the local validation and leader board was stable with 0.01 precision. It wasn’t as accurate as we wanted it to be, but still it was ok to trust our validation.
\n\nP.S. Some stats:\nOur group chat contains 1325 screenshots, 321 files, 428 links and thousands of messages.
", "messages": [ { "id": 453895, "postDate": "2019-01-11T00:03:06.417Z", "content": "First of all, thanks to Kaggle and HPA team for this interesting competition! Even leak couldn’t spoil it!
\n\nOur solution is an ensemble of 14 models. Most of them were trained on 512x512 RGB with additional data.
\n\nMany thanks Jeremy Howard for great fast.ai framework! It’s nice how easy it is to patch it. Fast.ai has been changed 40 times since the competition began, so it’d have hurt if it was not open sourced.
\n\nOur models are: Se-ResNext-50 trained on 256x256, 512x512 and 768x768 sizes, InceptionV4, BN-Inception and Xception (all trained on 512x512).
\nWe didn’t have enough resources to train models on high resolution images. There were two ways to deal with it: resizing and crops, but training on crops is risky since some organelles occur only once per image (e.g. cytokinetic bridge), and it is difficult to select proper crops, so we didn’t do it.
Things that worked:\n1. Learning rate finder and cyclic learning rate (long cycles at the beginning, short cycles at the end).\n2. Differential learning rate with gradual reducing (as described here: https://blog.slavv.com/differential-learning-rates-59eff5209a4f ) helped to preserve weights from ImageNet.\n3. Focal loss with default gamma, LSEP loss ( https://arxiv.org/pdf/1704.03135.pdf ).\n4. Simple one-layer network head.\n5. Brightness augmentations, D4 and wrap transforms.\n6. Average of 32 TTAs (we used same augmentations as during training).
\n\nThings that didn’t work:\n1. Training on RGBY.\n2. Training with sample pairing ( https://arxiv.org/abs/1801.02929 ).\n3. Mixup, which probably didn’t work because green channel is too important ( https://arxiv.org/abs/1710.09412 ).\n4. Complex network head.\n5. Large architectures such as Nasnet or Senet-154 (they would have probably worked, if we had more GPUs).\n6. Training one-vs-all models and training on subsets of similar classes.\n7. Training classifier over bottleneck features of networks (we tried lots of approaches here, but unfortunately all of them proved to be worse than our models).\n8. Complex augmentations, such as green channel modifications discussed here: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/75768
\n\nExternal data was acquired from Human Protein Atlas site. After trying to find duplicates between train and externaldata we found out that labels wasn't match some times. First of all they shouldn't be taken from subcellular_location.tsv as these locations is raleted to gene/protein not to the sample. But parsing labels from xml files also wasn't quite correct. More correct labels are actually on web-site, so we took them. And it looks like they merged some rare labels that wasn't presented in our task into others, so we do the same.
\n\nOur resources: 4x1080ti. It seems that the full training cycle consumed approximately two weeks of computing time.
\n\nFindings:\n1. Yellow channel wasn’t very helpfulas many participants noticed on the forum.\n2. Labeling was probably quite noisy (we found several contradictions between HPAv18 data and data provided in this competition).\n3. Several classes are extremely similar visually. It is almost impossible to distinguish between lysosomes and endosomes, for example (no wonder: endosomes are literally reborn into lysosomes at some point of their lifecycle). So it is not surprising that models don’t perform well enough on these classes too.
\n\nOur validation is a kind of an Adversarial Validation. We arranged train by similarity to the test using an NN with simple architecture and got 8K samples as a holdout. We used this holdout to fit the thresholds for single models and check the scores. We tried our best to avoid duplicates between train and our holdout. All leaked images from HPA were added to the validation set too.
\n\nThe most challenging part of the competition is how to deal with “small” classes with only a few positive cases. We couldn’t handle it better than ensembling models with linear models and voting between those stacks.
\n\nOther classes were stacked with LightGBM model. It was stable enough on the validation set because we followed the “folds-inside-folds” scheme (similar to Strategy C from https://www.kaggle.com/general/18793 ). The gap between the local validation and leader board was stable with 0.01 precision. It wasn’t as accurate as we wanted it to be, but still it was ok to trust our validation.
\n\nP.S. Some stats:\nOur group chat contains 1325 screenshots, 321 files, 428 links and thousands of messages.
", "rawMarkdown": "First of all, thanks to Kaggle and HPA team for this interesting competition! Even leak couldn’t spoil it! \n\nOur solution is an ensemble of 14 models. Most of them were trained on 512x512 RGB with additional data.\n\nMany thanks Jeremy Howard for great fast.ai framework! It’s nice how easy it is to patch it. Fast.ai has been changed 40 times since the competition began, so it’d have hurt if it was not open sourced.\n\nOur models are: Se-ResNext-50 trained on 256x256, 512x512 and 768x768 sizes, InceptionV4, BN-Inception and Xception (all trained on 512x512). \nWe didn’t have enough resources to train models on high resolution images. There were two ways to deal with it: resizing and crops, but training on crops is risky since some organelles occur only once per image (e.g. cytokinetic bridge), and it is difficult to select proper crops, so we didn’t do it.\n\nThings that worked:\n1. Learning rate finder and cyclic learning rate (long cycles at the beginning, short cycles at the end).\n2. Differential learning rate with gradual reducing (as described here: https://blog.slavv.com/differential-learning-rates-59eff5209a4f ) helped to preserve weights from ImageNet.\n3. Focal loss with default gamma, LSEP loss ( https://arxiv.org/pdf/1704.03135.pdf ).\n4. Simple one-layer network head.\n5. Brightness augmentations, D4 and wrap transforms.\n6. Average of 32 TTAs (we used same augmentations as during training).\n\nThings that didn’t work:\n1. Training on RGBY.\n2. Training with sample pairing ( https://arxiv.org/abs/1801.02929 ).\n3. Mixup, which probably didn’t work because green channel is too important ( https://arxiv.org/abs/1710.09412 ).\n4. Complex network head.\n5. Large architectures such as Nasnet or Senet-154 (they would have probably worked, if we had more GPUs).\n6. Training one-vs-all models and training on subsets of similar classes.\n7. Training classifier over bottleneck features of networks (we tried lots of approaches here, but unfortunately all of them proved to be worse than our models).\n8. Complex augmentations, such as green channel modifications discussed here: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/75768\n\nExternal data was acquired from Human Protein Atlas site. After trying to find duplicates between train and externaldata we found out that labels wasn't match some times. First of all they shouldn't be taken from subcellular_location.tsv as these locations is raleted to gene/protein not to the sample. But parsing labels from xml files also wasn't quite correct. More correct labels are actually on web-site, so we took them. And it looks like they merged some rare labels that wasn't presented in our task into others, so we do the same.\n\nOur resources: 4x1080ti. It seems that the full training cycle consumed approximately two weeks of computing time.\n\nFindings:\n1. Yellow channel wasn’t very helpfulas many participants noticed on the forum.\n2. Labeling was probably quite noisy (we found several contradictions between HPAv18 data and data provided in this competition).\n3. Several classes are extremely similar visually. It is almost impossible to distinguish between lysosomes and endosomes, for example (no wonder: endosomes are literally reborn into lysosomes at some point of their lifecycle). So it is not surprising that models don’t perform well enough on these classes too.\n\nOur validation is a kind of an Adversarial Validation. We arranged train by similarity to the test using an NN with simple architecture and got 8K samples as a holdout. We used this holdout to fit the thresholds for single models and check the scores. We tried our best to avoid duplicates between train and our holdout. All leaked images from HPA were added to the validation set too.\n\nThe most challenging part of the competition is how to deal with “small” classes with only a few positive cases. We couldn’t handle it better than ensembling models with linear models and voting between those stacks. \n\nOther classes were stacked with LightGBM model. It was stable enough on the validation set because we followed the “folds-inside-folds” scheme (similar to Strategy C from https://www.kaggle.com/general/18793 ). The gap between the local validation and leader board was stable with 0.01 precision. It wasn’t as accurate as we wanted it to be, but still it was ok to trust our validation.\n\nP.S. Some stats:\nOur group chat contains 1325 screenshots, 321 files, 428 links and thousands of messages.\n", "votes": 97 }, { "id": 454175, "postDate": "2019-01-11T07:54:35.253Z", "content": "Congratulations and good job! We had a nice fight in the public leaderboard ;).
", "rawMarkdown": "Congratulations and good job! We had a nice fight in the public leaderboard ;).", "votes": 4 }, { "id": 455672, "postDate": "2019-01-14T10:49:52.483Z", "content": "\n\nfirst of all congratulations to you and your team. It was a though race in the end. How did you come up with the stats of the slack channel?
", "rawMarkdown": "@sergeifironov\n\nfirst of all congratulations to you and your team. It was a though race in the end. How did you come up with the stats of the slack channel?", "votes": 1, "replies": [ { "id": 455686, "postDate": "2019-01-14T11:10:42.223Z", "content": "Our team used Telegram. It has \"shared media\" feature which gives you all the numbers.
", "rawMarkdown": "Our team used Telegram. It has \"shared media\" feature which gives you all the numbers." } ] }, { "id": 455249, "postDate": "2019-01-13T12:00:56.437Z", "content": "Congratulations!!\nI enjoyed your write up.
\n\nI have tried the Cadene library with fastai, but could not make it work. It seems the order of the arguments of the cadene models are different than the fastai models and will not work with create_cnn of fastai v1. For example in the resnet50 in cadene is:\npretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
while in fastai is:\nmodels.resnet50(pretrained=False, **kwargs)
How did you make it work?
\n\nIf you have a simple example of a code that you can share for importing any cadene model into fastai v1 , that would be great!
\n\nI am getting such sort of errors (attached)..
\n\nAnd here is my sample code sent to Jeremy for help.
\n\nUpdate: I could make create_cnn work with custom model. Here is my code.
I just used Learner() instead of create_cnn and worked with Cadene.
", "rawMarkdown": "I just used Learner() instead of create_cnn and worked with Cadene." }, { "id": 455652, "postDate": "2019-01-14T09:54:55.647Z", "content": "I could make them both work (create_cnn and Learner). The difference is that Learner will not do transfer learning for you, and you should train the network from scratch. It will work, but needs more time/epochs.
", "rawMarkdown": "I could make them both work (create_cnn and Learner). The difference is that Learner will not do transfer learning for you, and you should train the network from scratch. It will work, but needs more time/epochs." }, { "id": 455688, "postDate": "2019-01-14T11:14:42.130Z", "content": "\ndef body(pretrained = True):\n return pretrainedmodels.dict[_model_name](num_classes=1000, pretrained='imagenet')
\nlearner = create_cnn(data, arch=body, cut=-2, \n custom_head = create_head(4096, len(data.classes), ps=0.5))
This code works for me.
", "rawMarkdown": "\ndef body(pretrained = True):\n return pretrainedmodels.__dict__[_model_name](num_classes=1000, pretrained='imagenet')\n\n\nlearner = create_cnn(data, arch=body, cut=-2, \n custom_head = create_head(4096, len(data.classes), ps=0.5))\n\nThis code works for me.",
"votes": 2
}
]
},
{
"id": 454144,
"postDate": "2019-01-11T07:07:46.850Z",
"content": "Congratulations and well done !!
", "rawMarkdown": "Congratulations and well done !!", "votes": 1 }, { "id": 453929, "postDate": "2019-01-11T00:47:33.037Z", "content": "Great work and thanks for sharing. Brightness augmentation really made a big difference.
", "rawMarkdown": "Great work and thanks for sharing. Brightness augmentation really made a big difference. ", "votes": 1, "replies": [ { "id": 453932, "postDate": "2019-01-11T00:50:43.023Z", "content": "Indeed, it affected training a lot. Explanation is quite obvious, actually: as I can see from leaked data, there are pairs of images which are almost identical except slight shifts and different brightness. So brightness augmentation perfectly simulates natural data variability.
", "rawMarkdown": "Indeed, it affected training a lot. Explanation is quite obvious, actually: as I can see from leaked data, there are pairs of images which are almost identical except slight shifts and different brightness. So brightness augmentation perfectly simulates natural data variability.", "votes": 1 }, { "id": 454080, "postDate": "2019-01-11T05:23:06.527Z", "content": "Hi, may I ask how much LB can improve after using Brightness augmentation?
", "rawMarkdown": "Hi, may I ask how much LB can improve after using Brightness augmentation?" }, { "id": 454115, "postDate": "2019-01-11T06:26:24.300Z", "content": "We didn't benchmark separate augmentations, but brightness meant a lot, I think.
", "rawMarkdown": "We didn't benchmark separate augmentations, but brightness meant a lot, I think." }, { "id": 458255, "postDate": "2019-01-19T07:39:32.613Z", "content": "hi dmytro\ncould you paste teh snipped for brightness aug here please.
", "rawMarkdown": "hi dmytro\ncould you paste teh snipped for brightness aug here please." } ] }, { "id": 454016, "postDate": "2019-01-11T03:01:51.483Z", "content": "Thanks for sharing your amazing work, however the your blog link (Differential learning rate with gradual reducing) appears as an error, it has an additional \")\" towards the end of the link. Can you please update !! \nhttps://blog.slavv.com/differential-learning-rates-59eff5209a4f
", "rawMarkdown": "Thanks for sharing your amazing work, however the your blog link (Differential learning rate with gradual reducing) appears as an error, it has an additional \")\" towards the end of the link. Can you please update !! \nhttps://blog.slavv.com/differential-learning-rates-59eff5209a4f", "votes": 2 }, { "id": 624840, "postDate": "2019-09-12T12:29:54.457Z", "content": "Heyo! First off, thanks for the write up and congrats on the win! I'm currently working on a problem that I want to try LSEP loss for but can't seem to figure it out. I was wondering if you had a reference or link for your implementation? It would be greatly appreciated -- thanks :)
", "rawMarkdown": "Heyo! First off, thanks for the write up and congrats on the win! I'm currently working on a problem that I want to try LSEP loss for but can't seem to figure it out. I was wondering if you had a reference or link for your implementation? It would be greatly appreciated -- thanks :)" }, { "id": 458197, "postDate": "2019-01-19T03:21:27.217Z", "content": "amazing cooperation.
", "rawMarkdown": "amazing cooperation." }, { "id": 457319, "postDate": "2019-01-17T08:16:40.090Z", "content": "hi,i too used fastai but couldnt manage a better score\ncould you please throw light on ensembling if you used .\nsay a prediction from model 1 is 0.99.....0.2 0.3 and model 2 is 0.8...0.4. 0.1 result of ensembled model will be ?
", "rawMarkdown": "hi,i too used fastai but couldnt manage a better score\ncould you please throw light on ensembling if you used .\nsay a prediction from model 1 is 0.99.....0.2 0.3 and model 2 is 0.8...0.4. 0.1 result of ensembled model will be ?\n\n" }, { "id": 456214, "postDate": "2019-01-15T11:16:54.057Z", "content": "kudos for the group chat stats.
", "rawMarkdown": "kudos for the group chat stats." }, { "id": 455194, "postDate": "2019-01-13T07:39:55.277Z", "content": "Will you share your code of lsep loss ? I tried but failed.
", "rawMarkdown": "Will you share your code of lsep loss ? I tried but failed.", "replies": [ { "id": 624838, "postDate": "2019-09-12T12:26:51.600Z", "content": "Did you ever figure this out? Also trying to integrate LSEP w/ Fast AI with no luck :(
", "rawMarkdown": "Did you ever figure this out? Also trying to integrate LSEP w/ Fast AI with no luck :(" } ] }, { "id": 454402, "postDate": "2019-01-11T15:31:29.963Z", "content": "Congratulations! What score were you getting with your individual models? How did you decide which ones to choose for ensembling?
", "rawMarkdown": "Congratulations! What score were you getting with your individual models? How did you decide which ones to choose for ensembling?", "replies": [ { "id": 454404, "postDate": "2019-01-11T15:37:31.853Z", "content": "What score were you getting with your individual models?\n\n
Our top model is ~top-20 on both public and private LB.
\n\nHow did you decide which ones to choose for ensembling?\n\n
Use them all and let stack decide ;)
", "rawMarkdown": "What score were you getting with your individual models?\nOur top model is ~top-20 on both public and private LB.\n\n
How did you decide which ones to choose for ensembling?\nUse them all and let stack decide ;)" } ] }, { "id": 454340, "postDate": "2019-01-11T13:23:23.230Z", "content": "
Congratulations to you and your team. Excellent write up. Are you guys planning to put the complete solution on GitHub? It would be a great learning resource for beginners like me
", "rawMarkdown": "Congratulations to you and your team. Excellent write up. Are you guys planning to put the complete solution on GitHub? It would be a great learning resource for beginners like me" }, { "id": 454303, "postDate": "2019-01-11T11:51:32.990Z", "content": "Congratulations ! are you going to share your code?
", "rawMarkdown": "Congratulations ! are you going to share your code?" }, { "id": 454232, "postDate": "2019-01-11T09:20:47.620Z", "content": "Nice sharing!!
", "rawMarkdown": "Nice sharing!!" }, { "id": 454023, "postDate": "2019-01-11T03:16:52.050Z", "content": "Thank you! Very enlightening. Would you mind giving some more details about how you trained se-resnext50? Did you use the titu github? I couldn't get it to train properly no matter what I did.
", "rawMarkdown": "Thank you! Very enlightening. Would you mind giving some more details about how you trained se-resnext50? Did you use the titu github? I couldn't get it to train properly no matter what I did. ", "replies": [ { "id": 454102, "postDate": "2019-01-11T06:12:29.943Z", "content": "Actually we used Fast.ai frameworks with many tricks from Jeremy's sleeves and pretrained model from amazing https://github.com/Cadene/pretrained-models.pytorch . Our pipeline was strictily inspired by first or third lection from the new version of fast.ai course.
", "rawMarkdown": "Actually we used Fast.ai frameworks with many tricks from Jeremy's sleeves and pretrained model from amazing https://github.com/Cadene/pretrained-models.pytorch . Our pipeline was strictily inspired by first or third lection from the new version of fast.ai course.", "votes": 1 }, { "id": 454194, "postDate": "2019-01-11T08:21:21.613Z", "content": "In addition to what Sergei said, I'd like to add that Se-ResNeXT-50 was the best-performing model in our experiments.
", "rawMarkdown": "In addition to what Sergei said, I'd like to add that Se-ResNeXT-50 was the best-performing model in our experiments." } ] }, { "id": 453993, "postDate": "2019-01-11T02:29:43.563Z", "content": "Thx for ur sharing! Great work!
", "rawMarkdown": "Thx for ur sharing! Great work!" }, { "id": 453990, "postDate": "2019-01-11T02:22:26.703Z", "content": "Beautiful write up! Congratulations!
", "rawMarkdown": "Beautiful write up! Congratulations!" }, { "id": 453961, "postDate": "2019-01-11T01:47:33.523Z", "content": "Thanks for the write up. Interesting to see what it takes to be in top 10. Congrats!
", "rawMarkdown": "Thanks for the write up. Interesting to see what it takes to be in top 10. Congrats!" }, { "id": 453933, "postDate": "2019-01-11T00:51:41.077Z", "content": "Thanks very much for the writeup. Is there a general reference for the method of stacking with LightGBM? I am not familiar with it.
", "rawMarkdown": "Thanks very much for the writeup. Is there a general reference for the method of stacking with LightGBM? I am not familiar with it.", "replies": [ { "id": 454096, "postDate": "2019-01-11T05:59:12.803Z", "content": "To be honest, I don't really think the general reference exists.
\n\nWe treated probabilities of each class from each models as a features with some meta information about an images (brightness, blurness, size and so on). In this point we got standard tabular problem.
\n\nStacking models with F1 or ROC-AUC is a bit tricky. The main idea of our solution to split holdout to folds then split training part of the fold to folds again and voting between models trained on inner folds and validate on the validation part of the outter fold.
", "rawMarkdown": "To be honest, I don't really think the general reference exists. \n\nWe treated probabilities of each class from each models as a features with some meta information about an images (brightness, blurness, size and so on). In this point we got standard tabular problem.\n\nStacking models with F1 or ROC-AUC is a bit tricky. The main idea of our solution to split holdout to folds then split training part of the fold to folds again and voting between models trained on inner folds and validate on the validation part of the outter fold. " } ] }, { "id": 453930, "postDate": "2019-01-11T00:48:53.460Z", "content": "Only thousands of messages? Those are rookie numbers :P
", "rawMarkdown": "Only thousands of messages? Those are rookie numbers :P", "replies": [ { "id": 453934, "postDate": "2019-01-11T00:52:24.973Z", "content": "Thousands, plural ;)\nMy estimate is a couple hundred thousand at least.
", "rawMarkdown": "Thousands, plural ;)\nMy estimate is a couple hundred thousand at least.", "votes": 1 } ] }, { "id": 454133, "postDate": "2019-01-11T06:59:06.857Z", "content": "Thank you very much and Congratulations!!
", "rawMarkdown": "Thank you very much and Congratulations!!", "isDeleted": true }, { "id": 454006, "postDate": "2019-01-11T02:47:07.440Z", "content": "Thank you for your wonderful sharing
", "rawMarkdown": "Thank you for your wonderful sharing" }, { "id": 453997, "postDate": "2019-01-11T02:35:58.923Z", "content": "Thanks for sharing your knowledge :)
", "rawMarkdown": "Thanks for sharing your knowledge :)" } ], "comments": [ { "id": 454175, "author_name": "FlYM", "author_url": "", "post_date": "2019-01-11T07:54:35.253000", "content": "Congratulations and good job! We had a nice fight in the public leaderboard ;).
", "votes": 4, "replies": [] }, { "id": 455672, "author_name": "Dieter", "author_url": "", "post_date": "2019-01-14T10:49:52.483000", "content": "\n\nfirst of all congratulations to you and your team. It was a though race in the end. How did you come up with the stats of the slack channel?
", "votes": 1, "replies": [ { "id": 455686, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-14T11:10:42.223000", "content": "Our team used Telegram. It has \"shared media\" feature which gives you all the numbers.
", "votes": 0, "replies": [] } ] }, { "id": 455249, "author_name": "Haider Alwasiti", "author_url": "", "post_date": "2019-01-13T12:00:56.437000", "content": "Congratulations!!\nI enjoyed your write up.
\n\nI have tried the Cadene library with fastai, but could not make it work. It seems the order of the arguments of the cadene models are different than the fastai models and will not work with create_cnn of fastai v1. For example in the resnet50 in cadene is:\npretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
while in fastai is:\nmodels.resnet50(pretrained=False, **kwargs)
How did you make it work?
\n\nIf you have a simple example of a code that you can share for importing any cadene model into fastai v1 , that would be great!
\n\nI am getting such sort of errors (attached)..
\n\nAnd here is my sample code sent to Jeremy for help.
\n\nUpdate: I could make create_cnn work with custom model. Here is my code.
I just used Learner() instead of create_cnn and worked with Cadene.
", "votes": 0, "replies": [] }, { "id": 455652, "author_name": "Haider Alwasiti", "author_url": "", "post_date": "2019-01-14T09:54:55.647000", "content": "I could make them both work (create_cnn and Learner). The difference is that Learner will not do transfer learning for you, and you should train the network from scratch. It will work, but needs more time/epochs.
", "votes": 0, "replies": [] }, { "id": 455688, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-14T11:14:42.130000", "content": "\ndef body(pretrained = True):\n return pretrainedmodels.dict[_model_name](num_classes=1000, pretrained='imagenet')
\nlearner = create_cnn(data, arch=body, cut=-2, \n custom_head = create_head(4096, len(data.classes), ps=0.5))
This code works for me.
", "votes": 2, "replies": [] } ] }, { "id": 454144, "author_name": "Shai", "author_url": "", "post_date": "2019-01-11T07:07:46.850000", "content": "Congratulations and well done !!
", "votes": 1, "replies": [] }, { "id": 453929, "author_name": "Ben Lai", "author_url": "", "post_date": "2019-01-11T00:47:33.037000", "content": "Great work and thanks for sharing. Brightness augmentation really made a big difference.
", "votes": 1, "replies": [ { "id": 453932, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-11T00:50:43.023000", "content": "Indeed, it affected training a lot. Explanation is quite obvious, actually: as I can see from leaked data, there are pairs of images which are almost identical except slight shifts and different brightness. So brightness augmentation perfectly simulates natural data variability.
", "votes": 1, "replies": [] }, { "id": 454080, "author_name": "good good study", "author_url": "", "post_date": "2019-01-11T05:23:06.527000", "content": "Hi, may I ask how much LB can improve after using Brightness augmentation?
", "votes": 0, "replies": [] }, { "id": 454115, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-11T06:26:24.300000", "content": "We didn't benchmark separate augmentations, but brightness meant a lot, I think.
", "votes": 0, "replies": [] }, { "id": 458255, "author_name": "Jaideep", "author_url": "", "post_date": "2019-01-19T07:39:32.613000", "content": "hi dmytro\ncould you paste teh snipped for brightness aug here please.
", "votes": 0, "replies": [] } ] }, { "id": 454016, "author_name": "Vishy", "author_url": "", "post_date": "2019-01-11T03:01:51.483000", "content": "Thanks for sharing your amazing work, however the your blog link (Differential learning rate with gradual reducing) appears as an error, it has an additional \")\" towards the end of the link. Can you please update !! \nhttps://blog.slavv.com/differential-learning-rates-59eff5209a4f
", "votes": 2, "replies": [] }, { "id": 624840, "author_name": "Zach Eberhart", "author_url": "", "post_date": "2019-09-12T12:29:54.457000", "content": "Heyo! First off, thanks for the write up and congrats on the win! I'm currently working on a problem that I want to try LSEP loss for but can't seem to figure it out. I was wondering if you had a reference or link for your implementation? It would be greatly appreciated -- thanks :)
", "votes": 0, "replies": [] }, { "id": 458197, "author_name": "BenChur", "author_url": "", "post_date": "2019-01-19T03:21:27.217000", "content": "amazing cooperation.
", "votes": 0, "replies": [] }, { "id": 457319, "author_name": "Jaideep", "author_url": "", "post_date": "2019-01-17T08:16:40.090000", "content": "hi,i too used fastai but couldnt manage a better score\ncould you please throw light on ensembling if you used .\nsay a prediction from model 1 is 0.99.....0.2 0.3 and model 2 is 0.8...0.4. 0.1 result of ensembled model will be ?
", "votes": 0, "replies": [] }, { "id": 456214, "author_name": "RanaTallal", "author_url": "", "post_date": "2019-01-15T11:16:54.057000", "content": "kudos for the group chat stats.
", "votes": 0, "replies": [] }, { "id": 455194, "author_name": "zjuyang", "author_url": "", "post_date": "2019-01-13T07:39:55.277000", "content": "Will you share your code of lsep loss ? I tried but failed.
", "votes": 0, "replies": [ { "id": 624838, "author_name": "Zach Eberhart", "author_url": "", "post_date": "2019-09-12T12:26:51.600000", "content": "Did you ever figure this out? Also trying to integrate LSEP w/ Fast AI with no luck :(
", "votes": 0, "replies": [] } ] }, { "id": 454402, "author_name": "Gaurav", "author_url": "", "post_date": "2019-01-11T15:31:29.963000", "content": "Congratulations! What score were you getting with your individual models? How did you decide which ones to choose for ensembling?
", "votes": 0, "replies": [ { "id": 454404, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-11T15:37:31.853000", "content": "What score were you getting with your individual models?\n\n
Our top model is ~top-20 on both public and private LB.
\n\nHow did you decide which ones to choose for ensembling?\n\n
Use them all and let stack decide ;)
", "votes": 0, "replies": [] } ] }, { "id": 454340, "author_name": "sandy1112", "author_url": "", "post_date": "2019-01-11T13:23:23.230000", "content": "Congratulations to you and your team. Excellent write up. Are you guys planning to put the complete solution on GitHub? It would be a great learning resource for beginners like me
", "votes": 0, "replies": [] }, { "id": 454303, "author_name": "soywu", "author_url": "", "post_date": "2019-01-11T11:51:32.990000", "content": "Congratulations ! are you going to share your code?
", "votes": 0, "replies": [] }, { "id": 454232, "author_name": "nekoder", "author_url": "", "post_date": "2019-01-11T09:20:47.620000", "content": "Nice sharing!!
", "votes": 0, "replies": [] }, { "id": 454023, "author_name": "Moshel", "author_url": "", "post_date": "2019-01-11T03:16:52.050000", "content": "Thank you! Very enlightening. Would you mind giving some more details about how you trained se-resnext50? Did you use the titu github? I couldn't get it to train properly no matter what I did.
", "votes": 0, "replies": [ { "id": 454102, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2019-01-11T06:12:29.943000", "content": "Actually we used Fast.ai frameworks with many tricks from Jeremy's sleeves and pretrained model from amazing https://github.com/Cadene/pretrained-models.pytorch . Our pipeline was strictily inspired by first or third lection from the new version of fast.ai course.
", "votes": 1, "replies": [] }, { "id": 454194, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-11T08:21:21.613000", "content": "In addition to what Sergei said, I'd like to add that Se-ResNeXT-50 was the best-performing model in our experiments.
", "votes": 0, "replies": [] } ] }, { "id": 453993, "author_name": "Jun Liu", "author_url": "", "post_date": "2019-01-11T02:29:43.563000", "content": "Thx for ur sharing! Great work!
", "votes": 0, "replies": [] }, { "id": 453990, "author_name": "Hilal Shaath", "author_url": "", "post_date": "2019-01-11T02:22:26.703000", "content": "Beautiful write up! Congratulations!
", "votes": 0, "replies": [] }, { "id": 453961, "author_name": "David Wagner", "author_url": "", "post_date": "2019-01-11T01:47:33.523000", "content": "Thanks for the write up. Interesting to see what it takes to be in top 10. Congrats!
", "votes": 0, "replies": [] }, { "id": 453933, "author_name": "pete", "author_url": "", "post_date": "2019-01-11T00:51:41.077000", "content": "Thanks very much for the writeup. Is there a general reference for the method of stacking with LightGBM? I am not familiar with it.
", "votes": 0, "replies": [ { "id": 454096, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2019-01-11T05:59:12.803000", "content": "To be honest, I don't really think the general reference exists.
\n\nWe treated probabilities of each class from each models as a features with some meta information about an images (brightness, blurness, size and so on). In this point we got standard tabular problem.
\n\nStacking models with F1 or ROC-AUC is a bit tricky. The main idea of our solution to split holdout to folds then split training part of the fold to folds again and voting between models trained on inner folds and validate on the validation part of the outter fold.
", "votes": 0, "replies": [] } ] }, { "id": 453930, "author_name": "ryches", "author_url": "", "post_date": "2019-01-11T00:48:53.460000", "content": "Only thousands of messages? Those are rookie numbers :P
", "votes": 0, "replies": [ { "id": 453934, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-01-11T00:52:24.973000", "content": "Thousands, plural ;)\nMy estimate is a couple hundred thousand at least.
", "votes": 1, "replies": [] } ] }, { "id": 454133, "author_name": "", "author_url": "", "post_date": "2019-01-11T06:59:06.857000", "content": "Thank you very much and Congratulations!!
", "votes": 0, "replies": [] }, { "id": 454006, "author_name": "MountainClimber", "author_url": "", "post_date": "2019-01-11T02:47:07.440000", "content": "Thank you for your wonderful sharing
", "votes": 0, "replies": [] }, { "id": 453997, "author_name": "Minh Duy", "author_url": "", "post_date": "2019-01-11T02:35:58.923000", "content": "Thanks for sharing your knowledge :)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "453895": "First of all, thanks to Kaggle and HPA team for this interesting competition! Even leak couldn’t spoil it! \n\nOur solution is an ensemble of 14 models. Most of them were trained on 512x512 RGB with additional data.\n\nMany thanks Jeremy Howard for great fast.ai framework! It’s nice how easy it is to patch it. Fast.ai has been changed 40 times since the competition began, so it’d have hurt if it was not open sourced.\n\nOur models are: Se-ResNext-50 trained on 256x256, 512x512 and 768x768 sizes, InceptionV4, BN-Inception and Xception (all trained on 512x512). \nWe didn’t have enough resources to train models on high resolution images. There were two ways to deal with it: resizing and crops, but training on crops is risky since some organelles occur only once per image (e.g. cytokinetic bridge), and it is difficult to select proper crops, so we didn’t do it.\n\nThings that worked:\n1. Learning rate finder and cyclic learning rate (long cycles at the beginning, short cycles at the end).\n2. Differential learning rate with gradual reducing (as described here: https://blog.slavv.com/differential-learning-rates-59eff5209a4f ) helped to preserve weights from ImageNet.\n3. Focal loss with default gamma, LSEP loss ( https://arxiv.org/pdf/1704.03135.pdf ).\n4. Simple one-layer network head.\n5. Brightness augmentations, D4 and wrap transforms.\n6. Average of 32 TTAs (we used same augmentations as during training).\n\nThings that didn’t work:\n1. Training on RGBY.\n2. Training with sample pairing ( https://arxiv.org/abs/1801.02929 ).\n3. Mixup, which probably didn’t work because green channel is too important ( https://arxiv.org/abs/1710.09412 ).\n4. Complex network head.\n5. Large architectures such as Nasnet or Senet-154 (they would have probably worked, if we had more GPUs).\n6. Training one-vs-all models and training on subsets of similar classes.\n7. Training classifier over bottleneck features of networks (we tried lots of approaches here, but unfortunately all of them proved to be worse than our models).\n8. Complex augmentations, such as green channel modifications discussed here: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/75768\n\nExternal data was acquired from Human Protein Atlas site. After trying to find duplicates between train and externaldata we found out that labels wasn't match some times. First of all they shouldn't be taken from subcellular_location.tsv as these locations is raleted to gene/protein not to the sample. But parsing labels from xml files also wasn't quite correct. More correct labels are actually on web-site, so we took them. And it looks like they merged some rare labels that wasn't presented in our task into others, so we do the same.\n\nOur resources: 4x1080ti. It seems that the full training cycle consumed approximately two weeks of computing time.\n\nFindings:\n1. Yellow channel wasn’t very helpfulas many participants noticed on the forum.\n2. Labeling was probably quite noisy (we found several contradictions between HPAv18 data and data provided in this competition).\n3. Several classes are extremely similar visually. It is almost impossible to distinguish between lysosomes and endosomes, for example (no wonder: endosomes are literally reborn into lysosomes at some point of their lifecycle). So it is not surprising that models don’t perform well enough on these classes too.\n\nOur validation is a kind of an Adversarial Validation. We arranged train by similarity to the test using an NN with simple architecture and got 8K samples as a holdout. We used this holdout to fit the thresholds for single models and check the scores. We tried our best to avoid duplicates between train and our holdout. All leaked images from HPA were added to the validation set too.\n\nThe most challenging part of the competition is how to deal with “small” classes with only a few positive cases. We couldn’t handle it better than ensembling models with linear models and voting between those stacks. \n\nOther classes were stacked with LightGBM model. It was stable enough on the validation set because we followed the “folds-inside-folds” scheme (similar to Strategy C from https://www.kaggle.com/general/18793 ). The gap between the local validation and leader board was stable with 0.01 precision. It wasn’t as accurate as we wanted it to be, but still it was ok to trust our validation.\n\nP.S. Some stats:\nOur group chat contains 1325 screenshots, 321 files, 428 links and thousands of messages.\n", "454175": "Congratulations and good job! We had a nice fight in the public leaderboard ;).", "455672": "@sergeifironov\n\nfirst of all congratulations to you and your team. It was a though race in the end. How did you come up with the stats of the slack channel?", "455249": "Congratulations!!\nI enjoyed your write up.\n\nI have tried the Cadene library with fastai, but could not make it work. It seems the order of the arguments of the cadene models are different than the fastai models and will not work with `create_cnn` of fastai v1. For example in the resnet50 in cadene is:\n`pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')`\n\nwhile in fastai is:\n`models.resnet50(pretrained=False, **kwargs)`\n\nHow did you make it work?\n\nIf you have a simple example of a code that you can share for importing any cadene model into fastai v1 , that would be great!\n\nI am getting such sort of errors (attached)..\n\nAnd here is my [sample code][1] sent to Jeremy for help.\n\nUpdate: I could make `create_cnn` work with custom model. Here is my [code][2].\n\n\n [1]: https://forums.fast.ai/t/lesson-5-advanced-discussion/30865/39?u=hwasiti\n [2]: https://forums.fast.ai/t/lesson-5-advanced-discussion/30865/40?u=hwasiti", "454144": "Congratulations and well done !!", "453929": "Great work and thanks for sharing. Brightness augmentation really made a big difference. ", "454016": "Thanks for sharing your amazing work, however the your blog link (Differential learning rate with gradual reducing) appears as an error, it has an additional \")\" towards the end of the link. Can you please update !! \nhttps://blog.slavv.com/differential-learning-rates-59eff5209a4f", "624840": "Heyo! First off, thanks for the write up and congrats on the win! I'm currently working on a problem that I want to try LSEP loss for but can't seem to figure it out. I was wondering if you had a reference or link for your implementation? It would be greatly appreciated -- thanks :)", "458197": "amazing cooperation.", "457319": "hi,i too used fastai but couldnt manage a better score\ncould you please throw light on ensembling if you used .\nsay a prediction from model 1 is 0.99.....0.2 0.3 and model 2 is 0.8...0.4. 0.1 result of ensembled model will be ?\n\n", "456214": "kudos for the group chat stats.", "455194": "Will you share your code of lsep loss ? I tried but failed.", "454402": "Congratulations! What score were you getting with your individual models? How did you decide which ones to choose for ensembling?", "454340": "Congratulations to you and your team. Excellent write up. Are you guys planning to put the complete solution on GitHub? It would be a great learning resource for beginners like me", "454303": "Congratulations ! are you going to share your code?", "454232": "Nice sharing!!", "454023": "Thank you! Very enlightening. Would you mind giving some more details about how you trained se-resnext50? Did you use the titu github? I couldn't get it to train properly no matter what I did. ", "453993": "Thx for ur sharing! Great work!", "453990": "Beautiful write up! Congratulations!", "453961": "Thanks for the write up. Interesting to see what it takes to be in top 10. Congrats!", "453933": "Thanks very much for the writeup. Is there a general reference for the method of stacking with LightGBM? I am not familiar with it.", "453930": "Only thousands of messages? Those are rookie numbers :P", "454133": "Thank you very much and Congratulations!!", "454006": "Thank you for your wonderful sharing", "453997": "Thanks for sharing your knowledge :)" } }