First of all, thanks to Kaggle and HPA team for this meaningful competition! Thanks @Spytensor for sharing his pytorch baseline code. I started the competition by his code. Thanks @Allunia, @Iafoss, @jschnab, @kwentar for sharing their nice exploration and visualization kernels. Thanks @zhangmiao, @interneuron, @pascal1129, @maw501for, ... answered my question.
\n\nI am a new person in the field of data science and this is my first time to participate in competitions. I am very glad that I have chosen kaggle to enter the world of cv. Because there are too many people sharing their ideas here. I have learned too much in the past month. Because of the lack of knowledge, I don't have much trick to strengthen my model. You will find my model is very simple, maybe I got this place by luck haha.
\n\nTTA and oversampling is really helpful, and most importantly 5-fold cv improve score by 0.02. \n
Threshold is crucial, different threshold have a great influence on the LB score. The threshold I searched by valid set lowered my score at first. So I use the constant threshold (0.15) for a long time. The searched threshold for each class varies from 0.1 to 0.9 and different, and I cannot find any relationship between rare class and its threshold. I just want to give up to use searched threshold until I found that using smaller threshold would increase the score (not always). So I just multiply each searched threshold by a factor(~0.5). That mean I want predicted more target each class and get high recall. Although doing so may lower my f1 score, it does improve my public/private LB score. I think the reason may be that the wrong classification can be eliminated by tta and ensemble, and finally get more #TP and high score. My final submission ensemble 5-fold x 4-tta x 3-factor(0.3, 0.4, 0.5).\nnote: there are 5 version of threshold corresponding 5-fold cv. When using a model to predict, I choose the corresponding threshold. You can find them in my github repository data.py file(fold 12-16)\n
I think it's important to handle external data correctly. For me, the red, green and blue images are directly extracted from the corresponding channels of original jpg images ,the yellow image is combined with R and Y channel in original image. Finally save all of them into 512x512 gray png images separately. Because kaggle and external data have different distributions, so I normalized them with two different mean and std.\n
Put all images into SSD faster than HDD in training. and I want to know is there any other way to accelerate training? Because I train 1 fold for 30 hours (GTX 1070, 24 epochs)
Here is the github repository and I hope this is helpful for you.\nhttps://github.com/feifei9099/kaggle_human_protein\n
update: I am sorry that my grades have been cancelled. Because this is my first time to participate in the competition, there are some rules that I am not very clear. I promise to follow the rules of kaggle next time.
", "messages": [ { "id": 454393, "postDate": "2019-01-11T15:15:20.067Z", "content": "First of all, thanks to Kaggle and HPA team for this meaningful competition! Thanks @Spytensor for sharing his pytorch baseline code. I started the competition by his code. Thanks @Allunia, @Iafoss, @jschnab, @kwentar for sharing their nice exploration and visualization kernels. Thanks @zhangmiao, @interneuron, @pascal1129, @maw501for, ... answered my question.
\n\nI am a new person in the field of data science and this is my first time to participate in competitions. I am very glad that I have chosen kaggle to enter the world of cv. Because there are too many people sharing their ideas here. I have learned too much in the past month. Because of the lack of knowledge, I don't have much trick to strengthen my model. You will find my model is very simple, maybe I got this place by luck haha.
\n\nTTA and oversampling is really helpful, and most importantly 5-fold cv improve score by 0.02. \n
Threshold is crucial, different threshold have a great influence on the LB score. The threshold I searched by valid set lowered my score at first. So I use the constant threshold (0.15) for a long time. The searched threshold for each class varies from 0.1 to 0.9 and different, and I cannot find any relationship between rare class and its threshold. I just want to give up to use searched threshold until I found that using smaller threshold would increase the score (not always). So I just multiply each searched threshold by a factor(~0.5). That mean I want predicted more target each class and get high recall. Although doing so may lower my f1 score, it does improve my public/private LB score. I think the reason may be that the wrong classification can be eliminated by tta and ensemble, and finally get more #TP and high score. My final submission ensemble 5-fold x 4-tta x 3-factor(0.3, 0.4, 0.5).\nnote: there are 5 version of threshold corresponding 5-fold cv. When using a model to predict, I choose the corresponding threshold. You can find them in my github repository data.py file(fold 12-16)\n
I think it's important to handle external data correctly. For me, the red, green and blue images are directly extracted from the corresponding channels of original jpg images ,the yellow image is combined with R and Y channel in original image. Finally save all of them into 512x512 gray png images separately. Because kaggle and external data have different distributions, so I normalized them with two different mean and std.\n
Put all images into SSD faster than HDD in training. and I want to know is there any other way to accelerate training? Because I train 1 fold for 30 hours (GTX 1070, 24 epochs)
Here is the github repository and I hope this is helpful for you.\nhttps://github.com/feifei9099/kaggle_human_protein\n
update: I am sorry that my grades have been cancelled. Because this is my first time to participate in the competition, there are some rules that I am not very clear. I promise to follow the rules of kaggle next time.
", "rawMarkdown": "First of all, thanks to Kaggle and HPA team for this meaningful competition! Thanks @Spytensor for sharing his pytorch baseline code. I started the competition by his code. Thanks @Allunia, @Iafoss, @jschnab, @kwentar for sharing their nice exploration and visualization kernels. Thanks @zhangmiao, @interneuron, @pascal1129, @maw501for, ... answered my question.\n\nI am a new person in the field of data science and this is my first time to participate in competitions. I am very glad that I have chosen kaggle to enter the world of cv. Because there are too many people sharing their ideas here. I have learned too much in the past month. Because of the lack of knowledge, I don't have much trick to strengthen my model. You will find my model is very simple, maybe I got this place by luck haha.\n\n## Here is my solution:\n1. **data**: kaggle + external data 512x512 RGBY\n2. **model**: Densenet121 (conv0 3->4 channels, linear layers with batch normalization and dropout)\n3. **augmentation**: horizontal flip, vertical flip, rotate, shear, lighter/darker\n4. **normalize**: kaggle data and external data with different mean and std\n5. **optimizer**: SGD\n6. **loss**: BCE (no weight)\n7. **oversampling**: each image sampled by 'WeightedRandomSampler' function in pytorch. their weight is calculated by math.log(\\#all_target / \\#class-i).\n8. **cv**: 5 folds\n9. **threshold search**: I use the code supplied by Brian\n10. **TTA**: 2-horizontal flip, 2-vertical flip\n\n## I want to share my understanding to strategies:\n1. TTA and oversampling is really helpful, and most importantly 5-fold cv improve score by 0.02. \nThanks for sharing your solution. It is astonishing that you could achieve that high rank with such a a low number of ensembles and without to much tricks and tweaks. Sorry for what happened to your grades, but why? I am too knew in kaggle, and don't know the rules like you
", "rawMarkdown": "Thanks for sharing your solution. It is astonishing that you could achieve that high rank with such a a low number of ensembles and without to much tricks and tweaks. Sorry for what happened to your grades, but why? I am too knew in kaggle, and don't know the rules like you" }, { "id": 455111, "postDate": "2019-01-13T02:10:52.397Z", "content": "Thanks for sharing! Great work and such a nice and clear github README.\nSorry for what happened to your medal. Wish you good luck!
", "rawMarkdown": "Thanks for sharing! Great work and such a nice and clear github README.\nSorry for what happened to your medal. Wish you good luck!", "replies": [ { "id": 455138, "postDate": "2019-01-13T03:57:27.497Z", "content": "Thanks :-)
", "rawMarkdown": "Thanks :-)" } ] }, { "id": 454719, "postDate": "2019-01-12T05:00:02.177Z", "content": "Congrats!And if I can use your collections for top solutions in this competition to add it in my blog?
", "rawMarkdown": "Congrats!And if I can use your collections for top solutions in this competition to add it in my blog?", "replies": [ { "id": 454922, "postDate": "2019-01-12T14:02:38.140Z", "content": "Of course, this is my pleasure. I have put top solutions at the forefront of my README.md. And I will keep to update it
", "rawMarkdown": "Of course, this is my pleasure. I have put top solutions at the forefront of my README.md. And I will keep to update it" } ] }, { "id": 454687, "postDate": "2019-01-12T02:13:30.190Z", "content": "thanks for sharing. so you are doing voting ensemble on different folds? Have you tried averaging probability of all folds?\nI was always doing probability averaging during contest, this kind of ensemble can never boost me 0.02 on LB.
", "rawMarkdown": "thanks for sharing. so you are doing voting ensemble on different folds? Have you tried averaging probability of all folds?\nI was always doing probability averaging during contest, this kind of ensemble can never boost me 0.02 on LB.", "replies": [ { "id": 454694, "postDate": "2019-01-12T03:11:26.757Z", "content": "I use average mean ensemble of final target. I have never tried averaging probability.
", "rawMarkdown": "I use average mean ensemble of final target. I have never tried averaging probability." } ] }, { "id": 454496, "postDate": "2019-01-11T18:39:49.023Z", "content": "Great job, Femi! \nDense121 was one of my better performers, it worked strangely well. ibnDense121 was even better but densenets became incredibly slow for me as the image size increased.
\n\nI am also very new and this was my first competition I really worked on so I have much to learn. Thanks for sharing your solution, extremely informative!
", "rawMarkdown": "Great job, Femi! \nDense121 was one of my better performers, it worked strangely well. ibnDense121 was even better but densenets became incredibly slow for me as the image size increased. \n\nI am also very new and this was my first competition I really worked on so I have much to learn. Thanks for sharing your solution, extremely informative!", "replies": [ { "id": 454678, "postDate": "2019-01-12T01:17:51.933Z", "content": "Thanks, slow training has also plagued me. 256x256 took me 7 hours, 512x512 took me 30 hours. It seems that the time spent exponentially increases with the size of the image. I want to know if there is any way to speed up the training.\nI also have a lot of things to learn and work hard together :-)
", "rawMarkdown": "Thanks, slow training has also plagued me. 256x256 took me 7 hours, 512x512 took me 30 hours. It seems that the time spent exponentially increases with the size of the image. I want to know if there is any way to speed up the training.\nI also have a lot of things to learn and work hard together :-)" }, { "id": 454679, "postDate": "2019-01-12T01:25:57.803Z", "content": "512x512 has 4 times as much data as 256x256 so 28h seems reasonable.\nI also wonder how some of the top winners managed to train 14 models or so. You can't make any mistakes...
", "rawMarkdown": "512x512 has 4 times as much data as 256x256 so 28h seems reasonable.\nI also wonder how some of the top winners managed to train 14 models or so. You can't make any mistakes... " }, { "id": 454698, "postDate": "2019-01-12T03:15:34.013Z", "content": "thanks!
", "rawMarkdown": "thanks!" } ] }, { "id": 454464, "postDate": "2019-01-11T17:08:30.637Z", "content": "@femichen Thanks for sharing and great job being 28th with a single 5 fold model! Most of the write ups so far have many more models.
", "rawMarkdown": "@femichen Thanks for sharing and great job being 28th with a single 5 fold model! Most of the write ups so far have many more models.", "replies": [ { "id": 454675, "postDate": "2019-01-12T01:01:31.013Z", "content": "thanks, I think the key to my success is finding the right threshold.I will try more models in the next competition.
", "rawMarkdown": "thanks, I think the key to my success is finding the right threshold.I will try more models in the next competition." } ] }, { "id": 454411, "postDate": "2019-01-11T15:46:57.220Z", "content": "Congrats and thanks for your solution! It seems from your and other top-rated solutions , that preprocessing external data was extremely important.
", "rawMarkdown": "Congrats and thanks for your solution! It seems from your and other top-rated solutions , that preprocessing external data was extremely important. ", "replies": [ { "id": 454418, "postDate": "2019-01-11T16:04:09.567Z", "content": "Thanks, you are right. And I am curious how the gold medal winner uses external data. :-)
", "rawMarkdown": "Thanks, you are right. And I am curious how the gold medal winner uses external data. :-)" } ] }, { "id": 454989, "postDate": "2019-01-12T17:34:58.943Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 455116, "postDate": "2019-01-13T02:40:21.090Z", "content": "hah~, thanks! I am sorry that I have not tried stand-alone oversampling. My understanding of stand-alone oversampling is duplication of low frequency classes like code here. Am I right?\n\ntrain_df_orig=train_df.copy() \nSuppose my understanding is correct. I think such oversampling needs to decide which rare classes need to be oversampling and how many times they are duplicated. In other words, there are too many hyperparameters to be adjusted. You need to find the right parameters to get him to work. However, log weighted helped us decide all of this.\nI don't think log weighted is the most correct way, but I think it is the reasonable way. Hope to get your reply.
\nlows = [15,15,15,8,9,10,8,9,10,8,9,10,17,20,24,26,15,27,15,20,24,17,8,15,27,27,27]\nfor i in lows:\n target = str(i)\n indicies = train_df_orig.loc[train_df_orig['Target'] == target].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n indicies = train_df_orig.loc[train_df_orig['Target'].str.startswith(target+\" \")].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n indicies = train_df_orig.loc[train_df_orig['Target'].str.endswith(\" \"+target)].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n indicies = train_df_orig.loc[train_df_orig['Target'].str.contains(\" \"+target+\" \")].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n
Thanks for sharing your solution. It is astonishing that you could achieve that high rank with such a a low number of ensembles and without to much tricks and tweaks. Sorry for what happened to your grades, but why? I am too knew in kaggle, and don't know the rules like you
", "votes": 0, "replies": [] }, { "id": 455111, "author_name": "zhangboshen", "author_url": "", "post_date": "2019-01-13T02:10:52.397000", "content": "Thanks for sharing! Great work and such a nice and clear github README.\nSorry for what happened to your medal. Wish you good luck!
", "votes": 0, "replies": [ { "id": 455138, "author_name": "Femi", "author_url": "", "post_date": "2019-01-13T03:57:27.497000", "content": "Thanks :-)
", "votes": 0, "replies": [] } ] }, { "id": 454719, "author_name": "Spytensor", "author_url": "", "post_date": "2019-01-12T05:00:02.177000", "content": "Congrats!And if I can use your collections for top solutions in this competition to add it in my blog?
", "votes": 0, "replies": [ { "id": 454922, "author_name": "Femi", "author_url": "", "post_date": "2019-01-12T14:02:38.140000", "content": "Of course, this is my pleasure. I have put top solutions at the forefront of my README.md. And I will keep to update it
", "votes": 0, "replies": [] } ] }, { "id": 454687, "author_name": "good good study", "author_url": "", "post_date": "2019-01-12T02:13:30.190000", "content": "thanks for sharing. so you are doing voting ensemble on different folds? Have you tried averaging probability of all folds?\nI was always doing probability averaging during contest, this kind of ensemble can never boost me 0.02 on LB.
", "votes": 0, "replies": [ { "id": 454694, "author_name": "Femi", "author_url": "", "post_date": "2019-01-12T03:11:26.757000", "content": "I use average mean ensemble of final target. I have never tried averaging probability.
", "votes": 0, "replies": [] } ] }, { "id": 454496, "author_name": "interneuron", "author_url": "", "post_date": "2019-01-11T18:39:49.023000", "content": "Great job, Femi! \nDense121 was one of my better performers, it worked strangely well. ibnDense121 was even better but densenets became incredibly slow for me as the image size increased.
\n\nI am also very new and this was my first competition I really worked on so I have much to learn. Thanks for sharing your solution, extremely informative!
", "votes": 0, "replies": [ { "id": 454678, "author_name": "Femi", "author_url": "", "post_date": "2019-01-12T01:17:51.933000", "content": "Thanks, slow training has also plagued me. 256x256 took me 7 hours, 512x512 took me 30 hours. It seems that the time spent exponentially increases with the size of the image. I want to know if there is any way to speed up the training.\nI also have a lot of things to learn and work hard together :-)
", "votes": 0, "replies": [] }, { "id": 454679, "author_name": "Moshel", "author_url": "", "post_date": "2019-01-12T01:25:57.803000", "content": "512x512 has 4 times as much data as 256x256 so 28h seems reasonable.\nI also wonder how some of the top winners managed to train 14 models or so. You can't make any mistakes...
", "votes": 0, "replies": [] }, { "id": 454698, "author_name": "Femi", "author_url": "", "post_date": "2019-01-12T03:15:34.013000", "content": "thanks!
", "votes": 0, "replies": [] } ] }, { "id": 454464, "author_name": "David Wagner", "author_url": "", "post_date": "2019-01-11T17:08:30.637000", "content": "@femichen Thanks for sharing and great job being 28th with a single 5 fold model! Most of the write ups so far have many more models.
", "votes": 0, "replies": [ { "id": 454675, "author_name": "Femi", "author_url": "", "post_date": "2019-01-12T01:01:31.013000", "content": "thanks, I think the key to my success is finding the right threshold.I will try more models in the next competition.
", "votes": 0, "replies": [] } ] }, { "id": 454411, "author_name": "Vaagn Minasian", "author_url": "", "post_date": "2019-01-11T15:46:57.220000", "content": "Congrats and thanks for your solution! It seems from your and other top-rated solutions , that preprocessing external data was extremely important.
", "votes": 0, "replies": [ { "id": 454418, "author_name": "Femi", "author_url": "", "post_date": "2019-01-11T16:04:09.567000", "content": "Thanks, you are right. And I am curious how the gold medal winner uses external data. :-)
", "votes": 0, "replies": [] } ] }, { "id": 454989, "author_name": "", "author_url": "", "post_date": "2019-01-12T17:34:58.943000", "content": "", "votes": 0, "replies": [ { "id": 455116, "author_name": "Femi", "author_url": "", "post_date": "2019-01-13T02:40:21.090000", "content": "hah~, thanks! I am sorry that I have not tried stand-alone oversampling. My understanding of stand-alone oversampling is duplication of low frequency classes like code here. Am I right?\n\ntrain_df_orig=train_df.copy() \nSuppose my understanding is correct. I think such oversampling needs to decide which rare classes need to be oversampling and how many times they are duplicated. In other words, there are too many hyperparameters to be adjusted. You need to find the right parameters to get him to work. However, log weighted helped us decide all of this.\nI don't think log weighted is the most correct way, but I think it is the reasonable way. Hope to get your reply.
\nlows = [15,15,15,8,9,10,8,9,10,8,9,10,17,20,24,26,15,27,15,20,24,17,8,15,27,27,27]\nfor i in lows:\n target = str(i)\n indicies = train_df_orig.loc[train_df_orig['Target'] == target].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n indicies = train_df_orig.loc[train_df_orig['Target'].str.startswith(target+\" \")].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n indicies = train_df_orig.loc[train_df_orig['Target'].str.endswith(\" \"+target)].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n indicies = train_df_orig.loc[train_df_orig['Target'].str.contains(\" \"+target+\" \")].index\n train_df = pd.concat([train_df,train_df_orig.loc[indicies]], ignore_index=True)\n