{ "id": 189346, "title": "1st place \"mostly\" unpredictable solution", "url": "/competitions/osic-pulmonary-fibrosis-progression/writeups/art-1st-place-mostly-unpredictable-solution", "author_name": "", "post_date": "2020-10-07T10:18:13.432833600Z", "votes": 63, "comment_count": 23, "views": 0, "content": "

Hi all! I am really happy to be among the top scorers of this competition, even though I didn't expect that much. My best solution is heavily based on this notebook so kudos to @khoongweihao

\n

Introduction

\n

You might have noticed that during the competition there were a lot of public notebooks where authors just tuned the hyperparameters of the notebook mentioned above. When I started to solve this competition I noticed some unlogical parts in the notebooks, which for some reason resulted in a high score on the public leaderboard. My initial hypothesis was that on the private leaderboard those notebooks should fail, which turned out to be true.

\n

Here are some of the things that I have noticed:

\n\n

Validation

\n

I tried different techniques of validation and none of them really correlated with the leaderboard. Finally, I stopped on the following validation scheme:

\n

\"\"

\n

Even though I didn't notice any correlation with the public lb, I decided that this approach should be close to the one used in this competition for scoring. After creating this validation, if my model scored badly both on my validation and leaderboard - I disregarded it. But in terms of selecting the best submissions, it was still a black box for me.

\n

Training, models, and final solution

\n

I have tried a lot of things (I will add more about this below), but ironically the backbone to my best solution turned out to be this kernel :)

\n

Speaking about my final solution, here is what I have done to achieve this score. Firstly, I trained both models (Quantile Regression + EfficientNet b5) from scratch. For both models, I lowered the number of epochs. For effnet I decided to train for 30 epochs and for quantile regression for 600 epochs. Then, I changed the architecture of Quantile Regression a bit, because on validation my architecture worked better. Apart from that, I removed all the \"Percent\" related features for both models, it turned out that it gave a huge boost on private lb for me. The hardest decision was how to choose the weights for the blend. Well, I just decided to give a slightly higher weight to Quantile Regression because for me it seemed to work better. Finally, I did some more improvements for the backbone notebook, for example, there was a part with the quantile selection based on the best loglikelihood score for the EfficientNet models. This part took ages to finish and moreover, for me, it looked like not a good decision, so I have just set the quantile to 0.5 and didn't select anything, this allowed my Inference notebooks to run in just 3 minutes total.

\n

Also, I would like to give a small tip, on how I tend to select submissions and verify the prediction correctness, in general, this helps me a lot. When I have trained the new model and receive the submission file, I always draw distribution plots of prediction values, along with plots for predictions themselves. Here is how they look like:

\n

\"\"

\n

These are plots for the test set \"Confidence\" for a subset of my models, sometimes by looking at those plots you can identify strange model behavior and find a bug. In general, I always analyze the predictions really carefully and build a lot of graphs

\n

What didn't work

\n

As I have said I have tried a lot of stuff, but it almost always worked badly both on LB and CV. Here are a few things:

\n\n

\"\"

\n

It made total sense to remove them, but it didn't work on my validation, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.

\n

Final words

\n

This is my first gold on Kaggle and I am really happy about that. I am also happy about this huge shake-up which helped me land in the first place, which I wasn't expecting. I would like to thank all the Kaggle community for making so many notebooks public and being active on the forum! Without you guys, I wouldn't have learned that much during this competition and all other previous ones.

\n

And one last thing, don't ever track your public LB score, this mostly helps ;)

\n

Below I will attach links to my final submission notebooks, along with Medium writeup and Github repo.

\n

1st Place notebook

\n

Bronze zone very simple solution

\n

Medium writeup

\n

Github repository

", "messages": [ { "id": "1040736", "postDate": "10/07/2020 10:18:13", "content": "

Hi all! I am really happy to be among the top scorers of this competition, even though I didn't expect that much. My best solution is heavily based on this notebook so kudos to @khoongweihao

\n

Introduction

\n

You might have noticed that during the competition there were a lot of public notebooks where authors just tuned the hyperparameters of the notebook mentioned above. When I started to solve this competition I noticed some unlogical parts in the notebooks, which for some reason resulted in a high score on the public leaderboard. My initial hypothesis was that on the private leaderboard those notebooks should fail, which turned out to be true.

\n

Here are some of the things that I have noticed:

\n\n

Validation

\n

I tried different techniques of validation and none of them really correlated with the leaderboard. Finally, I stopped on the following validation scheme:

\n

\"\"

\n

Even though I didn't notice any correlation with the public lb, I decided that this approach should be close to the one used in this competition for scoring. After creating this validation, if my model scored badly both on my validation and leaderboard - I disregarded it. But in terms of selecting the best submissions, it was still a black box for me.

\n

Training, models, and final solution

\n

I have tried a lot of things (I will add more about this below), but ironically the backbone to my best solution turned out to be this kernel :)

\n

Speaking about my final solution, here is what I have done to achieve this score. Firstly, I trained both models (Quantile Regression + EfficientNet b5) from scratch. For both models, I lowered the number of epochs. For effnet I decided to train for 30 epochs and for quantile regression for 600 epochs. Then, I changed the architecture of Quantile Regression a bit, because on validation my architecture worked better. Apart from that, I removed all the \"Percent\" related features for both models, it turned out that it gave a huge boost on private lb for me. The hardest decision was how to choose the weights for the blend. Well, I just decided to give a slightly higher weight to Quantile Regression because for me it seemed to work better. Finally, I did some more improvements for the backbone notebook, for example, there was a part with the quantile selection based on the best loglikelihood score for the EfficientNet models. This part took ages to finish and moreover, for me, it looked like not a good decision, so I have just set the quantile to 0.5 and didn't select anything, this allowed my Inference notebooks to run in just 3 minutes total.

\n

Also, I would like to give a small tip, on how I tend to select submissions and verify the prediction correctness, in general, this helps me a lot. When I have trained the new model and receive the submission file, I always draw distribution plots of prediction values, along with plots for predictions themselves. Here is how they look like:

\n

\"\"

\n

These are plots for the test set \"Confidence\" for a subset of my models, sometimes by looking at those plots you can identify strange model behavior and find a bug. In general, I always analyze the predictions really carefully and build a lot of graphs

\n

What didn't work

\n

As I have said I have tried a lot of stuff, but it almost always worked badly both on LB and CV. Here are a few things:

\n\n

\"\"

\n

It made total sense to remove them, but it didn't work on my validation, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.

\n

Final words

\n

This is my first gold on Kaggle and I am really happy about that. I am also happy about this huge shake-up which helped me land in the first place, which I wasn't expecting. I would like to thank all the Kaggle community for making so many notebooks public and being active on the forum! Without you guys, I wouldn't have learned that much during this competition and all other previous ones.

\n

And one last thing, don't ever track your public LB score, this mostly helps ;)

\n

Below I will attach links to my final submission notebooks, along with Medium writeup and Github repo.

\n

1st Place notebook

\n

Bronze zone very simple solution

\n

Medium writeup

\n

Github repository

", "rawMarkdown": "Hi all! I am really happy to be among the top scorers of this competition, even though I didn't expect that much. My best solution is heavily based on [this notebook](https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference) so kudos to @khoongweihao\n\n**Introduction**\n\nYou might have noticed that during the competition there were a lot of public notebooks where authors just tuned the hyperparameters of the notebook mentioned above. When I started to solve this competition I noticed some unlogical parts in the notebooks, which for some reason resulted in a high score on the public leaderboard. My initial hypothesis was that on the private leaderboard those notebooks should fail, which turned out to be true. \n\nHere are some of the things that I have noticed:\n\n- Usage of the \"Percent\" feature, there were even discussions on the forum about the usefulness of this feature. While it wasn't obvious if it was really useful or not I decided not to use that\n- Strange blending weights for the models. To my mind, it was very illogical to give such a huge weight to the EfficientNet models because of 2 reasons. Firstly, we had only around 200 patients in the training set. Secondly, models were trained on the random slices of the initial 3d CT scans, which isn't a really reliable technique. \n\n**Validation**\n\nI tried different techniques of validation and none of them really correlated with the leaderboard. Finally, I stopped on the following validation scheme:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2139711%2Fe2fa663a66bb474595bbeb7b88fd7b7e%2Fscreen1.png?generation=1602065497962232&alt=media)\n\nEven though I didn't notice any correlation with the public lb, I decided that this approach should be close to the one used in this competition for scoring. After creating this validation, if my model scored badly both on my validation and leaderboard - I disregarded it. But in terms of selecting the best submissions, it was still a black box for me.\n\n**Training, models, and final solution**\n\nI have tried a lot of things (I will add more about this below), but ironically the backbone to my best solution turned out to be this [kernel](https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference) :)\n\nSpeaking about my final solution, here is what I have done to achieve this score. Firstly, I trained both models (Quantile Regression + EfficientNet b5) from scratch. For both models, I lowered the number of epochs. For effnet I decided to train for 30 epochs and for quantile regression for 600 epochs. Then, I changed the architecture of Quantile Regression a bit, because on validation my architecture worked better. Apart from that, I removed all the \"Percent\" related features for both models, it turned out that it gave a huge boost on private lb for me. The hardest decision was how to choose the weights for the blend. Well, I just decided to give a slightly higher weight to Quantile Regression because for me it seemed to work better. Finally, I did some more improvements for the backbone notebook, for example, there was a part with the quantile selection based on the best loglikelihood score for the EfficientNet models. This part took ages to finish and moreover, for me, it looked like not a good decision, so I have just set the quantile to 0.5 and didn't select anything, this allowed my Inference notebooks to run in just 3 minutes total.\n\nAlso, I would like to give a small tip, on how I tend to select submissions and verify the prediction correctness, in general, this helps me a lot. When I have trained the new model and receive the submission file, I always draw distribution plots of prediction values, along with plots for predictions themselves. Here is how they look like:\n\n \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2139711%2Fdb8ced80a746d1946238e60ba981273a%2FUntitled.png?generation=1602065623926772&alt=media)\n\nThese are plots for the test set \"Confidence\" for a subset of my models, sometimes by looking at those plots you can identify strange model behavior and find a bug. In general, I always analyze the predictions really carefully and build a lot of graphs\n\n**What didn't work**\n\nAs I have said I have tried a lot of stuff, but it almost always worked badly both on LB and CV. Here are a few things:\n\n- Calculated lung volume with methods from the public notebooks and passed it as features for both models\n- Tested other models, XGBoost, Log Regressions on tabular data. Thanks to my CV it immediately turned out that trees do not work here, so I didn't do anything with trees since the beginning of this competition.\n- Since I was testing simple models, my 2nd selected submission was a really simple logistic regression model, which by the way landed in the bronze zone\n- Augmentations for the CT scans worked bad, maybe I should have spent more time testing them\n- Histogram features of the image didn't work either\n- If you have analyzed model outputs, you might have noticed those spikes (both for Confidence and FVC)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2139711%2Fd0538eb2a86d7017f0042c79694ac4de%2FUntitled%20(1).png?generation=1602065688923432&alt=media)\n\nIt made total sense to remove them, but it didn't work on my validation, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.\n\n**Final words**\n\nThis is my first gold on Kaggle and I am really happy about that. I am also happy about this huge shake-up which helped me land in the first place, which I wasn't expecting. I would like to thank all the Kaggle community for making so many notebooks public and being active on the forum! Without you guys, I wouldn't have learned that much during this competition and all other previous ones.\n\nAnd one last thing, don't ever track your public LB score, this mostly helps ;)\n\nBelow I will attach links to my final submission notebooks, along with Medium writeup and Github repo.\n\n[1st Place notebook](https://www.kaggle.com/artkulak/inference-45-55-600-epochs-tuned-effnet-b5-30-ep)\n\n[Bronze zone very simple solution](https://www.kaggle.com/artkulak/simple-logreg?scriptVersionId=44081090)\n\n[Medium writeup](https://medium.com/@artkulakov/how-i-achieved-the-1st-place-in-kaggle-osic-pulmonary-fibrosis-progression-competition-e410962c4edc)\n\n[Github repository](https://github.com/artkulak/osic-pulmonary-fibrosis-progression)", "votes": null }, { "id": "1040829", "postDate": "10/07/2020 11:29:56", "content": "

Congratulation for the winning !

", "rawMarkdown": "Congratulation for the winning !", "votes": null }, { "id": "1040836", "postDate": "10/07/2020 11:36:04", "content": "

Thanks a lot!

", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "1040935", "postDate": "10/07/2020 13:03:09", "content": "

Congratulations.

\n

If I see that correctly, you are still leaking a lot of information through min_FVC
\nWithout leakage, you wouldn't see such a good prediction for your validation set.

\n

You should buy a lottery ticket today 😃

", "rawMarkdown": "Congratulations. \n\nIf I see that correctly, you are still leaking a lot of information through `min_FVC`\nWithout leakage, you wouldn't see such a good prediction for your validation set. \n\nYou should buy a lottery ticket today 😃", "votes": null }, { "id": "1040975", "postDate": "10/07/2020 13:30:30", "content": "

Thank you! Why do you think min_FVC leaks information? Isn't that just the initial measurement for each patient, which is present in the test set as well?

", "rawMarkdown": "Thank you! Why do you think min_FVC leaks information? Isn't that just the initial measurement for each patient, which is present in the test set as well?", "votes": null }, { "id": "1041013", "postDate": "10/07/2020 13:55:09", "content": "

I assumed by the name that it would be one of the last measurements ('min'). I could be mistaken. But you are clearly leaking information somewhere.

", "rawMarkdown": "I assumed by the name that it would be one of the last measurements ('min'). I could be mistaken. But you are clearly leaking information somewhere.", "votes": null }, { "id": "1041030", "postDate": "10/07/2020 14:05:45", "content": "

congrats for that

", "rawMarkdown": "congrats for that", "votes": null }, { "id": "1041298", "postDate": "10/07/2020 17:13:19", "content": "

Interesting. Would have never thought that one could win a competition this large by adapting the hyperparameters of a public notebook.
\nCongratulations!

", "rawMarkdown": "Interesting. Would have never thought that one could win a competition this large by adapting the hyperparameters of a public notebook.\nCongratulations!", "votes": null }, { "id": "1041467", "postDate": "10/07/2020 18:53:18", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "1041629", "postDate": "10/07/2020 21:02:34", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "1041666", "postDate": "10/07/2020 21:35:28", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "1041748", "postDate": "10/07/2020 22:33:51", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "1042386", "postDate": "10/08/2020 07:46:34", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "1042925", "postDate": "10/08/2020 14:58:08", "content": "

From the mentioned notebook :
\nbase = data.loc[data.Weeks == data.min_week]
\nbase = base[['Patient','FVC']].copy()
\nbase.columns = ['Patient','min_FVC']
\nWith min_week being the patient weeks minimum. I assume that's where the name comes from.

\n

@ilu000 do you think there are leakage because of those weird spikes ?

\n
\n

It made total sense to remove them, but it didn't work on my validation, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.

\n
\n

How did you try to remove them ? Where did this have a impact on your cv ?
\nI think it's just due to overfitting.

\n

There are 3 patterns in the confidence plots:
\n1) the gradually increasing confidence
\n2) the 5 very highly confident predictions
\n3) the close-to-stationary confidence

\n

For sure, 2) is related to those lines in the public kernel :
\notest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')
\nfor i in range(len(otest)):
\nsubm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]
\nsubm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1

\n

The blending is performed right after, it really looks like a perfect prediction attenuated by the blending.

\n

For 1) and 3), it looks like the 1) shows the effnet-based model dynamic (other public kernels using it eventually predict pretty large confidence values), and 3) is mostly due to the qreg model. I guess the qreg model is hardly overfitting on some weeks, predicting super tight confidence values. The model could even do something like \"after N weeks, be super confident\", noticing this pattern in the training set …

\n

Anyway, I guess that removing Percent from your model helped you to avoid overfitting on it (where so many of use did). Maybe the architecture modification you've made on the qreg model also had an impact; can you elaborate on that ?

\n

Congratulations on your win !

", "rawMarkdown": "From [the mentioned notebook](https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference) :\n`base = data.loc[data.Weeks == data.min_week]`\n`base = base[['Patient','FVC']].copy()`\n`base.columns = ['Patient','min_FVC']`\nWith min_week being the patient weeks minimum. I assume that's where the name comes from.\n\n@ilu000 do you think there are leakage because of those weird spikes ?\n\n\n> It made total sense to remove them, **but it didn't work on my validation**, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.\n\nHow did you try to remove them ? Where did this have a impact on your cv ?\nI think it's just due to overfitting. \n\nThere are 3 patterns in the confidence plots:\n1) the gradually increasing confidence\n2) the 5 very highly confident predictions\n3) the close-to-stationary confidence\n\nFor sure, 2) is related to those lines in the public kernel :\n`otest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')`\n`for i in range(len(otest)):`\n` subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]`\n` subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1`\n\nThe blending is performed right after, it really looks like a perfect prediction attenuated by the blending.\n\nFor 1) and 3), it looks like the 1) shows the effnet-based model dynamic (other public kernels using it eventually predict pretty large confidence values), and 3) is mostly due to the qreg model. I guess the qreg model is hardly overfitting on some weeks, predicting super tight confidence values. The model could even do something like \"after N weeks, be super confident\", noticing this pattern in the training set ...\n\nAnyway, I guess that removing `Percent` from your model helped you to avoid overfitting on it (where so many of use did). Maybe the architecture modification you've made on the qreg model also had an impact; can you elaborate on that ? \n\nCongratulations on your win !", "votes": null }, { "id": "1043640", "postDate": "10/09/2020 05:53:11", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "1043718", "postDate": "10/09/2020 07:33:45", "content": "

Thanks for sharing! Congrats!

", "rawMarkdown": "Thanks for sharing! Congrats!", "votes": null }, { "id": "1044235", "postDate": "10/09/2020 16:01:40", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "1045397", "postDate": "10/10/2020 15:28:15", "content": "

Thanks for sharing! Congrats!

", "rawMarkdown": "Thanks for sharing! Congrats!", "votes": null }, { "id": "1050534", "postDate": "10/15/2020 14:11:41", "content": "

Congrats this is awesome! Thank you for sharing!

", "rawMarkdown": "Congrats this is awesome! Thank you for sharing!", "votes": null }, { "id": "1050694", "postDate": "10/15/2020 16:24:41", "content": "

This is very interesting. worth a read!

", "rawMarkdown": "This is very interesting. worth a read!", "votes": null }, { "id": "1057059", "postDate": "10/22/2020 10:22:43", "content": "

What were your y-labels?

", "rawMarkdown": "What were your y-labels?", "votes": null }, { "id": "1063479", "postDate": "10/29/2020 01:09:34", "content": "

In your distribution plot of prediction values, what are x-axis and y-axis?

", "rawMarkdown": "In your distribution plot of prediction values, what are x-axis and y-axis?", "votes": null }, { "id": "1247173", "postDate": "03/21/2021 13:53:39", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "1509324", "postDate": "09/11/2021 06:19:56", "content": "

Dear please share your email address urgent work!

", "rawMarkdown": "Dear please share your email address urgent work!", "votes": null } ], "comments": [ { "id": 1040829, "author_name": "amedprof", "author_url": "", "post_date": "10/07/2020 11:29:56", "content": "

Congratulation for the winning !

", "votes": null, "replies": [] }, { "id": 1040836, "author_name": "vbaryshev", "author_url": "", "post_date": "10/07/2020 11:36:04", "content": "

Thanks a lot!

", "votes": null, "replies": [] }, { "id": 1040935, "author_name": "ilu000", "author_url": "", "post_date": "10/07/2020 13:03:09", "content": "

Congratulations.

\n

If I see that correctly, you are still leaking a lot of information through min_FVC
\nWithout leakage, you wouldn't see such a good prediction for your validation set.

\n

You should buy a lottery ticket today 😃

", "votes": null, "replies": [ { "id": 1040975, "author_name": "artkulak", "author_url": "", "post_date": "10/07/2020 13:30:30", "content": "

Thank you! Why do you think min_FVC leaks information? Isn't that just the initial measurement for each patient, which is present in the test set as well?

", "votes": null, "replies": [] }, { "id": 1041013, "author_name": "ilu000", "author_url": "", "post_date": "10/07/2020 13:55:09", "content": "

I assumed by the name that it would be one of the last measurements ('min'). I could be mistaken. But you are clearly leaking information somewhere.

", "votes": null, "replies": [] }, { "id": 1042925, "author_name": "johannhuber", "author_url": "", "post_date": "10/08/2020 14:58:08", "content": "

From the mentioned notebook :
\nbase = data.loc[data.Weeks == data.min_week]
\nbase = base[['Patient','FVC']].copy()
\nbase.columns = ['Patient','min_FVC']
\nWith min_week being the patient weeks minimum. I assume that's where the name comes from.

\n

@ilu000 do you think there are leakage because of those weird spikes ?

\n
\n

It made total sense to remove them, but it didn't work on my validation, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.

\n
\n

How did you try to remove them ? Where did this have a impact on your cv ?
\nI think it's just due to overfitting.

\n

There are 3 patterns in the confidence plots:
\n1) the gradually increasing confidence
\n2) the 5 very highly confident predictions
\n3) the close-to-stationary confidence

\n

For sure, 2) is related to those lines in the public kernel :
\notest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')
\nfor i in range(len(otest)):
\nsubm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]
\nsubm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1

\n

The blending is performed right after, it really looks like a perfect prediction attenuated by the blending.

\n

For 1) and 3), it looks like the 1) shows the effnet-based model dynamic (other public kernels using it eventually predict pretty large confidence values), and 3) is mostly due to the qreg model. I guess the qreg model is hardly overfitting on some weeks, predicting super tight confidence values. The model could even do something like \"after N weeks, be super confident\", noticing this pattern in the training set …

\n

Anyway, I guess that removing Percent from your model helped you to avoid overfitting on it (where so many of use did). Maybe the architecture modification you've made on the qreg model also had an impact; can you elaborate on that ?

\n

Congratulations on your win !

", "votes": null, "replies": [] } ] }, { "id": 1041030, "author_name": "elvinagammed", "author_url": "", "post_date": "10/07/2020 14:05:45", "content": "

congrats for that

", "votes": null, "replies": [] }, { "id": 1041298, "author_name": "bjaeger", "author_url": "", "post_date": "10/07/2020 17:13:19", "content": "

Interesting. Would have never thought that one could win a competition this large by adapting the hyperparameters of a public notebook.
\nCongratulations!

", "votes": null, "replies": [] }, { "id": 1041467, "author_name": "joewanga", "author_url": "", "post_date": "10/07/2020 18:53:18", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 1041629, "author_name": "marek3000", "author_url": "", "post_date": "10/07/2020 21:02:34", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 1041666, "author_name": "jonykarki", "author_url": "", "post_date": "10/07/2020 21:35:28", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 1041748, "author_name": "jiwongpark", "author_url": "", "post_date": "10/07/2020 22:33:51", "content": "

Thanks for sharing.

", "votes": null, "replies": [] }, { "id": 1042386, "author_name": "sunnykm", "author_url": "", "post_date": "10/08/2020 07:46:34", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 1043640, "author_name": "b06901135acac", "author_url": "", "post_date": "10/09/2020 05:53:11", "content": "

Thanks for sharing!

", "votes": null, "replies": [] }, { "id": 1043718, "author_name": "umarzubair95", "author_url": "", "post_date": "10/09/2020 07:33:45", "content": "

Thanks for sharing! Congrats!

", "votes": null, "replies": [ { "id": 1509324, "author_name": "chabdulrahman", "author_url": "", "post_date": "09/11/2021 06:19:56", "content": "

Dear please share your email address urgent work!

", "votes": null, "replies": [] } ] }, { "id": 1044235, "author_name": "jiazhen1496", "author_url": "", "post_date": "10/09/2020 16:01:40", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 1045397, "author_name": "", "author_url": "", "post_date": "10/10/2020 15:28:15", "content": "

Thanks for sharing! Congrats!

", "votes": null, "replies": [] }, { "id": 1050534, "author_name": "leonorfurtado", "author_url": "", "post_date": "10/15/2020 14:11:41", "content": "

Congrats this is awesome! Thank you for sharing!

", "votes": null, "replies": [] }, { "id": 1050694, "author_name": "gaurav2796", "author_url": "", "post_date": "10/15/2020 16:24:41", "content": "

This is very interesting. worth a read!

", "votes": null, "replies": [] }, { "id": 1057059, "author_name": "namanbansalcodes", "author_url": "", "post_date": "10/22/2020 10:22:43", "content": "

What were your y-labels?

", "votes": null, "replies": [] }, { "id": 1063479, "author_name": "dakomg", "author_url": "", "post_date": "10/29/2020 01:09:34", "content": "

In your distribution plot of prediction values, what are x-axis and y-axis?

", "votes": null, "replies": [] }, { "id": 1247173, "author_name": "hsuythomas", "author_url": "", "post_date": "03/21/2021 13:53:39", "content": "

Thanks for sharing!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1040736": "Hi all! I am really happy to be among the top scorers of this competition, even though I didn't expect that much. My best solution is heavily based on [this notebook](https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference) so kudos to @khoongweihao\n\n**Introduction**\n\nYou might have noticed that during the competition there were a lot of public notebooks where authors just tuned the hyperparameters of the notebook mentioned above. When I started to solve this competition I noticed some unlogical parts in the notebooks, which for some reason resulted in a high score on the public leaderboard. My initial hypothesis was that on the private leaderboard those notebooks should fail, which turned out to be true. \n\nHere are some of the things that I have noticed:\n\n- Usage of the \"Percent\" feature, there were even discussions on the forum about the usefulness of this feature. While it wasn't obvious if it was really useful or not I decided not to use that\n- Strange blending weights for the models. To my mind, it was very illogical to give such a huge weight to the EfficientNet models because of 2 reasons. Firstly, we had only around 200 patients in the training set. Secondly, models were trained on the random slices of the initial 3d CT scans, which isn't a really reliable technique. \n\n**Validation**\n\nI tried different techniques of validation and none of them really correlated with the leaderboard. Finally, I stopped on the following validation scheme:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2139711%2Fe2fa663a66bb474595bbeb7b88fd7b7e%2Fscreen1.png?generation=1602065497962232&alt=media)\n\nEven though I didn't notice any correlation with the public lb, I decided that this approach should be close to the one used in this competition for scoring. After creating this validation, if my model scored badly both on my validation and leaderboard - I disregarded it. But in terms of selecting the best submissions, it was still a black box for me.\n\n**Training, models, and final solution**\n\nI have tried a lot of things (I will add more about this below), but ironically the backbone to my best solution turned out to be this [kernel](https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference) :)\n\nSpeaking about my final solution, here is what I have done to achieve this score. Firstly, I trained both models (Quantile Regression + EfficientNet b5) from scratch. For both models, I lowered the number of epochs. For effnet I decided to train for 30 epochs and for quantile regression for 600 epochs. Then, I changed the architecture of Quantile Regression a bit, because on validation my architecture worked better. Apart from that, I removed all the \"Percent\" related features for both models, it turned out that it gave a huge boost on private lb for me. The hardest decision was how to choose the weights for the blend. Well, I just decided to give a slightly higher weight to Quantile Regression because for me it seemed to work better. Finally, I did some more improvements for the backbone notebook, for example, there was a part with the quantile selection based on the best loglikelihood score for the EfficientNet models. This part took ages to finish and moreover, for me, it looked like not a good decision, so I have just set the quantile to 0.5 and didn't select anything, this allowed my Inference notebooks to run in just 3 minutes total.\n\nAlso, I would like to give a small tip, on how I tend to select submissions and verify the prediction correctness, in general, this helps me a lot. When I have trained the new model and receive the submission file, I always draw distribution plots of prediction values, along with plots for predictions themselves. Here is how they look like:\n\n \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2139711%2Fdb8ced80a746d1946238e60ba981273a%2FUntitled.png?generation=1602065623926772&alt=media)\n\nThese are plots for the test set \"Confidence\" for a subset of my models, sometimes by looking at those plots you can identify strange model behavior and find a bug. In general, I always analyze the predictions really carefully and build a lot of graphs\n\n**What didn't work**\n\nAs I have said I have tried a lot of stuff, but it almost always worked badly both on LB and CV. Here are a few things:\n\n- Calculated lung volume with methods from the public notebooks and passed it as features for both models\n- Tested other models, XGBoost, Log Regressions on tabular data. Thanks to my CV it immediately turned out that trees do not work here, so I didn't do anything with trees since the beginning of this competition.\n- Since I was testing simple models, my 2nd selected submission was a really simple logistic regression model, which by the way landed in the bronze zone\n- Augmentations for the CT scans worked bad, maybe I should have spent more time testing them\n- Histogram features of the image didn't work either\n- If you have analyzed model outputs, you might have noticed those spikes (both for Confidence and FVC)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2139711%2Fd0538eb2a86d7017f0042c79694ac4de%2FUntitled%20(1).png?generation=1602065688923432&alt=media)\n\nIt made total sense to remove them, but it didn't work on my validation, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.\n\n**Final words**\n\nThis is my first gold on Kaggle and I am really happy about that. I am also happy about this huge shake-up which helped me land in the first place, which I wasn't expecting. I would like to thank all the Kaggle community for making so many notebooks public and being active on the forum! Without you guys, I wouldn't have learned that much during this competition and all other previous ones.\n\nAnd one last thing, don't ever track your public LB score, this mostly helps ;)\n\nBelow I will attach links to my final submission notebooks, along with Medium writeup and Github repo.\n\n[1st Place notebook](https://www.kaggle.com/artkulak/inference-45-55-600-epochs-tuned-effnet-b5-30-ep)\n\n[Bronze zone very simple solution](https://www.kaggle.com/artkulak/simple-logreg?scriptVersionId=44081090)\n\n[Medium writeup](https://medium.com/@artkulakov/how-i-achieved-the-1st-place-in-kaggle-osic-pulmonary-fibrosis-progression-competition-e410962c4edc)\n\n[Github repository](https://github.com/artkulak/osic-pulmonary-fibrosis-progression)", "1040829": "Congratulation for the winning !", "1040836": "Thanks a lot!", "1040935": "Congratulations. \n\nIf I see that correctly, you are still leaking a lot of information through `min_FVC`\nWithout leakage, you wouldn't see such a good prediction for your validation set. \n\nYou should buy a lottery ticket today 😃", "1040975": "Thank you! Why do you think min_FVC leaks information? Isn't that just the initial measurement for each patient, which is present in the test set as well?", "1041013": "I assumed by the name that it would be one of the last measurements ('min'). I could be mistaken. But you are clearly leaking information somewhere.", "1041030": "congrats for that", "1041298": "Interesting. Would have never thought that one could win a competition this large by adapting the hyperparameters of a public notebook.\nCongratulations!", "1041467": "Congratulations!", "1041629": "Congratulations!", "1041666": "Congratulations!", "1041748": "Thanks for sharing.", "1042386": "Congratulations!", "1042925": "From [the mentioned notebook](https://www.kaggle.com/khoongweihao/efficientnets-quantile-regression-inference) :\n`base = data.loc[data.Weeks == data.min_week]`\n`base = base[['Patient','FVC']].copy()`\n`base.columns = ['Patient','min_FVC']`\nWith min_week being the patient weeks minimum. I assume that's where the name comes from.\n\n@ilu000 do you think there are leakage because of those weird spikes ?\n\n\n> It made total sense to remove them, **but it didn't work on my validation**, so I left it as is. It turned out it wasn't working on the private test set as well. Though, I am still confused about the reason why it happened.\n\nHow did you try to remove them ? Where did this have a impact on your cv ?\nI think it's just due to overfitting. \n\nThere are 3 patterns in the confidence plots:\n1) the gradually increasing confidence\n2) the 5 very highly confident predictions\n3) the close-to-stationary confidence\n\nFor sure, 2) is related to those lines in the public kernel :\n`otest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')`\n`for i in range(len(otest)):`\n` subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]`\n` subm.loc[subm['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1`\n\nThe blending is performed right after, it really looks like a perfect prediction attenuated by the blending.\n\nFor 1) and 3), it looks like the 1) shows the effnet-based model dynamic (other public kernels using it eventually predict pretty large confidence values), and 3) is mostly due to the qreg model. I guess the qreg model is hardly overfitting on some weeks, predicting super tight confidence values. The model could even do something like \"after N weeks, be super confident\", noticing this pattern in the training set ...\n\nAnyway, I guess that removing `Percent` from your model helped you to avoid overfitting on it (where so many of use did). Maybe the architecture modification you've made on the qreg model also had an impact; can you elaborate on that ? \n\nCongratulations on your win !", "1043640": "Thanks for sharing!", "1043718": "Thanks for sharing! Congrats!", "1044235": "Congratulations!", "1045397": "Thanks for sharing! Congrats!", "1050534": "Congrats this is awesome! Thank you for sharing!", "1050694": "This is very interesting. worth a read!", "1057059": "What were your y-labels?", "1063479": "In your distribution plot of prediction values, what are x-axis and y-axis?", "1247173": "Thanks for sharing!", "1509324": "Dear please share your email address urgent work!" }, "source": "meta" }