Thanks for organizers to host this great competition, and good job for those who survived a big shake-up!
\nThis was a competition with medical data of a rare disease, in which the number of patients is relatively small. So the final standing was hard to predict, as the training sample might represent nothing about the private dataset.
\nMy concern was exactly this: Trusting LB is out of the question but should I really trust CV?
\nEventually I trusted my CV, and got the solo Bronze medal, which is alright. My solution used both tabular and image data, and first predicting FVC with many models (2DCNN, MLP, GBDT, linear…) then optimized confidence also with many models.
\nEventually it turned out that I also have a gold medal submission, which I could not choose because its LB as well as CV were not great. Basically the main difference between the two was
\nApparently the latter improves CV but also overfits to the training samples. The former had relatively worse CV but better LB.
\nSo how I missed my first gold medal?
\nThe answer seems to me that I overfitted the training samples and trusted my CV too much.
\nI should have taken into account the danger of overfitting to the training samples in this kind of datasets with small samples, and should have picked up a relatively robust way of ensemble such as stacking with linear model.
", "messages": [ { "id": "1040056", "postDate": "10/07/2020 00:56:44", "content": "Thanks for organizers to host this great competition, and good job for those who survived a big shake-up!
\nThis was a competition with medical data of a rare disease, in which the number of patients is relatively small. So the final standing was hard to predict, as the training sample might represent nothing about the private dataset.
\nMy concern was exactly this: Trusting LB is out of the question but should I really trust CV?
\nEventually I trusted my CV, and got the solo Bronze medal, which is alright. My solution used both tabular and image data, and first predicting FVC with many models (2DCNN, MLP, GBDT, linear…) then optimized confidence also with many models.
\nEventually it turned out that I also have a gold medal submission, which I could not choose because its LB as well as CV were not great. Basically the main difference between the two was
\nApparently the latter improves CV but also overfits to the training samples. The former had relatively worse CV but better LB.
\nSo how I missed my first gold medal?
\nThe answer seems to me that I overfitted the training samples and trusted my CV too much.
\nI should have taken into account the danger of overfitting to the training samples in this kind of datasets with small samples, and should have picked up a relatively robust way of ensemble such as stacking with linear model.
", "rawMarkdown": "Thanks for organizers to host this great competition, and good job for those who survived a big shake-up! \n\nThis was a competition with medical data of a rare disease, in which the number of patients is relatively small. So the final standing was hard to predict, as the training sample might represent nothing about the private dataset.\n\nMy concern was exactly this: **Trusting LB is out of the question but should I really trust CV?**\n\nEventually I trusted my CV, and got the solo Bronze medal, which is alright. My solution used both tabular and image data, and first predicting FVC with many models (2DCNN, MLP, GBDT, linear...) then optimized confidence also with many models. \n\nEventually it turned out that I also have **a gold medal submission, which I could not choose because its LB as well as CV were not great**. Basically the main difference between the two was \n\n- stacking with linear model (Bayesian Ridge) --> gold (CV: -6.9986, public: -6.9613, private: -6.8379)\n- multilayer stacking (MLP+XGB -> Bayesian Ridge) --> bronze (CV: -6.9415, public: -6.9822, private: -6.8680)\n\nApparently the latter improves CV but also overfits to the training samples. The former had relatively worse CV but better LB.\n\nSo how I missed my first gold medal? \n\nThe answer seems to me that **I overfitted the training samples and trusted my CV too much**. \n\nI should have taken into account the danger of overfitting to the training samples in this kind of datasets with small samples, and should have picked up a relatively robust way of ensemble such as stacking with linear model.", "votes": null }, { "id": "1040139", "postDate": "10/07/2020 01:58:33", "content": "Hi, can i know the details of your validation set in cv? as in was containing all the weeks or just latest 3 weeks datapoints?
", "rawMarkdown": "Hi, can i know the details of your validation set in cv? as in was containing all the weeks or just latest 3 weeks datapoints?", "votes": null }, { "id": "1040160", "postDate": "10/07/2020 02:17:27", "content": "See my solution (please have a look at Version 1) here.
", "rawMarkdown": "For FVC prediction: all weeks (stratified group kfold based on the degree of linear decay and patient as a group)\n- For optimized Confidence prediction: last 3 datapoints (same above but replacing non last 3 datapoints with randomly chosen last 3 datapoints)\n\nSee my solution (please have a look at **Version 1**) [here](https://www.kaggle.com/code1110/osic-cv-tab-cnn-ensemble-ocne-nop-lasts#Final-CV-score).", "votes": null } ], "comments": [ { "id": 1040139, "author_name": "mugdhahardikar", "author_url": "", "post_date": "10/07/2020 01:58:33", "content": "Hi, can i know the details of your validation set in cv? as in was containing all the weeks or just latest 3 weeks datapoints?
", "votes": null, "replies": [ { "id": 1040160, "author_name": "code1110", "author_url": "", "post_date": "10/07/2020 02:17:27", "content": "See my solution (please have a look at Version 1) here.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1040056": "Thanks for organizers to host this great competition, and good job for those who survived a big shake-up! \n\nThis was a competition with medical data of a rare disease, in which the number of patients is relatively small. So the final standing was hard to predict, as the training sample might represent nothing about the private dataset.\n\nMy concern was exactly this: **Trusting LB is out of the question but should I really trust CV?**\n\nEventually I trusted my CV, and got the solo Bronze medal, which is alright. My solution used both tabular and image data, and first predicting FVC with many models (2DCNN, MLP, GBDT, linear...) then optimized confidence also with many models. \n\nEventually it turned out that I also have **a gold medal submission, which I could not choose because its LB as well as CV were not great**. Basically the main difference between the two was \n\n- stacking with linear model (Bayesian Ridge) --> gold (CV: -6.9986, public: -6.9613, private: -6.8379)\n- multilayer stacking (MLP+XGB -> Bayesian Ridge) --> bronze (CV: -6.9415, public: -6.9822, private: -6.8680)\n\nApparently the latter improves CV but also overfits to the training samples. The former had relatively worse CV but better LB.\n\nSo how I missed my first gold medal? \n\nThe answer seems to me that **I overfitted the training samples and trusted my CV too much**. \n\nI should have taken into account the danger of overfitting to the training samples in this kind of datasets with small samples, and should have picked up a relatively robust way of ensemble such as stacking with linear model.", "1040139": "Hi, can i know the details of your validation set in cv? as in was containing all the weeks or just latest 3 weeks datapoints?", "1040160": "For FVC prediction: all weeks (stratified group kfold based on the degree of linear decay and patient as a group)\n- For optimized Confidence prediction: last 3 datapoints (same above but replacing non last 3 datapoints with randomly chosen last 3 datapoints)\n\nSee my solution (please have a look at **Version 1**) [here](https://www.kaggle.com/code1110/osic-cv-tab-cnn-ensemble-ocne-nop-lasts#Final-CV-score)." }, "source": "meta" }