![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2779944%2Fb308f42589cfdbe1a46171a6112bad94%2FUntitled%20Diagram.png?generation=1602038817737249&alt=media\")
First of all Congratulations to all the winners and everyone who survived the big shakeup . Thanks to the organizers for this competition , I surely learned a lot . As much as I am disappointed at my terrible failure in this competition , I more curious so as to why I failed . I worked hard in this competition and therefore its more critical for me to understand what went wrong so that I don't repeat it in future.
\nI would really appreciate if experts of the community could share some thoughts of what they think went wrong with my approach. Below is the writeup of my approach :-
\nWe had four underlying models which we used for blending for final submission . The final submission selection was done based on the cv score of the final blend .
\nSubmission 1 : Blend CV - 6.622 , Blend LB - 6.90 ( Quantile Reg + ImgQuantile Reg + Linear Decay )
\nSubmission 2 : Blend CV - 6.626 , Blend LB - 6.92 ( Quantile Reg + IMGQuantile Reg + Tabular data linear regression Model )
I used 5-fold Group cross validation strategy to validate my models ( Grouping has been done on the Patient - ID) . The folds can be found here
\nThe first model that we had was Ulrich's Quantile Regression which I adjusted to find the best CV based on our folds and added some post-processing ( copying the FVC wherever it was present in the test set) . The kernel for this model can be found here
\nCV - 6.602 , Public Lb - 6.9270 , Private LB - 7.03
One of the tricky parts of this competition was to extract features from the CT scans during this competition , So I thought why not make use of CNN's and learn features from Images and pass it through the Quantile Regressor Model . I also Pre-preprocessed the CT scans using the techniques mentioned in Laura's kernel , the figure below describes the model
\n
The code for the model can be found here
\nThe CNN architecture I used here is EFFNET B5
\nCV - 6.727
\nLB - 6.98
\nPrivate - 7.06
This model was trained on just the tabular data with some feature engineering , the code for this model can be found here
\nCV - 6.84
\nLB - 6.88
\nPrivate - 6.97
This was the fork of the lovely public model , we used B3 instead of B5 because it gave the best cv for us . Since this model predicted gradients and not the FVC directly we didn't have OOFS for this so I was a bit skeptical to use it in blend . I then finally decided to blend the above mentioned three models and then take a mean with Linear Decay model's prediction and then make one final submission with LD model and one without
\nI prepared the Following Three blends :
\nI can make public all the blend code as well , I just need some clarity on my end , any help from the community will be highly motivating for me
\nHere are the reasons I think what went wrong :-
\nFirst of all Congratulations to all the winners and everyone who survived the big shakeup . Thanks to the organizers for this competition , I surely learned a lot . As much as I am disappointed at my terrible failure in this competition , I more curious so as to why I failed . I worked hard in this competition and therefore its more critical for me to understand what went wrong so that I don't repeat it in future.
\nI would really appreciate if experts of the community could share some thoughts of what they think went wrong with my approach. Below is the writeup of my approach :-
\nWe had four underlying models which we used for blending for final submission . The final submission selection was done based on the cv score of the final blend .
\nSubmission 1 : Blend CV - 6.622 , Blend LB - 6.90 ( Quantile Reg + ImgQuantile Reg + Linear Decay )
\nSubmission 2 : Blend CV - 6.626 , Blend LB - 6.92 ( Quantile Reg + IMGQuantile Reg + Tabular data linear regression Model )
I used 5-fold Group cross validation strategy to validate my models ( Grouping has been done on the Patient - ID) . The folds can be found here
\nThe first model that we had was Ulrich's Quantile Regression which I adjusted to find the best CV based on our folds and added some post-processing ( copying the FVC wherever it was present in the test set) . The kernel for this model can be found here
\nCV - 6.602 , Public Lb - 6.9270 , Private LB - 7.03
One of the tricky parts of this competition was to extract features from the CT scans during this competition , So I thought why not make use of CNN's and learn features from Images and pass it through the Quantile Regressor Model . I also Pre-preprocessed the CT scans using the techniques mentioned in Laura's kernel , the figure below describes the model
\n
The code for the model can be found here
\nThe CNN architecture I used here is EFFNET B5
\nCV - 6.727
\nLB - 6.98
\nPrivate - 7.06
This model was trained on just the tabular data with some feature engineering , the code for this model can be found here
\nCV - 6.84
\nLB - 6.88
\nPrivate - 6.97
This was the fork of the lovely public model , we used B3 instead of B5 because it gave the best cv for us . Since this model predicted gradients and not the FVC directly we didn't have OOFS for this so I was a bit skeptical to use it in blend . I then finally decided to blend the above mentioned three models and then take a mean with Linear Decay model's prediction and then make one final submission with LD model and one without
\nI prepared the Following Three blends :
\nI can make public all the blend code as well , I just need some clarity on my end , any help from the community will be highly motivating for me
\nHere are the reasons I think what went wrong :-
\nBelow are a few things that come to my mind, hope you find it helpful:
\n\n\nWhile extracting features from images using CNN you could have randomly sampled multiple images from the middle 30-60 percentile CT scan images for each patient to reduce the variance.
\n
I am doing the same thing
\n\n\nHow did you use the Percent feature? While training the model did you include only the first Percent value or considered Percent from all weeks or completely ignored it?
\n
I have used the First Percent Value
\n\n\nValidation should have been done on last three weeks only.
\n
\n While blending the models, weights for the blend should have been choosen based on best OOF score for last 3 weeks.
Point Noted these two have been the major Things I missed
\nThanks a lot for your reply Abhishek
", "rawMarkdown": "> While extracting features from images using CNN you could have randomly sampled multiple images from the middle 30-60 percentile CT scan images for each patient to reduce the variance.\n\nI am doing the same thing\n\n> How did you use the Percent feature? While training the model did you include only the first Percent value or considered Percent from all weeks or completely ignored it?\n\nI have used the First Percent Value\n\n> Validation should have been done on last three weeks only.\nWhile blending the models, weights for the blend should have been choosen based on best OOF score for last 3 weeks.\n\nPoint Noted these two have been the major Things I missed\n\nThanks a lot for your reply Abhishek", "votes": null } ], "comments": [ { "id": 1040241, "author_name": "abhishekgbhat", "author_url": "", "post_date": "10/07/2020 03:39:39", "content": "Below are a few things that come to my mind, hope you find it helpful:
\n\n\nWhile extracting features from images using CNN you could have randomly sampled multiple images from the middle 30-60 percentile CT scan images for each patient to reduce the variance.
\n
I am doing the same thing
\n\n\nHow did you use the Percent feature? While training the model did you include only the first Percent value or considered Percent from all weeks or completely ignored it?
\n
I have used the First Percent Value
\n\n\nValidation should have been done on last three weeks only.
\n
\n While blending the models, weights for the blend should have been choosen based on best OOF score for last 3 weeks.
Point Noted these two have been the major Things I missed
\nThanks a lot for your reply Abhishek
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1040205": "First of all Congratulations to all the winners and everyone who survived the big shakeup . Thanks to the organizers for this competition , I surely learned a lot . As much as I am disappointed at my terrible failure in this competition , I more curious so as to why I failed . I worked hard in this competition and therefore its more critical for me to understand what went wrong so that I don't repeat it in future.\n\n I would really appreciate if experts of the community could share some thoughts of what they think went wrong with my approach. Below is the writeup of my approach :-\n\n# Introduction\n\nWe had four underlying models which we used for blending for final submission . The final submission selection was done based on the cv score of the final blend . \n\nSubmission 1 : Blend CV - 6.622 , Blend LB - 6.90 ( Quantile Reg + ImgQuantile Reg + Linear Decay )\nSubmission 2 : Blend CV - 6.626 , Blend LB - 6.92 ( Quantile Reg + IMGQuantile Reg + Tabular data linear regression Model )\n\n# CV Strategy\n\nI used 5-fold Group cross validation strategy to validate my models ( Grouping has been done on the Patient - ID) . The folds can be found [here] (https://www.kaggle.com/tanulsingh077/osic-folds)\n\n# Models\n\n* Quantile - Regression Model \n\nThe first model that we had was Ulrich's Quantile Regression which I adjusted to find the best CV based on our folds and added some post-processing ( copying the FVC wherever it was present in the test set) . The kernel for this model can be found [here] (https://www.kaggle.com/tanulsingh077/osic-quantile-regressor-baseline?scriptVersionId=43711584)\nCV - 6.602 , Public Lb - 6.9270 , Private LB - 7.03 \n\n* Image + Quantile Regressor\n\nOne of the tricky parts of this competition was to extract features from the CT scans during this competition , So I thought why not make use of CNN's and learn features from Images and pass it through the Quantile Regressor Model . I also Pre-preprocessed the CT scans using the techniques mentioned in Laura's kernel , the figure below describes the model \n\n\n\nThe code for the model can be found [here] (https://www.kaggle.com/tanulsingh077/osic-image-tabular-data-multi-quantile-fast)\n\nThe CNN architecture I used here is EFFNET B5\nCV - 6.727\nLB - 6.98\nPrivate - 7.06\n\n* Lasso Regressor\n\nThis model was trained on just the tabular data with some feature engineering , the code for this model can be found [here] (https://www.kaggle.com/tanulsingh077/osic-tabular-pipeline?scriptVersionId=44049809)\n\nCV - 6.84\nLB - 6.88\nPrivate - 6.97\n\n* Linear Decay Model \n\nThis was the fork of the lovely public model , we used B3 instead of B5 because it gave the best cv for us . Since this model predicted gradients and not the FVC directly we didn't have OOFS for this so I was a bit skeptical to use it in blend . I then finally decided to blend the above mentioned three models and then take a mean with Linear Decay model's prediction and then make one final submission with LD model and one without\n\n# Final Blend\n\nI prepared the Following Three blends :\n* Quant-Regression + IMG Quant-regression + Tabular Data Model \nOOF_CV - 6.625 \n* Quant- Regression + IMG Quant-Regression\nOOF_CV - 6.622\n* Quant-Regression + IMG Quant-Regression + Tabular Data Model + Linear Decay EffnetB3 model \n\nI can make public all the blend code as well , I just need some clarity on my end , any help from the community will be highly motivating for me\n\nHere are the reasons I think what went wrong :-\n* I should Have validated only on the last three weeks for every patient instead of just normal validation\n* I should have gone more stable single models which I missed which have score in the silver and gold zone", "1040241": "Below are a few things that come to my mind, hope you find it helpful:\n- While extracting features from images using CNN you could have randomly sampled multiple images from the middle 30-60 percentile CT scan images for each patient to reduce the variance.\n- How did you use the Percent feature? While training the model did you include only the first Percent value or considered Percent from all weeks or completely ignored it?\n- Validation should have been done on last three weeks only.\n- While blending the models, weights for the blend should have been choosen based on best OOF score for last 3 weeks.", "1040429": "> While extracting features from images using CNN you could have randomly sampled multiple images from the middle 30-60 percentile CT scan images for each patient to reduce the variance.\n\nI am doing the same thing\n\n> How did you use the Percent feature? While training the model did you include only the first Percent value or considered Percent from all weeks or completely ignored it?\n\nI have used the First Percent Value\n\n> Validation should have been done on last three weeks only.\nWhile blending the models, weights for the blend should have been choosen based on best OOF score for last 3 weeks.\n\nPoint Noted these two have been the major Things I missed\n\nThanks a lot for your reply Abhishek" }, "source": "meta" }