{
  "id": 189232,
  "title": "34th place solution - Using CT Scans (+ Code)",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/writeups/greatgamedota-34th-place-solution-using-ct-scans-c",
  "author_name": "",
  "post_date": "2020-10-10T16:23:37.500Z",
  "votes": 12,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I did not expect to win this comp so I was not immediately prepared with my solution or the code. I have made my code available here: <a href=\"https://github.com/GreatGameDota/OSIC-Pulmonary-Fibrosis-Prediction\" target=\"_blank\">https://github.com/GreatGameDota/OSIC-Pulmonary-Fibrosis-Prediction</a></p>\n<p>As always I want to thank Kaggle for another amazing competition as well as the hosts for putting this together!</p>\n<p>This was an interesting competition for me because I saw that everyone else were only using the tabular data. I have no experience at all with tabular data comps and I like CV so I was stubborn and forced myself to use the CT scans. I was discouraged by my poor LB score but I keep pushing and it turned out to be the correct decision!</p>\n<p>Anyway on to the solution:</p>\n<h2>Models</h2>\n<ul>\n<li>Resnet50</li>\n</ul>\n<h2>Dataset</h2>\n<p>Image resolution: 512x512<br>\nFor each scan I loaded in all the dicom files then converted them to HU. I then cropped all the images and reshaped them to the size 50x512x512. After doing that I then windowed each image into three parts. For a detailed explanation of the windowing take a look at <a href=\"https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930\" target=\"_blank\">this</a> post (I used the same function and window values as shared there, thanks so much Ian Pan!). After I got the three windowed images I simply saved them as pngs.<br>\nEach image looked similar to this:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F3b1f2461da6d31658806e1ace841dd5f%2Flung.png?generation=1602033034958099&amp;alt=media\" alt=\"\"></p>\n<p>I made the dataset I used public here: <a href=\"https://www.kaggle.com/greatgamedota/osic-windowed-lung-images\" target=\"_blank\">https://www.kaggle.com/greatgamedota/osic-windowed-lung-images</a></p>\n<p>For meta data I used the same meta data from <a href=\"https://www.kaggle.com/ulrich07\" target=\"_blank\">@ulrich07</a> 's <a href=\"https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter\" target=\"_blank\">baseline kernal</a> except I only used the base Percent value as it increased my CV to LB to PB correlation.</p>\n<h2>Augmentation</h2>\n<ul>\n<li>Coarse dropout</li>\n<li>SSR</li>\n<li>Horizontal + Vertical flip</li>\n<li>For one model: Random Saturation and Brightness</li>\n</ul>\n<p>No tabular/meta augmentation</p>\n<h2>Training</h2>\n<ul>\n<li>Adam optimizer with Reduce on Plateau scheduler</li>\n<li>Trained with an LR of .003 for 30 epochs</li>\n<li>Batch size of 16 (bs of 4 for one model)</li>\n<li>Trained using Quantile Regression with .8 qloss + .2 metric loss (same as Ulrich's kernal)</li>\n<li>Trained 5 fold for every model</li>\n<li>Didn't use any batch accumulation or mixed precision</li>\n<li>Saved checkpoint based on best validation score (all weeks)</li>\n</ul>\n<p>For training I removed 6 entire patients because their CT scans were broken. I then split each patient into a fold by randomly shuffling them then using GroupKFold.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F69a1967fbe3f4067d7fd9f6e5e91350c%2Ffolds.png?generation=1602036568284315&amp;alt=media\" alt=\"\"></p>\n<p>Then while training I pick a random unique patient and randomly select an image from 10-40 (since the first and last 10 images don't contain any lung info). I then made it so that each iteration lasted 4 * amount of patients. For validation I only picked the 15th image since it was the middle image that usually contains the most information.</p>\n<h2>Ensembling/Blending</h2>\n<p>Simple mean average of their FVC predictions and confidence</p>\n<h2>Final Submission</h2>\n<p>My final submission was a blend of 3 Resnet50 models:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F773b5cc6fdf39b2fe77ec08d4b295684%2Ffinal%20sub.png?generation=1602040599569002&amp;alt=media\" alt=\"\"><br>\nAnd another point is that the single model that would have scored gold had the same training parameters as the other models except the added brightness/saturation augmentation</p>\n<h2>What didn't work</h2>\n<ul>\n<li>3d resnets (I tried for at least a month with these)</li>\n<li>Linear Decay Regression</li>\n<li>Any other type of model besides Resnets and Efficientnets (determinism issues)</li>\n<li>Efficientnets</li>\n<li>Simple meta data head (just concat)</li>\n<li>Random erase augmentation</li>\n</ul>\n<h2>Final Thoughts</h2>\n<p>I want to say again that this was an awesome comp that I am so glad I participated in!! Very glad to get my third medal and second silver medal! :D</p>\n<h2>Some helpful links</h2>\n<p>Inference notebook: <a href=\"https://www.kaggle.com/greatgamedota/osic-inference2?scriptVersionId=44254523\" target=\"_blank\">https://www.kaggle.com/greatgamedota/osic-inference2?scriptVersionId=44254523</a></p>",
  "messages": [
    {
      "id": "1040217",
      "postDate": "10/07/2020 03:21:30",
      "content": "<p>I did not expect to win this comp so I was not immediately prepared with my solution or the code. I have made my code available here: <a href=\"https://github.com/GreatGameDota/OSIC-Pulmonary-Fibrosis-Prediction\" target=\"_blank\">https://github.com/GreatGameDota/OSIC-Pulmonary-Fibrosis-Prediction</a></p>\n<p>As always I want to thank Kaggle for another amazing competition as well as the hosts for putting this together!</p>\n<p>This was an interesting competition for me because I saw that everyone else were only using the tabular data. I have no experience at all with tabular data comps and I like CV so I was stubborn and forced myself to use the CT scans. I was discouraged by my poor LB score but I keep pushing and it turned out to be the correct decision!</p>\n<p>Anyway on to the solution:</p>\n<h2>Models</h2>\n<ul>\n<li>Resnet50</li>\n</ul>\n<h2>Dataset</h2>\n<p>Image resolution: 512x512<br>\nFor each scan I loaded in all the dicom files then converted them to HU. I then cropped all the images and reshaped them to the size 50x512x512. After doing that I then windowed each image into three parts. For a detailed explanation of the windowing take a look at <a href=\"https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930\" target=\"_blank\">this</a> post (I used the same function and window values as shared there, thanks so much Ian Pan!). After I got the three windowed images I simply saved them as pngs.<br>\nEach image looked similar to this:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F3b1f2461da6d31658806e1ace841dd5f%2Flung.png?generation=1602033034958099&amp;alt=media\" alt=\"\"></p>\n<p>I made the dataset I used public here: <a href=\"https://www.kaggle.com/greatgamedota/osic-windowed-lung-images\" target=\"_blank\">https://www.kaggle.com/greatgamedota/osic-windowed-lung-images</a></p>\n<p>For meta data I used the same meta data from <a href=\"https://www.kaggle.com/ulrich07\" target=\"_blank\">@ulrich07</a> 's <a href=\"https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter\" target=\"_blank\">baseline kernal</a> except I only used the base Percent value as it increased my CV to LB to PB correlation.</p>\n<h2>Augmentation</h2>\n<ul>\n<li>Coarse dropout</li>\n<li>SSR</li>\n<li>Horizontal + Vertical flip</li>\n<li>For one model: Random Saturation and Brightness</li>\n</ul>\n<p>No tabular/meta augmentation</p>\n<h2>Training</h2>\n<ul>\n<li>Adam optimizer with Reduce on Plateau scheduler</li>\n<li>Trained with an LR of .003 for 30 epochs</li>\n<li>Batch size of 16 (bs of 4 for one model)</li>\n<li>Trained using Quantile Regression with .8 qloss + .2 metric loss (same as Ulrich's kernal)</li>\n<li>Trained 5 fold for every model</li>\n<li>Didn't use any batch accumulation or mixed precision</li>\n<li>Saved checkpoint based on best validation score (all weeks)</li>\n</ul>\n<p>For training I removed 6 entire patients because their CT scans were broken. I then split each patient into a fold by randomly shuffling them then using GroupKFold.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F69a1967fbe3f4067d7fd9f6e5e91350c%2Ffolds.png?generation=1602036568284315&amp;alt=media\" alt=\"\"></p>\n<p>Then while training I pick a random unique patient and randomly select an image from 10-40 (since the first and last 10 images don't contain any lung info). I then made it so that each iteration lasted 4 * amount of patients. For validation I only picked the 15th image since it was the middle image that usually contains the most information.</p>\n<h2>Ensembling/Blending</h2>\n<p>Simple mean average of their FVC predictions and confidence</p>\n<h2>Final Submission</h2>\n<p>My final submission was a blend of 3 Resnet50 models:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F773b5cc6fdf39b2fe77ec08d4b295684%2Ffinal%20sub.png?generation=1602040599569002&amp;alt=media\" alt=\"\"><br>\nAnd another point is that the single model that would have scored gold had the same training parameters as the other models except the added brightness/saturation augmentation</p>\n<h2>What didn't work</h2>\n<ul>\n<li>3d resnets (I tried for at least a month with these)</li>\n<li>Linear Decay Regression</li>\n<li>Any other type of model besides Resnets and Efficientnets (determinism issues)</li>\n<li>Efficientnets</li>\n<li>Simple meta data head (just concat)</li>\n<li>Random erase augmentation</li>\n</ul>\n<h2>Final Thoughts</h2>\n<p>I want to say again that this was an awesome comp that I am so glad I participated in!! Very glad to get my third medal and second silver medal! :D</p>\n<h2>Some helpful links</h2>\n<p>Inference notebook: <a href=\"https://www.kaggle.com/greatgamedota/osic-inference2?scriptVersionId=44254523\" target=\"_blank\">https://www.kaggle.com/greatgamedota/osic-inference2?scriptVersionId=44254523</a></p>",
      "rawMarkdown": "I did not expect to win this comp so I was not immediately prepared with my solution or the code. I have made my code available here: https://github.com/GreatGameDota/OSIC-Pulmonary-Fibrosis-Prediction\n\nAs always I want to thank Kaggle for another amazing competition as well as the hosts for putting this together!\n\nThis was an interesting competition for me because I saw that everyone else were only using the tabular data. I have no experience at all with tabular data comps and I like CV so I was stubborn and forced myself to use the CT scans. I was discouraged by my poor LB score but I keep pushing and it turned out to be the correct decision!\n\nAnyway on to the solution:\n\n## Models\n\n- Resnet50\n\n## Dataset\n\nImage resolution: 512x512\nFor each scan I loaded in all the dicom files then converted them to HU. I then cropped all the images and reshaped them to the size 50x512x512. After doing that I then windowed each image into three parts. For a detailed explanation of the windowing take a look at [this](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930) post (I used the same function and window values as shared there, thanks so much Ian Pan!). After I got the three windowed images I simply saved them as pngs.\nEach image looked similar to this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F3b1f2461da6d31658806e1ace841dd5f%2Flung.png?generation=1602033034958099&alt=media)\n\nI made the dataset I used public here: https://www.kaggle.com/greatgamedota/osic-windowed-lung-images\n\nFor meta data I used the same meta data from @ulrich07 's [baseline kernal](https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter) except I only used the base Percent value as it increased my CV to LB to PB correlation.\n\n## Augmentation\n\n- Coarse dropout\n- SSR\n- Horizontal + Vertical flip\n- For one model: Random Saturation and Brightness\n\nNo tabular/meta augmentation\n\n## Training\n\n- Adam optimizer with Reduce on Plateau scheduler\n- Trained with an LR of .003 for 30 epochs\n- Batch size of 16 (bs of 4 for one model)\n- Trained using Quantile Regression with .8 qloss + .2 metric loss (same as Ulrich's kernal)\n- Trained 5 fold for every model\n- Didn't use any batch accumulation or mixed precision\n- Saved checkpoint based on best validation score (all weeks)\n\nFor training I removed 6 entire patients because their CT scans were broken. I then split each patient into a fold by randomly shuffling them then using GroupKFold.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F69a1967fbe3f4067d7fd9f6e5e91350c%2Ffolds.png?generation=1602036568284315&alt=media)\n\nThen while training I pick a random unique patient and randomly select an image from 10-40 (since the first and last 10 images don't contain any lung info). I then made it so that each iteration lasted 4 * amount of patients. For validation I only picked the 15th image since it was the middle image that usually contains the most information.\n\n## Ensembling/Blending\n\nSimple mean average of their FVC predictions and confidence\n\n## Final Submission\n\nMy final submission was a blend of 3 Resnet50 models:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F773b5cc6fdf39b2fe77ec08d4b295684%2Ffinal%20sub.png?generation=1602040599569002&alt=media)\nAnd another point is that the single model that would have scored gold had the same training parameters as the other models except the added brightness/saturation augmentation\n\n## What didn't work\n\n- 3d resnets (I tried for at least a month with these)\n- Linear Decay Regression\n- Any other type of model besides Resnets and Efficientnets (determinism issues)\n- Efficientnets\n- Simple meta data head (just concat)\n- Random erase augmentation\n\n## Final Thoughts\n\nI want to say again that this was an awesome comp that I am so glad I participated in!! Very glad to get my third medal and second silver medal! :D\n\n## Some helpful links\n\nInference notebook: https://www.kaggle.com/greatgamedota/osic-inference2?scriptVersionId=44254523",
      "votes": null
    },
    {
      "id": "1040224",
      "postDate": "10/07/2020 03:27:19",
      "content": "<p>Thanks for sharing ! nice work.</p>",
      "rawMarkdown": "Thanks for sharing ! nice work.",
      "votes": null
    },
    {
      "id": "1040235",
      "postDate": "10/07/2020 03:35:06",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/greatgamedota\" target=\"_blank\">@greatgamedota</a> on the silver! Waiting for the detailed writeup.</p>",
      "rawMarkdown": "Congratulations @greatgamedota on the silver! Waiting for the detailed writeup.",
      "votes": null
    },
    {
      "id": "1040248",
      "postDate": "10/07/2020 03:52:30",
      "content": "<p>Congratulations  to you and thx your sharing.</p>",
      "rawMarkdown": "Congratulations  to you and thx your sharing.",
      "votes": null
    },
    {
      "id": "1040262",
      "postDate": "10/07/2020 04:05:16",
      "content": "<p><a href=\"https://www.kaggle.com/greatgamedota\" target=\"_blank\">@greatgamedota</a> I wasn't expecting image models to do well on this one - congrats!</p>",
      "rawMarkdown": "greatgamedota I wasn't expecting image models to do well on this one - congrats!",
      "votes": null
    },
    {
      "id": "1040369",
      "postDate": "10/07/2020 05:19:38",
      "content": "<p>Thanks For sharing</p>",
      "rawMarkdown": "Thanks For sharing",
      "votes": null
    },
    {
      "id": "1040499",
      "postDate": "10/07/2020 07:09:02",
      "content": "<p>Thanks for sharing and congratulations! I’m looking forward to learning from your code!</p>",
      "rawMarkdown": "Thanks for sharing and congratulations! I’m looking forward to learning from your code!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1040224,
      "author_name": "tanagoolyenjai",
      "author_url": "",
      "post_date": "10/07/2020 03:27:19",
      "content": "<p>Thanks for sharing ! nice work.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1040235,
      "author_name": "aadhavvignesh",
      "author_url": "",
      "post_date": "10/07/2020 03:35:06",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/greatgamedota\" target=\"_blank\">@greatgamedota</a> on the silver! Waiting for the detailed writeup.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1040248,
      "author_name": "wanping7",
      "author_url": "",
      "post_date": "10/07/2020 03:52:30",
      "content": "<p>Congratulations  to you and thx your sharing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1040262,
      "author_name": "nxrprime",
      "author_url": "",
      "post_date": "10/07/2020 04:05:16",
      "content": "<p><a href=\"https://www.kaggle.com/greatgamedota\" target=\"_blank\">@greatgamedota</a> I wasn't expecting image models to do well on this one - congrats!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1040369,
      "author_name": "amrut11",
      "author_url": "",
      "post_date": "10/07/2020 05:19:38",
      "content": "<p>Thanks For sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1040499,
      "author_name": "douglaskgaraujo",
      "author_url": "",
      "post_date": "10/07/2020 07:09:02",
      "content": "<p>Thanks for sharing and congratulations! I’m looking forward to learning from your code!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1040217": "I did not expect to win this comp so I was not immediately prepared with my solution or the code. I have made my code available here: https://github.com/GreatGameDota/OSIC-Pulmonary-Fibrosis-Prediction\n\nAs always I want to thank Kaggle for another amazing competition as well as the hosts for putting this together!\n\nThis was an interesting competition for me because I saw that everyone else were only using the tabular data. I have no experience at all with tabular data comps and I like CV so I was stubborn and forced myself to use the CT scans. I was discouraged by my poor LB score but I keep pushing and it turned out to be the correct decision!\n\nAnyway on to the solution:\n\n## Models\n\n- Resnet50\n\n## Dataset\n\nImage resolution: 512x512\nFor each scan I loaded in all the dicom files then converted them to HU. I then cropped all the images and reshaped them to the size 50x512x512. After doing that I then windowed each image into three parts. For a detailed explanation of the windowing take a look at [this](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930) post (I used the same function and window values as shared there, thanks so much Ian Pan!). After I got the three windowed images I simply saved them as pngs.\nEach image looked similar to this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F3b1f2461da6d31658806e1ace841dd5f%2Flung.png?generation=1602033034958099&alt=media)\n\nI made the dataset I used public here: https://www.kaggle.com/greatgamedota/osic-windowed-lung-images\n\nFor meta data I used the same meta data from @ulrich07 's [baseline kernal](https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter) except I only used the base Percent value as it increased my CV to LB to PB correlation.\n\n## Augmentation\n\n- Coarse dropout\n- SSR\n- Horizontal + Vertical flip\n- For one model: Random Saturation and Brightness\n\nNo tabular/meta augmentation\n\n## Training\n\n- Adam optimizer with Reduce on Plateau scheduler\n- Trained with an LR of .003 for 30 epochs\n- Batch size of 16 (bs of 4 for one model)\n- Trained using Quantile Regression with .8 qloss + .2 metric loss (same as Ulrich's kernal)\n- Trained 5 fold for every model\n- Didn't use any batch accumulation or mixed precision\n- Saved checkpoint based on best validation score (all weeks)\n\nFor training I removed 6 entire patients because their CT scans were broken. I then split each patient into a fold by randomly shuffling them then using GroupKFold.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F69a1967fbe3f4067d7fd9f6e5e91350c%2Ffolds.png?generation=1602036568284315&alt=media)\n\nThen while training I pick a random unique patient and randomly select an image from 10-40 (since the first and last 10 images don't contain any lung info). I then made it so that each iteration lasted 4 * amount of patients. For validation I only picked the 15th image since it was the middle image that usually contains the most information.\n\n## Ensembling/Blending\n\nSimple mean average of their FVC predictions and confidence\n\n## Final Submission\n\nMy final submission was a blend of 3 Resnet50 models:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3543139%2F773b5cc6fdf39b2fe77ec08d4b295684%2Ffinal%20sub.png?generation=1602040599569002&alt=media)\nAnd another point is that the single model that would have scored gold had the same training parameters as the other models except the added brightness/saturation augmentation\n\n## What didn't work\n\n- 3d resnets (I tried for at least a month with these)\n- Linear Decay Regression\n- Any other type of model besides Resnets and Efficientnets (determinism issues)\n- Efficientnets\n- Simple meta data head (just concat)\n- Random erase augmentation\n\n## Final Thoughts\n\nI want to say again that this was an awesome comp that I am so glad I participated in!! Very glad to get my third medal and second silver medal! :D\n\n## Some helpful links\n\nInference notebook: https://www.kaggle.com/greatgamedota/osic-inference2?scriptVersionId=44254523",
    "1040224": "Thanks for sharing ! nice work.",
    "1040235": "Congratulations @greatgamedota on the silver! Waiting for the detailed writeup.",
    "1040248": "Congratulations  to you and thx your sharing.",
    "1040262": "greatgamedota I wasn't expecting image models to do well on this one - congrats!",
    "1040369": "Thanks For sharing",
    "1040499": "Thanks for sharing and congratulations! I’m looking forward to learning from your code!"
  },
  "source": "meta"
}