This was my first Kaggle silver medal, super happy about that 😄 This was a great learning experience for me. Kudos to all the winners and thanks to the entire Kaggle ecosystem.
\nTabular features and features obtained from CT scans were used in a Tabnet model, trained with pinball loss. As a form of data augmentation, every FVC score available for each person was assumed to be the first FVC test score, and other features were built accordingly.
\nTraining Notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361
\nInference Notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-inference/data?scriptVersionId=43574381
\nThis is fairly straightforward.
\nfirst_test_fvc - FVC score of the assumed first visit
\npredict_week - Weeks from the CT scan, Week feature from the train data
\nweeks_from_first_visit - number of weeks from the first FVC test.
\nexpected_fvc - fvc * percent this is a constant for each patient.
\npercent - Percent of the first_test_fvc when compared to the expected_fvc
\nage - Age of the patient
One-hot encoded values of Sex and SmokingStatus
3D rescaled segmented lung model was generated for each patient from which the image features were obtained.
\nSegmentation - The lung segmentation was performed using Marker-controlled watershed segmentation, thanks to the amazing work by Aadhav Vignesh
\nRescaling the voxel pixels - The rescaling was done using the resample method from Pulmonary Dicom Preprocessing. Thanks Laura Fink for the great kernel.
The only issue I faced with this approach was to stay within the 4 hours submission runtime while generating the 3D models for private dataset in the inference notebook. I was able to overcome this by switching scipy.ndimage.zoom in resample function with torch.nn.functional.interpolate, which was much faster.
Image features notebook - https://www.kaggle.com/gautham11/lung-volume-lung-height-hu-values-image-features
\ntrain Image features dataset - https://www.kaggle.com/gautham11/lung-image-features
\nImage features generated are
\nlung_volume_in_liters - Number of segmented lung voxel pixels in 3d model / 1e6.
\nlung_height - Difference between the first and last slice with more that 1000 lung voxel pixels.
\nlung_mean_hu - Mean of the HU values.
\nlung_skew - skew of the HU values.
\nlung_kurtosis - Kurtosis of the HU values.
And histogram bin values of the HU values in the 3D model were used as features - bin_-900, bin-800…bin_200.
The inspiration for most of these features was from the Domain expert's insights post by Dr. Konya
\nThese features gave me a good improvement in my CV score.
\nThe data was min-max scaled. The data was split into 5 folds, stratified on the expected_fvc, intuition being that split could be based on the Patients' characteristics, not sure if this was the best method. For each split, the Tabnet model was trained with SGD optimizer and One-cycle LR scheduler for 350 epochs.
Model weights dataset - https://www.kaggle.com/gautham11/gautham-quantmodel
\nI gave Pytorch-Lightning a try for the first time in this competition, and I'm fairly certain that I'll use it in every project from here on.
\nFinally, on selecting my submission, I selected my best CV score model and my best Public LB model. In retrospect, selecting the best Public LB model was a poor decision because it got a bad score in private LB, all the more affirmation to always trust the CV
", "messages": [ { "id": "1041491", "postDate": "10/07/2020 19:12:36", "content": "This was my first Kaggle silver medal, super happy about that 😄 This was a great learning experience for me. Kudos to all the winners and thanks to the entire Kaggle ecosystem.
\nTabular features and features obtained from CT scans were used in a Tabnet model, trained with pinball loss. As a form of data augmentation, every FVC score available for each person was assumed to be the first FVC test score, and other features were built accordingly.
\nTraining Notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361
\nInference Notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-inference/data?scriptVersionId=43574381
\nThis is fairly straightforward.
\nfirst_test_fvc - FVC score of the assumed first visit
\npredict_week - Weeks from the CT scan, Week feature from the train data
\nweeks_from_first_visit - number of weeks from the first FVC test.
\nexpected_fvc - fvc * percent this is a constant for each patient.
\npercent - Percent of the first_test_fvc when compared to the expected_fvc
\nage - Age of the patient
One-hot encoded values of Sex and SmokingStatus
3D rescaled segmented lung model was generated for each patient from which the image features were obtained.
\nSegmentation - The lung segmentation was performed using Marker-controlled watershed segmentation, thanks to the amazing work by Aadhav Vignesh
\nRescaling the voxel pixels - The rescaling was done using the resample method from Pulmonary Dicom Preprocessing. Thanks Laura Fink for the great kernel.
The only issue I faced with this approach was to stay within the 4 hours submission runtime while generating the 3D models for private dataset in the inference notebook. I was able to overcome this by switching scipy.ndimage.zoom in resample function with torch.nn.functional.interpolate, which was much faster.
Image features notebook - https://www.kaggle.com/gautham11/lung-volume-lung-height-hu-values-image-features
\ntrain Image features dataset - https://www.kaggle.com/gautham11/lung-image-features
\nImage features generated are
\nlung_volume_in_liters - Number of segmented lung voxel pixels in 3d model / 1e6.
\nlung_height - Difference between the first and last slice with more that 1000 lung voxel pixels.
\nlung_mean_hu - Mean of the HU values.
\nlung_skew - skew of the HU values.
\nlung_kurtosis - Kurtosis of the HU values.
And histogram bin values of the HU values in the 3D model were used as features - bin_-900, bin-800…bin_200.
The inspiration for most of these features was from the Domain expert's insights post by Dr. Konya
\nThese features gave me a good improvement in my CV score.
\nThe data was min-max scaled. The data was split into 5 folds, stratified on the expected_fvc, intuition being that split could be based on the Patients' characteristics, not sure if this was the best method. For each split, the Tabnet model was trained with SGD optimizer and One-cycle LR scheduler for 350 epochs.
Model weights dataset - https://www.kaggle.com/gautham11/gautham-quantmodel
\nI gave Pytorch-Lightning a try for the first time in this competition, and I'm fairly certain that I'll use it in every project from here on.
\nFinally, on selecting my submission, I selected my best CV score model and my best Public LB model. In retrospect, selecting the best Public LB model was a poor decision because it got a bad score in private LB, all the more affirmation to always trust the CV
", "rawMarkdown": "This was my first Kaggle silver medal, super happy about that 😄 This was a great learning experience for me. Kudos to all the winners and thanks to the entire Kaggle ecosystem.\n\n### **Solution Overview**\n\nTabular features and features obtained from CT scans were used in a [Tabnet model](https://github.com/dreamquark-ai/tabnet), trained with pinball loss. As a form of data augmentation, every FVC score available for each person was assumed to be the first FVC test score, and other features were built accordingly.\n\n**Training Notebook** - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361\n\n**Inference Notebook** - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-inference/data?scriptVersionId=43574381\n\n### **Tabular Features**\n\nThis is fairly straightforward.\n\n`first_test_fvc` - FVC score of the assumed first visit\n`predict_week` - Weeks from the CT scan, `Week` feature from the train data\n`weeks_from_first_visit` - number of weeks from the first FVC test.\n`expected_fvc` - `fvc * percent` this is a constant for each patient.\n`percent` - Percent of the `first_test_fvc` when compared to the `expected_fvc`\n`age` - Age of the patient\n\nOne-hot encoded values of `Sex` and `SmokingStatus`\n\n### **CT Scan Features**\n\n3D rescaled segmented lung model was generated for each patient from which the image features were obtained.\n\n**Segmentation** - The lung segmentation was performed using [Marker-controlled watershed segmentation](https://www.kaggle.com/aadhavvignesh/lung-segmentation-by-marker-controlled-watershed), thanks to the amazing work by [Aadhav Vignesh](https://www.kaggle.com/aadhavvignesh)\n\n**Rescaling the voxel pixels** - The rescaling was done using the `resample` method from [Pulmonary Dicom Preprocessing](https://www.kaggle.com/allunia/pulmonary-dicom-preprocessing). Thanks [Laura Fink](https://www.kaggle.com/allunia) for the great kernel.\n\nThe only issue I faced with this approach was to stay within the 4 hours submission runtime while generating the 3D models for private dataset in the inference notebook. I was able to overcome this by switching `scipy.ndimage.zoom` in `resample` function with `torch.nn.functional.interpolate`, which was much faster.\n\n**Image features notebook** - https://www.kaggle.com/gautham11/lung-volume-lung-height-hu-values-image-features\n\n**train Image features dataset** - https://www.kaggle.com/gautham11/lung-image-features\n\nImage features generated are\n\n`lung_volume_in_liters` - Number of segmented lung voxel pixels in 3d model / 1e6.\n`lung_height` - Difference between the first and last slice with more that 1000 lung voxel pixels.\n`lung_mean_hu` - Mean of the HU values.\n`lung_skew` - skew of the HU values.\n`lung_kurtosis` - Kurtosis of the HU values.\n\nAnd histogram bin values of the HU values in the 3D model were used as features - `bin_-900`, `bin-800`...`bin_200`.\n\nThe inspiration for most of these features was from the [Domain expert's insights](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165727) post by [Dr. Konya](https://www.kaggle.com/sandorkonya)\n\nThese features gave me a good improvement in my CV score.\n\n### **Model Training**\n\nThe data was min-max scaled. The data was split into 5 folds, stratified on the `expected_fvc`, intuition being that split could be based on the Patients' characteristics, not sure if this was the best method. For each split, the Tabnet model was trained with `SGD` optimizer and One-cycle LR scheduler for 350 epochs.\n\n**Model weights dataset** - https://www.kaggle.com/gautham11/gautham-quantmodel\n\nI gave Pytorch-Lightning a try for the first time in this competition, and I'm fairly certain that I'll use it in every project from here on.\n\nFinally, on selecting my submission, I selected my best CV score model and my best Public LB model. In retrospect, selecting the best Public LB model was a poor decision because it got a bad score in private LB, all the more affirmation to always **trust the CV**", "votes": null }, { "id": "1042040", "postDate": "10/08/2020 03:10:04", "content": "@gautham11 Congratulations on the silver! +1 for PyTorch-Lightning :P
", "rawMarkdown": "gautham11 Congratulations on the silver! +1 for PyTorch-Lightning :P", "votes": null }, { "id": "1042194", "postDate": "10/08/2020 06:01:54", "content": "Thanks, @aadhavvignesh! Good to see a fellow pytorch-lightning fan :P
", "rawMarkdown": "Thanks, @aadhavvignesh! Good to see a fellow pytorch-lightning fan :P", "votes": null }, { "id": "1042333", "postDate": "10/08/2020 07:18:22", "content": "Congratulations !
\nMy single tabnet model with PyTorch got top 8% only with tabular features. I do believe that tabnet is a robust solution for this challenge. Does pytorch lighthing provide a hyperparameter tuning for tabnet ?
Thanks! My tabnet model with only tabular features also scored in the same range.
\nGreat question! It seems pytorch-lightning supports hyperparameter tuning by integrating with other libraries. I didn't try it in this competition, but we should be able to tune tabnet parameters this way.
@optimo Told you Tabnet will do wonders
\n@gautham11 Thanks for the post , also congratulations on the medal
@gautham11 congratulations! will you make your solution public? I'd like to see how easy it was for you to use the library (I don't want to spoil, but some nice improvements are coming to make pytorch-tabnet easier to customize and so even more competitive! :))
\n@tanulsingh077 thanks for the mention, more wonders to come hopefully!
", "rawMarkdown": "gautham11 congratulations! will you make your solution public? I'd like to see how easy it was for you to use the library (I don't want to spoil, but some nice improvements are coming to make pytorch-tabnet easier to customize and so even more competitive! :))\n\n@tanulsingh077 thanks for the mention, more wonders to come hopefully!", "votes": null }, { "id": "1042974", "postDate": "10/08/2020 15:30:45", "content": "@optimo Very thankful for the work you've done, can't wait for the updates.
\nThe training code is available in this notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361
\nI didn't need the sklearn wrapper, so used TabNet directly.
Thanks @tanulsingh077 :)
", "rawMarkdown": "Thanks @tanulsingh077 :)", "votes": null }, { "id": "1043550", "postDate": "10/09/2020 04:14:14", "content": "Congratulations on the medal. Thank you sharing your experiance
", "rawMarkdown": "Congratulations on the medal. Thank you sharing your experiance", "votes": null } ], "comments": [ { "id": 1042040, "author_name": "aadhavvignesh", "author_url": "", "post_date": "10/08/2020 03:10:04", "content": "@gautham11 Congratulations on the silver! +1 for PyTorch-Lightning :P
", "votes": null, "replies": [ { "id": 1042194, "author_name": "gautham11", "author_url": "", "post_date": "10/08/2020 06:01:54", "content": "Thanks, @aadhavvignesh! Good to see a fellow pytorch-lightning fan :P
", "votes": null, "replies": [] } ] }, { "id": 1042333, "author_name": "alexj21", "author_url": "", "post_date": "10/08/2020 07:18:22", "content": "Congratulations !
\nMy single tabnet model with PyTorch got top 8% only with tabular features. I do believe that tabnet is a robust solution for this challenge. Does pytorch lighthing provide a hyperparameter tuning for tabnet ?
Thanks! My tabnet model with only tabular features also scored in the same range.
\nGreat question! It seems pytorch-lightning supports hyperparameter tuning by integrating with other libraries. I didn't try it in this competition, but we should be able to tune tabnet parameters this way.
@optimo Told you Tabnet will do wonders
\n@gautham11 Thanks for the post , also congratulations on the medal
@gautham11 congratulations! will you make your solution public? I'd like to see how easy it was for you to use the library (I don't want to spoil, but some nice improvements are coming to make pytorch-tabnet easier to customize and so even more competitive! :))
\n@tanulsingh077 thanks for the mention, more wonders to come hopefully!
", "votes": null, "replies": [] }, { "id": 1042974, "author_name": "gautham11", "author_url": "", "post_date": "10/08/2020 15:30:45", "content": "@optimo Very thankful for the work you've done, can't wait for the updates.
\nThe training code is available in this notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361
\nI didn't need the sklearn wrapper, so used TabNet directly.
Thanks @tanulsingh077 :)
", "votes": null, "replies": [] } ] }, { "id": 1043550, "author_name": "", "author_url": "", "post_date": "10/09/2020 04:14:14", "content": "Congratulations on the medal. Thank you sharing your experiance
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1041491": "This was my first Kaggle silver medal, super happy about that 😄 This was a great learning experience for me. Kudos to all the winners and thanks to the entire Kaggle ecosystem.\n\n### **Solution Overview**\n\nTabular features and features obtained from CT scans were used in a [Tabnet model](https://github.com/dreamquark-ai/tabnet), trained with pinball loss. As a form of data augmentation, every FVC score available for each person was assumed to be the first FVC test score, and other features were built accordingly.\n\n**Training Notebook** - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361\n\n**Inference Notebook** - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-inference/data?scriptVersionId=43574381\n\n### **Tabular Features**\n\nThis is fairly straightforward.\n\n`first_test_fvc` - FVC score of the assumed first visit\n`predict_week` - Weeks from the CT scan, `Week` feature from the train data\n`weeks_from_first_visit` - number of weeks from the first FVC test.\n`expected_fvc` - `fvc * percent` this is a constant for each patient.\n`percent` - Percent of the `first_test_fvc` when compared to the `expected_fvc`\n`age` - Age of the patient\n\nOne-hot encoded values of `Sex` and `SmokingStatus`\n\n### **CT Scan Features**\n\n3D rescaled segmented lung model was generated for each patient from which the image features were obtained.\n\n**Segmentation** - The lung segmentation was performed using [Marker-controlled watershed segmentation](https://www.kaggle.com/aadhavvignesh/lung-segmentation-by-marker-controlled-watershed), thanks to the amazing work by [Aadhav Vignesh](https://www.kaggle.com/aadhavvignesh)\n\n**Rescaling the voxel pixels** - The rescaling was done using the `resample` method from [Pulmonary Dicom Preprocessing](https://www.kaggle.com/allunia/pulmonary-dicom-preprocessing). Thanks [Laura Fink](https://www.kaggle.com/allunia) for the great kernel.\n\nThe only issue I faced with this approach was to stay within the 4 hours submission runtime while generating the 3D models for private dataset in the inference notebook. I was able to overcome this by switching `scipy.ndimage.zoom` in `resample` function with `torch.nn.functional.interpolate`, which was much faster.\n\n**Image features notebook** - https://www.kaggle.com/gautham11/lung-volume-lung-height-hu-values-image-features\n\n**train Image features dataset** - https://www.kaggle.com/gautham11/lung-image-features\n\nImage features generated are\n\n`lung_volume_in_liters` - Number of segmented lung voxel pixels in 3d model / 1e6.\n`lung_height` - Difference between the first and last slice with more that 1000 lung voxel pixels.\n`lung_mean_hu` - Mean of the HU values.\n`lung_skew` - skew of the HU values.\n`lung_kurtosis` - Kurtosis of the HU values.\n\nAnd histogram bin values of the HU values in the 3D model were used as features - `bin_-900`, `bin-800`...`bin_200`.\n\nThe inspiration for most of these features was from the [Domain expert's insights](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165727) post by [Dr. Konya](https://www.kaggle.com/sandorkonya)\n\nThese features gave me a good improvement in my CV score.\n\n### **Model Training**\n\nThe data was min-max scaled. The data was split into 5 folds, stratified on the `expected_fvc`, intuition being that split could be based on the Patients' characteristics, not sure if this was the best method. For each split, the Tabnet model was trained with `SGD` optimizer and One-cycle LR scheduler for 350 epochs.\n\n**Model weights dataset** - https://www.kaggle.com/gautham11/gautham-quantmodel\n\nI gave Pytorch-Lightning a try for the first time in this competition, and I'm fairly certain that I'll use it in every project from here on.\n\nFinally, on selecting my submission, I selected my best CV score model and my best Public LB model. In retrospect, selecting the best Public LB model was a poor decision because it got a bad score in private LB, all the more affirmation to always **trust the CV**", "1042040": "gautham11 Congratulations on the silver! +1 for PyTorch-Lightning :P", "1042194": "Thanks, @aadhavvignesh! Good to see a fellow pytorch-lightning fan :P", "1042333": "Congratulations ! \nMy single tabnet model with PyTorch got top 8% only with tabular features. I do believe that tabnet is a robust solution for this challenge. Does pytorch lighthing provide a hyperparameter tuning for tabnet ?", "1042461": "Thanks! My tabnet model with only tabular features also scored in the same range.\nGreat question! It seems pytorch-lightning supports [hyperparameter tuning by integrating with other libraries](https://pytorch-lightning.readthedocs.io/en/latest/hyperparameters.html#hyperparameter-optimization). I didn't try it in this competition, but we should be able to tune tabnet parameters this way.", "1042697": "optimo Told you Tabnet will do wonders\n@gautham11 Thanks for the post , also congratulations on the medal", "1042704": "gautham11 congratulations! will you make your solution public? I'd like to see how easy it was for you to use the library (I don't want to spoil, but some nice improvements are coming to make pytorch-tabnet easier to customize and so even more competitive! :))\n\n@tanulsingh077 thanks for the mention, more wonders to come hopefully!", "1042974": "optimo Very thankful for the work you've done, can't wait for the updates. \nThe training code is available in this notebook - https://www.kaggle.com/gautham11/quantile-regression-pytorch-lightning-training?scriptVersionId=44239361\nI didn't need the sklearn wrapper, so used `TabNet` directly.", "1042975": "Thanks @tanulsingh077 :)", "1043550": "Congratulations on the medal. Thank you sharing your experiance" }, "source": "meta" }