\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F987995%2Fd9b66b893e1847fa5daabe053e879b42%2Fmontage.jpeg?generation=1602104759897863&alt=media\")
Dear fellow kagglers,
\nTL; DR;
\nI would like to share segmentation datasets with you:\n1. fibrotic lung segmentation masks of 110 CT's\n2. whole heart segmentation of 87 of the above scans\n3. trachea segmentations (110)\nAs far as i know this is the largest existing public dataset of manual lung segmentations for fibrotic diseased lungs , i hope that many of you can use for further work and experiments.
\n
…and now the long version:
\nThis competition was one of the least rewarding experiences for me.
\nNot just because of the enormous work we put in it with my team
\n– and here I would like to thank to @authman @gunesevitan @sainatarajan7 for the 250+ hours we each spent on this competition and for all the things I learned from you along the way -
\nbut also due to the fact that some of my insights have involuntary mislead you fellow kagglers, making you believe, that with much work on the image data stronger features could be sourced.
\nI was convinced that despite the heterogeneity of the scans provided, there are features hidden in the images that help to predict better the course of the disease… that is the goal of the whole RADIOMICS – to get biological information that is not visible for the human eye!
\nAnd I still believe in it… then, I disregarded the first and most important rule: garbage in, garbage out. I could moan about the data quality, but we agreed to work on it from the beginning, so our own fault.
\nThe team made an amazing pipeline for the preparation of the CT data with cropping & rescaling, resampling and handling exceptions (which was excruciating in some cases), really grandiose, but it took like 2/3 of our time.
\nWe found some image derived features that seem to make better predictions, but these were not strong enough to \"steer the slope in the good direction\" when the clinical data over fitted.
\nI performed manual measurements on all the public scans, we tested the features and then we automatized some of them - if you would like we can post about these features to.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F987995%2Ff49ad61805407b6036ac1c056026dc5c%2FMy%20Drawing.sketchpad%20(1).jpeg?generation=1602105473039285&alt=media\")
In the meanwhile I managed to segment dozens of scans, and I would like to give something back to the community by uploading these, I hope it will be helpful for someone…
\nCold comfort, but in the end I think I made the world largest public segmentation database for fibrotic diseased lungs & heart – as far as I know.
\nLive long and prosper - oh and stay healthy :)
", "messages": [ { "id": "1041643", "postDate": "10/07/2020 21:19:47", "content": "Dear fellow kagglers,
\nTL; DR;
\nI would like to share segmentation datasets with you:\n1. fibrotic lung segmentation masks of 110 CT's\n2. whole heart segmentation of 87 of the above scans\n3. trachea segmentations (110)\nAs far as i know this is the largest existing public dataset of manual lung segmentations for fibrotic diseased lungs , i hope that many of you can use for further work and experiments.
\n
…and now the long version:
\nThis competition was one of the least rewarding experiences for me.
\nNot just because of the enormous work we put in it with my team
\n– and here I would like to thank to @authman @gunesevitan @sainatarajan7 for the 250+ hours we each spent on this competition and for all the things I learned from you along the way -
\nbut also due to the fact that some of my insights have involuntary mislead you fellow kagglers, making you believe, that with much work on the image data stronger features could be sourced.
\nI was convinced that despite the heterogeneity of the scans provided, there are features hidden in the images that help to predict better the course of the disease… that is the goal of the whole RADIOMICS – to get biological information that is not visible for the human eye!
\nAnd I still believe in it… then, I disregarded the first and most important rule: garbage in, garbage out. I could moan about the data quality, but we agreed to work on it from the beginning, so our own fault.
\nThe team made an amazing pipeline for the preparation of the CT data with cropping & rescaling, resampling and handling exceptions (which was excruciating in some cases), really grandiose, but it took like 2/3 of our time.
\nWe found some image derived features that seem to make better predictions, but these were not strong enough to \"steer the slope in the good direction\" when the clinical data over fitted.
\nI performed manual measurements on all the public scans, we tested the features and then we automatized some of them - if you would like we can post about these features to.
\n
In the meanwhile I managed to segment dozens of scans, and I would like to give something back to the community by uploading these, I hope it will be helpful for someone…
\nCold comfort, but in the end I think I made the world largest public segmentation database for fibrotic diseased lungs & heart – as far as I know.
\nLive long and prosper - oh and stay healthy :)
", "rawMarkdown": "Dear fellow kagglers, \n\n**TL; DR;**\n\n```\nI would like to share segmentation datasets with you:\n1. fibrotic lung segmentation masks of 110 CT's\n2. whole heart segmentation of 87 of the above scans\n3. trachea segmentations (110)\n```\nAs far as i know this is the [largest existing public dataset of manual lung segmentations for fibrotic diseased lungs](https://www.kaggle.com/sandorkonya/ct-lung-heart-trachea-segmentation) , i hope that many of you can use for further work and experiments.\n\n\n...and now the long version:\n \nThis competition was **one of the least rewarding** experiences for me.\n\nNot just because of the enormous work we put in it with my team\n\n– and here I would like to thank to @authman @gunesevitan @sainatarajan7 for the 250+ hours we each spent on this competition and for all the things I learned from you along the way -\n\nbut also due to the fact that **some of my insights** have **involuntary mislead you** fellow kagglers, making you believe, that with much work on the image data stronger features could be sourced.\n \n**I was convinced** that despite the heterogeneity of the scans provided, there are features hidden in the images **that help to predict** better the course of the disease... that is the goal of the whole RADIOMICS – to get biological information that is not visible for the human eye! \n\n**And I still believe in it…** then, I disregarded the first and most important rule: garbage in, garbage out. I could moan about the data quality, but we agreed to work on it from the beginning, so our own fault.\n \n**The team** made an amazing **pipeline for the preparation** of the CT data with cropping & rescaling, resampling and handling exceptions (which was excruciating in some cases), really grandiose, but it **took like 2/3 of our time**. \n \n**We found some image derived features that seem to make better predictions**, but these were not strong enough to \"steer the slope in the good direction\" when the clinical data over fitted.\n\nI performed **manual measurements on all the public scans**, we tested the features and then we automatized some of them - if you would like we can post about these features to.\n \n.jpeg?generation=1602105473039285&alt=media)\n\nIn the meanwhile I managed to segment dozens of scans, and I **would like to give something back to the community** by uploading these, I hope it will be helpful for someone... \n\nCold comfort, but in the end I think I made the world largest public segmentation database for fibrotic diseased lungs & heart – as far as I know.\n\nLive long and prosper - oh and stay healthy :)", "votes": null }, { "id": "1041648", "postDate": "10/07/2020 21:24:57", "content": "🖖🏾 will rise again
", "rawMarkdown": "🖖🏾 will rise again", "votes": null }, { "id": "1041672", "postDate": "10/07/2020 21:42:37", "content": "incredible efforts -- thanks for sharing!
", "rawMarkdown": "incredible efforts -- thanks for sharing!", "votes": null }, { "id": "1041723", "postDate": "10/07/2020 22:12:14", "content": "Thank you for your contribution!
", "rawMarkdown": "Thank you for your contribution!", "votes": null }, { "id": "1041863", "postDate": "10/08/2020 00:18:17", "content": "To borrow your words: This competition was one of the least rewarding experiences for me.
\nDespite this. I've learned and experimented a lot with your insights. I've left with the knowledge.
", "rawMarkdown": "To borrow your words: This competition was **one of the least rewarding experiences for me.**\n\nDespite this. I've learned and experimented a lot with your insights. I've left with the knowledge.", "votes": null }, { "id": "1042569", "postDate": "10/08/2020 10:20:52", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": null }, { "id": "1043195", "postDate": "10/08/2020 18:53:43", "content": "So essentially your conclusion is that there was nothing to learn about the progression of the disease from the CT scans?
", "rawMarkdown": "So essentially your conclusion is that there was nothing to learn about the progression of the disease from the CT scans?", "votes": null }, { "id": "1043217", "postDate": "10/08/2020 19:18:11", "content": "I can't speak for him, but I would say there is plenty you can infer about the state, and possible progression of the disease from the scans. But we had a small number of patients and a complex evaluation metric that heavily penalised non-conservative predictions, so the image features couldn't be used well for this competition.
", "rawMarkdown": "I can't speak for him, but I would say there is plenty you can infer about the state, and possible progression of the disease from the scans. But we had a small number of patients and a complex evaluation metric that heavily penalised non-conservative predictions, so the image features couldn't be used well for this competition.", "votes": null }, { "id": "1043247", "postDate": "10/08/2020 19:51:12", "content": "@bjaeger ,
\nmy conclusion is, that we derived many promising features but none of them was so substantial that could adjust the progression - even by the best performing linear models.
\nI am pretty sure that it is due to the heterogenity of the scans… on a more standardised and larger collection we would have more chance to find out.
Große Bauchlandung :P
", "rawMarkdown": "bjaeger ,\nmy conclusion is, that we derived many promising features but none of them was so substantial that could adjust the progression - even by the best performing linear models.\nI am pretty sure that it is due to the heterogenity of the scans... on a more standardised and larger collection we would have more chance to find out.\n\nGroße Bauchlandung :P", "votes": null }, { "id": "1043255", "postDate": "10/08/2020 19:57:05", "content": "@bmcinnovo
\nthank you very much, i appreciate your comment… even more in the ligh of how often you comment. ;)
@bigironsphere ,
\nthank you!
@ronaldokun ,
\nkudos! =)
@bjaeger I wouldn't say that. I'm pretty sure models could learn it with proper amount of data and better objective definition.
", "rawMarkdown": "bjaeger I wouldn't say that. I'm pretty sure models could learn it with proper amount of data and better objective definition.", "votes": null }, { "id": "1043752", "postDate": "10/09/2020 08:07:59", "content": "Wow…I'm impressed by all the effort you've put in this competition…your team really contributed a lot in the discussions here. I've learn a lot from you - especially about the task itself - and I have no regret putting so much effort trying to get something from those ct-scans.
\nIt is unfortunate that your commitment was not rewarded as deserved. I'm quite sure that many experimented kagglers decided not to compete after some EDA, understanding that OSIC was too submitted to randomness to be a good time investment for someone how target the top of the leaderboard. Too bad for the patients…
\n\nI performed manual measurements on all the public scans, we tested the features and then we automatized some of them - if you would like we can post about these features to.
\n
I am definitively one of those who are interested in further reading about it. Learning how ML could leverage on domain expertise to get better results is one of the reason why I was so excited about this competition. I guess we can learn more from \"what didn't work\" than a perfectly working solution, which could lead to … overfitting :)
", "rawMarkdown": "Wow...I'm impressed by all the effort you've put in this competition...your team really contributed a lot in the discussions here. I've learn a lot from you - especially about the task itself - and I have no regret putting so much effort trying to get something from those ct-scans.\nIt is unfortunate that your commitment was not rewarded as deserved. I'm quite sure that many experimented kagglers decided not to compete after some EDA, understanding that OSIC was too submitted to randomness to be a good time investment for someone how target the top of the leaderboard. Too bad for the patients...\n\n> I performed manual measurements on all the public scans, we tested the features and then we automatized some of them - if you would like we can post about these features to.\n\nI am definitively one of those who are interested in further reading about it. Learning how ML could leverage on domain expertise to get better results is one of the reason why I was so excited about this competition. I guess we can learn more from \"what didn't work\" than a perfectly working solution, which could lead to ... overfitting :)", "votes": null }, { "id": "1043955", "postDate": "10/09/2020 11:10:27", "content": "Good Work.
", "rawMarkdown": "Good Work.", "votes": null }, { "id": "1049252", "postDate": "10/14/2020 08:59:03", "content": "Thank you, that helped!
", "rawMarkdown": "Thank you, that helped!", "votes": null }, { "id": "1123988", "postDate": "12/23/2020 16:16:16", "content": "This is a great contribution! I would like to know if there is a segmentation of the fibrosis itself?
", "rawMarkdown": "This is a great contribution! I would like to know if there is a segmentation of the fibrosis itself?", "votes": null } ], "comments": [ { "id": 1041648, "author_name": "authman", "author_url": "", "post_date": "10/07/2020 21:24:57", "content": "🖖🏾 will rise again
", "votes": null, "replies": [] }, { "id": 1041672, "author_name": "bmcinnovo", "author_url": "", "post_date": "10/07/2020 21:42:37", "content": "incredible efforts -- thanks for sharing!
", "votes": null, "replies": [ { "id": 1043255, "author_name": "sandorkonya", "author_url": "", "post_date": "10/08/2020 19:57:05", "content": "@bmcinnovo
\nthank you very much, i appreciate your comment… even more in the ligh of how often you comment. ;)
Thank you for your contribution!
", "votes": null, "replies": [ { "id": 1043256, "author_name": "sandorkonya", "author_url": "", "post_date": "10/08/2020 19:57:39", "content": "@bigironsphere ,
\nthank you!
To borrow your words: This competition was one of the least rewarding experiences for me.
\nDespite this. I've learned and experimented a lot with your insights. I've left with the knowledge.
", "votes": null, "replies": [ { "id": 1043258, "author_name": "sandorkonya", "author_url": "", "post_date": "10/08/2020 19:58:35", "content": "@ronaldokun ,
\nkudos! =)
So essentially your conclusion is that there was nothing to learn about the progression of the disease from the CT scans?
", "votes": null, "replies": [ { "id": 1043217, "author_name": "bigironsphere", "author_url": "", "post_date": "10/08/2020 19:18:11", "content": "I can't speak for him, but I would say there is plenty you can infer about the state, and possible progression of the disease from the scans. But we had a small number of patients and a complex evaluation metric that heavily penalised non-conservative predictions, so the image features couldn't be used well for this competition.
", "votes": null, "replies": [] }, { "id": 1043247, "author_name": "sandorkonya", "author_url": "", "post_date": "10/08/2020 19:51:12", "content": "@bjaeger ,
\nmy conclusion is, that we derived many promising features but none of them was so substantial that could adjust the progression - even by the best performing linear models.
\nI am pretty sure that it is due to the heterogenity of the scans… on a more standardised and larger collection we would have more chance to find out.
Große Bauchlandung :P
", "votes": null, "replies": [] }, { "id": 1043693, "author_name": "gunesevitan", "author_url": "", "post_date": "10/09/2020 06:57:03", "content": "@bjaeger I wouldn't say that. I'm pretty sure models could learn it with proper amount of data and better objective definition.
", "votes": null, "replies": [] } ] }, { "id": 1043752, "author_name": "johannhuber", "author_url": "", "post_date": "10/09/2020 08:07:59", "content": "Wow…I'm impressed by all the effort you've put in this competition…your team really contributed a lot in the discussions here. I've learn a lot from you - especially about the task itself - and I have no regret putting so much effort trying to get something from those ct-scans.
\nIt is unfortunate that your commitment was not rewarded as deserved. I'm quite sure that many experimented kagglers decided not to compete after some EDA, understanding that OSIC was too submitted to randomness to be a good time investment for someone how target the top of the leaderboard. Too bad for the patients…
\n\nI performed manual measurements on all the public scans, we tested the features and then we automatized some of them - if you would like we can post about these features to.
\n
I am definitively one of those who are interested in further reading about it. Learning how ML could leverage on domain expertise to get better results is one of the reason why I was so excited about this competition. I guess we can learn more from \"what didn't work\" than a perfectly working solution, which could lead to … overfitting :)
", "votes": null, "replies": [] }, { "id": 1049252, "author_name": "andrey885", "author_url": "", "post_date": "10/14/2020 08:59:03", "content": "Thank you, that helped!
", "votes": null, "replies": [] }, { "id": 1123988, "author_name": "vexstar", "author_url": "", "post_date": "12/23/2020 16:16:16", "content": "This is a great contribution! I would like to know if there is a segmentation of the fibrosis itself?
", "votes": null, "replies": [] }, { "id": 1042569, "author_name": "sachin70992", "author_url": "", "post_date": "10/08/2020 10:20:52", "content": "Thanks
", "votes": null, "replies": [] }, { "id": 1043955, "author_name": "ashfaq892", "author_url": "", "post_date": "10/09/2020 11:10:27", "content": "Good Work.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1041643": "Dear fellow kagglers, \n\n**TL; DR;**\n\n```\nI would like to share segmentation datasets with you:\n1. fibrotic lung segmentation masks of 110 CT's\n2. whole heart segmentation of 87 of the above scans\n3. trachea segmentations (110)\n```\nAs far as i know this is the [largest existing public dataset of manual lung segmentations for fibrotic diseased lungs](https://www.kaggle.com/sandorkonya/ct-lung-heart-trachea-segmentation) , i hope that many of you can use for further work and experiments.\n\n\n...and now the long version:\n \nThis competition was **one of the least rewarding** experiences for me.\n\nNot just because of the enormous work we put in it with my team\n\n– and here I would like to thank to @authman @gunesevitan @sainatarajan7 for the 250+ hours we each spent on this competition and for all the things I learned from you along the way -\n\nbut also due to the fact that **some of my insights** have **involuntary mislead you** fellow kagglers, making you believe, that with much work on the image data stronger features could be sourced.\n \n**I was convinced** that despite the heterogeneity of the scans provided, there are features hidden in the images **that help to predict** better the course of the disease... that is the goal of the whole RADIOMICS – to get biological information that is not visible for the human eye! \n\n**And I still believe in it…** then, I disregarded the first and most important rule: garbage in, garbage out. I could moan about the data quality, but we agreed to work on it from the beginning, so our own fault.\n \n**The team** made an amazing **pipeline for the preparation** of the CT data with cropping & rescaling, resampling and handling exceptions (which was excruciating in some cases), really grandiose, but it **took like 2/3 of our time**. \n \n**We found some image derived features that seem to make better predictions**, but these were not strong enough to \"steer the slope in the good direction\" when the clinical data over fitted.\n\nI performed **manual measurements on all the public scans**, we tested the features and then we automatized some of them - if you would like we can post about these features to.\n \n.jpeg?generation=1602105473039285&alt=media)\n\nIn the meanwhile I managed to segment dozens of scans, and I **would like to give something back to the community** by uploading these, I hope it will be helpful for someone... \n\nCold comfort, but in the end I think I made the world largest public segmentation database for fibrotic diseased lungs & heart – as far as I know.\n\nLive long and prosper - oh and stay healthy :)", "1041648": "🖖🏾 will rise again", "1041672": "incredible efforts -- thanks for sharing!", "1041723": "Thank you for your contribution!", "1041863": "To borrow your words: This competition was **one of the least rewarding experiences for me.**\n\nDespite this. I've learned and experimented a lot with your insights. I've left with the knowledge.", "1042569": "Thanks", "1043195": "So essentially your conclusion is that there was nothing to learn about the progression of the disease from the CT scans?", "1043217": "I can't speak for him, but I would say there is plenty you can infer about the state, and possible progression of the disease from the scans. But we had a small number of patients and a complex evaluation metric that heavily penalised non-conservative predictions, so the image features couldn't be used well for this competition.", "1043247": "bjaeger ,\nmy conclusion is, that we derived many promising features but none of them was so substantial that could adjust the progression - even by the best performing linear models.\nI am pretty sure that it is due to the heterogenity of the scans... on a more standardised and larger collection we would have more chance to find out.\n\nGroße Bauchlandung :P", "1043255": "bmcinnovo\nthank you very much, i appreciate your comment... even more in the ligh of how often you comment. ;)", "1043256": "bigironsphere ,\nthank you!", "1043258": "ronaldokun , \nkudos! =)", "1043693": "bjaeger I wouldn't say that. I'm pretty sure models could learn it with proper amount of data and better objective definition.", "1043752": "Wow...I'm impressed by all the effort you've put in this competition...your team really contributed a lot in the discussions here. I've learn a lot from you - especially about the task itself - and I have no regret putting so much effort trying to get something from those ct-scans.\nIt is unfortunate that your commitment was not rewarded as deserved. I'm quite sure that many experimented kagglers decided not to compete after some EDA, understanding that OSIC was too submitted to randomness to be a good time investment for someone how target the top of the leaderboard. Too bad for the patients...\n\n> I performed manual measurements on all the public scans, we tested the features and then we automatized some of them - if you would like we can post about these features to.\n\nI am definitively one of those who are interested in further reading about it. Learning how ML could leverage on domain expertise to get better results is one of the reason why I was so excited about this competition. I guess we can learn more from \"what didn't work\" than a perfectly working solution, which could lead to ... overfitting :)", "1043955": "Good Work.", "1049252": "Thank you, that helped!", "1123988": "This is a great contribution! I would like to know if there is a segmentation of the fibrosis itself?" }, "source": "meta" }