![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2568215%2F04f8262565ef36522a3024495e515d16%2FScreenshot%202020-09-08%20at%209.59.26%20PM.png?generation=1599582732378028&alt=media\")
See my notebook.
\nI'm a late starter to this competition, so my apologies if this is old news to anyone. We are able to achieve a -7.1 Public LB score with a completely naive approach. The top score, as of yet, is about a -6.7. The theoretically best score is about a -4.6. So what are everyone's thoughts on this?
\nI have seen a lot of public kernels using some really leaky features and information that is not available for the final prediction. It appears we can easily get a -6.9 or so with relatively naive models --FVC as a linear function of Weeks for each of the five Patients in the test set. But to do much better we are going to have to incorporate information about the lungs.
I know no one likes to give away their secret sauce --especially if they're on the trail of a potentially good idea-- but I'd like to discuss ideas and/or approaches. At the very least we can talk about what we have tried and hasn't worked… and save newbies and late-comers like myself some time. :P
\nI was thinking we should be able to calculate the lung volume directly from a pixel count of the lungs. We would have to adjust for zoom/resolution/angle/etc., but that should be doable. I was also thinking that we can denoise the target in the training set (LOWESS?), remove outliers, and do some post-processing on the predictions --since it is just a time-series. Thoughts?
\nThere are a myriad of other issues to address too. So what does everyone think?
\nEDIT Ok. I guess I should also ask the question: LB Probing?
", "messages": [ { "id": 998754, "postDate": "2020-09-05T03:26:56.973Z", "content": "See my notebook.
\nI'm a late starter to this competition, so my apologies if this is old news to anyone. We are able to achieve a -7.1 Public LB score with a completely naive approach. The top score, as of yet, is about a -6.7. The theoretically best score is about a -4.6. So what are everyone's thoughts on this?
\nI have seen a lot of public kernels using some really leaky features and information that is not available for the final prediction. It appears we can easily get a -6.9 or so with relatively naive models --FVC as a linear function of Weeks for each of the five Patients in the test set. But to do much better we are going to have to incorporate information about the lungs.
I know no one likes to give away their secret sauce --especially if they're on the trail of a potentially good idea-- but I'd like to discuss ideas and/or approaches. At the very least we can talk about what we have tried and hasn't worked… and save newbies and late-comers like myself some time. :P
\nI was thinking we should be able to calculate the lung volume directly from a pixel count of the lungs. We would have to adjust for zoom/resolution/angle/etc., but that should be doable. I was also thinking that we can denoise the target in the training set (LOWESS?), remove outliers, and do some post-processing on the predictions --since it is just a time-series. Thoughts?
\nThere are a myriad of other issues to address too. So what does everyone think?
\nEDIT Ok. I guess I should also ask the question: LB Probing?
", "rawMarkdown": "See my [notebook](https://www.kaggle.com/puremath86/naive-baseline-lb-7-1?scriptVersionId=42010889).\n\nI'm a late starter to this competition, so my apologies if this is old news to anyone. We are able to achieve a -7.1 Public LB score with a completely naive approach. The top score, as of yet, is about a -6.7. The theoretically best score is about a -4.6. So what are everyone's thoughts on this?\n\nI have seen a lot of public kernels using some really leaky features and information that is not available for the final prediction. It appears we can easily get a -6.9 or so with relatively naive models --`FVC` as a linear function of `Weeks` for each of the five `Patient`s in the test set. But to do much better we are going to have to incorporate information about the lungs.\n\nI know no one likes to give away their secret sauce --especially if they're on the trail of a potentially good idea-- but I'd like to discuss ideas and/or approaches. At the very least we can talk about what we have tried and hasn't worked... and save newbies and late-comers like myself some time. :P\n\nI was thinking we should be able to calculate the lung volume directly from a pixel count of the lungs. We would have to adjust for zoom/resolution/angle/etc., but that should be doable. I was also thinking that we can denoise the target in the training set (LOWESS?), remove outliers, and do some post-processing on the predictions --since it is just a time-series. Thoughts?\n\nThere are a myriad of other issues to address too. So what does everyone think?\n\n**EDIT** Ok. I guess I should also ask the question: LB Probing?", "votes": 23 }, { "id": 1002191, "postDate": "2020-09-07T22:48:41.797Z", "content": "At least in my experiments, lung volume information doesn't bring me any improvement yet.
", "rawMarkdown": "At least in my experiments, lung volume information doesn't bring me any improvement yet.", "votes": 5, "replies": [ { "id": 1002268, "postDate": "2020-09-08T02:12:59.753Z", "content": "Thanks for sharing. I'm thinking the CT Scans won't be very useful for this competition due to lack of data and the noise in the FVC measurements.
", "rawMarkdown": "Thanks for sharing. I'm thinking the CT Scans won't be very useful for this competition due to lack of data and the noise in the FVC measurements.", "votes": 1 } ] }, { "id": 1002373, "postDate": "2020-09-08T04:54:05.500Z", "content": "I think this is an amazing competition. Especially because it is hard.
\nIMHO there are 2 key important things to create great models for this competition:
Fitting curves through the ~1500 FVC points in the training set is easy. The hard thing to do is (1) and (2). In (1), I've tried manual feature engineering (+21 features) and unsupervised learning techniques (Variational Autoencoder, +16 features). This code is not public yet, but I intend to do it once the competition ends. In (2), there are 2 approaches: quantile models/pinball loss, and Bayesian approach/probabilistic machine learning. Also tried both, and made some of the code already public.
", "rawMarkdown": "I think this is an amazing competition. Especially because it is hard.\nIMHO there are 2 key important things to create great models for this competition:\n1. Extracting relevant features from the CT scans\n2. Properly predicting uncertainty\n\nFitting curves through the ~1500 FVC points in the training set is easy. The hard thing to do is (1) and (2). In (1), I've tried manual feature engineering (+21 features) and unsupervised learning techniques (Variational Autoencoder, +16 features). This code is not public yet, but I intend to do it once the competition ends. In (2), there are 2 approaches: quantile models/pinball loss, and Bayesian approach/probabilistic machine learning. Also tried both, and made some of the code already public. \n", "votes": 3, "replies": [ { "id": 1002395, "postDate": "2020-09-08T05:23:45.973Z", "content": "Thanks for sharing your thoughts. I agree the competition is interesting. I'm just afraid the winning solutions will not be useful to the competition host. If computer vision-based approaches don't add substantially more value than some rudimentary approaches --or just collecting more spirometry measurements directly from patients-- then they aren't worth it. I'd loved to be wrong though. And maybe the feature engineering and approaches themselves will be of value.
\nIn my preliminary experiments, I'm finding that probing the LB should be able to attain a score of ~ -5.7 --if you can probe enough data for each of the five patients before the competition is over. And I'm not convinced I'll be able to achieve that since I started so late and only 15% of the LB is public. Fortunately, it doesn't look like many competitors are taking this approach. But who knows?
", "rawMarkdown": "Thanks for sharing your thoughts. I agree the competition is interesting. I'm just afraid the winning solutions will not be useful to the competition host. If computer vision-based approaches don't add substantially more value than some rudimentary approaches --or just collecting more spirometry measurements directly from patients-- then they aren't worth it. I'd loved to be wrong though. And maybe the feature engineering and approaches themselves will be of value.\n\nIn my preliminary experiments, I'm finding that probing the LB should be able to attain a score of ~ -5.7 --if you can probe enough data for each of the five patients before the competition is over. And I'm not convinced I'll be able to achieve that since I started so late and only 15% of the LB is public. Fortunately, it doesn't look like many competitors are taking this approach. But who knows?" }, { "id": 1002405, "postDate": "2020-09-08T05:29:25.657Z", "content": "If I were the host, I wouldn't care about the solutions. I'd care about the people who wrote them ;)
", "rawMarkdown": "If I were the host, I wouldn't care about the solutions. I'd care about the people who wrote them ;)", "votes": 2 }, { "id": 1008509, "postDate": "2020-09-13T07:08:22.723Z", "content": "Hi Carlos,
\n21 manual features!? 🤪 are those all from the ct scans? I can think only of: Lung volume, airway volume, some measure of texture. I know you are keeping your cards close to your chest on this until later in the competition but please could you give some hints of what other features are promising?
", "rawMarkdown": "Hi Carlos, \n\n21 manual features!? 🤪 are those all from the ct scans? I can think only of: Lung volume, airway volume, some measure of texture. I know you are keeping your cards close to your chest on this until later in the competition but please could you give some hints of what other features are promising?", "votes": 1 }, { "id": 1009355, "postDate": "2020-09-13T22:31:07.470Z", "content": "The manual features are no secret… I think I mentioned in a post or notebook:
\nThere's also the features from the unsupervised approach…
\nHope it helps! Cheers!
Doesn't work for me: Include simple image statistics (from segmented lungs) like mean, std, skew, kurtosis = didn't improve CV
", "rawMarkdown": "Doesn't work for me: Include simple image statistics (from segmented lungs) like mean, std, skew, kurtosis = didn't improve CV", "votes": 3, "replies": [ { "id": 1001381, "postDate": "2020-09-07T09:50:43.430Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing. " } ] }, { "id": 1003123, "postDate": "2020-09-08T17:11:57.063Z", "content": "Hello Bryan, I too recently joined the competition and have been trying to extract features form the CT scans. I am going to share some of my initial findings. So far I have managed to obtain the following features from images:
\nBelow are a couple of normalized distributions of the variables mentioned above:
\n
Thanks, Abhishek. That's interesting. I'm definitely going to look at lung volume features in the next few days.
", "rawMarkdown": "Thanks, Abhishek. That's interesting. I'm definitely going to look at lung volume features in the next few days." } ] }, { "id": 1002627, "postDate": "2020-09-08T09:33:58.670Z", "content": "I've found that the most important factor in this competition is \"gaming the metric\". By this I mean that a model that predicts future FVCs can be easily useless if given uncertainties aren't accurate enough (according to metric).
", "rawMarkdown": "I've found that the most important factor in this competition is \"gaming the metric\". By this I mean that a model that predicts future FVCs can be easily useless if given uncertainties aren't accurate enough (according to metric)." }, { "id": 1000771, "postDate": "2020-09-06T19:24:27.860Z", "content": "No one wants to share their recipe….
", "rawMarkdown": "No one wants to share their recipe....", "replies": [ { "id": 1000920, "postDate": "2020-09-06T23:05:08.420Z", "content": "No worries. I don't blame them. The lack of data and various other factors make this a bit of a challenge.
", "rawMarkdown": "No worries. I don't blame them. The lack of data and various other factors make this a bit of a challenge." } ] }, { "id": 1003355, "postDate": "2020-09-08T21:32:38.503Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1002191, "author_name": "Tsai29", "author_url": "", "post_date": "2020-09-07T22:48:41.797000", "content": "At least in my experiments, lung volume information doesn't bring me any improvement yet.
", "votes": 5, "replies": [ { "id": 1002268, "author_name": "Bryan Arnold", "author_url": "", "post_date": "2020-09-08T02:12:59.753000", "content": "Thanks for sharing. I'm thinking the CT Scans won't be very useful for this competition due to lack of data and the noise in the FVC measurements.
", "votes": 1, "replies": [] } ] }, { "id": 1002373, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-09-08T04:54:05.500000", "content": "I think this is an amazing competition. Especially because it is hard.
\nIMHO there are 2 key important things to create great models for this competition:
Fitting curves through the ~1500 FVC points in the training set is easy. The hard thing to do is (1) and (2). In (1), I've tried manual feature engineering (+21 features) and unsupervised learning techniques (Variational Autoencoder, +16 features). This code is not public yet, but I intend to do it once the competition ends. In (2), there are 2 approaches: quantile models/pinball loss, and Bayesian approach/probabilistic machine learning. Also tried both, and made some of the code already public.
", "votes": 3, "replies": [ { "id": 1002395, "author_name": "Bryan Arnold", "author_url": "", "post_date": "2020-09-08T05:23:45.973000", "content": "Thanks for sharing your thoughts. I agree the competition is interesting. I'm just afraid the winning solutions will not be useful to the competition host. If computer vision-based approaches don't add substantially more value than some rudimentary approaches --or just collecting more spirometry measurements directly from patients-- then they aren't worth it. I'd loved to be wrong though. And maybe the feature engineering and approaches themselves will be of value.
\nIn my preliminary experiments, I'm finding that probing the LB should be able to attain a score of ~ -5.7 --if you can probe enough data for each of the five patients before the competition is over. And I'm not convinced I'll be able to achieve that since I started so late and only 15% of the LB is public. Fortunately, it doesn't look like many competitors are taking this approach. But who knows?
", "votes": 0, "replies": [] }, { "id": 1002405, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-09-08T05:29:25.657000", "content": "If I were the host, I wouldn't care about the solutions. I'd care about the people who wrote them ;)
", "votes": 2, "replies": [] }, { "id": 1008509, "author_name": "Mark P", "author_url": "", "post_date": "2020-09-13T07:08:22.723000", "content": "Hi Carlos,
\n21 manual features!? 🤪 are those all from the ct scans? I can think only of: Lung volume, airway volume, some measure of texture. I know you are keeping your cards close to your chest on this until later in the competition but please could you give some hints of what other features are promising?
", "votes": 1, "replies": [] }, { "id": 1009355, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-09-13T22:31:07.470000", "content": "The manual features are no secret… I think I mentioned in a post or notebook:
\nThere's also the features from the unsupervised approach…
\nHope it helps! Cheers!
Doesn't work for me: Include simple image statistics (from segmented lungs) like mean, std, skew, kurtosis = didn't improve CV
", "votes": 3, "replies": [ { "id": 1001381, "author_name": "Bryan Arnold", "author_url": "", "post_date": "2020-09-07T09:50:43.430000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] } ] }, { "id": 1003123, "author_name": "Abhishek Bhat", "author_url": "", "post_date": "2020-09-08T17:11:57.063000", "content": "Hello Bryan, I too recently joined the competition and have been trying to extract features form the CT scans. I am going to share some of my initial findings. So far I have managed to obtain the following features from images:
\nBelow are a couple of normalized distributions of the variables mentioned above:
\n
Thanks, Abhishek. That's interesting. I'm definitely going to look at lung volume features in the next few days.
", "votes": 0, "replies": [] } ] }, { "id": 1002627, "author_name": "David Castillo", "author_url": "", "post_date": "2020-09-08T09:33:58.670000", "content": "I've found that the most important factor in this competition is \"gaming the metric\". By this I mean that a model that predicts future FVCs can be easily useless if given uncertainties aren't accurate enough (according to metric).
", "votes": 0, "replies": [] }, { "id": 1000771, "author_name": "Ahmmad Rashid", "author_url": "", "post_date": "2020-09-06T19:24:27.860000", "content": "No one wants to share their recipe….
", "votes": 0, "replies": [ { "id": 1000920, "author_name": "Bryan Arnold", "author_url": "", "post_date": "2020-09-06T23:05:08.420000", "content": "No worries. I don't blame them. The lack of data and various other factors make this a bit of a challenge.
", "votes": 0, "replies": [] } ] }, { "id": 1003355, "author_name": "", "author_url": "", "post_date": "2020-09-08T21:32:38.503000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "998754": "See my [notebook](https://www.kaggle.com/puremath86/naive-baseline-lb-7-1?scriptVersionId=42010889).\n\nI'm a late starter to this competition, so my apologies if this is old news to anyone. We are able to achieve a -7.1 Public LB score with a completely naive approach. The top score, as of yet, is about a -6.7. The theoretically best score is about a -4.6. So what are everyone's thoughts on this?\n\nI have seen a lot of public kernels using some really leaky features and information that is not available for the final prediction. It appears we can easily get a -6.9 or so with relatively naive models --`FVC` as a linear function of `Weeks` for each of the five `Patient`s in the test set. But to do much better we are going to have to incorporate information about the lungs.\n\nI know no one likes to give away their secret sauce --especially if they're on the trail of a potentially good idea-- but I'd like to discuss ideas and/or approaches. At the very least we can talk about what we have tried and hasn't worked... and save newbies and late-comers like myself some time. :P\n\nI was thinking we should be able to calculate the lung volume directly from a pixel count of the lungs. We would have to adjust for zoom/resolution/angle/etc., but that should be doable. I was also thinking that we can denoise the target in the training set (LOWESS?), remove outliers, and do some post-processing on the predictions --since it is just a time-series. Thoughts?\n\nThere are a myriad of other issues to address too. So what does everyone think?\n\n**EDIT** Ok. I guess I should also ask the question: LB Probing?", "1002191": "At least in my experiments, lung volume information doesn't bring me any improvement yet.", "1002373": "I think this is an amazing competition. Especially because it is hard.\nIMHO there are 2 key important things to create great models for this competition:\n1. Extracting relevant features from the CT scans\n2. Properly predicting uncertainty\n\nFitting curves through the ~1500 FVC points in the training set is easy. The hard thing to do is (1) and (2). In (1), I've tried manual feature engineering (+21 features) and unsupervised learning techniques (Variational Autoencoder, +16 features). This code is not public yet, but I intend to do it once the competition ends. In (2), there are 2 approaches: quantile models/pinball loss, and Bayesian approach/probabilistic machine learning. Also tried both, and made some of the code already public. \n", "1001377": "Doesn't work for me: Include simple image statistics (from segmented lungs) like mean, std, skew, kurtosis = didn't improve CV", "1003123": "Hello Bryan, I too recently joined the competition and have been trying to extract features form the CT scans. I am going to share some of my initial findings. So far I have managed to obtain the following features from images:\n1. **Lung Volume:** This variable is positively correlated with FVC and has a Pearson corr. coeff. of 0.35\n2. **Area of tissue pixels:** With some effort I managed to segment the tissue pixels and obtained the number of tissue pixels by setting some threshold. This variable is negatively correlated with FVC and a Pearson corr. coeff. of -0.02. I am planning to experiment with the threshold values in the future and include multiple features with different thresholds. \n3. **Other variables:** I have also extracted a few other tissue related features which show a good negative correlation with FVC. Might not be able reveal much at this point though.\n\nBelow are a couple of normalized distributions of the variables mentioned above:\n\n\n\n", "1002627": "I've found that the most important factor in this competition is \"gaming the metric\". By this I mean that a model that predicts future FVCs can be easily useless if given uncertainties aren't accurate enough (according to metric).", "1000771": "No one wants to share their recipe....", "1003355": "" } }