![\"Pairplot](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2488790%2Fcab84a01e7764f594cfb76ebc933b16c%2FWhatsApp%20Image%202020-09-19%20at%2007.15.10.jpeg?generation=1600527189718077&alt=media\")
I ran a bootstrap sampling, balancing the data and I got these funny looking graphs.
\nDoes smoking improve pulmonary capacity?
\nIs it correctly labeled?
I'm curious to discover what's happening here, or if I should start smoking before a marathon.
\nHere's my gist if you want to take a look at the code.
", "messages": [ { "id": "1018256", "postDate": "09/19/2020 14:59:26", "content": "I ran a bootstrap sampling, balancing the data and I got these funny looking graphs.
\nDoes smoking improve pulmonary capacity?
\nIs it correctly labeled?
I'm curious to discover what's happening here, or if I should start smoking before a marathon.
\nHere's my gist if you want to take a look at the code.
", "rawMarkdown": "I ran a bootstrap sampling, balancing the data and I got these funny looking graphs.\nDoes smoking improve pulmonary capacity?\nIs it correctly labeled?\n\n\n\nI'm curious to discover what's happening here, or if I should start smoking before a marathon.\n\nHere's [my gist](https://gist.github.com/Echo9k/ab13fba22011bde4031606f01ee3c4c2) if you want to take a look at the code.", "votes": null }, { "id": "1018570", "postDate": "09/19/2020 18:43:58", "content": "Hi! @echo9k,
\nDo you see the same difference between patients in training vs. the testing set? If this is the case, even if the labels are off, the model predictions will not be impacted, but it's also hard to know if they are flipped in the private validation sets.
I agree with you, as long as the labels are consistent it won't affect the result. However, if we use external data sources it would become noice for the algorithm.
\nThe test set has only 5 samples and it misses samples from current smokers.
My guess is that the disease is at an early stage for people that currently smokes, otherwise they would have stoped smoking already
", "rawMarkdown": "My guess is that the disease is at an early stage for people that currently smokes, otherwise they would have stoped smoking already", "votes": null }, { "id": "1018803", "postDate": "09/20/2020 01:56:47", "content": "@echo9k actually I was backcalculating some parameters using FVC models of the population, and it made my model quire bad. I hope this is not because the labels are off. Will be nice if someone from the competition team can confirm the situation.
", "rawMarkdown": "echo9k actually I was backcalculating some parameters using FVC models of the population, and it made my model quire bad. I hope this is not because the labels are off. Will be nice if someone from the competition team can confirm the situation.", "votes": null }, { "id": "1019023", "postDate": "09/20/2020 06:30:41", "content": "That's a good hypothesis, then \"currently smokers\" would have the same pulmonar capacity as \"never smoked\" patients.Yet smokers have better results.
", "rawMarkdown": "That's a good hypothesis, then \"currently smokers\" would have the same pulmonar capacity as \"never smoked\" patients.Yet smokers have better results.", "votes": null }, { "id": "1019650", "postDate": "09/20/2020 15:07:12", "content": "Not a pulmonologist expert, so I don’t know if the following makes sense, but there are some possibilities that could explain this observation:
\n1) maybe smokers go less to the doctor and therefore the only smokers for which we have an FVC reading are those that self-select into being healthier;
\n2) maybe smokers that get worse in fibrosis die first because of other diseases such as cancer;
\n3) maybe there is after all an effect of smoking on halting fibrosis even though it is bad for everything else, who knows?
Since I don’t know what is going on but have also observed this, I have chosen to model smokers and non-smokers separately. See my notebook for a possible implementation.
\nGood luck and success in the competition!
", "rawMarkdown": "Not a pulmonologist expert, so I don’t know if the following makes sense, but there are some possibilities that could explain this observation:\n\n1) maybe smokers go less to the doctor and therefore the only smokers for which we have an FVC reading are those that self-select into being healthier;\n2) maybe smokers that get worse in fibrosis die first because of other diseases such as cancer;\n3) maybe there is after all an effect of smoking on halting fibrosis even though it is bad for everything else, who knows?\n\nSince I don’t know what is going on but have also observed this, I have chosen to model smokers and non-smokers separately. See my notebook for a possible implementation. \n\nGood luck and success in the competition!", "votes": null }, { "id": "1020114", "postDate": "09/21/2020 00:06:41", "content": "Smoking does increase FVC but The FEV1/FVC ratio, also called Tiffeneau-Pinelli index, a calculated ratio used in the diagnosis of obstructive and restrictive lung disease, decreases.
\nA lower measured value corresponds to more severe lung abnormalities.
\nAlso I found an interesting research
\n\"Smokers reported significantly more wheezing and sputum production than nonsmokers but exhibited better FVC and FEV1 values. Those who smoked less than half a pack/d had significantly poorer FVC and FEV1 levels than nonsmokers. Smokers exhibited significantly lower FEV1/FVC ratios than nonsmokers.\"
\n", "rawMarkdown": "Smoking does increase FVC but The FEV1/FVC ratio, also called Tiffeneau-Pinelli index, a calculated ratio used in the diagnosis of obstructive and restrictive lung disease, decreases. \n\nA lower measured value corresponds to more severe lung abnormalities.\n\nAlso I found an interesting research \n\n\"Smokers reported significantly more wheezing and sputum production than nonsmokers but exhibited better FVC and FEV1 values. Those who smoked less than half a pack/d had significantly poorer FVC and FEV1 levels than nonsmokers. Smokers exhibited significantly lower FEV1/FVC ratios than nonsmokers.\"\n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC6709160/#:~:text=Smokers%20reported%20significantly%20more%20wheezing,FEV1%2FFVC%20ratios%20than%20nonsmokers.", "votes": null }, { "id": "1020143", "postDate": "09/21/2020 01:22:53", "content": "This sounds like a bias in data. Also you can think in a different way. As we are looking at a patient population, given they are developing this disease, if one never smoked, he usually have lower FVC (maybe he was not doing enough excise). If one is active smoker, he may have this disease even with a strong lung. So, do your marathon without smoke should be the way to go.
", "rawMarkdown": "This sounds like a bias in data. Also you can think in a different way. As we are looking at a patient population, given they are developing this disease, if one never smoked, he usually have lower FVC (maybe he was not doing enough excise). If one is active smoker, he may have this disease even with a strong lung. So, do your marathon without smoke should be the way to go.", "votes": null } ], "comments": [ { "id": 1018570, "author_name": "cv13j0", "author_url": "", "post_date": "09/19/2020 18:43:58", "content": "Hi! @echo9k,
\nDo you see the same difference between patients in training vs. the testing set? If this is the case, even if the labels are off, the model predictions will not be impacted, but it's also hard to know if they are flipped in the private validation sets.
I agree with you, as long as the labels are consistent it won't affect the result. However, if we use external data sources it would become noice for the algorithm.
\nThe test set has only 5 samples and it misses samples from current smokers.
@echo9k actually I was backcalculating some parameters using FVC models of the population, and it made my model quire bad. I hope this is not because the labels are off. Will be nice if someone from the competition team can confirm the situation.
", "votes": null, "replies": [] } ] }, { "id": 1018800, "author_name": "jsaguiar", "author_url": "", "post_date": "09/20/2020 01:44:32", "content": "My guess is that the disease is at an early stage for people that currently smokes, otherwise they would have stoped smoking already
", "votes": null, "replies": [ { "id": 1019023, "author_name": "echo9k", "author_url": "", "post_date": "09/20/2020 06:30:41", "content": "That's a good hypothesis, then \"currently smokers\" would have the same pulmonar capacity as \"never smoked\" patients.Yet smokers have better results.
", "votes": null, "replies": [] } ] }, { "id": 1019650, "author_name": "douglaskgaraujo", "author_url": "", "post_date": "09/20/2020 15:07:12", "content": "Not a pulmonologist expert, so I don’t know if the following makes sense, but there are some possibilities that could explain this observation:
\n1) maybe smokers go less to the doctor and therefore the only smokers for which we have an FVC reading are those that self-select into being healthier;
\n2) maybe smokers that get worse in fibrosis die first because of other diseases such as cancer;
\n3) maybe there is after all an effect of smoking on halting fibrosis even though it is bad for everything else, who knows?
Since I don’t know what is going on but have also observed this, I have chosen to model smokers and non-smokers separately. See my notebook for a possible implementation.
\nGood luck and success in the competition!
", "votes": null, "replies": [] }, { "id": 1020114, "author_name": "tkrsh09", "author_url": "", "post_date": "09/21/2020 00:06:41", "content": "Smoking does increase FVC but The FEV1/FVC ratio, also called Tiffeneau-Pinelli index, a calculated ratio used in the diagnosis of obstructive and restrictive lung disease, decreases.
\nA lower measured value corresponds to more severe lung abnormalities.
\nAlso I found an interesting research
\n\"Smokers reported significantly more wheezing and sputum production than nonsmokers but exhibited better FVC and FEV1 values. Those who smoked less than half a pack/d had significantly poorer FVC and FEV1 levels than nonsmokers. Smokers exhibited significantly lower FEV1/FVC ratios than nonsmokers.\"
\n", "votes": null, "replies": [] }, { "id": 1020143, "author_name": "libo9562", "author_url": "", "post_date": "09/21/2020 01:22:53", "content": "This sounds like a bias in data. Also you can think in a different way. As we are looking at a patient population, given they are developing this disease, if one never smoked, he usually have lower FVC (maybe he was not doing enough excise). If one is active smoker, he may have this disease even with a strong lung. So, do your marathon without smoke should be the way to go.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1018256": "I ran a bootstrap sampling, balancing the data and I got these funny looking graphs.\nDoes smoking improve pulmonary capacity?\nIs it correctly labeled?\n\n\n\nI'm curious to discover what's happening here, or if I should start smoking before a marathon.\n\nHere's [my gist](https://gist.github.com/Echo9k/ab13fba22011bde4031606f01ee3c4c2) if you want to take a look at the code.", "1018570": "Hi! @echo9k, \nDo you see the same difference between patients in training vs. the testing set? If this is the case, even if the labels are off, the model predictions will not be impacted, but it's also hard to know if they are flipped in the private validation sets.", "1018737": "I agree with you, as long as the labels are consistent it won't affect the result. However, if we use external data sources it would become noice for the algorithm.\nThe test set has only 5 samples and it misses samples from current smokers.", "1018800": "My guess is that the disease is at an early stage for people that currently smokes, otherwise they would have stoped smoking already", "1018803": "echo9k actually I was backcalculating some parameters using FVC models of the population, and it made my model quire bad. I hope this is not because the labels are off. Will be nice if someone from the competition team can confirm the situation.", "1019023": "That's a good hypothesis, then \"currently smokers\" would have the same pulmonar capacity as \"never smoked\" patients.Yet smokers have better results.", "1019650": "Not a pulmonologist expert, so I don’t know if the following makes sense, but there are some possibilities that could explain this observation:\n\n1) maybe smokers go less to the doctor and therefore the only smokers for which we have an FVC reading are those that self-select into being healthier;\n2) maybe smokers that get worse in fibrosis die first because of other diseases such as cancer;\n3) maybe there is after all an effect of smoking on halting fibrosis even though it is bad for everything else, who knows?\n\nSince I don’t know what is going on but have also observed this, I have chosen to model smokers and non-smokers separately. See my notebook for a possible implementation. \n\nGood luck and success in the competition!", "1020114": "Smoking does increase FVC but The FEV1/FVC ratio, also called Tiffeneau-Pinelli index, a calculated ratio used in the diagnosis of obstructive and restrictive lung disease, decreases. \n\nA lower measured value corresponds to more severe lung abnormalities.\n\nAlso I found an interesting research \n\n\"Smokers reported significantly more wheezing and sputum production than nonsmokers but exhibited better FVC and FEV1 values. Those who smoked less than half a pack/d had significantly poorer FVC and FEV1 levels than nonsmokers. Smokers exhibited significantly lower FEV1/FVC ratios than nonsmokers.\"\n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC6709160/#:~:text=Smokers%20reported%20significantly%20more%20wheezing,FEV1%2FFVC%20ratios%20than%20nonsmokers.", "1020143": "This sounds like a bias in data. Also you can think in a different way. As we are looking at a patient population, given they are developing this disease, if one never smoked, he usually have lower FVC (maybe he was not doing enough excise). If one is active smoker, he may have this disease even with a strong lung. So, do your marathon without smoke should be the way to go." }, "source": "meta" }