![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4256010%2F0c53f7742dcaecc7ea1037f418456ddb%2Fpic1.png?generation=1597774467598630&alt=media\")
Hey Kagglers! Would love to get your thoughts/comments on the analysis below.
\nIn the data description it says that the \"Percent\" field is:
\n\n\na computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics
\n
I was wondering what similar characteristics could mean, so I did a quick analysis to check whether the parameters we have could explain the relationship. These charts are scatter plots of Percent vs FCV for various ages, all ex-smokers, all male.
\n
Because each chart is narrowed down on one possible combination of parameters we might expect the relationship between FVC and % (FVC:% hereafter) to be fully explained. But that's far from the case. This suggests that there are hidden parameters used to define the FVC:% relationship.
\nWe can plot all ages together to get a sense for how much age is able to explain the FVC:% relationship:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4256010%2F988af5d5ad944ff676f537d9cc1f6568%2Fpic2.png?generation=1597774263178612&alt=media\")
As you can see, all of the points mix in together with quite some fanning going on. The only alignment comes from the fact that subsets of points belong to individual patients. What this suggests, is that the hidden parameters are a better indication of FVC:% than age is.
\nFinally, I decided to single out Sex and SmokingStatus to check how much they can explain FVC:%. This time I didn't fix all other parameters.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4256010%2F25d4cacebf97d8f8206650b5beb87dcc%2Fpic3.png?generation=1597773193084722&alt=media\")
At least Sex seems to be an important factor in explaining FVC:%, while Smoking Status doesn't seem to have much of a say.
\nSo there we have it. My conclusion from this analysis is that there are many hidden parameters which fed into the idea of \"persons of similar characteristics\". What this means is that FVC:% is actually a useful parameter in its own right.
\nINB4 - Yes, straight lines appear on the data because each point belonging to an individual patient has the same FVC:%. That's not what I'm getting at here. In fact, by the time I finished this analysis, I realised I should have just made another df column for FVC:% and deduplicated such that I'd have one row per patient. Then I could have made much more concise plots to reach the same conclusions.
", "messages": [ { "id": "976217", "postDate": "08/18/2020 18:14:31", "content": "Hey Kagglers! Would love to get your thoughts/comments on the analysis below.
\nIn the data description it says that the \"Percent\" field is:
\n\n\na computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics
\n
I was wondering what similar characteristics could mean, so I did a quick analysis to check whether the parameters we have could explain the relationship. These charts are scatter plots of Percent vs FCV for various ages, all ex-smokers, all male.
\n
Because each chart is narrowed down on one possible combination of parameters we might expect the relationship between FVC and % (FVC:% hereafter) to be fully explained. But that's far from the case. This suggests that there are hidden parameters used to define the FVC:% relationship.
\nWe can plot all ages together to get a sense for how much age is able to explain the FVC:% relationship:
\n
As you can see, all of the points mix in together with quite some fanning going on. The only alignment comes from the fact that subsets of points belong to individual patients. What this suggests, is that the hidden parameters are a better indication of FVC:% than age is.
\nFinally, I decided to single out Sex and SmokingStatus to check how much they can explain FVC:%. This time I didn't fix all other parameters.
\n
At least Sex seems to be an important factor in explaining FVC:%, while Smoking Status doesn't seem to have much of a say.
\nSo there we have it. My conclusion from this analysis is that there are many hidden parameters which fed into the idea of \"persons of similar characteristics\". What this means is that FVC:% is actually a useful parameter in its own right.
\nINB4 - Yes, straight lines appear on the data because each point belonging to an individual patient has the same FVC:%. That's not what I'm getting at here. In fact, by the time I finished this analysis, I realised I should have just made another df column for FVC:% and deduplicated such that I'd have one row per patient. Then I could have made much more concise plots to reach the same conclusions.
", "rawMarkdown": "Hey Kagglers! Would love to get your thoughts/comments on the analysis below.\n\nIn the [data description](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/data) it says that the \"Percent\" field is:\n\n> a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\n\nI was wondering what **similar characteristics** could mean, so I did a quick analysis to check whether the parameters we have could explain the relationship. These charts are scatter plots of Percent vs FCV for various ages, all ex-smokers, all male.\n\n\n\nBecause each chart is narrowed down on one possible combination of parameters we might expect the relationship between FVC and % (**FVC:%** hereafter) to be fully explained. But that's far from the case. **This suggests that there are hidden parameters used to define the FVC:% relationship**.\n\nWe can plot all ages together to get a sense for how much age is able to explain the **FVC:%** relationship:\n\n\n\nAs you can see, all of the points mix in together with quite some fanning going on. The only alignment comes from the fact that subsets of points belong to individual patients. What this suggests, is that the **hidden parameters** are a better indication of **FVC:%** than age is.\n\nFinally, I decided to single out Sex and SmokingStatus to check how much they can explain **FVC:%**. This time I didn't fix all other parameters.\n\n\n\nAt least Sex seems to be an important factor in explaining **FVC:%**, while Smoking Status doesn't seem to have much of a say.\n\nSo there we have it. My conclusion from this analysis is that there are many hidden parameters which fed into the idea of \"persons of similar characteristics\". What this means is that **FVC:%** is actually a useful parameter in its own right.\n\n**INB4** - Yes, straight lines appear on the data because each point belonging to an individual patient has the same **FVC:%**. That's not what I'm getting at here. In fact, by the time I finished this analysis, I realised I should have just made another df column for **FVC:%** and deduplicated such that I'd have one row per patient. Then I could have made much more concise plots to reach the same conclusions.", "votes": null }, { "id": "976532", "postDate": "08/18/2020 23:46:05", "content": "The relationship between FVC and percent just seems obvious to me. I don't really know how that could be useful in prediction.
", "rawMarkdown": "The relationship between FVC and percent just seems obvious to me. I don't really know how that could be useful in prediction.", "votes": null }, { "id": "976551", "postDate": "08/19/2020 00:14:01", "content": "Cool idea, and thanks for sharing @alexandersoare. Since the relationship between age and… \"patient\" (a single patients values) compared to percent seems to be a linear one, we could create a patientage-normal (perhaps more commonly referred to as de-seasonalized) FVC% column and feed that into our models. We won't know what those latent hidden variables encode, but at least we'll have the unadulterated information :-).
Hi thanks for sharing. Can you please elaborate. How's it obvious?
", "rawMarkdown": "Hi thanks for sharing. Can you please elaborate. How's it obvious?", "votes": null }, { "id": "976782", "postDate": "08/19/2020 05:03:53", "content": "Oh yeah sure. I was thinking about how FVC and the percent (which is compared to the percent of the FVC for a normal person) would be correlated you know. But I guess as the lung fibrosis progresses it isn't as obvious.
", "rawMarkdown": "Oh yeah sure. I was thinking about how FVC and the percent (which is compared to the percent of the FVC for a normal person) would be correlated you know. But I guess as the lung fibrosis progresses it isn't as obvious.", "votes": null }, { "id": "976878", "postDate": "08/19/2020 06:41:08", "content": "Yes I was referring to the \"normal person\" aspect. For any individual person the ratio is fixed throughout the whole time series. But across different people the ratio is not fixed, even if these people have the same characteristics as per the provided data.
", "rawMarkdown": "Yes I was referring to the \"normal person\" aspect. For any individual person the ratio is fixed throughout the whole time series. But across different people the ratio is not fixed, even if these people have the same characteristics as per the provided data.", "votes": null }, { "id": "977483", "postDate": "08/19/2020 14:00:12", "content": "In this excellent domain experts insight (https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166123) Dr. Konya mentions that 'In adults, age, height, sex and race are the main determinants of the reference values for spirometric measurements.' which in this case refers to the FVC percentage. In the data we have neither height nor 'race' so these might be all or some of the hidden parameters.
", "rawMarkdown": "In this excellent domain experts insight (https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166123) Dr. Konya mentions that 'In adults, age, height, sex and race are the main determinants of the reference values for spirometric measurements.' which in this case refers to the FVC percentage. In the data we have neither height nor 'race' so these might be all or some of the hidden parameters.", "votes": null }, { "id": "977489", "postDate": "08/19/2020 14:06:38", "content": "Excellent. Thank you!
", "rawMarkdown": "Excellent. Thank you!", "votes": null }, { "id": "977688", "postDate": "08/19/2020 16:39:39", "content": "The lines you're seeing in the data comes from plotting multiple (FVC, Percent) pairs for each patient. Since FVC and Percent are directly proportional, all measurements for a single patient lie on the same line.
\n(Not saying that Percent does not contain any useful information though.)
\nWith every measurement for each patient (with redundant information):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F94532%2Ff0ba5859d697682dca00318377d14fce%2FAll%20data.png?generation=1597855132214916&alt=media\")
Using only the first measurement for each patient (which provides the same information):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F94532%2F6508c4655f7639762110d9f7a280dd30%2FOnly%20the%20first%20measurement.png?generation=1597855167292344&alt=media\")
This is exactly what I was trying to say with my first comment.
", "rawMarkdown": "This is exactly what I was trying to say with my first comment.", "votes": null }, { "id": "977753", "postDate": "08/19/2020 17:30:06", "content": "@alexandersoare i came to your post after i found some thing unusual in FVC and percentage data .
\nin some cases inface quite a many given FVC is 100+ percentage of some reference value that too for people aged over 60 . i was wondering can a low FVC still be considered normal for a adult ,just bcz people belong to his age group show similar FVC normally?
Yep, I do mention that in the post. I'm more interested in the slope of the lines. If I had thought about it at the start of my analysis I would have just made another column called slope and deduplicated the data such that there is one row per patient.
", "rawMarkdown": "Yep, I do mention that in the post. I'm more interested in the slope of the lines. If I had thought about it at the start of my analysis I would have just made another column called slope and deduplicated the data such that there is one row per patient.", "votes": null }, { "id": "977763", "postDate": "08/19/2020 17:35:06", "content": "100% + just means they have a greater FVC than the \"norm\". And according @FrederikLaubisch's response on this thread, it is indeed suggested that age should affect FVC.
", "rawMarkdown": "100% + just means they have a greater FVC than the \"norm\". And according @FrederikLaubisch's response on this thread, it is indeed suggested that age should affect FVC.", "votes": null }, { "id": "978443", "postDate": "08/20/2020 07:23:01", "content": "@alexandersoare
\n Congrats for your silver medal in GWD on a solo basis. i am happy to inform you that with just 10 days of GWD participation i could get 225 rank on selected sub finally and 15th on unselected one . I am happy in last 5 days i could turn lot of things :)
\nLet me know if you want to team up with me this time which we couldnt in GWD . we can do lot of things..
@jaideepvalani congratulations! that's a lot to achieve in just 10 days. As you might notice, I haven't yet made a submission here as I'm not yet all in on competing. I'd like to test the waters and wait for some other things to settle before making a choice here. Thanks for the offer and I'll keep you in mind!
", "rawMarkdown": "jaideepvalani congratulations! that's a lot to achieve in just 10 days. As you might notice, I haven't yet made a submission here as I'm not yet all in on competing. I'd like to test the waters and wait for some other things to settle before making a choice here. Thanks for the offer and I'll keep you in mind!", "votes": null }, { "id": "978470", "postDate": "08/20/2020 07:40:49", "content": "sure no issues.. my intent is to learn with help of people from whom i think i can get good knowledge,and i can share same at my best i know. I will work till then.
\nRanks are volatile here 15 pct of data is nothing..
Thanks for this! Maybe we can assume the height and race based some logic…
", "rawMarkdown": "Thanks for this! Maybe we can assume the height and race based some logic...", "votes": null }, { "id": "988734", "postDate": "08/28/2020 08:35:04", "content": "Thank you for a good discussion on \"FVC\" and \"Percent\".
\nI'm worried that the test data will only give \"Percent\" data for the first week. If that's the case, I don't think it's appropriate to use \"Percent\" in a predictive model, is it?
\nOn the other hand, since the \"FVC/Percent\" values are always constant from patient to patient and determine the characteristics of each patient, could it be considered a valid variable even if the test data given is only for the first week?
The test data should give whatever it is giving now in the sample test.csv. So I don't think you need to worry about that
", "rawMarkdown": "The test data should give whatever it is giving now in the sample test.csv. So I don't think you need to worry about that", "votes": null }, { "id": "989553", "postDate": "08/28/2020 22:45:40", "content": "Hi,
\nthe relationship between \"FVC\" and \"Percent\" is that Percent = FVC / FEV1, and it represents the percent of the FVC that can be exhaled in one second.
Hi! @azzeineaftiss, I don't think the Percent = FVC/FEV1. from the competition description
\n\"Percent- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\"
I believe the Percent is something more like FVC calculated from a model like
\nHankinson, et al: Am J Respir Crit Care, 1999
\nIt will be nice to get clarification from the competition team.
On-line calculators can be found that compute the full volume and the Percent. A short paper can be found that show the formula for the full volume to be (volumes are in liters vs our data in ml). Google is your friend.
\nFull_Volume = LN(Age)A + AgeB +C/Height(in cm) + D
\nThe coefficients (A, B, C, D) are different by sex and by Race. Height is an accepted feaure - what is really wanted is the thoracic volume of the lungs - something that might be obtained from the CT scans for those skilled enough.
\nRace can be one of three groups, Caucasian, Hispanic, Black. When race in doubt or unknown the procedure calls for the use of Caucasian.
\nThe LN function is needed as full volume for youngsters increases with age - than following peak it declines for the remaining years of our lives.
\nWhen building an equation from our data to generate this full volume there are a few residuals. My guess is that age is one of the sources of error. Our patients have the same age over the full -12 to 133 weeks - that is clearly WRONG and likely accounts for the imperfect regression.
", "rawMarkdown": "On-line calculators can be found that compute the full volume and the Percent. A short paper can be found that show the formula for the full volume to be (volumes are in liters vs our data in ml). Google is your friend.\n\nFull_Volume = LN(Age)*A + Age*B +C/Height(in cm) + D\n\nThe coefficients (A, B, C, D) are different by sex and by Race. Height is an accepted feaure - what is really wanted is the thoracic volume of the lungs - something that might be obtained from the CT scans for those skilled enough.\n\nRace can be one of three groups, Caucasian, Hispanic, Black. When race in doubt or unknown the procedure calls for the use of Caucasian. \n\nThe LN function is needed as full volume for youngsters increases with age - than following peak it declines for the remaining years of our lives.\n\nWhen building an equation from our data to generate this full volume there are a few residuals. My guess is that age is one of the sources of error. Our patients have the same age over the full -12 to 133 weeks - that is clearly WRONG and likely accounts for the imperfect regression.", "votes": null } ], "comments": [ { "id": 976532, "author_name": "jonykarki", "author_url": "", "post_date": "08/18/2020 23:46:05", "content": "The relationship between FVC and percent just seems obvious to me. I don't really know how that could be useful in prediction.
", "votes": null, "replies": [ { "id": 976768, "author_name": "alexandersoare", "author_url": "", "post_date": "08/19/2020 04:44:45", "content": "Hi thanks for sharing. Can you please elaborate. How's it obvious?
", "votes": null, "replies": [] }, { "id": 976782, "author_name": "jonykarki", "author_url": "", "post_date": "08/19/2020 05:03:53", "content": "Oh yeah sure. I was thinking about how FVC and the percent (which is compared to the percent of the FVC for a normal person) would be correlated you know. But I guess as the lung fibrosis progresses it isn't as obvious.
", "votes": null, "replies": [] }, { "id": 976878, "author_name": "alexandersoare", "author_url": "", "post_date": "08/19/2020 06:41:08", "content": "Yes I was referring to the \"normal person\" aspect. For any individual person the ratio is fixed throughout the whole time series. But across different people the ratio is not fixed, even if these people have the same characteristics as per the provided data.
", "votes": null, "replies": [] }, { "id": 977753, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/19/2020 17:30:06", "content": "@alexandersoare i came to your post after i found some thing unusual in FVC and percentage data .
\nin some cases inface quite a many given FVC is 100+ percentage of some reference value that too for people aged over 60 . i was wondering can a low FVC still be considered normal for a adult ,just bcz people belong to his age group show similar FVC normally?
100% + just means they have a greater FVC than the \"norm\". And according @FrederikLaubisch's response on this thread, it is indeed suggested that age should affect FVC.
", "votes": null, "replies": [] }, { "id": 978443, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/20/2020 07:23:01", "content": "@alexandersoare
\n Congrats for your silver medal in GWD on a solo basis. i am happy to inform you that with just 10 days of GWD participation i could get 225 rank on selected sub finally and 15th on unselected one . I am happy in last 5 days i could turn lot of things :)
\nLet me know if you want to team up with me this time which we couldnt in GWD . we can do lot of things..
@jaideepvalani congratulations! that's a lot to achieve in just 10 days. As you might notice, I haven't yet made a submission here as I'm not yet all in on competing. I'd like to test the waters and wait for some other things to settle before making a choice here. Thanks for the offer and I'll keep you in mind!
", "votes": null, "replies": [] }, { "id": 978470, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/20/2020 07:40:49", "content": "sure no issues.. my intent is to learn with help of people from whom i think i can get good knowledge,and i can share same at my best i know. I will work till then.
\nRanks are volatile here 15 pct of data is nothing..
Cool idea, and thanks for sharing @alexandersoare. Since the relationship between age and… \"patient\" (a single patients values) compared to percent seems to be a linear one, we could create a patientage-normal (perhaps more commonly referred to as de-seasonalized) FVC% column and feed that into our models. We won't know what those latent hidden variables encode, but at least we'll have the unadulterated information :-).
In this excellent domain experts insight (https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/166123) Dr. Konya mentions that 'In adults, age, height, sex and race are the main determinants of the reference values for spirometric measurements.' which in this case refers to the FVC percentage. In the data we have neither height nor 'race' so these might be all or some of the hidden parameters.
", "votes": null, "replies": [ { "id": 977489, "author_name": "alexandersoare", "author_url": "", "post_date": "08/19/2020 14:06:38", "content": "Excellent. Thank you!
", "votes": null, "replies": [] }, { "id": 978960, "author_name": "srikanthpotukuchi", "author_url": "", "post_date": "08/20/2020 14:32:37", "content": "Thanks for this! Maybe we can assume the height and race based some logic…
", "votes": null, "replies": [] }, { "id": 1001894, "author_name": "pcjimmmy", "author_url": "", "post_date": "09/07/2020 16:49:40", "content": "On-line calculators can be found that compute the full volume and the Percent. A short paper can be found that show the formula for the full volume to be (volumes are in liters vs our data in ml). Google is your friend.
\nFull_Volume = LN(Age)A + AgeB +C/Height(in cm) + D
\nThe coefficients (A, B, C, D) are different by sex and by Race. Height is an accepted feaure - what is really wanted is the thoracic volume of the lungs - something that might be obtained from the CT scans for those skilled enough.
\nRace can be one of three groups, Caucasian, Hispanic, Black. When race in doubt or unknown the procedure calls for the use of Caucasian.
\nThe LN function is needed as full volume for youngsters increases with age - than following peak it declines for the remaining years of our lives.
\nWhen building an equation from our data to generate this full volume there are a few residuals. My guess is that age is one of the sources of error. Our patients have the same age over the full -12 to 133 weeks - that is clearly WRONG and likely accounts for the imperfect regression.
", "votes": null, "replies": [] } ] }, { "id": 977688, "author_name": "christoffer", "author_url": "", "post_date": "08/19/2020 16:39:39", "content": "The lines you're seeing in the data comes from plotting multiple (FVC, Percent) pairs for each patient. Since FVC and Percent are directly proportional, all measurements for a single patient lie on the same line.
\n(Not saying that Percent does not contain any useful information though.)
\nWith every measurement for each patient (with redundant information):
\n
Using only the first measurement for each patient (which provides the same information):
\n
This is exactly what I was trying to say with my first comment.
", "votes": null, "replies": [] }, { "id": 977755, "author_name": "alexandersoare", "author_url": "", "post_date": "08/19/2020 17:31:53", "content": "Yep, I do mention that in the post. I'm more interested in the slope of the lines. If I had thought about it at the start of my analysis I would have just made another column called slope and deduplicated the data such that there is one row per patient.
", "votes": null, "replies": [] } ] }, { "id": 988734, "author_name": "ti110106", "author_url": "", "post_date": "08/28/2020 08:35:04", "content": "Thank you for a good discussion on \"FVC\" and \"Percent\".
\nI'm worried that the test data will only give \"Percent\" data for the first week. If that's the case, I don't think it's appropriate to use \"Percent\" in a predictive model, is it?
\nOn the other hand, since the \"FVC/Percent\" values are always constant from patient to patient and determine the characteristics of each patient, could it be considered a valid variable even if the test data given is only for the first week?
The test data should give whatever it is giving now in the sample test.csv. So I don't think you need to worry about that
", "votes": null, "replies": [] } ] }, { "id": 989553, "author_name": "azzeineaftiss", "author_url": "", "post_date": "08/28/2020 22:45:40", "content": "Hi,
\nthe relationship between \"FVC\" and \"Percent\" is that Percent = FVC / FEV1, and it represents the percent of the FVC that can be exhaled in one second.
Hi! @azzeineaftiss, I don't think the Percent = FVC/FEV1. from the competition description
\n\"Percent- a computed field which approximates the patient's FVC as a percent of the typical FVC for a person of similar characteristics\"
I believe the Percent is something more like FVC calculated from a model like
\nHankinson, et al: Am J Respir Crit Care, 1999
\nIt will be nice to get clarification from the competition team.