![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4027889%2Faa081ebac8d31aa18575cc4dbc010b86%2Fhist.png?generation=1596061012216851&alt=media\")
Do you have any thoughts about the outlier in the training data? As the FVC distribution plot shows, there are bins with > 5500 value:
\n\n
which turns out to belong to one Patient - ID00219637202258203123958.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4027889%2F9e4f65198b29a47b0d61180d51db4f91%2Ftable.png?generation=1596061256309052&alt=media\")
Perhaps it is worthy to ommit this Patient in the training, doesn't it?
", "messages": [ { "id": "951116", "postDate": "07/29/2020 22:23:43", "content": "Do you have any thoughts about the outlier in the training data? As the FVC distribution plot shows, there are bins with > 5500 value:
\n\n
which turns out to belong to one Patient - ID00219637202258203123958.
\n\n
Perhaps it is worthy to ommit this Patient in the training, doesn't it?
", "rawMarkdown": "Do you have any thoughts about the outlier in the training data? As the FVC distribution plot shows, there are bins with > 5500 value:\n\n\n\nwhich turns out to belong to one Patient - ID00219637202258203123958.\n\n\n\nPerhaps it is worthy to ommit this Patient in the training, doesn't it?", "votes": null }, { "id": "951311", "postDate": "07/30/2020 03:59:58", "content": "FVC with values about 6000 is possible for swimmers.
", "rawMarkdown": "FVC with values about 6000 is possible for swimmers.", "votes": null }, { "id": "953214", "postDate": "07/31/2020 15:52:04", "content": "I don't think you have to drop these outlier patients because test set have baseline FVC and Percent for every patient. Your predictions will be closer to those baselines and slowly decrease (if you include a continuous time feature). As you see the first couple predictions are always close to baseline FVC.
![\"\"](\"https://i.ibb.co/ZWhxgyh/s2-1.jpg\")
Just wanted to say that I really like your visualization!
", "rawMarkdown": "Just wanted to say that I really like your visualization!", "votes": null }, { "id": "953461", "postDate": "07/31/2020 19:58:29", "content": "It make sense, I thought that if FVC ~6000 is an almost impossible case (as @miklgr500 mentioned, it doesn't have to be that rare), for the sake of model's generalization ability it would be beneficial to drop this one Patient.
", "rawMarkdown": "It make sense, I thought that if FVC ~6000 is an almost impossible case (as @miklgr500 mentioned, it doesn't have to be that rare), for the sake of model's generalization ability it would be beneficial to drop this one Patient.", "votes": null }, { "id": "953526", "postDate": "07/31/2020 21:18:04", "content": "Did you get any benefit by dropping it?
", "rawMarkdown": "Did you get any benefit by dropping it?", "votes": null }, { "id": "983642", "postDate": "08/24/2020 13:17:33", "content": "The more concerning outliers seem to be the FVC values that cause abrupt changes in the trends. If these outliers are not addressed, they will decrease models' performances.
\nAnother problem regarding the outliers is whether the private test FVC values have outliers too. I asked the competition host about this here.
", "rawMarkdown": "The more concerning outliers seem to be the FVC values that cause abrupt changes in the trends. If these outliers are not addressed, they will decrease models' performances.\n\nAnother problem regarding the outliers is whether the private test FVC values have outliers too. I asked the competition host about this [here](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/176826).", "votes": null } ], "comments": [ { "id": 983642, "author_name": "tolgadincer", "author_url": "", "post_date": "08/24/2020 13:17:33", "content": "The more concerning outliers seem to be the FVC values that cause abrupt changes in the trends. If these outliers are not addressed, they will decrease models' performances.
\nAnother problem regarding the outliers is whether the private test FVC values have outliers too. I asked the competition host about this here.
", "votes": null, "replies": [] }, { "id": 951311, "author_name": "miklgr500", "author_url": "", "post_date": "07/30/2020 03:59:58", "content": "FVC with values about 6000 is possible for swimmers.
", "votes": null, "replies": [] }, { "id": 953214, "author_name": "gunesevitan", "author_url": "", "post_date": "07/31/2020 15:52:04", "content": "I don't think you have to drop these outlier patients because test set have baseline FVC and Percent for every patient. Your predictions will be closer to those baselines and slowly decrease (if you include a continuous time feature). As you see the first couple predictions are always close to baseline FVC.
Just wanted to say that I really like your visualization!
", "votes": null, "replies": [] }, { "id": 953461, "author_name": "agnieszkalysak", "author_url": "", "post_date": "07/31/2020 19:58:29", "content": "It make sense, I thought that if FVC ~6000 is an almost impossible case (as @miklgr500 mentioned, it doesn't have to be that rare), for the sake of model's generalization ability it would be beneficial to drop this one Patient.
", "votes": null, "replies": [] }, { "id": 953526, "author_name": "jjinho", "author_url": "", "post_date": "07/31/2020 21:18:04", "content": "Did you get any benefit by dropping it?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "951116": "Do you have any thoughts about the outlier in the training data? As the FVC distribution plot shows, there are bins with > 5500 value:\n\n\n\nwhich turns out to belong to one Patient - ID00219637202258203123958.\n\n\n\nPerhaps it is worthy to ommit this Patient in the training, doesn't it?", "951311": "FVC with values about 6000 is possible for swimmers.", "953214": "I don't think you have to drop these outlier patients because test set have baseline `FVC` and `Percent` for every patient. Your predictions will be closer to those baselines and slowly decrease (if you include a continuous time feature). As you see the first couple predictions are always close to baseline `FVC`.\n\n", "953319": "Just wanted to say that I really like your visualization!", "953461": "It make sense, I thought that if FVC ~6000 is an almost impossible case (as @miklgr500 mentioned, it doesn't have to be that rare), for the sake of model's generalization ability it would be beneficial to drop this one Patient.", "953526": "Did you get any benefit by dropping it?", "983642": "The more concerning outliers seem to be the FVC values that cause abrupt changes in the trends. If these outliers are not addressed, they will decrease models' performances.\n\nAnother problem regarding the outliers is whether the private test FVC values have outliers too. I asked the competition host about this [here](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/176826)." }, "source": "meta" }