{ "id": 183626, "title": "Do you really think public solution might work?", "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/183626", "author_name": "", "post_date": "2020-09-17T14:02:29.344052700Z", "votes": 16, "comment_count": 20, "views": 0, "content": "

I want to remind, that public solution doesn't use patient splitting in cross validation, so surprisingly small MAE value were given:
\n\"\"

\n

It's unbelievable to get validation MAE<100 in this competition. So, you have to understand the next points in this case:

\n", "messages": [ { "id": "1014522", "postDate": "09/17/2020 14:02:29", "content": "

I want to remind, that public solution doesn't use patient splitting in cross validation, so surprisingly small MAE value were given:
\n\"\"

\n

It's unbelievable to get validation MAE<100 in this competition. So, you have to understand the next points in this case:

\n", "rawMarkdown": "I want to remind, that public solution doesn't use patient splitting in cross validation, so surprisingly small MAE value were given:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1659719%2Fb2da832811656473e6129474928f19c5%2FScreenshot%20from%202020-09-17%2016-55-06.png?generation=1600351027822753&alt=media)\n\nIt's unbelievable to get validation MAE<100 in this competition. So, you have to understand the next points in this case:\n- cross validation results are absolutely invalid\n- chances are, your model just repeat the last FVC value from train set for every patient\n- in my opinion, improving LB results reason is probably some noise aspects during training or just luck ;)", "votes": null }, { "id": "1014832", "postDate": "09/17/2020 18:08:58", "content": "

Aside from not properly splitting data for CV, most of the public solutions use percent as input feature, which is a huge leakage. I don't think these solution will work, as they are conceptually wrong.

", "rawMarkdown": "Aside from not properly splitting data for CV, most of the public solutions use percent as input feature, which is a huge leakage. I don't think these solution will work, as they are conceptually wrong.", "votes": null }, { "id": "1014856", "postDate": "09/17/2020 18:27:36", "content": "

The percent column is not leakage; it's a function of the initial FVC measurement and patient characteristics.

", "rawMarkdown": "The percent column is not leakage; it's a function of the initial FVC measurement and patient characteristics.", "votes": null }, { "id": "1014863", "postDate": "09/17/2020 18:31:52", "content": "

Agree, that the second reason for leakage. But the first one influence leakage so bigger. For instance, in my experiments with patient leak splitting I get about 60 CV MAE, with patient independent splitting - about 160 CV MAE, but really fair CV MAE (in my opinion) gives about 200 MAE.

", "rawMarkdown": "Agree, that the second reason for leakage. But the first one influence leakage so bigger. For instance, in my experiments with patient leak splitting I get about 60 CV MAE, with patient independent splitting - about 160 CV MAE, but really fair CV MAE (in my opinion) gives about 200 MAE.", "votes": null }, { "id": "1014866", "postDate": "09/17/2020 18:34:59", "content": "

It's not a leak in the sense that you cannot exploit it on the test set. However, you can build a model, in which you use the Percent in a week to predict the FVC in that week. If you do that, you should be able to get a perfect score. Essentially you just need to teach a model that
\n (FVC in a week) = baseline FVC * Percent in the week / Percent at baseline
\nYou could do a nice cross-validation (never mind if you \"validate\" your model on your training data) and still fool yourself into thinking that you have a great model.

\n

Of course, that will not work on the test set, because you do not have Percent for the weeks you need to predict. So, if a model gives good results on the public leaderboard, if cannot be due to exploiting this (because you cannot exploit it on the leaderboard).

", "rawMarkdown": "It's not a leak in the sense that you cannot exploit it on the test set. However, you can build a model, in which you use the `Percent` in a week to predict the `FVC` in that week. If you do that, you should be able to get a perfect score. Essentially you just need to teach a model that \n `(FVC in a week) = baseline FVC * Percent in the week / Percent at baseline` \nYou could do a nice cross-validation (never mind if you \"validate\" your model on your training data) and still fool yourself into thinking that you have a great model.\n\nOf course, that will not work on the test set, because you do not have `Percent` for the weeks you need to predict. So, if a model gives good results on the public leaderboard, if cannot be due to exploiting this (because you cannot exploit it on the leaderboard).", "votes": null }, { "id": "1014867", "postDate": "09/17/2020 18:36:40", "content": "

During inference time it's not a leakage, cause you have an access to first measurement row only. But during training, where all measurement are available, that's a leakage on my side, cause model can use Percent feature only, get nice CV score and fails in the inference time then. But if you use only first rows for each patients, it's OK.

", "rawMarkdown": "During inference time it's not a leakage, cause you have an access to first measurement row only. But during training, where all measurement are available, that's a leakage on my side, cause model can use Percent feature only, get nice CV score and fails in the inference time then. But if you use only first rows for each patients, it's OK.", "votes": null }, { "id": "1014869", "postDate": "09/17/2020 18:38:12", "content": "

Absolutely the same understanding)

", "rawMarkdown": "Absolutely the same understanding)", "votes": null }, { "id": "1014949", "postDate": "09/17/2020 20:17:01", "content": "

These two screenshots are taken from one of the most popular notebooks: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter which scored 6.83X on LB.

\n

Training set:
\n\"\"

\n

Testing set:
\n\"\"

\n

This is the leakage I'm talking about. You will not have the percent feature for each time step for a given patient, only the first value. However, these notebooks are training the model as if the percent value were available at all time steps.

", "rawMarkdown": "These two screenshots are taken from one of the most popular notebooks: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter which scored 6.83X on LB. \n\nTraining set:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F350756%2F1fff4b5d57cb4971385d100134c74ce1%2FScreen%20Shot%202020-09-17%20at%2015.08.57.png?generation=1600373457125816&alt=media)\n\nTesting set:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F350756%2F26f2a06b39a88524c2dbaae60d9876fd%2FScreen%20Shot%202020-09-17%20at%2015.09.20.png?generation=1600373481936507&alt=media)\n\nThis is the leakage I'm talking about. You will not have the percent feature for each time step for a given patient, only the first value. However, these notebooks are training the model as if the percent value were available at all time steps.", "votes": null }, { "id": "1014953", "postDate": "09/17/2020 20:18:36", "content": "

@wcukierski I know isn't a leakage itself, but the way is being used for training produces leakage. See my post below.

", "rawMarkdown": "wcukierski I know isn't a leakage itself, but the way is being used for training produces leakage. See my post below.", "votes": null }, { "id": "1014958", "postDate": "09/17/2020 20:21:37", "content": "

Agree, so clearly

", "rawMarkdown": "Agree, so clearly", "votes": null }, { "id": "1016135", "postDate": "09/18/2020 17:29:08", "content": "

I think these notebooks are really a good starting point in this competition as they provide a decent base code. One will definitely have to make a few changes to these readily available notebooks like use GroupKFold CV to avoid leakage, use only first Percent value, experiment with other model parameters. Most probably these changes would give a lower LB score but they will surely generalize better on unseen data.

", "rawMarkdown": "I think these notebooks are really a good starting point in this competition as they provide a decent base code. One will definitely have to make a few changes to these readily available notebooks like use GroupKFold CV to avoid leakage, use only first Percent value, experiment with other model parameters. Most probably these changes would give a lower LB score but they will surely generalize better on unseen data.", "votes": null }, { "id": "1016179", "postDate": "09/18/2020 18:11:29", "content": "

Agree, but there are some public notebooks with hoping that it's a type of optimization - changing the hyperparameters with same cross validation techinique. It's more for preventing high hopes for these cases and an argument for changing this strategy.

", "rawMarkdown": "Agree, but there are some public notebooks with hoping that it's a type of optimization - changing the hyperparameters with same cross validation techinique. It's more for preventing high hopes for these cases and an argument for changing this strategy.", "votes": null }, { "id": "1016393", "postDate": "09/18/2020 23:04:19", "content": "

@koza4ukdmitrij
\nMy models (only using tabular data) score CV 194-197 MAE with fair CV (last 3 observations), so your CV method with 200 MAE is probably accurate.

", "rawMarkdown": "koza4ukdmitrij \nMy models (only using tabular data) score CV 194-197 MAE with fair CV (last 3 observations), so your CV method with 200 MAE is probably accurate.", "votes": null }, { "id": "1018342", "postDate": "09/19/2020 16:12:28", "content": "

This has been my experience too, my MAE on each fold is 150-200 using tabular data, but these don't give me good Public LB score

", "rawMarkdown": "This has been my experience too, my MAE on each fold is 150-200 using tabular data, but these don't give me good Public LB score", "votes": null }, { "id": "1018427", "postDate": "09/19/2020 17:15:57", "content": "

That's good result, I have about 200. In my opinion, you don't have to take into account LB result, medium result on the LB is OK

", "rawMarkdown": "That's good result, I have about 200. In my opinion, you don't have to take into account LB result, medium result on the LB is OK", "votes": null }, { "id": "1019466", "postDate": "09/20/2020 13:09:42", "content": "

This is an interesting thread. I think that LB score is basically useless in this competition, so it's nice to see what scores others are getting with leakless CV.

\n

BTW would you also care to share what competition metric value are you getting in CV?
\n@koza4ukdmitrij @gautham11 @resistance0108

\n

These are my current results, computed on last 3 observations per patient. To stabilize CV score I'm rerunning each training multiple times with different seed.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
competition metricMAE
single model-6.93192
ensemble-6.90189
\n

I hope to still improve these scores, but I doubt I'll be able to get anywhere near the score which people are getting on public LB. There's a ton of teams with scores around -6.80, but I suppose those are either overfitting, or some random lucky submissions. I wonder how realistic it is to get near -6.80 with stable, leakless CV.

", "rawMarkdown": "This is an interesting thread. I think that LB score is basically useless in this competition, so it's nice to see what scores others are getting with leakless CV.\n\nBTW would you also care to share what competition metric value are you getting in CV?\n@koza4ukdmitrij @gautham11 @resistance0108 \n\nThese are my current results, computed on last 3 observations per patient. To stabilize CV score I'm rerunning each training multiple times with different seed.\n\n| | competition metric | MAE |\n|---|---|\n| single model | -6.93 | 192 |\n| ensemble | -6.90 | 189 |\n\nI hope to still improve these scores, but I doubt I'll be able to get anywhere near the score which people are getting on public LB. There's a ton of teams with scores around -6.80, but I suppose those are either overfitting, or some random lucky submissions. I wonder how realistic it is to get near -6.80 with stable, leakless CV.", "votes": null }, { "id": "1019482", "postDate": "09/20/2020 13:28:41", "content": "

nice

", "rawMarkdown": "nice", "votes": null }, { "id": "1019487", "postDate": "09/20/2020 13:32:10", "content": "

@maciejbudys
\nMy result is here.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
competition metricMAE
single model-6.962-194.4
ensemble-6.944-190.9
\n

Single model score is mean of scores of 3 seeds, ensemble is result of seed averaging.
\nI haven't use image data yet and my teammates scored better CV by adding image data(-6.98ish without image data, -6.913 with image data), so there's lots of room to improve CV for me.
\nDid you use image data?

", "rawMarkdown": "maciejbudys \nMy result is here.\n\n| | competition metric|MAE|\n| --- | --- |---|\n| single model | -6.962 | -194.4|\n| ensemble| -6.944| -190.9|\n\nSingle model score is mean of scores of 3 seeds, ensemble is result of seed averaging.\nI haven't use image data yet and my teammates scored better CV by adding image data(-6.98ish without image data, -6.913 with image data), so there's lots of room to improve CV for me.\nDid you use image data?", "votes": null }, { "id": "1019509", "postDate": "09/20/2020 13:47:33", "content": "

Yes, I have used some features extracted from images.

", "rawMarkdown": "Yes, I have used some features extracted from images.", "votes": null }, { "id": "1019525", "postDate": "09/20/2020 13:59:28", "content": "

I think image data can give CV +0.07~.
\nI think we can reach -6.92 - -6.93 without image data, so maybe around -6.83 is prize line or gold line?

", "rawMarkdown": "I think image data can give CV +0.07~.\nI think we can reach -6.92 - -6.93 without image data, so maybe around -6.83 is prize line or gold line?", "votes": null }, { "id": "1020737", "postDate": "09/21/2020 11:36:02", "content": "

Agree ! I've read from the authors that using all the percent values during training leads to a better LB score. That could explain why so many public notebooks have this leakage. However I'm pretty sure that they just got lucky because public LB set (15% => ~30 patients) is so small

", "rawMarkdown": "Agree ! I've read from the authors that using all the percent values during training leads to a better LB score. That could explain why so many public notebooks have this leakage. However I'm pretty sure that they just got lucky because public LB set (15% => ~30 patients) is so small", "votes": null } ], "comments": [ { "id": 1014832, "author_name": "mavillan", "author_url": "", "post_date": "09/17/2020 18:08:58", "content": "

Aside from not properly splitting data for CV, most of the public solutions use percent as input feature, which is a huge leakage. I don't think these solution will work, as they are conceptually wrong.

", "votes": null, "replies": [ { "id": 1014856, "author_name": "wcukierski", "author_url": "", "post_date": "09/17/2020 18:27:36", "content": "

The percent column is not leakage; it's a function of the initial FVC measurement and patient characteristics.

", "votes": null, "replies": [] }, { "id": 1014863, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/17/2020 18:31:52", "content": "

Agree, that the second reason for leakage. But the first one influence leakage so bigger. For instance, in my experiments with patient leak splitting I get about 60 CV MAE, with patient independent splitting - about 160 CV MAE, but really fair CV MAE (in my opinion) gives about 200 MAE.

", "votes": null, "replies": [] }, { "id": 1014866, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "09/17/2020 18:34:59", "content": "

It's not a leak in the sense that you cannot exploit it on the test set. However, you can build a model, in which you use the Percent in a week to predict the FVC in that week. If you do that, you should be able to get a perfect score. Essentially you just need to teach a model that
\n (FVC in a week) = baseline FVC * Percent in the week / Percent at baseline
\nYou could do a nice cross-validation (never mind if you \"validate\" your model on your training data) and still fool yourself into thinking that you have a great model.

\n

Of course, that will not work on the test set, because you do not have Percent for the weeks you need to predict. So, if a model gives good results on the public leaderboard, if cannot be due to exploiting this (because you cannot exploit it on the leaderboard).

", "votes": null, "replies": [] }, { "id": 1014867, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/17/2020 18:36:40", "content": "

During inference time it's not a leakage, cause you have an access to first measurement row only. But during training, where all measurement are available, that's a leakage on my side, cause model can use Percent feature only, get nice CV score and fails in the inference time then. But if you use only first rows for each patients, it's OK.

", "votes": null, "replies": [] }, { "id": 1014869, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/17/2020 18:38:12", "content": "

Absolutely the same understanding)

", "votes": null, "replies": [] }, { "id": 1014953, "author_name": "mavillan", "author_url": "", "post_date": "09/17/2020 20:18:36", "content": "

@wcukierski I know isn't a leakage itself, but the way is being used for training produces leakage. See my post below.

", "votes": null, "replies": [] }, { "id": 1016393, "author_name": "resistance0108", "author_url": "", "post_date": "09/18/2020 23:04:19", "content": "

@koza4ukdmitrij
\nMy models (only using tabular data) score CV 194-197 MAE with fair CV (last 3 observations), so your CV method with 200 MAE is probably accurate.

", "votes": null, "replies": [] }, { "id": 1018342, "author_name": "gautham11", "author_url": "", "post_date": "09/19/2020 16:12:28", "content": "

This has been my experience too, my MAE on each fold is 150-200 using tabular data, but these don't give me good Public LB score

", "votes": null, "replies": [] }, { "id": 1018427, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/19/2020 17:15:57", "content": "

That's good result, I have about 200. In my opinion, you don't have to take into account LB result, medium result on the LB is OK

", "votes": null, "replies": [] }, { "id": 1019466, "author_name": "maciejbudys", "author_url": "", "post_date": "09/20/2020 13:09:42", "content": "

This is an interesting thread. I think that LB score is basically useless in this competition, so it's nice to see what scores others are getting with leakless CV.

\n

BTW would you also care to share what competition metric value are you getting in CV?
\n@koza4ukdmitrij @gautham11 @resistance0108

\n

These are my current results, computed on last 3 observations per patient. To stabilize CV score I'm rerunning each training multiple times with different seed.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
competition metricMAE
single model-6.93192
ensemble-6.90189
\n

I hope to still improve these scores, but I doubt I'll be able to get anywhere near the score which people are getting on public LB. There's a ton of teams with scores around -6.80, but I suppose those are either overfitting, or some random lucky submissions. I wonder how realistic it is to get near -6.80 with stable, leakless CV.

", "votes": null, "replies": [] }, { "id": 1019487, "author_name": "resistance0108", "author_url": "", "post_date": "09/20/2020 13:32:10", "content": "

@maciejbudys
\nMy result is here.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
competition metricMAE
single model-6.962-194.4
ensemble-6.944-190.9
\n

Single model score is mean of scores of 3 seeds, ensemble is result of seed averaging.
\nI haven't use image data yet and my teammates scored better CV by adding image data(-6.98ish without image data, -6.913 with image data), so there's lots of room to improve CV for me.
\nDid you use image data?

", "votes": null, "replies": [] }, { "id": 1019509, "author_name": "maciejbudys", "author_url": "", "post_date": "09/20/2020 13:47:33", "content": "

Yes, I have used some features extracted from images.

", "votes": null, "replies": [] }, { "id": 1019525, "author_name": "resistance0108", "author_url": "", "post_date": "09/20/2020 13:59:28", "content": "

I think image data can give CV +0.07~.
\nI think we can reach -6.92 - -6.93 without image data, so maybe around -6.83 is prize line or gold line?

", "votes": null, "replies": [] } ] }, { "id": 1014949, "author_name": "mavillan", "author_url": "", "post_date": "09/17/2020 20:17:01", "content": "

These two screenshots are taken from one of the most popular notebooks: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter which scored 6.83X on LB.

\n

Training set:
\n\"\"

\n

Testing set:
\n\"\"

\n

This is the leakage I'm talking about. You will not have the percent feature for each time step for a given patient, only the first value. However, these notebooks are training the model as if the percent value were available at all time steps.

", "votes": null, "replies": [ { "id": 1014958, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/17/2020 20:21:37", "content": "

Agree, so clearly

", "votes": null, "replies": [] }, { "id": 1020737, "author_name": "alexj21", "author_url": "", "post_date": "09/21/2020 11:36:02", "content": "

Agree ! I've read from the authors that using all the percent values during training leads to a better LB score. That could explain why so many public notebooks have this leakage. However I'm pretty sure that they just got lucky because public LB set (15% => ~30 patients) is so small

", "votes": null, "replies": [] } ] }, { "id": 1016135, "author_name": "abhishekgbhat", "author_url": "", "post_date": "09/18/2020 17:29:08", "content": "

I think these notebooks are really a good starting point in this competition as they provide a decent base code. One will definitely have to make a few changes to these readily available notebooks like use GroupKFold CV to avoid leakage, use only first Percent value, experiment with other model parameters. Most probably these changes would give a lower LB score but they will surely generalize better on unseen data.

", "votes": null, "replies": [ { "id": 1016179, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/18/2020 18:11:29", "content": "

Agree, but there are some public notebooks with hoping that it's a type of optimization - changing the hyperparameters with same cross validation techinique. It's more for preventing high hopes for these cases and an argument for changing this strategy.

", "votes": null, "replies": [] } ] }, { "id": 1019482, "author_name": "vijaysimhareddyp", "author_url": "", "post_date": "09/20/2020 13:28:41", "content": "

nice

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1014522": "I want to remind, that public solution doesn't use patient splitting in cross validation, so surprisingly small MAE value were given:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1659719%2Fb2da832811656473e6129474928f19c5%2FScreenshot%20from%202020-09-17%2016-55-06.png?generation=1600351027822753&alt=media)\n\nIt's unbelievable to get validation MAE<100 in this competition. So, you have to understand the next points in this case:\n- cross validation results are absolutely invalid\n- chances are, your model just repeat the last FVC value from train set for every patient\n- in my opinion, improving LB results reason is probably some noise aspects during training or just luck ;)", "1014832": "Aside from not properly splitting data for CV, most of the public solutions use percent as input feature, which is a huge leakage. I don't think these solution will work, as they are conceptually wrong.", "1014856": "The percent column is not leakage; it's a function of the initial FVC measurement and patient characteristics.", "1014863": "Agree, that the second reason for leakage. But the first one influence leakage so bigger. For instance, in my experiments with patient leak splitting I get about 60 CV MAE, with patient independent splitting - about 160 CV MAE, but really fair CV MAE (in my opinion) gives about 200 MAE.", "1014866": "It's not a leak in the sense that you cannot exploit it on the test set. However, you can build a model, in which you use the `Percent` in a week to predict the `FVC` in that week. If you do that, you should be able to get a perfect score. Essentially you just need to teach a model that \n `(FVC in a week) = baseline FVC * Percent in the week / Percent at baseline` \nYou could do a nice cross-validation (never mind if you \"validate\" your model on your training data) and still fool yourself into thinking that you have a great model.\n\nOf course, that will not work on the test set, because you do not have `Percent` for the weeks you need to predict. So, if a model gives good results on the public leaderboard, if cannot be due to exploiting this (because you cannot exploit it on the leaderboard).", "1014867": "During inference time it's not a leakage, cause you have an access to first measurement row only. But during training, where all measurement are available, that's a leakage on my side, cause model can use Percent feature only, get nice CV score and fails in the inference time then. But if you use only first rows for each patients, it's OK.", "1014869": "Absolutely the same understanding)", "1014949": "These two screenshots are taken from one of the most popular notebooks: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter which scored 6.83X on LB. \n\nTraining set:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F350756%2F1fff4b5d57cb4971385d100134c74ce1%2FScreen%20Shot%202020-09-17%20at%2015.08.57.png?generation=1600373457125816&alt=media)\n\nTesting set:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F350756%2F26f2a06b39a88524c2dbaae60d9876fd%2FScreen%20Shot%202020-09-17%20at%2015.09.20.png?generation=1600373481936507&alt=media)\n\nThis is the leakage I'm talking about. You will not have the percent feature for each time step for a given patient, only the first value. However, these notebooks are training the model as if the percent value were available at all time steps.", "1014953": "wcukierski I know isn't a leakage itself, but the way is being used for training produces leakage. See my post below.", "1014958": "Agree, so clearly", "1016135": "I think these notebooks are really a good starting point in this competition as they provide a decent base code. One will definitely have to make a few changes to these readily available notebooks like use GroupKFold CV to avoid leakage, use only first Percent value, experiment with other model parameters. Most probably these changes would give a lower LB score but they will surely generalize better on unseen data.", "1016179": "Agree, but there are some public notebooks with hoping that it's a type of optimization - changing the hyperparameters with same cross validation techinique. It's more for preventing high hopes for these cases and an argument for changing this strategy.", "1016393": "koza4ukdmitrij \nMy models (only using tabular data) score CV 194-197 MAE with fair CV (last 3 observations), so your CV method with 200 MAE is probably accurate.", "1018342": "This has been my experience too, my MAE on each fold is 150-200 using tabular data, but these don't give me good Public LB score", "1018427": "That's good result, I have about 200. In my opinion, you don't have to take into account LB result, medium result on the LB is OK", "1019466": "This is an interesting thread. I think that LB score is basically useless in this competition, so it's nice to see what scores others are getting with leakless CV.\n\nBTW would you also care to share what competition metric value are you getting in CV?\n@koza4ukdmitrij @gautham11 @resistance0108 \n\nThese are my current results, computed on last 3 observations per patient. To stabilize CV score I'm rerunning each training multiple times with different seed.\n\n| | competition metric | MAE |\n|---|---|\n| single model | -6.93 | 192 |\n| ensemble | -6.90 | 189 |\n\nI hope to still improve these scores, but I doubt I'll be able to get anywhere near the score which people are getting on public LB. There's a ton of teams with scores around -6.80, but I suppose those are either overfitting, or some random lucky submissions. I wonder how realistic it is to get near -6.80 with stable, leakless CV.", "1019482": "nice", "1019487": "maciejbudys \nMy result is here.\n\n| | competition metric|MAE|\n| --- | --- |---|\n| single model | -6.962 | -194.4|\n| ensemble| -6.944| -190.9|\n\nSingle model score is mean of scores of 3 seeds, ensemble is result of seed averaging.\nI haven't use image data yet and my teammates scored better CV by adding image data(-6.98ish without image data, -6.913 with image data), so there's lots of room to improve CV for me.\nDid you use image data?", "1019509": "Yes, I have used some features extracted from images.", "1019525": "I think image data can give CV +0.07~.\nI think we can reach -6.92 - -6.93 without image data, so maybe around -6.83 is prize line or gold line?", "1020737": "Agree ! I've read from the authors that using all the percent values during training leads to a better LB score. That could explain why so many public notebooks have this leakage. However I'm pretty sure that they just got lucky because public LB set (15% => ~30 patients) is so small" }, "source": "meta" }