I've encountered two issues:
\nThis competition adopt a special evaluation metric focused on stability, which might be the reason, but the Public LB score is quite unstable.
\nFor instance, even when the local CV score increases, the Public LB score often decreases, showing no strong positive correlation.
\nMinor parameter changes can also lead to significant fluctuations in the Public LB.
Additionally, submitting the exact same code can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570.
\nThis might be due to the lack of reproducibility when using GPU for calculations in LightGBM, but regardless, it indicates that the Public LB is unstable.
I have no idea how to address this issue.
\nMany public notebooks simply split the data into 5-fold without considering the time-series.
\nThis can lead to evaluating past data using future data, which generally isn't appropriate due to the leakage problem.
\nIdeally, future data should be used as the validation set, and past data as the training set.
\nI tried this approach, but the Public LB score unchanged, so I was confused.
For these reasons, I'm uncertain about how to proceed in this competition.
\nI think it's risky that we can't find a correlation between the public LB and local CV, which means
\nthat many of the currently high-scoring public notebooks might be overfitting to the public LB and could shake for Private LB.
Does anyone have any opinion?
", "messages": [ { "id": "2729018", "postDate": "04/02/2024 14:12:36", "content": "I've encountered two issues:
\nThis competition adopt a special evaluation metric focused on stability, which might be the reason, but the Public LB score is quite unstable.
\nFor instance, even when the local CV score increases, the Public LB score often decreases, showing no strong positive correlation.
\nMinor parameter changes can also lead to significant fluctuations in the Public LB.
Additionally, submitting the exact same code can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570.
\nThis might be due to the lack of reproducibility when using GPU for calculations in LightGBM, but regardless, it indicates that the Public LB is unstable.
I have no idea how to address this issue.
\nMany public notebooks simply split the data into 5-fold without considering the time-series.
\nThis can lead to evaluating past data using future data, which generally isn't appropriate due to the leakage problem.
\nIdeally, future data should be used as the validation set, and past data as the training set.
\nI tried this approach, but the Public LB score unchanged, so I was confused.
For these reasons, I'm uncertain about how to proceed in this competition.
\nI think it's risky that we can't find a correlation between the public LB and local CV, which means
\nthat many of the currently high-scoring public notebooks might be overfitting to the public LB and could shake for Private LB.
Does anyone have any opinion?
", "rawMarkdown": "I've encountered two issues:\n\n## 1. The Public LB is unstable, and no correlation with local CV can be found.\nThis competition adopt a special evaluation metric focused on stability, which might be the reason, but the Public LB score is quite unstable.\n\nFor instance, even when the local CV score increases, the Public LB score often decreases, showing no strong positive correlation. \nMinor parameter changes can also lead to significant fluctuations in the Public LB.\n\nAdditionally, submitting **the exact same code** can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570. \nThis might be due to the lack of reproducibility when using GPU for calculations in LightGBM, but regardless, it indicates that the Public LB is unstable.\n\nI have no idea how to address this issue.\n\n## 2. Should Cross Validation consider time series data?\n\nMany public notebooks simply split the data into 5-fold without considering the time-series.\nThis can lead to evaluating past data using future data, which generally isn't appropriate due to the leakage problem. \nIdeally, future data should be used as the validation set, and past data as the training set.\nI tried this approach, but the Public LB score unchanged, so I was confused.\n\n## Summary\nFor these reasons, I'm uncertain about how to proceed in this competition. \nI think it's risky that we can't find a correlation between the public LB and local CV, which means\nthat many of the currently high-scoring public notebooks might be overfitting to the public LB and could shake for Private LB.\n\nDoes anyone have any opinion?", "votes": null }, { "id": "2729076", "postDate": "04/02/2024 14:38:49", "content": "@tetsuro731 I totally agree with this too.
\nI also accidently did your experiment lately and have experienced code result fluctuations in the same range. Is everyone's public leaderboard the same?
\nI also concur on the shakeup potential and CV strategy too. I think building a good CV is of prime importance here and I doubt if this stratified grouped K-fold lives up to the task. Perhaps this thwarts past future surrogacy and this could be a problem in the private evaluation.
@tetsuro731 Due to the fact that I could not find a correlation between the leaderboard and local tests, I lost interest in the task.
\nI come in periodically to see if anyone has a solution.
Due to the fact that we do not understand what the test sample represents, perhaps everyone is standing still.
\nIt’s not interesting to come up with new cool features when they locally give an increase, but not lb
", "rawMarkdown": "tetsuro731 Due to the fact that I could not find a correlation between the leaderboard and local tests, I lost interest in the task.\nI come in periodically to see if anyone has a solution.\n\nDue to the fact that we do not understand what the test sample represents, perhaps everyone is standing still.\n\nIt’s not interesting to come up with new cool features when they locally give an increase, but not lb", "votes": null }, { "id": "2750193", "postDate": "04/13/2024 14:04:09", "content": "\n\nIt’s not interesting to come up with new cool features when they locally give an increase, but not lb
\n
That totally makes sense to me, but I'd like to discover and figure out better approach for this competition.
", "rawMarkdown": ">It’s not interesting to come up with new cool features when they locally give an increase, but not lb\n\nThat totally makes sense to me, but I'd like to discover and figure out better approach for this competition.", "votes": null }, { "id": "2750194", "postDate": "04/13/2024 14:06:53", "content": "\n\nIs everyone's public leaderboard the same?
\n
Good point.
\nI guess there is fluctuation even if using exactly the same code, which could be the problem.
\nOn the other hand, you know, final score will be calculated by the other private test data so creating robust model could be the better way to win the competition in my understanding.
Hi,
\nsplit public - private leaderboard is fixed and same for every Kaggler. So difference in LB score is only caused by different model, not data.
\n\nAdditionally, submitting the exact same code can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570.
\n
I've also noticed this issue. I think this is caused by the randomness of the GPU version of CatBoost. I am now starting to save the model each time it runs for future use.
", "rawMarkdown": ">Additionally, submitting the exact same code can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570.\n\nI've also noticed this issue. I think this is caused by the randomness of the GPU version of CatBoost. I am now starting to save the model each time it runs for future use.", "votes": null }, { "id": "2753675", "postDate": "04/15/2024 16:00:16", "content": "That is exactly what is this competition about, a way how to create new features that would be able to take into account instabilities and the same for a model.
", "rawMarkdown": "That is exactly what is this competition about, a way how to create new features that would be able to take into account instabilities and the same for a model.", "votes": null }, { "id": "2756105", "postDate": "04/16/2024 21:29:53", "content": "I think using a time series split may give more accurate loss and and metric estimates but at the same time you're not using the most up to date data for training (just validation), which could be a detriment when testing on LB data. I wonder if it's better to use time series splitting for hyperparameter tuning and stratified k-fold for actual training for LB.
", "rawMarkdown": "I think using a time series split may give more accurate loss and and metric estimates but at the same time you're not using the most up to date data for training (just validation), which could be a detriment when testing on LB data. I wonder if it's better to use time series splitting for hyperparameter tuning and stratified k-fold for actual training for LB.", "votes": null } ], "comments": [ { "id": 2729076, "author_name": "ravi20076", "author_url": "", "post_date": "04/02/2024 14:38:49", "content": "@tetsuro731 I totally agree with this too.
\nI also accidently did your experiment lately and have experienced code result fluctuations in the same range. Is everyone's public leaderboard the same?
\nI also concur on the shakeup potential and CV strategy too. I think building a good CV is of prime importance here and I doubt if this stratified grouped K-fold lives up to the task. Perhaps this thwarts past future surrogacy and this could be a problem in the private evaluation.
\n\nIs everyone's public leaderboard the same?
\n
Good point.
\nI guess there is fluctuation even if using exactly the same code, which could be the problem.
\nOn the other hand, you know, final score will be calculated by the other private test data so creating robust model could be the better way to win the competition in my understanding.
Hi,
\nsplit public - private leaderboard is fixed and same for every Kaggler. So difference in LB score is only caused by different model, not data.
@tetsuro731 Due to the fact that I could not find a correlation between the leaderboard and local tests, I lost interest in the task.
\nI come in periodically to see if anyone has a solution.
Due to the fact that we do not understand what the test sample represents, perhaps everyone is standing still.
\nIt’s not interesting to come up with new cool features when they locally give an increase, but not lb
", "votes": null, "replies": [ { "id": 2750193, "author_name": "tetsuro731", "author_url": "", "post_date": "04/13/2024 14:04:09", "content": "\n\nIt’s not interesting to come up with new cool features when they locally give an increase, but not lb
\n
That totally makes sense to me, but I'd like to discover and figure out better approach for this competition.
", "votes": null, "replies": [ { "id": 2753675, "author_name": "jetakow", "author_url": "", "post_date": "04/15/2024 16:00:16", "content": "That is exactly what is this competition about, a way how to create new features that would be able to take into account instabilities and the same for a model.
", "votes": null, "replies": [] } ] } ] }, { "id": 2753406, "author_name": "bruceqdu", "author_url": "", "post_date": "04/15/2024 13:23:41", "content": "\n\nAdditionally, submitting the exact same code can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570.
\n
I've also noticed this issue. I think this is caused by the randomness of the GPU version of CatBoost. I am now starting to save the model each time it runs for future use.
", "votes": null, "replies": [] }, { "id": 2756105, "author_name": "caelhasse", "author_url": "", "post_date": "04/16/2024 21:29:53", "content": "I think using a time series split may give more accurate loss and and metric estimates but at the same time you're not using the most up to date data for training (just validation), which could be a detriment when testing on LB data. I wonder if it's better to use time series splitting for hyperparameter tuning and stratified k-fold for actual training for LB.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2729018": "I've encountered two issues:\n\n## 1. The Public LB is unstable, and no correlation with local CV can be found.\nThis competition adopt a special evaluation metric focused on stability, which might be the reason, but the Public LB score is quite unstable.\n\nFor instance, even when the local CV score increases, the Public LB score often decreases, showing no strong positive correlation. \nMinor parameter changes can also lead to significant fluctuations in the Public LB.\n\nAdditionally, submitting **the exact same code** can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570. \nThis might be due to the lack of reproducibility when using GPU for calculations in LightGBM, but regardless, it indicates that the Public LB is unstable.\n\nI have no idea how to address this issue.\n\n## 2. Should Cross Validation consider time series data?\n\nMany public notebooks simply split the data into 5-fold without considering the time-series.\nThis can lead to evaluating past data using future data, which generally isn't appropriate due to the leakage problem. \nIdeally, future data should be used as the validation set, and past data as the training set.\nI tried this approach, but the Public LB score unchanged, so I was confused.\n\n## Summary\nFor these reasons, I'm uncertain about how to proceed in this competition. \nI think it's risky that we can't find a correlation between the public LB and local CV, which means\nthat many of the currently high-scoring public notebooks might be overfitting to the public LB and could shake for Private LB.\n\nDoes anyone have any opinion?", "2729076": "tetsuro731 I totally agree with this too.\n\nI also accidently did your experiment lately and have experienced code result fluctuations in the same range. **Is everyone's public leaderboard the same?** \nI also concur on the shakeup potential and CV strategy too. I think building a good CV is of prime importance here and I doubt if this stratified grouped K-fold lives up to the task. Perhaps this thwarts past future surrogacy and this could be a problem in the private evaluation.", "2729255": "tetsuro731 Due to the fact that I could not find a correlation between the leaderboard and local tests, I lost interest in the task.\nI come in periodically to see if anyone has a solution.\n\nDue to the fact that we do not understand what the test sample represents, perhaps everyone is standing still.\n\nIt’s not interesting to come up with new cool features when they locally give an increase, but not lb", "2750193": ">It’s not interesting to come up with new cool features when they locally give an increase, but not lb\n\nThat totally makes sense to me, but I'd like to discover and figure out better approach for this competition.", "2750194": ">Is everyone's public leaderboard the same?\n\nGood point.\nI guess there is fluctuation even if using exactly the same code, which could be the problem.\nOn the other hand, you know, final score will be calculated by the other private test data so creating robust model could be the better way to win the competition in my understanding.", "2752894": "Hi,\nsplit public - private leaderboard is fixed and same for every Kaggler. So difference in LB score is only caused by different model, not data.", "2753406": ">Additionally, submitting the exact same code can result in varying scores. I've submitted a publicl notebook as a baseline multiple times and observed variations from 0.568 to 0.570.\n\nI've also noticed this issue. I think this is caused by the randomness of the GPU version of CatBoost. I am now starting to save the model each time it runs for future use.", "2753675": "That is exactly what is this competition about, a way how to create new features that would be able to take into account instabilities and the same for a model.", "2756105": "I think using a time series split may give more accurate loss and and metric estimates but at the same time you're not using the most up to date data for training (just validation), which could be a detriment when testing on LB data. I wonder if it's better to use time series splitting for hyperparameter tuning and stratified k-fold for actual training for LB." }, "source": "meta" }