![\"https://raw.githubusercontent.com/nyanp/mlb-player-digital-engagement/main/docs/img/time%20series%20gap.png\"](\"https://raw.githubusercontent.com/nyanp/mlb-player-digital-engagement/main/docs/img/time%20series%20gap.png\")
2021-09-10 updated
\nThe final ranking is 3rd! Thank you all :) My source code is a bit messy, but you can see it here: https://github.com/nyanp/mlb-player-digital-engagement and https://www.kaggle.com/nyanpn/3rd-place-solution-inference-only .
First of all, I would like to thank the hosts for organizing this competition.
\nIt was a very tough competition but I learned a lot from it.
I have no idea what my final standings will be, but I will share with you what I did.
\nLag features are very useful in this competition, but we cannot use target information in the test data period. So we need to set an appropriate \"gap\" between the prediction date and the lag feature data.
\nFor example, if you want to make a prediction for 8/4,
\nyou need to create a gap of at least 3 days because you cannot use the target information from 8/1 to 8/3.
I divided the test data into 8 periods and used different models with different gaps in each period (gap: 0, 3, 7, 14, 21, 28, 35 ,45days).
\n
The three models were ensembled with different weights for each target.
\n1DCNN is the same as the 2nd place solution of MoA Competition.
\nTime based split.
\nBasically, I only adopted ideas that improved the score in all periods.
\nI used ~440 features. In addition to joins and asof merge of basic tables, the following features were used:
\nThere is a strange correlation between the cumcount of the dataframe retrieved from the Time-Series API and the target.
\nI noticed this problem 3 days before the competition ended. I did not post it in the discussion as it might confuse the participants, but contacted the host immediately.
\nAdding this cumcount to the features only improves the CV a little bit, so it's probably some kind of artifact or something, but even if it doesn't improve the CV much, it's better to shuffle the test data since it's nonsense that the order of the rows makes sense.
I did not end up using this leak for final submission.
\nBuilding a complex data pipeline in Jupyter Notebook with the Time Series API can be a big pain. I'll share some of my efforts.
\nThanks to all of this, I was able to finish the competition 1st stage without making a single submission error.
", "messages": [ { "id": "1408712", "postDate": "08/02/2021 15:16:17", "content": "2021-09-10 updated
\nThe final ranking is 3rd! Thank you all :) My source code is a bit messy, but you can see it here: https://github.com/nyanp/mlb-player-digital-engagement and https://www.kaggle.com/nyanpn/3rd-place-solution-inference-only .
First of all, I would like to thank the hosts for organizing this competition.
\nIt was a very tough competition but I learned a lot from it.
I have no idea what my final standings will be, but I will share with you what I did.
\nLag features are very useful in this competition, but we cannot use target information in the test data period. So we need to set an appropriate \"gap\" between the prediction date and the lag feature data.
\nFor example, if you want to make a prediction for 8/4,
\nyou need to create a gap of at least 3 days because you cannot use the target information from 8/1 to 8/3.
I divided the test data into 8 periods and used different models with different gaps in each period (gap: 0, 3, 7, 14, 21, 28, 35 ,45days).
\n
The three models were ensembled with different weights for each target.
\n1DCNN is the same as the 2nd place solution of MoA Competition.
\nTime based split.
\nBasically, I only adopted ideas that improved the score in all periods.
\nI used ~440 features. In addition to joins and asof merge of basic tables, the following features were used:
\nThere is a strange correlation between the cumcount of the dataframe retrieved from the Time-Series API and the target.
\nI noticed this problem 3 days before the competition ended. I did not post it in the discussion as it might confuse the participants, but contacted the host immediately.
\nAdding this cumcount to the features only improves the CV a little bit, so it's probably some kind of artifact or something, but even if it doesn't improve the CV much, it's better to shuffle the test data since it's nonsense that the order of the rows makes sense.
I did not end up using this leak for final submission.
\nBuilding a complex data pipeline in Jupyter Notebook with the Time Series API can be a big pain. I'll share some of my efforts.
\nThanks to all of this, I was able to finish the competition 1st stage without making a single submission error.
", "rawMarkdown": "***2021-09-10 updated***\nThe final ranking is 3rd! Thank you all :) My source code is a bit messy, but you can see it here: https://github.com/nyanp/mlb-player-digital-engagement and https://www.kaggle.com/nyanpn/3rd-place-solution-inference-only .\n\n---\n\nFirst of all, I would like to thank the hosts for organizing this competition.\nIt was a very tough competition but I learned a lot from it.\n\nI have no idea what my final standings will be, but I will share with you what I did.\n\n## Model strategy\nLag features are very useful in this competition, but we cannot use target information in the test data period. So we need to set an appropriate \"gap\" between the prediction date and the lag feature data.\n\nFor example, if you want to make a prediction for 8/4, \nyou need to create a gap of at least 3 days because you cannot use the target information from 8/1 to 8/3.\n\nI divided the test data into 8 periods and used different models with different gaps in each period (gap: 0, 3, 7, 14, 21, 28, 35 ,45days).\n\n\n\n## Model\nThe three models were ensembled with different weights for each target.\n\n- LightGBM\n- MLP\n- 1DCNN\n\n1DCNN is the same as [the 2nd place solution of MoA Competition](https://www.kaggle.com/c/lish-moa/discussion/202256).\n\n### Validation\nTime based split.\n- validation before update: 2019/8, 2020/8, 2021/4\n- validation after update: 2020/8, 2021/6, 2021/7\n\nBasically, I only adopted ideas that improved the score in all periods.\n\n## Features\nI used ~440 features. In addition to joins and asof merge of basic tables, the following features were used:\n\n- lag features per player\n - Average of the last 7/28/70/360/720 days\n - Average over the on-seasons\n - Average for the same period in the previous year\n - Average of days with/without a game\n- number of events, pitch events, action events\n- days from last rosters, awards, transactions and box scores\n- sum of box scores in the last 7/30/90 days\n- number of games and events in the day\n- event-level meta feature\n - aggregation of predictions of model trained on event table\n - group by (date, playerId), (date, teamId) and (date)\n\n### Cumcount Leakage\nThere is a strange correlation between the cumcount of the dataframe retrieved from the Time-Series API and the target.\n\nI noticed this problem 3 days before the competition ended. I did not post it in the discussion as it might confuse the participants, but contacted the host immediately.\nAdding this cumcount to the features only improves the CV a little bit, so it's probably some kind of artifact or something, but even if it doesn't improve the CV much, it's better to shuffle the test data since it's nonsense that the order of the rows makes sense.\n\nI did not end up using this leak for final submission.\n\n### Implementation Note\nBuilding a complex data pipeline in Jupyter Notebook with the Time Series API can be a big pain. I'll share some of my efforts.\n\n- Maintain the source code on GitHub and paste the BASE64-encoded code into the jupyter notebook\n - see: https://github.com/lopuhin/kaggle-imet-2019\n- The inference notebook is also maintained on GitHub and automatically uploaded as the Kaggle Kernel through GitHub Actions\n- Avoid the use of pandas and instead use a dictionary of numpy arrays to manage state updates\n- Use the same feature generation function for training data and inference\n - Both training and test are treated as streaming data, and features were generated using for-loop.\n - This is the most important point to get a stable and bug-free data pipeline\n - see: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/196942\n- Debug code locally using the API emulator\n - Test the robustness of my inference pipeline by \"dropout\" some of the data returned by the emulator (a kind of \"Chaos Engineering\")\n- Catch exceptions in various functions and convert them to appropriate \"default\" values\n\nThanks to all of this, I was able to finish the competition 1st stage without making a single submission error.", "votes": null }, { "id": "1408746", "postDate": "08/02/2021 15:28:05", "content": "I’ve shared the notebook about cumcount leakage.
\nhttps://www.kaggle.com/nyanpn/cumcount-leakage
I also found that the 1D CNN that you mentioned worked pretty well for me, although it didn't make it into my final submission. Thanks for the writeup!
", "rawMarkdown": "I also found that the 1D CNN that you mentioned worked pretty well for me, although it didn't make it into my final submission. Thanks for the writeup!", "votes": null }, { "id": "1502702", "postDate": "09/04/2021 14:53:40", "content": "Hello, how do you deal with nan problem with this 1D CNN?
", "rawMarkdown": "Hello, how do you deal with nan problem with this 1D CNN?", "votes": null }, { "id": "1508189", "postDate": "09/10/2021 01:41:12", "content": "Awesome, great job @nyanpn . Congrats on solo gold 3rd place!
", "rawMarkdown": "Awesome, great job @nyanpn . Congrats on solo gold 3rd place!", "votes": null }, { "id": "1508741", "postDate": "09/10/2021 14:16:41", "content": "Congratulations @nyanpn . Use of github actions to streamline inference pipeline is really creative.👍
", "rawMarkdown": "Congratulations @nyanpn . Use of github actions to streamline inference pipeline is really creative.👍", "votes": null } ], "comments": [ { "id": 1408746, "author_name": "nyanpn", "author_url": "", "post_date": "08/02/2021 15:28:05", "content": "I’ve shared the notebook about cumcount leakage.
\nhttps://www.kaggle.com/nyanpn/cumcount-leakage
I also found that the 1D CNN that you mentioned worked pretty well for me, although it didn't make it into my final submission. Thanks for the writeup!
", "votes": null, "replies": [ { "id": 1502702, "author_name": "yuanzhezhou", "author_url": "", "post_date": "09/04/2021 14:53:40", "content": "Hello, how do you deal with nan problem with this 1D CNN?
", "votes": null, "replies": [] } ] }, { "id": 1508189, "author_name": "cdeotte", "author_url": "", "post_date": "09/10/2021 01:41:12", "content": "Awesome, great job @nyanpn . Congrats on solo gold 3rd place!
", "votes": null, "replies": [] }, { "id": 1508741, "author_name": "tezdhar", "author_url": "", "post_date": "09/10/2021 14:16:41", "content": "Congratulations @nyanpn . Use of github actions to streamline inference pipeline is really creative.👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1408712": "***2021-09-10 updated***\nThe final ranking is 3rd! Thank you all :) My source code is a bit messy, but you can see it here: https://github.com/nyanp/mlb-player-digital-engagement and https://www.kaggle.com/nyanpn/3rd-place-solution-inference-only .\n\n---\n\nFirst of all, I would like to thank the hosts for organizing this competition.\nIt was a very tough competition but I learned a lot from it.\n\nI have no idea what my final standings will be, but I will share with you what I did.\n\n## Model strategy\nLag features are very useful in this competition, but we cannot use target information in the test data period. So we need to set an appropriate \"gap\" between the prediction date and the lag feature data.\n\nFor example, if you want to make a prediction for 8/4, \nyou need to create a gap of at least 3 days because you cannot use the target information from 8/1 to 8/3.\n\nI divided the test data into 8 periods and used different models with different gaps in each period (gap: 0, 3, 7, 14, 21, 28, 35 ,45days).\n\n\n\n## Model\nThe three models were ensembled with different weights for each target.\n\n- LightGBM\n- MLP\n- 1DCNN\n\n1DCNN is the same as [the 2nd place solution of MoA Competition](https://www.kaggle.com/c/lish-moa/discussion/202256).\n\n### Validation\nTime based split.\n- validation before update: 2019/8, 2020/8, 2021/4\n- validation after update: 2020/8, 2021/6, 2021/7\n\nBasically, I only adopted ideas that improved the score in all periods.\n\n## Features\nI used ~440 features. In addition to joins and asof merge of basic tables, the following features were used:\n\n- lag features per player\n - Average of the last 7/28/70/360/720 days\n - Average over the on-seasons\n - Average for the same period in the previous year\n - Average of days with/without a game\n- number of events, pitch events, action events\n- days from last rosters, awards, transactions and box scores\n- sum of box scores in the last 7/30/90 days\n- number of games and events in the day\n- event-level meta feature\n - aggregation of predictions of model trained on event table\n - group by (date, playerId), (date, teamId) and (date)\n\n### Cumcount Leakage\nThere is a strange correlation between the cumcount of the dataframe retrieved from the Time-Series API and the target.\n\nI noticed this problem 3 days before the competition ended. I did not post it in the discussion as it might confuse the participants, but contacted the host immediately.\nAdding this cumcount to the features only improves the CV a little bit, so it's probably some kind of artifact or something, but even if it doesn't improve the CV much, it's better to shuffle the test data since it's nonsense that the order of the rows makes sense.\n\nI did not end up using this leak for final submission.\n\n### Implementation Note\nBuilding a complex data pipeline in Jupyter Notebook with the Time Series API can be a big pain. I'll share some of my efforts.\n\n- Maintain the source code on GitHub and paste the BASE64-encoded code into the jupyter notebook\n - see: https://github.com/lopuhin/kaggle-imet-2019\n- The inference notebook is also maintained on GitHub and automatically uploaded as the Kaggle Kernel through GitHub Actions\n- Avoid the use of pandas and instead use a dictionary of numpy arrays to manage state updates\n- Use the same feature generation function for training data and inference\n - Both training and test are treated as streaming data, and features were generated using for-loop.\n - This is the most important point to get a stable and bug-free data pipeline\n - see: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/196942\n- Debug code locally using the API emulator\n - Test the robustness of my inference pipeline by \"dropout\" some of the data returned by the emulator (a kind of \"Chaos Engineering\")\n- Catch exceptions in various functions and convert them to appropriate \"default\" values\n\nThanks to all of this, I was able to finish the competition 1st stage without making a single submission error.", "1408746": "I’ve shared the notebook about cumcount leakage.\nhttps://www.kaggle.com/nyanpn/cumcount-leakage", "1408755": "I also found that the 1D CNN that you mentioned worked pretty well for me, although it didn't make it into my final submission. Thanks for the writeup!", "1502702": "Hello, how do you deal with nan problem with this 1D CNN?", "1508189": "Awesome, great job @nyanpn . Congrats on solo gold 3rd place!", "1508741": "Congratulations @nyanpn . Use of github actions to streamline inference pipeline is really creative.👍" }, "source": "meta" }