Hello MLB participants,
\nWe have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.
\nApproximately half of the candidate rerun submissions did not succeed. A large failure rate during a rerun can happen for any number of reasons, and even involve a combination of reasons. Most commonly:
\nWe have performed an initial investigation into the potential error causes here. The failures appear to largely stem from widespread forking of a notebook that used lagged target values, yet failed to handle the case where the train and test set are not adjacent in time. We provided train_updated.csv and messaged its existence expressly for this use case. The second stage was additionally clearly described as occurring months after the end of train.csv from the onset of the competition. As such, we will not be making accommodations to fix notebooks affected by this issue.
On the investigation, I would like to clarify two important points:
\nWe know this is frustrating for those whose notebooks did not succeed. Thank you for your participation and see you for the next rerun, the week of August 23.
", "messages": [ { "id": "1467211", "postDate": "08/11/2021 21:06:37", "content": "Hello MLB participants,
\nWe have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.
\nApproximately half of the candidate rerun submissions did not succeed. A large failure rate during a rerun can happen for any number of reasons, and even involve a combination of reasons. Most commonly:
\nWe have performed an initial investigation into the potential error causes here. The failures appear to largely stem from widespread forking of a notebook that used lagged target values, yet failed to handle the case where the train and test set are not adjacent in time. We provided train_updated.csv and messaged its existence expressly for this use case. The second stage was additionally clearly described as occurring months after the end of train.csv from the onset of the competition. As such, we will not be making accommodations to fix notebooks affected by this issue.
On the investigation, I would like to clarify two important points:
\nWe know this is frustrating for those whose notebooks did not succeed. Thank you for your participation and see you for the next rerun, the week of August 23.
", "rawMarkdown": "Hello MLB participants,\n\nWe have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.\n\nApproximately half of the candidate rerun submissions did not succeed. A large failure rate during a rerun can happen for any number of reasons, and even involve a combination of reasons. Most commonly:\n\n - new data that does not conform to the structure of the first-stage data\n - a bug in the pipelines that created the data\n - mass adoption of a flawed notebook or dataset\n\nWe have performed an initial investigation into the potential error causes here. The failures appear to largely stem from widespread forking of a notebook that used lagged target values, yet failed to handle the case where the train and test set are not adjacent in time. We provided `train_updated.csv` and messaged its existence expressly for this use case. The second stage was additionally clearly described as occurring months after the end of `train.csv` from the onset of the competition. As such, we will not be making accommodations to fix notebooks affected by this issue.\n\nOn the investigation, I would like to clarify two important points:\n\n- I describe the investigation as \"initial\" because we have more reruns to go and there is always a chance we discover other problems before the final rerun. At this time we do not have reason to believe the data is nonconforming or that we have introduced any new bugs when creating the datasets.\n- We have had discussions in previous reruns (prior this competition) about what level of error constitutes fair grounds for intervention. While we ultimately own the decision to intervene or not, we conduct these reviews in the most neutral manner possible. We do not consider the hypothetical impacts on rankings nor do we examine who is affected by any resulting actions.\n\nWe know this is frustrating for those whose notebooks did not succeed. Thank you for your participation and see you for the next rerun, the week of August 23.", "votes": null }, { "id": "1467360", "postDate": "08/12/2021 00:17:58", "content": "Good luck to all participants 🙏
", "rawMarkdown": "Good luck to all participants 🙏", "votes": null }, { "id": "1467405", "postDate": "08/12/2021 01:04:22", "content": "For competition medals, I am assuming the number of teams that participated (not succeeded) is what influences the number of medals handed out? https://www.kaggle.com/progression
", "rawMarkdown": "For competition medals, I am assuming the number of teams that participated (not succeeded) is what influences the number of medals handed out? https://www.kaggle.com/progression", "votes": null }, { "id": "1467407", "postDate": "08/12/2021 01:06:30", "content": "Correct. 852 is the \"official\" team count for points/medals.
", "rawMarkdown": "Correct. 852 is the \"official\" team count for points/medals.", "votes": null }, { "id": "1467858", "postDate": "08/12/2021 06:56:32", "content": "Tough love wins again
", "rawMarkdown": "Tough love wins again", "votes": null }, { "id": "1471188", "postDate": "08/14/2021 03:02:48", "content": "Thanks for the info.
\nSo the next rerun will start from 08-07 to 08-23 or from 08-01 to 08-23.
Good luck to all remaining participants
\nAlmost half have been left out after the first run, as I notice 438 in the LB vs 852 participants!!
While a bug in the pipelines that created the data & mass adoption of a flawed notebook or dataset are in some ways the participants fault
\nnew data that does not conform to the structure of the first-stage data -- that is a concern, no?
\n", "rawMarkdown": "> At this time we do not have reason to believe the data is nonconforming", "votes": null }, { "id": "1475271", "postDate": "08/16/2021 14:44:56", "content": "At this time we do not have reason to believe the data is nonconforming
\n
thanks for the update @wcukierski
\nhighlighted as you have noted it as the first reason in your post
\n\nWe have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.
\n
Three swings and three misses.
\nMy best submission struck out.
", "rawMarkdown": "kmldas \n\n> We have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.\n\nThree swings and three misses.\n\nMy best submission struck out.", "votes": null }, { "id": "1482907", "postDate": "08/20/2021 10:47:52", "content": "Can we use submision system for the dataset for evaluation phase after this competition?
\nOr can you publish the dataset for evaluation phase after this competition?
I want to use it for my study.
", "rawMarkdown": "Can we use submision system for the dataset for evaluation phase after this competition?\nOr can you publish the dataset for evaluation phase after this competition?\n\nI want to use it for my study.", "votes": null } ], "comments": [ { "id": 1467360, "author_name": "iniestamoh", "author_url": "", "post_date": "08/12/2021 00:17:58", "content": "Good luck to all participants 🙏
", "votes": null, "replies": [] }, { "id": 1467405, "author_name": "kaito510", "author_url": "", "post_date": "08/12/2021 01:04:22", "content": "For competition medals, I am assuming the number of teams that participated (not succeeded) is what influences the number of medals handed out? https://www.kaggle.com/progression
", "votes": null, "replies": [ { "id": 1467407, "author_name": "wcukierski", "author_url": "", "post_date": "08/12/2021 01:06:30", "content": "Correct. 852 is the \"official\" team count for points/medals.
", "votes": null, "replies": [] } ] }, { "id": 1467858, "author_name": "pcjimmmy", "author_url": "", "post_date": "08/12/2021 06:56:32", "content": "Tough love wins again
", "votes": null, "replies": [] }, { "id": 1471188, "author_name": "fangyu67", "author_url": "", "post_date": "08/14/2021 03:02:48", "content": "Thanks for the info.
\nSo the next rerun will start from 08-07 to 08-23 or from 08-01 to 08-23.
Good luck to all remaining participants
\nAlmost half have been left out after the first run, as I notice 438 in the LB vs 852 participants!!
While a bug in the pipelines that created the data & mass adoption of a flawed notebook or dataset are in some ways the participants fault
\nnew data that does not conform to the structure of the first-stage data -- that is a concern, no?
\n", "votes": null, "replies": [] }, { "id": 1475271, "author_name": "kmldas", "author_url": "", "post_date": "08/16/2021 14:44:56", "content": "At this time we do not have reason to believe the data is nonconforming
\n
thanks for the update @wcukierski
\nhighlighted as you have noted it as the first reason in your post
\n\nWe have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.
\n
Three swings and three misses.
\nMy best submission struck out.
", "votes": null, "replies": [] } ] }, { "id": 1482907, "author_name": "stadtfruss", "author_url": "", "post_date": "08/20/2021 10:47:52", "content": "Can we use submision system for the dataset for evaluation phase after this competition?
\nOr can you publish the dataset for evaluation phase after this competition?
I want to use it for my study.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1467211": "Hello MLB participants,\n\nWe have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.\n\nApproximately half of the candidate rerun submissions did not succeed. A large failure rate during a rerun can happen for any number of reasons, and even involve a combination of reasons. Most commonly:\n\n - new data that does not conform to the structure of the first-stage data\n - a bug in the pipelines that created the data\n - mass adoption of a flawed notebook or dataset\n\nWe have performed an initial investigation into the potential error causes here. The failures appear to largely stem from widespread forking of a notebook that used lagged target values, yet failed to handle the case where the train and test set are not adjacent in time. We provided `train_updated.csv` and messaged its existence expressly for this use case. The second stage was additionally clearly described as occurring months after the end of `train.csv` from the onset of the competition. As such, we will not be making accommodations to fix notebooks affected by this issue.\n\nOn the investigation, I would like to clarify two important points:\n\n- I describe the investigation as \"initial\" because we have more reruns to go and there is always a chance we discover other problems before the final rerun. At this time we do not have reason to believe the data is nonconforming or that we have introduced any new bugs when creating the datasets.\n- We have had discussions in previous reruns (prior this competition) about what level of error constitutes fair grounds for intervention. While we ultimately own the decision to intervene or not, we conduct these reviews in the most neutral manner possible. We do not consider the hypothetical impacts on rankings nor do we examine who is affected by any resulting actions.\n\nWe know this is frustrating for those whose notebooks did not succeed. Thank you for your participation and see you for the next rerun, the week of August 23.", "1467360": "Good luck to all participants 🙏", "1467405": "For competition medals, I am assuming the number of teams that participated (not succeeded) is what influences the number of medals handed out? https://www.kaggle.com/progression", "1467407": "Correct. 852 is the \"official\" team count for points/medals.", "1467858": "Tough love wins again", "1471188": "Thanks for the info. \nSo the next rerun will start from 08-07 to 08-23 or from 08-01 to 08-23.", "1471244": "Good luck to all remaining participants\nAlmost half have been left out after the first run, as I notice 438 in the LB vs 852 participants!!\n\nWhile a bug in the pipelines that created the data & mass adoption of a flawed notebook or dataset are in some ways the participants fault\nnew data that does not conform to the structure of the first-stage data -- that is a concern, no?", "1475218": "> At this time we do not have reason to believe the data is nonconforming", "1475271": "thanks for the update @wcukierski \nhighlighted as you have noted it as the first reason in your post", "1482798": "kmldas \n\n> We have just published the results of the first rerun, which included live data from 2021-08-01 to 2021-08-07. As is customary, each submission was provided up to three attempts to succeed.\n\nThree swings and three misses.\n\nMy best submission struck out.", "1482907": "Can we use submision system for the dataset for evaluation phase after this competition?\nOr can you publish the dataset for evaluation phase after this competition?\n\nI want to use it for my study." }, "source": "meta" }