{ "id": 251126, "title": "Please tell us about the evaluation phase", "url": "/competitions/mlb-player-digital-engagement-forecasting/discussion/251126", "author_name": "", "post_date": "2021-07-06T03:05:40.091209500Z", "votes": 38, "comment_count": 27, "views": 0, "content": "

Hi.
\nI remember that the host of this competition used to advise in a (now defunct) discussion 'not to assume continuity in testing'.
\nI'd like to ask again, is it correct that what this competition is looking for is a complete prediction of engagement in August-September using training data up to July 31?
\nSpecifically, in the evaluation phase

\n\n

If the evaluation phases were more clearly defined, it would be easier to know what we need to focus on to grow the model, and participants could improve the final score.
\nThis would be beneficial to both the host and the participants, so please provide as much information as possible.
\nThank you very much.

", "messages": [ { "id": "1377634", "postDate": "07/06/2021 03:05:40", "content": "

Hi.
\nI remember that the host of this competition used to advise in a (now defunct) discussion 'not to assume continuity in testing'.
\nI'd like to ask again, is it correct that what this competition is looking for is a complete prediction of engagement in August-September using training data up to July 31?
\nSpecifically, in the evaluation phase

\n\n

If the evaluation phases were more clearly defined, it would be easier to know what we need to focus on to grow the model, and participants could improve the final score.
\nThis would be beneficial to both the host and the participants, so please provide as much information as possible.
\nThank you very much.

", "rawMarkdown": "Hi.\n\nI remember that the host of this competition used to advise in a (now defunct) discussion 'not to assume continuity in testing'.\n\nI'd like to ask again, is it correct that what this competition is looking for is a complete prediction of engagement in August-September using training data up to July 31?\nSpecifically, in the evaluation phase\n- The grand truth before the evaluation date will not be provided.\n- Evaluation is not done continuously.\n- If the evaluation is not continuous, it will not be re-run from the first date of the evaluation.\n(e.g., when evaluating 8/20~30, test re-runs from 8/1 will not be done) \n
\n

\nIf the evaluation phases were more clearly defined, it would be easier to know what we need to focus on to grow the model, and participants could improve the final score.\nThis would be beneficial to both the host and the participants, so please provide as much information as possible.\nThank you very much.\n

", "votes": null }, { "id": "1377824", "postDate": "07/06/2021 06:29:19", "content": "

I'd also want to ask the similar question:

\n
    \n
  1. Whether the public test set ground truth(2021 May) will be provided when the final version of training data released(~ 2021 July 31th)
    2. \n
  2. How often will the evaluation proceed(once a day/once a week/…)? And whether it will be re-run from the first date of the evaluation.
    3. \n
\n

If the questions are confirmed, we can decide if we should add some season accumulate statistics in our model training.
\nThanks:)

", "rawMarkdown": "I'd also want to ask the similar question: \n1. Whether the public test set ground truth(2021 May) will be provided when the final version of training data released(~ 2021 July 31th)\n2. How often will the evaluation proceed(once a day/once a week/...)? And whether it will be re-run from the first date of the evaluation.\n\nIf the questions are confirmed, we can decide if we should add some season accumulate statistics in our model training.\nThanks:)", "votes": null }, { "id": "1377870", "postDate": "07/06/2021 07:28:14", "content": "

Yes we have the same question @juliaelliott @wcukierski @ryanholbrook @philculliton @alokpattani. During the evaluation phase, will we always be scored from auguts 1st to the evaluation date, or will we only be scored at the evaluation date. This question is crucial for people using lag features. If need be, it is also good to know the last evaluation date. Thks in advance.

", "rawMarkdown": "Yes we have the same question @juliaelliott @wcukierski @ryanholbrook @philculliton @alokpattani. During the evaluation phase, will we always be scored from auguts 1st to the evaluation date, or will we only be scored at the evaluation date. This question is crucial for people using lag features. If need be, it is also good to know the last evaluation date. Thks in advance.", "votes": null }, { "id": "1378239", "postDate": "07/06/2021 11:51:58", "content": "

August-September is before the post-season. So, the number of win is more important than other period?

", "rawMarkdown": "August-September is before the post-season. So, the number of win is more important than other period?", "votes": null }, { "id": "1379948", "postDate": "07/07/2021 17:25:12", "content": "

It appears the poster of this post which originally raised the test set continuity question has since deleted their account/comment, which provided the best thread that discussed this topic. I recommend reviewing it. The key points are:

\n", "rawMarkdown": "It appears the poster of [this post which originally raised the test set continuity question](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/245606) has since deleted their account/comment, which provided the best thread that discussed this topic. I recommend reviewing it. The key points are:\n- The training data will be updated to the data page in a separate `train_updated.csv` file on roughly July 20th. This will be in the same format as the existing `train.csv`. After the deadline, `train_updated.csv` will be updated again to contain data up through July 31st.\n- Because of ^, we have intentionally timed this to be a late release, in order to minimize public leaderboard contamination. Note, however, that your final ranking will only be based on the future test evaluation period.\n- It is up to you whether to make use of that additional training data.\n- We have not (and will not) specified the precise dates of the test set. You should not make any assumptions about the continuity of the test set.\n- The frequency of rerun and leaderboard update has not been determined. Assuming we opt to support a >1 rerun frequency, each successive rerun would be cumulative, so inclusive of the evaluation period in the prior rerun(s).", "votes": null }, { "id": "1380240", "postDate": "07/08/2021 00:14:26", "content": "

You had me understanding until the last bullet. It would seem to me that greater than a single rerun will just give more opportunities for us to have a failed kernel. I understand the need for the API and the almost total lack of error feedback on failures. Giving me more chances to fail (beyond 1) does not seem like a desirable idea. The API would seem a huge barrier to folks getting good submissions and a heck of a nice time waster trying to debug - please don't give me more than one chance to fail on the final LB.

\n

If you decide to run more than once - your statement on successive being cumulative is very confusing to this old man. Please expand.

", "rawMarkdown": "You had me understanding until the last bullet. It would seem to me that greater than a single rerun will just give more opportunities for us to have a failed kernel. I understand the need for the API and the almost total lack of error feedback on failures. Giving me more chances to fail (beyond 1) does not seem like a desirable idea. The API would seem a huge barrier to folks getting good submissions and a heck of a nice time waster trying to debug - please don't give me more than one chance to fail on the final LB.\n\nIf you decide to run more than once - your statement on successive being cumulative is very confusing to this old man. Please expand.", "votes": null }, { "id": "1381310", "postDate": "07/08/2021 20:58:38", "content": "

The final scores and rankings will be based on some (undisclosed) timeframe after July 31st. So another way of rephrasing the last bullet is: If we were to do any interim evaluation period leaderboard updates (as is being done in the Jane Street competition), they would only represent a subset of the full evaluation period. It's ultimately the final rerun encompassing the full evaluation period which would determine final scores.

", "rawMarkdown": "The final scores and rankings will be based on some (undisclosed) timeframe after July 31st. So another way of rephrasing the last bullet is: If we were to do any interim evaluation period leaderboard updates (as is being done in the Jane Street competition), they would only represent a subset of the full evaluation period. It's ultimately the final rerun encompassing the full evaluation period which would determine final scores.", "votes": null }, { "id": "1381451", "postDate": "07/09/2021 03:01:31", "content": "

Got it - thanks

", "rawMarkdown": "Got it - thanks", "votes": null }, { "id": "1381892", "postDate": "07/09/2021 10:21:08", "content": "

Kind of confused why this is not being run like the NFL Big DataBowl ? It seemed much clearer but perhaps more games over all days of the week for baseball and availability of targets?

\n

While it is understandable what is in the submission for date and playerId could be subsets or only some days and not revealed which,but why would what is in the test data not be continuous? What would be the point of that? It should only have data for players, teams, games when played of course.

\n

Could I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right?

", "rawMarkdown": "Kind of confused why this is not being run like the [NFL Big DataBowl](https://www.kaggle.com/c/nfl-big-data-bowl-2020/overview/timeline) ? It seemed much clearer but perhaps more games over all days of the week for baseball and availability of targets?\n\nWhile it is understandable what is in the submission for date and playerId could be subsets or only some days and not revealed which,but why would what is in the test data not be continuous? What would be the point of that? It should only have data for players, teams, games when played of course. \n\nCould I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right?", "votes": null }, { "id": "1383534", "postDate": "07/11/2021 01:48:12", "content": "

Forgive me if i missed something, but what is the point of updating the training set after the deadline? @juliaelliott

", "rawMarkdown": "Forgive me if i missed something, but what is the point of updating the training set after the deadline? @juliaelliott", "votes": null }, { "id": "1383773", "postDate": "07/11/2021 08:19:55", "content": "

@shujun717 - see what Julia replies, but some information on this so far -

\n

From the overview timeline page copied below the idea is an update for training prior to Final Submission deadline -

\n

July 20, 2021 (estimated date) - Training Set Update. The Data page will be updated with the latest release of training data. We aim to complete the ~1.5 weeks prior to the Final Submission Deadline, exact date to be communicated via the forums.

\n

July 24, 2021 - Entry Deadline. You must accept the competition rules before this date in order to compete.

\n

July 24, 2021 - Team Merger Deadline. This is the last day participants may join or merge teams.

\n

July 31, 2021 - Final Submission Deadline. This is also the deadline to select your 2 final submissions which will be rerun during the evaluation period.

\n

Will C. also posted here that updated train would be opt-in so have a different name like train_updated.csv so people could use it and retrain or not.

", "rawMarkdown": "shujun717 - see what Julia replies, but some information on this so far - \n\nFrom the [overview timeline page](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/timeline) copied below the idea is an update for training prior to Final Submission deadline -\n\nJuly 20, 2021 (estimated date) - Training Set Update. The Data page will be updated with the latest release of training data. We aim to complete the ~1.5 weeks prior to the Final Submission Deadline, exact date to be communicated via the forums.\n\nJuly 24, 2021 - Entry Deadline. You must accept the competition rules before this date in order to compete.\n\nJuly 24, 2021 - Team Merger Deadline. This is the last day participants may join or merge teams.\n\nJuly 31, 2021 - Final Submission Deadline. This is also the deadline to select your 2 final submissions which will be rerun during the evaluation period.\n\nWill C. also posted [here](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/245606#1350775) that updated train would be opt-in so have a different name like train_updated.csv so people could use it and retrain or not.", "votes": null }, { "id": "1384529", "postDate": "07/12/2021 01:01:23", "content": "

Thank you for answering, @juliaelliott .
\nWhat about the following points?

\n
\n \n
\n

For example, when we use the API to predict August 15 during the evaluation phase, can we get the ground truth for August 14, or not?

\n

Sorry if this has already been mentioned in other discussions.

", "rawMarkdown": "Thank you for answering, @juliaelliott .\nWhat about the following points?\n\n> - The grand truth before the evaluation date will not be provided.\n\nFor example, when we use the API to predict August 15 during the evaluation phase, can we get the ground truth for August 14, or not?\n\nSorry if this has already been mentioned in other discussions.", "votes": null }, { "id": "1393292", "postDate": "07/19/2021 14:23:37", "content": "

@nomorevotch No, you will not receive any lagged ground truth; the latest you will receive ground truth up to is July 31st for a future prediction period thereafter.

", "rawMarkdown": "nomorevotch No, you will not receive any lagged ground truth; the latest you will receive ground truth up to is July 31st for a future prediction period thereafter.", "votes": null }, { "id": "1393438", "postDate": "07/19/2021 16:33:02", "content": "

@juliaelliott OK, thank you very much.

", "rawMarkdown": "juliaelliott OK, thank you very much.", "votes": null }, { "id": "1393601", "postDate": "07/19/2021 18:46:51", "content": "

@juliaelliott Assume this means no lagged features either?

", "rawMarkdown": "juliaelliott Assume this means no lagged features either?", "votes": null }, { "id": "1393616", "postDate": "07/19/2021 18:59:19", "content": "

You can still do lags - its just a lot harder :)

", "rawMarkdown": "You can still do lags - its just a lot harder :)", "votes": null }, { "id": "1393838", "postDate": "07/20/2021 00:50:50", "content": "

Hello, When you say we cannot assume data to be in order, will some days be not available or do we get data out of order or both? @juliaelliott.

", "rawMarkdown": "Hello, When you say we cannot assume data to be in order, will some days be not available or do we get data out of order or both? @juliaelliott.", "votes": null }, { "id": "1394829", "postDate": "07/20/2021 16:50:46", "content": "

Input features that mirror the same data that has been provided on the Data page for the training period will be updated in the evaluation period re-run for the data that has been specified as daily (non-static).

", "rawMarkdown": "Input features that mirror the same data that has been provided on the Data page for the training period *will* be updated in the evaluation period re-run for the data that has been specified as daily (non-static).", "votes": null }, { "id": "1394928", "postDate": "07/20/2021 18:23:48", "content": "

@juliaelliott Sorry Julia, just to clarify, are you saying we will have features up to the test evaluation date, just not ground truth? e.g. If test date starts on 15th aug, we will receive features up to 15th Aug?

", "rawMarkdown": "juliaelliott Sorry Julia, just to clarify, are you saying we will have features up to the test evaluation date, just not ground truth? e.g. If test date starts on 15th aug, we will receive features up to 15th Aug?", "votes": null }, { "id": "1395216", "postDate": "07/21/2021 03:37:04", "content": "

@jacobhowardparker Yes, for the future test set, you will have the same test features (no ground truth) available on date d to make your d+1 predictions as have been made available for the April test set currently being used for the public leaderboard. For the training set, you will only receive the specified features and their ground truth up to July 31st during the evaluation period. This is as detailed on the data page.

", "rawMarkdown": "jacobhowardparker Yes, for the future test set, you will have the same test features (no ground truth) available on date d to make your d+1 predictions as have been made available for the April test set currently being used for the public leaderboard. For the training set, you will only receive the specified features and their ground truth up to July 31st during the evaluation period. This is as detailed on the [data page](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data).", "votes": null }, { "id": "1395236", "postDate": "07/21/2021 04:21:52", "content": "

Sorry Julia @juliaelliott, i still dont understand how it works. If evaluation period starts at 7th aug, ends at 31th aug, will we recieve features(no ground truth) which cover from 1st aug to 6th aug? if we will, how these features are given to us? The API can only give us features of evaldates, but the period from 1st aug to 6th aug isn't included in evaluation period.

", "rawMarkdown": "Sorry Julia @juliaelliott, i still dont understand how it works. If evaluation period starts at 7th aug, ends at 31th aug, will we recieve features(no ground truth) which cover from 1st aug to 6th aug? if we will, how these features are given to us? The API can only give us features of evaldates, but the period from 1st aug to 6th aug isn't included in evaluation period.", "votes": null }, { "id": "1396047", "postDate": "07/21/2021 17:50:34", "content": "

@juliaelliott, So there could potentially be a gap?
\ni.e. say I am using feature_a as a feature, but I'm also using the players feature_a from the previous day, and the day prior to that etc. Then the first few days of private test could run into issues with that?

", "rawMarkdown": "juliaelliott, So there could potentially be a gap?\ni.e. say I am using *feature_a* as a feature, but I'm also using the players *feature_a* from the previous day, and the day prior to that etc. Then the first few days of private test could run into issues with that?", "votes": null }, { "id": "1396393", "postDate": "07/22/2021 04:44:33", "content": "

Based on Will C.'s post updated training set soon available
\n\"We will update train_updated.csv again prior to the evaluation period (containing data up to July 31).\"

\n

And according to the Timeline page -
\n\"Evaluation Timeline

\n

Starting after the Final Submission Deadline, there will be periodic updates to the leaderboard to reflect the future date range of this competition's evaluation period, at which time each team's selected notebooks will be rerun on that future data. These reruns will include the training data updated through July 31, 2021. \"

\n

So it makes it tricky if not opting in for new train data and training models again.

\n

I am considering from what has been said in various posts - the evaluation will start from 1 Aug and may or may not be done on a periodic basis ( >1 TBD) till competition ends. But if more than 1 rerun it would include all evaluation dates prior to that rerun, so if 7 Aug rerun it would still have from 1 Aug thru to 7 Aug just like now the public LB has all of May. But no targets 1-4 for anything beyond 31 July.

\n

For players and teams static csv files, these should not be updated again during evaluation period. Whether or not they are with updated train now will have to check. After 31 July update will not know.
\nBut players box scores, they would only have players data only if they played on the day, so looking for previous info needs to consider how to handle that. Similar for team box scores.

\n

Whatever dates and player combinations are in the submission file for each evaluation periodic rerun could be just a subset of players, so if trying to use predicted features for target1-4 in future would also have to consider how to handle that. Meaning predict for all not just what is in submission files.

\n

@juliaelliott, is that pretty close to how evaluation will work? Do you think we could have a pinned thread for evaluation phase sometime soon so everyone has the latest advice, information? Thanks!!

", "rawMarkdown": "Based on Will C.'s post [updated training set soon available](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/254265)\n\"We will update train_updated.csv again prior to the evaluation period (containing data up to July 31).\"\n\nAnd according to the Timeline page - \n\"Evaluation Timeline\n\nStarting after the Final Submission Deadline, there will be periodic updates to the leaderboard to reflect the future date range of this competition's evaluation period, at which time each team's selected notebooks will be rerun on that future data. These reruns will include the training data updated through July 31, 2021. \"\n\nSo it makes it tricky if not opting in for new train data and training models again. \n\nI am considering from what has been said in various posts - the evaluation will start from 1 Aug and may or may not be done on a periodic basis ( >1 TBD) till competition ends. But if more than 1 rerun it would include all evaluation dates prior to that rerun, so if 7 Aug rerun it would still have from 1 Aug thru to 7 Aug just like now the public LB has all of May. But no targets 1-4 for anything beyond 31 July.\n\nFor players and teams static csv files, these should not be updated again during evaluation period. Whether or not they are with updated train now will have to check. After 31 July update will not know. \nBut players box scores, they would only have players data only if they played on the day, so looking for previous info needs to consider how to handle that. Similar for team box scores.\n\nWhatever dates and player combinations are in the submission file for each evaluation periodic rerun could be just a subset of players, so if trying to use predicted features for target1-4 in future would also have to consider how to handle that. Meaning predict for all not just what is in submission files. \n\n@juliaelliott, is that pretty close to how evaluation will work? Do you think we could have a pinned thread for evaluation phase sometime soon so everyone has the latest advice, information? Thanks!!", "votes": null }, { "id": "1397096", "postDate": "07/22/2021 19:05:53", "content": "

@something4kag You appear to have reiterated points that have been addressed previously already. I believe I have confirmed all of this in my post below. Rather than respond to your re-articulation which I am not sure is precisely how I would explain it, I believe these points have already been made clear in this thread. If you have a precise question about not understanding a statement I have made, you're welcome to ask for that clarification.

", "rawMarkdown": "something4kag You appear to have reiterated points that have been addressed previously already. I believe I have confirmed all of this in [my post below](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/251126#1379948). Rather than respond to your re-articulation which I am not sure is precisely how I would explain it, I believe these points have already been made clear in this thread. If you have a precise question about not understanding a statement I have made, you're welcome to ask for that clarification.", "votes": null }, { "id": "1397295", "postDate": "07/23/2021 03:45:20", "content": "

@juliaelliott - some of the repeat is for others that may not have seen the information and since one of the earlier posts was by someone deleted so it kind of got lost. Have no problem with not replying to me. Just was suggesting a pinned post for others in the competition that may like to understand evaluation period since it is coming up soon. A synopsis of the points might help everyone, but if not that is no problem too.

", "rawMarkdown": "juliaelliott - some of the repeat is for others that may not have seen the information and since one of the earlier posts was by someone deleted so it kind of got lost. Have no problem with not replying to me. Just was suggesting a pinned post for others in the competition that may like to understand evaluation period since it is coming up soon. A synopsis of the points might help everyone, but if not that is no problem too.", "votes": null }, { "id": "1400901", "postDate": "07/26/2021 17:39:09", "content": "

Realizing I'd skipped over @ben7252 and @jacobhowardparker 's follow-up questions RE: test feature gap. To clarify, during the evaluation period, you will receive test features for the full post-July 31st period, up until the conclusion of that not-yet-specified evaluation timeframe, even if there are gap dates which do not count towards scoring. So if the evaluation period runs from Aug 7-20, you will still receive test features (without ground truth) for the full timeframe of Aug 1-20, even though you will only be scored the subset of that.

", "rawMarkdown": "Realizing I'd skipped over @ben7252 and @jacobhowardparker 's follow-up questions RE: test feature gap. To clarify, during the evaluation period, you will receive test features for the full post-July 31st period, up until the conclusion of that not-yet-specified evaluation timeframe, even if there are gap dates which do not count towards scoring. So if the evaluation period runs from Aug 7-20, you will still receive test features (without ground truth) for the full timeframe of Aug 1-20, even though you will only be scored the subset of that.", "votes": null }, { "id": "1400920", "postDate": "07/26/2021 17:56:30", "content": "

Thanks Julia. This makes sense now :)

", "rawMarkdown": "Thanks Julia. This makes sense now :)", "votes": null }, { "id": "1402669", "postDate": "07/28/2021 12:15:00", "content": "

Dear Kaggle Staff

\n

I checked the following page and almost understood it.
\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data

\n

However, one point is that I don't understand the behavior of the data in the mlb module when you rerun the program.

\n

Specifically, there is a program example of how to submit using the mlb module on the following page,
\nbut I don't know what period of time the data is stored in the iter_test variable.
\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/evaluation

\n
\n

import mlb
\n env = mlb.make_env() # initialize the environment
\n iter_test = env.iter_test() # iterator which loops over each date in test set
\n for (test_df, sample_prediction_df) in iter_test:
\n sample_prediction_df['target1'] = 100 #make predictions here
\n env.predict(sample_prediction_df)

\n
\n

For example, when you re-run the notebook on 8/7 first,
\nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/7 (except for nextDayPlayerEngagement) and the data for submitting prediction scores?
\nBy running \"for (test_df, sample_prediction_df) in iter_test:\",
\ncan we refer to each data for one day at a time in date order?

\n

Also, if you re-run the notebook for the second time on 8/21,
\nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/21 (except for nextDayPlayerEngagement) and the data for submitting prediction scores or from 8/8 to 8/21?

\n

As for running the notebook after the deadline,
\nI believe that the env of the mlb module will be only updated to the latest version,
\nbut the other input files will not be updated.

\n

Is any of the above incorrect?

\n

I would appreciate it if you could help me.
\nThank you very much for your help.

", "rawMarkdown": "Dear Kaggle Staff\n\nI checked the following page and almost understood it.\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data\n\nHowever, one point is that I don't understand the behavior of the data in the mlb module when you rerun the program.\n\nSpecifically, there is a program example of how to submit using the mlb module on the following page, \nbut I don't know what period of time the data is stored in the iter_test variable.\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/evaluation\n\n> import mlb\nenv = mlb.make_env() # initialize the environment\niter_test = env.iter_test() # iterator which loops over each date in test set\nfor (test_df, sample_prediction_df) in iter_test:\n sample_prediction_df['target1'] = 100 #make predictions here\n env.predict(sample_prediction_df)\n\nFor example, when you re-run the notebook on 8/7 first, \nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/7 (except for nextDayPlayerEngagement) and the data for submitting prediction scores?\nBy running \"for (test_df, sample_prediction_df) in iter_test:\", \ncan we refer to each data for one day at a time in date order?\n\nAlso, if you re-run the notebook for the second time on 8/21, \nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/21 (except for nextDayPlayerEngagement) and the data for submitting prediction scores or from 8/8 to 8/21?\n\nAs for running the notebook after the deadline,\nI believe that the env of the mlb module will be only updated to the latest version, \nbut the other input files will not be updated.\n\nIs any of the above incorrect?\n\nI would appreciate it if you could help me.\nThank you very much for your help.", "votes": null } ], "comments": [ { "id": 1377824, "author_name": "joshspchang", "author_url": "", "post_date": "07/06/2021 06:29:19", "content": "

I'd also want to ask the similar question:

\n
    \n
  1. Whether the public test set ground truth(2021 May) will be provided when the final version of training data released(~ 2021 July 31th)
    2. \n
  2. How often will the evaluation proceed(once a day/once a week/…)? And whether it will be re-run from the first date of the evaluation.
    3. \n
\n

If the questions are confirmed, we can decide if we should add some season accumulate statistics in our model training.
\nThanks:)

", "votes": null, "replies": [] }, { "id": 1377870, "author_name": "ulrich07", "author_url": "", "post_date": "07/06/2021 07:28:14", "content": "

Yes we have the same question @juliaelliott @wcukierski @ryanholbrook @philculliton @alokpattani. During the evaluation phase, will we always be scored from auguts 1st to the evaluation date, or will we only be scored at the evaluation date. This question is crucial for people using lag features. If need be, it is also good to know the last evaluation date. Thks in advance.

", "votes": null, "replies": [] }, { "id": 1378239, "author_name": "kohashi0000", "author_url": "", "post_date": "07/06/2021 11:51:58", "content": "

August-September is before the post-season. So, the number of win is more important than other period?

", "votes": null, "replies": [] }, { "id": 1379948, "author_name": "juliaelliott", "author_url": "", "post_date": "07/07/2021 17:25:12", "content": "

It appears the poster of this post which originally raised the test set continuity question has since deleted their account/comment, which provided the best thread that discussed this topic. I recommend reviewing it. The key points are:

\n", "votes": null, "replies": [ { "id": 1380240, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/08/2021 00:14:26", "content": "

You had me understanding until the last bullet. It would seem to me that greater than a single rerun will just give more opportunities for us to have a failed kernel. I understand the need for the API and the almost total lack of error feedback on failures. Giving me more chances to fail (beyond 1) does not seem like a desirable idea. The API would seem a huge barrier to folks getting good submissions and a heck of a nice time waster trying to debug - please don't give me more than one chance to fail on the final LB.

\n

If you decide to run more than once - your statement on successive being cumulative is very confusing to this old man. Please expand.

", "votes": null, "replies": [] }, { "id": 1381310, "author_name": "juliaelliott", "author_url": "", "post_date": "07/08/2021 20:58:38", "content": "

The final scores and rankings will be based on some (undisclosed) timeframe after July 31st. So another way of rephrasing the last bullet is: If we were to do any interim evaluation period leaderboard updates (as is being done in the Jane Street competition), they would only represent a subset of the full evaluation period. It's ultimately the final rerun encompassing the full evaluation period which would determine final scores.

", "votes": null, "replies": [] }, { "id": 1381451, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/09/2021 03:01:31", "content": "

Got it - thanks

", "votes": null, "replies": [] }, { "id": 1383534, "author_name": "shujun717", "author_url": "", "post_date": "07/11/2021 01:48:12", "content": "

Forgive me if i missed something, but what is the point of updating the training set after the deadline? @juliaelliott

", "votes": null, "replies": [] }, { "id": 1383773, "author_name": "something4kag", "author_url": "", "post_date": "07/11/2021 08:19:55", "content": "

@shujun717 - see what Julia replies, but some information on this so far -

\n

From the overview timeline page copied below the idea is an update for training prior to Final Submission deadline -

\n

July 20, 2021 (estimated date) - Training Set Update. The Data page will be updated with the latest release of training data. We aim to complete the ~1.5 weeks prior to the Final Submission Deadline, exact date to be communicated via the forums.

\n

July 24, 2021 - Entry Deadline. You must accept the competition rules before this date in order to compete.

\n

July 24, 2021 - Team Merger Deadline. This is the last day participants may join or merge teams.

\n

July 31, 2021 - Final Submission Deadline. This is also the deadline to select your 2 final submissions which will be rerun during the evaluation period.

\n

Will C. also posted here that updated train would be opt-in so have a different name like train_updated.csv so people could use it and retrain or not.

", "votes": null, "replies": [] }, { "id": 1393838, "author_name": "chinta", "author_url": "", "post_date": "07/20/2021 00:50:50", "content": "

Hello, When you say we cannot assume data to be in order, will some days be not available or do we get data out of order or both? @juliaelliott.

", "votes": null, "replies": [] } ] }, { "id": 1381892, "author_name": "something4kag", "author_url": "", "post_date": "07/09/2021 10:21:08", "content": "

Kind of confused why this is not being run like the NFL Big DataBowl ? It seemed much clearer but perhaps more games over all days of the week for baseball and availability of targets?

\n

While it is understandable what is in the submission for date and playerId could be subsets or only some days and not revealed which,but why would what is in the test data not be continuous? What would be the point of that? It should only have data for players, teams, games when played of course.

\n

Could I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right?

", "votes": null, "replies": [] }, { "id": 1384529, "author_name": "nomorevotch", "author_url": "", "post_date": "07/12/2021 01:01:23", "content": "

Thank you for answering, @juliaelliott .
\nWhat about the following points?

\n
\n \n
\n

For example, when we use the API to predict August 15 during the evaluation phase, can we get the ground truth for August 14, or not?

\n

Sorry if this has already been mentioned in other discussions.

", "votes": null, "replies": [ { "id": 1393292, "author_name": "juliaelliott", "author_url": "", "post_date": "07/19/2021 14:23:37", "content": "

@nomorevotch No, you will not receive any lagged ground truth; the latest you will receive ground truth up to is July 31st for a future prediction period thereafter.

", "votes": null, "replies": [] }, { "id": 1393438, "author_name": "nomorevotch", "author_url": "", "post_date": "07/19/2021 16:33:02", "content": "

@juliaelliott OK, thank you very much.

", "votes": null, "replies": [] }, { "id": 1393601, "author_name": "jacobhowardparker", "author_url": "", "post_date": "07/19/2021 18:46:51", "content": "

@juliaelliott Assume this means no lagged features either?

", "votes": null, "replies": [] }, { "id": 1393616, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/19/2021 18:59:19", "content": "

You can still do lags - its just a lot harder :)

", "votes": null, "replies": [] }, { "id": 1394829, "author_name": "juliaelliott", "author_url": "", "post_date": "07/20/2021 16:50:46", "content": "

Input features that mirror the same data that has been provided on the Data page for the training period will be updated in the evaluation period re-run for the data that has been specified as daily (non-static).

", "votes": null, "replies": [] }, { "id": 1394928, "author_name": "jacobhowardparker", "author_url": "", "post_date": "07/20/2021 18:23:48", "content": "

@juliaelliott Sorry Julia, just to clarify, are you saying we will have features up to the test evaluation date, just not ground truth? e.g. If test date starts on 15th aug, we will receive features up to 15th Aug?

", "votes": null, "replies": [] }, { "id": 1395216, "author_name": "juliaelliott", "author_url": "", "post_date": "07/21/2021 03:37:04", "content": "

@jacobhowardparker Yes, for the future test set, you will have the same test features (no ground truth) available on date d to make your d+1 predictions as have been made available for the April test set currently being used for the public leaderboard. For the training set, you will only receive the specified features and their ground truth up to July 31st during the evaluation period. This is as detailed on the data page.

", "votes": null, "replies": [] }, { "id": 1395236, "author_name": "ben7252", "author_url": "", "post_date": "07/21/2021 04:21:52", "content": "

Sorry Julia @juliaelliott, i still dont understand how it works. If evaluation period starts at 7th aug, ends at 31th aug, will we recieve features(no ground truth) which cover from 1st aug to 6th aug? if we will, how these features are given to us? The API can only give us features of evaldates, but the period from 1st aug to 6th aug isn't included in evaluation period.

", "votes": null, "replies": [] }, { "id": 1396047, "author_name": "jacobhowardparker", "author_url": "", "post_date": "07/21/2021 17:50:34", "content": "

@juliaelliott, So there could potentially be a gap?
\ni.e. say I am using feature_a as a feature, but I'm also using the players feature_a from the previous day, and the day prior to that etc. Then the first few days of private test could run into issues with that?

", "votes": null, "replies": [] }, { "id": 1396393, "author_name": "something4kag", "author_url": "", "post_date": "07/22/2021 04:44:33", "content": "

Based on Will C.'s post updated training set soon available
\n\"We will update train_updated.csv again prior to the evaluation period (containing data up to July 31).\"

\n

And according to the Timeline page -
\n\"Evaluation Timeline

\n

Starting after the Final Submission Deadline, there will be periodic updates to the leaderboard to reflect the future date range of this competition's evaluation period, at which time each team's selected notebooks will be rerun on that future data. These reruns will include the training data updated through July 31, 2021. \"

\n

So it makes it tricky if not opting in for new train data and training models again.

\n

I am considering from what has been said in various posts - the evaluation will start from 1 Aug and may or may not be done on a periodic basis ( >1 TBD) till competition ends. But if more than 1 rerun it would include all evaluation dates prior to that rerun, so if 7 Aug rerun it would still have from 1 Aug thru to 7 Aug just like now the public LB has all of May. But no targets 1-4 for anything beyond 31 July.

\n

For players and teams static csv files, these should not be updated again during evaluation period. Whether or not they are with updated train now will have to check. After 31 July update will not know.
\nBut players box scores, they would only have players data only if they played on the day, so looking for previous info needs to consider how to handle that. Similar for team box scores.

\n

Whatever dates and player combinations are in the submission file for each evaluation periodic rerun could be just a subset of players, so if trying to use predicted features for target1-4 in future would also have to consider how to handle that. Meaning predict for all not just what is in submission files.

\n

@juliaelliott, is that pretty close to how evaluation will work? Do you think we could have a pinned thread for evaluation phase sometime soon so everyone has the latest advice, information? Thanks!!

", "votes": null, "replies": [] }, { "id": 1397096, "author_name": "juliaelliott", "author_url": "", "post_date": "07/22/2021 19:05:53", "content": "

@something4kag You appear to have reiterated points that have been addressed previously already. I believe I have confirmed all of this in my post below. Rather than respond to your re-articulation which I am not sure is precisely how I would explain it, I believe these points have already been made clear in this thread. If you have a precise question about not understanding a statement I have made, you're welcome to ask for that clarification.

", "votes": null, "replies": [] }, { "id": 1397295, "author_name": "something4kag", "author_url": "", "post_date": "07/23/2021 03:45:20", "content": "

@juliaelliott - some of the repeat is for others that may not have seen the information and since one of the earlier posts was by someone deleted so it kind of got lost. Have no problem with not replying to me. Just was suggesting a pinned post for others in the competition that may like to understand evaluation period since it is coming up soon. A synopsis of the points might help everyone, but if not that is no problem too.

", "votes": null, "replies": [] }, { "id": 1400901, "author_name": "juliaelliott", "author_url": "", "post_date": "07/26/2021 17:39:09", "content": "

Realizing I'd skipped over @ben7252 and @jacobhowardparker 's follow-up questions RE: test feature gap. To clarify, during the evaluation period, you will receive test features for the full post-July 31st period, up until the conclusion of that not-yet-specified evaluation timeframe, even if there are gap dates which do not count towards scoring. So if the evaluation period runs from Aug 7-20, you will still receive test features (without ground truth) for the full timeframe of Aug 1-20, even though you will only be scored the subset of that.

", "votes": null, "replies": [] }, { "id": 1400920, "author_name": "jacobhowardparker", "author_url": "", "post_date": "07/26/2021 17:56:30", "content": "

Thanks Julia. This makes sense now :)

", "votes": null, "replies": [] } ] }, { "id": 1402669, "author_name": "tomfuj", "author_url": "", "post_date": "07/28/2021 12:15:00", "content": "

Dear Kaggle Staff

\n

I checked the following page and almost understood it.
\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data

\n

However, one point is that I don't understand the behavior of the data in the mlb module when you rerun the program.

\n

Specifically, there is a program example of how to submit using the mlb module on the following page,
\nbut I don't know what period of time the data is stored in the iter_test variable.
\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/evaluation

\n
\n

import mlb
\n env = mlb.make_env() # initialize the environment
\n iter_test = env.iter_test() # iterator which loops over each date in test set
\n for (test_df, sample_prediction_df) in iter_test:
\n sample_prediction_df['target1'] = 100 #make predictions here
\n env.predict(sample_prediction_df)

\n
\n

For example, when you re-run the notebook on 8/7 first,
\nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/7 (except for nextDayPlayerEngagement) and the data for submitting prediction scores?
\nBy running \"for (test_df, sample_prediction_df) in iter_test:\",
\ncan we refer to each data for one day at a time in date order?

\n

Also, if you re-run the notebook for the second time on 8/21,
\nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/21 (except for nextDayPlayerEngagement) and the data for submitting prediction scores or from 8/8 to 8/21?

\n

As for running the notebook after the deadline,
\nI believe that the env of the mlb module will be only updated to the latest version,
\nbut the other input files will not be updated.

\n

Is any of the above incorrect?

\n

I would appreciate it if you could help me.
\nThank you very much for your help.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1377634": "Hi.\n\nI remember that the host of this competition used to advise in a (now defunct) discussion 'not to assume continuity in testing'.\n\nI'd like to ask again, is it correct that what this competition is looking for is a complete prediction of engagement in August-September using training data up to July 31?\nSpecifically, in the evaluation phase\n- The grand truth before the evaluation date will not be provided.\n- Evaluation is not done continuously.\n- If the evaluation is not continuous, it will not be re-run from the first date of the evaluation.\n(e.g., when evaluating 8/20~30, test re-runs from 8/1 will not be done) \n
\n

\nIf the evaluation phases were more clearly defined, it would be easier to know what we need to focus on to grow the model, and participants could improve the final score.\nThis would be beneficial to both the host and the participants, so please provide as much information as possible.\nThank you very much.\n

", "1377824": "I'd also want to ask the similar question: \n1. Whether the public test set ground truth(2021 May) will be provided when the final version of training data released(~ 2021 July 31th)\n2. How often will the evaluation proceed(once a day/once a week/...)? And whether it will be re-run from the first date of the evaluation.\n\nIf the questions are confirmed, we can decide if we should add some season accumulate statistics in our model training.\nThanks:)", "1377870": "Yes we have the same question @juliaelliott @wcukierski @ryanholbrook @philculliton @alokpattani. During the evaluation phase, will we always be scored from auguts 1st to the evaluation date, or will we only be scored at the evaluation date. This question is crucial for people using lag features. If need be, it is also good to know the last evaluation date. Thks in advance.", "1378239": "August-September is before the post-season. So, the number of win is more important than other period?", "1379948": "It appears the poster of [this post which originally raised the test set continuity question](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/245606) has since deleted their account/comment, which provided the best thread that discussed this topic. I recommend reviewing it. The key points are:\n- The training data will be updated to the data page in a separate `train_updated.csv` file on roughly July 20th. This will be in the same format as the existing `train.csv`. After the deadline, `train_updated.csv` will be updated again to contain data up through July 31st.\n- Because of ^, we have intentionally timed this to be a late release, in order to minimize public leaderboard contamination. Note, however, that your final ranking will only be based on the future test evaluation period.\n- It is up to you whether to make use of that additional training data.\n- We have not (and will not) specified the precise dates of the test set. You should not make any assumptions about the continuity of the test set.\n- The frequency of rerun and leaderboard update has not been determined. Assuming we opt to support a >1 rerun frequency, each successive rerun would be cumulative, so inclusive of the evaluation period in the prior rerun(s).", "1380240": "You had me understanding until the last bullet. It would seem to me that greater than a single rerun will just give more opportunities for us to have a failed kernel. I understand the need for the API and the almost total lack of error feedback on failures. Giving me more chances to fail (beyond 1) does not seem like a desirable idea. The API would seem a huge barrier to folks getting good submissions and a heck of a nice time waster trying to debug - please don't give me more than one chance to fail on the final LB.\n\nIf you decide to run more than once - your statement on successive being cumulative is very confusing to this old man. Please expand.", "1381310": "The final scores and rankings will be based on some (undisclosed) timeframe after July 31st. So another way of rephrasing the last bullet is: If we were to do any interim evaluation period leaderboard updates (as is being done in the Jane Street competition), they would only represent a subset of the full evaluation period. It's ultimately the final rerun encompassing the full evaluation period which would determine final scores.", "1381451": "Got it - thanks", "1381892": "Kind of confused why this is not being run like the [NFL Big DataBowl](https://www.kaggle.com/c/nfl-big-data-bowl-2020/overview/timeline) ? It seemed much clearer but perhaps more games over all days of the week for baseball and availability of targets?\n\nWhile it is understandable what is in the submission for date and playerId could be subsets or only some days and not revealed which,but why would what is in the test data not be continuous? What would be the point of that? It should only have data for players, teams, games when played of course. \n\nCould I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right?", "1383534": "Forgive me if i missed something, but what is the point of updating the training set after the deadline? @juliaelliott", "1383773": "shujun717 - see what Julia replies, but some information on this so far - \n\nFrom the [overview timeline page](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/timeline) copied below the idea is an update for training prior to Final Submission deadline -\n\nJuly 20, 2021 (estimated date) - Training Set Update. The Data page will be updated with the latest release of training data. We aim to complete the ~1.5 weeks prior to the Final Submission Deadline, exact date to be communicated via the forums.\n\nJuly 24, 2021 - Entry Deadline. You must accept the competition rules before this date in order to compete.\n\nJuly 24, 2021 - Team Merger Deadline. This is the last day participants may join or merge teams.\n\nJuly 31, 2021 - Final Submission Deadline. This is also the deadline to select your 2 final submissions which will be rerun during the evaluation period.\n\nWill C. also posted [here](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/245606#1350775) that updated train would be opt-in so have a different name like train_updated.csv so people could use it and retrain or not.", "1384529": "Thank you for answering, @juliaelliott .\nWhat about the following points?\n\n> - The grand truth before the evaluation date will not be provided.\n\nFor example, when we use the API to predict August 15 during the evaluation phase, can we get the ground truth for August 14, or not?\n\nSorry if this has already been mentioned in other discussions.", "1393292": "nomorevotch No, you will not receive any lagged ground truth; the latest you will receive ground truth up to is July 31st for a future prediction period thereafter.", "1393438": "juliaelliott OK, thank you very much.", "1393601": "juliaelliott Assume this means no lagged features either?", "1393616": "You can still do lags - its just a lot harder :)", "1393838": "Hello, When you say we cannot assume data to be in order, will some days be not available or do we get data out of order or both? @juliaelliott.", "1394829": "Input features that mirror the same data that has been provided on the Data page for the training period *will* be updated in the evaluation period re-run for the data that has been specified as daily (non-static).", "1394928": "juliaelliott Sorry Julia, just to clarify, are you saying we will have features up to the test evaluation date, just not ground truth? e.g. If test date starts on 15th aug, we will receive features up to 15th Aug?", "1395216": "jacobhowardparker Yes, for the future test set, you will have the same test features (no ground truth) available on date d to make your d+1 predictions as have been made available for the April test set currently being used for the public leaderboard. For the training set, you will only receive the specified features and their ground truth up to July 31st during the evaluation period. This is as detailed on the [data page](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data).", "1395236": "Sorry Julia @juliaelliott, i still dont understand how it works. If evaluation period starts at 7th aug, ends at 31th aug, will we recieve features(no ground truth) which cover from 1st aug to 6th aug? if we will, how these features are given to us? The API can only give us features of evaldates, but the period from 1st aug to 6th aug isn't included in evaluation period.", "1396047": "juliaelliott, So there could potentially be a gap?\ni.e. say I am using *feature_a* as a feature, but I'm also using the players *feature_a* from the previous day, and the day prior to that etc. Then the first few days of private test could run into issues with that?", "1396393": "Based on Will C.'s post [updated training set soon available](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/254265)\n\"We will update train_updated.csv again prior to the evaluation period (containing data up to July 31).\"\n\nAnd according to the Timeline page - \n\"Evaluation Timeline\n\nStarting after the Final Submission Deadline, there will be periodic updates to the leaderboard to reflect the future date range of this competition's evaluation period, at which time each team's selected notebooks will be rerun on that future data. These reruns will include the training data updated through July 31, 2021. \"\n\nSo it makes it tricky if not opting in for new train data and training models again. \n\nI am considering from what has been said in various posts - the evaluation will start from 1 Aug and may or may not be done on a periodic basis ( >1 TBD) till competition ends. But if more than 1 rerun it would include all evaluation dates prior to that rerun, so if 7 Aug rerun it would still have from 1 Aug thru to 7 Aug just like now the public LB has all of May. But no targets 1-4 for anything beyond 31 July.\n\nFor players and teams static csv files, these should not be updated again during evaluation period. Whether or not they are with updated train now will have to check. After 31 July update will not know. \nBut players box scores, they would only have players data only if they played on the day, so looking for previous info needs to consider how to handle that. Similar for team box scores.\n\nWhatever dates and player combinations are in the submission file for each evaluation periodic rerun could be just a subset of players, so if trying to use predicted features for target1-4 in future would also have to consider how to handle that. Meaning predict for all not just what is in submission files. \n\n@juliaelliott, is that pretty close to how evaluation will work? Do you think we could have a pinned thread for evaluation phase sometime soon so everyone has the latest advice, information? Thanks!!", "1397096": "something4kag You appear to have reiterated points that have been addressed previously already. I believe I have confirmed all of this in [my post below](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/251126#1379948). Rather than respond to your re-articulation which I am not sure is precisely how I would explain it, I believe these points have already been made clear in this thread. If you have a precise question about not understanding a statement I have made, you're welcome to ask for that clarification.", "1397295": "juliaelliott - some of the repeat is for others that may not have seen the information and since one of the earlier posts was by someone deleted so it kind of got lost. Have no problem with not replying to me. Just was suggesting a pinned post for others in the competition that may like to understand evaluation period since it is coming up soon. A synopsis of the points might help everyone, but if not that is no problem too.", "1400901": "Realizing I'd skipped over @ben7252 and @jacobhowardparker 's follow-up questions RE: test feature gap. To clarify, during the evaluation period, you will receive test features for the full post-July 31st period, up until the conclusion of that not-yet-specified evaluation timeframe, even if there are gap dates which do not count towards scoring. So if the evaluation period runs from Aug 7-20, you will still receive test features (without ground truth) for the full timeframe of Aug 1-20, even though you will only be scored the subset of that.", "1400920": "Thanks Julia. This makes sense now :)", "1402669": "Dear Kaggle Staff\n\nI checked the following page and almost understood it.\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/data\n\nHowever, one point is that I don't understand the behavior of the data in the mlb module when you rerun the program.\n\nSpecifically, there is a program example of how to submit using the mlb module on the following page, \nbut I don't know what period of time the data is stored in the iter_test variable.\nhttps://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/overview/evaluation\n\n> import mlb\nenv = mlb.make_env() # initialize the environment\niter_test = env.iter_test() # iterator which loops over each date in test set\nfor (test_df, sample_prediction_df) in iter_test:\n sample_prediction_df['target1'] = 100 #make predictions here\n env.predict(sample_prediction_df)\n\nFor example, when you re-run the notebook on 8/7 first, \nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/7 (except for nextDayPlayerEngagement) and the data for submitting prediction scores?\nBy running \"for (test_df, sample_prediction_df) in iter_test:\", \ncan we refer to each data for one day at a time in date order?\n\nAlso, if you re-run the notebook for the second time on 8/21, \nis the iter_test variable contains the data from the train.csv layout from 8/1 to 8/21 (except for nextDayPlayerEngagement) and the data for submitting prediction scores or from 8/8 to 8/21?\n\nAs for running the notebook after the deadline,\nI believe that the env of the mlb module will be only updated to the latest version, \nbut the other input files will not be updated.\n\nIs any of the above incorrect?\n\nI would appreciate it if you could help me.\nThank you very much for your help." }, "source": "meta" }