Hi. My first submission test ended with error \"Notebook Exceeded Allowed Compute\". I made a second attempt with the same notebook, but using GPU. The result was the same.
\nThis caught me by surprise because:
\nI understand the error messages are vague to avoid poking into the scoring data. My ability to run tests is constrained by the 5 submissions per day limit. I'd greatly appreciate if the kaggle team can provide any pointers as to what the problem might be. The notebook link is https://www.kaggle.com/vialactea/mlb-submit-2. It is private, but I assume you have access to all notebooks regardless of visibility. If that is not the case, please let me know.
\nAny advice from fellow kagglers is also greatly appreciated.
\nRegards
", "messages": [ { "id": 1390486, "postDate": "2021-07-16T17:21:38.510Z", "content": "Hey,
\nI've been getting the 'Notebook Exceeded Allowed Compute' error frequently in this competition due to having duplicate playerId in the feature matrix during a given prediction step. In my opinion this should be giving 'Submission Scoring Error' rather than compute. However I suppose it depends on how you attach your predictions to the submission df.
\nI would recommend prior to each join in your test pipeline, either dropping duplicates on playerId or doing a group by with some meaningful aggregation depending on the table causing you issue.
", "rawMarkdown": "Hey,\n\nI've been getting the 'Notebook Exceeded Allowed Compute' error frequently in this competition due to having duplicate playerId in the feature matrix during a given prediction step. In my opinion this should be giving 'Submission Scoring Error' rather than compute. However I suppose it depends on how you attach your predictions to the submission df.\n\nI would recommend prior to each join in your test pipeline, either dropping duplicates on playerId or doing a group by with some meaningful aggregation depending on the table causing you issue.", "votes": 2, "replies": [ { "id": 1390516, "postDate": "2021-07-16T18:09:31.343Z", "content": "Thanks. This aligns with my suspicion that the issue is with my submission data, rather than with the resources my notebook uses. That helps a lot. I wish this competition would not count errored submissions towards the daily limit. In my first competition that was the case (this is my 2nd). Next time, I'll do it sooner so that I have time (and submission attempts) to run tests.
\nA couple of days ago I ran into issues with duplicates during training and I thought I had none left, but upon a quick review I noticed that was not the case. I'll fix it and I'll give it a go.
\nBefore reading your message I was wondering if the problem was caused by my assumption that each call to iter_test only returns one date. It makes sense to avoid peeking into the future, but maybe that is not how it works. Would you know if only one date is returned each time?
", "rawMarkdown": "Thanks. This aligns with my suspicion that the issue is with my submission data, rather than with the resources my notebook uses. That helps a lot. I wish this competition would not count errored submissions towards the daily limit. In my first competition that was the case (this is my 2nd). Next time, I'll do it sooner so that I have time (and submission attempts) to run tests.\n\nA couple of days ago I ran into issues with duplicates during training and I thought I had none left, but upon a quick review I noticed that was not the case. I'll fix it and I'll give it a go. \n\nBefore reading your message I was wondering if the problem was caused by my assumption that each call to iter_test only returns one date. It makes sense to avoid peeking into the future, but maybe that is not how it works. Would you know if only one date is returned each time?" }, { "id": 1390558, "postDate": "2021-07-16T19:06:12.453Z", "content": "@vialactea I am fairly certain it is one date per call since my test pipeline only does joins on playerId and not playerId and date as my train pipeline does. I think if there were multiple dates in a single call my test pipeline would fail.
", "rawMarkdown": "@vialactea I am fairly certain it is one date per call since my test pipeline only does joins on playerId and not playerId and date as my train pipeline does. I think if there were multiple dates in a single call my test pipeline would fail. ", "votes": 1 } ] }, { "id": 1390299, "postDate": "2021-07-16T14:33:43.817Z", "content": "Hi. My first submission test ended with error \"Notebook Exceeded Allowed Compute\". I made a second attempt with the same notebook, but using GPU. The result was the same.
\nThis caught me by surprise because:
\nI understand the error messages are vague to avoid poking into the scoring data. My ability to run tests is constrained by the 5 submissions per day limit. I'd greatly appreciate if the kaggle team can provide any pointers as to what the problem might be. The notebook link is https://www.kaggle.com/vialactea/mlb-submit-2. It is private, but I assume you have access to all notebooks regardless of visibility. If that is not the case, please let me know.
\nAny advice from fellow kagglers is also greatly appreciated.
\nRegards
", "rawMarkdown": "Hi. My first submission test ended with error \"Notebook Exceeded Allowed Compute\". I made a second attempt with the same notebook, but using GPU. The result was the same.\n\nThis caught me by surprise because:\n- The notebook took <3m to run when I saved it with CPU.\n- Each prediction takes less than 1s with CPU. I assume the scoring will be done in a date range no larger than 3 months. That means the notebook should run in a few minutes, well below the 6h limit.\n- The memory usage before starting the predictions is 1.5 GB. In the few examples provided by iter_test it never exceeds that. I saved 4 of the entries provided by iter_test, made deep copies to create two samples of 120 and 400 days, ran the prediction on these samples and in both cases the memory usage stayed exactly the same (1.5 GB).\n- I'm using a simple pretrained keras model for the predictions (<1M parameters). The model input has a shape of about 2000x300. My intention with this submission was just to test the waters on the process before moving to more sophisticated models.\n- I wonder if the problem is lack of resources or something else, such as some issue with the submission data.\n\nI understand the error messages are vague to avoid poking into the scoring data. My ability to run tests is constrained by the 5 submissions per day limit. I'd greatly appreciate if the kaggle team can provide any pointers as to what the problem might be. The notebook link is https://www.kaggle.com/vialactea/mlb-submit-2. It is private, but I assume you have access to all notebooks regardless of visibility. If that is not the case, please let me know.\n\nAny advice from fellow kagglers is also greatly appreciated.\n\nRegards" }, { "id": 1393604, "postDate": "2021-07-19T18:52:33.223Z", "content": "Very strange that folks who would be smart enough to probe this test set would do it just to get a better OPTICS on the leader board during the public test period. Seems like a heck of a lot of probing needed to get a perfect score - that's some talent or perhaps its just a lot easier to cheat than I think it is?
\nIs all this effort to prevent probing intended to keep GM's and Experts from cheating each other?
", "rawMarkdown": "Very strange that folks who would be smart enough to probe this test set would do it just to get a better OPTICS on the leader board during the public test period. Seems like a heck of a lot of probing needed to get a perfect score - that's some talent or perhaps its just a lot easier to cheat than I think it is?\n\nIs all this effort to prevent probing intended to keep GM's and Experts from cheating each other? " }, { "id": 1391567, "postDate": "2021-07-17T18:01:14.030Z", "content": "The notebook in the link is not accessible
", "rawMarkdown": "The notebook in the link is not accessible" }, { "id": 1390446, "postDate": "2021-07-16T16:45:35.370Z", "content": "Sorry, but out of fairness, we are not in a position to debug your specific code/submission. Instead, we encourage you to review our code debugging tips or seek the support of others on the forums.
", "rawMarkdown": "Sorry, but out of fairness, we are not in a position to debug your specific code/submission. Instead, we encourage you to review our [code debugging tips](https://www.kaggle.com/code-competition-debugging) or seek the support of others on the forums.", "replies": [ { "id": 1390526, "postDate": "2021-07-16T18:24:21.123Z", "content": "Thanks for the quick response. I understand your point.
\nA fellow kaggler gave me the pointer I was looking for: the issue might be with the submission's data and not necessarily with the resources used. I'll focus my tests on that possibility.
\nI'd like to take the opportunity to leave two suggestions for future competitions:
\nJulia - your not in a position to help debug but you are in a position to NOT COUNT failed attempts as part of the daily quota.
\nI assume you could make that change today.
\nProviding more help by way of error information would certaily be a longer time frame task - but IMO these API type submissions are chasing away the rookies - heck I been here for several years and I HATE them and suffer my fair share of submission failures.
\nI assume API's are the way of the future - how about some more help with errors codes?
", "rawMarkdown": "[Julia](https://www.kaggle.com/juliaelliott) - your not in a position to help debug but you are in a position to NOT COUNT failed attempts as part of the daily quota.\n\nI assume you could make that change today.\n\nProviding more help by way of error information would certaily be a longer time frame task - but IMO these API type submissions are chasing away the rookies - heck I been here for several years and I HATE them and suffer my fair share of submission failures.\n\nI assume API's are the way of the future - how about some more help with errors codes?" }, { "id": 1390615, "postDate": "2021-07-16T20:58:18.800Z", "content": "This has been discussed a lot in past competitions but the problem with not counting failed submissions and/or having more specific error messages is that it can be used for probing private datasets.
", "rawMarkdown": "This has been discussed a lot in past competitions but the problem with not counting failed submissions and/or having more specific error messages is that it can be used for probing private datasets.", "votes": 2 }, { "id": 1390634, "postDate": "2021-07-16T21:57:15.917Z", "content": "I get what you are saying, although I'm not sure you can fully prevent probing. For example, one can build a simple and very efficient model (with no possibility of any unintentional errors) that depending on the data just raises an exception or sleeps \"forever\". You can certainly make the effort less efficient by having just one error or a small set of errors with a \"fuzzy\" meaning.
\nIf you ask me, I'd rather have some more help fixing submission errors, even if it comes at a higher risk of someone probing. I got involved in competitions for the fun and learning experience. Obviously, I'll be more pleased if I get a good position on a competition, but I care a lot more about what I learn and submission errors get in the way.
\nIn the specific case of this competition, I'd think that there would be little to no harm from probing anyway:
\nIn case it helps others, in my 3rd attempt I couldn't fix my problem by eliminating the only obvious source of duplication left, but at least in my 4th attempt I confirmed my problem is in the submission data - I got a successful submission by keeping all my code and replacing my submission with sample_prediction_dffilled with 0s. I'm glad I didn't spend my time and submission quota trying to optimize resources. Thanks @jacobhowardparker.
Now I need to figure out how I'm going to use my 5th and last attempt for today to learn more about my problem.
", "rawMarkdown": "I get what you are saying, although I'm not sure you can fully prevent probing. For example, one can build a simple and very efficient model (with no possibility of any unintentional errors) that depending on the data just raises an exception or sleeps \"forever\". You can certainly make the effort less efficient by having just one error or a small set of errors with a \"fuzzy\" meaning.\n\nIf you ask me, I'd rather have some more help fixing submission errors, even if it comes at a higher risk of someone probing. I got involved in competitions for the fun and learning experience. Obviously, I'll be more pleased if I get a good position on a competition, but I care a lot more about what I learn and submission errors get in the way.\n\nIn the specific case of this competition, I'd think that there would be little to no harm from probing anyway:\n- You can only probe the data in the public LB and the evaluation will be based on the data in the private LB, which I understood is entirely different. You could still use the probing to build lagging features for future dates, but I wonder if that could be done in a way that would make a difference.\n- I believe our inputs have no bearing in past engagement. People engage based on what already happened, not on what will occur in a future they ignore (with a caveat for games early enough in the day to make a difference for that day). Obviously this wouldn't be the case in other types of competitions, e.g., those involving the prediction of financial market assets.\n\nIn case it helps others, in my 3rd attempt I couldn't fix my problem by eliminating the only obvious source of duplication left, but at least in my 4th attempt I confirmed my problem is in the submission data - I got a successful submission by keeping all my code and replacing my submission with `sample_prediction_df `filled with 0s. I'm glad I didn't spend my time and submission quota trying to optimize resources. Thanks @jacobhowardparker.\n\nNow I need to figure out how I'm going to use my 5th and last attempt for today to learn more about my problem." }, { "id": 1390645, "postDate": "2021-07-16T22:28:07.533Z", "content": "At what point does the cure become worse than the disease. A percentage of the world consists of POS - they will do all possible to cheat.
\nThe other end of the range are the folks wanting to learn ML - for many of the rookies on this site the only successful submissions they have are forks with almost no changes. If these type of API's had been the rule when I first start a couple of years ago - for me - it would have been a very short start and finish.
\nAlways end up as a question - who is the target audience for the product. Kaggle making it clear to me that the target is not novice learners - it's professional staffs and companies.
", "rawMarkdown": "At what point does the cure become worse than the disease. A percentage of the world consists of POS - they will do all possible to cheat. \n\nThe other end of the range are the folks wanting to learn ML - for many of the rookies on this site the only successful submissions they have are forks with almost no changes. If these type of API's had been the rule when I first start a couple of years ago - for me - it would have been a very short start and finish.\n\nAlways end up as a question - who is the target audience for the product. Kaggle making it clear to me that the target is not novice learners - it's professional staffs and companies. " }, { "id": 1390651, "postDate": "2021-07-16T23:14:31.027Z", "content": "The private data set for this MLB has yet to occur - pretty darn hard to probe it.
", "rawMarkdown": "The private data set for this MLB has yet to occur - pretty darn hard to probe it.", "votes": 2 }, { "id": 1391177, "postDate": "2021-07-17T11:12:44.973Z", "content": "@juliaelliott I mentioned this in another thread but I think a lot of the issues stem from having such a small public test set (5? days it runs on during commit), which also happens to not contain most of the things that cause trip-ups.
\nI wonder if in future API competitions we can get a slightly larger public test to make debugging easier?
\nI understand in this situation it is difficult with data so recent but even an additional 5 days may have helped?
", "rawMarkdown": "@juliaelliott I mentioned this in another thread but I think a lot of the issues stem from having such a small public test set (5? days it runs on during commit), which also happens to not contain most of the things that cause trip-ups. \n\nI wonder if in future API competitions we can get a slightly larger public test to make debugging easier? \n\nI understand in this situation it is difficult with data so recent but even an additional 5 days may have helped?" }, { "id": 1391770, "postDate": "2021-07-18T04:20:04.460Z", "content": "I had posted here but no response recently on that thread on evaluation phase (which still seems a bit vague) -
\n\"Could I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right? \"
", "rawMarkdown": "I had posted [here](https://www.kaggle.com/c/mlb-player-digital-engagement-forecasting/discussion/251126#1381892) but no response recently on that thread on evaluation phase (which still seems a bit vague) -\n\n\"Could I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right? \"" }, { "id": 1391830, "postDate": "2021-07-18T06:03:11.903Z", "content": "Can probably still do submissions but you need to keep May and later in your validation hold-out so you can get an idea of where you would have scored - keeping checking the LB and write down the last real test - when the update occurs people will use it to jump up the leader board.
", "rawMarkdown": "Can probably still do submissions but you need to keep May and later in your validation hold-out so you can get an idea of where you would have scored - keeping checking the LB and write down the last real test - when the update occurs people will use it to jump up the leader board.\n" }, { "id": 1393586, "postDate": "2021-07-19T18:37:39.350Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1393707, "postDate": "2021-07-19T20:12:22.977Z", "content": "@jacobhowardparker Fully understand your struggle and hear your feedback. We agonized extensively over the design of the competition to balance multiple factors. We recognize that from a real-time debugging standpoint, the public test is not of an ideal size. Unfortunately, we also have to preemptively combat probing and the optics of a public leaderboard which is rendered meaningless due to perfect scores. When we have a larger overall test set at our disposal which is not reliant on \"future data,\" we are at liberty to be more flexible in our design & split.
", "rawMarkdown": "@jacobhowardparker Fully understand your struggle and hear your feedback. We agonized extensively over the design of the competition to balance multiple factors. We recognize that from a real-time debugging standpoint, the public test is not of an ideal size. Unfortunately, we also have to preemptively combat probing and the optics of a public leaderboard which is rendered meaningless due to perfect scores. When we have a larger overall test set at our disposal which is not reliant on \"future data,\" we are at liberty to be more flexible in our design & split. ", "votes": 1 } ] } ], "comments": [ { "id": 1390486, "author_name": "JacobW", "author_url": "", "post_date": "2021-07-16T17:21:38.510000", "content": "Hey,
\nI've been getting the 'Notebook Exceeded Allowed Compute' error frequently in this competition due to having duplicate playerId in the feature matrix during a given prediction step. In my opinion this should be giving 'Submission Scoring Error' rather than compute. However I suppose it depends on how you attach your predictions to the submission df.
\nI would recommend prior to each join in your test pipeline, either dropping duplicates on playerId or doing a group by with some meaningful aggregation depending on the table causing you issue.
", "votes": 2, "replies": [ { "id": 1390516, "author_name": "vialactea", "author_url": "", "post_date": "2021-07-16T18:09:31.343000", "content": "Thanks. This aligns with my suspicion that the issue is with my submission data, rather than with the resources my notebook uses. That helps a lot. I wish this competition would not count errored submissions towards the daily limit. In my first competition that was the case (this is my 2nd). Next time, I'll do it sooner so that I have time (and submission attempts) to run tests.
\nA couple of days ago I ran into issues with duplicates during training and I thought I had none left, but upon a quick review I noticed that was not the case. I'll fix it and I'll give it a go.
\nBefore reading your message I was wondering if the problem was caused by my assumption that each call to iter_test only returns one date. It makes sense to avoid peeking into the future, but maybe that is not how it works. Would you know if only one date is returned each time?
", "votes": 0, "replies": [] }, { "id": 1390558, "author_name": "JacobW", "author_url": "", "post_date": "2021-07-16T19:06:12.453000", "content": "@vialactea I am fairly certain it is one date per call since my test pipeline only does joins on playerId and not playerId and date as my train pipeline does. I think if there were multiple dates in a single call my test pipeline would fail.
", "votes": 1, "replies": [] } ] }, { "id": 1393604, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-07-19T18:52:33.223000", "content": "Very strange that folks who would be smart enough to probe this test set would do it just to get a better OPTICS on the leader board during the public test period. Seems like a heck of a lot of probing needed to get a perfect score - that's some talent or perhaps its just a lot easier to cheat than I think it is?
\nIs all this effort to prevent probing intended to keep GM's and Experts from cheating each other?
", "votes": 0, "replies": [] }, { "id": 1391567, "author_name": "Lu Bin Liu", "author_url": "", "post_date": "2021-07-17T18:01:14.030000", "content": "The notebook in the link is not accessible
", "votes": 0, "replies": [] }, { "id": 1390446, "author_name": "Julia Elliott", "author_url": "", "post_date": "2021-07-16T16:45:35.370000", "content": "Sorry, but out of fairness, we are not in a position to debug your specific code/submission. Instead, we encourage you to review our code debugging tips or seek the support of others on the forums.
", "votes": 0, "replies": [ { "id": 1390526, "author_name": "vialactea", "author_url": "", "post_date": "2021-07-16T18:24:21.123000", "content": "Thanks for the quick response. I understand your point.
\nA fellow kaggler gave me the pointer I was looking for: the issue might be with the submission's data and not necessarily with the resources used. I'll focus my tests on that possibility.
\nI'd like to take the opportunity to leave two suggestions for future competitions:
\nJulia - your not in a position to help debug but you are in a position to NOT COUNT failed attempts as part of the daily quota.
\nI assume you could make that change today.
\nProviding more help by way of error information would certaily be a longer time frame task - but IMO these API type submissions are chasing away the rookies - heck I been here for several years and I HATE them and suffer my fair share of submission failures.
\nI assume API's are the way of the future - how about some more help with errors codes?
", "votes": 0, "replies": [] }, { "id": 1390615, "author_name": "Christoffer Karlsson", "author_url": "", "post_date": "2021-07-16T20:58:18.800000", "content": "This has been discussed a lot in past competitions but the problem with not counting failed submissions and/or having more specific error messages is that it can be used for probing private datasets.
", "votes": 2, "replies": [] }, { "id": 1390634, "author_name": "vialactea", "author_url": "", "post_date": "2021-07-16T21:57:15.917000", "content": "I get what you are saying, although I'm not sure you can fully prevent probing. For example, one can build a simple and very efficient model (with no possibility of any unintentional errors) that depending on the data just raises an exception or sleeps \"forever\". You can certainly make the effort less efficient by having just one error or a small set of errors with a \"fuzzy\" meaning.
\nIf you ask me, I'd rather have some more help fixing submission errors, even if it comes at a higher risk of someone probing. I got involved in competitions for the fun and learning experience. Obviously, I'll be more pleased if I get a good position on a competition, but I care a lot more about what I learn and submission errors get in the way.
\nIn the specific case of this competition, I'd think that there would be little to no harm from probing anyway:
\nIn case it helps others, in my 3rd attempt I couldn't fix my problem by eliminating the only obvious source of duplication left, but at least in my 4th attempt I confirmed my problem is in the submission data - I got a successful submission by keeping all my code and replacing my submission with sample_prediction_dffilled with 0s. I'm glad I didn't spend my time and submission quota trying to optimize resources. Thanks @jacobhowardparker.
Now I need to figure out how I'm going to use my 5th and last attempt for today to learn more about my problem.
", "votes": 0, "replies": [] }, { "id": 1390645, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-07-16T22:28:07.533000", "content": "At what point does the cure become worse than the disease. A percentage of the world consists of POS - they will do all possible to cheat.
\nThe other end of the range are the folks wanting to learn ML - for many of the rookies on this site the only successful submissions they have are forks with almost no changes. If these type of API's had been the rule when I first start a couple of years ago - for me - it would have been a very short start and finish.
\nAlways end up as a question - who is the target audience for the product. Kaggle making it clear to me that the target is not novice learners - it's professional staffs and companies.
", "votes": 0, "replies": [] }, { "id": 1390651, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-07-16T23:14:31.027000", "content": "The private data set for this MLB has yet to occur - pretty darn hard to probe it.
", "votes": 2, "replies": [] }, { "id": 1391177, "author_name": "JacobW", "author_url": "", "post_date": "2021-07-17T11:12:44.973000", "content": "@juliaelliott I mentioned this in another thread but I think a lot of the issues stem from having such a small public test set (5? days it runs on during commit), which also happens to not contain most of the things that cause trip-ups.
\nI wonder if in future API competitions we can get a slightly larger public test to make debugging easier?
\nI understand in this situation it is difficult with data so recent but even an additional 5 days may have helped?
", "votes": 0, "replies": [] }, { "id": 1391770, "author_name": "something4kag", "author_url": "", "post_date": "2021-07-18T04:20:04.460000", "content": "I had posted here but no response recently on that thread on evaluation phase (which still seems a bit vague) -
\n\"Could I suggest that maybe when the roughly up to July 20th data is available it is first tried as a new test set like currently the May data is? That might give people an idea if their code is going to break before other deadlines. It would seem like once that training set update occurs there will be no more LB for submissions right? \"
", "votes": 0, "replies": [] }, { "id": 1391830, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-07-18T06:03:11.903000", "content": "Can probably still do submissions but you need to keep May and later in your validation hold-out so you can get an idea of where you would have scored - keeping checking the LB and write down the last real test - when the update occurs people will use it to jump up the leader board.
", "votes": 0, "replies": [] }, { "id": 1393586, "author_name": "", "author_url": "", "post_date": "2021-07-19T18:37:39.350000", "content": "", "votes": 0, "replies": [] }, { "id": 1393707, "author_name": "Julia Elliott", "author_url": "", "post_date": "2021-07-19T20:12:22.977000", "content": "@jacobhowardparker Fully understand your struggle and hear your feedback. We agonized extensively over the design of the competition to balance multiple factors. We recognize that from a real-time debugging standpoint, the public test is not of an ideal size. Unfortunately, we also have to preemptively combat probing and the optics of a public leaderboard which is rendered meaningless due to perfect scores. When we have a larger overall test set at our disposal which is not reliant on \"future data,\" we are at liberty to be more flexible in our design & split.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1390486": "Hey,\n\nI've been getting the 'Notebook Exceeded Allowed Compute' error frequently in this competition due to having duplicate playerId in the feature matrix during a given prediction step. In my opinion this should be giving 'Submission Scoring Error' rather than compute. However I suppose it depends on how you attach your predictions to the submission df.\n\nI would recommend prior to each join in your test pipeline, either dropping duplicates on playerId or doing a group by with some meaningful aggregation depending on the table causing you issue.", "1390299": "Hi. My first submission test ended with error \"Notebook Exceeded Allowed Compute\". I made a second attempt with the same notebook, but using GPU. The result was the same.\n\nThis caught me by surprise because:\n- The notebook took <3m to run when I saved it with CPU.\n- Each prediction takes less than 1s with CPU. I assume the scoring will be done in a date range no larger than 3 months. That means the notebook should run in a few minutes, well below the 6h limit.\n- The memory usage before starting the predictions is 1.5 GB. In the few examples provided by iter_test it never exceeds that. I saved 4 of the entries provided by iter_test, made deep copies to create two samples of 120 and 400 days, ran the prediction on these samples and in both cases the memory usage stayed exactly the same (1.5 GB).\n- I'm using a simple pretrained keras model for the predictions (<1M parameters). The model input has a shape of about 2000x300. My intention with this submission was just to test the waters on the process before moving to more sophisticated models.\n- I wonder if the problem is lack of resources or something else, such as some issue with the submission data.\n\nI understand the error messages are vague to avoid poking into the scoring data. My ability to run tests is constrained by the 5 submissions per day limit. I'd greatly appreciate if the kaggle team can provide any pointers as to what the problem might be. The notebook link is https://www.kaggle.com/vialactea/mlb-submit-2. It is private, but I assume you have access to all notebooks regardless of visibility. If that is not the case, please let me know.\n\nAny advice from fellow kagglers is also greatly appreciated.\n\nRegards", "1393604": "Very strange that folks who would be smart enough to probe this test set would do it just to get a better OPTICS on the leader board during the public test period. Seems like a heck of a lot of probing needed to get a perfect score - that's some talent or perhaps its just a lot easier to cheat than I think it is?\n\nIs all this effort to prevent probing intended to keep GM's and Experts from cheating each other? ", "1391567": "The notebook in the link is not accessible", "1390446": "Sorry, but out of fairness, we are not in a position to debug your specific code/submission. Instead, we encourage you to review our [code debugging tips](https://www.kaggle.com/code-competition-debugging) or seek the support of others on the forums." } }