\n@sohier
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1611824%2Fa8b62bfef2d1d84751ab5840b04180a3%2FScreen%20Shot%202020-11-11%20at%201.04.13%20PM.png?generation=1605096393773567&alt=media\")
I've uploaded an update to respond to the issues raised here. It will start affecting new submissions in the next half an hour or so.
\nYour notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a simple sample submission still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.
\nPut differently, you will now see your results after the real runtime plus some random additional time, with the random additional time never pushing the total over nine hours.
The updates only affect the hidden rerun version of the API.
I'll rerun all notebooks published during the relevant time window over the rest of this week.
As a reminder, we will be reviewing submissions for cheating attempts at the end of the competition. Any other vulnerabilities that are discovered and applied but not reported to us before then may lead to a ban from the competition and other sanctions.
Edit: I introduced a bug in the initial copy of the update. That should be fixed as of 13:00 PST today. Apologies for the inconvenience!
", "messages": [ { "id": 1073758, "postDate": "2020-11-09T22:24:06.647Z", "content": "I've uploaded an update to respond to the issues raised here. It will start affecting new submissions in the next half an hour or so.
\nYour notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a simple sample submission still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.
\nPut differently, you will now see your results after the real runtime plus some random additional time, with the random additional time never pushing the total over nine hours.
The updates only affect the hidden rerun version of the API.
I'll rerun all notebooks published during the relevant time window over the rest of this week.
As a reminder, we will be reviewing submissions for cheating attempts at the end of the competition. Any other vulnerabilities that are discovered and applied but not reported to us before then may lead to a ban from the competition and other sanctions.
Edit: I introduced a bug in the initial copy of the update. That should be fixed as of 13:00 PST today. Apologies for the inconvenience!
", "rawMarkdown": "I've uploaded an update to respond to [the issues raised here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195815). It will start affecting new submissions in the next half an hour or so.\n\n- Your notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a [simple sample submission](https://www.kaggle.com/sohier/quick-sample-submission) still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.\n Put differently, you will now see your results after the real runtime plus some random additional time, with the random additional time never pushing the total over nine hours.\n\n- The updates only affect the hidden rerun version of the API.\n- I'll rerun all notebooks published during the relevant time window over the rest of this week.\n- As a reminder, we will be reviewing submissions for cheating attempts at the end of the competition. Any other vulnerabilities that are discovered and applied but not reported to us before then may lead to a ban from the competition and other sanctions.\n\nEdit: I introduced a bug in the initial copy of the update. That should be fixed as of 13:00 PST today. Apologies for the inconvenience!\n", "votes": 59 }, { "id": 1074153, "postDate": "2020-11-10T10:17:11.187Z", "content": "Why my Previously successfully executed notebook failed using new api?
", "rawMarkdown": "Why my Previously successfully executed notebook failed using new api?", "votes": 9, "replies": [ { "id": 1074164, "postDate": "2020-11-10T10:39:51.697Z", "content": "also get error with new api with a notebook previously run successfully … Submission Scoring Error.
", "rawMarkdown": "also get error with new api with a notebook previously run successfully ... Submission Scoring Error.", "votes": 1 }, { "id": 1074236, "postDate": "2020-11-10T12:18:31.233Z", "content": "At least you have a successful run :) Mine got Submission Scoring Error after 3 hours of submitting :?
", "rawMarkdown": "At least you have a successful run :) Mine got Submission Scoring Error after 3 hours of submitting :?", "votes": 1 }, { "id": 1074391, "postDate": "2020-11-10T15:48:48.237Z", "content": "Same here after seeing the other comment, I resubmitted my best submission to date and get the \"Submission Scoring Error\" message. 😔
", "rawMarkdown": "Same here after seeing the other comment, I resubmitted my best submission to date and get the \"Submission Scoring Error\" message. 😔", "votes": 1 }, { "id": 1087076, "postDate": "2020-11-22T10:13:13.890Z", "content": "I think some new lectures are added in the test data ?
", "rawMarkdown": "I think some new lectures are added in the test data ? " } ] }, { "id": 1075024, "postDate": "2020-11-11T10:28:30.640Z", "content": "The same version notebook LB 0.757->0.758, submit time about 1h -> about 3h
\nIt seems much more slower
\nI need to find a faster way
I can confirm this, submissions take considerately longer now, ~1h -> ~4h
", "rawMarkdown": "I can confirm this, submissions take considerately longer now, ~1h -> ~4h", "votes": 2 } ] }, { "id": 1074186, "postDate": "2020-11-10T11:07:02.310Z", "content": "Get error with new api with a notebook previously run successfully … Submission Scoring Error.
", "rawMarkdown": "Get error with new api with a notebook previously run successfully … Submission Scoring Error.", "votes": 6 }, { "id": 1074115, "postDate": "2020-11-10T09:16:40.127Z", "content": "Thank you for the quick fix @sohier
\nI have some questions about
\n\n\n
\n- Your notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a simple sample submission still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.
\n
I'm a little unsure of what this means. As far as I knew once we submitted it would finish within the next 9hr or we get a \"Notebook Exceeded Allowed Compute\" error.
\n2 hours past, still running. My implementation is quite fast, I monitored the time of predicting on sample submission:
\n[INFO 2020-11-15 01:18:14.094 I]: total iterations: 4, cost 0.1564 secs, 0.0391 secs/iteration\nIf the group size is the same, the iterations of the private dataset are unbelievably large.
\nThis submission utilizes offline generated data and model, all the data loading and statistics calculating can be done in a few seconds.
", "rawMarkdown": "2 hours past, still running. My implementation is quite fast, I monitored the time of predicting on sample submission:\n```\n[INFO 2020-11-15 01:18:14.094 I]: total iterations: 4, cost 0.1564 secs, 0.0391 secs/iteration\n```\nIf the group size is the same, the iterations of the private dataset are unbelievably large.\n\nThis submission utilizes offline generated data and model, all the data loading and statistics calculating can be done in a few seconds.", "votes": 3 }, { "id": 1075126, "postDate": "2020-11-11T12:12:33.107Z", "content": "Still getting Submission Scoring Error if we submit Simple submissions from CSV file
\n@sohier
\n
I put the submission result manually and it still making error
\nhttps://www.kaggle.com/ahmed3991/quick-sample-submission-error
@ahmed3991 you can't do this for submission. when submit, there is a hidden test dataset, which is different from the example_test.cvs.
", "rawMarkdown": "@ahmed3991 you can't do this for submission. when submit, there is a hidden test dataset, which is different from the example_test.cvs.", "votes": 2 }, { "id": 1075502, "postDate": "2020-11-11T18:50:07.247Z", "content": "I know, But the hidden test Dataset now is the same example_test.csv.
", "rawMarkdown": "I know, But the hidden test Dataset now is the same **example_test.csv**." }, { "id": 1075506, "postDate": "2020-11-11T18:52:57.887Z", "content": "@yihdarshieh You can check it in the Notebooks output with the row_id
", "rawMarkdown": "@yihdarshieh You can check it in the Notebooks output with the row_id" }, { "id": 1075544, "postDate": "2020-11-11T19:14:25.483Z", "content": "you can't modify the content of hidden dataset, it makes no sense.
", "rawMarkdown": "you can't modify the content of hidden dataset, it makes no sense.", "votes": 2 } ] }, { "id": 1074251, "postDate": "2020-11-10T12:40:11.820Z", "content": "Submission Scoring Error… I don't think I've changed anything
", "rawMarkdown": "Submission Scoring Error... I don't think I've changed anything", "votes": 3 }, { "id": 1074249, "postDate": "2020-11-10T12:34:29.693Z", "content": "Cannot make a successful submit neither, even with previous successful submission.
", "rawMarkdown": "Cannot make a successful submit neither, even with previous successful submission.\n", "votes": 3 }, { "id": 1075237, "postDate": "2020-11-11T14:11:00.033Z", "content": "Thank you for updating.
\nI have confirmed that if the inference is < 0.55 sec/iter, it is submitted successfully even for new API.
\nhttps://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter?scriptVersionId=46552376
What was the run time of your submission with time.sleep(0.6) ? Was it 9 hours or you got the error message earlier?
", "rawMarkdown": "What was the run time of your submission with time.sleep(0.6) ? Was it 9 hours or you got the error message earlier?" }, { "id": 1075310, "postDate": "2020-11-11T15:15:46.990Z", "content": "I don't check it for new API, but the notebook kept running for 9 hours (or more??) and I got error message 'Submission Scoring Error' and 'No additional message for this error' for previous API.
", "rawMarkdown": "I don't check it for new API, but the notebook kept running for 9 hours (or more??) and I got error message 'Submission Scoring Error' and 'No additional message for this error' for previous API.", "votes": 1 } ] }, { "id": 1074356, "postDate": "2020-11-10T15:05:10.573Z", "content": "Even \"Quick Sample Submission\" has the same Error.
\nSo let's wait for the fix.
Thanks for flagging, taking a look now.
", "rawMarkdown": "Thanks for flagging, taking a look now.", "votes": 3 }, { "id": 1074481, "postDate": "2020-11-10T17:37:55.643Z", "content": "I've posted a patch for that problem, but it will take a bit longer to validate the fix with the quick sample submission.
", "rawMarkdown": "I've posted a patch for that problem, but it will take a bit longer to validate the fix with the quick sample submission." }, { "id": 1074486, "postDate": "2020-11-10T17:42:08.830Z", "content": "@sohier once the patch is confirmed to work, could you also update this thread, so we can start to submit again, please?
", "rawMarkdown": "@sohier once the patch is confirmed to work, could you also update this thread, so we can start to submit again, please?" }, { "id": 1074489, "postDate": "2020-11-10T17:51:55.660Z", "content": "Yes, I'll provide an update then.
", "rawMarkdown": "Yes, I'll provide an update then. ", "votes": 2 }, { "id": 1074548, "postDate": "2020-11-10T19:42:14.147Z", "content": "The patch is mostly working well, but I'm tweaking one parameter now.
", "rawMarkdown": "The patch is mostly working well, but I'm tweaking one parameter now.", "votes": 1 }, { "id": 1074601, "postDate": "2020-11-10T20:57:53.737Z", "content": "The version with the parameter tweak has passing submissions, it should be alright to resume normal submissions. I've gone ahead and dropped the 1.00 submission from the leaderboard.
", "rawMarkdown": "The version with the parameter tweak has passing submissions, it should be alright to resume normal submissions. I've gone ahead and dropped the 1.00 submission from the leaderboard. ", "votes": 4 }, { "id": 1074677, "postDate": "2020-11-10T23:42:00.567Z", "content": "Cool. Thanks
", "rawMarkdown": "Cool. Thanks", "votes": 1 } ] }, { "id": 1074188, "postDate": "2020-11-10T11:08:16.370Z", "content": "me too.The official submission also has the same problem
", "rawMarkdown": "me too.The official submission also has the same problem", "votes": 4 }, { "id": 1074184, "postDate": "2020-11-10T11:04:59.703Z", "content": "Same problem here! I get Submission Scoring Error.
", "rawMarkdown": "Same problem here! I get Submission Scoring Error.", "votes": 4, "replies": [ { "id": 1075010, "postDate": "2020-11-11T10:08:59.260Z", "content": "I can submit now.
", "rawMarkdown": "I can submit now." } ] }, { "id": 1075665, "postDate": "2020-11-11T21:38:09.460Z", "content": "I get a notebook timeout
", "rawMarkdown": "I get a notebook timeout ", "votes": 1, "replies": [ { "id": 1075675, "postDate": "2020-11-11T21:43:48.080Z", "content": "Was the previous submission run of that notebook close to 9 hours? Not sure, but in this case, you might get timeout. According to @sohier,
\n\n\nThe real runtime has barely changed, but the reported runtime has been obfuscated. For example, if your real runtime was one hour you might see two hours. If your real runtime was 8 hours, you could see anything between 8 and 9 hours.
\n
If the real runtime is close to 9 hours, I don't know if the extra waiting still occurs.
", "rawMarkdown": "Was the previous submission run of that notebook close to 9 hours? Not sure, but in this case, you might get timeout. According to @sohier, \n\n> The real runtime has barely changed, but the reported runtime has been obfuscated. For example, if your real runtime was one hour you might see two hours. If your real runtime was 8 hours, you could see anything between 8 and 9 hours.\n\nIf the real runtime is close to 9 hours, I don't know if the extra waiting still occurs." } ] }, { "id": 1074986, "postDate": "2020-11-11T09:44:00.417Z", "content": "Submissions last forever since fix and finally end up in a \"Notebook Timeout\" error, the exact same submissions that worked perfectly fine before.
", "rawMarkdown": "Submissions last forever since fix and finally end up in a \"Notebook Timeout\" error, the exact same submissions that worked perfectly fine before.", "votes": 1 }, { "id": 1074011, "postDate": "2020-11-10T06:26:26.803Z", "content": "thanks for the fix .Are there still 2.5 million rows in all test sets？
", "rawMarkdown": "thanks for the fix .Are there still 2.5 million rows in all test sets？", "votes": 1, "replies": [ { "id": 1074418, "postDate": "2020-11-10T16:27:44.653Z", "content": "The data hasn't changed at all; only the API has been updated.
", "rawMarkdown": "The data hasn't changed at all; only the API has been updated.", "votes": 3 } ] }, { "id": 1073834, "postDate": "2020-11-10T00:45:29.713Z", "content": "Thanks for the fix. Is the content of the hidden dataset the same as before, and we will still get the same score? I mean for a normal prediction method.
", "rawMarkdown": "Thanks for the fix. Is the content of the hidden dataset the same as before, and we will still get the same score? I mean for a normal prediction method.", "votes": 1, "replies": [ { "id": 1074417, "postDate": "2020-11-10T16:27:16.370Z", "content": "Yes, you will get the same score for a normal prediction.
", "rawMarkdown": "Yes, you will get the same score for a normal prediction.", "votes": 2 } ] }, { "id": 1081782, "postDate": "2020-11-17T09:28:27.200Z", "content": "Would be great if you guide me a bit more on this competition. I am able to understand some points but not all…
", "rawMarkdown": "Would be great if you guide me a bit more on this competition. I am able to understand some points but not all...", "votes": 2 }, { "id": 1075138, "postDate": "2020-11-11T12:30:11.443Z", "content": "I run one submission, which is the same as a previous one, and it should be finish much less than 3 hours before. However, after the API update, it finishes in EXACTLY 3 hours (yes, I monitored the timing by myself). @sohier , could you confirm this is expected, please? And would it be really necessary for the fix of the hack reported??
Mine finishes in exactly 3 hours too. Hmmm
", "rawMarkdown": "Mine finishes in exactly 3 hours too. Hmmm", "votes": 2 }, { "id": 1075421, "postDate": "2020-11-11T17:55:04.320Z", "content": "I took a look at the runtimes and there's a reasonable spread. For example, the first notebook I spot checked finished in a little over an hour.
", "rawMarkdown": "I took a look at the runtimes and there's a reasonable spread. For example, the first notebook I spot checked finished in a little over an hour.", "votes": 1 }, { "id": 1077164, "postDate": "2020-11-13T09:35:01.860Z", "content": "Submit took more than 3 hours for me. I suppose it depends on the model one uses too … took more time for some models then for other ones.
", "rawMarkdown": "Submit took more than 3 hours for me. I suppose it depends on the model one uses too ... took more time for some models then for other ones.", "votes": 1 }, { "id": 1077247, "postDate": "2020-11-13T11:46:57.440Z", "content": "@catadanna Sure, model size counts. I was just surprised that a previous submission took << 3 hours and now it takes 3 hours. But sohier already explain this is expected.
", "rawMarkdown": "@catadanna Sure, model size counts. I was just surprised that a previous submission took << 3 hours and now it takes 3 hours. But sohier already explain this is expected.", "votes": 1 }, { "id": 1077249, "postDate": "2020-11-13T11:50:05.313Z", "content": "@yihdarshieh I suppose it depends on the model size as you say but also on the model type. Not to mention that GPU speeds processing time for neural networks but not for regressors or classifiers.
", "rawMarkdown": "@yihdarshieh I suppose it depends on the model size as you say but also on the model type. Not to mention that GPU speeds processing time for neural networks but not for regressors or classifiers.", "votes": 1 } ] }, { "id": 1075055, "postDate": "2020-11-11T11:06:30.703Z", "content": "Every submission takes atleast 3 hours to be successful
", "rawMarkdown": "Every submission takes atleast 3 hours to be successful", "votes": 2, "replies": [ { "id": 1075059, "postDate": "2020-11-11T11:11:19.440Z", "content": "For you previous submissions, how long it took usually?
", "rawMarkdown": "For you previous submissions, how long it took usually?" }, { "id": 1075107, "postDate": "2020-11-11T11:52:42.937Z", "content": "3 hours to see the result (10 hours ago). Now it has been more than 4 hours and i am yet to see the results.
", "rawMarkdown": "3 hours to see the result (10 hours ago). Now it has been more than 4 hours and i am yet to see the results." } ] }, { "id": 1075002, "postDate": "2020-11-11T09:55:37.543Z", "content": "@sohier Would you mind explain why the submission becomes so slow after the API update? The original version is already slow, and we don't have many options to do more complex modeling due to this. Are the causes of the slowdown coming from really necessary fix for the hack reported?
", "rawMarkdown": "@sohier Would you mind explain why the submission becomes so slow after the API update? The original version is already slow, and we don't have many options to do more complex modeling due to this. Are the causes of the slowdown coming from really necessary fix for the hack reported?", "votes": 2, "replies": [ { "id": 1075008, "postDate": "2020-11-11T10:07:18.430Z", "content": "@sohier Please could we be provided the runtime of the submissions, so we know how close we are to the limit. I don't think knowing this would compromise the API by leaking any information.
", "rawMarkdown": "@sohier Please could we be provided the runtime of the submissions, so we know how close we are to the limit. I don't think knowing this would compromise the API by leaking any information.", "votes": 2 }, { "id": 1075102, "postDate": "2020-11-11T11:41:36.563Z", "content": "I start to doubt that the API fix uses some delay between 2 predictions, (or some minimal submission time). If it is the case, isn't there a better way for the fix without affecting the submission timing (and/or scores)?
", "rawMarkdown": "I start to doubt that the API fix uses some delay between 2 predictions, (or some minimal submission time). If it is the case, isn't there a better way for the fix without affecting the submission timing (and/or scores)? " }, { "id": 1075422, "postDate": "2020-11-11T17:58:13.380Z", "content": "The real runtime has barely changed, but the reported runtime has been obfuscated. For example, if your real runtime was one hour you might see two hours. If your real runtime was 8 hours, you could see anything between 8 and 9 hours.
", "rawMarkdown": "The real runtime has barely changed, but the reported runtime has been obfuscated. For example, if your real runtime was one hour you might see two hours. If your real runtime was 8 hours, you could see anything between 8 and 9 hours." }, { "id": 1075433, "postDate": "2020-11-11T18:09:44.830Z", "content": "@sohier Where can we see the reported runtime that you mention? Only while a submission is running? Or is there any log available?
", "rawMarkdown": "@sohier Where can we see the reported runtime that you mention? Only while a submission is running? Or is there any log available?", "votes": 1 }, { "id": 1075438, "postDate": "2020-11-11T18:14:36.013Z", "content": "As well as @alvaromunoz question, @sohier could you explain why are you obfuscating the runtime?
", "rawMarkdown": "As well as @alvaromunoz question, @sohier could you explain why are you obfuscating the runtime?", "votes": 1 }, { "id": 1075446, "postDate": "2020-11-11T18:19:14.530Z", "content": "If the real runtime is the same, it is kind ok for me, although waiting longer is not something I prefer. But still curious about the reason.
", "rawMarkdown": "If the real runtime is the same, it is kind ok for me, although waiting longer is not something I prefer. But still curious about the reason.", "votes": 5 } ] }, { "id": 1074463, "postDate": "2020-11-10T17:14:22.173Z", "content": "Still having Submission Scoring Error with a notebook that worked before
", "rawMarkdown": "Still having Submission Scoring Error with a notebook that worked before", "votes": 2 }, { "id": 1074135, "postDate": "2020-11-10T09:54:06.243Z", "content": "Ok, cheating is not good
", "rawMarkdown": "Ok, cheating is not good", "votes": 2 }, { "id": 1073802, "postDate": "2020-11-09T23:45:38.653Z", "content": "@sohier Thank you for your clarification!
", "rawMarkdown": "@sohier Thank you for your clarification!", "votes": 2 }, { "id": 1074517, "postDate": "2020-11-10T18:45:21.833Z", "content": "Will it be possible to allow user to return more rows per test_chunk when we iterate over the same? Ty!
", "rawMarkdown": "Will it be possible to allow user to return more rows per test_chunk when we iterate over the same? Ty!", "votes": -2, "replies": [ { "id": 1074528, "postDate": "2020-11-10T19:03:48.110Z", "content": "No.
", "rawMarkdown": "No. " } ] }, { "id": 1120368, "postDate": "2020-12-20T19:15:42.953Z", "content": "I am new to data scientist, but I spent quite a lot of time working on the project, my first submission takes longer than 8 hours, then got notebook timeout error, my second trail is still running after 6 hours, is that normal? REALLY appreciate your help!
", "rawMarkdown": "I am new to data scientist, but I spent quite a lot of time working on the project, my first submission takes longer than 8 hours, then got notebook timeout error, my second trail is still running after 6 hours, is that normal? REALLY appreciate your help!" }, { "id": 1111507, "postDate": "2020-12-13T18:59:31.803Z", "content": "@sohier It is a good thing that notebook runtime is now obfuscated. However submissions made before could have allowed some people to gather info about test data.
\nAnd even with this obfuscation method, it is still possible to answer any binary question one might have about test data (by exiting immediately after iteration or doing a sleep(time left until timeout) depending on the answer).
\nThus EVERY submissions of all winning teams should be reviewed at the end to look for \"cheating\" code like sleep or wait calls inside them.
\nIdeally all submissions of teams getting a medal should be reviewed, at least with an automatic search for some keywords. Deeper inspection could be done for participants with a \"suspiciously\" high number of submissions.
Test dataset probing is not forbidden as far as i know, unless the target is obtained.
", "rawMarkdown": "Test dataset probing is not forbidden as far as i know, unless the target is obtained." }, { "id": 1111547, "postDate": "2020-12-13T19:36:46.503Z", "content": "Ok but if it is not then it should be for future competitions like this one. Because in real life you cannot know the future and test data probing is kind of looking into the future.
", "rawMarkdown": "Ok but if it is not then it should be for future competitions like this one. Because in real life you cannot know the future and test data probing is kind of looking into the future." }, { "id": 1111563, "postDate": "2020-12-13T19:50:06.397Z", "content": "Yep, this can be done easily! But trust me, people here are smart enough to not have such things in their final sub for sure :)
\nPlus I am not sure how probing can actually be stopped. As long as you have access, you can always do it. But yes, if you limit the number of subs like one each day ONLY, then it can be constrained…
", "rawMarkdown": "Yep, this can be done easily! But trust me, people here are smart enough to not have such things in their final sub for sure :)\n\nPlus I am not sure how probing can actually be stopped. As long as you have access, you can always do it. But yes, if you limit the number of subs like one each day ONLY, then it can be constrained... ", "votes": 1 } ] }, { "id": 1093180, "postDate": "2020-11-27T14:19:23.787Z", "content": "Can't we make submission with datatable? Should we strictly use pandas?
", "rawMarkdown": "Can't we make submission with datatable? Should we strictly use pandas?" }, { "id": 1092518, "postDate": "2020-11-26T23:17:32.167Z", "content": "Probably a dumb question but I'm not sure what I'm doing wrong.
\nI am using a Kaggle kernel and when I try to import riiideducation I get the error \"No module named 'riiideducation'\". How am I supposed to address this?
", "rawMarkdown": "Probably a dumb question but I'm not sure what I'm doing wrong.\n\nI am using a Kaggle kernel and when I try to import riiideducation I get the error \"No module named 'riiideducation'\". How am I supposed to address this? ", "replies": [ { "id": 1093015, "postDate": "2020-11-27T11:42:00.337Z", "content": "Did you insert the competition data?
", "rawMarkdown": "Did you insert the competition data?" } ] }, { "id": 1085029, "postDate": "2020-11-20T16:01:40.457Z", "content": "I only use 3000 rows of training data to train my neural model, it takes 9 hours for my submission to run, and then a submission error is reported.
\nI assure my code has no error. I don't know why this problem appears.
\nI have tried to solve this problem for one week.
Do you have a time consuming process e.g. feature creation etc.? The hidden testset is quite big and it could cause an Notebook Timeout
I have tried to set my testset to 2.5 M, and it did not time out
", "rawMarkdown": "I have tried to set my testset to 2.5 M, and it did not time out" }, { "id": 1092802, "postDate": "2020-11-27T07:21:41.547Z", "content": "But in the test set data comes with batches
", "rawMarkdown": "But in the test set data comes with batches", "votes": 1 } ] }, { "id": 1075435, "postDate": "2020-11-11T18:11:12.300Z", "content": "comment deleted
", "rawMarkdown": "comment deleted" }, { "id": 1074682, "postDate": "2020-11-10T23:47:44.323Z", "content": "Has anyone get new (lower) scores for previous submission after the API is updated? It seems the case to me, but I am waiting for more previous submission to be re-run.
", "rawMarkdown": "Has anyone get new (lower) scores for previous submission after the API is updated? It seems the case to me, but I am waiting for more previous submission to be re-run.", "replies": [ { "id": 1075009, "postDate": "2020-11-11T10:07:36.380Z", "content": "My BSF back to back was a repeat with the new API. However my new submission to move me up the leaderboard got a lower score 😔. Don't think that has anything do with the API tho!
", "rawMarkdown": "My BSF back to back was a repeat with the new API. However my new submission to move me up the leaderboard got a lower score 😔. Don't think that has anything do with the API tho!", "votes": 1 } ] }, { "id": 1111494, "postDate": "2020-12-13T18:43:45.917Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1093767, "postDate": "2020-11-28T02:04:15.307Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1085026, "postDate": "2020-11-20T16:00:26.787Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1076346, "postDate": "2020-11-12T13:14:41.867Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1090697, "postDate": "2020-11-25T14:40:07.607Z", "content": "thank you for this info
", "rawMarkdown": "thank you for this info" } ], "comments": [ { "id": 1074153, "author_name": "toxu", "author_url": "", "post_date": "2020-11-10T10:17:11.187000", "content": "Why my Previously successfully executed notebook failed using new api?
", "votes": 9, "replies": [ { "id": 1074164, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-10T10:39:51.697000", "content": "also get error with new api with a notebook previously run successfully … Submission Scoring Error.
", "votes": 1, "replies": [] }, { "id": 1074236, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2020-11-10T12:18:31.233000", "content": "At least you have a successful run :) Mine got Submission Scoring Error after 3 hours of submitting :?
", "votes": 1, "replies": [] }, { "id": 1074391, "author_name": "GhostSkipper", "author_url": "", "post_date": "2020-11-10T15:48:48.237000", "content": "Same here after seeing the other comment, I resubmitted my best submission to date and get the \"Submission Scoring Error\" message. 😔
", "votes": 1, "replies": [] }, { "id": 1087076, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-11-22T10:13:13.890000", "content": "I think some new lectures are added in the test data ?
", "votes": 0, "replies": [] } ] }, { "id": 1075024, "author_name": "Solomon Xian", "author_url": "", "post_date": "2020-11-11T10:28:30.640000", "content": "The same version notebook LB 0.757->0.758, submit time about 1h -> about 3h
\nIt seems much more slower
\nI need to find a faster way
I can confirm this, submissions take considerately longer now, ~1h -> ~4h
", "votes": 2, "replies": [] } ] }, { "id": 1074186, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-10T11:07:02.310000", "content": "Get error with new api with a notebook previously run successfully … Submission Scoring Error.
", "votes": 6, "replies": [] }, { "id": 1074115, "author_name": "GhostSkipper", "author_url": "", "post_date": "2020-11-10T09:16:40.127000", "content": "Thank you for the quick fix @sohier
\nI have some questions about
\n\n\n
\n- Your notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a simple sample submission still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.
\n
I'm a little unsure of what this means. As far as I knew once we submitted it would finish within the next 9hr or we get a \"Notebook Exceeded Allowed Compute\" error.
\n2 hours past, still running. My implementation is quite fast, I monitored the time of predicting on sample submission:
\n[INFO 2020-11-15 01:18:14.094 I]: total iterations: 4, cost 0.1564 secs, 0.0391 secs/iteration\nIf the group size is the same, the iterations of the private dataset are unbelievably large.
\nThis submission utilizes offline generated data and model, all the data loading and statistics calculating can be done in a few seconds.
", "votes": 3, "replies": [] }, { "id": 1075126, "author_name": "Ahmed Ghenabzia", "author_url": "", "post_date": "2020-11-11T12:12:33.107000", "content": "Still getting Submission Scoring Error if we submit Simple submissions from CSV file
\n@sohier
\n
I put the submission result manually and it still making error
\nhttps://www.kaggle.com/ahmed3991/quick-sample-submission-error
@ahmed3991 you can't do this for submission. when submit, there is a hidden test dataset, which is different from the example_test.cvs.
", "votes": 2, "replies": [] }, { "id": 1075502, "author_name": "Ahmed Ghenabzia", "author_url": "", "post_date": "2020-11-11T18:50:07.247000", "content": "I know, But the hidden test Dataset now is the same example_test.csv.
", "votes": 0, "replies": [] }, { "id": 1075506, "author_name": "Ahmed Ghenabzia", "author_url": "", "post_date": "2020-11-11T18:52:57.887000", "content": "@yihdarshieh You can check it in the Notebooks output with the row_id
", "votes": 0, "replies": [] }, { "id": 1075544, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T19:14:25.483000", "content": "you can't modify the content of hidden dataset, it makes no sense.
", "votes": 2, "replies": [] } ] }, { "id": 1074251, "author_name": "Solomon Xian", "author_url": "", "post_date": "2020-11-10T12:40:11.820000", "content": "Submission Scoring Error… I don't think I've changed anything
", "votes": 3, "replies": [] }, { "id": 1074249, "author_name": "chlxyd", "author_url": "", "post_date": "2020-11-10T12:34:29.693000", "content": "Cannot make a successful submit neither, even with previous successful submission.
", "votes": 3, "replies": [] }, { "id": 1075237, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2020-11-11T14:11:00.033000", "content": "Thank you for updating.
\nI have confirmed that if the inference is < 0.55 sec/iter, it is submitted successfully even for new API.
\nhttps://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter?scriptVersionId=46552376
What was the run time of your submission with time.sleep(0.6) ? Was it 9 hours or you got the error message earlier?
", "votes": 0, "replies": [] }, { "id": 1075310, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2020-11-11T15:15:46.990000", "content": "I don't check it for new API, but the notebook kept running for 9 hours (or more??) and I got error message 'Submission Scoring Error' and 'No additional message for this error' for previous API.
", "votes": 1, "replies": [] } ] }, { "id": 1074356, "author_name": "Mihail Parushkin", "author_url": "", "post_date": "2020-11-10T15:05:10.573000", "content": "Even \"Quick Sample Submission\" has the same Error.
\nSo let's wait for the fix.
Thanks for flagging, taking a look now.
", "votes": 3, "replies": [] }, { "id": 1074481, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T17:37:55.643000", "content": "I've posted a patch for that problem, but it will take a bit longer to validate the fix with the quick sample submission.
", "votes": 0, "replies": [] }, { "id": 1074486, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-10T17:42:08.830000", "content": "@sohier once the patch is confirmed to work, could you also update this thread, so we can start to submit again, please?
", "votes": 0, "replies": [] }, { "id": 1074489, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T17:51:55.660000", "content": "Yes, I'll provide an update then.
", "votes": 2, "replies": [] }, { "id": 1074548, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T19:42:14.147000", "content": "The patch is mostly working well, but I'm tweaking one parameter now.
", "votes": 1, "replies": [] }, { "id": 1074601, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T20:57:53.737000", "content": "The version with the parameter tweak has passing submissions, it should be alright to resume normal submissions. I've gone ahead and dropped the 1.00 submission from the leaderboard.
", "votes": 4, "replies": [] }, { "id": 1074677, "author_name": "GhostSkipper", "author_url": "", "post_date": "2020-11-10T23:42:00.567000", "content": "Cool. Thanks
", "votes": 1, "replies": [] } ] }, { "id": 1074188, "author_name": "sayoulala", "author_url": "", "post_date": "2020-11-10T11:08:16.370000", "content": "me too.The official submission also has the same problem
", "votes": 4, "replies": [] }, { "id": 1074184, "author_name": "Catadanna", "author_url": "", "post_date": "2020-11-10T11:04:59.703000", "content": "Same problem here! I get Submission Scoring Error.
", "votes": 4, "replies": [ { "id": 1075010, "author_name": "Catadanna", "author_url": "", "post_date": "2020-11-11T10:08:59.260000", "content": "I can submit now.
", "votes": 0, "replies": [] } ] }, { "id": 1075665, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-11-11T21:38:09.460000", "content": "I get a notebook timeout
", "votes": 1, "replies": [ { "id": 1075675, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T21:43:48.080000", "content": "Was the previous submission run of that notebook close to 9 hours? Not sure, but in this case, you might get timeout. According to @sohier,
\n\n\nThe real runtime has barely changed, but the reported runtime has been obfuscated. For example, if your real runtime was one hour you might see two hours. If your real runtime was 8 hours, you could see anything between 8 and 9 hours.
\n
If the real runtime is close to 9 hours, I don't know if the extra waiting still occurs.
", "votes": 0, "replies": [] } ] }, { "id": 1074986, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2020-11-11T09:44:00.417000", "content": "Submissions last forever since fix and finally end up in a \"Notebook Timeout\" error, the exact same submissions that worked perfectly fine before.
", "votes": 1, "replies": [] }, { "id": 1074011, "author_name": "sayoulala", "author_url": "", "post_date": "2020-11-10T06:26:26.803000", "content": "thanks for the fix .Are there still 2.5 million rows in all test sets？
", "votes": 1, "replies": [ { "id": 1074418, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T16:27:44.653000", "content": "The data hasn't changed at all; only the API has been updated.
", "votes": 3, "replies": [] } ] }, { "id": 1073834, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-10T00:45:29.713000", "content": "Thanks for the fix. Is the content of the hidden dataset the same as before, and we will still get the same score? I mean for a normal prediction method.
", "votes": 1, "replies": [ { "id": 1074417, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T16:27:16.370000", "content": "Yes, you will get the same score for a normal prediction.
", "votes": 2, "replies": [] } ] }, { "id": 1081782, "author_name": "Kashishbhagat", "author_url": "", "post_date": "2020-11-17T09:28:27.200000", "content": "Would be great if you guide me a bit more on this competition. I am able to understand some points but not all…
", "votes": 2, "replies": [] }, { "id": 1075138, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T12:30:11.443000", "content": "I run one submission, which is the same as a previous one, and it should be finish much less than 3 hours before. However, after the API update, it finishes in EXACTLY 3 hours (yes, I monitored the timing by myself). @sohier , could you confirm this is expected, please? And would it be really necessary for the fix of the hack reported??
Mine finishes in exactly 3 hours too. Hmmm
", "votes": 2, "replies": [] }, { "id": 1075421, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-11T17:55:04.320000", "content": "I took a look at the runtimes and there's a reasonable spread. For example, the first notebook I spot checked finished in a little over an hour.
", "votes": 1, "replies": [] }, { "id": 1077164, "author_name": "Catadanna", "author_url": "", "post_date": "2020-11-13T09:35:01.860000", "content": "Submit took more than 3 hours for me. I suppose it depends on the model one uses too … took more time for some models then for other ones.
", "votes": 1, "replies": [] }, { "id": 1077247, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-13T11:46:57.440000", "content": "@catadanna Sure, model size counts. I was just surprised that a previous submission took << 3 hours and now it takes 3 hours. But sohier already explain this is expected.
", "votes": 1, "replies": [] }, { "id": 1077249, "author_name": "Catadanna", "author_url": "", "post_date": "2020-11-13T11:50:05.313000", "content": "@yihdarshieh I suppose it depends on the model size as you say but also on the model type. Not to mention that GPU speeds processing time for neural networks but not for regressors or classifiers.
", "votes": 1, "replies": [] } ] }, { "id": 1075055, "author_name": "ShudharsananMuthuraj", "author_url": "", "post_date": "2020-11-11T11:06:30.703000", "content": "Every submission takes atleast 3 hours to be successful
", "votes": 2, "replies": [ { "id": 1075059, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T11:11:19.440000", "content": "For you previous submissions, how long it took usually?
", "votes": 0, "replies": [] }, { "id": 1075107, "author_name": "ShudharsananMuthuraj", "author_url": "", "post_date": "2020-11-11T11:52:42.937000", "content": "3 hours to see the result (10 hours ago). Now it has been more than 4 hours and i am yet to see the results.
", "votes": 0, "replies": [] } ] }, { "id": 1075002, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T09:55:37.543000", "content": "@sohier Would you mind explain why the submission becomes so slow after the API update? The original version is already slow, and we don't have many options to do more complex modeling due to this. Are the causes of the slowdown coming from really necessary fix for the hack reported?
", "votes": 2, "replies": [ { "id": 1075008, "author_name": "GhostSkipper", "author_url": "", "post_date": "2020-11-11T10:07:18.430000", "content": "@sohier Please could we be provided the runtime of the submissions, so we know how close we are to the limit. I don't think knowing this would compromise the API by leaking any information.
", "votes": 2, "replies": [] }, { "id": 1075102, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T11:41:36.563000", "content": "I start to doubt that the API fix uses some delay between 2 predictions, (or some minimal submission time). If it is the case, isn't there a better way for the fix without affecting the submission timing (and/or scores)?
", "votes": 0, "replies": [] }, { "id": 1075422, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-11T17:58:13.380000", "content": "The real runtime has barely changed, but the reported runtime has been obfuscated. For example, if your real runtime was one hour you might see two hours. If your real runtime was 8 hours, you could see anything between 8 and 9 hours.
", "votes": 0, "replies": [] }, { "id": 1075433, "author_name": "Alvaro", "author_url": "", "post_date": "2020-11-11T18:09:44.830000", "content": "@sohier Where can we see the reported runtime that you mention? Only while a submission is running? Or is there any log available?
", "votes": 1, "replies": [] }, { "id": 1075438, "author_name": "GhostSkipper", "author_url": "", "post_date": "2020-11-11T18:14:36.013000", "content": "As well as @alvaromunoz question, @sohier could you explain why are you obfuscating the runtime?
", "votes": 1, "replies": [] }, { "id": 1075446, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-11T18:19:14.530000", "content": "If the real runtime is the same, it is kind ok for me, although waiting longer is not something I prefer. But still curious about the reason.
", "votes": 5, "replies": [] } ] }, { "id": 1074463, "author_name": "Iuryck Santos", "author_url": "", "post_date": "2020-11-10T17:14:22.173000", "content": "Still having Submission Scoring Error with a notebook that worked before
", "votes": 2, "replies": [] }, { "id": 1074135, "author_name": "Yernar Knayatov", "author_url": "", "post_date": "2020-11-10T09:54:06.243000", "content": "Ok, cheating is not good
", "votes": 2, "replies": [] }, { "id": 1073802, "author_name": "Bayartsogt Yadamsuren", "author_url": "", "post_date": "2020-11-09T23:45:38.653000", "content": "@sohier Thank you for your clarification!
", "votes": 2, "replies": [] }, { "id": 1074517, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-10T18:45:21.833000", "content": "Will it be possible to allow user to return more rows per test_chunk when we iterate over the same? Ty!
", "votes": -2, "replies": [ { "id": 1074528, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-11-10T19:03:48.110000", "content": "No.
", "votes": 0, "replies": [] } ] }, { "id": 1120368, "author_name": "Amyee", "author_url": "", "post_date": "2020-12-20T19:15:42.953000", "content": "I am new to data scientist, but I spent quite a lot of time working on the project, my first submission takes longer than 8 hours, then got notebook timeout error, my second trail is still running after 6 hours, is that normal? REALLY appreciate your help!
", "votes": 0, "replies": [] }, { "id": 1111507, "author_name": "ISMAX", "author_url": "", "post_date": "2020-12-13T18:59:31.803000", "content": "@sohier It is a good thing that notebook runtime is now obfuscated. However submissions made before could have allowed some people to gather info about test data.
\nAnd even with this obfuscation method, it is still possible to answer any binary question one might have about test data (by exiting immediately after iteration or doing a sleep(time left until timeout) depending on the answer).
\nThus EVERY submissions of all winning teams should be reviewed at the end to look for \"cheating\" code like sleep or wait calls inside them.
\nIdeally all submissions of teams getting a medal should be reviewed, at least with an automatic search for some keywords. Deeper inspection could be done for participants with a \"suspiciously\" high number of submissions.
Test dataset probing is not forbidden as far as i know, unless the target is obtained.
", "votes": 0, "replies": [] }, { "id": 1111547, "author_name": "ISMAX", "author_url": "", "post_date": "2020-12-13T19:36:46.503000", "content": "Ok but if it is not then it should be for future competitions like this one. Because in real life you cannot know the future and test data probing is kind of looking into the future.
", "votes": 0, "replies": [] }, { "id": 1111563, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-13T19:50:06.397000", "content": "Yep, this can be done easily! But trust me, people here are smart enough to not have such things in their final sub for sure :)
\nPlus I am not sure how probing can actually be stopped. As long as you have access, you can always do it. But yes, if you limit the number of subs like one each day ONLY, then it can be constrained…
", "votes": 1, "replies": [] } ] }, { "id": 1093180, "author_name": "bturan19", "author_url": "", "post_date": "2020-11-27T14:19:23.787000", "content": "Can't we make submission with datatable? Should we strictly use pandas?
", "votes": 0, "replies": [] }, { "id": 1092518, "author_name": "Patrick Brazil", "author_url": "", "post_date": "2020-11-26T23:17:32.167000", "content": "Probably a dumb question but I'm not sure what I'm doing wrong.
\nI am using a Kaggle kernel and when I try to import riiideducation I get the error \"No module named 'riiideducation'\". How am I supposed to address this?
", "votes": 0, "replies": [ { "id": 1093015, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-11-27T11:42:00.337000", "content": "Did you insert the competition data?
", "votes": 0, "replies": [] } ] }, { "id": 1085029, "author_name": "suxiaohe", "author_url": "", "post_date": "2020-11-20T16:01:40.457000", "content": "I only use 3000 rows of training data to train my neural model, it takes 9 hours for my submission to run, and then a submission error is reported.
\nI assure my code has no error. I don't know why this problem appears.
\nI have tried to solve this problem for one week.
Do you have a time consuming process e.g. feature creation etc.? The hidden testset is quite big and it could cause an Notebook Timeout
I have tried to set my testset to 2.5 M, and it did not time out
", "votes": 0, "replies": [] }, { "id": 1092802, "author_name": "bturan19", "author_url": "", "post_date": "2020-11-27T07:21:41.547000", "content": "But in the test set data comes with batches
", "votes": 1, "replies": [] } ] }, { "id": 1075435, "author_name": "", "author_url": "", "post_date": "2020-11-11T18:11:12.300000", "content": "", "votes": 0, "replies": [] }, { "id": 1074682, "author_name": "", "author_url": "", "post_date": "2020-11-10T23:47:44.323000", "content": "", "votes": 0, "replies": [ { "id": 1075009, "author_name": "", "author_url": "", "post_date": "2020-11-11T10:07:36.380000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1111494, "author_name": "", "author_url": "", "post_date": "2020-12-13T18:43:45.917000", "content": "", "votes": 0, "replies": [] }, { "id": 1093767, "author_name": "", "author_url": "", "post_date": "2020-11-28T02:04:15.307000", "content": "", "votes": 0, "replies": [] }, { "id": 1085026, "author_name": "", "author_url": "", "post_date": "2020-11-20T16:00:26.787000", "content": "", "votes": 0, "replies": [] }, { "id": 1076346, "author_name": "", "author_url": "", "post_date": "2020-11-12T13:14:41.867000", "content": "", "votes": 1, "replies": [] }, { "id": 1090697, "author_name": "", "author_url": "", "post_date": "2020-11-25T14:40:07.607000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1073758": "I've uploaded an update to respond to [the issues raised here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195815). It will start affecting new submissions in the next half an hour or so.\n\n- Your notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a [simple sample submission](https://www.kaggle.com/sohier/quick-sample-submission) still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.\n Put differently, you will now see your results after the real runtime plus some random additional time, with the random additional time never pushing the total over nine hours.\n\n- The updates only affect the hidden rerun version of the API.\n- I'll rerun all notebooks published during the relevant time window over the rest of this week.\n- As a reminder, we will be reviewing submissions for cheating attempts at the end of the competition. Any other vulnerabilities that are discovered and applied but not reported to us before then may lead to a ban from the competition and other sanctions.\n\nEdit: I introduced a bug in the initial copy of the update. That should be fixed as of 13:00 PST today. Apologies for the inconvenience!\n", "1074153": "Why my Previously successfully executed notebook failed using new api?", "1075024": "The same version notebook LB 0.757->0.758, submit time about 1h -> about 3h\nIt seems much more slower\nI need to find a faster way", "1074186": "Get error with new api with a notebook previously run successfully … Submission Scoring Error.", "1074115": "Thank you for the quick fix @sohier \nI have some questions about\n> - Your notebook's runtime will now be somewhat obfuscated. Unfortunately, this means it will be more difficult to do performance tuning. Making a [simple sample submission](https://www.kaggle.com/sohier/quick-sample-submission) still takes about 15 minutes; I recommend using that as a baseline for offline performance estimation.\n\nI'm a little unsure of what this means. As far as I knew once we submitted it would finish within the next 9hr or we get a \"Notebook Exceeded Allowed Compute\" error. \n- Was it possible to get the runtime of a submission that had finished?\n- Do you mean submission will run a bit slower or there is now a more random element to the submission timings due to the extra checks?", "1078555": "2 hours past, still running. My implementation is quite fast, I monitored the time of predicting on sample submission:\n```\n[INFO 2020-11-15 01:18:14.094 I]: total iterations: 4, cost 0.1564 secs, 0.0391 secs/iteration\n```\nIf the group size is the same, the iterations of the private dataset are unbelievably large.\n\nThis submission utilizes offline generated data and model, all the data loading and statistics calculating can be done in a few seconds.", "1075126": "Still getting **Submission Scoring Error** if we submit Simple submissions from CSV file\n@sohier \n", "1074251": "Submission Scoring Error... I don't think I've changed anything", "1074249": "Cannot make a successful submit neither, even with previous successful submission.\n", "1075237": "Thank you for updating.\nI have confirmed that if the inference is < 0.55 sec/iter, it is submitted successfully even for new API.\nhttps://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter?scriptVersionId=46552376\n", "1074356": "Even [\"Quick Sample Submission\"](https://www.kaggle.com/sohier/quick-sample-submission) has the same Error.\nSo let's wait for the fix.", "1074188": "me too.The official submission also has the same problem", "1074184": "Same problem here! I get Submission Scoring Error.", "1075665": "I get a notebook timeout ", "1074986": "Submissions last forever since fix and finally end up in a \"Notebook Timeout\" error, the exact same submissions that worked perfectly fine before.", "1074011": "thanks for the fix .Are there still 2.5 million rows in all test sets？", "1073834": "Thanks for the fix. Is the content of the hidden dataset the same as before, and we will still get the same score? I mean for a normal prediction method.", "1081782": "Would be great if you guide me a bit more on this competition. I am able to understand some points but not all...", "1075138": "I run one submission, which is the same as a previous one, and it should be finish much less than 3 hours before. However, after the API update, it finishes in `EXACTLY` 3 hours (yes, I monitored the timing by myself). @sohier , could you confirm this is expected, please? And would it be really necessary for the fix of the hack reported??", "1075055": "Every submission takes atleast 3 hours to be successful", "1075002": "@sohier Would you mind explain why the submission becomes so slow after the API update? The original version is already slow, and we don't have many options to do more complex modeling due to this. Are the causes of the slowdown coming from really necessary fix for the hack reported?", "1074463": "Still having Submission Scoring Error with a notebook that worked before", "1074135": "Ok, cheating is not good", "1073802": "@sohier Thank you for your clarification!", "1074517": "Will it be possible to allow user to return more rows per test_chunk when we iterate over the same? Ty!", "1120368": "I am new to data scientist, but I spent quite a lot of time working on the project, my first submission takes longer than 8 hours, then got notebook timeout error, my second trail is still running after 6 hours, is that normal? REALLY appreciate your help!", "1111507": "@sohier It is a good thing that notebook runtime is now obfuscated. However submissions made before could have allowed some people to gather info about test data.\nAnd even with this obfuscation method, **it is still possible to answer any binary question one might have about test data** (by exiting immediately after iteration or doing a sleep(time left until timeout) depending on the answer).\n**Thus EVERY submissions of all winning teams should be reviewed at the end** to look for \"cheating\" code like sleep or wait calls inside them.\nIdeally all submissions of teams getting a medal should be reviewed, at least with an automatic search for some keywords. Deeper inspection could be done for participants with a \"suspiciously\" high number of submissions.", "1093180": "Can't we make submission with datatable? Should we strictly use pandas?", "1092518": "Probably a dumb question but I'm not sure what I'm doing wrong.\n\nI am using a Kaggle kernel and when I try to import riiideducation I get the error \"No module named 'riiideducation'\". How am I supposed to address this? ", "1085029": "I only use 3000 rows of training data to train my neural model, it takes 9 hours for my submission to run, and then a submission error is reported.\nI assure my code has no error. I don't know why this problem appears.\nI have tried to solve this problem for one week.", "1075435": "comment deleted", "1074682": "Has anyone get new (lower) scores for previous submission after the API is updated? It seems the case to me, but I am waiting for more previous submission to be re-run.", "1111494": "", "1093767": "", "1085026": "", "1076346": "", "1090697": "thank you for this info" } }