{ "id": 208473, "title": "Score submission error (read discussions and used emulators and nothing works)", "url": "/competitions/riiid-test-answer-prediction/discussion/208473", "author_name": "", "post_date": "2021-01-03T16:26:04.200537100Z", "votes": null, "comment_count": 6, "views": 0, "content": "

I created a submission notebook with features added but I get a score submission error. I used two emulators so far and they are working fine :(
\nI really have no idea what to do. Did any one face something like this before ?

", "messages": [ { "id": "1137053", "postDate": "01/03/2021 16:26:04", "content": "

I created a submission notebook with features added but I get a score submission error. I used two emulators so far and they are working fine :(
\nI really have no idea what to do. Did any one face something like this before ?

", "rawMarkdown": "I created a submission notebook with features added but I get a score submission error. I used two emulators so far and they are working fine :(\nI really have no idea what to do. Did any one face something like this before ?", "votes": null }, { "id": "1138711", "postDate": "01/04/2021 22:08:47", "content": "

Use simulator from Tito with different configurations in addition to existing one:

\n\n

Also, watch inference time with simulator. If it's too long in simulator you will get submission scoring error after 9h.

", "rawMarkdown": "Use [simulator from Tito](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator) with different configurations in addition to existing one:\n- **Configuration 1**: `max_user=1`, it allows testing small batch. It helped me quite a lot for `array/shape` issues.\n- **Configuration 2**: `max_user=1000` and tune simulator code to make it returns more than 1 to 30 rows per batch (even if same user is duplicated in same batch). I've modified it to have 20000 rows per batch to simulate large API batches and it helped me to fix OOM.\n\nAlso, watch inference time with simulator. If it's too long in simulator you will get submission scoring error after 9h.", "votes": null }, { "id": "1138715", "postDate": "01/04/2021 22:32:15", "content": "

I tried the simulator but didn't configure it. I will try the configurations. Thanks a lot

", "rawMarkdown": "I tried the simulator but didn't configure it. I will try the configurations. Thanks a lot", "votes": null }, { "id": "1138724", "postDate": "01/04/2021 22:43:58", "content": "

This simulator is really powerful, you can simulate almost everything that could happen with the API.

", "rawMarkdown": "This simulator is really powerful, you can simulate almost everything that could happen with the API.", "votes": null }, { "id": "1139376", "postDate": "01/05/2021 10:58:11", "content": "

Do you check your inference is < 0.55 sec/iter (or < 9 hours in total)?
\nIf so, how about your memory usage?
\nThis discussion topic would be also helpful for you.

", "rawMarkdown": "Do you check [your inference is < 0.55 sec/iter](https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter) (or < 9 hours in total)?\nIf so, how about your memory usage?\n[This discussion topic](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192124) would be also helpful for you.", "votes": null }, { "id": "1139382", "postDate": "01/05/2021 11:03:15", "content": "

I checked every thing. It turned out to be an index out of range problem in some cases. I found it by revising my code again and again and thinking about possible problems.

", "rawMarkdown": "I checked every thing. It turned out to be an index out of range problem in some cases. I found it by revising my code again and again and thinking about possible problems.", "votes": null }, { "id": "1139385", "postDate": "01/05/2021 11:04:37", "content": "

I also divided my code into sections and submitted with one section uncommented multiple times to know where the problem is (took time and trials but solved it finally).

", "rawMarkdown": "I also divided my code into sections and submitted with one section uncommented multiple times to know where the problem is (took time and trials but solved it finally).", "votes": null } ], "comments": [ { "id": 1138711, "author_name": "mpware", "author_url": "", "post_date": "01/04/2021 22:08:47", "content": "

Use simulator from Tito with different configurations in addition to existing one:

\n\n

Also, watch inference time with simulator. If it's too long in simulator you will get submission scoring error after 9h.

", "votes": null, "replies": [ { "id": 1138715, "author_name": "ahmedsalah142", "author_url": "", "post_date": "01/04/2021 22:32:15", "content": "

I tried the simulator but didn't configure it. I will try the configurations. Thanks a lot

", "votes": null, "replies": [ { "id": 1138724, "author_name": "mpware", "author_url": "", "post_date": "01/04/2021 22:43:58", "content": "

This simulator is really powerful, you can simulate almost everything that could happen with the API.

", "votes": null, "replies": [] } ] } ] }, { "id": 1139376, "author_name": "tomooinubushi", "author_url": "", "post_date": "01/05/2021 10:58:11", "content": "

Do you check your inference is < 0.55 sec/iter (or < 9 hours in total)?
\nIf so, how about your memory usage?
\nThis discussion topic would be also helpful for you.

", "votes": null, "replies": [ { "id": 1139382, "author_name": "ahmedsalah142", "author_url": "", "post_date": "01/05/2021 11:03:15", "content": "

I checked every thing. It turned out to be an index out of range problem in some cases. I found it by revising my code again and again and thinking about possible problems.

", "votes": null, "replies": [] }, { "id": 1139385, "author_name": "ahmedsalah142", "author_url": "", "post_date": "01/05/2021 11:04:37", "content": "

I also divided my code into sections and submitted with one section uncommented multiple times to know where the problem is (took time and trials but solved it finally).

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1137053": "I created a submission notebook with features added but I get a score submission error. I used two emulators so far and they are working fine :(\nI really have no idea what to do. Did any one face something like this before ?", "1138711": "Use [simulator from Tito](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator) with different configurations in addition to existing one:\n- **Configuration 1**: `max_user=1`, it allows testing small batch. It helped me quite a lot for `array/shape` issues.\n- **Configuration 2**: `max_user=1000` and tune simulator code to make it returns more than 1 to 30 rows per batch (even if same user is duplicated in same batch). I've modified it to have 20000 rows per batch to simulate large API batches and it helped me to fix OOM.\n\nAlso, watch inference time with simulator. If it's too long in simulator you will get submission scoring error after 9h.", "1138715": "I tried the simulator but didn't configure it. I will try the configurations. Thanks a lot", "1138724": "This simulator is really powerful, you can simulate almost everything that could happen with the API.", "1139376": "Do you check [your inference is < 0.55 sec/iter](https://www.kaggle.com/tomooinubushi/inference-must-be-0-55-sec-iter) (or < 9 hours in total)?\nIf so, how about your memory usage?\n[This discussion topic](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192124) would be also helpful for you.", "1139382": "I checked every thing. It turned out to be an index out of range problem in some cases. I found it by revising my code again and again and thinking about possible problems.", "1139385": "I also divided my code into sections and submitted with one section uncommented multiple times to know where the problem is (took time and trials but solved it finally)." }, "source": "meta" }