{ "id": 433175, "title": "Inference time is not consistent", "url": "/competitions/asl-fingerspelling/discussion/433175", "author_name": "", "post_date": "2023-08-20T15:55:26.725164600Z", "votes": 1, "comment_count": 13, "views": 0, "content": "

In the exact same setting, sometimes inference time is fast and sometimes it is slow.
\nThere's up to a 2 times difference. Is anyone else experiencing the same issue?

", "messages": [ { "id": "2399831", "postDate": "08/20/2023 15:55:26", "content": "

In the exact same setting, sometimes inference time is fast and sometimes it is slow.
\nThere's up to a 2 times difference. Is anyone else experiencing the same issue?

", "rawMarkdown": "In the exact same setting, sometimes inference time is fast and sometimes it is slow. \nThere's up to a 2 times difference. Is anyone else experiencing the same issue?", "votes": null }, { "id": "2400016", "postDate": "08/20/2023 18:18:34", "content": "

I did observe it when inferencing on Kaggle's notebook. But it is very rare.

", "rawMarkdown": "I did observe it when inferencing on Kaggle's notebook. But it is very rare.", "votes": null }, { "id": "2400440", "postDate": "08/21/2023 04:35:48", "content": "

Thanks for your response.
\nFor me, it happens quite often.
\nBut I don't know why. I hope it doesn't affect submission inference time…

", "rawMarkdown": "Thanks for your response.\nFor me, it happens quite often.\nBut I don't know why. I hope it doesn't affect submission inference time...", "votes": null }, { "id": "2401827", "postDate": "08/21/2023 20:16:36", "content": "

It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?

", "rawMarkdown": "It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?", "votes": null }, { "id": "2401998", "postDate": "08/22/2023 00:27:05", "content": "

I have observed variation in model speed too. The following code with my model produced about 140ms when i committed just now. And about 350ms when i committed 2 hours ago with the same model (that's 2x difference!). I have also observed variation in submit time too. But i think variation in submit is less than variation in interactive and commit notebooks.

\n
%%timeit -n \n  prediction_fn(inputsframe)\n
", "rawMarkdown": "I have observed variation in model speed too. The following code with my model produced about 140ms when i **committed** just now. And about 350ms when i **committed** 2 hours ago **with the same model** (that's 2x difference!). I have also observed variation in **submit** time too. But i think variation in **submit** is less than variation in **interactive** and **commit** notebooks.\n\n %%timeit -n 10\n output = prediction_fn(inputs=frame)", "votes": null }, { "id": "2404370", "postDate": "08/23/2023 07:55:21", "content": "

I also observed variation in submit is quite consistent.

", "rawMarkdown": "I also observed variation in submit is quite consistent.", "votes": null }, { "id": "2406236", "postDate": "08/24/2023 09:41:40", "content": "

Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime…

", "rawMarkdown": "Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime...", "votes": null }, { "id": "2406263", "postDate": "08/24/2023 10:14:33", "content": "

Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?

", "rawMarkdown": "Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?", "votes": null }, { "id": "2406271", "postDate": "08/24/2023 10:21:29", "content": "

I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.

", "rawMarkdown": "I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.", "votes": null }, { "id": "2406292", "postDate": "08/24/2023 10:38:47", "content": "

yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.

", "rawMarkdown": "yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.", "votes": null }, { "id": "2406349", "postDate": "08/24/2023 11:19:22", "content": "

Makes sense, thanks for the details!

", "rawMarkdown": "Makes sense, thanks for the details!", "votes": null }, { "id": "2406474", "postDate": "08/24/2023 13:09:55", "content": "

@cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?

", "rawMarkdown": "cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?", "votes": null }, { "id": "2406554", "postDate": "08/24/2023 13:59:11", "content": "

@goldenlock The overtime error does not say overtime. The overtime error says:

\n
\n

Submission Scoring Error
\n Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips

\n
\n

This is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error.

", "rawMarkdown": "goldenlock The overtime error does not say overtime. The overtime error says:\n\n>Submission Scoring Error\nYour notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\n\nThis is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error.", "votes": null }, { "id": "2406583", "postDate": "08/24/2023 14:24:07", "content": "

@cdeotte thanks! Your explanation helped a lot!

", "rawMarkdown": "cdeotte thanks! Your explanation helped a lot!", "votes": null } ], "comments": [ { "id": 2400016, "author_name": "nightsh4de", "author_url": "", "post_date": "08/20/2023 18:18:34", "content": "

I did observe it when inferencing on Kaggle's notebook. But it is very rare.

", "votes": null, "replies": [ { "id": 2400440, "author_name": "wonjunjg", "author_url": "", "post_date": "08/21/2023 04:35:48", "content": "

Thanks for your response.
\nFor me, it happens quite often.
\nBut I don't know why. I hope it doesn't affect submission inference time…

", "votes": null, "replies": [] } ] }, { "id": 2401827, "author_name": "yassinealouini", "author_url": "", "post_date": "08/21/2023 20:16:36", "content": "

It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?

", "votes": null, "replies": [] }, { "id": 2401998, "author_name": "cdeotte", "author_url": "", "post_date": "08/22/2023 00:27:05", "content": "

I have observed variation in model speed too. The following code with my model produced about 140ms when i committed just now. And about 350ms when i committed 2 hours ago with the same model (that's 2x difference!). I have also observed variation in submit time too. But i think variation in submit is less than variation in interactive and commit notebooks.

\n
%%timeit -n \n  prediction_fn(inputsframe)\n
", "votes": null, "replies": [ { "id": 2404370, "author_name": "wonjunjg", "author_url": "", "post_date": "08/23/2023 07:55:21", "content": "

I also observed variation in submit is quite consistent.

", "votes": null, "replies": [] } ] }, { "id": 2406236, "author_name": "goldenlock", "author_url": "", "post_date": "08/24/2023 09:41:40", "content": "

Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime…

", "votes": null, "replies": [ { "id": 2406263, "author_name": "yassinealouini", "author_url": "", "post_date": "08/24/2023 10:14:33", "content": "

Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?

", "votes": null, "replies": [ { "id": 2406271, "author_name": "cdeotte", "author_url": "", "post_date": "08/24/2023 10:21:29", "content": "

I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.

", "votes": null, "replies": [ { "id": 2406292, "author_name": "goldenlock", "author_url": "", "post_date": "08/24/2023 10:38:47", "content": "

yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.

", "votes": null, "replies": [] }, { "id": 2406349, "author_name": "yassinealouini", "author_url": "", "post_date": "08/24/2023 11:19:22", "content": "

Makes sense, thanks for the details!

", "votes": null, "replies": [] }, { "id": 2406474, "author_name": "goldenlock", "author_url": "", "post_date": "08/24/2023 13:09:55", "content": "

@cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?

", "votes": null, "replies": [ { "id": 2406554, "author_name": "cdeotte", "author_url": "", "post_date": "08/24/2023 13:59:11", "content": "

@goldenlock The overtime error does not say overtime. The overtime error says:

\n
\n

Submission Scoring Error
\n Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips

\n
\n

This is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error.

", "votes": null, "replies": [ { "id": 2406583, "author_name": "goldenlock", "author_url": "", "post_date": "08/24/2023 14:24:07", "content": "

@cdeotte thanks! Your explanation helped a lot!

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2399831": "In the exact same setting, sometimes inference time is fast and sometimes it is slow. \nThere's up to a 2 times difference. Is anyone else experiencing the same issue?", "2400016": "I did observe it when inferencing on Kaggle's notebook. But it is very rare.", "2400440": "Thanks for your response.\nFor me, it happens quite often.\nBut I don't know why. I hope it doesn't affect submission inference time...", "2401827": "It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?", "2401998": "I have observed variation in model speed too. The following code with my model produced about 140ms when i **committed** just now. And about 350ms when i **committed** 2 hours ago **with the same model** (that's 2x difference!). I have also observed variation in **submit** time too. But i think variation in **submit** is less than variation in **interactive** and **commit** notebooks.\n\n %%timeit -n 10\n output = prediction_fn(inputs=frame)", "2404370": "I also observed variation in submit is quite consistent.", "2406236": "Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime...", "2406263": "Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?", "2406271": "I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.", "2406292": "yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.", "2406349": "Makes sense, thanks for the details!", "2406474": "cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?", "2406554": "goldenlock The overtime error does not say overtime. The overtime error says:\n\n>Submission Scoring Error\nYour notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\n\nThis is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error.", "2406583": "cdeotte thanks! Your explanation helped a lot!" }, "source": "meta" }