\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F3907320890673c4e24f948c0fe320ccd%2Foutoftime.jpg?generation=1679491011243306&alt=media\")
We discovered that, due to a version control mistake the metric constraints around model runtime and size weren't being applied as we intended. I've patched the metric:
\nAbout 60 submissions received scores when they should have been rejected due to the time constraint; I will be invalidating those submissions shortly.
\nApologies for the disruption and confusion.
", "messages": [ { "id": 2175328, "postDate": "2023-03-09T19:24:17.803Z", "content": "We discovered that, due to a version control mistake the metric constraints around model runtime and size weren't being applied as we intended. I've patched the metric:
\nAbout 60 submissions received scores when they should have been rejected due to the time constraint; I will be invalidating those submissions shortly.
\nApologies for the disruption and confusion.
", "rawMarkdown": "We discovered that, due to a version control mistake the metric constraints around model runtime and size weren't being applied as we intended. I've patched the metric:\n- You should now get an error message more promptly if your model is too large\n- Submissions will be rejected properly if their runtime exceeds the cap (about 75 minutes). \n\nAbout 60 submissions received scores when they should have been rejected due to the time constraint; I will be invalidating those submissions shortly. \n\nApologies for the disruption and confusion.", "votes": 30 }, { "id": 2210090, "postDate": "2023-04-05T05:57:22.693Z", "content": "@sohier The submission seems to be kicked in 60min. Is this behavior right?
", "rawMarkdown": "@sohier The submission seems to be kicked in 60min. Is this behavior right?", "votes": 9, "replies": [ { "id": 2237236, "postDate": "2023-04-27T13:20:30.487Z", "content": "The same happened. Last week I submitted the model successfully. Now it failed a few times in a row. I have a strong feeling something changed. It also fails after 60 minutes, however, 70 min expected…
", "rawMarkdown": "The same happened. Last week I submitted the model successfully. Now it failed a few times in a row. I have a strong feeling something changed. It also fails after 60 minutes, however, 70 min expected..." } ] }, { "id": 2204385, "postDate": "2023-03-31T14:19:54.663Z", "content": "Hi @sohier !
\nI was wondering what was going on when submitting. I've had models returning the \"Evaluation Exception: Inference time cap exceeded.error\" error after a ~1h03 run, which is weird since the error should be raised after ~75 minutes of runtime.
\n100ms / it + 10min margin implies there are about 32k images in the test data, is that actually the case ? Or could it be that the 10 min buffer from the evaluation page is not taken into account ?
\nOthers have also reported similar timings to receive the error :)
\nThanks !!
if the evalution script uses parallel processes, then time limit is 75/num of process
", "rawMarkdown": "if the evalution script uses parallel processes, then time limit is 75/num of process", "votes": 1 }, { "id": 2220495, "postDate": "2023-04-13T12:56:43.840Z", "content": "Poke @sohier ! 😊
", "rawMarkdown": "Poke @sohier ! 😊", "votes": 2, "replies": [ { "id": 2234125, "postDate": "2023-04-24T22:42:08.050Z", "content": "@theoviel did you (or anyone else) find clarity on this issue? Did @sohier response somewhere else?
\nI have sumissions that fail at ~62 minutes. The same submissions run in less than 100ms/it in the kaggle notebook with TensorFlow Lite Runtime v2.9.1.
\nThe evaluation page says \"Expect to see approximately 40,000 videos in the test set. We allow an additional 10 minute buffer for loading the data and miscellaneous overhead.\" Shouldn't this equate to well above 60 minutes even without the 10 minute buffer?
", "rawMarkdown": "@theoviel did you (or anyone else) find clarity on this issue? Did @sohier response somewhere else?\n\nI have sumissions that fail at ~62 minutes. The same submissions run in less than 100ms/it in the kaggle notebook with TensorFlow Lite Runtime v2.9.1.\n\nThe [evaluation page](https://www.kaggle.com/competitions/asl-signs/overview/evaluation) says \"Expect to see approximately 40,000 videos in the test set. We allow an additional 10 minute buffer for loading the data and miscellaneous overhead.\" Shouldn't this equate to well above 60 minutes even without the 10 minute buffer?", "votes": 2 }, { "id": 2234147, "postDate": "2023-04-25T00:34:56.433Z", "content": "Same here. My last submission timed out at exactly 1 hour. According to the evaluation page here, our submit notebooks should be allowed approximately 40000 * 0.1 / 60 + 10 minutes which equals approximately 76.7 minutes. That means a submit should not fail before 1 hour and 16.7 minutes (if there are 40000 test videos). Can anyone explain what is happening?
One answer is that test data only has 30000 videos. Then we would only have 60 minutes = 30000 * 0.1 / 60 + 10 minutes. Another answer is that the server uses some parallel processing.
\n", "rawMarkdown": "Same here. My last submission timed out at exactly 1 hour. According to the evaluation page [here][1], our submit notebooks should be allowed approximately `40000 * 0.1 / 60 + 10 minutes` which equals approximately `76.7` minutes. That means a submit should not fail before 1 hour and 16.7 minutes (if there are 40000 test videos). Can anyone explain what is happening?\n\nOne answer is that test data only has 30000 videos. Then we would only have `60 minutes = 30000 * 0.1 / 60 + 10 minutes`. Another answer is that the server uses some parallel processing.\n\n>Your model must also perform inference with less than 100 milliseconds of latency per video on average and use less than 40 MB of storage space. Expect to see approximately 40,000 videos in the test set. We allow an additional 10 minute buffer for loading the data and miscellaneous overhead.\n\n[1]: https://www.kaggle.com/competitions/asl-signs/overview/evaluation", "votes": 3 }, { "id": 2234169, "postDate": "2023-04-25T01:15:04.757Z", "content": "Your model must also perform inference with less than 100 milliseconds of latency per video on average and use less than 40 MB of storage space. Expect to see approximately 40,000 videos in the test set. We allow an additional 10 minute buffer for loading the data and miscellaneous overhead.
\n
This was posted few days back https://www.kaggle.com/competitions/asl-signs/discussion/403110#2229730
\nThey have been consistent with ~60 mins time .. it's possible that \"75 mins\" calculation was incorrectly mentioned in evaluation page - but this is just me speculating
", "rawMarkdown": "This was posted few days back https://www.kaggle.com/competitions/asl-signs/discussion/403110#2229730\n\nThey have been consistent with ~60 mins time .. it's possible that \"75 mins\" calculation was incorrectly mentioned in evaluation page - but this is just me speculating\n", "votes": 1 }, { "id": 2234473, "postDate": "2023-04-25T08:18:52.537Z", "content": "No update on this, and I don't think we will get one. Maybe there is an inconsistency between Sohier's code an what is written on the evaluation page, or maybe Kaggle is doing optimizations during submissions that speed up inference.
\nThis should have been addressed a month ago though, one week before the deadline is too late to change anything.
", "rawMarkdown": "No update on this, and I don't think we will get one. Maybe there is an inconsistency between Sohier's code an what is written on the evaluation page, or maybe Kaggle is doing optimizations during submissions that speed up inference.\n\nThis should have been addressed a month ago though, one week before the deadline is too late to change anything.", "votes": 5 } ] } ] }, { "id": 2191434, "postDate": "2023-03-22T01:12:21.780Z", "content": "It seems we are getting \"Submission Scoring Error\" if we exceed runtime … Is that correct ?
\nReason I ask is my 2 fold model worked in ~ 42 mins and 3 fold model returned \"submission scoring error\" in ~60 minutes.
", "rawMarkdown": "It seems we are getting \"Submission Scoring Error\" if we exceed runtime ... Is that correct ? \n\nReason I ask is my 2 fold model worked in ~ 42 mins and 3 fold model returned \"submission scoring error\" in ~60 minutes. ", "votes": 3, "replies": [ { "id": 2191724, "postDate": "2023-03-22T06:58:15.537Z", "content": "Personally, I received a Submission Scoring Error in two cases:
\nYes, I was puzzled because we typically get \"runtime exceeded error\" for going beyond.
", "rawMarkdown": "Yes, I was puzzled because we typically get \"runtime exceeded error\" for going beyond. ", "votes": 1 }, { "id": 2191995, "postDate": "2023-03-22T11:09:27.563Z", "content": "We gettin \"runtime exceeded error\" when you making predicts in your notebook. But in this competition in notebook you only load submission.zip and all computations are in \"Scoring\" part. Where usually Kaggle just calculate metric from your submission.csv predictions.
", "rawMarkdown": "We gettin \"runtime exceeded error\" when you making predicts in your notebook. But in this competition in notebook you only load submission.zip and all computations are in \"Scoring\" part. Where usually Kaggle just calculate metric from your submission.csv predictions.", "votes": 4 } ] }, { "id": 2191991, "postDate": "2023-03-22T11:05:10.397Z", "content": "Agree. I keep getting Submission Scoring Error. After 60-65 minutes after submit. And the same code have succeeded once.
\nI think that means that model converting is correct, model size is correct.
Also @sohier said about ~75 minutes, how it can return error after 60?
", "rawMarkdown": "Agree. I keep getting Submission Scoring Error. After 60-65 minutes after submit. And the same code have succeeded once.\nI think that means that model converting is correct, model size is correct.\n\nAlso @sohier said about ~75 minutes, how it can return error after 60?", "votes": 3, "replies": [ { "id": 2192157, "postDate": "2023-03-22T13:17:24.513Z", "content": "For example today's submission. In my location submissions counter reloads at 03:00 , so i have submitted my notebook at 03:01, ~2 minutes for preparing submission.zip. At ~03:02 scoring has started
\nAnd at 04:04 I already found \"Submission Scoring Error\"
\n
Local scoring time for 40,000 samples:
\nMean time including data loading time, only infer counts only model runtime.
st = time.time()\ncnt = \ntotal = \nmodel_time = \n\n i tqdm(np.linspace(, (train)-, total, dtype=)):\n sample = train.loc[i] \n\n yy = load_relevant_data_subset(base_dir + sample[])\n md_st = time.time()\n output = prediction_fn(inputs=yy)\n model_time += time.time() - md_st\n sign = np.argmax(output[])\n \n\n sample[] == sign: cnt +=\n\n() \n\n()\n()\n%|██████████| / [:06:<:, it/s]\nMean time: \nMean time only infer: \nI'm having a similar problem too. I don't know yet, but I suspect that the submission time is getting unexpectedly longer when combining multiple models into one model with tflite conversion(e.g. average of each fold).
\nFor example, my 50ms single fold model completed submission within around 40 minutes, but the 5-fold average model of 50 ms(5x 10ms models) got a submission scoring error after 75minutes.
\ncertain thing is that there is some difference between the submission time tested in local kaggle notebooks and the actual submission time under certain circumstances.
", "rawMarkdown": "I'm having a similar problem too. I don't know yet, but I suspect that the submission time is getting unexpectedly longer when combining multiple models into one model with tflite conversion(e.g. average of each fold). \n\nFor example, my 50ms single fold model completed submission within around 40 minutes, but the 5-fold average model of 50 ms(5x 10ms models) got a submission scoring error after 75minutes. \n\ncertain thing is that there is some difference between the submission time tested in local kaggle notebooks and the actual submission time under certain circumstances.", "replies": [ { "id": 2194220, "postDate": "2023-03-23T20:14:06.490Z", "content": "Interesting point. But this happens for me with one 1-fold model as well. I have submitted one ~100ms model and it was successful in only one attempt from ~10.
\nDon't know, some kind of magic :)
", "rawMarkdown": "Interesting point. But this happens for me with one 1-fold model as well. I have submitted one ~100ms model and it was successful in only one attempt from ~10.\n\nDon't know, some kind of magic :)" } ] } ] }, { "id": 2224142, "postDate": "2023-04-17T02:58:03.047Z", "content": "@sohier My submission is still gettimg time out around 60min as everyone reported. Have you already figured out why?
", "rawMarkdown": "@sohier My submission is still gettimg time out around 60min as everyone reported. Have you already figured out why?", "votes": 1 }, { "id": 2238475, "postDate": "2023-04-28T15:02:22.213Z", "content": "@sohier, (or if anyone else has ideas) I have made a few changes in my best model. It is only about 12 code line changes in the tflite model class which is submitted. The validation scores on the hold-out dataset are good when I run in kaggle kernels using the Evaluation page code to score. However when I submit the score drops massively. I have tried a lot of things to fix this. The same model without the lines changes are consistent on validation and leaderboard score.
\nNo custom_ops and fp32 is used in the tflite model creation. The submission run finishes in ~25 mins. I test in kaggle kernel to get validation score with the code on the Evaluation page, all except the below line… this one did not work in the kernels with any of my models but it was not a problem for previous models.
if REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\nThe new lines I added have a couple of extra tf functions, tf.tensor_scatter_nd_update, tf.logical_not, I tried to strip out any other extra tf functions.
\nCan you give any indication of the environment where submissions are executed to simulate the submission ? I understand it does not exactly use the code in the evaluation page. Are the tflite pkg versions the same ?
\nLet me know if the evaluation.py you use is released I could not see it; or even a similar script to reference.
ps. I also benchmark memory using the tflite benchmark tool and it is almost the same as the good model.
", "rawMarkdown": "@sohier, (or if anyone else has ideas) I have made a few changes in my best model. It is only about 12 code line changes in the tflite model class which is submitted. The validation scores on the hold-out dataset are good when I run in kaggle kernels using the Evaluation page code to score. However when I submit the score drops massively. I have tried a lot of things to fix this. The same model without the lines changes are consistent on validation and leaderboard score. \nNo `custom_ops` and fp32 is used in the tflite model creation. The submission run finishes in ~25 mins. I test in kaggle kernel to get validation score with the code on the Evaluation page, all except the below line... this one did not work in the kernels with any of my models but it was not a problem for previous models. \n```\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n```\nThe new lines I added have a couple of extra tf functions, `tf.tensor_scatter_nd_update`, `tf.logical_not`, I tried to strip out any other extra tf functions. \nCan you give any indication of the environment where submissions are executed to simulate the submission ? I understand it does not exactly use the code in the evaluation page. Are the tflite pkg versions the same ?\nLet me know if the `evaluation.py` you use is released I could not see it; or even a similar script to reference.\n\nps. I also benchmark memory using the [tflite benchmark tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark) and it is almost the same as the good model. " }, { "id": 2237243, "postDate": "2023-04-27T13:31:14.730Z", "content": "As team with highest amount of submissions I can recommend to just accept it and:
\nWe have got in total more than 100 errors. But when we started loading only solutions with 70-80ms, our results began to improve because we weren't wasting attempts on errors
", "rawMarkdown": "As team with highest amount of submissions I can recommend to just accept it and:\n1. use only stable submissions (for us it's like 70-80ms)\n2. manage submissions order and be ready for 90% errors if you are submitting 85-100ms\n\nWe have got in total more than 100 errors. But when we started loading only solutions with 70-80ms, our results began to improve because we weren't wasting attempts on errors", "replies": [ { "id": 2237248, "postDate": "2023-04-27T13:38:14.513Z", "content": "May I ask you how you test your file performance??
\nI've tried to measure performance on the training dataset with kaggle kernels, but even models with <70ms (around 68ms) still fail very often. I am wondering what I am missing…
\nWould love to hear your advice, please
We just use this simple code 3-5 times. And better in different time of day (because at night it faster for example)
\nst = time.time()\ncnt = \ntotal = \nmodel_time = \n\n i tqdm(np.linspace(, (train)-, total, dtype=)):\n sample = train.loc[i] \n\n yy = load_relevant_data_subset(base_dir + sample[])\n md_st = time.time()\n output = prediction_fn(inputs=yy)\n model_time += time.time() - md_st\n sign = np.argmax(output[])\n\n sample[] == sign: cnt +=\n\n() \n\n()\n()\nAlso note that (as far as I know) the machines you have access to during submissions are not the same as in the notebook editor or during commit.
", "rawMarkdown": "Also note that (as far as I know) the machines you have access to during submissions are not the same as in the notebook editor or during commit.", "votes": 1 }, { "id": 2237332, "postDate": "2023-04-27T14:38:17.850Z", "content": "Ah yeah, you are right. Sometimes we also used \"Save version\" button. But mostly local tests are enough
", "rawMarkdown": "Ah yeah, you are right. Sometimes we also used \"Save version\" button. But mostly local tests are enough ", "votes": 1 }, { "id": 2237374, "postDate": "2023-04-27T15:22:02.313Z", "content": "Thank you very much. I really appreciate your help!
", "rawMarkdown": "Thank you very much. I really appreciate your help!" }, { "id": 2237484, "postDate": "2023-04-27T16:53:36.043Z", "content": "Thank you for sharing it. I noticed you also calculate inference time and data loading time. Is data loading also considered a model inference time?
", "rawMarkdown": "Thank you for sharing it. I noticed you also calculate inference time and data loading time. Is data loading also considered a model inference time?" }, { "id": 2238201, "postDate": "2023-04-28T09:52:51.370Z", "content": "We mostly looking on only inference time without data loading. For example our last submission time:
\nMean time: \nMean time only infer: \nI have a feeling that data loading is considered as a loading time as well..
", "rawMarkdown": "I have a feeling that data loading is considered as a loading time as well.." } ] } ] }, { "id": 2237125, "postDate": "2023-04-27T11:49:33.937Z", "content": "@sohier I am facing the same problem as described in the comments below.
\nMy submission fails even when the time is less than 60 minutes.
\nMy last submission started at 14:38, and it failed at 15:39. So it fails within 60-61 minutes after the start, but you mentioned that we are given about 75 minutes.
I have checked the model inference time (for the tf-lite converted model in Kaggle CPU kernels), and it is 56ms. However, according to the rules, we should have a limit of 100ms. These measurements were made using the entire training dataset.
\nAnother strange thing is that the exact same model scored successfully yesterday (with the same weights).
", "rawMarkdown": "@sohier I am facing the same problem as described in the comments below. \n\nMy submission fails even when the time is **less than 60 minutes**. \nMy last submission started at 14:38, and it failed at 15:39. So it fails within 60-61 minutes after the start, but you mentioned that we are given about 75 minutes.\n\nI have checked the model inference time (for the tf-lite converted model in Kaggle CPU kernels), and it is 56ms. However, according to the rules, we should have a limit of 100ms. These measurements were made using the entire training dataset.\n\n**Another strange thing is that the exact same model scored successfully yesterday (with the same weights).**", "replies": [ { "id": 2238134, "postDate": "2023-04-28T08:36:40.960Z", "content": "I'm facing the same error, I got error exactly before it hits 60 minutes. Not as described 75minutes.
", "rawMarkdown": "I'm facing the same error, I got error exactly before it hits 60 minutes. Not as described 75minutes.", "replies": [ { "id": 2238135, "postDate": "2023-04-28T08:37:02.273Z", "content": "@sohier Please look into it .
", "rawMarkdown": "@sohier Please look into it .", "votes": 1 } ] } ] }, { "id": 2191152, "postDate": "2023-03-21T18:36:23.510Z", "content": "As per model evaluation page:
\nimport tflite_runtime.interpreter as tflite\ninterpreter = tflite.Interpreter(model_path)\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput = prediction_fn(inputs=frames)\nsign = np.argmax(output[\"outputs\"])\nWhat's the REQUIRED_SIGNATURE please?
", "rawMarkdown": "As per model evaluation page: \n\n```\nimport tflite_runtime.interpreter as tflite\ninterpreter = tflite.Interpreter(model_path)\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput = prediction_fn(inputs=frames)\nsign = np.argmax(output[\"outputs\"])\n```\n\nWhat's the REQUIRED_SIGNATURE please?", "replies": [ { "id": 2191431, "postDate": "2023-03-22T01:10:35.327Z", "content": "serving_default given here https://www.kaggle.com/competitions/asl-signs/overview/evaluation
Have you completed the invalidation yet @sohier? If not, please can you update this thread once it's complete so that we know whether we're still needing to checking our submissions?
", "rawMarkdown": "Have you completed the invalidation yet @sohier? If not, please can you update this thread once it's complete so that we know whether we're still needing to checking our submissions?" }, { "id": 2178979, "postDate": "2023-03-12T20:13:30.033Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2210090, "author_name": "arutema47", "author_url": "", "post_date": "2023-04-05T05:57:22.693000", "content": "@sohier The submission seems to be kicked in 60min. Is this behavior right?
", "votes": 9, "replies": [ { "id": 2237236, "author_name": "Mykola", "author_url": "", "post_date": "2023-04-27T13:20:30.487000", "content": "The same happened. Last week I submitted the model successfully. Now it failed a few times in a row. I have a strong feeling something changed. It also fails after 60 minutes, however, 70 min expected…
", "votes": 0, "replies": [] } ] }, { "id": 2204385, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-03-31T14:19:54.663000", "content": "Hi @sohier !
\nI was wondering what was going on when submitting. I've had models returning the \"Evaluation Exception: Inference time cap exceeded.error\" error after a ~1h03 run, which is weird since the error should be raised after ~75 minutes of runtime.
\n100ms / it + 10min margin implies there are about 32k images in the test data, is that actually the case ? Or could it be that the 10 min buffer from the evaluation page is not taken into account ?
\nOthers have also reported similar timings to receive the error :)
\nThanks !!
if the evalution script uses parallel processes, then time limit is 75/num of process
", "votes": 1, "replies": [] }, { "id": 2220495, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-04-13T12:56:43.840000", "content": "Poke @sohier ! 😊
", "votes": 2, "replies": [ { "id": 2234125, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-04-24T22:42:08.050000", "content": "@theoviel did you (or anyone else) find clarity on this issue? Did @sohier response somewhere else?
\nI have sumissions that fail at ~62 minutes. The same submissions run in less than 100ms/it in the kaggle notebook with TensorFlow Lite Runtime v2.9.1.
\nThe evaluation page says \"Expect to see approximately 40,000 videos in the test set. We allow an additional 10 minute buffer for loading the data and miscellaneous overhead.\" Shouldn't this equate to well above 60 minutes even without the 10 minute buffer?
", "votes": 2, "replies": [] }, { "id": 2234147, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-04-25T00:34:56.433000", "content": "Same here. My last submission timed out at exactly 1 hour. According to the evaluation page here, our submit notebooks should be allowed approximately 40000 * 0.1 / 60 + 10 minutes which equals approximately 76.7 minutes. That means a submit should not fail before 1 hour and 16.7 minutes (if there are 40000 test videos). Can anyone explain what is happening?
One answer is that test data only has 30000 videos. Then we would only have 60 minutes = 30000 * 0.1 / 60 + 10 minutes. Another answer is that the server uses some parallel processing.
\n", "votes": 3, "replies": [] }, { "id": 2234169, "author_name": "RB", "author_url": "", "post_date": "2023-04-25T01:15:04.757000", "content": "Your model must also perform inference with less than 100 milliseconds of latency per video on average and use less than 40 MB of storage space. Expect to see approximately 40,000 videos in the test set. We allow an additional 10 minute buffer for loading the data and miscellaneous overhead.
\n
This was posted few days back https://www.kaggle.com/competitions/asl-signs/discussion/403110#2229730
\nThey have been consistent with ~60 mins time .. it's possible that \"75 mins\" calculation was incorrectly mentioned in evaluation page - but this is just me speculating
", "votes": 1, "replies": [] }, { "id": 2234473, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-04-25T08:18:52.537000", "content": "No update on this, and I don't think we will get one. Maybe there is an inconsistency between Sohier's code an what is written on the evaluation page, or maybe Kaggle is doing optimizations during submissions that speed up inference.
\nThis should have been addressed a month ago though, one week before the deadline is too late to change anything.
", "votes": 5, "replies": [] } ] } ] }, { "id": 2191434, "author_name": "RB", "author_url": "", "post_date": "2023-03-22T01:12:21.780000", "content": "It seems we are getting \"Submission Scoring Error\" if we exceed runtime … Is that correct ?
\nReason I ask is my 2 fold model worked in ~ 42 mins and 3 fold model returned \"submission scoring error\" in ~60 minutes.
", "votes": 3, "replies": [ { "id": 2191724, "author_name": "Andrij", "author_url": "", "post_date": "2023-03-22T06:58:15.537000", "content": "Personally, I received a Submission Scoring Error in two cases:
\nYes, I was puzzled because we typically get \"runtime exceeded error\" for going beyond.
", "votes": 1, "replies": [] }, { "id": 2191995, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-03-22T11:09:27.563000", "content": "We gettin \"runtime exceeded error\" when you making predicts in your notebook. But in this competition in notebook you only load submission.zip and all computations are in \"Scoring\" part. Where usually Kaggle just calculate metric from your submission.csv predictions.
", "votes": 4, "replies": [] } ] }, { "id": 2191991, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-03-22T11:05:10.397000", "content": "Agree. I keep getting Submission Scoring Error. After 60-65 minutes after submit. And the same code have succeeded once.
\nI think that means that model converting is correct, model size is correct.
Also @sohier said about ~75 minutes, how it can return error after 60?
", "votes": 3, "replies": [ { "id": 2192157, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-03-22T13:17:24.513000", "content": "For example today's submission. In my location submissions counter reloads at 03:00 , so i have submitted my notebook at 03:01, ~2 minutes for preparing submission.zip. At ~03:02 scoring has started
\nAnd at 04:04 I already found \"Submission Scoring Error\"
\n
Local scoring time for 40,000 samples:
\nMean time including data loading time, only infer counts only model runtime.
st = time.time()\ncnt = \ntotal = \nmodel_time = \n\n i tqdm(np.linspace(, (train)-, total, dtype=)):\n sample = train.loc[i] \n\n yy = load_relevant_data_subset(base_dir + sample[])\n md_st = time.time()\n output = prediction_fn(inputs=yy)\n model_time += time.time() - md_st\n sign = np.argmax(output[])\n \n\n sample[] == sign: cnt +=\n\n() \n\n()\n()\n%|██████████| / [:06:<:, it/s]\nMean time: \nMean time only infer: \nI'm having a similar problem too. I don't know yet, but I suspect that the submission time is getting unexpectedly longer when combining multiple models into one model with tflite conversion(e.g. average of each fold).
\nFor example, my 50ms single fold model completed submission within around 40 minutes, but the 5-fold average model of 50 ms(5x 10ms models) got a submission scoring error after 75minutes.
\ncertain thing is that there is some difference between the submission time tested in local kaggle notebooks and the actual submission time under certain circumstances.
", "votes": 0, "replies": [ { "id": 2194220, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-03-23T20:14:06.490000", "content": "Interesting point. But this happens for me with one 1-fold model as well. I have submitted one ~100ms model and it was successful in only one attempt from ~10.
\nDon't know, some kind of magic :)
", "votes": 0, "replies": [] } ] } ] }, { "id": 2224142, "author_name": "Camaro", "author_url": "", "post_date": "2023-04-17T02:58:03.047000", "content": "@sohier My submission is still gettimg time out around 60min as everyone reported. Have you already figured out why?
", "votes": 1, "replies": [] }, { "id": 2238475, "author_name": "Darragh", "author_url": "", "post_date": "2023-04-28T15:02:22.213000", "content": "@sohier, (or if anyone else has ideas) I have made a few changes in my best model. It is only about 12 code line changes in the tflite model class which is submitted. The validation scores on the hold-out dataset are good when I run in kaggle kernels using the Evaluation page code to score. However when I submit the score drops massively. I have tried a lot of things to fix this. The same model without the lines changes are consistent on validation and leaderboard score.
\nNo custom_ops and fp32 is used in the tflite model creation. The submission run finishes in ~25 mins. I test in kaggle kernel to get validation score with the code on the Evaluation page, all except the below line… this one did not work in the kernels with any of my models but it was not a problem for previous models.
if REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\nThe new lines I added have a couple of extra tf functions, tf.tensor_scatter_nd_update, tf.logical_not, I tried to strip out any other extra tf functions.
\nCan you give any indication of the environment where submissions are executed to simulate the submission ? I understand it does not exactly use the code in the evaluation page. Are the tflite pkg versions the same ?
\nLet me know if the evaluation.py you use is released I could not see it; or even a similar script to reference.
ps. I also benchmark memory using the tflite benchmark tool and it is almost the same as the good model.
", "votes": 0, "replies": [] }, { "id": 2237243, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-04-27T13:31:14.730000", "content": "As team with highest amount of submissions I can recommend to just accept it and:
\nWe have got in total more than 100 errors. But when we started loading only solutions with 70-80ms, our results began to improve because we weren't wasting attempts on errors
", "votes": 0, "replies": [ { "id": 2237248, "author_name": "Mykola", "author_url": "", "post_date": "2023-04-27T13:38:14.513000", "content": "May I ask you how you test your file performance??
\nI've tried to measure performance on the training dataset with kaggle kernels, but even models with <70ms (around 68ms) still fail very often. I am wondering what I am missing…
\nWould love to hear your advice, please
We just use this simple code 3-5 times. And better in different time of day (because at night it faster for example)
\nst = time.time()\ncnt = \ntotal = \nmodel_time = \n\n i tqdm(np.linspace(, (train)-, total, dtype=)):\n sample = train.loc[i] \n\n yy = load_relevant_data_subset(base_dir + sample[])\n md_st = time.time()\n output = prediction_fn(inputs=yy)\n model_time += time.time() - md_st\n sign = np.argmax(output[])\n\n sample[] == sign: cnt +=\n\n() \n\n()\n()\nAlso note that (as far as I know) the machines you have access to during submissions are not the same as in the notebook editor or during commit.
", "votes": 1, "replies": [] }, { "id": 2237332, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-04-27T14:38:17.850000", "content": "Ah yeah, you are right. Sometimes we also used \"Save version\" button. But mostly local tests are enough
", "votes": 1, "replies": [] }, { "id": 2237374, "author_name": "Mykola", "author_url": "", "post_date": "2023-04-27T15:22:02.313000", "content": "Thank you very much. I really appreciate your help!
", "votes": 0, "replies": [] }, { "id": 2237484, "author_name": "Mykola", "author_url": "", "post_date": "2023-04-27T16:53:36.043000", "content": "Thank you for sharing it. I noticed you also calculate inference time and data loading time. Is data loading also considered a model inference time?
", "votes": 0, "replies": [] }, { "id": 2238201, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-04-28T09:52:51.370000", "content": "We mostly looking on only inference time without data loading. For example our last submission time:
\nMean time: \nMean time only infer: \nI have a feeling that data loading is considered as a loading time as well..
", "votes": 0, "replies": [] } ] } ] }, { "id": 2237125, "author_name": "Ilya Novoselskiy", "author_url": "", "post_date": "2023-04-27T11:49:33.937000", "content": "@sohier I am facing the same problem as described in the comments below.
\nMy submission fails even when the time is less than 60 minutes.
\nMy last submission started at 14:38, and it failed at 15:39. So it fails within 60-61 minutes after the start, but you mentioned that we are given about 75 minutes.
I have checked the model inference time (for the tf-lite converted model in Kaggle CPU kernels), and it is 56ms. However, according to the rules, we should have a limit of 100ms. These measurements were made using the entire training dataset.
\nAnother strange thing is that the exact same model scored successfully yesterday (with the same weights).
", "votes": 0, "replies": [ { "id": 2238134, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2023-04-28T08:36:40.960000", "content": "I'm facing the same error, I got error exactly before it hits 60 minutes. Not as described 75minutes.
", "votes": 0, "replies": [ { "id": 2238135, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2023-04-28T08:37:02.273000", "content": "@sohier Please look into it .
", "votes": 1, "replies": [] } ] } ] }, { "id": 2191152, "author_name": "Aman Arora", "author_url": "", "post_date": "2023-03-21T18:36:23.510000", "content": "As per model evaluation page:
\nimport tflite_runtime.interpreter as tflite\ninterpreter = tflite.Interpreter(model_path)\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput = prediction_fn(inputs=frames)\nsign = np.argmax(output[\"outputs\"])\nWhat's the REQUIRED_SIGNATURE please?
", "votes": 0, "replies": [ { "id": 2191431, "author_name": "RB", "author_url": "", "post_date": "2023-03-22T01:10:35.327000", "content": "serving_default given here https://www.kaggle.com/competitions/asl-signs/overview/evaluation
Have you completed the invalidation yet @sohier? If not, please can you update this thread once it's complete so that we know whether we're still needing to checking our submissions?
", "votes": 0, "replies": [] }, { "id": 2178979, "author_name": "", "author_url": "", "post_date": "2023-03-12T20:13:30.033000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2175328": "We discovered that, due to a version control mistake the metric constraints around model runtime and size weren't being applied as we intended. I've patched the metric:\n- You should now get an error message more promptly if your model is too large\n- Submissions will be rejected properly if their runtime exceeds the cap (about 75 minutes). \n\nAbout 60 submissions received scores when they should have been rejected due to the time constraint; I will be invalidating those submissions shortly. \n\nApologies for the disruption and confusion.", "2210090": "@sohier The submission seems to be kicked in 60min. Is this behavior right?", "2204385": "Hi @sohier !\n\nI was wondering what was going on when submitting. I've had models returning the \"Evaluation Exception: Inference time cap exceeded.error\" error after a ~1h03 run, which is weird since the error should be raised after ~75 minutes of runtime.\n\n100ms / it + 10min margin implies there are about 32k images in the test data, is that actually the case ? Or could it be that the 10 min buffer from the evaluation page is not taken into account ?\n\nOthers have also reported similar timings to receive the error :)\nThanks !!", "2191434": "It seems we are getting \"Submission Scoring Error\" if we exceed runtime ... Is that correct ? \n\nReason I ask is my 2 fold model worked in ~ 42 mins and 3 fold model returned \"submission scoring error\" in ~60 minutes. ", "2224142": "@sohier My submission is still gettimg time out around 60min as everyone reported. Have you already figured out why?", "2238475": "@sohier, (or if anyone else has ideas) I have made a few changes in my best model. It is only about 12 code line changes in the tflite model class which is submitted. The validation scores on the hold-out dataset are good when I run in kaggle kernels using the Evaluation page code to score. However when I submit the score drops massively. I have tried a lot of things to fix this. The same model without the lines changes are consistent on validation and leaderboard score. \nNo `custom_ops` and fp32 is used in the tflite model creation. The submission run finishes in ~25 mins. I test in kaggle kernel to get validation score with the code on the Evaluation page, all except the below line... this one did not work in the kernels with any of my models but it was not a problem for previous models. \n```\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n```\nThe new lines I added have a couple of extra tf functions, `tf.tensor_scatter_nd_update`, `tf.logical_not`, I tried to strip out any other extra tf functions. \nCan you give any indication of the environment where submissions are executed to simulate the submission ? I understand it does not exactly use the code in the evaluation page. Are the tflite pkg versions the same ?\nLet me know if the `evaluation.py` you use is released I could not see it; or even a similar script to reference.\n\nps. I also benchmark memory using the [tflite benchmark tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark) and it is almost the same as the good model. ", "2237243": "As team with highest amount of submissions I can recommend to just accept it and:\n1. use only stable submissions (for us it's like 70-80ms)\n2. manage submissions order and be ready for 90% errors if you are submitting 85-100ms\n\nWe have got in total more than 100 errors. But when we started loading only solutions with 70-80ms, our results began to improve because we weren't wasting attempts on errors", "2237125": "@sohier I am facing the same problem as described in the comments below. \n\nMy submission fails even when the time is **less than 60 minutes**. \nMy last submission started at 14:38, and it failed at 15:39. So it fails within 60-61 minutes after the start, but you mentioned that we are given about 75 minutes.\n\nI have checked the model inference time (for the tf-lite converted model in Kaggle CPU kernels), and it is 56ms. However, according to the rules, we should have a limit of 100ms. These measurements were made using the entire training dataset.\n\n**Another strange thing is that the exact same model scored successfully yesterday (with the same weights).**", "2191152": "As per model evaluation page: \n\n```\nimport tflite_runtime.interpreter as tflite\ninterpreter = tflite.Interpreter(model_path)\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput = prediction_fn(inputs=frames)\nsign = np.argmax(output[\"outputs\"])\n```\n\nWhat's the REQUIRED_SIGNATURE please?", "2176764": "Have you completed the invalidation yet @sohier? If not, please can you update this thread once it's complete so that we know whether we're still needing to checking our submissions?", "2178979": "" } }