Hi Greetings,
\nI've been struggling with the submission scoring error, for a long time, does anyone please give me why it's happening and how we can solve it? I got some reference discussion but doesn't help. If anyone can help me always welcome them to my team I do have a great model so we can fix the issue together and make a good submission
Hi Greetings,
\nI've been struggling with the submission scoring error, for a long time, does anyone please give me why it's happening and how we can solve it? I got some reference discussion but doesn't help. If anyone can help me always welcome them to my team I do have a great model so we can fix the issue together and make a good submission
I finally find out the solution for the error, The issue is the inference time for the model, My current model's times
\nWER: 9.56117
\nMean time: 3.0161214
\nMean time only infer: 2.6765670
depending on my research on the competition rules the model should run within 5 hours
\n\n\nYour model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.
\n
the approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.
", "rawMarkdown": "I finally find out the solution for the error, The issue is the inference time for the model, My current model's times\nWER: 9.56117\nMean time: 3.0161214\nMean time only infer: 2.6765670\n\ndepending on my research on the competition rules the model should run within 5 hours\n\n>Your model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.\n\nthe approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.", "votes": null }, { "id": "2379387", "postDate": "08/08/2023 06:25:13", "content": "Here is the sample code to check your Mean inference time
\nCreate your TFLite model
time\n json\n tqdm.auto tqdm\n Levenshtein Lev\n\n\nSEL_FEATURES = json.load(())[]\n\n ():\n pd.read_parquet(pq_path, columns=SEL_FEATURES) \n\n (, ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\n\n\ndf = pd.read_csv()\n\nidx = \nsample = df.loc[idx]\nloaded = load_relevant_data_subset( + sample[])\nloaded = loaded[loaded.index==sample[]].values\n(loaded.shape)\nframes = loaded\n\n ():\n w1 = (s1.split())\n lvd = Lev.distance(s1, s2)\n lvd / w1\n\n\n\nfound_signatures = (interpreter.get_signature_list().keys())\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n REQUIRED_SIGNATURE found_signatures:\n KernelEvalException()\n\nprediction_fn = interpreter.get_signature_runner()\noutput_lite = prediction_fn(inputs=frames)\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_lite[REQUIRED_OUTPUT], axis=)])\n(prediction_str)\n\n\nst = time.time()\ncnt = \ntotal = \nmodel_time = \n\nlevs = []\n\n i tqdm(((df.iloc[:total]))):\n sample = df.loc[i]\n loaded = load_relevant_data_subset( + sample[])\n loaded = loaded[loaded.index==sample[]].values\n\n md_st = time.time()\n output_ = prediction_fn(inputs=loaded)\n model_time += time.time() - md_st\n\n\n prediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_[REQUIRED_OUTPUT], axis=)])\n cur_lev = wer__(sample[], prediction_str) \n\n levs.append(cur_lev)\n\n()\n()\n()\nI'm also struggling with this issue right now. I got submission scoring error 1min after the submission. Importing the model with tf.lite.Interpreter works fine inside the notebooks. I hypothesized it could be a versioning problem. Indeed, installing tflite-runtime-2.9.1 and importing my model failed (using tflite_runtime.interpreter). I think this is so because I built my model in collab and some ops aren't supported by tflight 2.9.1. Maybe, this finding helps somebody
", "rawMarkdown": "I'm also struggling with this issue right now. I got submission scoring error 1min after the submission. Importing the model with tf.lite.Interpreter works fine inside the notebooks. I hypothesized it could be a versioning problem. Indeed, installing tflite-runtime-2.9.1 and importing my model failed (using tflite_runtime.interpreter). I think this is so because I built my model in collab and some ops aren't supported by tflight 2.9.1. Maybe, this finding helps somebody", "votes": null }, { "id": "2380287", "postDate": "08/08/2023 14:25:07", "content": "Just read the solution which I posted below, the inference time is causing the submission error, just use the below code to check you inference time and make sure that the mean only inference time is below 0.6
", "rawMarkdown": "Just read the solution which I posted below, the inference time is causing the submission error, just use the below code to check you inference time and make sure that the mean only inference time is below 0.6", "votes": null }, { "id": "2392725", "postDate": "08/15/2023 20:41:53", "content": "Hello Noel - Your \"the approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.\" tracks well with my Submission Scoring Error conundrums. Appreciate the code snippet to track the times! Thanks, Matthew
", "rawMarkdown": "Hello Noel - Your \"the approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.\" tracks well with my Submission Scoring Error conundrums. Appreciate the code snippet to track the times! Thanks, Matthew", "votes": null } ], "comments": [ { "id": 2379383, "author_name": "noelsjacob", "author_url": "", "post_date": "08/08/2023 06:23:18", "content": "I finally find out the solution for the error, The issue is the inference time for the model, My current model's times
\nWER: 9.56117
\nMean time: 3.0161214
\nMean time only infer: 2.6765670
depending on my research on the competition rules the model should run within 5 hours
\n\n\nYour model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.
\n
the approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.
", "votes": null, "replies": [] }, { "id": 2379387, "author_name": "noelsjacob", "author_url": "", "post_date": "08/08/2023 06:25:13", "content": "Here is the sample code to check your Mean inference time
\nCreate your TFLite model
time\n json\n tqdm.auto tqdm\n Levenshtein Lev\n\n\nSEL_FEATURES = json.load(())[]\n\n ():\n pd.read_parquet(pq_path, columns=SEL_FEATURES) \n\n (, ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\n\n\ndf = pd.read_csv()\n\nidx = \nsample = df.loc[idx]\nloaded = load_relevant_data_subset( + sample[])\nloaded = loaded[loaded.index==sample[]].values\n(loaded.shape)\nframes = loaded\n\n ():\n w1 = (s1.split())\n lvd = Lev.distance(s1, s2)\n lvd / w1\n\n\n\nfound_signatures = (interpreter.get_signature_list().keys())\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n REQUIRED_SIGNATURE found_signatures:\n KernelEvalException()\n\nprediction_fn = interpreter.get_signature_runner()\noutput_lite = prediction_fn(inputs=frames)\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_lite[REQUIRED_OUTPUT], axis=)])\n(prediction_str)\n\n\nst = time.time()\ncnt = \ntotal = \nmodel_time = \n\nlevs = []\n\n i tqdm(((df.iloc[:total]))):\n sample = df.loc[i]\n loaded = load_relevant_data_subset( + sample[])\n loaded = loaded[loaded.index==sample[]].values\n\n md_st = time.time()\n output_ = prediction_fn(inputs=loaded)\n model_time += time.time() - md_st\n\n\n prediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_[REQUIRED_OUTPUT], axis=)])\n cur_lev = wer__(sample[], prediction_str) \n\n levs.append(cur_lev)\n\n()\n()\n()\nI'm also struggling with this issue right now. I got submission scoring error 1min after the submission. Importing the model with tf.lite.Interpreter works fine inside the notebooks. I hypothesized it could be a versioning problem. Indeed, installing tflite-runtime-2.9.1 and importing my model failed (using tflite_runtime.interpreter). I think this is so because I built my model in collab and some ops aren't supported by tflight 2.9.1. Maybe, this finding helps somebody
", "votes": null, "replies": [] }, { "id": 2380287, "author_name": "noelsjacob", "author_url": "", "post_date": "08/08/2023 14:25:07", "content": "Just read the solution which I posted below, the inference time is causing the submission error, just use the below code to check you inference time and make sure that the mean only inference time is below 0.6
", "votes": null, "replies": [] }, { "id": 2392725, "author_name": "matucker", "author_url": "", "post_date": "08/15/2023 20:41:53", "content": "Hello Noel - Your \"the approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.\" tracks well with my Submission Scoring Error conundrums. Appreciate the code snippet to track the times! Thanks, Matthew
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2379123": "Hi Greetings,\nI've been struggling with the submission scoring error, for a long time, does anyone please give me why it's happening and how we can solve it? I got some reference discussion but doesn't help. If anyone can help me always welcome them to my team I do have a great model so we can fix the issue together and make a good submission", "2379383": "I finally find out the solution for the error, The issue is the inference time for the model, My current model's times\nWER: 9.56117\nMean time: 3.0161214\nMean time only infer: 2.6765670\n\ndepending on my research on the competition rules the model should run within 5 hours\n\n>Your model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.\n\nthe approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.", "2379387": "Here is the sample code to check your Mean inference time\nCreate your TFLite model\n\n\n```python\nimport time\nimport json\nfrom tqdm.auto import tqdm\nimport Levenshtein as Lev\n\n\nSEL_FEATURES = json.load(open('/kaggle/working/inference_args.json'))['selected_columns']\n\ndef load_relevant_data_subset(pq_path):\n return pd.read_parquet(pq_path, columns=SEL_FEATURES) #selected_columns)\n\nwith open (\"/kaggle/input/asl-fingerspelling/character_to_prediction_index.json\", \"r\") as f:\n character_map = json.load(f)\nrev_character_map = {j:i for i,j in character_map.items()}\n\n\ndf = pd.read_csv('/kaggle/input/asl-fingerspelling/train.csv')\n\nidx = 0\nsample = df.loc[idx]\nloaded = load_relevant_data_subset('/kaggle/input/asl-fingerspelling/' + sample['path'])\nloaded = loaded[loaded.index==sample['sequence_id']].values\nprint(loaded.shape)\nframes = loaded\n\ndef wer__(s1, s2):\n w1 = len(s1.split())\n lvd = Lev.distance(s1, s2)\n return lvd / w1\n\n# import tflite_runtime.interpreter as tflite\n# interpreter = tflite.Interpreter('model.tflite')\nfound_signatures = list(interpreter.get_signature_list().keys())\n\nREQUIRED_SIGNATURE = 'serving_default'\nREQUIRED_OUTPUT = 'outputs'\nif REQUIRED_SIGNATURE not in found_signatures:\n raise KernelEvalException('Required input signature not found.')\n\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\noutput_lite = prediction_fn(inputs=frames)\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output_lite[REQUIRED_OUTPUT], axis=1)])\nprint(prediction_str)\n\n\nst = time.time()\ncnt = 0\ntotal = 100\nmodel_time = 0\n\nlevs = []\n\nfor i in tqdm(range(len(df.iloc[:total]))):\n sample = df.loc[i]\n loaded = load_relevant_data_subset('/kaggle/input/asl-fingerspelling/' + sample['path'])\n loaded = loaded[loaded.index==sample['sequence_id']].values\n\n md_st = time.time()\n output_ = prediction_fn(inputs=loaded)\n model_time += time.time() - md_st\n\n\n prediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output_[REQUIRED_OUTPUT], axis=1)])\n cur_lev = wer__(sample['phrase'], prediction_str) \n\n levs.append(cur_lev)\n\nprint(f'WER: {np.mean(levs):.5f}')\nprint(f'Mean time: {(time.time() - st)/total:.7f}')\nprint(f'Mean time only infer: {model_time/total:.7f}')\n\n```", "2379985": "I'm also struggling with this issue right now. I got submission scoring error 1min after the submission. Importing the model with tf.lite.Interpreter works fine inside the notebooks. I hypothesized it could be a versioning problem. Indeed, installing tflite-runtime-2.9.1 and importing my model failed (using tflite_runtime.interpreter). I think this is so because I built my model in collab and some ops aren't supported by tflight 2.9.1. Maybe, this finding helps somebody", "2380287": "Just read the solution which I posted below, the inference time is causing the submission error, just use the below code to check you inference time and make sure that the mean only inference time is below 0.6", "2392725": "Hello Noel - Your \"the approximate maximum Mean Only Inference Time for the model is nearly 0.6 seconds from my calculation.\" tracks well with my Submission Scoring Error conundrums. Appreciate the code snippet to track the times! Thanks, Matthew" }, "source": "meta" }