Hello everyone! Nice to see that type of competition again
\nI have one question. Could hosts please clarify meaning of
\n\n\nYour model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.
\n
How to calculate how many phrases 35 hours includes? I would like to estimate one phrase inference time or how many phrases we have in that 35 hours.
\nThanks
\nUPDATE1. Also faced issues with submission. But can't detect the reason with info from Evaluation page. More info posted below
\nUPDATE2. Fixed
\nUPDATE3. Approximation of inference time. here and here
", "messages": [ { "id": "2256169", "postDate": "05/12/2023 09:50:59", "content": "Hello everyone! Nice to see that type of competition again
\nI have one question. Could hosts please clarify meaning of
\n\n\nYour model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.
\n
How to calculate how many phrases 35 hours includes? I would like to estimate one phrase inference time or how many phrases we have in that 35 hours.
\nThanks
\nUPDATE1. Also faced issues with submission. But can't detect the reason with info from Evaluation page. More info posted below
\nUPDATE2. Fixed
\nUPDATE3. Approximation of inference time. here and here
", "rawMarkdown": "Hello everyone! Nice to see that type of competition again\n\nI have one question. Could hosts please clarify meaning of \n>Your model must also perform inference in less than 5 hours and use less than 40 MB of storage space. Expect to see approximately 35 hours of video in the test set.\n\nHow to calculate how many phrases 35 hours includes? I would like to estimate one phrase inference time or how many phrases we have in that 35 hours.\n\nThanks\n\n\nUPDATE1. Also faced issues with submission. But can't detect the reason with info from Evaluation page. More info posted [below](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/409722#2273788)\n\n\nUPDATE2. [Fixed](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/409722#2276829)\n\nUPDATE3. Approximation of inference time. [here](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/409722#2276907) and [here](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/409722#2308349)", "votes": null }, { "id": "2272974", "postDate": "05/24/2023 22:17:09", "content": "After submissions became available I think this topic is actual again.
\nI have few submits with ~40-42 minutes with Submission Scoring Error. But:
\n\nYour model must also perform inference in less than 5 hours
\n
Or maybe there are some kind of multiprocessing and the time cap is 45 minutes?
\nDoes anyone have successful submissions longer than 50 minutes? Any ideas about mean per sample inference time?
\nupd. Error in my submission not because of 'out of time' . I deleted most of layers from my model and it has Error at ~8 minutes.
\nSo I don't understand the problem. My input is (time_dim, len(selected_columns)) and output with (new_time_dim, 59) and still getting errors.
\nModel under 40 MB, capable with tflite, same environment with successful public kernels. Maybe here some another limitations I didn't know? At Kaggle kernels it works fine with Evaluation example code 😨
the only think I was capable of submitting were dummy submissions not processing the frames.
", "rawMarkdown": "the only think I was capable of submitting were dummy submissions not processing the frames.", "votes": null }, { "id": "2273492", "postDate": "05/25/2023 07:33:02", "content": "if you click on the triangle with error - do you get any more information under Submission Scoring Error in the Submission Details popup? e.g. \"Your notebook generated a submission file with incorrect format.\" or other message? Know usually very little is give when submissions fail.
\nI can confirm using inference_args.json for selected columns works ok, but not if you only were using frame column. Tried that for some simple tests. Had issues with preprocessing layer though so far.
\nIf for some reason your output is empty, you can have error for not an iterator here -
\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])
\nSo perhaps a default output can be returned like a space for exceptions.
Think there are still open questions about what is provided in the video via
\ndef load_relevant_data_subset(pq_path):
\nis it a parquet file of many sequence ids like in train? or one parquet file for each sequence id to predict like in GISLR?
\nIf the former, then is the expectation to split the output strings per sequence id? In train there is one parquet that has 287 sequence ids ( /kaggle/input/asl-fingerspelling/train_landmarks/450474571.parquet ), the rest have 1000 but perhaps in test all have 1000.
@georgemegre seems to have been able to do more than dummy submissions. Without sharing too many details maybe has some thoughts on tips for true submissions under 5hrs? Thanks in advance.
", "rawMarkdown": "if you click on the triangle with error - do you get any more information under Submission Scoring Error in the Submission Details popup? e.g. \"Your notebook generated a submission file with incorrect format.\" or other message? Know usually very little is give when submissions fail.\n\nI can confirm using inference_args.json for selected columns works ok, but not if you only were using frame column. Tried that for some simple tests. Had issues with preprocessing layer though so far. \n\nIf for some reason your output is empty, you can have error for not an iterator here -\n`prediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])`\nSo perhaps a default output can be returned like a space for exceptions.\n \nThink there are still open questions about what is provided in the video via \n`def load_relevant_data_subset(pq_path):`\nis it a parquet file of many sequence ids like in train? or one parquet file for each sequence id to predict like in GISLR?\nIf the former, then is the expectation to split the output strings per sequence id? In train there is one parquet that has 287 sequence ids ( /kaggle/input/asl-fingerspelling/train_landmarks/450474571.parquet ), the rest have 1000 but perhaps in test all have 1000. \n\n@georgemegre seems to have been able to do more than dummy submissions. Without sharing too many details maybe has some thoughts on tips for true submissions under 5hrs? Thanks in advance.", "votes": null }, { "id": "2273547", "postDate": "05/25/2023 08:14:38", "content": "Error message has nothing helpful, this message was prepared for submitting csv files and just told about rows and columns.
\n\n\nIf for some reason your output is empty, you can have error
\n
I tested empty input with time_dim=0 and with full nan input. There are no way that my model has zero or empty output. Only if selected columns are wrong, but this scoring for about 40 minutes I think everything is ok with this part.
\nAlso my model capable for any length of input. For single frame or multiple frames from one csv. Output phrases would be just as one big string if so.
", "rawMarkdown": "Error message has nothing helpful, this message was prepared for submitting csv files and just told about rows and columns.\n\n> If for some reason your output is empty, you can have error\n\nI tested empty input with time_dim=0 and with full nan input. There are no way that my model has zero or empty output. Only if selected columns are wrong, but this scoring for about 40 minutes I think everything is ok with this part.\n\nAlso my model capable for any length of input. For single frame or multiple frames from one csv. Output phrases would be just as one big string if so.", "votes": null }, { "id": "2273788", "postDate": "05/25/2023 12:06:40", "content": "\n
\n
\n
WIth fixed preprocessing code it works
\n!pip install tflite-runtime==\n tflite_runtime.interpreter tflite\n\n time\n json\n pandas pd\n tqdm.auto tqdm\n Levenshtein Lev\n tensorflow tf\n\nconverter = tf.lite.TFLiteConverter.from_keras_model(infr) \ntflite_model = converter.convert()\n (, ) f:\n f.write(tflite_model)\n! submission. \n\n\nSEL_FEATURES = json.load(())[]\n\n ():\n pd.read_parquet(pq_path, columns=SEL_FEATURES) \n\n (, ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\n\n\ndf = pd.read_csv()\n\nidx = \nsample = df.loc[idx]\nloaded = load_relevant_data_subset( + sample[])\nloaded = loaded[loaded.index==sample[]].values\n(loaded.shape)\nframes = loaded\n\n ():\n w1 = (s1.split())\n lvd = Lev.distance(s1, s2)\n lvd / w1\n\n tflite_runtime.interpreter tflite\ninterpreter = tflite.Interpreter()\nfound_signatures = (interpreter.get_signature_list().keys())\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n REQUIRED_SIGNATURE found_signatures:\n KernelEvalException()\n\nprediction_fn = interpreter.get_signature_runner()\noutput_lite = prediction_fn(inputs=frames)\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_lite[REQUIRED_OUTPUT], axis=)])\n(prediction_str)\n\n\nst = time.time()\ncnt = \ntotal = \nmodel_time = \n\nlevs = []\n\n i tqdm(((df.iloc[:total]))):\n sample = df.loc[i]\n loaded = load_relevant_data_subset( + sample[])\n loaded = loaded[loaded.index==sample[]].values\n\n md_st = time.time()\n output_ = prediction_fn(inputs=loaded)\n model_time += time.time() - md_st\n\n\n prediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_[REQUIRED_OUTPUT], axis=)])\n cur_lev = wer__(sample[], prediction_str) \n \n \n\n levs.append(cur_lev)\n\n()\n()\n()\nMy submission run about 1 hour, the pipeline is very similar to previous competition, except that output is (time, 59). It also can be a problem in tflite version, it should be 2.9.1. i've checked my sub with this code (hope this will helpful):
\npython\n ():\n df = pd.read_parquet(pq_path, columns=selected_columns)\n df = df.loc[]\n df.values\n\n (json_fn) f:\n cols = json.load(f)[]\n\nframes = load_relevant_data_subset(, cols)\n\ninterpreter = tf.lite.Interpreter(sub_model_path)\n\nprediction_fn = interpreter.get_signature_runner()\nt = time()\noutput = prediction_fn(inputs=frames)\n()\n(*(time() - t) )\nt = time()\noutput = prediction_fn(inputs=frames)\n()\n(*(time() - t) )\n\nout_2 = output[]\n(out_2.shape)\n(np.argmax(out_2, axis=))\n\n (, ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(out_2, axis=)])\nprediction_str\nThanks for info. Could you please share frames shape and out_2 shape? They are same by time axis or different?
hm, to be honest i don't understand how you calculate it, but i can explain my understanding by two examples (hope this will be helpful):
\n1)
\ny_true = my bare face in the wind
\ny_pred = my bare face sa the wind
\nN = 24
\nLEVD = 2
\nMetric = 22/24 = ~0.91
But if we have:
\ny_true = my bare face in the wind
\ny_pred = \"sssssssssssssssssssssssss wdaedaj\"
\nN = 24
\nLEVD = 30
\nMetric = 24-30 / 24 = -0.25
So, it should be in the range (-d_min, 1).
", "rawMarkdown": "hm, to be honest i don't understand how you calculate it, but i can explain my understanding by two examples (hope this will be helpful):\n1)\ny_true = my bare face in the wind\ny_pred = my bare face sa the wind\nN = 24\nLEVD = 2\nMetric = 22/24 = ~0.91\n\nBut if we have:\ny_true = my bare face in the wind\ny_pred = \"sssssssssssssssssssssssss wdaedaj\"\nN = 24\nLEVD = 30\nMetric = 24-30 / 24 = -0.25\n\nSo, it should be in the range (-d_min, 1).", "votes": null }, { "id": "2274139", "postDate": "05/25/2023 16:41:15", "content": "Sure, here it is:
\nframes shape: (236, 252)
\nout_2 shape: (19, 59)
Thanks again. So now I'm totally don't understand why I can't submit my model)
", "rawMarkdown": "Thanks again. So now I'm totally don't understand why I can't submit my model)", "votes": null }, { "id": "2276829", "postDate": "05/27/2023 08:32:56", "content": "I finally found the problem with submission.
\nIn previous competition I filled nans with this code. And also I unsqueezed tensor like this. (I changed both lines, didn't try change it one by one)
\nx = tf.where(tfnp.isnan(x), , x)\nx = tf.expand_dims(x, axis=)\nIt worked fine in previous competition. And it work fine here in local kernel. But has error while submission scoring
\nI just changed this lines to:
\nx = tf.where(tfnp.isnan(x), tf.zeros_like(x), x)\nx = x[]\nMaybe it will help someone. Not so obvious reasons, as for me
", "rawMarkdown": "I finally found the problem with submission. \n\nIn previous competition I filled nans with this code. And also I unsqueezed tensor like this. (I changed both lines, didn't try change it one by one)\n\n```python\nx = tf.where(tfnp.isnan(x), 0.0, x)\nx = tf.expand_dims(x, axis=0)\n```\n\n\nIt worked fine in previous competition. And it work fine here in local kernel. But has error while submission scoring\n\nI just changed this lines to:\n```python\nx = tf.where(tfnp.isnan(x), tf.zeros_like(x), x)\nx = x[None]\n```\n\nMaybe it will help someone. Not so obvious reasons, as for me", "votes": null }, { "id": "2276907", "postDate": "05/27/2023 09:56:56", "content": "I also calculated the approximate time of one sample inference.
\nSubmission scoring lasts ~3hours, mean inference time ~0.42 seconds. So:
\n3*3600 seconds / 0.42 ~= 25714 samples
We have 5 hours limitation. With this logic approximate maximum Mean Only Inference Time for model ~ 0.7 seconds
", "rawMarkdown": "I also calculated the approximate time of one sample inference. \nSubmission scoring lasts ~3hours, mean inference time ~0.42 seconds. So:\n`3*3600 seconds / 0.42 ~= 25714` samples\n\nWe have 5 hours limitation. With this logic approximate maximum Mean Only Inference Time for model ~ 0.7 seconds", "votes": null }, { "id": "2276976", "postDate": "05/27/2023 10:57:17", "content": "Thank you for sharing this, but I am just a beginner and I am a little confused why do we need to Un squeeze or expand dimensions of tensor for this problem??
", "rawMarkdown": "Thank you for sharing this, but I am just a beginner and I am a little confused why do we need to Un squeeze or expand dimensions of tensor for this problem??", "votes": null }, { "id": "2277083", "postDate": "05/27/2023 12:39:32", "content": "@kolyaforrat It's bit off topic but lb0.734 is awesome! May I ask what's your validation score?
", "rawMarkdown": "kolyaforrat It's bit off topic but lb0.734 is awesome! May I ask what's your validation score?", "votes": null }, { "id": "2277165", "postDate": "05/27/2023 13:53:40", "content": "Actually you may need this for most models as batch dimension. But it depends of your architecture and approach. You may use time dimension as batch dimension for example
", "rawMarkdown": "Actually you may need this for most models as batch dimension. But it depends of your architecture and approach. You may use time dimension as batch dimension for example", "votes": null }, { "id": "2277169", "postDate": "05/27/2023 13:55:16", "content": "Valid score: 0.743 | LB score: 0.734
", "rawMarkdown": "Valid score: 0.743 | LB score: 0.734", "votes": null }, { "id": "2277198", "postDate": "05/27/2023 14:25:32", "content": "Wow, thanks for sharing!
\nI need to reproduce your solution at prev competition at first;)
Could you please share your TFLiteModel
\n\npython\nclass TFLiteModel(tf.Module):\n
After the competition deadline of course if you'll still need it :)
", "rawMarkdown": "After the competition deadline of course if you'll still need it :)", "votes": null }, { "id": "2277338", "postDate": "05/27/2023 16:18:24", "content": "yeah I was thinking to just use time dimension as batch dimension, thank you for clarifying
", "rawMarkdown": "yeah I was thinking to just use time dimension as batch dimension, thank you for clarifying", "votes": null }, { "id": "2306719", "postDate": "06/17/2023 14:02:22", "content": "Sorry to bother you.
\nMy model.tflite works wll in the Notebook with tensorflow 2.11.0. But it gets Submission Scoring Error soon after submit.
\nMy model.tflite is converted from a PyTorch model.
\nShould I use tensorflow 2.12 or 2.14? Is there a safe way to convert torch model to tflite?
\nIf you were me, what is your choice?
", "rawMarkdown": "Sorry to bother you.\n\nMy model.tflite works wll in the Notebook with tensorflow 2.11.0. But it gets Submission Scoring Error soon after submit.\n\nMy model.tflite is converted from a PyTorch model.\n\nShould I use tensorflow 2.12 or 2.14? Is there a safe way to convert torch model to tflite?\n\nIf you were me, what is your choice?", "votes": null }, { "id": "2307042", "postDate": "06/17/2023 19:00:46", "content": "Hi, I just keep using old environment with
\nWorks fine for me
", "rawMarkdown": "Hi, I just keep using old environment with \n- python 3.7\n- tensorflow 2.11.0\n- tflite-runtime==2.9.1\n\nWorks fine for me", "votes": null }, { "id": "2307205", "postDate": "06/18/2023 02:00:09", "content": "Thank you, The information you shared helped me a lot.
", "rawMarkdown": "Thank you, The information you shared helped me a lot.", "votes": null }, { "id": "2307211", "postDate": "06/18/2023 02:07:40", "content": "Well, I used the Kaggle notebook to test my model's inference speed, and my model with ~0.6s inference time exceeds the time limit ……
", "rawMarkdown": "Well, I used the Kaggle notebook to test my model's inference speed, and my model with ~0.6s inference time exceeds the time limit ......", "votes": null }, { "id": "2307955", "postDate": "06/18/2023 14:40:58", "content": "Maybe it depends of which subset of data do you use for speed test? And which tflite import you are usingimport tflite_runtime.interpreter as tflite or import tensorflow.lite as tflite
\nBelow I shared the way I tested it. For first 1000 samples from train.csv
Anyways, my calculations are approximate. And if you for example share your results for the same subset we can make some more accurate prediction of maximum inference time.
", "rawMarkdown": "Maybe it depends of which subset of data do you use for speed test? And which tflite import you are using` import tflite_runtime.interpreter as tflite` or `import tensorflow.lite as tflite`\n[Below](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/409722#2273788) I shared the way I tested it. For first 1000 samples from train.csv\n\nAnyways, my calculations are approximate. And if you for example share your results for the same subset we can make some more accurate prediction of maximum inference time.", "votes": null }, { "id": "2308304", "postDate": "06/18/2023 19:24:17", "content": "I'm using import tensorflow.lite as tflite as I fail to import tflite_runtime in notebook. And I tried your code
!pip install tflite-runtime==\n tflite_runtime.interpreter tflite\nbut still can't install tflite-runtime:
\nCould not find a version that satisfies the requirement tflite-runtime==2.9.1 (from versions: none)\nNo matching distribution found for tflite-runtime==2.9.1\n---------------------------------------------------------------------------\nModuleNotFoundError Traceback (most recent call last)\nCell In[11], line 2\n 1 get_ipython().system('pip install tflite-runtime==2.9.1')\n----> 2 import tflite_runtime.interpreter as tflite\nModuleNotFoundError: No module named 'tflite_runtime'\nI think the remaining code is basically the same, my code is like:
\nimport pandas as pd\nimport tensorflow as tf\nimport glob\nimport json\nimport numpy as np\nimport time\n\n\ndf = pd.read_csv()\n\n\ninterpreter = tf.lite.Interpreter()\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n\nwith open (, ) as f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\n REQUIRED_SIGNATURE found_signatures:\n raise KernelEvalException()\n\nwith open (, ) as f:\n load_cols = json.load(f)\nSEL_COLS = load_cols.()\n\nprediction_fn = interpreter.get_signature_runner()\ntot = time.time()\n i,(path,file_id,sequence_id,participant_id,phrase) df.iterrows():\n data = pd.read_parquet(f, =SEL_COLS)\n start_time = time.time()\n output = prediction_fn(=data[data.==sequence_id])\n prediction_str = .join([rev_character_map.(s, ) s np.argmax(output[REQUIRED_OUTPUT], =1)])\n i%==0:\n (, time.time()-start_time, )\n (, prediction_str)\n (,phrase)\n ()\n ==100:\n break\n(,(time.time()-tot)/i)\nand my result of first 100 samples are:
\n time (inference plus IO) .\n inference time .\nAh, so that's the reason.
\nFor me import tensorflow.lite as tflite :
And for import tflite_runtime.interpreter as tflite :
It has very different time. I wrote about import tflite_runtime.interpreter as tflite timings.
\nWith this approximation your time with import tflite_runtime.interpreter as tflite might be about 2 seconds
To install it I use previous python 3.7 environment.
\nPS. Mean time only infer: 0.4468766 has submission time ~3:40 , I think that maximum inference time not 0.7, maybe 0.6
", "rawMarkdown": "Ah, so that's the reason. \nFor me `import tensorflow.lite as tflite` :\n- Mean time: 0.5808348\n- Mean time only infer: 0.1342919\n\nAnd for `import tflite_runtime.interpreter as tflite` :\n- Mean time: 0.8799638\n- Mean time only infer: 0.4468766\n\nIt has very different time. I wrote about `import tflite_runtime.interpreter as tflite` timings.\nWith this approximation your time with `import tflite_runtime.interpreter as tflite` might be about 2 seconds\n\nTo install it I use previous python 3.7 environment.\n\nPS. Mean time only infer: 0.4468766 has submission time ~3:40 , I think that maximum inference time not 0.7, maybe 0.6", "votes": null }, { "id": "2308370", "postDate": "06/18/2023 21:24:36", "content": "Yes, you're right, there is a huge difference. import tflite_runtime.interpreter as tflite changed my inference time from ~0.55s to ~0.9s. Thank you for your information, really appreciate it!
I see public notebook run 5-6 hours, where is 5 hour limit? Notebook still has 9 hour limit submissiong? A bit confused here.
", "rawMarkdown": "I see public notebook run 5-6 hours, where is 5 hour limit? Notebook still has 9 hour limit submissiong? A bit confused here.", "votes": null }, { "id": "2324390", "postDate": "06/30/2023 15:10:52", "content": "Do you mean kernel runtime or scoring time?
\n5 hours limit only for scoring time
ah hello I have a noob question :( . Your submission takes 3 hours to run but we are currently using only 50% of the test files on public board so if we run it full it will take around 6 hours(private + public test). But the contest is limited to 5 hours. Will this affect the final result.
", "rawMarkdown": "ah hello I have a noob question :( . Your submission takes 3 hours to run but we are currently using only 50% of the test files on public board so if we run it full it will take around 6 hours(private + public test). But the contest is limited to 5 hours. Will this affect the final result.", "votes": null }, { "id": "2344595", "postDate": "07/14/2023 15:21:02", "content": "Hi.
\nSubmissions scored on the whole data. But you only see the metric of 50% on LB. So don't worry, it won't affect the final result
ah I got it. Thanks for your answer
", "rawMarkdown": "ah I got it. Thanks for your answer", "votes": null }, { "id": "2362828", "postDate": "07/28/2023 10:24:03", "content": "Sorry to bother you.
\nhow do you keep using python 3.7 ig now i can only pin it to the latest version ?
\nany workarounds without 3.7?? or to get 3.7?
I just copied some public notebook with this environment. Don't actually remember which one. You could try to find it in early stage discussions
\nI'm also like you don't know how to change environment to any other except last :)
", "rawMarkdown": "I just copied some public notebook with this environment. Don't actually remember which one. You could try to find it in early stage discussions\n\nI'm also like you don't know how to change environment to any other except last :)", "votes": null }, { "id": "2362932", "postDate": "07/28/2023 11:40:52", "content": "oooh thanks !
", "rawMarkdown": "oooh thanks !", "votes": null } ], "comments": [ { "id": 2272974, "author_name": "kolyaforrat", "author_url": "", "post_date": "05/24/2023 22:17:09", "content": "After submissions became available I think this topic is actual again.
\nI have few submits with ~40-42 minutes with Submission Scoring Error. But:
\n\nYour model must also perform inference in less than 5 hours
\n
Or maybe there are some kind of multiprocessing and the time cap is 45 minutes?
\nDoes anyone have successful submissions longer than 50 minutes? Any ideas about mean per sample inference time?
\nupd. Error in my submission not because of 'out of time' . I deleted most of layers from my model and it has Error at ~8 minutes.
\nSo I don't understand the problem. My input is (time_dim, len(selected_columns)) and output with (new_time_dim, 59) and still getting errors.
\nModel under 40 MB, capable with tflite, same environment with successful public kernels. Maybe here some another limitations I didn't know? At Kaggle kernels it works fine with Evaluation example code 😨
the only think I was capable of submitting were dummy submissions not processing the frames.
", "votes": null, "replies": [] }, { "id": 2273492, "author_name": "something4kag", "author_url": "", "post_date": "05/25/2023 07:33:02", "content": "if you click on the triangle with error - do you get any more information under Submission Scoring Error in the Submission Details popup? e.g. \"Your notebook generated a submission file with incorrect format.\" or other message? Know usually very little is give when submissions fail.
\nI can confirm using inference_args.json for selected columns works ok, but not if you only were using frame column. Tried that for some simple tests. Had issues with preprocessing layer though so far.
\nIf for some reason your output is empty, you can have error for not an iterator here -
\nprediction_str = \"\".join([rev_character_map.get(s, \"\") for s in np.argmax(output[REQUIRED_OUTPUT], axis=1)])
\nSo perhaps a default output can be returned like a space for exceptions.
Think there are still open questions about what is provided in the video via
\ndef load_relevant_data_subset(pq_path):
\nis it a parquet file of many sequence ids like in train? or one parquet file for each sequence id to predict like in GISLR?
\nIf the former, then is the expectation to split the output strings per sequence id? In train there is one parquet that has 287 sequence ids ( /kaggle/input/asl-fingerspelling/train_landmarks/450474571.parquet ), the rest have 1000 but perhaps in test all have 1000.
@georgemegre seems to have been able to do more than dummy submissions. Without sharing too many details maybe has some thoughts on tips for true submissions under 5hrs? Thanks in advance.
", "votes": null, "replies": [ { "id": 2273547, "author_name": "kolyaforrat", "author_url": "", "post_date": "05/25/2023 08:14:38", "content": "Error message has nothing helpful, this message was prepared for submitting csv files and just told about rows and columns.
\n\n\nIf for some reason your output is empty, you can have error
\n
I tested empty input with time_dim=0 and with full nan input. There are no way that my model has zero or empty output. Only if selected columns are wrong, but this scoring for about 40 minutes I think everything is ok with this part.
\nAlso my model capable for any length of input. For single frame or multiple frames from one csv. Output phrases would be just as one big string if so.
", "votes": null, "replies": [ { "id": 2274136, "author_name": "georgemegre", "author_url": "", "post_date": "05/25/2023 16:38:53", "content": "hm, to be honest i don't understand how you calculate it, but i can explain my understanding by two examples (hope this will be helpful):
\n1)
\ny_true = my bare face in the wind
\ny_pred = my bare face sa the wind
\nN = 24
\nLEVD = 2
\nMetric = 22/24 = ~0.91
But if we have:
\ny_true = my bare face in the wind
\ny_pred = \"sssssssssssssssssssssssss wdaedaj\"
\nN = 24
\nLEVD = 30
\nMetric = 24-30 / 24 = -0.25
So, it should be in the range (-d_min, 1).
", "votes": null, "replies": [] } ] }, { "id": 2274122, "author_name": "georgemegre", "author_url": "", "post_date": "05/25/2023 16:27:15", "content": "My submission run about 1 hour, the pipeline is very similar to previous competition, except that output is (time, 59). It also can be a problem in tflite version, it should be 2.9.1. i've checked my sub with this code (hope this will helpful):
\npython\n ():\n df = pd.read_parquet(pq_path, columns=selected_columns)\n df = df.loc[]\n df.values\n\n (json_fn) f:\n cols = json.load(f)[]\n\nframes = load_relevant_data_subset(, cols)\n\ninterpreter = tf.lite.Interpreter(sub_model_path)\n\nprediction_fn = interpreter.get_signature_runner()\nt = time()\noutput = prediction_fn(inputs=frames)\n()\n(*(time() - t) )\nt = time()\noutput = prediction_fn(inputs=frames)\n()\n(*(time() - t) )\n\nout_2 = output[]\n(out_2.shape)\n(np.argmax(out_2, axis=))\n\n (, ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(out_2, axis=)])\nprediction_str\nThanks for info. Could you please share frames shape and out_2 shape? They are same by time axis or different?
Sure, here it is:
\nframes shape: (236, 252)
\nout_2 shape: (19, 59)
Thanks again. So now I'm totally don't understand why I can't submit my model)
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2273788, "author_name": "kolyaforrat", "author_url": "", "post_date": "05/25/2023 12:06:40", "content": "\n
\n
\n
WIth fixed preprocessing code it works
\n!pip install tflite-runtime==\n tflite_runtime.interpreter tflite\n\n time\n json\n pandas pd\n tqdm.auto tqdm\n Levenshtein Lev\n tensorflow tf\n\nconverter = tf.lite.TFLiteConverter.from_keras_model(infr) \ntflite_model = converter.convert()\n (, ) f:\n f.write(tflite_model)\n! submission. \n\n\nSEL_FEATURES = json.load(())[]\n\n ():\n pd.read_parquet(pq_path, columns=SEL_FEATURES) \n\n (, ) f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\n\n\ndf = pd.read_csv()\n\nidx = \nsample = df.loc[idx]\nloaded = load_relevant_data_subset( + sample[])\nloaded = loaded[loaded.index==sample[]].values\n(loaded.shape)\nframes = loaded\n\n ():\n w1 = (s1.split())\n lvd = Lev.distance(s1, s2)\n lvd / w1\n\n tflite_runtime.interpreter tflite\ninterpreter = tflite.Interpreter()\nfound_signatures = (interpreter.get_signature_list().keys())\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n REQUIRED_SIGNATURE found_signatures:\n KernelEvalException()\n\nprediction_fn = interpreter.get_signature_runner()\noutput_lite = prediction_fn(inputs=frames)\nprediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_lite[REQUIRED_OUTPUT], axis=)])\n(prediction_str)\n\n\nst = time.time()\ncnt = \ntotal = \nmodel_time = \n\nlevs = []\n\n i tqdm(((df.iloc[:total]))):\n sample = df.loc[i]\n loaded = load_relevant_data_subset( + sample[])\n loaded = loaded[loaded.index==sample[]].values\n\n md_st = time.time()\n output_ = prediction_fn(inputs=loaded)\n model_time += time.time() - md_st\n\n\n prediction_str = .join([rev_character_map.get(s, ) s np.argmax(output_[REQUIRED_OUTPUT], axis=)])\n cur_lev = wer__(sample[], prediction_str) \n \n \n\n levs.append(cur_lev)\n\n()\n()\n()\nI finally found the problem with submission.
\nIn previous competition I filled nans with this code. And also I unsqueezed tensor like this. (I changed both lines, didn't try change it one by one)
\nx = tf.where(tfnp.isnan(x), , x)\nx = tf.expand_dims(x, axis=)\nIt worked fine in previous competition. And it work fine here in local kernel. But has error while submission scoring
\nI just changed this lines to:
\nx = tf.where(tfnp.isnan(x), tf.zeros_like(x), x)\nx = x[]\nMaybe it will help someone. Not so obvious reasons, as for me
", "votes": null, "replies": [ { "id": 2276976, "author_name": "aadityabansalcodes", "author_url": "", "post_date": "05/27/2023 10:57:17", "content": "Thank you for sharing this, but I am just a beginner and I am a little confused why do we need to Un squeeze or expand dimensions of tensor for this problem??
", "votes": null, "replies": [ { "id": 2277165, "author_name": "kolyaforrat", "author_url": "", "post_date": "05/27/2023 13:53:40", "content": "Actually you may need this for most models as batch dimension. But it depends of your architecture and approach. You may use time dimension as batch dimension for example
", "votes": null, "replies": [ { "id": 2277204, "author_name": "glitchr", "author_url": "", "post_date": "05/27/2023 14:31:02", "content": "Could you please share your TFLiteModel
\n\npython\nclass TFLiteModel(tf.Module):\n
After the competition deadline of course if you'll still need it :)
", "votes": null, "replies": [] } ] }, { "id": 2277338, "author_name": "aadityabansalcodes", "author_url": "", "post_date": "05/27/2023 16:18:24", "content": "yeah I was thinking to just use time dimension as batch dimension, thank you for clarifying
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2276907, "author_name": "kolyaforrat", "author_url": "", "post_date": "05/27/2023 09:56:56", "content": "I also calculated the approximate time of one sample inference.
\nSubmission scoring lasts ~3hours, mean inference time ~0.42 seconds. So:
\n3*3600 seconds / 0.42 ~= 25714 samples
We have 5 hours limitation. With this logic approximate maximum Mean Only Inference Time for model ~ 0.7 seconds
", "votes": null, "replies": [ { "id": 2307211, "author_name": "nightsh4de", "author_url": "", "post_date": "06/18/2023 02:07:40", "content": "Well, I used the Kaggle notebook to test my model's inference speed, and my model with ~0.6s inference time exceeds the time limit ……
", "votes": null, "replies": [ { "id": 2307955, "author_name": "kolyaforrat", "author_url": "", "post_date": "06/18/2023 14:40:58", "content": "Maybe it depends of which subset of data do you use for speed test? And which tflite import you are usingimport tflite_runtime.interpreter as tflite or import tensorflow.lite as tflite
\nBelow I shared the way I tested it. For first 1000 samples from train.csv
Anyways, my calculations are approximate. And if you for example share your results for the same subset we can make some more accurate prediction of maximum inference time.
", "votes": null, "replies": [ { "id": 2308304, "author_name": "nightsh4de", "author_url": "", "post_date": "06/18/2023 19:24:17", "content": "I'm using import tensorflow.lite as tflite as I fail to import tflite_runtime in notebook. And I tried your code
!pip install tflite-runtime==\n tflite_runtime.interpreter tflite\nbut still can't install tflite-runtime:
\nCould not find a version that satisfies the requirement tflite-runtime==2.9.1 (from versions: none)\nNo matching distribution found for tflite-runtime==2.9.1\n---------------------------------------------------------------------------\nModuleNotFoundError Traceback (most recent call last)\nCell In[11], line 2\n 1 get_ipython().system('pip install tflite-runtime==2.9.1')\n----> 2 import tflite_runtime.interpreter as tflite\nModuleNotFoundError: No module named 'tflite_runtime'\nI think the remaining code is basically the same, my code is like:
\nimport pandas as pd\nimport tensorflow as tf\nimport glob\nimport json\nimport numpy as np\nimport time\n\n\ndf = pd.read_csv()\n\n\ninterpreter = tf.lite.Interpreter()\n\nREQUIRED_SIGNATURE = \nREQUIRED_OUTPUT = \n\nwith open (, ) as f:\n character_map = json.load(f)\nrev_character_map = {j:i i,j character_map.items()}\n\nfound_signatures = list(interpreter.get_signature_list().keys())\n\n REQUIRED_SIGNATURE found_signatures:\n raise KernelEvalException()\n\nwith open (, ) as f:\n load_cols = json.load(f)\nSEL_COLS = load_cols.()\n\nprediction_fn = interpreter.get_signature_runner()\ntot = time.time()\n i,(path,file_id,sequence_id,participant_id,phrase) df.iterrows():\n data = pd.read_parquet(f, =SEL_COLS)\n start_time = time.time()\n output = prediction_fn(=data[data.==sequence_id])\n prediction_str = .join([rev_character_map.(s, ) s np.argmax(output[REQUIRED_OUTPUT], =1)])\n i%==0:\n (, time.time()-start_time, )\n (, prediction_str)\n (,phrase)\n ()\n ==100:\n break\n(,(time.time()-tot)/i)\nand my result of first 100 samples are:
\n time (inference plus IO) .\n inference time .\nAh, so that's the reason.
\nFor me import tensorflow.lite as tflite :
And for import tflite_runtime.interpreter as tflite :
It has very different time. I wrote about import tflite_runtime.interpreter as tflite timings.
\nWith this approximation your time with import tflite_runtime.interpreter as tflite might be about 2 seconds
To install it I use previous python 3.7 environment.
\nPS. Mean time only infer: 0.4468766 has submission time ~3:40 , I think that maximum inference time not 0.7, maybe 0.6
", "votes": null, "replies": [ { "id": 2308370, "author_name": "nightsh4de", "author_url": "", "post_date": "06/18/2023 21:24:36", "content": "Yes, you're right, there is a huge difference. import tflite_runtime.interpreter as tflite changed my inference time from ~0.55s to ~0.9s. Thank you for your information, really appreciate it!
ah hello I have a noob question :( . Your submission takes 3 hours to run but we are currently using only 50% of the test files on public board so if we run it full it will take around 6 hours(private + public test). But the contest is limited to 5 hours. Will this affect the final result.
", "votes": null, "replies": [ { "id": 2344595, "author_name": "kolyaforrat", "author_url": "", "post_date": "07/14/2023 15:21:02", "content": "Hi.
\nSubmissions scored on the whole data. But you only see the metric of 50% on LB. So don't worry, it won't affect the final result
ah I got it. Thanks for your answer
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2277083, "author_name": "bamps53", "author_url": "", "post_date": "05/27/2023 12:39:32", "content": "@kolyaforrat It's bit off topic but lb0.734 is awesome! May I ask what's your validation score?
", "votes": null, "replies": [ { "id": 2277169, "author_name": "kolyaforrat", "author_url": "", "post_date": "05/27/2023 13:55:16", "content": "Valid score: 0.743 | LB score: 0.734
", "votes": null, "replies": [ { "id": 2277198, "author_name": "bamps53", "author_url": "", "post_date": "05/27/2023 14:25:32", "content": "Wow, thanks for sharing!
\nI need to reproduce your solution at prev competition at first;)
Sorry to bother you.
\nMy model.tflite works wll in the Notebook with tensorflow 2.11.0. But it gets Submission Scoring Error soon after submit.
\nMy model.tflite is converted from a PyTorch model.
\nShould I use tensorflow 2.12 or 2.14? Is there a safe way to convert torch model to tflite?
\nIf you were me, what is your choice?
", "votes": null, "replies": [ { "id": 2307042, "author_name": "kolyaforrat", "author_url": "", "post_date": "06/17/2023 19:00:46", "content": "Hi, I just keep using old environment with
\nWorks fine for me
", "votes": null, "replies": [ { "id": 2307205, "author_name": "jiangjixiang", "author_url": "", "post_date": "06/18/2023 02:00:09", "content": "Thank you, The information you shared helped me a lot.
", "votes": null, "replies": [] }, { "id": 2362828, "author_name": "jainam213", "author_url": "", "post_date": "07/28/2023 10:24:03", "content": "Sorry to bother you.
\nhow do you keep using python 3.7 ig now i can only pin it to the latest version ?
\nany workarounds without 3.7?? or to get 3.7?
I just copied some public notebook with this environment. Don't actually remember which one. You could try to find it in early stage discussions
\nI'm also like you don't know how to change environment to any other except last :)
", "votes": null, "replies": [ { "id": 2362932, "author_name": "jainam213", "author_url": "", "post_date": "07/28/2023 11:40:52", "content": "oooh thanks !
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2324330, "author_name": "goldenlock", "author_url": "", "post_date": "06/30/2023 14:34:15", "content": "I see public notebook run 5-6 hours, where is 5 hour limit? Notebook still has 9 hour limit submissiong? A bit confused here.
", "votes": null, "replies": [ { "id": 2324390, "author_name": "kolyaforrat", "author_url": "", "post_date": "06/30/2023 15:10:52", "content": "Do you mean kernel runtime or scoring time?
\n5 hours limit only for scoring time