{ "id": 398531, "title": "Submission fail", "url": "/competitions/asl-signs/discussion/398531", "author_name": "", "post_date": "2023-03-30T14:03:15.182131300Z", "votes": 1, "comment_count": 5, "views": 0, "content": "

Hello Kagglers!

\n

I am facing a 'Submission scoring error' when i try to submit my model.
\nThere is my notebook.
\nThe latest version (24 and above) are submission tests, i try several different think to get submission to work, but nothing work.

\n

Does someone know what is wrong ?

", "messages": [ { "id": "2203093", "postDate": "03/30/2023 14:03:15", "content": "

Hello Kagglers!

\n

I am facing a 'Submission scoring error' when i try to submit my model.
\nThere is my notebook.
\nThe latest version (24 and above) are submission tests, i try several different think to get submission to work, but nothing work.

\n

Does someone know what is wrong ?

", "rawMarkdown": "Hello Kagglers! \n\nI am facing a 'Submission scoring error' when i try to submit my model. \nThere is my [notebook](https://www.kaggle.com/code/dorianmb/islr-convlstm).\nThe latest version (24 and above) are submission tests, i try several different think to get submission to work, but nothing work.\n\nDoes someone know what is wrong ?", "votes": null }, { "id": "2203635", "postDate": "03/31/2023 02:14:09", "content": "

Hi @dorianmb, there are several reasons that could lead to a scoring error, here are the most common ones:

\nMain Requirements

\n\n

Potential Technical Issues

\n\n

Your best bet would to debug your code using the following code and see if everything works fine (it is the code used by the competition hosts to test your model)

\n
 tflite_runtime.interpreter  tflite\n\n\ninterpreter = tflite.Interpreter()\nfound_signatures = (interpreter.get_signature_list().keys())\nprediction_fn = interpreter.get_signature_runner()\n\n\noutput = prediction_fn(inputs=load_relevant_data_subset(CFG.data_dir + train_data.path[]))\nsign = np.argmax(output[])\n\n(, decoder(sign))\n(, train_data.sign[])\n
\n

Hope that helps!

", "rawMarkdown": "Hi @dorianmb, there are several reasons that could lead to a scoring error, here are the most common ones:
\n**Main Requirements**\n- Model must also perform inference with less than 100 milliseconds of latency per video on average \n- Model must use less than 40 MB of storage space\n- Model should be compatible with the TensorFlow Lite Runtime v2.9.1\n\n**Potential Technical Issues**\n- Use of preprocessing layer that is not compatible with tensorflow operators (also known as custom operators)\n- Model does not take the competition data as inputs\n- Error in preprocessing steps\n\nYour best bet would to debug your code using the following code and see if everything works fine (it is the code used by the competition hosts to test your model)\n\n```python\nimport tflite_runtime.interpreter as tflite\n\n# load you tflite model\ninterpreter = tflite.Interpreter(\"/kaggle/input/asl-sign-model/asl_model/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\n# predict class of the relevant video\noutput = prediction_fn(inputs=load_relevant_data_subset(CFG.data_dir + train_data.path[0]))\nsign = np.argmax(output[\"outputs\"])\n\nprint(\"PREDICTION : \", decoder(sign))\nprint(\"ACTUAL : \", train_data.sign[0])\n```\n\nHope that helps!", "votes": null }, { "id": "2205436", "postDate": "04/01/2023 14:48:50", "content": "

I @josephzahar thank for helping me !

\n

I just upload a new version (V31) of my notebook here

\n\n

i also try with a different classification layers, base on LSTM layer instead of ConvLSTM1D layer, but the submission still not working.

\n

Do you or anyone else know what is wrong ?

", "rawMarkdown": "I @josephzahar thank for helping me !\n\nI just upload a new version (V31) of my notebook [here](https://www.kaggle.com/code/dorianmb/islr-convlstm?scriptVersionId=124202197)\n\n- I check the performance < 100ms\n- I check storage space < 40MB\n- I'm not sure if my model is compatible with the TensorFlow Lite Runtime v2.9.1 .
But the code you gave me works fine\n\ni also try with a different classification layers, base on `LSTM` layer instead of `ConvLSTM1D` layer, but the submission still not working.\n\nDo you or anyone else know what is wrong ?", "votes": null }, { "id": "2205592", "postDate": "04/01/2023 17:54:48", "content": "

Hi Dorian,

\n

I'm not sure if it has been discussed elsewhere, but I have also gotten 'Submission scoring error' when using:

\n
converter.target_spec.supported_ops = [\n    tf.lite.OpsSet.TFLITE_BUILTINS,\n    tf.lite.OpsSet.SELECT_TF_OPS,\n]\n
\n

Does your model convert to tflite without this configuration?

\n

Regards,
\nPeter

", "rawMarkdown": "Hi Dorian, \n\nI'm not sure if it has been discussed elsewhere, but I have also gotten 'Submission scoring error' when using: \n\n```\nconverter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS,\n tf.lite.OpsSet.SELECT_TF_OPS,\n]\n```\n\nDoes your model convert to tflite without this configuration?\n\nRegards,\nPeter", "votes": null }, { "id": "2205702", "postDate": "04/01/2023 19:52:57", "content": "

Hi @dorianmb,
\n@koehlepe is right, you should try converting it without the additional supported ops. I would also try one more thing, your TFLite model is a subclass of tf.keras.Model, try tf.Module instead:

\n
 (tf.Module):\n     ():\n        ().__init__()\n        self.prep_inputs = FeatureGen()\n        self.model = model\n\n\n     ():\n        x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n        x = tf.expand_dims(x, axis=)\n        outputs = self.model(x)[, :]\n        \n         {: outputs}\n
\n

Let me know if it still doesn't work so we can investigate it further.

", "rawMarkdown": "Hi @dorianmb, \n@koehlepe is right, you should try converting it without the additional supported ops. I would also try one more thing, your TFLite model is a subclass of `tf.keras.Model`, try `tf.Module` instead:\n\n```python\nclass TFLiteModel(tf.Module):\n def __init__(self, model):\n super().__init__()\n self.prep_inputs = FeatureGen()\n self.model = model\n \n @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n def call(self, inputs):\n x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n x = tf.expand_dims(x, axis=0)\n outputs = self.model(x)[0, :]\n # Return a dictionary with the output tensor\n return {'outputs': outputs}\n```\n\nLet me know if it still doesn't work so we can investigate it further.", "votes": null }, { "id": "2206728", "postDate": "04/02/2023 19:32:59", "content": "

Hi @josephzahar, @koehlepe

\n

I have also gotten 'Submission scoring error' when removing :

\n
converter.target_spec.supported_ops = [\n    tf.lite.OpsSet.TFLITE_BUILTINS,\n    tf.lite.OpsSet.SELECT_TF_OPS,\n]\n
\n

But i use a classification layers, base on LSTM layer instead of ConvLSTM1D layer.
\n(Without this line and with ConvLSTM1D layer, model cant be convert to TFLite model)
\n

\n

Same 'Submission scoring error' happen with tf.Module insteed of tf.keras.Model.
\n

\n

But in this two cases the submission ran for i think 2h, maybe less.
\nHowever I'm not sur if it a good sign.

", "rawMarkdown": "Hi @josephzahar, @koehlepe \n\nI have also gotten 'Submission scoring error' when removing : \n```\nconverter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS,\n tf.lite.OpsSet.SELECT_TF_OPS,\n]\n```\nBut i use a classification layers, base on `LSTM` layer instead of `ConvLSTM1D` layer.\n(Without this line and with `ConvLSTM1D` layer, model cant be convert to TFLite model) \n
\n\nSame 'Submission scoring error' happen with `tf.Module` insteed of `tf.keras.Model`.\n
\n\nBut in this two cases the submission ran for i think 2h, maybe less.\nHowever I'm not sur if it a good sign.", "votes": null } ], "comments": [ { "id": 2203635, "author_name": "josephzahar", "author_url": "", "post_date": "03/31/2023 02:14:09", "content": "

Hi @dorianmb, there are several reasons that could lead to a scoring error, here are the most common ones:

\nMain Requirements

\n\n

Potential Technical Issues

\n\n

Your best bet would to debug your code using the following code and see if everything works fine (it is the code used by the competition hosts to test your model)

\n
 tflite_runtime.interpreter  tflite\n\n\ninterpreter = tflite.Interpreter()\nfound_signatures = (interpreter.get_signature_list().keys())\nprediction_fn = interpreter.get_signature_runner()\n\n\noutput = prediction_fn(inputs=load_relevant_data_subset(CFG.data_dir + train_data.path[]))\nsign = np.argmax(output[])\n\n(, decoder(sign))\n(, train_data.sign[])\n
\n

Hope that helps!

", "votes": null, "replies": [ { "id": 2205436, "author_name": "dorianmb", "author_url": "", "post_date": "04/01/2023 14:48:50", "content": "

I @josephzahar thank for helping me !

\n

I just upload a new version (V31) of my notebook here

\n\n

i also try with a different classification layers, base on LSTM layer instead of ConvLSTM1D layer, but the submission still not working.

\n

Do you or anyone else know what is wrong ?

", "votes": null, "replies": [ { "id": 2205702, "author_name": "josephzahar", "author_url": "", "post_date": "04/01/2023 19:52:57", "content": "

Hi @dorianmb,
\n@koehlepe is right, you should try converting it without the additional supported ops. I would also try one more thing, your TFLite model is a subclass of tf.keras.Model, try tf.Module instead:

\n
 (tf.Module):\n     ():\n        ().__init__()\n        self.prep_inputs = FeatureGen()\n        self.model = model\n\n\n     ():\n        x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n        x = tf.expand_dims(x, axis=)\n        outputs = self.model(x)[, :]\n        \n         {: outputs}\n
\n

Let me know if it still doesn't work so we can investigate it further.

", "votes": null, "replies": [ { "id": 2206728, "author_name": "dorianmb", "author_url": "", "post_date": "04/02/2023 19:32:59", "content": "

Hi @josephzahar, @koehlepe

\n

I have also gotten 'Submission scoring error' when removing :

\n
converter.target_spec.supported_ops = [\n    tf.lite.OpsSet.TFLITE_BUILTINS,\n    tf.lite.OpsSet.SELECT_TF_OPS,\n]\n
\n

But i use a classification layers, base on LSTM layer instead of ConvLSTM1D layer.
\n(Without this line and with ConvLSTM1D layer, model cant be convert to TFLite model)
\n

\n

Same 'Submission scoring error' happen with tf.Module insteed of tf.keras.Model.
\n

\n

But in this two cases the submission ran for i think 2h, maybe less.
\nHowever I'm not sur if it a good sign.

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2205592, "author_name": "koehlepe", "author_url": "", "post_date": "04/01/2023 17:54:48", "content": "

Hi Dorian,

\n

I'm not sure if it has been discussed elsewhere, but I have also gotten 'Submission scoring error' when using:

\n
converter.target_spec.supported_ops = [\n    tf.lite.OpsSet.TFLITE_BUILTINS,\n    tf.lite.OpsSet.SELECT_TF_OPS,\n]\n
\n

Does your model convert to tflite without this configuration?

\n

Regards,
\nPeter

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2203093": "Hello Kagglers! \n\nI am facing a 'Submission scoring error' when i try to submit my model. \nThere is my [notebook](https://www.kaggle.com/code/dorianmb/islr-convlstm).\nThe latest version (24 and above) are submission tests, i try several different think to get submission to work, but nothing work.\n\nDoes someone know what is wrong ?", "2203635": "Hi @dorianmb, there are several reasons that could lead to a scoring error, here are the most common ones:
\n**Main Requirements**\n- Model must also perform inference with less than 100 milliseconds of latency per video on average \n- Model must use less than 40 MB of storage space\n- Model should be compatible with the TensorFlow Lite Runtime v2.9.1\n\n**Potential Technical Issues**\n- Use of preprocessing layer that is not compatible with tensorflow operators (also known as custom operators)\n- Model does not take the competition data as inputs\n- Error in preprocessing steps\n\nYour best bet would to debug your code using the following code and see if everything works fine (it is the code used by the competition hosts to test your model)\n\n```python\nimport tflite_runtime.interpreter as tflite\n\n# load you tflite model\ninterpreter = tflite.Interpreter(\"/kaggle/input/asl-sign-model/asl_model/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\n# predict class of the relevant video\noutput = prediction_fn(inputs=load_relevant_data_subset(CFG.data_dir + train_data.path[0]))\nsign = np.argmax(output[\"outputs\"])\n\nprint(\"PREDICTION : \", decoder(sign))\nprint(\"ACTUAL : \", train_data.sign[0])\n```\n\nHope that helps!", "2205436": "I @josephzahar thank for helping me !\n\nI just upload a new version (V31) of my notebook [here](https://www.kaggle.com/code/dorianmb/islr-convlstm?scriptVersionId=124202197)\n\n- I check the performance < 100ms\n- I check storage space < 40MB\n- I'm not sure if my model is compatible with the TensorFlow Lite Runtime v2.9.1 .
But the code you gave me works fine\n\ni also try with a different classification layers, base on `LSTM` layer instead of `ConvLSTM1D` layer, but the submission still not working.\n\nDo you or anyone else know what is wrong ?", "2205592": "Hi Dorian, \n\nI'm not sure if it has been discussed elsewhere, but I have also gotten 'Submission scoring error' when using: \n\n```\nconverter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS,\n tf.lite.OpsSet.SELECT_TF_OPS,\n]\n```\n\nDoes your model convert to tflite without this configuration?\n\nRegards,\nPeter", "2205702": "Hi @dorianmb, \n@koehlepe is right, you should try converting it without the additional supported ops. I would also try one more thing, your TFLite model is a subclass of `tf.keras.Model`, try `tf.Module` instead:\n\n```python\nclass TFLiteModel(tf.Module):\n def __init__(self, model):\n super().__init__()\n self.prep_inputs = FeatureGen()\n self.model = model\n \n @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n def call(self, inputs):\n x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n x = tf.expand_dims(x, axis=0)\n outputs = self.model(x)[0, :]\n # Return a dictionary with the output tensor\n return {'outputs': outputs}\n```\n\nLet me know if it still doesn't work so we can investigate it further.", "2206728": "Hi @josephzahar, @koehlepe \n\nI have also gotten 'Submission scoring error' when removing : \n```\nconverter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS,\n tf.lite.OpsSet.SELECT_TF_OPS,\n]\n```\nBut i use a classification layers, base on `LSTM` layer instead of `ConvLSTM1D` layer.\n(Without this line and with `ConvLSTM1D` layer, model cant be convert to TFLite model) \n
\n\nSame 'Submission scoring error' happen with `tf.Module` insteed of `tf.keras.Model`.\n
\n\nBut in this two cases the submission ran for i think 2h, maybe less.\nHowever I'm not sur if it a good sign." }, "source": "meta" }