{
  "id": 402534,
  "title": "Submission Scoring Error: Your notebook generated a submission file with incorrect format.",
  "url": "/competitions/asl-signs/discussion/402534",
  "author_name": "",
  "post_date": "2023-04-18T19:18:08.889580400Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>My notebook generates a submission.zip file from the tflite model.<br>\nWhen I try the model in the notebook it works. But in the submission comes the error from the title. <br>\nI tried to output the input and output layer and it seems to be ok for me. Do any of you notice anything? Or what other possibilities do I have to debug the submission or to get more info about the error?</p>\n<pre><code>keras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflite_keras_model)\ntflite_model = keras_model_converter.convert()\nwith open('/kaggle/working/model.tflite', 'wb') as f:\n    f.write(tflite_model)\n\n!zip submission.zip /kaggle/working/model.tflite\n</code></pre>\n<p>Output: updating: kaggle/working/model.tflite (deflated 8%)</p>\n<pre><code># Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=demo_raw_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n</code></pre>\n<p>works without error and found_signatures = ['serving_default'] </p>\n<p><strong>Test input/output Tensor:</strong></p>\n<pre><code># Load the TFLite model\ntflite_model_file = '/kaggle/working/model.tflite'\ninterpreter = tflite.Interpreter(model_path=tflite_model_file)\n\n# Allocate tensors\n#interpreter.allocate_tensors()\n\n# Get input and output tensors\ninput_details = interpreter.get_input_details()\noutput_details = interpreter.get_output_details()\nprint(input_details)\nprint(\"----output----\")\nprint(output_details)\n</code></pre>\n<p>This gives me:</p>\n<pre><code>[{'name': 'inputs', 'index': 0, 'shape': array([  1, 543,   3], dtype=int32), 'shape_signature': array([ -1, 543,   3], dtype=int32), 'dtype': &lt;class 'numpy.float32'&gt;, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n\n----output----\n\n[{'name': 'Identity', 'index': 77, 'shape': array([250], dtype=int32), 'shape_signature': array([250], dtype=int32), 'dtype': &lt;class 'numpy.float32'&gt;, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n</code></pre>",
  "messages": [
    {
      "id": "2226250",
      "postDate": "04/18/2023 19:18:08",
      "content": "<p>My notebook generates a submission.zip file from the tflite model.<br>\nWhen I try the model in the notebook it works. But in the submission comes the error from the title. <br>\nI tried to output the input and output layer and it seems to be ok for me. Do any of you notice anything? Or what other possibilities do I have to debug the submission or to get more info about the error?</p>\n<pre><code>keras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflite_keras_model)\ntflite_model = keras_model_converter.convert()\nwith open('/kaggle/working/model.tflite', 'wb') as f:\n    f.write(tflite_model)\n\n!zip submission.zip /kaggle/working/model.tflite\n</code></pre>\n<p>Output: updating: kaggle/working/model.tflite (deflated 8%)</p>\n<pre><code># Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=demo_raw_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n</code></pre>\n<p>works without error and found_signatures = ['serving_default'] </p>\n<p><strong>Test input/output Tensor:</strong></p>\n<pre><code># Load the TFLite model\ntflite_model_file = '/kaggle/working/model.tflite'\ninterpreter = tflite.Interpreter(model_path=tflite_model_file)\n\n# Allocate tensors\n#interpreter.allocate_tensors()\n\n# Get input and output tensors\ninput_details = interpreter.get_input_details()\noutput_details = interpreter.get_output_details()\nprint(input_details)\nprint(\"----output----\")\nprint(output_details)\n</code></pre>\n<p>This gives me:</p>\n<pre><code>[{'name': 'inputs', 'index': 0, 'shape': array([  1, 543,   3], dtype=int32), 'shape_signature': array([ -1, 543,   3], dtype=int32), 'dtype': &lt;class 'numpy.float32'&gt;, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n\n----output----\n\n[{'name': 'Identity', 'index': 77, 'shape': array([250], dtype=int32), 'shape_signature': array([250], dtype=int32), 'dtype': &lt;class 'numpy.float32'&gt;, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n</code></pre>",
      "rawMarkdown": "My notebook generates a submission.zip file from the tflite model.\nWhen I try the model in the notebook it works. But in the submission comes the error from the title. \nI tried to output the input and output layer and it seems to be ok for me. Do any of you notice anything? Or what other possibilities do I have to debug the submission or to get more info about the error?\n\n\n```\nkeras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflite_keras_model)\ntflite_model = keras_model_converter.convert()\nwith open('/kaggle/working/model.tflite', 'wb') as f:\n    f.write(tflite_model)\n\n!zip submission.zip /kaggle/working/model.tflite\n\n```\n\nOutput: updating: kaggle/working/model.tflite (deflated 8%)\n\n\n```\n# Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=demo_raw_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n\n```\nworks without error and found_signatures = ['serving_default'] \n\n\n**Test input/output Tensor:**\n\n```\n# Load the TFLite model\ntflite_model_file = '/kaggle/working/model.tflite'\ninterpreter = tflite.Interpreter(model_path=tflite_model_file)\n\n# Allocate tensors\n#interpreter.allocate_tensors()\n\n# Get input and output tensors\ninput_details = interpreter.get_input_details()\noutput_details = interpreter.get_output_details()\nprint(input_details)\nprint(\"----output----\")\nprint(output_details)\n\n```\n\nThis gives me:\n\n```\n[{'name': 'inputs', 'index': 0, 'shape': array([  1, 543,   3], dtype=int32), 'shape_signature': array([ -1, 543,   3], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n\n----output----\n\n[{'name': 'Identity', 'index': 77, 'shape': array([250], dtype=int32), 'shape_signature': array([250], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n\n```",
      "votes": null
    },
    {
      "id": "2226304",
      "postDate": "04/18/2023 20:27:34",
      "content": "<p>I was able to fix it. <br>\nProblem was my preprocessing layer and a training dataset that only allowed a certain number of frames.<br>\nWith this code I was able to detect the problem. See <code># Create dummy data to play with shapes</code></p>\n<pre><code># Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model2.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\n# Create dummy data to play with shapes\ninput_data = np.random.rand(78, 543, 3).astype(np.float32)\n\noutput = prediction_fn(inputs=input_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n</code></pre>",
      "rawMarkdown": "I was able to fix it. \nProblem was my preprocessing layer and a training dataset that only allowed a certain number of frames.\nWith this code I was able to detect the problem. See `# Create dummy data to play with shapes`\n```\n# Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model2.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\n# Create dummy data to play with shapes\ninput_data = np.random.rand(78, 543, 3).astype(np.float32)\n\noutput = prediction_fn(inputs=input_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n```",
      "votes": null
    },
    {
      "id": "2229489",
      "postDate": "04/21/2023 12:13:29",
      "content": "<p>How much time usually scoring gets for you? My scoring fails after 2-3 hours. I have same issue, but my preprocessing layer working good with any length of inputs</p>",
      "rawMarkdown": "How much time usually scoring gets for you? My scoring fails after 2-3 hours. I have same issue, but my preprocessing layer working good with any length of inputs",
      "votes": null
    },
    {
      "id": "2229552",
      "postDate": "04/21/2023 13:13:49",
      "content": "<p>For me it was the error in the preprocessing layer.<br>\nAfter the fix I was able to submit a submission successfully.<br>\nSince then, however, it also takes me 2 + x hours and then it breaks. Others here have already reported that it is due to the resources of Kaggle. </p>\n<p>There is a limit of 1 hour for vaildation. Therefore, if your validation takes longer than 1 hour, it will always fail. My current understanding is that the validation runs longer because the resources are missing or queued.</p>\n<p>The whole thing is very frustrating and I now have 13 failed submissions in a row.<br>\nWhen I test the TFLite model, I get a runtime of less than 100ms. I have also tried quantization models but without success.</p>",
      "rawMarkdown": "For me it was the error in the preprocessing layer.\nAfter the fix I was able to submit a submission successfully.\nSince then, however, it also takes me 2 + x hours and then it breaks. Others here have already reported that it is due to the resources of Kaggle. \n\nThere is a limit of 1 hour for vaildation. Therefore, if your validation takes longer than 1 hour, it will always fail. My current understanding is that the validation runs longer because the resources are missing or queued.\n\nThe whole thing is very frustrating and I now have 13 failed submissions in a row.\nWhen I test the TFLite model, I get a runtime of less than 100ms. I have also tried quantization models but without success.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2226304,
      "author_name": "mariosommer",
      "author_url": "",
      "post_date": "04/18/2023 20:27:34",
      "content": "<p>I was able to fix it. <br>\nProblem was my preprocessing layer and a training dataset that only allowed a certain number of frames.<br>\nWith this code I was able to detect the problem. See <code># Create dummy data to play with shapes</code></p>\n<pre><code># Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model2.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\n# Create dummy data to play with shapes\ninput_data = np.random.rand(78, 543, 3).astype(np.float32)\n\noutput = prediction_fn(inputs=input_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 2229489,
          "author_name": "ch3rkasov",
          "author_url": "",
          "post_date": "04/21/2023 12:13:29",
          "content": "<p>How much time usually scoring gets for you? My scoring fails after 2-3 hours. I have same issue, but my preprocessing layer working good with any length of inputs</p>",
          "votes": null,
          "replies": [
            {
              "id": 2229552,
              "author_name": "mariosommer",
              "author_url": "",
              "post_date": "04/21/2023 13:13:49",
              "content": "<p>For me it was the error in the preprocessing layer.<br>\nAfter the fix I was able to submit a submission successfully.<br>\nSince then, however, it also takes me 2 + x hours and then it breaks. Others here have already reported that it is due to the resources of Kaggle. </p>\n<p>There is a limit of 1 hour for vaildation. Therefore, if your validation takes longer than 1 hour, it will always fail. My current understanding is that the validation runs longer because the resources are missing or queued.</p>\n<p>The whole thing is very frustrating and I now have 13 failed submissions in a row.<br>\nWhen I test the TFLite model, I get a runtime of less than 100ms. I have also tried quantization models but without success.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2226250": "My notebook generates a submission.zip file from the tflite model.\nWhen I try the model in the notebook it works. But in the submission comes the error from the title. \nI tried to output the input and output layer and it seems to be ok for me. Do any of you notice anything? Or what other possibilities do I have to debug the submission or to get more info about the error?\n\n\n```\nkeras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflite_keras_model)\ntflite_model = keras_model_converter.convert()\nwith open('/kaggle/working/model.tflite', 'wb') as f:\n    f.write(tflite_model)\n\n!zip submission.zip /kaggle/working/model.tflite\n\n```\n\nOutput: updating: kaggle/working/model.tflite (deflated 8%)\n\n\n```\n# Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=demo_raw_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n\n```\nworks without error and found_signatures = ['serving_default'] \n\n\n**Test input/output Tensor:**\n\n```\n# Load the TFLite model\ntflite_model_file = '/kaggle/working/model.tflite'\ninterpreter = tflite.Interpreter(model_path=tflite_model_file)\n\n# Allocate tensors\n#interpreter.allocate_tensors()\n\n# Get input and output tensors\ninput_details = interpreter.get_input_details()\noutput_details = interpreter.get_output_details()\nprint(input_details)\nprint(\"----output----\")\nprint(output_details)\n\n```\n\nThis gives me:\n\n```\n[{'name': 'inputs', 'index': 0, 'shape': array([  1, 543,   3], dtype=int32), 'shape_signature': array([ -1, 543,   3], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n\n----output----\n\n[{'name': 'Identity', 'index': 77, 'shape': array([250], dtype=int32), 'shape_signature': array([250], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]\n\n```",
    "2226304": "I was able to fix it. \nProblem was my preprocessing layer and a training dataset that only allowed a certain number of frames.\nWith this code I was able to detect the problem. See `# Create dummy data to play with shapes`\n```\n# Verify TFLite model can be loaded and used for prediction\n#!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/model2.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprint(found_signatures)  # Print the available signatures\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\n# Create dummy data to play with shapes\ninput_data = np.random.rand(78, 543, 3).astype(np.float32)\n\noutput = prediction_fn(inputs=input_data)\nsign = output['outputs'].argmax()\n\nprint(\"PRED : \", ORD2SIGN.get(sign), f'[{sign}]')\nprint(\"TRUE : \", train.sign.values[0], f'[{train.sign_ord.values[0]}]')\n```",
    "2229489": "How much time usually scoring gets for you? My scoring fails after 2-3 hours. I have same issue, but my preprocessing layer working good with any length of inputs",
    "2229552": "For me it was the error in the preprocessing layer.\nAfter the fix I was able to submit a submission successfully.\nSince then, however, it also takes me 2 + x hours and then it breaks. Others here have already reported that it is due to the resources of Kaggle. \n\nThere is a limit of 1 hour for vaildation. Therefore, if your validation takes longer than 1 hour, it will always fail. My current understanding is that the validation runs longer because the resources are missing or queued.\n\nThe whole thing is very frustrating and I now have 13 failed submissions in a row.\nWhen I test the TFLite model, I get a runtime of less than 100ms. I have also tried quantization models but without success."
  },
  "source": "meta"
}