{
  "id": 401202,
  "title": "Submission scoring error",
  "url": "/competitions/asl-signs/discussion/401202",
  "author_name": "",
  "post_date": "2023-04-12T08:04:44.659158200Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi! I'm trying to understand why my submission fails when I try to submit ensemble.<br>\nkernel with the model that fails during sub: <a href=\"https://www.kaggle.com/code/martynoveduard/asl-sub-debug\" target=\"_blank\">https://www.kaggle.com/code/martynoveduard/asl-sub-debug</a> </p>\n<p>Single model submission goes fine and gets a score in 27 minutes of runtime, <br>\nhere's the benchmark output, thanks <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for sharing the code (time measured in microseconds):</p>\n<pre><code>INFO: STARTING!\nWARN: Unconsumed cmdline flags: \\\nINFO: Log parameter values verbosely: [0]\nINFO: Num threads: [4]\nINFO: Report the peak memory footprint: [1]\nINFO: Graph: [/home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite]\nINFO: Input layers: [input1]\nINFO: Input shapes: [256,543,3]\nINFO: Enable op profiling: [1]\nINFO: CSV File to export profiling data to: [xxx.csv]\nINFO: #threads used for CPU inference: [4]\nINFO: Loaded model /home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite\nWARN: Tensor # 0 is named serving_default_inputs:0 but flags call it input1\nINFO: The input model file size (MB): 4.2064\nINFO: Initialized session in 19.582ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=13 first=50299 curr=38168 min=37058 max=50299 avg=39155.6 std=3311\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=38385 curr=35680 min=34800 max=49320 avg=37269.1 std=2353\n\nINFO: Inference timings in us: Init: 19582, First inference: 50299, Warmup (avg): 39155.6, Inference (avg): 37269.1\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=5.44922 overall=32.3633\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 36.8047\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 32 MB\nINFO: + RssAnnon           : 21 MB\nINFO: + RssFile + RssShmem : 11 MB\n\n\nrun_tflite_benchmark() ok !!!\n</code></pre>\n<p>However when I try to submit 2 folds of the same model it fails :(<br>\nWorth noting that submission fails very early (~2min after kernel starts running)</p>\n<pre><code>INFO: The input model file size (MB): 8.39862\nINFO: Initialized session in 27.941ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=8 first=84834 curr=62929 min=61489 max=84834 avg=65416.2 std=7357\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=62985 curr=62698 min=60979 max=63673 avg=62735.5 std=431\n\nINFO: Inference timings in us: Init: 27941, First inference: 84834, Warmup (avg): 65416.2, Inference (avg): 62735.5\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=8.03906 overall=56.0391\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 60.4531\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 47 MB\nINFO: + RssAnnon           : 32 MB\nINFO: + RssFile + RssShmem : 15 MB\n\n\nrun_tflite_benchmark() ok !!!\n</code></pre>\n<p>Are there any obvious problems</p>",
  "messages": [
    {
      "id": "2219029",
      "postDate": "04/12/2023 08:04:44",
      "content": "<p>Hi! I'm trying to understand why my submission fails when I try to submit ensemble.<br>\nkernel with the model that fails during sub: <a href=\"https://www.kaggle.com/code/martynoveduard/asl-sub-debug\" target=\"_blank\">https://www.kaggle.com/code/martynoveduard/asl-sub-debug</a> </p>\n<p>Single model submission goes fine and gets a score in 27 minutes of runtime, <br>\nhere's the benchmark output, thanks <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> for sharing the code (time measured in microseconds):</p>\n<pre><code>INFO: STARTING!\nWARN: Unconsumed cmdline flags: \\\nINFO: Log parameter values verbosely: [0]\nINFO: Num threads: [4]\nINFO: Report the peak memory footprint: [1]\nINFO: Graph: [/home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite]\nINFO: Input layers: [input1]\nINFO: Input shapes: [256,543,3]\nINFO: Enable op profiling: [1]\nINFO: CSV File to export profiling data to: [xxx.csv]\nINFO: #threads used for CPU inference: [4]\nINFO: Loaded model /home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite\nWARN: Tensor # 0 is named serving_default_inputs:0 but flags call it input1\nINFO: The input model file size (MB): 4.2064\nINFO: Initialized session in 19.582ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=13 first=50299 curr=38168 min=37058 max=50299 avg=39155.6 std=3311\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=38385 curr=35680 min=34800 max=49320 avg=37269.1 std=2353\n\nINFO: Inference timings in us: Init: 19582, First inference: 50299, Warmup (avg): 39155.6, Inference (avg): 37269.1\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=5.44922 overall=32.3633\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 36.8047\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 32 MB\nINFO: + RssAnnon           : 21 MB\nINFO: + RssFile + RssShmem : 11 MB\n\n\nrun_tflite_benchmark() ok !!!\n</code></pre>\n<p>However when I try to submit 2 folds of the same model it fails :(<br>\nWorth noting that submission fails very early (~2min after kernel starts running)</p>\n<pre><code>INFO: The input model file size (MB): 8.39862\nINFO: Initialized session in 27.941ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=8 first=84834 curr=62929 min=61489 max=84834 avg=65416.2 std=7357\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=62985 curr=62698 min=60979 max=63673 avg=62735.5 std=431\n\nINFO: Inference timings in us: Init: 27941, First inference: 84834, Warmup (avg): 65416.2, Inference (avg): 62735.5\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=8.03906 overall=56.0391\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 60.4531\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 47 MB\nINFO: + RssAnnon           : 32 MB\nINFO: + RssFile + RssShmem : 15 MB\n\n\nrun_tflite_benchmark() ok !!!\n</code></pre>\n<p>Are there any obvious problems</p>",
      "rawMarkdown": "Hi! I'm trying to understand why my submission fails when I try to submit ensemble.\nkernel with the model that fails during sub: https://www.kaggle.com/code/martynoveduard/asl-sub-debug \n\nSingle model submission goes fine and gets a score in 27 minutes of runtime, \nhere's the benchmark output, thanks @hengck23 for sharing the code (time measured in microseconds):\n```\nINFO: STARTING!\nWARN: Unconsumed cmdline flags: \\\nINFO: Log parameter values verbosely: [0]\nINFO: Num threads: [4]\nINFO: Report the peak memory footprint: [1]\nINFO: Graph: [/home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite]\nINFO: Input layers: [input1]\nINFO: Input shapes: [256,543,3]\nINFO: Enable op profiling: [1]\nINFO: CSV File to export profiling data to: [xxx.csv]\nINFO: #threads used for CPU inference: [4]\nINFO: Loaded model /home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite\nWARN: Tensor # 0 is named serving_default_inputs:0 but flags call it input1\nINFO: The input model file size (MB): 4.2064\nINFO: Initialized session in 19.582ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=13 first=50299 curr=38168 min=37058 max=50299 avg=39155.6 std=3311\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=38385 curr=35680 min=34800 max=49320 avg=37269.1 std=2353\n\nINFO: Inference timings in us: Init: 19582, First inference: 50299, Warmup (avg): 39155.6, Inference (avg): 37269.1\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=5.44922 overall=32.3633\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 36.8047\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 32 MB\nINFO: + RssAnnon           : 21 MB\nINFO: + RssFile + RssShmem : 11 MB\n\n\nrun_tflite_benchmark() ok !!!\n```\n\nHowever when I try to submit 2 folds of the same model it fails :(\nWorth noting that submission fails very early (~2min after kernel starts running)\n\n```\nINFO: The input model file size (MB): 8.39862\nINFO: Initialized session in 27.941ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=8 first=84834 curr=62929 min=61489 max=84834 avg=65416.2 std=7357\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=62985 curr=62698 min=60979 max=63673 avg=62735.5 std=431\n\nINFO: Inference timings in us: Init: 27941, First inference: 84834, Warmup (avg): 65416.2, Inference (avg): 62735.5\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=8.03906 overall=56.0391\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 60.4531\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 47 MB\nINFO: + RssAnnon           : 32 MB\nINFO: + RssFile + RssShmem : 15 MB\n\n\nrun_tflite_benchmark() ok !!!\n```\n\nAre there any obvious problems",
      "votes": null
    },
    {
      "id": "2219320",
      "postDate": "04/12/2023 13:39:23",
      "content": "<blockquote>\n  <blockquote>\n    <p>Worth noting that submission fails very early (~2min after kernel starts running)</p>\n  </blockquote>\n</blockquote>\n<p>This means something wrong with your tflite file.<br>\nyou should try to loop over all your train samples in the kaggle notebook in development.</p>\n<p>If submission failed after 40 min (or the time you estimated to complete all test samples), it means that there is nothing wrong with your tflite file and all test test samples passed. However you did not meet the resource limitation (e.g. 40 mb, 100ms, etc), so the submission still fail</p>",
      "rawMarkdown": ">>Worth noting that submission fails very early (~2min after kernel starts running)\n\n\nThis means something wrong with your tflite file.\nyou should try to loop over all your train samples in the kaggle notebook in development.\n\n\nIf submission failed after 40 min (or the time you estimated to complete all test samples), it means that there is nothing wrong with your tflite file and all test test samples passed. However you did not meet the resource limitation (e.g. 40 mb, 100ms, etc), so the submission still fail",
      "votes": null
    },
    {
      "id": "2219753",
      "postDate": "04/12/2023 20:51:53",
      "content": "<p>I found an op that was expanded into 1800 compute nodes, I removed it and I got only 170 left, hope it'll help <br>\nI inspected the benchmark output csv file to find this op</p>\n<p>I'll report when I fix the issue</p>",
      "rawMarkdown": "I found an op that was expanded into 1800 compute nodes, I removed it and I got only 170 left, hope it'll help \nI inspected the benchmark output csv file to find this op\n\nI'll report when I fix the issue",
      "votes": null
    },
    {
      "id": "2220009",
      "postDate": "04/13/2023 04:17:23",
      "content": "<p>I replaced this op with a simplified version of it, and it didn't work, single model gets score after 20 minutes only (0.74) and 2-3 model ens fails in 2min<br>\nbenchmark output for the 2-model ens<br>\nthe amount of nodes in the final graph is not the issue</p>\n<pre><code>INFO: The input model file size (MB): 8.30289\n\nINFO: Inference timings in us: Init: 20318, First inference: 72438, Warmup (avg): 59304.7, Inference (avg): 57206.2\nINFO: Memory footprint delta from the start of the tool (MB): init=3.85938 overall=44.0508\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.4805\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 46 MB\nINFO: + RssAnnon           : 31 MB\nINFO: + RssFile + RssShmem : 15 MB\n</code></pre>",
      "rawMarkdown": "I replaced this op with a simplified version of it, and it didn't work, single model gets score after 20 minutes only (0.74) and 2-3 model ens fails in 2min\nbenchmark output for the 2-model ens\nthe amount of nodes in the final graph is not the issue\n```\nINFO: The input model file size (MB): 8.30289\n\nINFO: Inference timings in us: Init: 20318, First inference: 72438, Warmup (avg): 59304.7, Inference (avg): 57206.2\nINFO: Memory footprint delta from the start of the tool (MB): init=3.85938 overall=44.0508\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.4805\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 46 MB\nINFO: + RssAnnon           : 31 MB\nINFO: + RssFile + RssShmem : 15 MB\n```",
      "votes": null
    },
    {
      "id": "2220022",
      "postDate": "04/13/2023 04:35:52",
      "content": "<p>minimalistic run code for a failing 2-model ensemble mentioned in the post<br>\n<a href=\"https://www.kaggle.com/code/martynoveduard/asl-sub-debug\" target=\"_blank\">https://www.kaggle.com/code/martynoveduard/asl-sub-debug</a> <br>\nit seems that the model is compatible with tflite-2.9.1</p>",
      "rawMarkdown": "minimalistic run code for a failing 2-model ensemble mentioned in the post\nhttps://www.kaggle.com/code/martynoveduard/asl-sub-debug \nit seems that the model is compatible with tflite-2.9.1",
      "votes": null
    },
    {
      "id": "2220074",
      "postDate": "04/13/2023 05:27:55",
      "content": "<p>I fixed it, it appears that the error was simple, - evaluation script depends on the file extension to be exactly '.tflite', it wasn't obvious considering the fact that we're submitting the single file and it isn't mentioned on the evaluation page, however I agree that changing the file extension name is an anti-pattern</p>",
      "rawMarkdown": "I fixed it, it appears that the error was simple, - evaluation script depends on the file extension to be exactly '.tflite', it wasn't obvious considering the fact that we're submitting the single file and it isn't mentioned on the evaluation page, however I agree that changing the file extension name is an anti-pattern",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2219320,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "04/12/2023 13:39:23",
      "content": "<blockquote>\n  <blockquote>\n    <p>Worth noting that submission fails very early (~2min after kernel starts running)</p>\n  </blockquote>\n</blockquote>\n<p>This means something wrong with your tflite file.<br>\nyou should try to loop over all your train samples in the kaggle notebook in development.</p>\n<p>If submission failed after 40 min (or the time you estimated to complete all test samples), it means that there is nothing wrong with your tflite file and all test test samples passed. However you did not meet the resource limitation (e.g. 40 mb, 100ms, etc), so the submission still fail</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2219753,
      "author_name": "martynoveduard",
      "author_url": "",
      "post_date": "04/12/2023 20:51:53",
      "content": "<p>I found an op that was expanded into 1800 compute nodes, I removed it and I got only 170 left, hope it'll help <br>\nI inspected the benchmark output csv file to find this op</p>\n<p>I'll report when I fix the issue</p>",
      "votes": null,
      "replies": [
        {
          "id": 2220009,
          "author_name": "martynoveduard",
          "author_url": "",
          "post_date": "04/13/2023 04:17:23",
          "content": "<p>I replaced this op with a simplified version of it, and it didn't work, single model gets score after 20 minutes only (0.74) and 2-3 model ens fails in 2min<br>\nbenchmark output for the 2-model ens<br>\nthe amount of nodes in the final graph is not the issue</p>\n<pre><code>INFO: The input model file size (MB): 8.30289\n\nINFO: Inference timings in us: Init: 20318, First inference: 72438, Warmup (avg): 59304.7, Inference (avg): 57206.2\nINFO: Memory footprint delta from the start of the tool (MB): init=3.85938 overall=44.0508\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.4805\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 46 MB\nINFO: + RssAnnon           : 31 MB\nINFO: + RssFile + RssShmem : 15 MB\n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2220022,
      "author_name": "martynoveduard",
      "author_url": "",
      "post_date": "04/13/2023 04:35:52",
      "content": "<p>minimalistic run code for a failing 2-model ensemble mentioned in the post<br>\n<a href=\"https://www.kaggle.com/code/martynoveduard/asl-sub-debug\" target=\"_blank\">https://www.kaggle.com/code/martynoveduard/asl-sub-debug</a> <br>\nit seems that the model is compatible with tflite-2.9.1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2220074,
      "author_name": "martynoveduard",
      "author_url": "",
      "post_date": "04/13/2023 05:27:55",
      "content": "<p>I fixed it, it appears that the error was simple, - evaluation script depends on the file extension to be exactly '.tflite', it wasn't obvious considering the fact that we're submitting the single file and it isn't mentioned on the evaluation page, however I agree that changing the file extension name is an anti-pattern</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2219029": "Hi! I'm trying to understand why my submission fails when I try to submit ensemble.\nkernel with the model that fails during sub: https://www.kaggle.com/code/martynoveduard/asl-sub-debug \n\nSingle model submission goes fine and gets a score in 27 minutes of runtime, \nhere's the benchmark output, thanks @hengck23 for sharing the code (time measured in microseconds):\n```\nINFO: STARTING!\nWARN: Unconsumed cmdline flags: \\\nINFO: Log parameter values verbosely: [0]\nINFO: Num threads: [4]\nINFO: Report the peak memory footprint: [1]\nINFO: Graph: [/home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite]\nINFO: Input layers: [input1]\nINFO: Input shapes: [256,543,3]\nINFO: Enable op profiling: [1]\nINFO: CSV File to export profiling data to: [xxx.csv]\nINFO: #threads used for CPU inference: [4]\nINFO: Loaded model /home/slime/kaggle/asl-signs/exps/exp_85/converts/model_0.70547.tflite\nWARN: Tensor # 0 is named serving_default_inputs:0 but flags call it input1\nINFO: The input model file size (MB): 4.2064\nINFO: Initialized session in 19.582ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=13 first=50299 curr=38168 min=37058 max=50299 avg=39155.6 std=3311\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=38385 curr=35680 min=34800 max=49320 avg=37269.1 std=2353\n\nINFO: Inference timings in us: Init: 19582, First inference: 50299, Warmup (avg): 39155.6, Inference (avg): 37269.1\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=5.44922 overall=32.3633\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 36.8047\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 32 MB\nINFO: + RssAnnon           : 21 MB\nINFO: + RssFile + RssShmem : 11 MB\n\n\nrun_tflite_benchmark() ok !!!\n```\n\nHowever when I try to submit 2 folds of the same model it fails :(\nWorth noting that submission fails very early (~2min after kernel starts running)\n\n```\nINFO: The input model file size (MB): 8.39862\nINFO: Initialized session in 27.941ms.\nINFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.\nINFO: count=8 first=84834 curr=62929 min=61489 max=84834 avg=65416.2 std=7357\n\nINFO: Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.\nINFO: count=50 first=62985 curr=62698 min=60979 max=63673 avg=62735.5 std=431\n\nINFO: Inference timings in us: Init: 27941, First inference: 84834, Warmup (avg): 65416.2, Inference (avg): 62735.5\nINFO: Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.\nINFO: Memory footprint delta from the start of the tool (MB): init=8.03906 overall=56.0391\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 60.4531\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 47 MB\nINFO: + RssAnnon           : 32 MB\nINFO: + RssFile + RssShmem : 15 MB\n\n\nrun_tflite_benchmark() ok !!!\n```\n\nAre there any obvious problems",
    "2219320": ">>Worth noting that submission fails very early (~2min after kernel starts running)\n\n\nThis means something wrong with your tflite file.\nyou should try to loop over all your train samples in the kaggle notebook in development.\n\n\nIf submission failed after 40 min (or the time you estimated to complete all test samples), it means that there is nothing wrong with your tflite file and all test test samples passed. However you did not meet the resource limitation (e.g. 40 mb, 100ms, etc), so the submission still fail",
    "2219753": "I found an op that was expanded into 1800 compute nodes, I removed it and I got only 170 left, hope it'll help \nI inspected the benchmark output csv file to find this op\n\nI'll report when I fix the issue",
    "2220009": "I replaced this op with a simplified version of it, and it didn't work, single model gets score after 20 minutes only (0.74) and 2-3 model ens fails in 2min\nbenchmark output for the 2-model ens\nthe amount of nodes in the final graph is not the issue\n```\nINFO: The input model file size (MB): 8.30289\n\nINFO: Inference timings in us: Init: 20318, First inference: 72438, Warmup (avg): 59304.7, Inference (avg): 57206.2\nINFO: Memory footprint delta from the start of the tool (MB): init=3.85938 overall=44.0508\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.4805\nINFO: Memory status at the end of exeution:\nINFO: - VmRSS              : 46 MB\nINFO: + RssAnnon           : 31 MB\nINFO: + RssFile + RssShmem : 15 MB\n```",
    "2220022": "minimalistic run code for a failing 2-model ensemble mentioned in the post\nhttps://www.kaggle.com/code/martynoveduard/asl-sub-debug \nit seems that the model is compatible with tflite-2.9.1",
    "2220074": "I fixed it, it appears that the error was simple, - evaluation script depends on the file extension to be exactly '.tflite', it wasn't obvious considering the fact that we're submitting the single file and it isn't mentioned on the evaluation page, however I agree that changing the file extension name is an anti-pattern"
  },
  "source": "meta"
}