{
  "id": 596265,
  "title": "How to fix this error? ",
  "url": "/competitions/aeroclub-recsys-2025/discussion/596265",
  "author_name": "",
  "post_date": "2025-08-02T16:01:04.543501800Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>NotFoundError: Graph execution error:</p>\n<p>Detected at node StatefulPartitionedCall defined at (most recent call last):<br>\n  File \"/usr/local/lib/python3.10/runpy.py\", line 196, in _run_module_as_main</p>\n<p>File \"/usr/local/lib/python3.10/runpy.py\", line 86, in _run_code</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in </p>\n<p>File \"/usr/local/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 765, in start</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py\", line 74, in run</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 2310, in run</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 217, in run</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/events.py\", line 80, in _run</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 435, in process_shell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 512, in process_shell_message</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 361, in execute_request</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 766, in execute_request</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 457, in do_execute</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 606, in run_cell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3077, in run_cell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3132, in _run_cell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3336, in run_cell_async</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3519, in run_ast_nodes</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3579, in run_code</p>\n<p>File \"/tmp/ipykernel_10/883686475.py\", line 148, in </p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 377, in fit</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 220, in function</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 133, in multi_step_on_iterator</p>\n<p>could not find registered transfer manager for platform Host -- check target linkage<br>\n     [[{{node StatefulPartitionedCall}}]] [Op:__inference_multi_step_on_iterator_30330]<br>\nadd Codeadd Markdown</p>",
  "messages": [
    {
      "id": "3261953",
      "postDate": "08/02/2025 16:01:04",
      "content": "<p>NotFoundError: Graph execution error:</p>\n<p>Detected at node StatefulPartitionedCall defined at (most recent call last):<br>\n  File \"/usr/local/lib/python3.10/runpy.py\", line 196, in _run_module_as_main</p>\n<p>File \"/usr/local/lib/python3.10/runpy.py\", line 86, in _run_code</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in </p>\n<p>File \"/usr/local/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 765, in start</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py\", line 74, in run</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 2310, in run</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 217, in run</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once</p>\n<p>File \"/usr/local/lib/python3.10/asyncio/events.py\", line 80, in _run</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 435, in process_shell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 512, in process_shell_message</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 361, in execute_request</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 766, in execute_request</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 457, in do_execute</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 606, in run_cell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3077, in run_cell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3132, in _run_cell</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3336, in run_cell_async</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3519, in run_ast_nodes</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3579, in run_code</p>\n<p>File \"/tmp/ipykernel_10/883686475.py\", line 148, in </p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 377, in fit</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 220, in function</p>\n<p>File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 133, in multi_step_on_iterator</p>\n<p>could not find registered transfer manager for platform Host -- check target linkage<br>\n     [[{{node StatefulPartitionedCall}}]] [Op:__inference_multi_step_on_iterator_30330]<br>\nadd Codeadd Markdown</p>",
      "rawMarkdown": "NotFoundError: Graph execution error:\n\nDetected at node StatefulPartitionedCall defined at (most recent call last):\n  File \"/usr/local/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n\n  File \"/usr/local/lib/python3.10/runpy.py\", line 86, in _run_code\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in <module>\n\n  File \"/usr/local/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 765, in start\n\n  File \"/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py\", line 74, in run\n\n  File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 2310, in run\n\n  File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 217, in run\n\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete\n\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n\n  File \"/usr/local/lib/python3.10/asyncio/events.py\", line 80, in _run\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 435, in process_shell\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 512, in process_shell_message\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 361, in execute_request\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 766, in execute_request\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 457, in do_execute\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 606, in run_cell\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3077, in run_cell\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3132, in _run_cell\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3336, in run_cell_async\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3519, in run_ast_nodes\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3579, in run_code\n\n  File \"/tmp/ipykernel_10/883686475.py\", line 148, in <module>\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 377, in fit\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 220, in function\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 133, in multi_step_on_iterator\n\ncould not find registered transfer manager for platform Host -- check target linkage\n\t [[{{node StatefulPartitionedCall}}]] [Op:__inference_multi_step_on_iterator_30330]\nadd Codeadd Markdown",
      "votes": null
    },
    {
      "id": "3262236",
      "postDate": "08/03/2025 08:25:48",
      "content": "<p><a href=\"https://www.kaggle.com/pranava2458\" target=\"_blank\">@pranava2458</a> I faced something similar, I presume you are using GPU T4x2 if that is the case this is due to imbalanced distribution of samples in both the GPUs resulting in different shaped vectors/ results out of those GPUs. So it is unable to combine the results and proceed further.<br>\nSolutions-</p>\n<ul>\n<li>You can use shuffle or repeat certain data points to work it out. (Not preferred but is an option.)</li>\n<li>You can switch to GPU P100 which is a single entity and edit the code likewise. (Better solution)<br>\nDont forget to set seeds in tensorflow etc. as well since I learnt it the hard way and not being able to produce the good results again.</li>\n</ul>",
      "rawMarkdown": "pranava2458 I faced something similar, I presume you are using GPU T4x2 if that is the case this is due to imbalanced distribution of samples in both the GPUs resulting in different shaped vectors/ results out of those GPUs. So it is unable to combine the results and proceed further.\nSolutions-\n- You can use shuffle or repeat certain data points to work it out. (Not preferred but is an option.)\n- You can switch to GPU P100 which is a single entity and edit the code likewise. (Better solution)\nDont forget to set seeds in tensorflow etc. as well since I learnt it the hard way and not being able to produce the good results again.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3262236,
      "author_name": "shubhdas",
      "author_url": "",
      "post_date": "08/03/2025 08:25:48",
      "content": "<p><a href=\"https://www.kaggle.com/pranava2458\" target=\"_blank\">@pranava2458</a> I faced something similar, I presume you are using GPU T4x2 if that is the case this is due to imbalanced distribution of samples in both the GPUs resulting in different shaped vectors/ results out of those GPUs. So it is unable to combine the results and proceed further.<br>\nSolutions-</p>\n<ul>\n<li>You can use shuffle or repeat certain data points to work it out. (Not preferred but is an option.)</li>\n<li>You can switch to GPU P100 which is a single entity and edit the code likewise. (Better solution)<br>\nDont forget to set seeds in tensorflow etc. as well since I learnt it the hard way and not being able to produce the good results again.</li>\n</ul>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3261953": "NotFoundError: Graph execution error:\n\nDetected at node StatefulPartitionedCall defined at (most recent call last):\n  File \"/usr/local/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n\n  File \"/usr/local/lib/python3.10/runpy.py\", line 86, in _run_code\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in <module>\n\n  File \"/usr/local/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 765, in start\n\n  File \"/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py\", line 74, in run\n\n  File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 2310, in run\n\n  File \"/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py\", line 217, in run\n\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete\n\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n\n  File \"/usr/local/lib/python3.10/asyncio/events.py\", line 80, in _run\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 435, in process_shell\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 512, in process_shell_message\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 361, in execute_request\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 766, in execute_request\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 457, in do_execute\n\n  File \"/usr/local/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 606, in run_cell\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3077, in run_cell\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3132, in _run_cell\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3336, in run_cell_async\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3519, in run_ast_nodes\n\n  File \"/usr/local/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3579, in run_code\n\n  File \"/tmp/ipykernel_10/883686475.py\", line 148, in <module>\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 377, in fit\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 220, in function\n\n  File \"/usr/local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 133, in multi_step_on_iterator\n\ncould not find registered transfer manager for platform Host -- check target linkage\n\t [[{{node StatefulPartitionedCall}}]] [Op:__inference_multi_step_on_iterator_30330]\nadd Codeadd Markdown",
    "3262236": "pranava2458 I faced something similar, I presume you are using GPU T4x2 if that is the case this is due to imbalanced distribution of samples in both the GPUs resulting in different shaped vectors/ results out of those GPUs. So it is unable to combine the results and proceed further.\nSolutions-\n- You can use shuffle or repeat certain data points to work it out. (Not preferred but is an option.)\n- You can switch to GPU P100 which is a single entity and edit the code likewise. (Better solution)\nDont forget to set seeds in tensorflow etc. as well since I learnt it the hard way and not being able to produce the good results again."
  },
  "source": "meta"
}