{
  "id": 498110,
  "title": "Notebook exception error",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/498110",
  "author_name": "",
  "post_date": "2024-04-26T22:16:28.198947200Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>This is the worst error to face as per me. But I am writing this post to share some ideas to tackle it.</p>\n<p>Lets say ur previous notebook was running smoothly, now u made some changes and this error pops up, what to do?<br>\nstep 1 - check ur recent changes that u made and think of all possible ways things can go wrong, like u assumed a particular datatype which was not consistent while execution, or the elements of list are ordered differently, etc.<br>\nstep 2 - check the running time of the notebook, after much of execution u got the error, it helps in giving a rough idea of the location of error.<br>\nstep 3 - check the recent changes from memory point of view, cold those recent changes exponentially increase memory usage, specially considering the unknown test data size.<br>\nstep 4 -  make prediction in batches, always.</p>\n<p>Now lets say ur wrote a fresh new notebook and u got the error, now u have no idea where to look for.<br>\nstep 1 - assuming the notebook ran successfully locally so now the only difference is the test data. this gives a rough idea hat it has to somewhere related to handling of test data.<br>\nstep 2 - test data might have different datatype than expected, encontering None or Nan or null or inf. check all the places where we are assuming about test data.<br>\nstep 3 - check the handling of test data from memory point of view.<br>\nstep 4 - make prediction in batches, always</p>\n<p>Now lets say u have tried everything, and failed, but  want the notebook to be submitted. and u can do anything for it.<br>\nstep 1 - hash the entire code<br>\nstep 2 - make a dummy submission and place it at the end or start (unhashed)<br>\nstep 3 - unhash the top few blocks of code and submit<br>\nstep 4 - than unhash next few blocks of code and submit, continue this till u get the error<br>\nwith this progressive hashing, u will be able pin point which block of code is causing the error.<br>\nthis will take away few submissions but will give an idea where the error is.</p>\n<p>hope this helps.</p>",
  "messages": [
    {
      "id": "2777977",
      "postDate": "04/26/2024 22:16:28",
      "content": "<p>This is the worst error to face as per me. But I am writing this post to share some ideas to tackle it.</p>\n<p>Lets say ur previous notebook was running smoothly, now u made some changes and this error pops up, what to do?<br>\nstep 1 - check ur recent changes that u made and think of all possible ways things can go wrong, like u assumed a particular datatype which was not consistent while execution, or the elements of list are ordered differently, etc.<br>\nstep 2 - check the running time of the notebook, after much of execution u got the error, it helps in giving a rough idea of the location of error.<br>\nstep 3 - check the recent changes from memory point of view, cold those recent changes exponentially increase memory usage, specially considering the unknown test data size.<br>\nstep 4 -  make prediction in batches, always.</p>\n<p>Now lets say ur wrote a fresh new notebook and u got the error, now u have no idea where to look for.<br>\nstep 1 - assuming the notebook ran successfully locally so now the only difference is the test data. this gives a rough idea hat it has to somewhere related to handling of test data.<br>\nstep 2 - test data might have different datatype than expected, encontering None or Nan or null or inf. check all the places where we are assuming about test data.<br>\nstep 3 - check the handling of test data from memory point of view.<br>\nstep 4 - make prediction in batches, always</p>\n<p>Now lets say u have tried everything, and failed, but  want the notebook to be submitted. and u can do anything for it.<br>\nstep 1 - hash the entire code<br>\nstep 2 - make a dummy submission and place it at the end or start (unhashed)<br>\nstep 3 - unhash the top few blocks of code and submit<br>\nstep 4 - than unhash next few blocks of code and submit, continue this till u get the error<br>\nwith this progressive hashing, u will be able pin point which block of code is causing the error.<br>\nthis will take away few submissions but will give an idea where the error is.</p>\n<p>hope this helps.</p>",
      "rawMarkdown": "This is the worst error to face as per me. But I am writing this post to share some ideas to tackle it.\n\nLets say ur previous notebook was running smoothly, now u made some changes and this error pops up, what to do?\nstep 1 - check ur recent changes that u made and think of all possible ways things can go wrong, like u assumed a particular datatype which was not consistent while execution, or the elements of list are ordered differently, etc.\nstep 2 - check the running time of the notebook, after much of execution u got the error, it helps in giving a rough idea of the location of error.\nstep 3 - check the recent changes from memory point of view, cold those recent changes exponentially increase memory usage, specially considering the unknown test data size.\nstep 4 -  make prediction in batches, always.\n\nNow lets say ur wrote a fresh new notebook and u got the error, now u have no idea where to look for.\nstep 1 - assuming the notebook ran successfully locally so now the only difference is the test data. this gives a rough idea hat it has to somewhere related to handling of test data.\nstep 2 - test data might have different datatype than expected, encontering None or Nan or null or inf. check all the places where we are assuming about test data.\nstep 3 - check the handling of test data from memory point of view.\nstep 4 - make prediction in batches, always\n\nNow lets say u have tried everything, and failed, but  want the notebook to be submitted. and u can do anything for it.\nstep 1 - hash the entire code\nstep 2 - make a dummy submission and place it at the end or start (unhashed)\nstep 3 - unhash the top few blocks of code and submit\nstep 4 - than unhash next few blocks of code and submit, continue this till u get the error\nwith this progressive hashing, u will be able pin point which block of code is causing the error.\nthis will take away few submissions but will give an idea where the error is.\n\nhope this helps.",
      "votes": null
    },
    {
      "id": "2778094",
      "postDate": "04/27/2024 00:18:51",
      "content": "<p>Thank you for providing the method.</p>",
      "rawMarkdown": "Thank you for providing the method.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2778094,
      "author_name": "yunsuxiaozi",
      "author_url": "",
      "post_date": "04/27/2024 00:18:51",
      "content": "<p>Thank you for providing the method.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2777977": "This is the worst error to face as per me. But I am writing this post to share some ideas to tackle it.\n\nLets say ur previous notebook was running smoothly, now u made some changes and this error pops up, what to do?\nstep 1 - check ur recent changes that u made and think of all possible ways things can go wrong, like u assumed a particular datatype which was not consistent while execution, or the elements of list are ordered differently, etc.\nstep 2 - check the running time of the notebook, after much of execution u got the error, it helps in giving a rough idea of the location of error.\nstep 3 - check the recent changes from memory point of view, cold those recent changes exponentially increase memory usage, specially considering the unknown test data size.\nstep 4 -  make prediction in batches, always.\n\nNow lets say ur wrote a fresh new notebook and u got the error, now u have no idea where to look for.\nstep 1 - assuming the notebook ran successfully locally so now the only difference is the test data. this gives a rough idea hat it has to somewhere related to handling of test data.\nstep 2 - test data might have different datatype than expected, encontering None or Nan or null or inf. check all the places where we are assuming about test data.\nstep 3 - check the handling of test data from memory point of view.\nstep 4 - make prediction in batches, always\n\nNow lets say u have tried everything, and failed, but  want the notebook to be submitted. and u can do anything for it.\nstep 1 - hash the entire code\nstep 2 - make a dummy submission and place it at the end or start (unhashed)\nstep 3 - unhash the top few blocks of code and submit\nstep 4 - than unhash next few blocks of code and submit, continue this till u get the error\nwith this progressive hashing, u will be able pin point which block of code is causing the error.\nthis will take away few submissions but will give an idea where the error is.\n\nhope this helps.",
    "2778094": "Thank you for providing the method."
  },
  "source": "meta"
}