{
  "id": 342622,
  "title": "Suddenly... I cannot read plain old test file",
  "url": "/competitions/amex-default-prediction/discussion/342622",
  "author_name": "",
  "post_date": "2022-08-08T05:06:18.884339300Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi. I have submitted a handful of notebooks, the last one 12 hours ago, without a problem.</p>\n<p>I use <a href=\"https://www.kaggle.com/raddar\" target=\"_blank\">@raddar</a> versión of test (3.3GB, parquet) and I delete everything about training (except the model) before procesing the test file. </p>\n<p>Now, I get the message: \"Your notebook tried to allocate…\"</p>\n<p>¿Has something change in the last hours?</p>\n<p>Here is my notebook. <a href=\"https://www.kaggle.com/code/jsmithperera/amex-lightgbm-v3\" target=\"_blank\">https://www.kaggle.com/code/jsmithperera/amex-lightgbm-v3</a></p>\n<p><strong>Update 08/08</strong>: Apparently it's the model. My current model has 926 features. When I ran it with 576 features, it read test without problem.  But if test requieres 7 GB, then apparently the model needs 9 GB. Is that possible?</p>\n<p><strong>Update 08/09</strong>: I am still getting the out of memory message but discovered a way to know how much RAM is being used by the NB while the NB is run by hand. You just need to go to the three vertical dots and press View Session Metrics. One of the metric is RAM.</p>",
  "messages": [
    {
      "id": "1889242",
      "postDate": "08/08/2022 05:06:18",
      "content": "<p>Hi. I have submitted a handful of notebooks, the last one 12 hours ago, without a problem.</p>\n<p>I use <a href=\"https://www.kaggle.com/raddar\" target=\"_blank\">@raddar</a> versión of test (3.3GB, parquet) and I delete everything about training (except the model) before procesing the test file. </p>\n<p>Now, I get the message: \"Your notebook tried to allocate…\"</p>\n<p>¿Has something change in the last hours?</p>\n<p>Here is my notebook. <a href=\"https://www.kaggle.com/code/jsmithperera/amex-lightgbm-v3\" target=\"_blank\">https://www.kaggle.com/code/jsmithperera/amex-lightgbm-v3</a></p>\n<p><strong>Update 08/08</strong>: Apparently it's the model. My current model has 926 features. When I ran it with 576 features, it read test without problem.  But if test requieres 7 GB, then apparently the model needs 9 GB. Is that possible?</p>\n<p><strong>Update 08/09</strong>: I am still getting the out of memory message but discovered a way to know how much RAM is being used by the NB while the NB is run by hand. You just need to go to the three vertical dots and press View Session Metrics. One of the metric is RAM.</p>",
      "rawMarkdown": "Hi. I have submitted a handful of notebooks, the last one 12 hours ago, without a problem.\n\nI use @raddar versión of test (3.3GB, parquet) and I delete everything about training (except the model) before procesing the test file. \n\nNow, I get the message: \"Your notebook tried to allocate...\"\n\n¿Has something change in the last hours?\n\nHere is my notebook. https://www.kaggle.com/code/jsmithperera/amex-lightgbm-v3\n\n**Update 08/08**: Apparently it's the model. My current model has 926 features. When I ran it with 576 features, it read test without problem.  But if test requieres 7 GB, then apparently the model needs 9 GB. Is that possible?\n\n**Update 08/09**: I am still getting the out of memory message but discovered a way to know how much RAM is being used by the NB while the NB is run by hand. You just need to go to the three vertical dots and press View Session Metrics. One of the metric is RAM.",
      "votes": null
    },
    {
      "id": "1889293",
      "postDate": "08/08/2022 05:54:57",
      "content": "<p>It seems like a technical issue. I suggest to wait for some time or use it elsewhere like Colab/ local device </p>",
      "rawMarkdown": "It seems like a technical issue. I suggest to wait for some time or use it elsewhere like Colab/ local device",
      "votes": null
    },
    {
      "id": "1889866",
      "postDate": "08/08/2022 12:10:17",
      "content": "<p>Ok. Thank you</p>",
      "rawMarkdown": "Ok. Thank you",
      "votes": null
    },
    {
      "id": "1906342",
      "postDate": "08/19/2022 19:32:28",
      "content": "<p>I have same issue. My new model performs much better but when I read parquest test files, it restarts.<br>\nDask dataframe doesn't work as it load data fast but then I can't run any commands from dask.dataframe.read_parquet(…….)    dd.group_by(col_id).set_index().. It considers set_index command as column and says no such attributes.</p>",
      "rawMarkdown": "I have same issue. My new model performs much better but when I read parquest test files, it restarts.\nDask dataframe doesn't work as it load data fast but then I can't run any commands from dask.dataframe.read_parquet(.......)    dd.group_by(col_id).set_index().. It considers set_index command as column and says no such attributes.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1889293,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "08/08/2022 05:54:57",
      "content": "<p>It seems like a technical issue. I suggest to wait for some time or use it elsewhere like Colab/ local device </p>",
      "votes": null,
      "replies": [
        {
          "id": 1889866,
          "author_name": "jsmithperera",
          "author_url": "",
          "post_date": "08/08/2022 12:10:17",
          "content": "<p>Ok. Thank you</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1906342,
      "author_name": "gauravsingh37",
      "author_url": "",
      "post_date": "08/19/2022 19:32:28",
      "content": "<p>I have same issue. My new model performs much better but when I read parquest test files, it restarts.<br>\nDask dataframe doesn't work as it load data fast but then I can't run any commands from dask.dataframe.read_parquet(…….)    dd.group_by(col_id).set_index().. It considers set_index command as column and says no such attributes.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1889242": "Hi. I have submitted a handful of notebooks, the last one 12 hours ago, without a problem.\n\nI use @raddar versión of test (3.3GB, parquet) and I delete everything about training (except the model) before procesing the test file. \n\nNow, I get the message: \"Your notebook tried to allocate...\"\n\n¿Has something change in the last hours?\n\nHere is my notebook. https://www.kaggle.com/code/jsmithperera/amex-lightgbm-v3\n\n**Update 08/08**: Apparently it's the model. My current model has 926 features. When I ran it with 576 features, it read test without problem.  But if test requieres 7 GB, then apparently the model needs 9 GB. Is that possible?\n\n**Update 08/09**: I am still getting the out of memory message but discovered a way to know how much RAM is being used by the NB while the NB is run by hand. You just need to go to the three vertical dots and press View Session Metrics. One of the metric is RAM.",
    "1889293": "It seems like a technical issue. I suggest to wait for some time or use it elsewhere like Colab/ local device",
    "1889866": "Ok. Thank you",
    "1906342": "I have same issue. My new model performs much better but when I read parquest test files, it restarts.\nDask dataframe doesn't work as it load data fast but then I can't run any commands from dask.dataframe.read_parquet(.......)    dd.group_by(col_id).set_index().. It considers set_index command as column and says no such attributes."
  },
  "source": "meta"
}