{
  "id": 335222,
  "title": "how to load big file in Rapid  ?",
  "url": "/competitions/amex-default-prediction/discussion/335222",
  "author_name": "",
  "post_date": "2022-07-05T07:42:52.485214Z",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I got this error.</p>\n<p>MemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory</p>",
  "messages": [
    {
      "id": "1843912",
      "postDate": "07/05/2022 07:42:52",
      "content": "<p>I got this error.</p>\n<p>MemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory</p>",
      "rawMarkdown": "I got this error.\n\nMemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory",
      "votes": null
    },
    {
      "id": "1844071",
      "postDate": "07/05/2022 10:23:59",
      "content": "<p><a href=\"https://www.kaggle.com/dragonzhang\" target=\"_blank\">@dragonzhang</a> Do you mean RAPIDS (rapids.ai)?</p>",
      "rawMarkdown": "dragonzhang Do you mean RAPIDS (rapids.ai)?",
      "votes": null
    },
    {
      "id": "1844378",
      "postDate": "07/05/2022 14:01:45",
      "content": "<p>This URL may be helpful for you:<br>\n<a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790</a></p>",
      "rawMarkdown": "This URL may be helpful for you:\n[https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790](url)",
      "votes": null
    },
    {
      "id": "1844980",
      "postDate": "07/06/2022 00:53:26",
      "content": "<p>Yes, the link above uses RAPIDS to read the dataset and convert it into a 3D NumPy array to train an RNN or Transformer. And the link <a href=\"https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793\" target=\"_blank\">here</a> uses RAPIDS to read the dataset (and leave as a 2D dataframe) to train GBT like XGB, LGBM, or CatBoost</p>",
      "rawMarkdown": "Yes, the link above uses RAPIDS to read the dataset and convert it into a 3D NumPy array to train an RNN or Transformer. And the link [here][1] uses RAPIDS to read the dataset (and leave as a 2D dataframe) to train GBT like XGB, LGBM, or CatBoost\n\n[1]: https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793",
      "votes": null
    },
    {
      "id": "1845711",
      "postDate": "07/06/2022 14:27:44",
      "content": "<p>rapid cudf.  </p>\n<p>when I add use some dataset larger than the one used in forked notebook, It reports that Error.<br>\nthe dataset is about 5/6 GB, however during data processing, OOM。</p>",
      "rawMarkdown": "rapid cudf.  \n\nwhen I add use some dataset larger than the one used in forked notebook, It reports that Error.\nthe dataset is about 5/6 GB, however during data processing, OOM。",
      "votes": null
    },
    {
      "id": "1845713",
      "postDate": "07/06/2022 14:30:08",
      "content": "<p>thanks.  I forked. however, when I use bigger dataset, then OOM. </p>",
      "rawMarkdown": "thanks.  I forked. however, when I use bigger dataset, then OOM.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1844071,
      "author_name": "mirfanazam",
      "author_url": "",
      "post_date": "07/05/2022 10:23:59",
      "content": "<p><a href=\"https://www.kaggle.com/dragonzhang\" target=\"_blank\">@dragonzhang</a> Do you mean RAPIDS (rapids.ai)?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1845711,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "07/06/2022 14:27:44",
          "content": "<p>rapid cudf.  </p>\n<p>when I add use some dataset larger than the one used in forked notebook, It reports that Error.<br>\nthe dataset is about 5/6 GB, however during data processing, OOM。</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1844378,
      "author_name": "yamashitamotokazu",
      "author_url": "",
      "post_date": "07/05/2022 14:01:45",
      "content": "<p>This URL may be helpful for you:<br>\n<a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1844980,
          "author_name": "cdeotte",
          "author_url": "",
          "post_date": "07/06/2022 00:53:26",
          "content": "<p>Yes, the link above uses RAPIDS to read the dataset and convert it into a 3D NumPy array to train an RNN or Transformer. And the link <a href=\"https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793\" target=\"_blank\">here</a> uses RAPIDS to read the dataset (and leave as a 2D dataframe) to train GBT like XGB, LGBM, or CatBoost</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1845713,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "07/06/2022 14:30:08",
          "content": "<p>thanks.  I forked. however, when I use bigger dataset, then OOM. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1843912": "I got this error.\n\nMemoryError: std::bad_alloc: CUDA error at: /opt/conda/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory",
    "1844071": "dragonzhang Do you mean RAPIDS (rapids.ai)?",
    "1844378": "This URL may be helpful for you:\n[https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790](url)",
    "1844980": "Yes, the link above uses RAPIDS to read the dataset and convert it into a 3D NumPy array to train an RNN or Transformer. And the link [here][1] uses RAPIDS to read the dataset (and leave as a 2D dataframe) to train GBT like XGB, LGBM, or CatBoost\n\n[1]: https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793",
    "1845711": "rapid cudf.  \n\nwhen I add use some dataset larger than the one used in forked notebook, It reports that Error.\nthe dataset is about 5/6 GB, however during data processing, OOM。",
    "1845713": "thanks.  I forked. however, when I use bigger dataset, then OOM."
  },
  "source": "meta"
}