{
  "id": 513519,
  "title": "Dataset is too large. How to load it?",
  "url": "/competitions/leash-BELKA/discussion/513519",
  "author_name": "",
  "post_date": "2024-06-20T12:33:46.214844600Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Dataset is too large. How to load it?</p>",
  "messages": [
    {
      "id": "2880881",
      "postDate": "06/20/2024 12:33:46",
      "content": "<p>Dataset is too large. How to load it?</p>",
      "rawMarkdown": "Dataset is too large. How to load it?",
      "votes": null
    },
    {
      "id": "2881687",
      "postDate": "06/20/2024 22:54:52",
      "content": "<p>Hi Nitishi,<br>\nCheck this Notebook:</p>\n<p><a href=\"https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest/notebook\" target=\"_blank\">Leash Tutorial - ECFPs and Random Forest</a>  By Andrew D. Blevins</p>",
      "rawMarkdown": "Hi Nitishi,\nCheck this Notebook:\n\n[Leash Tutorial - ECFPs and Random Forest](https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest/notebook)  By Andrew D. Blevins",
      "votes": null
    },
    {
      "id": "2899841",
      "postDate": "07/01/2024 22:06:29",
      "content": "<p>This tutorial has you use a duckdb connection, but I am still unable to load the entire dataset in this way, is there a good way to load the whole dataset?</p>",
      "rawMarkdown": "This tutorial has you use a duckdb connection, but I am still unable to load the entire dataset in this way, is there a good way to load the whole dataset?",
      "votes": null
    },
    {
      "id": "2899917",
      "postDate": "07/01/2024 23:27:03",
      "content": "<p>Hi  MarvinMeng,<br>\nUnfortunately, I don't know. Maybe you could ask to Andrew D. Blevins  (Competition host).</p>",
      "rawMarkdown": "Hi  MarvinMeng,\nUnfortunately, I don't know. Maybe you could ask to Andrew D. Blevins  (Competition host).",
      "votes": null
    },
    {
      "id": "2900769",
      "postDate": "07/02/2024 13:21:35",
      "content": "<p>yeah so youll need to set up a custom streaming iterator, heres a <a href=\"https://medium.com/@icarusabiding/neurips-2024-belka-challenge-innovating-small-molecule-binding-prediction-593aefc7a310\" target=\"_blank\">guide</a> on this exact subject, basically your expected to populate a db with it, have enough ram to load it, or figure out how to stream it.</p>",
      "rawMarkdown": "yeah so youll need to set up a custom streaming iterator, heres a [guide](https://medium.com/@icarusabiding/neurips-2024-belka-challenge-innovating-small-molecule-binding-prediction-593aefc7a310) on this exact subject, basically your expected to populate a db with it, have enough ram to load it, or figure out how to stream it.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2881687,
      "author_name": "mpwolke",
      "author_url": "",
      "post_date": "06/20/2024 22:54:52",
      "content": "<p>Hi Nitishi,<br>\nCheck this Notebook:</p>\n<p><a href=\"https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest/notebook\" target=\"_blank\">Leash Tutorial - ECFPs and Random Forest</a>  By Andrew D. Blevins</p>",
      "votes": null,
      "replies": [
        {
          "id": 2899841,
          "author_name": "marvinmeng",
          "author_url": "",
          "post_date": "07/01/2024 22:06:29",
          "content": "<p>This tutorial has you use a duckdb connection, but I am still unable to load the entire dataset in this way, is there a good way to load the whole dataset?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2899917,
              "author_name": "mpwolke",
              "author_url": "",
              "post_date": "07/01/2024 23:27:03",
              "content": "<p>Hi  MarvinMeng,<br>\nUnfortunately, I don't know. Maybe you could ask to Andrew D. Blevins  (Competition host).</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2900769,
      "author_name": "mbanuna",
      "author_url": "",
      "post_date": "07/02/2024 13:21:35",
      "content": "<p>yeah so youll need to set up a custom streaming iterator, heres a <a href=\"https://medium.com/@icarusabiding/neurips-2024-belka-challenge-innovating-small-molecule-binding-prediction-593aefc7a310\" target=\"_blank\">guide</a> on this exact subject, basically your expected to populate a db with it, have enough ram to load it, or figure out how to stream it.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2880881": "Dataset is too large. How to load it?",
    "2881687": "Hi Nitishi,\nCheck this Notebook:\n\n[Leash Tutorial - ECFPs and Random Forest](https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest/notebook)  By Andrew D. Blevins",
    "2899841": "This tutorial has you use a duckdb connection, but I am still unable to load the entire dataset in this way, is there a good way to load the whole dataset?",
    "2899917": "Hi  MarvinMeng,\nUnfortunately, I don't know. Maybe you could ask to Andrew D. Blevins  (Competition host).",
    "2900769": "yeah so youll need to set up a custom streaming iterator, heres a [guide](https://medium.com/@icarusabiding/neurips-2024-belka-challenge-innovating-small-molecule-binding-prediction-593aefc7a310) on this exact subject, basically your expected to populate a db with it, have enough ram to load it, or figure out how to stream it."
  },
  "source": "meta"
}