{
  "id": 202460,
  "title": "Training with the Full Dataset",
  "url": "/competitions/riiid-test-answer-prediction/discussion/202460",
  "author_name": "",
  "post_date": "2020-12-10T04:34:36.219269400Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>For those who are currently using the entire dataset to train your model, I was wondering what your setup is like, and how much of an improvement training with the full dataset gets over just 5-10M rows. I'm currently using 10M rows and it takes approx 14GB/25GB of RAM on Colab to train my model, is GCP the next step to take?</p>",
  "messages": [
    {
      "id": "1107910",
      "postDate": "12/10/2020 04:34:36",
      "content": "<p>For those who are currently using the entire dataset to train your model, I was wondering what your setup is like, and how much of an improvement training with the full dataset gets over just 5-10M rows. I'm currently using 10M rows and it takes approx 14GB/25GB of RAM on Colab to train my model, is GCP the next step to take?</p>",
      "rawMarkdown": "For those who are currently using the entire dataset to train your model, I was wondering what your setup is like, and how much of an improvement training with the full dataset gets over just 5-10M rows. I'm currently using 10M rows and it takes approx 14GB/25GB of RAM on Colab to train my model, is GCP the next step to take?",
      "votes": null
    },
    {
      "id": "1108018",
      "postDate": "12/10/2020 07:27:59",
      "content": "<p>I am training on 40M rows today and it is taking 8 to 10 gb of memory in colab with xgboost gpu.<br>\nDataframe is almost 2.5 Gb with 15 features</p>",
      "rawMarkdown": "I am training on 40M rows today and it is taking 8 to 10 gb of memory in colab with xgboost gpu.\nDataframe is almost 2.5 Gb with 15 features",
      "votes": null
    },
    {
      "id": "1108023",
      "postDate": "12/10/2020 07:33:38",
      "content": "<p>Repeated training with different features is taking now 12 to 13 gb.<br>\nSo i think almost 75 % data can be trained in colab with 25gb</p>",
      "rawMarkdown": "Repeated training with different features is taking now 12 to 13 gb.\nSo i think almost 75 % data can be trained in colab with 25gb",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1108018,
      "author_name": "ptrikp",
      "author_url": "",
      "post_date": "12/10/2020 07:27:59",
      "content": "<p>I am training on 40M rows today and it is taking 8 to 10 gb of memory in colab with xgboost gpu.<br>\nDataframe is almost 2.5 Gb with 15 features</p>",
      "votes": null,
      "replies": [
        {
          "id": 1108023,
          "author_name": "ptrikp",
          "author_url": "",
          "post_date": "12/10/2020 07:33:38",
          "content": "<p>Repeated training with different features is taking now 12 to 13 gb.<br>\nSo i think almost 75 % data can be trained in colab with 25gb</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1107910": "For those who are currently using the entire dataset to train your model, I was wondering what your setup is like, and how much of an improvement training with the full dataset gets over just 5-10M rows. I'm currently using 10M rows and it takes approx 14GB/25GB of RAM on Colab to train my model, is GCP the next step to take?",
    "1108018": "I am training on 40M rows today and it is taking 8 to 10 gb of memory in colab with xgboost gpu.\nDataframe is almost 2.5 Gb with 15 features",
    "1108023": "Repeated training with different features is taking now 12 to 13 gb.\nSo i think almost 75 % data can be trained in colab with 25gb"
  },
  "source": "meta"
}