{
  "id": 553229,
  "title": "incremental learning",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/553229",
  "author_name": "zhongkaiqi",
  "post_date": "2024-12-24T14:24:57.045000",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Has anyone considered using incremental learning to address the issue of insufficient memory during Kaggle model training? When training on the entire dataset at once, memory often becomes a bottleneck. I haven't tried it yet, so I'm not sure if it would be effective. So far, I haven't seen anyone mention this approach either.</p>",
  "messages": [
    {
      "id": 3080030,
      "postDate": "2024-12-24T14:24:57.047Z",
      "content": "<p>Has anyone considered using incremental learning to address the issue of insufficient memory during Kaggle model training? When training on the entire dataset at once, memory often becomes a bottleneck. I haven't tried it yet, so I'm not sure if it would be effective. So far, I haven't seen anyone mention this approach either.</p>",
      "rawMarkdown": "Has anyone considered using incremental learning to address the issue of insufficient memory during Kaggle model training? When training on the entire dataset at once, memory often becomes a bottleneck. I haven't tried it yet, so I'm not sure if it would be effective. So far, I haven't seen anyone mention this approach either.",
      "votes": 1
    },
    {
      "id": 3080184,
      "postDate": "2024-12-24T20:25:28.913Z",
      "content": "<p>I believe dedicated participants are all using their own machine to run local training, where memory is not really a limit (128GB memory is quite sufficient). In this case, money is the easier solution comparing to any technical tricks. :(<br>\nAs for many participants who only copied the public notebook, I believe it is not their consideration to implement those advanced tricks neither.</p>",
      "rawMarkdown": "I believe dedicated participants are all using their own machine to run local training, where memory is not really a limit (128GB memory is quite sufficient). In this case, money is the easier solution comparing to any technical tricks. :(\nAs for many participants who only copied the public notebook, I believe it is not their consideration to implement those advanced tricks neither.",
      "votes": 2,
      "replies": [
        {
          "id": 3080476,
          "postDate": "2024-12-25T08:56:19.947Z",
          "content": "<p>Currently, I can only rent servers to perform operations, otherwise I frequently run into memory shortage issues. :(</p>",
          "rawMarkdown": "Currently, I can only rent servers to perform operations, otherwise I frequently run into memory shortage issues. :("
        }
      ]
    },
    {
      "id": 3080815,
      "postDate": "2024-12-25T19:13:40.357Z",
      "content": "<p>if you're training a neural net you can stream your data and avoid any memory issues</p>",
      "rawMarkdown": "if you're training a neural net you can stream your data and avoid any memory issues"
    },
    {
      "id": 3082489,
      "postDate": "2024-12-28T08:06:05.023Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 3080184,
      "author_name": "SLi",
      "author_url": "",
      "post_date": "2024-12-24T20:25:28.913000",
      "content": "<p>I believe dedicated participants are all using their own machine to run local training, where memory is not really a limit (128GB memory is quite sufficient). In this case, money is the easier solution comparing to any technical tricks. :(<br>\nAs for many participants who only copied the public notebook, I believe it is not their consideration to implement those advanced tricks neither.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 3080476,
          "author_name": "zhongkaiqi",
          "author_url": "",
          "post_date": "2024-12-25T08:56:19.947000",
          "content": "<p>Currently, I can only rent servers to perform operations, otherwise I frequently run into memory shortage issues. :(</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3080815,
      "author_name": "Lu Bin Liu",
      "author_url": "",
      "post_date": "2024-12-25T19:13:40.357000",
      "content": "<p>if you're training a neural net you can stream your data and avoid any memory issues</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3082489,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-12-28T08:06:05.023000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3080030": "Has anyone considered using incremental learning to address the issue of insufficient memory during Kaggle model training? When training on the entire dataset at once, memory often becomes a bottleneck. I haven't tried it yet, so I'm not sure if it would be effective. So far, I haven't seen anyone mention this approach either.",
    "3080184": "I believe dedicated participants are all using their own machine to run local training, where memory is not really a limit (128GB memory is quite sufficient). In this case, money is the easier solution comparing to any technical tricks. :(\nAs for many participants who only copied the public notebook, I believe it is not their consideration to implement those advanced tricks neither.",
    "3080815": "if you're training a neural net you can stream your data and avoid any memory issues",
    "3082489": ""
  }
}