{
  "id": 197619,
  "title": "how fast is your data pipeline for NN model",
  "url": "/competitions/riiid-test-answer-prediction/discussion/197619",
  "author_name": "YL",
  "post_date": "2020-11-17T09:46:18.410000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>At the moment I have a simple RNN based sequence model and I developed the pipeline around it. In order to minimize memory consumption as much as I can, I have some features generating on the fly (still reaching 30GB plus when all the datasets are loaded). With that I am only able to achieve on average 4 batches (batch of 64) per second. And this means anything will take forever 😭😭😭</p>\n<p>Just wondering how fast can your data pipeline reach, I.e. how long does your sequence model take to train?</p>",
  "messages": [
    {
      "id": 1081794,
      "postDate": "2020-11-17T09:46:18.410Z",
      "content": "<p>At the moment I have a simple RNN based sequence model and I developed the pipeline around it. In order to minimize memory consumption as much as I can, I have some features generating on the fly (still reaching 30GB plus when all the datasets are loaded). With that I am only able to achieve on average 4 batches (batch of 64) per second. And this means anything will take forever 😭😭😭</p>\n<p>Just wondering how fast can your data pipeline reach, I.e. how long does your sequence model take to train?</p>",
      "rawMarkdown": "At the moment I have a simple RNN based sequence model and I developed the pipeline around it. In order to minimize memory consumption as much as I can, I have some features generating on the fly (still reaching 30GB plus when all the datasets are loaded). With that I am only able to achieve on average 4 batches (batch of 64) per second. And this means anything will take forever 😭😭😭\n\nJust wondering how fast can your data pipeline reach, I.e. how long does your sequence model take to train?"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1081794": "At the moment I have a simple RNN based sequence model and I developed the pipeline around it. In order to minimize memory consumption as much as I can, I have some features generating on the fly (still reaching 30GB plus when all the datasets are loaded). With that I am only able to achieve on average 4 batches (batch of 64) per second. And this means anything will take forever 😭😭😭\n\nJust wondering how fast can your data pipeline reach, I.e. how long does your sequence model take to train?"
  }
}