The datasets provided to this project are simply huge in size, especially the training set. I want to try LSTM, but it would take too much memory to load the data -- my desktop computer was not able to even start the first epoch after a whole night of data loading. So I am thinking if it is a good idea to load part of the data to train the model, throw away the used data and load the next chunk of data into memory to train the model. So on and on until the data is all used chunk by chunk. Technically speaking, is this approach identical to loading all data into memory at once?
", "messages": [ { "id": "458438", "postDate": "01/19/2019 16:15:26", "content": "The datasets provided to this project are simply huge in size, especially the training set. I want to try LSTM, but it would take too much memory to load the data -- my desktop computer was not able to even start the first epoch after a whole night of data loading. So I am thinking if it is a good idea to load part of the data to train the model, throw away the used data and load the next chunk of data into memory to train the model. So on and on until the data is all used chunk by chunk. Technically speaking, is this approach identical to loading all data into memory at once?
", "rawMarkdown": "The datasets provided to this project are simply huge in size, especially the training set. I want to try LSTM, but it would take too much memory to load the data -- my desktop computer was not able to even start the first epoch after a whole night of data loading. So I am thinking if it is a good idea to load part of the data to train the model, throw away the used data and load the next chunk of data into memory to train the model. So on and on until the data is all used chunk by chunk. Technically speaking, is this approach identical to loading all data into memory at once?", "votes": null }, { "id": "458500", "postDate": "01/19/2019 19:45:47", "content": "Hi! Mousheng, you can build a Generator that can feed subsequences of the dataset, without consuming all your memory there are a few Kernels with this strategy below one of them, also you can work with tf.data:\nKernel RNN\nhttps://www.kaggle.com/mayer79/rnn-starter\ntf.data\nhttps://www.tensorflow.org/guide/datasets
", "rawMarkdown": "Hi! Mousheng, you can build a Generator that can feed subsequences of the dataset, without consuming all your memory there are a few Kernels with this strategy below one of them, also you can work with tf.data:\nKernel RNN\nhttps://www.kaggle.com/mayer79/rnn-starter\ntf.data\nhttps://www.tensorflow.org/guide/datasets", "votes": null }, { "id": "458565", "postDate": "01/19/2019 23:39:26", "content": "One can try to obtain a spectrum first, e.g. STFT, MFC, MFCC ...\nFrom there, do a CNN or RNN. That would be feasible. \nAnd also that's the standard way to model the similar tasks using neural nets
\n\nI haven't tried neural nets on this competition yet. \nWould love to know it's performance if someone tests them out.
", "rawMarkdown": "One can try to obtain a spectrum first, e.g. STFT, MFC, MFCC ...\nFrom there, do a CNN or RNN. That would be feasible. \nAnd also that's the standard way to model the similar tasks using neural nets\n\nI haven't tried neural nets on this competition yet. \nWould love to know it's performance if someone tests them out.", "votes": null }, { "id": "458673", "postDate": "01/20/2019 08:42:33", "content": "It seems to all of the problems with big-data, the neural networks still can not import all of the data by one-time. Shuffling the data into different part could be a way to solve this problem. Acutally i am curiosity about what kind of LSTM architecture you are preparing use in this problem.
", "rawMarkdown": "It seems to all of the problems with big-data, the neural networks still can not import all of the data by one-time. Shuffling the data into different part could be a way to solve this problem. Acutally i am curiosity about what kind of LSTM architecture you are preparing use in this problem.", "votes": null }, { "id": "459036", "postDate": "01/21/2019 04:09:14", "content": "Haven't started yet, but thinking about the very basic one.
", "rawMarkdown": "Haven't started yet, but thinking about the very basic one.", "votes": null }, { "id": "459037", "postDate": "01/21/2019 04:12:02", "content": "Thanks a lot. Probably I will come back with questions. :)
", "rawMarkdown": "Thanks a lot. Probably I will come back with questions. :)", "votes": null }, { "id": "504542", "postDate": "03/31/2019 19:47:49", "content": "You can use cudnnLstm.
", "rawMarkdown": "You can use cudnnLstm.", "votes": null } ], "comments": [ { "id": 458500, "author_name": "cv13j0", "author_url": "", "post_date": "01/19/2019 19:45:47", "content": "Hi! Mousheng, you can build a Generator that can feed subsequences of the dataset, without consuming all your memory there are a few Kernels with this strategy below one of them, also you can work with tf.data:\nKernel RNN\nhttps://www.kaggle.com/mayer79/rnn-starter\ntf.data\nhttps://www.tensorflow.org/guide/datasets
", "votes": null, "replies": [ { "id": 459037, "author_name": "moushengxu", "author_url": "", "post_date": "01/21/2019 04:12:02", "content": "Thanks a lot. Probably I will come back with questions. :)
", "votes": null, "replies": [] } ] }, { "id": 458565, "author_name": "tclf90", "author_url": "", "post_date": "01/19/2019 23:39:26", "content": "One can try to obtain a spectrum first, e.g. STFT, MFC, MFCC ...\nFrom there, do a CNN or RNN. That would be feasible. \nAnd also that's the standard way to model the similar tasks using neural nets
\n\nI haven't tried neural nets on this competition yet. \nWould love to know it's performance if someone tests them out.
", "votes": null, "replies": [] }, { "id": 458673, "author_name": "amdur757", "author_url": "", "post_date": "01/20/2019 08:42:33", "content": "It seems to all of the problems with big-data, the neural networks still can not import all of the data by one-time. Shuffling the data into different part could be a way to solve this problem. Acutally i am curiosity about what kind of LSTM architecture you are preparing use in this problem.
", "votes": null, "replies": [ { "id": 459036, "author_name": "moushengxu", "author_url": "", "post_date": "01/21/2019 04:09:14", "content": "Haven't started yet, but thinking about the very basic one.
", "votes": null, "replies": [] } ] }, { "id": 504542, "author_name": "alexus1000", "author_url": "", "post_date": "03/31/2019 19:47:49", "content": "You can use cudnnLstm.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "458438": "The datasets provided to this project are simply huge in size, especially the training set. I want to try LSTM, but it would take too much memory to load the data -- my desktop computer was not able to even start the first epoch after a whole night of data loading. So I am thinking if it is a good idea to load part of the data to train the model, throw away the used data and load the next chunk of data into memory to train the model. So on and on until the data is all used chunk by chunk. Technically speaking, is this approach identical to loading all data into memory at once?", "458500": "Hi! Mousheng, you can build a Generator that can feed subsequences of the dataset, without consuming all your memory there are a few Kernels with this strategy below one of them, also you can work with tf.data:\nKernel RNN\nhttps://www.kaggle.com/mayer79/rnn-starter\ntf.data\nhttps://www.tensorflow.org/guide/datasets", "458565": "One can try to obtain a spectrum first, e.g. STFT, MFC, MFCC ...\nFrom there, do a CNN or RNN. That would be feasible. \nAnd also that's the standard way to model the similar tasks using neural nets\n\nI haven't tried neural nets on this competition yet. \nWould love to know it's performance if someone tests them out.", "458673": "It seems to all of the problems with big-data, the neural networks still can not import all of the data by one-time. Shuffling the data into different part could be a way to solve this problem. Acutally i am curiosity about what kind of LSTM architecture you are preparing use in this problem.", "459036": "Haven't started yet, but thinking about the very basic one.", "459037": "Thanks a lot. Probably I will come back with questions. :)", "504542": "You can use cudnnLstm." }, "source": "meta" }