Hello Kagglers,
\nThis is my first time competing and this may be a silly question. I was trying to run stratified k-fold cross-validation on train.csv and it gives me a MemoryError. Specifically, MemoryError: Unable to allocate 6.03 GiB for an array with shape (8, 101230332) and data type int64.
\nMy specs:
\n16 GB RAM
\nGTX 1060
\nCore i7
I do not think it is much of a hardware issue. I ran the fold code on Spyder. Any help is appreciated. Thank you very much.
", "messages": [ { "id": "1047346", "postDate": "10/12/2020 13:52:35", "content": "Hello Kagglers,
\nThis is my first time competing and this may be a silly question. I was trying to run stratified k-fold cross-validation on train.csv and it gives me a MemoryError. Specifically, MemoryError: Unable to allocate 6.03 GiB for an array with shape (8, 101230332) and data type int64.
\nMy specs:
\n16 GB RAM
\nGTX 1060
\nCore i7
I do not think it is much of a hardware issue. I ran the fold code on Spyder. Any help is appreciated. Thank you very much.
", "rawMarkdown": "Hello Kagglers,\n\nThis is my first time competing and this may be a silly question. I was trying to run stratified k-fold cross-validation on train.csv and it gives me a MemoryError. Specifically, MemoryError: Unable to allocate 6.03 GiB for an array with shape (8, 101230332) and data type int64. \n\nMy specs:\n16 GB RAM\nGTX 1060\nCore i7 \n\nI do not think it is much of a hardware issue. I ran the fold code on Spyder. Any help is appreciated. Thank you very much.", "votes": null }, { "id": "1047379", "postDate": "10/12/2020 14:34:58", "content": "It is a hardware issue.
\nEven kaggle kernels also have 16GB of RAM and by loading this much size of any arrays will have memory errors
\"MemoryError: Unable to allocate 6.03 GiB for an array with shape (8, 101230332)\" - This error usually arises when you initialize a numpy array (by doing something like np.zeros((8, 101230332)) ), however such an array cant be fit on the memory, so this step fails. More details - https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type
", "rawMarkdown": "\"MemoryError: Unable to allocate 6.03 GiB for an array with shape (8, 101230332)\" - This error usually arises when you initialize a numpy array (by doing something like np.zeros((8, 101230332)) ), however such an array cant be fit on the memory, so this step fails. More details - https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type", "votes": null }, { "id": "1047943", "postDate": "10/13/2020 04:36:23", "content": "I have also noticed that the program first tries to allocate a much larger memory, may be while making copies of the original dataset i.e. memory goes from 6GB to 20GB and then back down to 14GB. So, even though actual requirement is less than 16GB, training code throws an error at the intermediate stage where a larger memory is allocated.
\nEven though train data size is about 7.5GB, we were not able to load it in kaggle kernel, which is the primary reason for several previous discussions on using Dask, cuDF, etc. Also see pandas solution and kernel using chunksize.
\nIt is a hardware issue.
\nEven kaggle kernels also have 16GB of RAM and by loading this much size of any arrays will have memory errors
\"MemoryError: Unable to allocate 6.03 GiB for an array with shape (8, 101230332)\" - This error usually arises when you initialize a numpy array (by doing something like np.zeros((8, 101230332)) ), however such an array cant be fit on the memory, so this step fails. More details - https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type
", "votes": null, "replies": [] }, { "id": 1047943, "author_name": "sirishks", "author_url": "", "post_date": "10/13/2020 04:36:23", "content": "I have also noticed that the program first tries to allocate a much larger memory, may be while making copies of the original dataset i.e. memory goes from 6GB to 20GB and then back down to 14GB. So, even though actual requirement is less than 16GB, training code throws an error at the intermediate stage where a larger memory is allocated.
\nEven though train data size is about 7.5GB, we were not able to load it in kaggle kernel, which is the primary reason for several previous discussions on using Dask, cuDF, etc. Also see pandas solution and kernel using chunksize.
\n