Hi,
\n\nI have been using keras for this competition.my submission scored 1.13 on leader board. Right now, am trying to increase the dimension of the image(from 32*32 to 224*224) .but i can't fit this to memory.Please suggest some good idea to have this loaded in RAM (at a stretch or as a generator).
\n\nMy Laptop RAM - 8GB
\n\nNJ/-
", "messages": [ { "id": "118349", "postDate": "05/03/2016 11:34:49", "content": "Hi,
\n\nI have been using keras for this competition.my submission scored 1.13 on leader board. Right now, am trying to increase the dimension of the image(from 32*32 to 224*224) .but i can't fit this to memory.Please suggest some good idea to have this loaded in RAM (at a stretch or as a generator).
\n\nMy Laptop RAM - 8GB
\n\nNJ/-
", "rawMarkdown": "Hi,\r\n\r\nI have been using keras for this competition.my submission scored 1.13 on leader board. Right now, am trying to increase the dimension of the image(from 32*32 to 224*224) .but i can't fit this to memory.Please suggest some good idea to have this loaded in RAM (at a stretch or as a generator).\r\n\r\n\r\n\r\nMy Laptop RAM - 8GB\r\n\r\n\r\nNJ/-", "votes": null }, { "id": "118692", "postDate": "05/04/2016 19:16:53", "content": "You can use AWS but you have to pay to use the service.
", "rawMarkdown": "You can use AWS but you have to pay to use the service.", "votes": null }, { "id": "118731", "postDate": "05/05/2016 01:16:29", "content": "Maybe you can run a manual train.\nAnd every time you train a batch, then you load the batch data to ram.
", "rawMarkdown": "Maybe you can run a manual train.\r\nAnd every time you train a batch, then you load the batch data to ram.", "votes": null }, { "id": "118754", "postDate": "05/05/2016 06:22:25", "content": "Check what data type your array is using. Maybe you can more efficiently store the information without loss of integrity.
", "rawMarkdown": "Check what data type your array is using. Maybe you can more efficiently store the information without loss of integrity.", "votes": null }, { "id": "118783", "postDate": "05/05/2016 09:29:28", "content": "You don't have to load all the dataset to memory, you could load only the batch you are going to feed through the network.
", "rawMarkdown": "You don't have to load all the dataset to memory, you could load only the batch you are going to feed through the network.", "votes": null }, { "id": "119096", "postDate": "05/07/2016 06:55:34", "content": "[quote=NJ/-;118349]
\n\nHi,
\n\nI have been using keras for this competition.my submission scored 1.13 on leader board. Right now, am trying to increase the dimension of the image(from 32*32 to 224*224) .but i can't fit this to memory.Please suggest some good idea to have this loaded in RAM (at a stretch or as a generator).
\n\nMy Laptop RAM - 8GB
\n\nNJ/-
\n\n[/quote]\nAny success with this? I am facing the same problem. My starter script is that shared by ZFTurbo. What about you?
", "rawMarkdown": "[quote=NJ/-;118349]\r\n\r\nHi,\r\n\r\nI have been using keras for this competition.my submission scored 1.13 on leader board. Right now, am trying to increase the dimension of the image(from 32*32 to 224*224) .but i can't fit this to memory.Please suggest some good idea to have this loaded in RAM (at a stretch or as a generator).\r\n\r\n\r\n\r\nMy Laptop RAM - 8GB\r\n\r\n\r\nNJ/-\r\n\r\n[/quote]\r\nAny success with this? I am facing the same problem. My starter script is that shared by ZFTurbo. What about you?", "votes": null }, { "id": "119137", "postDate": "05/07/2016 14:43:26", "content": "@Abhijay
\n\nI'm Trying with generator now.
", "rawMarkdown": "Abhijay\r\n\r\nI'm Trying with generator now.", "votes": null }, { "id": "119147", "postDate": "05/07/2016 16:37:01", "content": "You can use memory mapped files to load and work the training data. It's a bit like a swap file; memory mapped files create a window and a virtual address space larger than RAM. When data is accessed that's not loaded into the window, the kernel swaps in the required pages from the file so you can work on that piece of data. So with this method, random access can be a bit slow, sequential access should be slightly better.
\n\nWith this method, I can access a 48gb file (test data) using only 24gb of RAM without running out of memory, because the kernel unloads and reloads data for me automatically.
\n\nHere's how to do this in python:
\n\nAs I'm using numpy to represent my arrays/matrix, I create a file of the correct size first. Notice how it's opened write/append:
\n\ntrain_data = np.memmap(ROOT_DIR+"train.driver", dtype='float32', mode='w+', shape=(TRAIN_ROWS, IMG_SIZE))\nThen you can assign items row by row or any way you want. Eventually the numpy array is flushed and written to disk:
\n\ntrain_data[tidx,:] = <my_numpy_array_structure_also_of_float32>\nFlush and close the file at the end:
\n\ntrain_data.flush()\ntrain_data.close()\nOpen it when you're trying to access data, again through a memmap, but this time in read mode:
\n\ntrain = np.memmap(trainfilename, dtype='float32', mode='r', shape=(TRAIN_ROWS, IMG_SIZE))\nCreate an index to access the array and shuffle the indices to get random data:
\n\ntrain_idxs = [i for i in range(train.shape[0])]\nshuffle(train_idxs)\nIt works with ZFTurbo's code, but you can write your own method to access train_data in a batched way with tensorflow:
\n\ndef next_batch(start,train,labels,batch_size=250):\n newstart = start+batch_size\n if newstart > train.shape[0]:\n newstart = 0\n idxs = train_idxs[start:start+batch_size]\n return train[idxs,:], labels[idxs,:], newstart\nObviously with random access and a large matrix there will be a delay accessing memory because the kernel needs to move the window across your data. The larger you specify your window, the quicker this can be done.
", "rawMarkdown": "You can use memory mapped files to load and work the training data. It's a bit like a swap file; memory mapped files create a window and a virtual address space larger than RAM. When data is accessed that's not loaded into the window, the kernel swaps in the required pages from the file so you can work on that piece of data. So with this method, random access can be a bit slow, sequential access should be slightly better. \r\n\r\nWith this method, I can access a 48gb file (test data) using only 24gb of RAM without running out of memory, because the kernel unloads and reloads data for me automatically.\r\n\r\nHere's how to do this in python:\r\n\r\nAs I'm using numpy to represent my arrays/matrix, I create a file of the correct size first. Notice how it's opened write/append:\r\n\r\n train_data = np.memmap(ROOT_DIR+\"train.driver\", dtype='float32', mode='w+', shape=(TRAIN_ROWS, IMG_SIZE))\r\n\r\nThen you can assign items row by row or any way you want. Eventually the numpy array is flushed and written to disk:\r\n\r\n train_data[tidx,:] =I've been preparing a keras tutorial for the MNIST dataset which is aimed at folks attempting the Painter By Numbers competition - but perhaps those of you working on the Distracted Driver competition might also find it helpful. The network architecture is a little different from this competition because Painter By Numbers needs to examine two images at the same time using a siamese network. However, the process of using a generator to load chunks of the training set is the same.
\n\nCode is at https://github.com/small-yellow-duck/kaggle_art/blob/master/mnist_siamese_generator2.py:
", "rawMarkdown": "I've been preparing a keras tutorial for the MNIST dataset which is aimed at folks attempting the [Painter By Numbers][1] competition - but perhaps those of you working on the Distracted Driver competition might also find it helpful. The network architecture is a little different from this competition because Painter By Numbers needs to examine two images at the same time using a siamese network. However, the process of using a generator to load chunks of the training set is the same.\r\n\r\nCode is at [https://github.com/small-yellow-duck/kaggle_art/blob/master/mnist_siamese_generator2.py][2]: \r\n\r\n\r\n [1]: https://www.kaggle.com/c/painter-by-numbers\r\n [2]: http://%20https://github.com/small-yellow-duck/kaggle_art/blob/master/mnist_siamese_generator2.py", "votes": null }, { "id": "121895", "postDate": "05/30/2016 19:21:02", "content": "@Gerard Toonstra would you be willing to post more of your code? I can't get it running.
", "rawMarkdown": "Gerard Toonstra would you be willing to post more of your code? I can't get it running.", "votes": null }, { "id": "121900", "postDate": "05/30/2016 19:43:10", "content": "Evan,
\n\nStart with a toy example. Create an array of increasing numbers, then assign that to the memory mapped file. Open the memory mapped file with a hex editor and see if it's all there. Then attempt to read lines from it when you open it from a different script.
\n\nOnce you have that going, apply it to the problem at hand.
\n\nimport numpy as np\n\nTOY_ROWS=5\nTOY_SIZE=5\ntrain_data = np.memmap("toy.example", dtype='float32', mode='w+', shape=(TOY_ROWS, TOY_SIZE))\nfor x in xrange(TOY_ROWS):\n train_data[x,:] = np.arange(x*TOY_SIZE, (x+1)*TOY_SIZE)\ntrain_data.flush()\nI didn't actually run that code, so there may be minor bugs, but it should produce an example file locally. Then read this out as:
\n\nimport numpy as np\n\ntrain = np.memmap("toy.example", dtype='float32', mode='r', shape=(TOY_ROWS, TOY_SIZE))\nprint(train.shape)\nfor x in xrange(TOY_ROWS):\n print(train[x,:])\nthanks Gerard. I'll see if I can get it to work.
", "rawMarkdown": "thanks Gerard. I'll see if I can get it to work.", "votes": null }, { "id": "121941", "postDate": "05/31/2016 05:40:02", "content": "Another solution which I find much more convenient than memmaps is hdf5 with h5py.
\n\nKeras can run directly on h5py files, eliminating the need to manually specify the training on each batch.
\n\nf = h5py.File('cache.hdf5', 'w')\n\ndataset = f.create_dataset('cacheName', (nImgs, nChannels, imgHeight, imgWidth), dtype='float32')\n\nfor i, img in enumerate(images):\n # fill the thing\n dataset[i] = img\n\n#read it back\ndataset = h5py.File('cache.hdf5', 'r')\ndataset = dataset.get('cacheName')\nYou can use AWS but you have to pay to use the service.
", "votes": null, "replies": [] }, { "id": 118731, "author_name": "yelrose", "author_url": "", "post_date": "05/05/2016 01:16:29", "content": "Maybe you can run a manual train.\nAnd every time you train a batch, then you load the batch data to ram.
", "votes": null, "replies": [] }, { "id": 118754, "author_name": "austin6", "author_url": "", "post_date": "05/05/2016 06:22:25", "content": "Check what data type your array is using. Maybe you can more efficiently store the information without loss of integrity.
", "votes": null, "replies": [] }, { "id": 118783, "author_name": "galloguille", "author_url": "", "post_date": "05/05/2016 09:29:28", "content": "You don't have to load all the dataset to memory, you could load only the batch you are going to feed through the network.
", "votes": null, "replies": [] }, { "id": 119096, "author_name": "abhijayvuyyuru", "author_url": "", "post_date": "05/07/2016 06:55:34", "content": "[quote=NJ/-;118349]
\n\nHi,
\n\nI have been using keras for this competition.my submission scored 1.13 on leader board. Right now, am trying to increase the dimension of the image(from 32*32 to 224*224) .but i can't fit this to memory.Please suggest some good idea to have this loaded in RAM (at a stretch or as a generator).
\n\nMy Laptop RAM - 8GB
\n\nNJ/-
\n\n[/quote]\nAny success with this? I am facing the same problem. My starter script is that shared by ZFTurbo. What about you?
", "votes": null, "replies": [] }, { "id": 119137, "author_name": "nithinjames", "author_url": "", "post_date": "05/07/2016 14:43:26", "content": "@Abhijay
\n\nI'm Trying with generator now.
", "votes": null, "replies": [] }, { "id": 119147, "author_name": "remap1", "author_url": "", "post_date": "05/07/2016 16:37:01", "content": "You can use memory mapped files to load and work the training data. It's a bit like a swap file; memory mapped files create a window and a virtual address space larger than RAM. When data is accessed that's not loaded into the window, the kernel swaps in the required pages from the file so you can work on that piece of data. So with this method, random access can be a bit slow, sequential access should be slightly better.
\n\nWith this method, I can access a 48gb file (test data) using only 24gb of RAM without running out of memory, because the kernel unloads and reloads data for me automatically.
\n\nHere's how to do this in python:
\n\nAs I'm using numpy to represent my arrays/matrix, I create a file of the correct size first. Notice how it's opened write/append:
\n\ntrain_data = np.memmap(ROOT_DIR+"train.driver", dtype='float32', mode='w+', shape=(TRAIN_ROWS, IMG_SIZE))\nThen you can assign items row by row or any way you want. Eventually the numpy array is flushed and written to disk:
\n\ntrain_data[tidx,:] = <my_numpy_array_structure_also_of_float32>\nFlush and close the file at the end:
\n\ntrain_data.flush()\ntrain_data.close()\nOpen it when you're trying to access data, again through a memmap, but this time in read mode:
\n\ntrain = np.memmap(trainfilename, dtype='float32', mode='r', shape=(TRAIN_ROWS, IMG_SIZE))\nCreate an index to access the array and shuffle the indices to get random data:
\n\ntrain_idxs = [i for i in range(train.shape[0])]\nshuffle(train_idxs)\nIt works with ZFTurbo's code, but you can write your own method to access train_data in a batched way with tensorflow:
\n\ndef next_batch(start,train,labels,batch_size=250):\n newstart = start+batch_size\n if newstart > train.shape[0]:\n newstart = 0\n idxs = train_idxs[start:start+batch_size]\n return train[idxs,:], labels[idxs,:], newstart\nObviously with random access and a large matrix there will be a delay accessing memory because the kernel needs to move the window across your data. The larger you specify your window, the quicker this can be done.
", "votes": null, "replies": [] }, { "id": 119195, "author_name": "smallyellowduck", "author_url": "", "post_date": "05/07/2016 23:32:50", "content": "I've been preparing a keras tutorial for the MNIST dataset which is aimed at folks attempting the Painter By Numbers competition - but perhaps those of you working on the Distracted Driver competition might also find it helpful. The network architecture is a little different from this competition because Painter By Numbers needs to examine two images at the same time using a siamese network. However, the process of using a generator to load chunks of the training set is the same.
\n\nCode is at https://github.com/small-yellow-duck/kaggle_art/blob/master/mnist_siamese_generator2.py:
", "votes": null, "replies": [] }, { "id": 121895, "author_name": "evanvanness", "author_url": "", "post_date": "05/30/2016 19:21:02", "content": "@Gerard Toonstra would you be willing to post more of your code? I can't get it running.
", "votes": null, "replies": [] }, { "id": 121900, "author_name": "remap1", "author_url": "", "post_date": "05/30/2016 19:43:10", "content": "Evan,
\n\nStart with a toy example. Create an array of increasing numbers, then assign that to the memory mapped file. Open the memory mapped file with a hex editor and see if it's all there. Then attempt to read lines from it when you open it from a different script.
\n\nOnce you have that going, apply it to the problem at hand.
\n\nimport numpy as np\n\nTOY_ROWS=5\nTOY_SIZE=5\ntrain_data = np.memmap("toy.example", dtype='float32', mode='w+', shape=(TOY_ROWS, TOY_SIZE))\nfor x in xrange(TOY_ROWS):\n train_data[x,:] = np.arange(x*TOY_SIZE, (x+1)*TOY_SIZE)\ntrain_data.flush()\nI didn't actually run that code, so there may be minor bugs, but it should produce an example file locally. Then read this out as:
\n\nimport numpy as np\n\ntrain = np.memmap("toy.example", dtype='float32', mode='r', shape=(TOY_ROWS, TOY_SIZE))\nprint(train.shape)\nfor x in xrange(TOY_ROWS):\n print(train[x,:])\nthanks Gerard. I'll see if I can get it to work.
", "votes": null, "replies": [] }, { "id": 121941, "author_name": "gpistre", "author_url": "", "post_date": "05/31/2016 05:40:02", "content": "Another solution which I find much more convenient than memmaps is hdf5 with h5py.
\n\nKeras can run directly on h5py files, eliminating the need to manually specify the training on each batch.
\n\nf = h5py.File('cache.hdf5', 'w')\n\ndataset = f.create_dataset('cacheName', (nImgs, nChannels, imgHeight, imgWidth), dtype='float32')\n\nfor i, img in enumerate(images):\n # fill the thing\n dataset[i] = img\n\n#read it back\ndataset = h5py.File('cache.hdf5', 'r')\ndataset = dataset.get('cacheName')\n