{ "id": 167086, "title": "Ram got crashed. ", "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/167086", "author_name": "SAURABH THAKUR", "post_date": "2020-07-15T07:06:02.770000", "votes": 1, "comment_count": 5, "views": 0, "content": "

My Ram get crashed while converting image into np.array. How do i preprocess my image without running out of memory.

\n\n

Please suggest alternative way.

", "messages": [ { "id": 930068, "postDate": "2020-07-15T07:06:02.770Z", "content": "

My Ram get crashed while converting image into np.array. How do i preprocess my image without running out of memory.

\n\n

Please suggest alternative way.

", "rawMarkdown": "My Ram get crashed while converting image into np.array. How do i preprocess my image without running out of memory.\n\nPlease suggest alternative way.", "votes": 1 }, { "id": 930851, "postDate": "2020-07-15T18:45:34.480Z", "content": "

I'm having a similar issue, and what I've found so far isn't very helpful: using the tracemalloc library, I took snapshots of my memory allocations at different points in my code. The biggest jump was at an unsurprising place, but here's the rub: my draft session statistics say I'm using about 7.6 times as much RAM as the number I get from summing all my current memory allocations' sizes using *tracemalloc.take_snapshot()*. What am I to make of this? How do I resolve this issue?\n\"\"

", "rawMarkdown": "I'm having a similar issue, and what I've found so far isn't very helpful: using the *tracemalloc* library, I took snapshots of my memory allocations at different points in my code. The biggest jump was at an unsurprising place, but here's the rub: my draft session statistics say I'm using about 7.6 times as much RAM as the number I get from summing all my current memory allocations' sizes using *tracemalloc.take_snapshot()*. What am I to make of this? How do I resolve this issue?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3503700%2F2492bed192222f6d5143f9a3f0216be2%2FScreenshot%20from%202020-07-15%2014-34-29.png?generation=1594838649442895&alt=media)\n" }, { "id": 930333, "postDate": "2020-07-15T11:18:19.433Z", "content": "

numpy append makes a copy of the data each time you add a row. If that is what you are doing, look at other methods.

", "rawMarkdown": "numpy append makes a copy of the data each time you add a row. If that is what you are doing, look at other methods." }, { "id": 930103, "postDate": "2020-07-15T07:42:09.290Z", "content": "

You can try to load the images in smaller batches.

", "rawMarkdown": "You can try to load the images in smaller batches.", "replies": [ { "id": 930197, "postDate": "2020-07-15T09:08:26.503Z", "content": "

i tried with batchsize =16, still ram gets out of memory

", "rawMarkdown": "i tried with batchsize =16, still ram gets out of memory" } ] }, { "id": 930343, "postDate": "2020-07-15T11:32:06.333Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 930851, "author_name": "Mathieu Beaudoin", "author_url": "", "post_date": "2020-07-15T18:45:34.480000", "content": "

I'm having a similar issue, and what I've found so far isn't very helpful: using the tracemalloc library, I took snapshots of my memory allocations at different points in my code. The biggest jump was at an unsurprising place, but here's the rub: my draft session statistics say I'm using about 7.6 times as much RAM as the number I get from summing all my current memory allocations' sizes using *tracemalloc.take_snapshot()*. What am I to make of this? How do I resolve this issue?\n\"\"

", "votes": 0, "replies": [] }, { "id": 930333, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-07-15T11:18:19.433000", "content": "

numpy append makes a copy of the data each time you add a row. If that is what you are doing, look at other methods.

", "votes": 0, "replies": [] }, { "id": 930103, "author_name": "Der Informatiker", "author_url": "", "post_date": "2020-07-15T07:42:09.290000", "content": "

You can try to load the images in smaller batches.

", "votes": 0, "replies": [ { "id": 930197, "author_name": "SAURABH THAKUR", "author_url": "", "post_date": "2020-07-15T09:08:26.503000", "content": "

i tried with batchsize =16, still ram gets out of memory

", "votes": 0, "replies": [] } ] }, { "id": 930343, "author_name": "", "author_url": "", "post_date": "2020-07-15T11:32:06.333000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "930068": "My Ram get crashed while converting image into np.array. How do i preprocess my image without running out of memory.\n\nPlease suggest alternative way.", "930851": "I'm having a similar issue, and what I've found so far isn't very helpful: using the *tracemalloc* library, I took snapshots of my memory allocations at different points in my code. The biggest jump was at an unsurprising place, but here's the rub: my draft session statistics say I'm using about 7.6 times as much RAM as the number I get from summing all my current memory allocations' sizes using *tracemalloc.take_snapshot()*. What am I to make of this? How do I resolve this issue?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3503700%2F2492bed192222f6d5143f9a3f0216be2%2FScreenshot%20from%202020-07-15%2014-34-29.png?generation=1594838649442895&alt=media)\n", "930333": "numpy append makes a copy of the data each time you add a row. If that is what you are doing, look at other methods.", "930103": "You can try to load the images in smaller batches.", "930343": "" } }