![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3428553%2Fd413ad6ec6e512c0b8bd8f7e6a66c0ae%2FScreenshot%20(351).png?generation=1597654783539402&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3428553%2F278ef66fc2a9e5ba27294f155bab2d47%2FScreenshot%20(353).png?generation=1597654817993383&alt=media\")
\ni wrote a function load_images (look in screenshot) ..but cant load images all at a tym in case of test data
i cant load test data ! i am new to this but I am trying my best ..so someone help to load test data and predict the target by model weights
", "messages": [ { "id": "973313", "postDate": "08/17/2020 08:26:51", "content": "i cant load test data ! i am new to this but I am trying my best ..so someone help to load test data and predict the target by model weights
", "rawMarkdown": "i cant load test data ! i am new to this but I am trying my best ..so someone help to load test data and predict the target by model weights", "votes": null }, { "id": "973319", "postDate": "08/17/2020 08:38:48", "content": "Can you share any error or anything which explains the problem better?
\nBtw, if you are using the competition's main input data, use the already resized data.
", "rawMarkdown": "Can you share any error or anything which explains the problem better?\n\nBtw, if you are using the competition's main input data, use the already resized data.", "votes": null }, { "id": "973343", "postDate": "08/17/2020 09:01:22", "content": "
\n
\ni wrote a function load_images (look in screenshot) ..but cant load images all at a tym in case of test data
The thing is that you are trying to read all the images at once, that's why the program is giving a memory error. What we do in such a case is we use batches, we use one batch at a time, i.e. if our batch size is 32, then we are only reading 32 images at a time, this way you will not have any memory issues.
\nAs for now, just try to work on some already public notebook, if you want to work for the competition even if its just about to end, there are many baselines available.
\nAlso, use already resized images and not the images from the input! There are many available like this and this.
", "rawMarkdown": "The thing is that you are trying to read all the images at once, that's why the program is giving a memory error. What we do in such a case is we use batches, we use one batch at a time, i.e. if our batch size is 32, then we are only reading 32 images at a time, this way you will not have any memory issues.\n\nAs for now, just try to work on some already public notebook, if you want to work for the competition even if its just about to end, there are many baselines available.\n\nAlso, use already resized images and not the images from the input! There are many available like [this](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161043) and [this](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092).", "votes": null }, { "id": "973412", "postDate": "08/17/2020 09:34:00", "content": "I have same problem too i find out a solution to do it.
\nI split testdataset 22 times to predict model in partwise.
\nJust a sample working
\nhere
\nIf any one found better idea to overcome it please share it too.
I did it by batch size even 16 and 8 but still it say more memory allocated. I do not know the reason why.
", "rawMarkdown": "I did it by batch size even 16 and 8 but still it say more memory allocated. I do not know the reason why.", "votes": null }, { "id": "973420", "postDate": "08/17/2020 09:39:17", "content": "You may be using a heavy model, you see all the elements in a program take memory, it is not limited to data or model, you just need to manage the memory.
", "rawMarkdown": "You may be using a heavy model, you see all the elements in a program take memory, it is not limited to data or model, you just need to manage the memory.", "votes": null }, { "id": "973424", "postDate": "08/17/2020 09:41:31", "content": "I used simple model too still test data is huge
", "rawMarkdown": "I used simple model too still test data is huge", "votes": null } ], "comments": [ { "id": 973319, "author_name": "sarques", "author_url": "", "post_date": "08/17/2020 08:38:48", "content": "Can you share any error or anything which explains the problem better?
\nBtw, if you are using the competition's main input data, use the already resized data.
", "votes": null, "replies": [] }, { "id": 973343, "author_name": "yagnasrip2", "author_url": "", "post_date": "08/17/2020 09:01:22", "content": "
\n
\ni wrote a function load_images (look in screenshot) ..but cant load images all at a tym in case of test data
The thing is that you are trying to read all the images at once, that's why the program is giving a memory error. What we do in such a case is we use batches, we use one batch at a time, i.e. if our batch size is 32, then we are only reading 32 images at a time, this way you will not have any memory issues.
\nAs for now, just try to work on some already public notebook, if you want to work for the competition even if its just about to end, there are many baselines available.
\nAlso, use already resized images and not the images from the input! There are many available like this and this.
", "votes": null, "replies": [ { "id": 973417, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "08/17/2020 09:36:11", "content": "I did it by batch size even 16 and 8 but still it say more memory allocated. I do not know the reason why.
", "votes": null, "replies": [] }, { "id": 973420, "author_name": "sarques", "author_url": "", "post_date": "08/17/2020 09:39:17", "content": "You may be using a heavy model, you see all the elements in a program take memory, it is not limited to data or model, you just need to manage the memory.
", "votes": null, "replies": [ { "id": 973424, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "08/17/2020 09:41:31", "content": "I used simple model too still test data is huge
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 973412, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "08/17/2020 09:34:00", "content": "I have same problem too i find out a solution to do it.
\nI split testdataset 22 times to predict model in partwise.
\nJust a sample working
\nhere
\nIf any one found better idea to overcome it please share it too.