I want to do some preprocessor for each image in train set, but when the code read images from the train set, the problem of out of memory(oom) appeared. \nJust the following code,(even not include the image preprocessor part) makes the kernel out of memory. when these code run completed, it takes up about 7G memory. \n'''\ni =0\nwhile True:\n imss = cv2.imread(\"../input/aptos2019-blindness-detection/train_images/%s.png\" % train['id_code'][i])
\n ## this is the space for image preprocessor\n del imss\n gc.collect()\n i = i+1\n print(i,' ',end='')\n if i == len(train):\n break
Thanks for the report, see https://www.kaggle.com/product-feedback/104464#1058238 for update.
", "rawMarkdown": "Thanks for the report, see https://www.kaggle.com/product-feedback/104464#1058238 for update." }, { "id": 577643, "postDate": "2019-07-16T22:00:58.917Z", "content": "To be clear, I created a simple kernel, the whole code is below, and the problem is still exist, as mentioned in last reply, this problem only occured on kaggle kernel(python version 3.6.6), but not on local computer(python version 3.6.4), Does the version of python will lead to this? Is it possible to change the python version in kaggle kernel?\nimport os\nimport cv2\nimport numpy as np\nimport pandas as pd
train = pd.read_csv('../input/train.csv')
i =0\nwhile True:\nimss = cv2.imread(\"../input/train_images/%s.png\" % train['id_code'][i])\ni = i+1\nprint(i,' ',end='')\nif i == len(train):\nbreak
If this is the complete code - that is strange, because you are only allocating memory for one image at most at each iteration.\nBut if you save your preprocessed image somewhere - that's not surprising at all, providing the size of the images.
", "rawMarkdown": "If this is the complete code - that is strange, because you are only allocating memory for one image at most at each iteration.\nBut if you save your preprocessed image somewhere - that's not surprising at all, providing the size of the images.", "replies": [ { "id": 577630, "postDate": "2019-07-16T21:25:58.513Z", "content": "yes, it's strange, when the same code running on local computer, this problem doesn't appeard.
", "rawMarkdown": "yes, it's strange, when the same code running on local computer, this problem doesn't appeard." } ] }, { "id": 577364, "postDate": "2019-07-16T16:24:33.157Z", "content": "That could be due to the image size you are currently using. You can either try reducing it or process the training data in smaller buckets.
", "rawMarkdown": "That could be due to the image size you are currently using. You can either try reducing it or process the training data in smaller buckets." }, { "id": 577250, "postDate": "2019-07-16T14:40:14.107Z", "content": "I want to do some preprocessor for each image in train set, but when the code read images from the train set, the problem of out of memory(oom) appeared. \nJust the following code,(even not include the image preprocessor part) makes the kernel out of memory. when these code run completed, it takes up about 7G memory. \n'''\ni =0\nwhile True:\n imss = cv2.imread(\"../input/aptos2019-blindness-detection/train_images/%s.png\" % train['id_code'][i])
\n ## this is the space for image preprocessor\n del imss\n gc.collect()\n i = i+1\n print(i,' ',end='')\n if i == len(train):\n break
Thanks for the report, see https://www.kaggle.com/product-feedback/104464#1058238 for update.
", "votes": 0, "replies": [] }, { "id": 577643, "author_name": "daniel", "author_url": "", "post_date": "2019-07-16T22:00:58.917000", "content": "To be clear, I created a simple kernel, the whole code is below, and the problem is still exist, as mentioned in last reply, this problem only occured on kaggle kernel(python version 3.6.6), but not on local computer(python version 3.6.4), Does the version of python will lead to this? Is it possible to change the python version in kaggle kernel?\nimport os\nimport cv2\nimport numpy as np\nimport pandas as pd
train = pd.read_csv('../input/train.csv')
i =0\nwhile True:\nimss = cv2.imread(\"../input/train_images/%s.png\" % train['id_code'][i])\ni = i+1\nprint(i,' ',end='')\nif i == len(train):\nbreak
If this is the complete code - that is strange, because you are only allocating memory for one image at most at each iteration.\nBut if you save your preprocessed image somewhere - that's not surprising at all, providing the size of the images.
", "votes": 0, "replies": [ { "id": 577630, "author_name": "daniel", "author_url": "", "post_date": "2019-07-16T21:25:58.513000", "content": "yes, it's strange, when the same code running on local computer, this problem doesn't appeard.
", "votes": 0, "replies": [] } ] }, { "id": 577364, "author_name": "Federico Raimondi Cominesi", "author_url": "", "post_date": "2019-07-16T16:24:33.157000", "content": "That could be due to the image size you are currently using. You can either try reducing it or process the training data in smaller buckets.
", "votes": 0, "replies": [] }, { "id": 617518, "author_name": "", "author_url": "", "post_date": "2019-09-04T07:51:32.350000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1058283": "Thanks for the report, see https://www.kaggle.com/product-feedback/104464#1058238 for update.", "577643": "To be clear, I created a simple kernel, the whole code is below, and the problem is still exist, as mentioned in last reply, this problem only occured on kaggle kernel(python version 3.6.6), but not on local computer(python version 3.6.4), Does the version of python will lead to this? Is it possible to change the python version in kaggle kernel?\n`import os`\n`import cv2`\n`import numpy as np`\n`import pandas as pd`\n \n`train = pd.read_csv('../input/train.csv')`\n \n`i =0`\n`while True:`\n` imss = cv2.imread(\"../input/train_images/%s.png\" % train['id_code'][i]) `\n` i = i+1`\n` print(i,' ',end='')`\n` if i == len(train):`\n` break`", "577530": "If this is the complete code - that is strange, because you are only allocating memory for one image at most at each iteration.\nBut if you save your preprocessed image somewhere - that's not surprising at all, providing the size of the images.", "577364": "That could be due to the image size you are currently using. You can either try reducing it or process the training data in smaller buckets.", "577250": "I want to do some preprocessor for each image in train set, but when the code read images from the train set, the problem of out of memory(oom) appeared. \nJust the following code,(even not include the image preprocessor part) makes the kernel out of memory. when these code run completed, it takes up about 7G memory. \n'''\ni =0\nwhile True:\n imss = cv2.imread(\"../input/aptos2019-blindness-detection/train_images/%s.png\" % train['id_code'][i]) \n ## this is the space for image preprocessor\n del imss\n gc.collect()\n i = i+1\n print(i,' ',end='')\n if i == len(train):\n break", "617518": "" } }