{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Fast Image Reading via Multiprocessing & Getting Image Sizes\n\nThe train dataset contains ~1.5 million images. The popular image libraries, such as imageio or cv2, read ~20 images in a second on average (the average is based on first 38k images in the train dataset). Reading 1.5 million images with this average rate would give a total reading time of 20.8 hr, which is impossibly long to keep the kaggle notebebook alive.\n\nThis reading time can be reduced by multiprocessing as can be shown in this notebook. Exploiting this faster reading time, I present here a full list of image size (xsize, ysize) and depth (channel) information for all train images.\n\nP.S. Please don't forget to upvote, if you find the notebook useful.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport tqdm\nimport imageio\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom multiprocessing import Pool\nimport seaborn as sns; sns.set(style=\"white\", color_codes=True)\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"base_dir = '../input/landmark-recognition-2020/'\ntrain_csv = pd.read_csv(base_dir + 'train.csv')\nsample_submission = pd.read_csv(base_dir + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_image_features(lid):\n    impath = base_dir + 'train/' + '/'.join(list(lid[:3])) + '/' + lid + '.jpg'\n    im = imageio.imread(impath)\n    xsize = im.shape[-3]\n    ysize = im.shape[-2]\n    depth = im.shape[-1]\n    return xsize, ysize, depth","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with Pool(4) as p:\n    r = list(tqdm.tqdm(p.imap(get_image_features, train_csv.id), total=1000))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv['xsize'] = np.array(r).T[0]\ntrain_csv['ysize'] = np.array(r).T[1]\ntrain_csv['depth'] = np.array(r).T[2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.to_csv('train_featured.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Train Image Size Distribution**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.jointplot(x=\"xsize\", y=\"ysize\", data=train_csv)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}