{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How to load images quickly ?\n*This is my first large scale computer vision competition so feel free to prove me wrong in the comment section*😉\n\nSo I've created my first code for this competition and came across a little problem:<br/>\n**IT'S SOOOOO SLOOOWWWWW**<br/>\n\nSo I wanted to share with you my solution to iterate quickly on this competition. \n\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport time","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T18:24:49.050927Z","iopub.execute_input":"2022-02-04T18:24:49.051327Z","iopub.status.idle":"2022-02-04T18:24:49.426505Z","shell.execute_reply.started":"2022-02-04T18:24:49.051217Z","shell.execute_reply":"2022-02-04T18:24:49.425676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Why slow ?\n## 1.1 The size of images\nSo first of all I didn't even suspected that loading images would be a problem. As a matter of fact, when you start tutorials on computer vision, you come across some random MNIST tutorial that loads the entire dataset into RAM without worrying at all about this problem.","metadata":{}},{"cell_type":"code","source":"N_IMAGES = len(os.listdir('../input/happy-whale-and-dolphin/test_images')) + len(os.listdir('../input/happy-whale-and-dolphin/train_images'))\nprint(\"Total number of images in the dataset:\", N_IMAGES)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T18:24:49.428033Z","iopub.execute_input":"2022-02-04T18:24:49.428251Z","iopub.status.idle":"2022-02-04T18:24:50.565914Z","shell.execute_reply.started":"2022-02-04T18:24:49.428225Z","shell.execute_reply":"2022-02-04T18:24:50.564414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But if you do that here, you'll be in big troubles because your dataset is **BIG**, like ~80k images big...<br>\nSo maybe now you want to tell me:<br>\n*-But MNIST is about the same number of images*<br>\nAnd you would be right but the *little* difference is that here it's far from being 28\\*28 grayscale images...<br>\nLet's open a random image to find out:","metadata":{}},{"cell_type":"code","source":"def read_and_plot_image(image):\n    t = time.time()\n    image = cv2.imread(f\"../input/happy-whale-and-dolphin/train_images/{image}.jpg\")\n    print(\"Opened the image in {:.2f}ms\".format((time.time()-t)*1000))\n    plt.imshow(image)\n    plt.show()\n    print(\"Image array shape:\", image.shape)\n    print(\"Image data type:\", image.dtype)\nread_and_plot_image(\"00144776eb476d\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T18:24:50.568155Z","iopub.execute_input":"2022-02-04T18:24:50.568374Z","iopub.status.idle":"2022-02-04T18:24:51.827708Z","shell.execute_reply.started":"2022-02-04T18:24:50.568347Z","shell.execute_reply":"2022-02-04T18:24:51.826800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Yeah, that's a *lot* of pixels, and that's why the kaggle dataset is 62.06 GB large...\nAnd in fact, if you take the raw space taken by the images it should be even bigger than that. Which leads me to...","metadata":{}},{"cell_type":"markdown","source":"## 1.2 The image compression\nIf you have a look at the images, you will notice that thay are encoded in the *.jpg* format. This format encode an image very efficiently. It loses some information in the process, but it's doing so in a way that visually will not change much.<br>\nFor example, take the following image:","metadata":{}},{"cell_type":"code","source":"read_and_plot_image(\"00144776eb476d\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T18:24:51.830134Z","iopub.execute_input":"2022-02-04T18:24:51.831043Z","iopub.status.idle":"2022-02-04T18:24:53.148018Z","shell.execute_reply.started":"2022-02-04T18:24:51.830999Z","shell.execute_reply":"2022-02-04T18:24:53.147103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The image size on disk is 1.22 MB. But by performing a very clever multiplication, you will notice that...","metadata":{}},{"cell_type":"code","source":"print(\"2336 * 3504 * 3 =\", 2336 * 3504 * 3)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T18:24:53.149243Z","iopub.execute_input":"2022-02-04T18:24:53.149478Z","iopub.status.idle":"2022-02-04T18:24:53.154670Z","shell.execute_reply.started":"2022-02-04T18:24:53.149443Z","shell.execute_reply":"2022-02-04T18:24:53.153691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"... it should take 24.56 MB. That's the magic of jpeg happening right here.<br>\nHowever, it comes with a cost. Indeed, while it takes less space on disk, it will however be slower to load because the computer needs to recompute the original image using fancy clever matrix computations.","metadata":{}},{"cell_type":"markdown","source":"# 2. So how do you make it **fast** ?\n## 2.1 Less pixels\nWell the answer is: do you really need 2336x3504 images to identify a dolphin fin ?\nWell *maybe*...<br>\nbut let's pretend that we can do it with far less pixels !<br>\nThe goal is to reduce the image size. Their are multiple ways to do it like cropping only interesting areas of the image and / or resizing the image.<br>\nAs I don't know how to identify precisely the interesting parts of the image (and it's posssible that the very clever solutions that will win this competition will find a way to do it at some point), I will take the easy path and just resize every image to 224x224 squares. It's a bad idea and you probably shouldn't use it in your final solution\n\n**Why is it a bad idea ?**<br>\nWell it's a bad idea because images aren't all squares, and so this solution will squeeze images and lose some information. Furthermore, they may be details that will not be captured at this resolution\n\n**Why do I do it anyway ?**<br>\nI still do it because it will be a solid base to test quickly new ideas. Then I will bother creating better crafed datasets for the final solutions.\n\n**Why 224x224 ?**<br>\nIt's completely arbitrary, feel free to fork this notebook and change the resolution (next code cell) if you're unhappy with it. (Running time is less than 2 hours)","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = 224","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:24:53.156187Z","iopub.execute_input":"2022-02-04T18:24:53.156493Z","iopub.status.idle":"2022-02-04T18:24:53.166126Z","shell.execute_reply.started":"2022-02-04T18:24:53.156450Z","shell.execute_reply":"2022-02-04T18:24:53.165265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = \"00144776eb476d\"\nimage = cv2.imread(f\"../input/happy-whale-and-dolphin/train_images/{image}.jpg\")\nimage = cv2.resize(image, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_CUBIC)\nprint(image.shape)\nplt.imshow(image)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T18:24:53.167486Z","iopub.execute_input":"2022-02-04T18:24:53.167758Z","iopub.status.idle":"2022-02-04T18:24:53.500613Z","shell.execute_reply.started":"2022-02-04T18:24:53.167723Z","shell.execute_reply":"2022-02-04T18:24:53.499788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, the image is a little squeezed, but we can still see a fair amount of details so I'll go for it.","metadata":{}},{"cell_type":"markdown","source":"## 2.2 Image format\nAs we saw jpeg may not be ideal to load images quickly. So here we'll try to see if there are some formats that are faster to load:","metadata":{}},{"cell_type":"code","source":"cv2.imwrite(\"image.ppm\", image)\ncv2.imwrite(\"image.bmp\", image)\ncv2.imwrite(\"image.dib\", image)\ncv2.imwrite(\"image.jpg\", image)\ncv2.imwrite(\"image.jp2\", image)\ncv2.imwrite(\"image.png\", image)\ncv2.imwrite(\"image.tiff\", image)\ncv2.imwrite(\"image.tif\", image)\nnp.save(\"image.npy\", image)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:24:53.502138Z","iopub.execute_input":"2022-02-04T18:24:53.502573Z","iopub.status.idle":"2022-02-04T18:24:53.573229Z","shell.execute_reply.started":"2022-02-04T18:24:53.502528Z","shell.execute_reply":"2022-02-04T18:24:53.572487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit -n 100 -r 10 cv2.imread(f\"./image.ppm\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.bmp\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.dib\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.jpg\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.jp2\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.png\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.tiff\")\n%timeit -n 100 -r 10 cv2.imread(f\"./image.tif\")\n%timeit -n 100 -r 10 np.load(f\"./image.npy\")","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:24:53.574364Z","iopub.execute_input":"2022-02-04T18:24:53.574597Z","iopub.status.idle":"2022-02-04T18:25:09.749690Z","shell.execute_reply.started":"2022-02-04T18:24:53.574570Z","shell.execute_reply":"2022-02-04T18:25:09.748883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well the least we can say is that for the same image, the format make a huge difference...<br>\nSo we will probably go with \".bmp\" but it seems that we could go with \".dib\" without noticing the difference.\nThese formats are faster because the pixels are stored as such into the memory, so no computing is needed to obtain the original image.\n\nBefore we convert everything, we have to make sure that the total dataset will fit on the harddrive:","metadata":{}},{"cell_type":"code","source":"ls -l --block-size=K","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:25:09.752054Z","iopub.execute_input":"2022-02-04T18:25:09.752290Z","iopub.status.idle":"2022-02-04T18:25:10.524095Z","shell.execute_reply.started":"2022-02-04T18:25:09.752261Z","shell.execute_reply":"2022-02-04T18:25:10.523002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Expected total size of the dataset:\", N_IMAGES * 148000)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:25:10.525945Z","iopub.execute_input":"2022-02-04T18:25:10.526195Z","iopub.status.idle":"2022-02-04T18:25:10.531255Z","shell.execute_reply.started":"2022-02-04T18:25:10.526168Z","shell.execute_reply":"2022-02-04T18:25:10.530378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So everything should be ok because the expected output size of 11.69 GB is less than the notebook output limit (19.6GB)","metadata":{}},{"cell_type":"markdown","source":"# 3. Okay let's do it !","metadata":{}},{"cell_type":"code","source":"!cp ../input/happy-whale-and-dolphin/sample_submission.csv sample_submission.csv\n!cp ../input/happy-whale-and-dolphin/train.csv train.csv\n!rm ./image.* \n!mkdir test_images\n!mkdir train_images","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:34:15.112800Z","iopub.execute_input":"2022-02-04T18:34:15.113547Z","iopub.status.idle":"2022-02-04T18:34:18.936080Z","shell.execute_reply.started":"2022-02-04T18:34:15.113498Z","shell.execute_reply":"2022-02-04T18:34:18.935201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"./train.csv\")\ntrain[\"image\"] = train[\"image\"].str[:-3] + \"bmp\"\ntrain.to_csv(\"./train.csv\", index=False)\nsample_submission = pd.read_csv(\"./sample_submission.csv\")\nsample_submission[\"inference_image\"] = sample_submission[\"image\"].str[:-3] + \"bmp\"\nsample_submission.to_csv(\"./sample_submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T18:34:23.270512Z","iopub.execute_input":"2022-02-04T18:34:23.271359Z","iopub.status.idle":"2022-02-04T18:34:23.770358Z","shell.execute_reply.started":"2022-02-04T18:34:23.271318Z","shell.execute_reply":"2022-02-04T18:34:23.769310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def copy_dir(dirname, base_path=\"../input/happy-whale-and-dolphin/\"):\n    print(\"Copying\", dirname)\n    path = os.path.join(base_path, dirname)\n    images = list(os.listdir(path))\n    n = len(images)\n    for i, f in enumerate(images):\n        print(f\"{i}/{n}\", end=\"\\r\")\n        image_path = os.path.join(path, f)\n        image = cv2.imread(image_path)\n        image = cv2.resize(image, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_CUBIC)\n        new_path = os.path.join(\"./\", dirname, f.split('.')[0] + \".bmp\")\n        cv2.imwrite(new_path, image)\n    \ncopy_dir(\"test_images\")\ncopy_dir(\"train_images\")","metadata":{"execution":{"iopub.status.busy":"2022-02-04T13:59:49.505875Z","iopub.execute_input":"2022-02-04T13:59:49.506183Z","iopub.status.idle":"2022-02-04T13:59:52.371811Z","shell.execute_reply.started":"2022-02-04T13:59:49.50615Z","shell.execute_reply":"2022-02-04T13:59:52.370485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And Done !<br>\nNow, all you have to do to use this fast dataset is to add this notebook as input data and change the path of the files to use the output of the notebook instead of the classical dataset.<br>\nThis can speed up your pipeline by a lot (theoreticaly up to 1000x if image reading is the only bottleneck, which is never the case in practice). If the rest of the code is not well optimized, it can still fairly speed it up as image reading is usually a huge bottlneck.<br>\nTo give a practical idea, on my (very bad and unoptimized) pipeline, it went 5x faster just by replacing the competition data with this one (because other bottlenecks remains)","metadata":{}}]}