{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "e37c04e5-e322-0a44-1aad-ecba52d11797" }, "source": [ "# 0. First of all\n", "\n", "This kernel is the tutorial to explore and visualize datasets and train Convolutional Neural Network (CNN) on keras.\n", "\n", "I'm not good at English. So, **please post a comment if there are any unknown points:)** \n", "\n", "Additionally, this kernel is unfinished, still writing. I will do my best! \n", "This kernel is getting better little by little. **Many thanks to all of the comments.**" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e05ea04f-6e18-acb2-fdc2-13ff526e46b8" }, "source": [ "# 1. Environment construction\n", "\n", "In this kernel, you will mainly compute with python on Colfax Cluster. \n", "\n", "The datasets (test, train, additional) had extracted and placed in /data/kaggle/ directory on Colfax Cluster. \n", "So, you don't have to download datasets on your local machine. \n", "Of course, you can download them while reading this kernel for killing time.\n", "\n", "Datasets are here: \n", "[https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/data](https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/data)\n", "\n", "Note: \n", "You can use extracted datasets on kaggle's kernel too. \n", "But I recommend using Colfax Cluster from the point of computing speed and making your original submission." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "649ba9be-160a-e84c-5e65-d039f6c3bee1" }, "source": [ "## 1-1. Setting ssh connection to Colfax Cluster\n", "\n", "Sign up to Colfax Cluster:\n", "[https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening#Intel-Tutorial](https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening#Intel-Tutorial) \n", "\n", "Reference to ssh connection:[https://access.colfaxresearch.com/?p=connect](https://access.colfaxresearch.com/?p=connect) \n", "\n", "Remember:\n", "\n", " chmod 600 ~/Downloads/colfax-access-key-****\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "49b7c97c-75cd-2cde-8c6f-9cbe298c0743" }, "source": [ "## 1-2. Build enviroment after connect to Colfax Cluster by ssh colfax\n", "\n", "Make own environment, install opencv, etc.\n", "\n", " ssh colfax\n", " conda create --name test_env jupyter\n", " source activate test_env\n", " conda install numpy pandas opencv scikit-learn matplotlib tensorflow keras jupyter\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ead30ed3-e123-69da-0fce-34fe66832b06" }, "source": [ "## 1-3. Configure Jupyter Notebook, port and password (Thanks to everyone commented)\n", "\n", "For avoid port collision and access by other user's access.\n", "\n", "### Select port (recommended)\n", "\n", "Select port number, not likely to make collision. Default is 8888. \n", "If port collides, another port will be used (like 8888 -> 8889). \n", "It's a hassle, so use unique port.\n", "\n", "If you are not familiar with network port configurations, I think ephemeral ports (49152 - 65535) are useful. \n", "Here is the script to choose random ephemeral ports.\n", "Of course, you can choose your favorite number in 49152 - 65535.\n", "\n", " python -c \"import random; ports = range(49152, 65535 + 1); random.shuffle(ports); print ports[0]\"\n", "\n", "### Make Password hash (recommended)\n", "\n", "Run the bellow command, input password twice, then you get hashed-password.\n", "\n", " python -c \"from notebook.auth import passwd; print passwd()\"\n", " # e.g.) => sha1:237ca8abda58:9aef98cbcbae988caab4b9f86084ff22a1b2b373\n", "\n", "### Generate and edit config file\n", "\n", "Generate config file (~/.jupyter/jupyter_notebook_config.py)\n", "\n", " jupyter notebook --generate-config\n", "\n", "Edit via vi like this\n", "\n", " vi ~/.jupyter/jupyter_notebook_config.py\n", " ....\n", " # c.NotebookApp.password = u''\n", " c.NotebookApp.password = u'sha1:237ca8abda58:9aef98cbcbae988caab4b9f86084ff22a1b2b373'\n", " ....\n", " # c.NotebookApp.port = 8888\n", " c.NotebookApp.port = 1234\n", "\n", "If you are not familiar with vi, use the bellow scripts (need to edit)\n", "\n", " echo \"c.NotebookApp.password = u'sha1:237ca8abda58:9aef98cbcbae988caab4b9f86084ff22a1b2b373'\\n\" >> ~/.jupyter/jupyter_notebook_config.py\n", " echo \"c.NotebookApp.port = 1234\\n\" >> ~/.jupyter/jupyter_notebook_config.py\n", "\n", "If you missed something, you can regenerate (over-write) config file.\n", "\n", " jupyter notebook --generate-config\n", "\n", "Note: If your setting port makes port collision unfortunately, another port will be used.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7418d784-7a92-4672-120f-7755d995007c" }, "source": [ "## 1-4. Connect to Colfax Cluster by ssh tunneling, and run Jupyter Notebook\n", "\n", "Before runnig Jupyter Notebook, once logout.\n", "\n", " logout\n", "\n", "And, runnig Jupyter Notebook via ssh tunneling\n", "\n", " ssh -L 1234:localhost:1234 colfax -Y\n", " source activate test_env\n", " jupyter notebook --no-browser\n", "\n", "**Note: The window (ran command) should be kept opened!**\n", "\n", "Note: You can change port this step (Special thanks to Sriracha's comment)\n", "\n", " ssh -L 4321:localhost:4321 colfax -Y\n", " source activate test_env\n", " jupyter notebook --no-browser --port=4321\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9aa997e6-3838-aa3b-18af-266173090262" }, "source": [ "## 1-5. Access to Jupyter Notebook on Colfax Cluste by your local machine's browser\n", "\n", " Access by your web browser (e.g. Google Chrome) on your local machine (e.g. Windows, Mac...)\n", "\n", "[http://localhost:1234/](http://localhost:1234/)\n", " or [http://127.0.0.1:1234/](http://127.0.0.1:1234/) (if you can't)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "adb6d1f9-2f05-0373-3dd3-4cb3f210cf7d" }, "source": [ "# 2. Listing dataset image files" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4160d39c-8fdb-2bc2-c2b4-94ef885aa90c" }, "source": [ "## 2-0. Setting of dataset's directories" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "bb2dee24-9a17-21ce-077d-47326ebf3b41" }, "outputs": [], "source": [ "import platform\n", "import os\n", "\n", "if 'c001' in platform.node(): \n", " # platform.node() => 'c001' or like 'c001-n030' on Colfax\n", " abspath_dataset_dir_train_1 = '/data/kaggle/train/Type_1'\n", " abspath_dataset_dir_train_2 = '/data/kaggle/train/Type_2'\n", " abspath_dataset_dir_train_3 = '/data/kaggle/train/Type_3'\n", " abspath_dataset_dir_test = '/data/kaggle/test/'\n", " abspath_dataset_dir_add_1 = '/data/kaggle/additional/Type_1'\n", " abspath_dataset_dir_add_2 = '/data/kaggle/additional/Type_2'\n", " abspath_dataset_dir_add_3 = '/data/kaggle/additional/Type_3'\n", "elif '.local' in platform.node():\n", " # platform.node() => '*.local' on my local MacBook Air\n", " abspath_dataset_dir_train_1 = '/abspath/to/train/Type_1'\n", " abspath_dataset_dir_train_2 = '/abspath/to/train/Type_2'\n", " abspath_dataset_dir_train_3 = '/abspath/to/train/Type_3'\n", " abspath_dataset_dir_test = '/abspath/to/test/'\n", " abspath_dataset_dir_add_1 = '/abspath/to/additional/Type_1'\n", " abspath_dataset_dir_add_2 = '/abspath/to/additional/Type_2'\n", " abspath_dataset_dir_add_3 = '/abspath/to/additional/Type_3'\n", "else:\n", " # For kaggle's kernels environment (docker container?)\n", " abspath_dataset_dir_train_1 = '/kaggle/input/train/Type_1'\n", " abspath_dataset_dir_train_2 = '/kaggle/input/train/Type_2'\n", " abspath_dataset_dir_train_3 = '/kaggle/input/train/Type_3'\n", " abspath_dataset_dir_test = '/kaggle/input/test/'\n", " abspath_dataset_dir_add_1 = '/kaggle/input/additional/Type_1'\n", " abspath_dataset_dir_add_2 = '/kaggle/input/additional/Type_2'\n", " abspath_dataset_dir_add_3 = '/kaggle/input/additional/Type_3'\n", "\n", " \n", "def get_list_abspath_img(abspath_dataset_dir):\n", " list_abspath_img = []\n", " for str_name_file_or_dir in os.listdir(abspath_dataset_dir):\n", " if ('.jpg' in str_name_file_or_dir) == True:\n", " list_abspath_img.append(os.path.join(abspath_dataset_dir, str_name_file_or_dir))\n", " list_abspath_img.sort()\n", " return list_abspath_img\n", "\n", "\n", "list_abspath_img_train_1 = get_list_abspath_img(abspath_dataset_dir_train_1)\n", "list_abspath_img_train_2 = get_list_abspath_img(abspath_dataset_dir_train_2)\n", "list_abspath_img_train_3 = get_list_abspath_img(abspath_dataset_dir_train_3)\n", "list_abspath_img_train = list_abspath_img_train_1 + list_abspath_img_train_2 + list_abspath_img_train_3\n", "\n", "list_abspath_img_test = get_list_abspath_img(abspath_dataset_dir_test)\n", "\n", "list_abspath_img_add_1 = get_list_abspath_img(abspath_dataset_dir_add_1)\n", "list_abspath_img_add_2 = get_list_abspath_img(abspath_dataset_dir_add_2)\n", "list_abspath_img_add_3 = get_list_abspath_img(abspath_dataset_dir_add_3)\n", "list_abspath_img_add = list_abspath_img_add_1 + list_abspath_img_add_2 + list_abspath_img_add_3\n", "\n", "# 0: Type_1, 1: Type_2, 2: Type_3\n", "list_answer_train = [0] * len(list_abspath_img_train_1) + [1] * len(list_abspath_img_train_2) + [2] * len(list_abspath_img_train_3)\n", "list_answer_add = [0] * len(list_abspath_img_add_1) + [1] * len(list_abspath_img_add_2) + [2] * len(list_abspath_img_add_3)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "91bf2531-3e23-dd49-c059-802bc4b6c104" }, "source": [ "## 2-1. Check the (small part of) absolute paths" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0af1c716-ce52-0315-d2b7-b9c533dbad25" }, "outputs": [], "source": [ "print(list_abspath_img_train_1[0:2])\n", "print(list_abspath_img_train_2[0:2])\n", "print(list_abspath_img_train_3[0:2])\n", "print(list_abspath_img_train[0:4])\n", "print(list_abspath_img_test[0:3])\n", "print(list_abspath_img_add_1[0:2])\n", "print(list_abspath_img_add_2[0:2])\n", "print(list_abspath_img_add_3[0:2])\n", "print(list_abspath_img_add[0:4])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "41059f09-9166-e036-102b-86d4117c85ad" }, "source": [ "## 2-2. Counting number of image files\n", "\n", "Pandas is powerful data analysis toolkit. It is very useful to input, output and analyze csv files.\n", "\n", "Check [10 Minutes to pandas](http://pandas.pydata.org/pandas-docs/stable/10min.html) if you have time." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d5c205ed-434f-c9d7-7830-b27a33ace592" }, "outputs": [], "source": [ "import pandas\n", "\n", "\n", "pandas_columns = ['Number of image files']\n", "pandas_index = ['train_1', 'train_2', 'train_3', 'train', 'test', 'add_1', 'add_2', 'add_3', 'add', 'train + add', 'total']\n", "pandas_data = [len(list_abspath_img_train_1), len(list_abspath_img_train_2), len(list_abspath_img_train_3), len(list_abspath_img_train), len(list_abspath_img_test), len(list_abspath_img_add_1), len(list_abspath_img_add_2), len(list_abspath_img_add_3), len(list_abspath_img_add), len(list_abspath_img_train) + len(list_abspath_img_add), len(list_abspath_img_train) + len(list_abspath_img_test) + len(list_abspath_img_add)]\n", "\n", "pandas.DataFrame(pandas_data, index = pandas_index, columns = pandas_columns)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a2fc5f20-3a5a-545a-2edb-8d133cc6ed55" }, "source": [ "## 2-3. Showing the ratio (Type 1, Type 2, Type 3)\n", "\n", "It\u2019s usually a good idea to check the deviation of dataset. \n", "In my experience of another competition, my model's training accuracy was more than 80%, but all prediction were same \n", "(my bullshit model was the master of selecting the majority \ud83d\ude2d )." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ebb3300c-922a-9d5a-46da-5f78c988c275" }, "outputs": [], "source": [ "pandas_columns = ['Type_1', 'Type_2', 'Type_3']\n", "pandas_index = ['train', 'test', 'add']\n", "\n", "ratio_train = [x / len(list_abspath_img_train) for x in [len(list_abspath_img_train_1), len(list_abspath_img_train_2), len(list_abspath_img_train_3)]]\n", "ratio_test = ['?', '?', '?']\n", "ratio_add = [x / len(list_abspath_img_add) for x in [len(list_abspath_img_add_1), len(list_abspath_img_add_2), len(list_abspath_img_add_3)]]\n", "\n", "pandas_data = [ratio_train, ratio_test, ratio_add]\n", "\n", "pandas.DataFrame(pandas_data, index = pandas_index, columns = pandas_columns)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "90cc6a1e-7ffb-1861-4f04-dc053496e56c" }, "source": [ "# Check dataset image pixel sizes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "81f1eb55-6535-6e00-3422-dedc870c737c" }, "outputs": [], "source": [ "'''\n", "import cv2\n", "\n", "\n", "abspath_output_csv = './check_img_shape.csv'\n", "\n", "file_output_csv = open(abspath_output_csv, 'w')\n", "file_output_csv.write('abspath,shape_1,shape_2,shape_3\\n')\n", "file_output_csv.close()\n", "\n", "for abspath_img in (list_abspath_img_train + list_abspath_img_test):\n", " str_shape = str(cv2.imread(abspath_img).shape)\n", " str_shape = str_shape.replace('(', '').replace(')', '').replace(' ', '')\n", " file_output_csv = open(abspath_output_csv, 'a')\n", " file_output_csv.write('%s,%s\\n' % (abspath_img, str_shape))\n", " file_output_csv.close()\n", "'''\n", "\n", "'''\n", "It will spend a lot of time to run. So I comment-out in the kernel notebook.\n", "I uploaded './check_img_shape.csv' on Google Drive.\n", "'''\n", "\n", "\"https://drive.google.com/open?id=0B2kJp7wSl9SIZTgtOWlTSmtDT2s\"" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "daa7cdb1-4e37-1f8f-46f4-ff10de767b8a" }, "source": [ "# 3. Show images by matplotlib" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7c8bb625-14f3-e1e6-38fa-506bc6cbd49f" }, "outputs": [], "source": [ "import cv2\n", "import matplotlib.pyplot\n", "\n", "\n", "def sub_func_load_img(abspath_img):\n", " img_rgb = cv2.cvtColor(cv2.imread(abspath_img), cv2.COLOR_BGR2RGB)\n", " return img_rgb\n", "\n", "def show_img(abspath_img):\n", " matplotlib.pyplot.imshow(sub_func_load_img(abspath_img))\n", " matplotlib.pyplot.show()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3ed1c14d-3c22-2430-6eb1-fb93fd13f4b5" }, "outputs": [], "source": [ "# Show the first image\n", "\n", "show_img(list_abspath_img_train[0])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "48d1ebbb-9d23-e3eb-38b5-27c03f0f2848" }, "outputs": [], "source": [ "# Another Usage, using string of image file's path\n", "\n", "if 'c001' in platform.node():\n", " abspath_img = '/data/kaggle/test/81.jpg' # on Colfax Cluster\n", "else:\n", " abspath_img = '../input/test/81.jpg' # on Kaggle's Kernel\n", "\n", "show_img(abspath_img)\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9e133eed-7e20-ce4c-bd60-df0caf68f628" }, "source": [ "## Resampling images\n", "\n", "For input CNN, unify all images into 640 * 480 RGB images\n", " (fixed aspect-ratio and filled blank with black color).\n", "\n", "This is just only one example out of many.\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "38be759c-4495-40a6-3172-cde62b4cc43b" }, "source": [ "## Step 0: Unify sidelong images into vertically long images \n", "\n", "This step can be skipped." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f58a5adf-91eb-eb0a-b12d-fb15bfa375bd" }, "outputs": [], "source": [ "import numpy\n", "\n", "\n", "def sub_func_rotate_img_if_need(img_rgb):\n", " if img_rgb.shape[0] >= img_rgb.shape[1]:\n", " return img_rgb\n", " else:\n", " return numpy.rot90(img_rgb)\n", "\n", "\n", "\n", "if 'c001' in platform.node():\n", " abspath_img = '/data/kaggle/test/81.jpg' # on Colfax Cluster\n", "else:\n", " abspath_img = '../input/test/81.jpg' # on Kaggle Kernel\n", "\n", " \n", "img_rgb = sub_func_load_img(abspath_img)\n", "\n", "matplotlib.pyplot.imshow(img_rgb)\n", "matplotlib.pyplot.show()\n", "\n", "matplotlib.pyplot.imshow(sub_func_rotate_img_if_need(img_rgb))\n", "matplotlib.pyplot.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c6474e82-0141-0a2e-fedb-3c07c7afa408" }, "source": [ "## Step 1: Resize image with same aspect-ratio\n", "\n", "sidelong images -> (640, *, 3)\n", "\n", "vertically long images -> (*, 480, 3)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "63ed40b8-436f-5dec-8b3c-d66a1d52f3a8" }, "outputs": [], "source": [ "def sub_func_resize_img_same_ratio(img_rgb):\n", " if img_rgb.shape[0] / 640.0 >= img_rgb.shape[1] / 480.0:\n", " img_resized_rgb = cv2.resize(img_rgb, (int(640.0 * img_rgb.shape[1] / img_rgb.shape[0]), 640)) # (640, *, 3)\n", " else:\n", " img_resized_rgb = cv2.resize(img_rgb, (480, int(480.0 * img_rgb.shape[0] / img_rgb.shape[1]))) # (*, 480, 3)\n", " return img_resized_rgb\n", "\n", "\n", "if 'c001' in platform.node():\n", " abspath_img = '/data/kaggle/test/81.jpg' # on Colfax Cluster\n", "else:\n", " abspath_img = '../input/test/81.jpg' # on Kaggle Kernel\n", "\n", " \n", "img_rgb = sub_func_load_img(abspath_img)\n", "\n", "matplotlib.pyplot.imshow(img_rgb)\n", "matplotlib.pyplot.show()\n", "print(img_rgb.shape)\n", "\n", "matplotlib.pyplot.imshow(sub_func_resize_img_same_ratio(img_rgb))\n", "matplotlib.pyplot.show()\n", "print(sub_func_resize_img_same_ratio(img_rgb).shape)\n", "\n", "# Step 0 + Step 1 -> (*, 480, 3), Accidentally this example -> (640 ,480, 3)\n", "matplotlib.pyplot.imshow(sub_func_resize_img_same_ratio(sub_func_rotate_img_if_need(img_rgb)))\n", "matplotlib.pyplot.show()\n", "print(sub_func_resize_img_same_ratio(sub_func_rotate_img_if_need(img_rgb)).shape)\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4deccd52-832c-b794-2cdd-c015d5fbdd74" }, "source": [ "Step 2: Fill blank with black-color" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0fb477f6-fc57-59ef-8aab-c5998588a28a" }, "outputs": [], "source": [ "def sub_func_fill_img(img_rgb):\n", " if img_rgb.shape[0] == 640:\n", " int_resize_1 = img_rgb.shape[1]\n", " int_fill_1 = (480 - int_resize_1 ) // 2\n", " int_fill_2 = 480 - int_resize_1 - int_fill_1\n", " numpy_fill_1 = numpy.zeros((640, int_fill_1, 3), dtype=numpy.uint8)\n", " numpy_fill_2 = numpy.zeros((640, int_fill_2, 3), dtype=numpy.uint8)\n", " img_filled_rgb = numpy.concatenate((numpy_fill_1, img_rgb, numpy_fill_1), axis=1)\n", " elif img_rgb.shape[1] == 480:\n", " int_resize_0 = img_rgb.shape[0]\n", " int_fill_1 = (640 - int_resize_0 ) // 2\n", " int_fill_2 = 640 - int_resize_0 - int_fill_1\n", " numpy_fill_1 = numpy.zeros((int_fill_1, 480, 3), dtype=numpy.uint8)\n", " numpy_fill_2 = numpy.zeros((int_fill_2, 480, 3), dtype=numpy.uint8)\n", " img_filled_rgb = numpy.concatenate((numpy_fill_1, img_rgb, numpy_fill_1), axis=0)\n", " else:\n", " raise ValueError\n", " return img_filled_rgb\n", "\n", "\n", "matplotlib.pyplot.imshow(img_rgb)\n", "matplotlib.pyplot.show()\n", "print(img_rgb.shape)\n", "\n", "# Step 1 + Step 2\n", "matplotlib.pyplot.imshow(sub_func_fill_img(sub_func_resize_img_same_ratio(img_rgb)))\n", "matplotlib.pyplot.show()\n", "print(sub_func_fill_img(sub_func_resize_img_same_ratio(img_rgb)).shape)\n", "\n", "\n", "# Step 0 + Step 1 + Step 2\n", "matplotlib.pyplot.imshow(sub_func_fill_img(sub_func_resize_img_same_ratio(sub_func_rotate_img_if_need(img_rgb))))\n", "matplotlib.pyplot.show()\n", "print(sub_func_fill_img(sub_func_resize_img_same_ratio(sub_func_rotate_img_if_need(img_rgb))).shape)\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d9fc1466-b7ec-9766-517a-fe3de79e01c7" }, "source": [ "## Finally: Step 0 + Step 1 + Step 2" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "34f0fc26-0bcd-12e4-cab1-96aaff28ea24" }, "outputs": [], "source": [ "def sub_func_resample_img(abspath_img):\n", " img = sub_func_load_img(abspath_img)\n", " img = sub_func_rotate_img_if_need(img)\n", " img = sub_func_resize_img_same_ratio(img)\n", " img = sub_func_fill_img(img)\n", " return img\n", "\n", "def show_resample_img(abspath_img):\n", " matplotlib.pyplot.imshow(sub_func_resample_img(abspath_img))\n", " matplotlib.pyplot.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "005daf60-a5b7-45e8-e553-6e50348da6be" }, "outputs": [], "source": [ "show_img(list_abspath_img_train[0])\n", "print(sub_func_load_img(list_abspath_img_train[0]).shape)\n", "\n", "show_resample_img(list_abspath_img_train[0])\n", "print(sub_func_resample_img(list_abspath_img_train[0]).shape)\n", "\n", "if 'c001' in platform.node():\n", " abspath_img = '/data/kaggle/test/81.jpg' # on Colfax Cluster\n", "else:\n", " abspath_img = '../input/test/81.jpg' # on Kaggle Kernel\n", "\n", "show_img(abspath_img)\n", "print(sub_func_load_img(abspath_img).shape)\n", "\n", "show_resample_img(abspath_img)\n", "print(sub_func_resample_img(abspath_img).shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c067f1ef-7377-1f5f-3d49-cb35db74a9b2" }, "outputs": [], "source": [ "matplotlib.pyplot.imshow(cv2.resize(sub_func_resample_img(abspath_img), (224, 224)))\n", "matplotlib.pyplot.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "652d4180-870c-62d0-6ec4-1ffddef4663a" }, "source": [ "## Parallel computation\n", "\n", " multiprocessing.cpu_count() -> 8 # Jupyter Notebook on Colfax Cluster\n", " multiprocessing.cpu_count() -> 256 # on Colfax Cluster, using `qsub` to run script" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "90091940-5e8a-7d95-b5b8-3e5457f47808" }, "outputs": [], "source": [ "import multiprocessing\n", "\n", "\n", "def multi_func_resample_img(list_abspath_img):\n", " multiprocessing_pool = multiprocessing.Pool(max(1, multiprocessing.cpu_count() - 1))\n", " return multiprocessing_pool.map(sub_func_resample_img, list_abspath_img)\n", "\n", "\n", "list_img_train = multi_func_resample_img(list_abspath_img_train[0:4])\n", "\n", "for resample_img in list_img_train:\n", " matplotlib.pyplot.imshow(resample_img)\n", " matplotlib.pyplot.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "498469ef-7f50-c75e-a0e7-6253ec5dfe15" }, "source": [ "## To Be Continued..." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f0078de4-cf09-86f6-d05e-6046086ae8e8" }, "source": [ "## Run script on Colfax Cluster (MEMO, Re-write at training section)\n", "\n", "We can run `python ./check_img_shape.py` on terminal directory. \n", "But this method can use only 8 cpu-core, and the process will be terminated if spent long-time. \n", "\n", "If you run `qsub ./check_img_shape.sh`, you can use 256 cpu-core. \n", "\n", "### Make check_img_shape.sh (via `vi`, `echo -e`, etc.). \n", "\n", "The contents of check_img_shape.sh is bellow. \n", "u???? is your user name like u2000. \n", "\n", " source activate test_env\n", " python /home/u????/check_img_shape.py\n", "\n", "### Compute by qsub\n", "\n", " qsub ./check_img_shape.sh\n", "\n", "### Check runnning status:\n", "\n", " qstat\n", "\n", "### After runnning:\n", "\n", " STDOUT -> ./check_img_shape.sh.o0000\n", " STDERR -> ./check_img_shape.sh.e0000\n", "\n", "If you need, check it:\n", "\n", " cat ./check_img_shape.sh.o*\n", " cat ./check_img_shape.sh.e*y " ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }