{"cells":[{"metadata":{},"cell_type":"markdown","source":"# PRE-PROCESSING FILES FROM APTOS 2019\n\n### The aim of this kernel is to compress, convert into arrays and normalise all the provided images and splitting those arrays into training and cross-validation subsets.\n\n#### The process will be as follows:\n\n* One-hot encoding of the training labels.\n* Convert, compress and normalise the training set.\n* Convert, compress and normalise the test set.\n* Split the preprocessed training set into random train and test subsets.\n* Save the arrays in compressed NPZ files.\n\n - Let's import some useful modules..."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nfrom time import time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport imageio as io\nimport cv2 as cv\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\n\nprint(\"Setup complete!\")\n\n# Input data files are available in the \"../input/\" directory.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" - Now we need to create a useful function..."},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_seconds_to_time(seconds):\n    \"\"\"\n    (float -> str)\n    \n    Converts seconds (float) in days, hours, minutes and seconds and returns a string with the result.  \n    \"\"\"\n    if seconds < float(86400) and seconds >= float(3600):\n        h, sec = divmod(int(round(seconds)), 3600)\n        m, sec = divmod(int(sec), 60)\n        return f'{int(h)} hours, {int(m)} minutes and {round(sec)} seconds'\n    \n    elif seconds < float(86400) and seconds < float(3600):\n        if seconds >= float(60):\n            m, sec = divmod(int(round(seconds)), 60)\n            return f'{int(m)} minutes and {round(sec)} seconds'\n        else:\n            return f'{round(seconds)} seconds'\n    else:\n        d, sec = divmod(int(round(seconds)), 86400)\n        return f'{int(d)} days, {convert_seconds(float(sec))}'\n\ndef diab_retin(prediction):\n    \"\"\"\n    (int -> str)\n    \n    Returns a string with information of the type of diabetic retinopathy, if present, \n    according to an integer which is the prediction given by the model.\n    \"\"\"\n    if prediction == 0:\n        return \"No diabetic retinopathy\"\n    elif prediction == 1:\n        return \"Mild non-proliferative diabetic retinopathy\"\n    elif prediction == 2:\n        return \"Moderate non-proliferative diabetic retinopathy\"\n    elif prediction == 3:\n        return \"Severe non-proliferative diabetic retinopathy\"\n    elif prediction == 4:\n        return \"Proliferative diabetic retinopathy\"\n    else:\n        raise ValueError(\"The argument should be an integer from 0 to 4, both included.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of images in the training set:\", len(os.listdir(\"../input/train_images\")))\nprint(\"Number of images in the test set:\", len(os.listdir(\"../input/test_images\")))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" - We need to one-hot encode the categorical values of the labels for the training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"url = r\"../input/train.csv\"\ntrain = pd.read_csv(url)\n\nprint(\"One-hot encoding of the provided labels...\", end = \" \")\ny_train_noncat = train[\"diagnosis\"].values\ny_train_cat = to_categorical(y_train_noncat, 5)\nprint(\"Done!\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- It's time to compress the images from the training set, compress, and normalise them.\n\n**The result will be a 4-Dimensional tensor with shape (batch_size, height, width, channels)**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_id_codes = train[\"id_code\"].tolist()\nlist_train_img = []\na = 0\n\ntimea = time() \nprint(\"Converting and adding training images to a 4-d tensor (batch_size, height, width, channels)...\")\nfor im in train_id_codes:\n    uri = glob.glob(\"../input/train_images/\" + im + \".*\")\n    image = io.imread(uri[0])\n    image = cv.resize(image, (256, 256), interpolation = cv.INTER_AREA) / 255 #Normalising...\n    list_train_img.append(image)\n    a += 1\n    if a % 500 == 0:\n        print(f\"\\t{a} images from the training set converted and added to the tensor\")\n        \ntimeb = time()\ntotal_time = timeb- timea\nprint(f\"\\nIt took {convert_seconds_to_time(total_time)} to complete the process\")\nprint(\"All images from the training set converted and added!\")\n\ntrain_im = np.asarray(list_train_img)\nprint(\"The shape of the input training set before splitting is\", train_im.shape)\nprint(\"The shape of the array containing the labels of the training set before splitting is\", y_train_cat.shape)\ndel list_train_img\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- And now let's see some examples taken from the provided training set after the whole process:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#rand_samples = np.random.randint(0, 3663, size = 3)\n\n#fig, arr = plt.subplots(1,3, figsize = (20, 10), sharey = \"all\")\n#plt.suptitle(\"3 random examples from the provided training dataset\", fontsize = 20)\n\n#arr[0].imshow(train_im[rand_samples[0]])\n#arr[0].set_title(diab_retin(y_train_noncat[rand_samples[0]]))\n#arr[0].axis(\"off\")\n#arr[1].imshow(train_im[rand_samples[1]])\n#arr[1].set_title(diab_retin(y_train_noncat[rand_samples[1]]))\n#arr[1].axis(\"off\")\n#arr[2].imshow(train_im[rand_samples[2]])\n#arr[2].set_title(diab_retin(y_train_noncat[rand_samples[2]]))\n#arr[2].axis(\"off\")\n\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Now it's time to compress the images from the test set, compress, and normalise them.\n\n**The result will be again a 4-Dimensional tensor with shape (batch_size, height, width, channels)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#NOW THE SAME FOR THE TEST SET OF IMAGES:\nurl = r\"../input/test.csv\"\ntest = pd.read_csv(url)\ntest_id_codes = test[\"id_code\"].tolist()\n\nlist_test_img = []\nb = 0\n\ntimec = time()\nprint(\"Converting and adding test images to a 4-D tensor (batch_size, height, width, channels)...\")\nfor im_test in test_id_codes:\n    uri = glob.glob(\"../input/test_images/\" + im_test + \".*\")\n    image = io.imread(uri[0])\n    image = cv.resize(image, (256, 256), interpolation = cv.INTER_AREA) / 255 #Normalising...\n    list_test_img.append(image)\n    b += 1\n    if b % 500 == 0:\n        print(f\"\\t{b} images from the test set converted and added to the tensor\")      \n\ntimed = time()\ntotal_time = timed - timec\nprint(f\"It took {convert_seconds_to_time(total_time)} to complete the process\")        \nprint(\"All images from the test set converted and added!\\n\")\n\ntest_im = np.asarray(list_test_img)\nprint(\"The shape of the test set is\", test_im.shape)\ndel list_test_img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- We can split the processed training set into random training and cross-validation subsets."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Splitting input tensor and labels into random train and cross-validation subsets...\", end = \" \")\nX_train, X_val, Y_train, Y_val = train_test_split(train_im, y_train_cat, test_size = 0.2)\nprint(\"Done!\\n\")\nprint(\"Shape of the training set inputs (X_train):\", X_train.shape)\nprint(\"Shape of the cross-validation set (X_val):\", X_val.shape)\nprint(\"Shape of the training subset labels after one-hot encoding (Y_train):\", Y_train.shape)\nprint(\"Shape of the cross-validation subset labels after one-hot encoding (Y_val):\", Y_val.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Let's save the complete training dataset, test set of images and the aforementioned subsets in the corresponding NPZ compressed files..."},{"metadata":{"trusted":true},"cell_type":"code","source":"np.savez_compressed(file = \"train_set\", X_train = X_train, Y_train = Y_train)\nnp.savez_compressed(file = \"val_set\", X_val = X_val, Y_val = Y_val)\nnp.savez_compressed(file = \"input_dataset\", train_im = train_im)\nnp.savez_compressed(file = \"dataset_labels\", dataset_labels = y_train_cat)\nnp.savez_compressed(file = \"test_set\", test_im = test_im)\n\nprint(\"Name of the file containing the full provided set of images: input_dataset.npz\")\nprint(\"Name of the file containing all the provided labels: dataset_labels.npz\")\nprint(\"Name of the file containing the training subset (inputs and labels): train_set.npz\")\nprint(\"Name of the file containing the cross-validation subset (inputs and labels): val_set.npz\")\nprint(\"Name of the file containing the test set of images: test_set.npz\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}