{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Gastly Resliced Image\n## Of [SIIM-ISIC-Melanoma-classification](https://www.kaggle.com/c/siim-isic-melanoma-classification)\n\n> This competition offers very fine grained source image, it's a shame we have to all of them into less than 512x512 to put it into the model. And in some common sense, the texture and pattern of the such detail really should matter.\n\n> In this notebook, we relice image in 4 shots of zoom in version\n\n> The output data of this notebook is made into [this dataset](https://www.kaggle.com/raynardj/gastly-detailed-512x-4shots)\n\n\n**WARNING** THIS NOTEBOOK CONTAINS VERY UNCOMFORTABLE GRAPHIC DETAIL ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Imports\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install -q forgebox","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from forgebox.imports import *\nfrom joblib import Parallel, delayed\nfrom random import choice","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Locations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT = Path('/kaggle/input')\nDATA = INPUT/\"siim-isic-melanoma-classification\"\nTRAIN = DATA/\"jpeg\"/'train'\nTEST  = DATA/\"jpeg\"/'test'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA.ls()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN.ls()[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST.ls()[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helpers\nOpen image in train folder","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef open_img(path,parent = TRAIN):\n    return Image.open(parent/path).convert('RGB')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"open_img('ISIC_2679975.jpg').resize((300,300))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# resize/ rotate image in to the proper range, with w>h is a must\ndef proper_size(img):\n    w,h = img.size\n    if h>w:\n        img = img.transpose(Image.ROTATE_90)\n    check = False\n    while check == False:\n        if min(w,h)>1599:\n            img = img.resize((w//2,h//2))\n        if min(w,h)<256:\n            img = img.resize((w*2,h*2))\n            \n        w,h = img.size\n        check = True\n    return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image size EDA\n\nAs we can see by the following value counts, most original image has original size way bigger than our usual model can handle.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_meta_df = pd.DataFrame(dict(fname = TRAIN.ls()))\n# test_meta_df = pd.DataFrame(dict(fname = TEST.ls()))\n\n# train_meta_df['img_size'] = train_meta_df.fname.apply(lambda fname: open_img(fname).size)\n\n# test_meta_df['img_size'] = test_meta_df.fname.apply(lambda fname: open_img(fname,parent=TEST).size)\n\n# train_meta_df.vc(\"img_size\").head(20)\n\n# test_meta_df.vc(\"img_size\").head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\nimg_size\n(6000, 4000)\t14703\n(1872, 1053)\t7534\n(640, 480)\t4147\n(5184, 3456)\t3418\n(3264, 2448)\t1483\n(4288, 2848)\t729\n(2592, 1936)\t674\n(3888, 2592)\t140\n(4032, 3024)\t84\n(2317, 2317)\t29\n(2848, 4288)\t17\n(3456, 5184)\t16\n(4608, 3072)\t10\n(1761, 1761)\t7\n(1775, 1775)\t7\n(2329, 2329)\t6\n(1763, 1763)\t6\n(1769, 1769)\t5\n(1773, 1773)\t4\n(3872, 2592)\t4\n```","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Helpers for 4-shots zoom in ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def find_center(img,size = 256):\n    w,h = img.size\n    left = w//2-size//2\n    upper = h//2-size//2\n    right = left+size\n    lower = upper+size\n    return img.crop((left, upper, right, lower))\n\ndef find_ratio(img,size = 256,ratio = .3):\n    w,h = img.size\n    h2 = int(h*ratio)\n    upper = (h-h2)//2\n    lower = upper+h2\n    \n    w2 = int(w*ratio)\n    wpad = (w-w2)//2\n    start = choice(list(range(max(1,w2-h2))))\n    left = wpad+start\n    right = left+h2\n\n    return img.crop((left, upper, right, lower)).resize((size,size))\n\ndef combine_4in1(*imgs,size = 256):\n    \"\"\"\n    combining 4 images of 'size' into image (2*size x 2*size)\n    \"\"\"\n    dst = Image.new('RGB', (size*2,size*2))\n    dst.paste(imgs[0], (0, 0))\n    dst.paste(imgs[1], (0, size))\n    dst.paste(imgs[2], (size, 0))\n    dst.paste(imgs[3], (size, size))\n    return dst\n\ndef different_scale_crop(img,size=512):\n    \"\"\"\n    process for 4 shots and combine into 1\n    \"\"\"\n    img = proper_size(img)\n    return combine_4in1(*map(lambda i:find_ratio(img,size=size//2,ratio = 1-2*(i/10)),range(1,5)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let's take some preview","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(100,150):\n    img = different_scale_crop(open_img(TRAIN.ls()[i*2]),size = 512)\n    display(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create new dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"HOME = Path(\".\")\n\nTRAIN_SAVE = HOME/\"img/train\"\nTEST_SAVE = HOME/\"img/test\"\n\n# !mkdir -p {TRAIN_SAVE}\n# !mkdir -p {TEST_SAVE}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_files = TRAIN.ls()\ndef process_file_train(fname):\n    img = open_img(fname,parent = TRAIN)\n    img = different_scale_crop(img,size = 512)\n    newname = fname.split(\".\")[0]+\".jpg\"\n    img.save(TRAIN_SAVE/f\"{newname}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_files = TEST.ls()\ndef process_file_test(fname):\n    img = open_img(fname,parent = TEST)\n    img = different_scale_crop(img,size = 512)\n    newname = fname.split(\".\")[0]+\".jpg\"\n    img.save(TEST_SAVE/f\"{newname}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Process the files in parallel\n\nYou don't have to run this step, you can use [this dataset here is the exact output](https://www.kaggle.com/raynardj/gastly-detailed-512x-4shots), you can use your favourite contest kernel on this dataset as the image_id all match","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parallel(n_jobs=8)(delayed(process_file_train)(fname) for fname in train_files)\n\n# Parallel(n_jobs=8)(delayed(process_file_test)(fname) for fname in test_files)\n\n# !ls -l {TRAIN_SAVE}|wc -l\n\n# !ls -l {TEST_SAVE}|wc -l","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compress & Cleaning up","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# !tar -czvf train_data.tar.gz {TRAIN_SAVE} > /dev/null\n\n# !rm -rf {TRAIN_SAVE}\n\n# !tar -czvf test_data.tar.gz {TEST_SAVE} > /dev/null\n\n# !rm -rf {TEST_SAVE}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}