{"cells":[{"metadata":{},"cell_type":"markdown","source":"To explore and analyze all the characters in the Kuzushiji Comeptition, you would have to cut out each character. I wanted to save you some time, so I created this notebook and found a pretty fast way to cut out each char.<br> To save you even some more time I saved the computed results of this notebooks in [this dataset](https://www.kaggle.com/christianwallenwein/kuzushiji-characters). If you want to analyze the Kuzushiji Characters, simple use the dataset."},{"metadata":{},"cell_type":"markdown","source":"## Setup"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport shutil\nfrom pathlib import Path\n\nimport pandas as pd\nimport numpy as np\n\nfrom PIL import Image\nimport cv2\n\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tnrange, tqdm_notebook\n\nimport time\nimport random\n\n# hide warnings\nimport warnings\nwarnings.simplefilter('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# all important folders\nINPUT = Path(\"../input\")\nTEST = INPUT/'test_images'\nTRAIN = INPUT/'train_images'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CHARS = Path(\"../chars\")\ntry:\n    os.makedirs(CHARS)\nexcept:\n    pass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(INPUT/\"train.csv\"); train_df.tail(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Crop all images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def cropImage(labels, loop1_index):\n    if isinstance(labels, float):\n        return None\n\n    filepath = str(f\"{str(TRAIN)}/{row.image_id}.jpg\")\n    \n    img = cv2.imread(filepath)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\n    labels = labels.split(\" \")\n\n#     change unicode char to integer representing the class\n#     labels[::5] = map(unicodeToInt, labels[::5])\n\n\n    # to get the highest speed, we use numpy for the cropping of the images\n    # numpy doesn't support strings in their ndarrays\n    # that's why we move unicodes outside of labels\n    unicode = labels[::5]\n    del labels[::5]\n    \n\n    labels = np.array(labels, dtype=np.int16)\n\n    labels = labels.reshape(-1, 4)\n\n    labels[:, 2] = np.sum(a=labels[:,[0,2]], axis=1)\n    labels[:, 3] = np.sum(a=labels[:,[1,3]], axis=1)\n\n    [Image.fromarray(img[label[1]:label[3], label[0]:label[2]]).save(f\"{CHARS}/{unicode[loop2_index]}_{loop1_index}-{loop2_index}.jpg\") for loop2_index, label in enumerate(labels)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The **first cell** will always work no matter which operating system. Use this one **on Windows**.\n<br>Linux has better support for Pythons multiprocessing library. Therefore use the **second cell** if you are **on Linux**.\n\nbtw Kaggle Kernels are based on Linux"},{"metadata":{"trusted":true},"cell_type":"code","source":"# pbar = tqdm_notebook(total=len(os.listdir(TRAIN)))\n# for loop1_index, (list_index, row) in enumerate(train_df.iterrows()):\n#     cropImage(row.labels, loop1_index)\n#     pbar.update(1)\n# pbar.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from multiprocessing import Process, current_process\n\nprocesses = []\n\npbar = tqdm_notebook(total=len(os.listdir(TRAIN)))\nfor loop1_index, (list_index, row) in enumerate(train_df.iterrows()):\n    process = Process(target=cropImage, args=(row.labels, loop1_index))\n    pbar.update(1)\n    processes.append(process)\n    \n    process.start()\npbar.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(os.listdir(CHARS))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create ZIP-file from all chars"},{"metadata":{"trusted":true},"cell_type":"code","source":"shutil.make_archive(\"chars\", \"zip\", CHARS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Useful functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def displayRandomImagesFromFolder(directory):\n    images = os.listdir(directory)\n\n    rows = 3\n    columns = 3\n    fig = plt.figure(figsize=(20, 20))\n\n    for i in range(1, rows*columns + 1):\n        randomNumber = random.randint(0, len(images)-1)\n        image = Image.open(directory/images[randomNumber])\n        fig.add_subplot(rows, columns, i)\n        plt.imshow(image, aspect='equal')\n    \n    plt.show()\n    \n    \ndisplayRandomImagesFromFolder(CHARS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unicode_df = pd.read_csv(INPUT/'unicode_translation.csv')\ndisplay(unicode_df.head(10))\nprint(len(unicode_df))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = \"U+5DDE_2802-291.jpg\"\ndef getInfoFromFilename(filename):\n    unicode, rest = filename.split(\"_\")\n    char = unicode_df[unicode_df.Unicode == unicode].char.values[0]\n    rest = rest.split(\".\")[0]\n    image_nbr = int(rest.split(\"-\")[0])\n    image_filename = os.listdir(TRAIN)[image_nbr]\n    print(f\"This image displays the following char: {char} \\nIts unicode is: {unicode} \\nThis char has been taken from {image_filename}\")\n    \n    \ngetInfoFromFilename(x)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}