{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n
\n**You open doors when you open books... doors that swing wide to unlimited horizons of knowledge, wisdom, and inspiration that will enlarge the dimensions of your life.** ~ Wilferd Peterson\n
\n
"},{"metadata":{},"cell_type":"markdown","source":"1. [Introduction](#Introduction:-Kuzushiji-Recognition)\n2. [Data](#The-Data)\n3. [Preparation](#Preparation)\n4. [Useful functions](#Useful-functions)\n5. [Visualize training data](#Visualize-training-data)\n6. [Inspect single Kuzushiji Characters](#Inspect-Kuzushiji-Characters)\n7. [EDA (tSNE, SVD, UMAP)](#EDA)"},{"metadata":{},"cell_type":"markdown","source":"## Introduction: Kuzushiji Recognition\nKuzushiji is a Japanese cursive writing style that has stopped being used 150 years ago when the Japanese education system had a reformation. Kuzushiji has been used for over 1000 years. A million books have been published and more than one billion unregistered books have been written in Kuzushiji\nNow only 0.01% of Japanese natives are able to read Kuzushiji and thus it is hard to transcribe the documents into modern Japanese characters.\nThe advancements in Machine Learning and Computer Vision are the perfect opportunity to finally solve this challenge.\n\n**Our task** is to **build a model** that can **locate and classify** **Kuzushiji-characters** on images.\n\nI'm happy to help everyone, so if you have any questions or explanations weren't completely clear, don't hesitate to ask in the comments. Upvotes are as always appreciated if you learned something new :)"},{"metadata":{},"cell_type":"markdown","source":"## The Data\n\n### train.csv\n* image_id: The filename without the filextension to uniquely identify each image\n* labels: A string containing all labels for the given image. This can be used to draw a bounding box around each character on the image. The string contains each Unicode character, X-coordinate, Y-coordinate, With and Heigt. The information are space seperated.\n\n### sample_submission.csv\n\n* image_id: The filename without the filextension to uniquely identify each image\n* labels\n\n### unicode_translations.csv\nmapping unicode ID and Japanese character\n\n### train_images.zip\nTraining images. Thanks to train.csv we have the information about each Kuzushiji character on there\n\n### test_images.zip\nTesting images. Our task is to locate each character on the image and classify them afterwards."},{"metadata":{},"cell_type":"markdown","source":"TODO:\n- add more comments\n- remove all useless lines of code\n- explain SVD, tSNE and UMAP"},{"metadata":{},"cell_type":"markdown","source":"## Preparation"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline\n\n# hide warnings\nimport warnings\nwarnings.simplefilter('ignore')\n\nimport os\nimport shutil\nfrom pathlib import Path\n\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\n\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nfrom matplotlib.patches import Rectangle\n\nfrom PIL import Image, ImageDraw, ImageFont\nimport cv2\n\nimport regex as re\nimport math\nimport random\n\nfrom itertools import compress\n\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications.inception_v3 import preprocess_input\nfrom keras.utils.data_utils import GeneratorEnqueuer\n\nfrom tqdm import tnrange, tqdm_notebook\n\nfrom sklearn.manifold import TSNE\nfrom sklearn.decomposition import TruncatedSVD","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Setup font, so that characters can be displayed"},{"metadata":{"trusted":true},"cell_type":"code","source":"# download a font that can display the characters\n\n# From https://www.google.com/get/noto/\n!wget -q --show-progress https://noto-website-2.storage.googleapis.com/pkgs/NotoSansCJKjp-hinted.zip\n!unzip -p NotoSansCJKjp-hinted.zip NotoSansCJKjp-Regular.otf > NotoSansCJKjp-Regular.otf\n!rm NotoSansCJKjp-hinted.zip\n\n# enable font for matplotlib\nimport matplotlib.font_manager as font_manager\npath = './NotoSansCJKjp-Regular.otf'\nprop = font_manager.FontProperties(fname=path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save and inspect input directory\nINPUT = Path(\"../input/kuzushiji-recognition\")\nprint(os.listdir(INPUT))\n\n# save and inspect sub-folders of input directory\nTEST = INPUT/'test_images'\nTRAIN = INPUT/'train_images'\nprint(os.listdir(TEST)[:3])\nprint(os.listdir(TRAIN)[:3])\n\n#Check the number of training and testing images\nprint(f\"images in training dataset: {len(os.listdir(TRAIN))}\")\nprint(f\"images in test dataset: {len(os.listdir(TEST))}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Inspect train.csv\ntrain_df = pd.read_csv(INPUT/'train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# setup an image_id test variable\ntest_img_id = train_df.image_id[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Useful functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def toPath(string):\n ''' image_id to the path to image '''\n if \".jpg\" not in string:\n string = string + \".jpg\"\n return string\n\ndef toID(string):\n ''' image path to image_id '''\n if string[-4:] ==\".jpg\":\n string = string[:-4]\n return string\n\nprint(toPath(\"0123\"))\nprint(toID(\"0123.jpg\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# In the training data, we get an entire string with all the characters in it and that needs to be splitted\n\n# new list, every element is one char + all the information needed to create the bounding box\ndef splitEachChar(string):\n string = str(string)\n string = (re.findall(r\"(?:\\S*\\s){5}\", string))\n return [line[:-1]for line in string]\n\n# new list, split everything by a blank\ndef splitEachInformation(string):\n string = str(string)\n string = string.split(\" \")\n return string\n \n \n \nprint(splitEachChar(train_df.labels[0])[:2])\nprint(splitEachInformation(train_df.labels[0])[:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_unicodes(string):\n \"\"\"function to get all unicode chars from a string with regex\"\"\"\n string = str(string)\n return re.findall(r'U[+][\\S]*', string)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getImageSize(image):\n \"\"\"returns the image size given the image_id or path to the image\"\"\"\n path = toPath(image)\n width, height = Image.open(TRAIN/path).size\n return [width, height]\n\ngetImageSize(test_img_id)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unicode_map = {codepoint: char for codepoint, char in pd.read_csv(INPUT/'unicode_translation.csv').values}\nunicode_list = list(unicode_map)\n\ndef unicodeToCharacter(unicode):\n ''' turns the unicode into the actual character '''\n return unicode_map[unicode]\n\n# unicode to int conversion and the other way around\n# unique identifier for every unicode character\ndef unicodeToInt(unicode):\n return unicode_list.index(unicode)\n\ndef intToUnicode(integer):\n return unicode_list[integer]\n\ntest_unicode = unicode_list[10]\n\nprint(test_unicode)\nprint(unicodeToCharacter(test_unicode))\nprint(unicodeToInt(test_unicode))\nprint(intToUnicode(10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def isUnicode(string):\n ''' check whether the passed string is a unicode or not '''\n string = string.strip()\n if re.match(\"^U\\+\\w{4,5}$\", string):\n return(True)\n else:\n return(False)\n \n\ntestUnicode1 = intToUnicode(10)\ntestUnicode2 = intToUnicode(20)\n\nprint(isUnicode(testUnicode1))\nprint(isUnicode(testUnicode2))\nprint(isUnicode(testUnicode1+\"abc\"))\nprint(isUnicode(testUnicode2+\" \"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Function to display images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def displayImage(filepath=None, directory=None, image_id=None):\n \"\"\"\n display one image with matplotlib\n \n Parameters:\n - either specify the entire filepath or (the direcory and the image_id)\n \n Returns:\n - matplotlib.pyplot figure of the image\n \"\"\"\n \n if filepath == None:\n if (directory == None) and (image_id==None):\n print(\"path to file not specified\")\n return None\n else:\n filepath=directory/toPath(image_id)\n \n plt.figure(figsize=(15,15))\n this_img = Image.open(filepath)\n plt.imshow(this_img)\n return plt\n\ndisplayImage(directory=TRAIN, image_id=test_img_id)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def displayRandomImages(directory, paths=None , rows=3, columns=3):\n \"\"\"\n display random images from a folder\n \n parameters:\n - directory (string or Path) of images\n - paths (string of Path) of images inside of directory\n that should be viewed. If not specified, these are\n all files inside of the directory.\n - rows (int): the number of rows that should be displayed\n - columns (int): the number of columns that should be displayed\n \"\"\"\n fig = plt.figure(figsize=(20, 20))\n \n # if path is not specified, display all files in directory\n if paths == None:\n paths = os.listdir(directory)\n \n for i in range(1, rows*columns + 1):\n randomNumber = random.randint(0, len(paths)-1)\n image = Image.open(directory/paths[randomNumber])\n fig.add_subplot(rows, columns, i)\n plt.imshow(image, aspect='equal')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## inspect unicode_translation.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"unicode_df = pd.read_csv(INPUT/'unicode_translation.csv')\ndisplay(unicode_df.head(6))\nprint(len(unicode_df))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 4787 unique characters. But how many of those are in the training set?"},{"metadata":{"trusted":true},"cell_type":"code","source":"#concatenate all labels to one string\nall_labels = train_df.labels.str.cat(sep=\" \")\n\n# get all unicodes in that string\nall_unicodes = get_unicodes(all_labels)\n\n# get the number of unique values from all unicodes\nlen(set(all_unicodes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only 4212 of the 4787 characters are in the training set. Thus some characters definitely can't be predicted."},{"metadata":{},"cell_type":"markdown","source":"## Display image data"},{"metadata":{},"cell_type":"markdown","source":"### Inspect random image from testset"},{"metadata":{"trusted":true},"cell_type":"code","source":"displayRandomImages(TRAIN)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Display images where label is NaN in training set "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Check whether there are NaN columns in the training set\ntrain_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images_nan_labels = train_df[train_df.isna().labels]['image_id'].tolist()\nimages_nan_paths = [str(label)+\".jpg\" for label in images_nan_labels]\ndisplayRandomImages(directory=TRAIN, paths=images_nan_paths)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Images with NaN labels are either images or empty pages"},{"metadata":{},"cell_type":"markdown","source":"### Display single Kuzushiji Characters\nTo inspect all characters in the dataset you would usually have to cut out each character. To speed up the time to run the notebook, I created a [notebook](https://www.kaggle.com/christianwallenwein/fastest-way-to-crop-all-images) that cuts out each character. I saved the results in a seperate [datataset](https://www.kaggle.com/christianwallenwein/kuzushiji-characters)"},{"metadata":{"trusted":true},"cell_type":"code","source":"CHAR = Path(\"../input/kuzushiji-characters\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"displayRandomImages(CHAR)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize training data with bounding boxes and unicodes"},{"metadata":{"trusted":true},"cell_type":"code","source":"# take the image_id return an image with bounding boxes around each character\n# image_id is the filename without the file extension (in this case .jpg)\n\n# get all the characters and the position of the bounding boxes for an image\ndef getLabels(image_id):\n allLabels = train_df.loc[train_df[\"image_id\"]==image_id].labels[0]\n allLabels = np.array(allLabels.split(\" \")).reshape(-1, 5)\n return allLabels\n\ndef drawBoxAndText(ax, label):\n codepoint, x, y, w, h = label\n x, y, w, h = int(x), int(y), int(w), int(h)\n rect = Rectangle((x, y), w, h, linewidth=1, edgecolor=\"r\", facecolor=\"none\")\n ax.add_patch(rect)\n ax.text(x+w+25, y+(h/2)+20, unicodeToCharacter(codepoint),\n fontproperties=prop,\n color=\"r\",\n size=16)\n return ax\n\ndef displayTrainingData(image_id):\n labels = getLabels(image_id)\n plt = displayImage(directory=TRAIN, image_id=image_id)\n ax = plt.gca()\n\n for label in labels:\n ax = drawBoxAndText(ax, label)\n \n \ndisplayTrainingData(test_img_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EDA\nbased on [this](https://www.kaggle.com/aakashnain/kmnist-mnist-replacement)\n### Preparation"},{"metadata":{"trusted":true},"cell_type":"code","source":"noOfChars = 10\nnoOfSamplesPerChar = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"width = 60\nheight = 80","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def filenameToUnicodeInt(string):\n '''\n filename to integer representing a unicode\n '''\n unicode = string.split(\"_\")[0]\n unicodeInteger = unicodeToInt(unicode)\n return unicodeInteger","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"char_filenames = os.listdir(CHAR)\nchars = [filenameToUnicodeInt(filename) for filename in char_filenames]\n\nfrom collections import Counter\ncountAll = Counter(chars)\n\ndef getNmostCommonCharacters(n=10, countAll=countAll):\n \"\"\"\n get a list of the most common characters in the Kuzushiji dataset\n \"\"\"\n NmostCommon = countAll.most_common(n)\n NmostCommon = [unicodeID for unicodeID,frequency in NmostCommon]\n return NmostCommon","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getPathsFromCharID(char_id, noOfSamples=100):\n \"\"\"\n get n(=noOfSamples) paths to chars for every char in char_id\n \"\"\"\n char_id = int(char_id)\n isCharIdList = [filenameToUnicodeInt(filename)==char_id for filename in os.listdir(CHAR)]\n allCharIdPaths = list(compress(os.listdir(CHAR), isCharIdList))\n return allCharIdPaths[:noOfSamples]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths = [getPathsFromCharID(charID, noOfSamplesPerChar) for charID in getNmostCommonCharacters(noOfChars)]\n# flatten the list\npaths = sum(paths, [])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create empty images and empty labels\nimages = np.zeros(shape=(noOfChars * noOfSamplesPerChar, width*height), dtype=np.uint8)\nlabels = np.zeros(shape=(noOfChars * noOfSamplesPerChar,), dtype=np.uint8)\n\n# flatten images + make images black and white\nfor index, imageName in enumerate(paths):\n filepath = str(CHAR/imageName)\n img = cv2.imread(filepath)\n img = cv2.resize(img, (width, height))\n img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n img = img.reshape(-1, width*height)\n images[index] = img\n labels[index] = filenameToUnicodeInt(imageName)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uniqueLabels = np.array([int(i) for i in list(set(labels))], dtype=np.uint8)\nuniqueUnicodeLabels = [unicodeToCharacter(intToUnicode(unicode)) for unicode in uniqueLabels]\nuniqueUnicodeLabels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### tSNE with 10 most common characters in dataset\n\nto learn more about tSNE, watch [this video](https://www.youtube.com/watch?v=NEaUSP4YerM)"},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne = TSNE(n_components=2, perplexity=30)\nrandom_train_2D = tsne.fit_transform(images)\nfig = plt.figure(figsize=(10, 8))\nfor i in uniqueLabels:\n sns.scatterplot(random_train_2D[labels == i, 0], \n random_train_2D[labels == i, 1], \n label=i, s=18)\n \nplt.title(\"Visualizating embeddings from the 10 most common Kuzushiji characters using tSNE\", fontsize=16)\nplt.legend(uniqueUnicodeLabels,prop=prop)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### SVD with 10 most common characters dataset\nto learn more about tSNE, watch [this video](https://www.youtube.com/watch?v=CQbbsKK1kus)"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10, 8))\nX_pca = TruncatedSVD(n_components=2).fit_transform(images)\nfor i in uniqueLabels:\n sns.scatterplot(X_pca[labels == i, 0], \n X_pca[labels == i, 1], \n label=i, s=18)\n \nplt.title(\"Principal Component projection of the 10 most common Kuzushiji characters\", fontsize=16)\nplt.legend(uniqueUnicodeLabels,prop=prop)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### UMAP with 10 most common characters in dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"import umap\nreduce = umap.UMAP()\nembedding = reduce.fit_transform(images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10, 8))\n\nfor i in uniqueLabels:\n sns.scatterplot(embedding[labels == i, 0], \n embedding[labels == i, 1], \n label=i, s=18)\n\nplt.title(\"UMAP of the 10 most common Kuzushiji characters\", fontsize=16)\nplt.legend(uniqueUnicodeLabels,prop=prop)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fun Fact\n\n[source](https://www.learnsomethingeveryday.co.uk/#/26-march-2019/)","attachments":{}}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}