{"cells":[{"metadata":{},"cell_type":"markdown","source":"***The following is just a rough draft/incomplete solution to the problem. I don't know if I'll be able to complete on time or not becuase of University stuff. But here it is as of now. I put heavy emphasis on the preprocessing. A lot of the preprocessing is stuff I learnt from my college course \"Digital Image Processing\".***","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n        print(dirname)\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Importing the training CSV file**","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"base_path = \"../input/kuzushiji-recognition/\"\ntrain = pd.read_csv(\"../input/kuzushiji-recognition/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's observe the labels of any one element","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train[\"labels\"][0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On observing we see that, each of the of labels gives us bounding boxes in the format - \"class id\", \"X_c\", \"Y_c\", \"width\", \"height\". ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at one of the training images. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nimport matplotlib.pyplot as plt\nplt.figure(figsize=(20,20))\ns = train[\"image_id\"][0]\nimg_0_path = base_path + \"train_images/\" + s + \".jpg\"\nimg_0 = cv2.imread(img_0_path)\nimg_0 = cv2.cvtColor(img_0, cv2.COLOR_BGR2RGB)\nplt.imshow(img_0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We observe some issues, which have been stated in the problem statement also.\n\n1. You can occasionally see through especially thin paper and read characters from the opposite side of the page. Those characters should also be ignored.\n2. Also, the annotations which are in smaller font size, they need to be ignored.\n\nThe first problem can be tackled by using thresholding. Let's see what happens thresholding is applied.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#preprocessing on a single image\n\nplt.figure(figsize=(20,20))\nimg_0_orig = cv2.imread(img_0_path)\nimg_0_orig = cv2.cvtColor(img_0_orig, cv2.COLOR_BGR2RGB)\nimg_0 = cv2.imread(img_0_path)\nimg_0 = cv2.cvtColor(img_0, cv2.COLOR_BGR2GRAY)\nblur = cv2.GaussianBlur(img_0,(3,3),0)\nsharp_mask = np.subtract(img_0, blur)\nimg_0 = cv2.addWeighted(img_0,1, sharp_mask,10, 0)\nret,th = cv2.threshold(blur,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)\nkernel_1 = np.ones((5,5),np.uint8)\nkernel_2 = np.ones((1,1),np.uint8)\nopening = cv2.morphologyEx(th, cv2.MORPH_OPEN, kernel_1)\nclosing = cv2.morphologyEx(opening, cv2.MORPH_CLOSE, kernel_2)\nmask = cv2.bitwise_not(closing)\nmask = cv2.cvtColor(closing, cv2.COLOR_GRAY2RGB)\nimg = cv2.add(img_0_orig,mask)\nblur_1 = cv2.GaussianBlur(img, (13,13), 0)\nsharp_mask_1 = np.subtract(img,blur_1)\nsharp_mask_1 = cv2.GaussianBlur(sharp_mask_1, (7,7), 0)\nimg = cv2.addWeighted(img,1,sharp_mask_1,-10, 0)\nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\nplt.imshow(img)\nprint(img_0_orig.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So before putting any image we need to preprocess our image so that the print on the back page is not passed when the model is predicting.\n\nIntuitively, I can see that the intensity values for that are higher than the Kuzushiji print on a page so, by using Otsu's Thresholding and I do binary thresholding of the images and I get the above result. The binary thresholding involves passing the image through a (3,3) Gaussian filter and then Otsu's thresholding is applied. On observing, we were right and we can no more see any from the opposite side of the page. To further improve the quality of the input, the image is passed through 2 morphological transformations, i.e. opening using a (5,5) one's kernel and a (1,1) one's closing kernel.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**End of all preprocessing**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pwd = os.getcwd()\nprint(pwd)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Making a list of all training images**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_imgs_path = base_path + \"train_images/\"\nfilelist = os.listdir(train_imgs_path)\n#print(filelist[:5])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainlist = train[\"image_id\"].tolist()\ntrainlist = [x + \".jpg\" for x in trainlist]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(trainlist[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_out = train[\"labels\"].tolist()\n#train_out[:5]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"yolo_labels = []\nimg_label = []\nfor l in train_out:\n    voc_out = str(l).split()\n    for i in range(len(voc_out)//5):\n        start_idx = 5*i\n        img_label.append(voc_out[start_idx:start_idx+5])\n    yolo_labels.append(img_label)\n    img_label = []\n#print(yolo_labels[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainlist[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def pre(Image):\n    img_0_orig = cv2.imread(Image)\n    img_0_orig = cv2.cvtColor(img_0_orig, cv2.COLOR_BGR2RGB)\n    img_0 = cv2.imread(Image)\n    img_0 = cv2.cvtColor(img_0, cv2.COLOR_BGR2GRAY)\n    blur = cv2.GaussianBlur(img_0,(3,3),0)\n    sharp_mask = np.subtract(img_0, blur)\n    img_0 = cv2.addWeighted(img_0,1, sharp_mask,10, 0)\n    ret,th = cv2.threshold(blur,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)\n    kernel_1 = np.ones((5,5),np.uint8)\n    kernel_2 = np.ones((1,1),np.uint8)\n    opening = cv2.morphologyEx(th, cv2.MORPH_OPEN, kernel_1)\n    closing = cv2.morphologyEx(opening, cv2.MORPH_CLOSE, kernel_2)\n    mask = cv2.bitwise_not(closing)\n    mask = cv2.cvtColor(closing, cv2.COLOR_GRAY2RGB)\n    img = cv2.add(img_0_orig,mask)\n    blur_1 = cv2.GaussianBlur(img, (13,13), 0)\n    sharp_mask_1 = np.subtract(img,blur_1)\n    sharp_mask_1 = cv2.GaussianBlur(sharp_mask_1, (7,7), 0)\n    img = cv2.addWeighted(img,1,sharp_mask_1,-10, 0)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n    return img\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(pre(train_imgs_path+trainlist[0]), cmap = 'gray')\nprint(pre(train_imgs_path+trainlist[0]).shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#yolo_model = make_yolov3_model()\n#yolo_model.load_weights(\"yolo.h5\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#yolo_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uni_lab = pd.read_csv(\"../input/kuzushiji-recognition/unicode_translation.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#uni_lab.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#uni_lab.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uni_list = uni_lab[\"Unicode\"].to_list()\n#print(uni_list[:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Saving all of these in a class.names file**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('class.names', 'w') as f:\n    for item in uni_list:\n        f.writelines(item+\"\\n\")\n#!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#open('class.names', 'w').close()\nwith open(\"class.names\", \"r\") as f_r:\n    test_list = f_r.readlines()\n#print(test_list[0][:-1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Creating a dictionary to map class_id to class_name**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"uni_dict = ((uni_list[i],i) for i in range(len(uni_list)))\nuni_dict = dict(uni_dict)        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#uni_dict[\"U+0031\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Changing the classnames to class ids in the yolo_labels list including the groundtruths**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for labels in yolo_labels:\n    for label in labels:\n        label[0] = uni_dict[label[0]]\n        label[1:5] = list(map(int, label[1:5]))\nyolo_labels[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Converting the annotations into VOC format*","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(trainlist)):\n    labels = yolo_labels[i]\n    img = trainlist[i]\n    img_path = base_path + \"train_images/\" + img\n    image = Image.open(img_path)\n    w, h = image.size\n    for label in labels:\n        id_1 = label[0]\n        label[0],label[2] = list(map(int,[label[1]-label[3]/2,label[1]+label[3]/2]))\n        label[1],label[3] = list(map(int,[label[2]-label[4]/2,label[2]+label[4]/2]))\n        label[4] = id_1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#yolo_labels[:3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#str(yolo_labels[0][0][0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Creating a dictionary which maps image ids to bounding boxes/detections**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lines = [train_imgs_path+img_name for img_name in trainlist]\n#print(lines[:3])\nannot = []\nfor labels in yolo_labels:\n    ann_fin = \"\"\n    for label in labels:\n        ann = \" \" + str(label[0])+','+str(label[1])+','+str(label[2])+','+str(label[3])+','+str(label[4])\n        ann_fin += ann\n    annot.append(ann_fin)\n#print(annot[0])\nfor i in range(len(lines)):\n    lines[i] = lines[i]+annot[i]\n#print(lines[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is all I could do before the end of the competition. Hope this helps someone.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}