{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Introduction

\n","metadata":{}},{"cell_type":"markdown","source":"

In the process of solving the problem we are facing in this competition, there are various paths to be taken.

\n

I believe that for a future model, a narrower search space can be obtained by narrowing down the translation process to a smaller subset of options, which include only the observable elements in the image stated by letter.

\n


\n

This notebook presents a suggestion for an image processing pipeline that can retrieve the letters from a given chemical image.

\n


","metadata":{}},{"cell_type":"markdown","source":"

Libraries And Utilities

\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom collections import Counter, defaultdict\nimport cv2, os\nimport skimage.io as io\nfrom tqdm.auto import tqdm\ntqdm.pandas()\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.rc('figure',figsize=(18,11))\nplt.rc('image',cmap='Blues')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Loading The Data

\n","metadata":{}},{"cell_type":"code","source":"labels = pd.read_csv('../input/bms-molecular-translation/train_labels.csv')\nss = pd.read_csv('../input/bms-molecular-translation/sample_submission.csv', index_col = 0)\n\nlabels['path'] = labels['image_id'].progress_apply(\n lambda x: \"../input/bms-molecular-translation/train/{}/{}/{}/{}.png\".format(\n x[0], x[1], x[2], x))\nlabels.head()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Going Through The Logic

\n","metadata":{}},{"cell_type":"markdown","source":"

Our letter retrieval pipeline will consist of 3 main steps that will be explained in the following section:

\n
    \n
  1. Preprocess the image using morphological transformations.
    2. \n
  2. Detect contours in the processed image
    3. \n
  3. Remove contours that deviate from our criteria
    4. \n
","metadata":{}},{"cell_type":"markdown","source":"

1 - Preprocessing The Image

\n

We will use dilation and erosion to fix the inconsistency in lines and the noise accompanied by most of the images.\nThe entire purpose of the preprocessing stage is to create better structures for our contour detection stage.

\n


","metadata":{}},{"cell_type":"code","source":"plt.subplot(2,2,1)\nplt.title('Before Preprocessing',fontsize=16,fontweight='bold')\nplt.imshow(plt.imread(labels.path[14]))\nplt.subplot(2,2,2)\nt1 = cv2.erode(plt.imread(labels.path[14]),np.ones((2,2)))\nplt.title('After Preprocessing',fontsize=16,fontweight='bold')\nplt.imshow(t1)\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

2 - Detecting Contours

\n","metadata":{}},{"cell_type":"code","source":"plt.subplot(2,2,1)\nplt.title('Preprocessing Image',fontsize=16,fontweight='bold')\nplt.imshow(t1)\nplt.subplot(2,2,2)\n\nt1 = 255 + t1\ncontours, hierarchy = cv2.findContours(t1.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE )\nimg = cv2.cvtColor(np.zeros_like(t1),cv2.COLOR_GRAY2RGB)\nwc = cv2.drawContours(img, contours, -1, (0,255,0), 2)\n\nplt.title('Detected Contours Colored Green',fontsize=16,fontweight='bold')\nplt.imshow(wc)\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3 - Process Detected Contours

\n\n

We will make the following assumption, every letter present in a given image can be bound by a rectangle which tends to be a square, i.e., the difference between orthogonal sides tends to be equal to zero.

\n

We also know that we can bound a square with a minimum-sized circle which will grow larger in a rectangle case.

\n

Using these assumptions, we will bound every contour we find with a rectangle and a circle; we will then condition the validity of a contour based on the bounding circle area and the difference between orthogonal sides.

","metadata":{}},{"cell_type":"code","source":"cnts = []\nbarea = lambda x: np.linalg.norm(x[0]-x[1])*np.linalg.norm(x[0]-x[3])\nbsd = lambda x: np.abs(np.linalg.norm(x[0]-x[1])-np.linalg.norm(x[0]-x[3]))\nfor cnt in contours:\n (x,y),radius = cv2.minEnclosingCircle(cnt)\n rect = cv2.minAreaRect(cnt)\n box = cv2.boxPoints(rect)\n\n #if contour bonding box sides are not more different then alpha\n #and the bounding circle area is between beta1 and beta 2\n #save contour else ignore it\n if (20Letter Retrieval Pipeline\n","metadata":{}},{"cell_type":"code","source":"def clean_image(img,alpha = 10,beta1=20,beta2=450,pr_mode=1,kernel_size=2):\n \n \"\"\"\n \n Parameters\n ----------\n img : np.array\n an image of a chemical compound\n alpha : float\n the maximum allowed difference between bouding rectangle sides ,0 differnce means all sides are equal i.e square\n beta1 : float\n the minimum area of the circle which bound the contour found \n beta2 : int\n the maximum area of the circle which bound the contour found \n pr_mode : int\n preprocessing preset value can be 1,2 or 3 \n kernel_size : int\n kernel size for morphological operations in preprocessing \n \n \"\"\"\n \n t1 =img.copy().astype(np.uint8)\n #preprocess the image so better contours can be detected\n if pr_mode ==1 :\n t1 = cv2.erode(t1,np.ones((kernel_size,kernel_size)))\n elif pr_mode == 2:\n t1 = cv2.morphologyEx(t1,cv2.MORPH_OPEN,np.ones((kernel_size,kernel_size)))\n else:\n t1 = cv2.morphologyEx(t1,cv2.MORPH_GRADIENT,np.ones((kernel_size,kernel_size)))\n\n\n t1 = t1 + 255\n \n #create canvas to draw contours on\n img = cv2.cvtColor(np.zeros_like(t1),cv2.COLOR_GRAY2RGB)\n \n #find contours\n contours, hierarchy = cv2.findContours(t1.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE )\n\n cnts = []\n \n #function for box area calculation\n barea = lambda x: np.linalg.norm(x[0]-x[1])*np.linalg.norm(x[0]-x[3])\n #function for box side difference calculation\n bsd = lambda x: np.abs(np.linalg.norm(x[0]-x[1])-np.linalg.norm(x[0]-x[3]))\n\n #preprocess contours\n for cnt in contours:\n (x,y),radius = cv2.minEnclosingCircle(cnt)\n rect = cv2.minAreaRect(cnt)\n box = cv2.boxPoints(rect)\n \n #if contour bonding box sides are not more different then alpha\n #and the bounding circle area is between beta1 and beta 2\n #save contour else ignore it\n if (beta1Examples\n","metadata":{}},{"cell_type":"code","source":"imgs = [10,5,55]\nfig,axs = plt.subplots(2,3)\nfor ix,img in enumerate(imgs):\n axs[1,ix].imshow(clean_image(plt.imread(labels.path[img])))\n axs[0,ix].imshow((plt.imread(labels.path[img])))\n ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Extracting Only The Letters

\n","metadata":{}},{"cell_type":"code","source":"plt.subplot(2,2,1)\nplt.imshow((plt.imread(labels.path[2])))\nplt.subplot(2,2,2)\nplt.imshow(clean_image(plt.imread(labels.path[2])))\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"letters = get_letters(clean_image(plt.imread(labels.path[2])),1.5) \nax,ay = 0,0\nL = int(np.sqrt(len(letters)))\nfig,axs = plt.subplots(L+1,L)\nfor l in letters:\n if axConclusions and Further Directions\n","metadata":{}},{"cell_type":"markdown","source":"

The current version of the pipeline does a fair job given the default parameters; the image's resolution needs to be taken into account as it directly affects the alpha,beta1, and beta2 parameters.

\n

Future work on this path will include running a detection pipeline on the resulting image to convert the retrieved letters into a string that can be fed into an independent process.

\n


\n


","metadata":{}}]}