{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction","metadata":{}},{"cell_type":"markdown","source":"Many studies on **Keypoint based detector** are conducted these days. One of the well-known keypoint-based detectors is **CornerNet**. The concept of ConerNet is predicting the coner of the bounding boxes using heatmaps like semantic segmentation. This detector is relatively easy to implement since it **does not need multiple ankers** unlike other well-known one-stage detectors such as RCNN, YOLO and SSD. \n**CenterNet (Object as Points)** I show here is one of keypoint based detectors derieved from CornerNet. CenterNet directly predicts the center pixel of the object with heatmap. It's quite similar to YOLO with single anker, but the way of using heatmap is interesting.\nFor further information, please refer to the original papers.\n\n\n*   CornerNet:https://arxiv.org/abs/1808.01244\n*   CenterNet (Object as Points):https://arxiv.org/abs/1904.07850\n\nI'm newbie on image detection/classification task.\nPlease let me know if you find any mistake. Thanks!","metadata":{"id":"kgIyy83KgE4O"}},{"cell_type":"markdown","source":"*-----This kernel is written in both English and Japanese.------*\n\n初心者カーネルですが、日本語でも並記します。皆様のご参考になれば幸いです。\n\n本カーネルでは、最近話題になっているキーポイントベースの検出器を試してみました。CornerNet派生の『CenterNet』と呼ばれるもので、YOLOなどのようにアンカーを使用せず、セグメンテーション(U-Net)のようなヒートマップで対象物の中心点を検出する手法です。(シングルアンカーのような雰囲気ですが、ヒートマップだけでいいので実装しやすい印象です)\n\nDeNAさんはじめとした、様々な日本語の記事でも勉強させていただいておりますので、この場を借りてお礼申し上げます。","metadata":{"id":"UKL-mKbyk3e0"}},{"cell_type":"code","source":"import numpy as np\nimport json\nimport pandas as pd\nfrom PIL import Image, ImageDraw\nimport matplotlib.pyplot as plt\nfrom pandas.io.json import json_normalize\nimport random\nimport tensorflow as tf\nfrom sklearn.utils import shuffle\nfrom sklearn.model_selection import KFold,train_test_split\nimport matplotlib.pyplot as plt\nimport glob\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dense,Dropout, Conv2D,Conv2DTranspose, BatchNormalization, Activation,AveragePooling2D,GlobalAveragePooling2D, Input, Concatenate, MaxPool2D, Add, UpSampling2D, LeakyReLU,ZeroPadding2D\nfrom keras.models import Model\nfrom keras.objectives import mean_squared_error\nfrom keras import backend as K\nfrom keras.losses import binary_crossentropy\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard, ReduceLROnPlateau,LearningRateScheduler\nimport os  \n\nfrom keras.optimizers import Adam, RMSprop, SGD\nfrom tensorflow.compat.v1 import ConfigProto\nfrom tensorflow.compat.v1 import InteractiveSession\n","metadata":{"id":"eumVCfpDlWwf","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-14T13:54:04.287165Z","iopub.execute_input":"2021-06-14T13:54:04.287508Z","iopub.status.idle":"2021-06-14T13:54:06.850535Z","shell.execute_reply.started":"2021-06-14T13:54:04.28746Z","shell.execute_reply":"2021-06-14T13:54:06.849664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First of all, let's convert the input data into the labels for CenterNet.\n\nCenterNet用に入力データを形成しておきます。","metadata":{"id":"-rtA4BBQqDGy"}},{"cell_type":"code","source":"path_1=\"../input/train.csv\"\npath_2=\"../input/train_images/\"\npath_3=\"../input/test_images/\"\npath_4=\"../input/sample_submission.csv\"\ndf_train=pd.read_csv(path_1)\n#print(df_train.head())\n#print(df_train.shape)\ndf_train=df_train.dropna(axis=0, how='any')#you can use nan data(page with no letter)\ndf_train=df_train.reset_index(drop=True)\n#print(df_train.shape)\n\nannotation_list_train=[]\ncategory_names=set()\n\nfor i in range(len(df_train)):\n  ann=np.array(df_train.loc[i,\"labels\"].split(\" \")).reshape(-1,5)#cat,x,y,width,height for each picture\n  category_names=category_names.union({i for i in ann[:,0]})\n\ncategory_names=sorted(category_names)\ndict_cat={list(category_names)[j]:str(j) for j in range(len(category_names))}\ninv_dict_cat={str(j):list(category_names)[j] for j in range(len(category_names))}\n#print(dict_cat)\n  \nfor i in range(len(df_train)):\n  ann=np.array(df_train.loc[i,\"labels\"].split(\" \")).reshape(-1,5)#cat,left,top,width,height for each picture\n  for j,category_name in enumerate(ann[:,0]):\n    ann[j,0]=int(dict_cat[category_name])  \n  ann=ann.astype('int32')\n  ann[:,1]+=ann[:,3]//2#center_x\n  ann[:,2]+=ann[:,4]//2#center_y\n  annotation_list_train.append([\"{}{}.jpg\".format(path_2,df_train.loc[i,\"image_id\"]),ann])\n\nprint(\"sample image\")\ninput_width,input_height=512, 512\nimg = np.asarray(Image.open(annotation_list_train[0][0]).resize((input_width,input_height)).convert('RGB'))\nplt.imshow(img)\nplt.show()\n","metadata":{"id":"viYi-cjclZ2D","execution":{"iopub.status.busy":"2021-06-14T14:13:16.342845Z","iopub.execute_input":"2021-06-14T14:13:16.343172Z","iopub.status.idle":"2021-06-14T14:13:21.971391Z","shell.execute_reply.started":"2021-06-14T14:13:16.343118Z","shell.execute_reply":"2021-06-14T14:13:21.970233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get directory of test images\ndf_submission=pd.read_csv(path_4)\nid_test=path_3+df_submission[\"image_id\"]+\".jpg\"","metadata":{"id":"KtKhxahpqXLZ","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## STEP 1: Preprocessing (Check Object Size)","metadata":{"id":"dlTW8tO9R7vJ"}},{"cell_type":"markdown","source":"**Goal of Step1 is to check the object size and determine the size of input image for detector.**","metadata":{"id":"p1066wEtR_g_"}},{"cell_type":"markdown","source":"I'd like to check the size of objects before creating the model. If the object size is too small, detector cannot find the target. \n\nまず、検出モデルを作る前に、文字サイズをチェックしておきます。CenterNetの出力方式に対して過少に小さい文字は、検出できませんので。","metadata":{"id":"UTOoH3Cy9UkN"}},{"cell_type":"code","source":"aspect_ratio_pic_all=[]\naspect_ratio_pic_all_test=[]\naverage_letter_size_all=[]\ntrain_input_for_size_estimate=[]\nresize_dir=\"resized/\"\nif os.path.exists(resize_dir) == False:os.mkdir(resize_dir)\nfor i in range(len(annotation_list_train)):\n    with Image.open(annotation_list_train[i][0]) as f:\n        width,height=f.size\n        area=width*height\n        aspect_ratio_pic=height/width\n        aspect_ratio_pic_all.append(aspect_ratio_pic)\n        letter_size=annotation_list_train[i][1][:,3]*annotation_list_train[i][1][:,4]\n        letter_size_ratio=letter_size/area\n    \n        average_letter_size=np.mean(letter_size_ratio)\n        average_letter_size_all.append(average_letter_size)\n        train_input_for_size_estimate.append([annotation_list_train[i][0],np.log(average_letter_size)])#logにしとく\n    \n\nfor i in range(len(id_test)):\n    with Image.open(id_test[i]) as f:\n        width,height=f.size\n        aspect_ratio_pic=height/width\n        aspect_ratio_pic_all_test.append(aspect_ratio_pic)\n\n\nplt.hist(np.log(average_letter_size_all),bins=100)\nplt.title('log(ratio of letter_size to picture_size))',loc='center',fontsize=12)\nplt.show()","metadata":{"id":"rP6_99JH6zoU","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see the average size of the objects varies among pictures. So it would be better to split the picture into several parts. So cropping pictures with appropriate ratio would be important when detecting letters.  One way to find the best cropping size is creating the model to predict the average letter size.\n\n図は、文字サイズ÷ピクチャサイズの比率の分布を示したものですが、ピクチャ毎に結構ばらついています。相対的な文字の大きさが大きくなったり小さくなったりするよりも統一されていた方が検出しやすいので、統一方法を考えます。ここでは、やや強引に、CNNに文字の平均サイズを算出してもらえるか試してみることにします。","metadata":{"id":"QcTNHyXZJqZY"}},{"cell_type":"markdown","source":"### Create Model","metadata":{"id":"t_Vq2P0BUAkn"}},{"cell_type":"markdown","source":"Let's create CNN model(almost ResNet).\n\n*   Input: Image (resized into 512x512x3)\n*   Output: Ratio of letter_size to picture_size\n\nResNetっぽいモデルで試してみます。入力は1ページごとの画像。出力は文字とピクチャのサイズ比とします。","metadata":{"id":"wcmN-Yu4P-9i"}},{"cell_type":"code","source":"\ncategory_n=1\nimport cv2\ninput_width,input_height=512, 512\n\ndef Datagen_sizecheck_model(filenames, batch_size, size_detection_mode=True, is_train=True,random_crop=True):\n  x=[]\n  y=[]\n  \n  count=0\n\n  while True:\n    for i in range(len(filenames)):\n      if random_crop:\n        crop_ratio=np.random.uniform(0.7,1)\n      else:\n        crop_ratio=1\n      with Image.open(filenames[i][0]) as f:\n        #random crop\n        if random_crop and is_train:\n          pic_width,pic_height=f.size\n          f=np.asarray(f.convert('RGB'),dtype=np.uint8)\n          top_offset=np.random.randint(0,pic_height-int(crop_ratio*pic_height))\n          left_offset=np.random.randint(0,pic_width-int(crop_ratio*pic_width))\n          bottom_offset=top_offset+int(crop_ratio*pic_height)\n          right_offset=left_offset+int(crop_ratio*pic_width)\n          f=cv2.resize(f[top_offset:bottom_offset,left_offset:right_offset,:],(input_height,input_width))\n        else:\n          f=f.resize((input_width, input_height))\n          f=np.asarray(f.convert('RGB'),dtype=np.uint8)          \n        x.append(f)\n      \n      \n      if random_crop and is_train:\n        y.append(filenames[i][1]-np.log(crop_ratio))\n      else:\n        y.append(filenames[i][1])\n      \n      count+=1\n      if count==batch_size:\n        x=np.array(x, dtype=np.float32)\n        y=np.array(y, dtype=np.float32)\n\n        inputs=x/255\n        targets=y       \n        x=[]\n        y=[]\n        count=0\n        yield inputs, targets\n\n\n\ndef aggregation_block(x_shallow, x_deep, deep_ch, out_ch):\n  x_deep= Conv2DTranspose(deep_ch, kernel_size=2, strides=2, padding='same', use_bias=False)(x_deep)\n  x_deep = BatchNormalization()(x_deep)   \n  x_deep = LeakyReLU(alpha=0.1)(x_deep)\n  x = Concatenate()([x_shallow, x_deep])\n  x=Conv2D(out_ch, kernel_size=1, strides=1, padding=\"same\")(x)\n  x = BatchNormalization()(x)   \n  x = LeakyReLU(alpha=0.1)(x)\n  return x\n  \n\n\ndef cbr(x, out_layer, kernel, stride):\n  x=Conv2D(out_layer, kernel_size=kernel, strides=stride, padding=\"same\")(x)\n  x = BatchNormalization()(x)\n  x = LeakyReLU(alpha=0.1)(x)\n  return x\n\ndef resblock(x_in,layer_n):\n  x=cbr(x_in,layer_n,3,1)\n  x=cbr(x,layer_n,3,1)\n  x=Add()([x,x_in])\n  return x  \n\n\n#I use the same network at CenterNet\ndef create_model(input_shape, size_detection_mode=True, aggregation=True):\n    input_layer = Input(input_shape)\n    \n    #resized input\n    input_layer_1=AveragePooling2D(2)(input_layer)\n    input_layer_2=AveragePooling2D(2)(input_layer_1)\n\n    #### ENCODER ####\n\n    x_0= cbr(input_layer, 16, 3, 2)#512->256\n    concat_1 = Concatenate()([x_0, input_layer_1])\n\n    x_1= cbr(concat_1, 32, 3, 2)#256->128\n    concat_2 = Concatenate()([x_1, input_layer_2])\n\n    x_2= cbr(concat_2, 64, 3, 2)#128->64\n    \n    x=cbr(x_2,64,3,1)\n    x=resblock(x,64)\n    x=resblock(x,64)\n    \n    x_3= cbr(x, 128, 3, 2)#64->32\n    x= cbr(x_3, 128, 3, 1)\n    x=resblock(x,128)\n    x=resblock(x,128)\n    x=resblock(x,128)\n    \n    x_4= cbr(x, 256, 3, 2)#32->16\n    x= cbr(x_4, 256, 3, 1)\n    x=resblock(x,256)\n    x=resblock(x,256)\n    x=resblock(x,256)\n    x=resblock(x,256)\n    x=resblock(x,256)\n \n    x_5= cbr(x, 512, 3, 2)#16->8\n    x= cbr(x_5, 512, 3, 1)\n    \n    x=resblock(x,512)\n    x=resblock(x,512)\n    x=resblock(x,512)\n    \n    if size_detection_mode:\n      x=GlobalAveragePooling2D()(x)\n      x=Dropout(0.2)(x)\n      out=Dense(1,activation=\"linear\")(x)\n    \n    else:#centernet mode\n    #### DECODER ####\n      x_1= cbr(x_1, output_layer_n, 1, 1)\n      x_1 = aggregation_block(x_1, x_2, output_layer_n, output_layer_n)\n      x_2= cbr(x_2, output_layer_n, 1, 1)\n      x_2 = aggregation_block(x_2, x_3, output_layer_n, output_layer_n)\n      x_1 = aggregation_block(x_1, x_2, output_layer_n, output_layer_n)\n      x_3= cbr(x_3, output_layer_n, 1, 1)\n      x_3 = aggregation_block(x_3, x_4, output_layer_n, output_layer_n) \n      x_2 = aggregation_block(x_2, x_3, output_layer_n, output_layer_n)\n      x_1 = aggregation_block(x_1, x_2, output_layer_n, output_layer_n)\n      \n      x_4= cbr(x_4, output_layer_n, 1, 1)\n\n      x=cbr(x, output_layer_n, 1, 1)\n      x= UpSampling2D(size=(2, 2))(x)#8->16 tconvのがいいか\n\n      x = Concatenate()([x, x_4])\n      x=cbr(x, output_layer_n, 3, 1)\n      x= UpSampling2D(size=(2, 2))(x)#16->32\n    \n      x = Concatenate()([x, x_3])\n      x=cbr(x, output_layer_n, 3, 1)\n      x= UpSampling2D(size=(2, 2))(x)#32->64   128のがいいかも？ \n    \n      x = Concatenate()([x, x_2])\n      x=cbr(x, output_layer_n, 3, 1)\n      x= UpSampling2D(size=(2, 2))(x)#64->128 \n      \n      x = Concatenate()([x, x_1])\n      x=Conv2D(output_layer_n, kernel_size=3, strides=1, padding=\"same\")(x)\n      out = Activation(\"sigmoid\")(x)\n    \n    model=Model(input_layer, out)\n    \n    return model\n  \n    \n\n\ndef model_fit_sizecheck_model(model,train_list,cv_list,n_epoch,batch_size=32):\n    hist = model.fit_generator(\n        Datagen_sizecheck_model(train_list,batch_size, is_train=True,random_crop=True),\n        steps_per_epoch = len(train_list) // batch_size,\n        epochs = n_epoch,\n        validation_data=Datagen_sizecheck_model(cv_list,batch_size, is_train=False,random_crop=False),\n        validation_steps = len(cv_list) // batch_size,\n        callbacks = [lr_schedule, model_checkpoint],#[early_stopping, reduce_lr, model_checkpoint],\n        shuffle = True,\n        verbose = 1\n    )\n    return hist\n\n  \n","metadata":{"id":"jVix-CRbIXPf","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"K.clear_session()\nmodel=create_model(input_shape=(input_height,input_width,3),size_detection_mode=True)\n\"\"\"\n# EarlyStopping\nearly_stopping = EarlyStopping(monitor = 'val_loss', min_delta=0, patience = 10, verbose = 1)\n# ModelCheckpoint\nweights_dir = '/model_1/'\nif os.path.exists(weights_dir) == False:os.mkdir(weights_dir)\nmodel_checkpoint = ModelCheckpoint(weights_dir + \"val_loss{val_loss:.3f}.hdf5\", monitor = 'val_loss', verbose = 1,\n                                      save_best_only = True, save_weights_only = True, period = 1)\n# reduce learning rate\nreduce_lr = ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, patience = 10, verbose = 1)\n\"\"\"\ndef lrs(epoch):\n    lr = 0.0005\n    if epoch>10:\n        lr = 0.0001\n    return lr\n\nlr_schedule = LearningRateScheduler(lrs)\nmodel_checkpoint = ModelCheckpoint(\"final_weights_step1.hdf5\", monitor = 'val_loss', verbose = 1,\n                                      save_best_only = True, save_weights_only = True, period = 1)\nprint(model.summary())","metadata":{"id":"1F1a3C_LW_UY","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training","metadata":{"id":"1UMVc89HSNNO"}},{"cell_type":"markdown","source":"Step1 is not main topic of this kernel. Run only 15 epochs.\n\n本カーネルの主目的はstep2(CenterNet)なので、少し短めのエポックで切り上げます。","metadata":{"id":"7126xqIkoI-Q"}},{"cell_type":"code","source":"train_list, cv_list = train_test_split(train_input_for_size_estimate, random_state = 111,test_size = 0.2)\n\n\nlearning_rate=0.0005\nn_epoch=12\nbatch_size=32\n\nmodel.compile(loss=mean_squared_error, optimizer=Adam(lr=learning_rate))\nhist = model_fit_sizecheck_model(model,train_list,cv_list,n_epoch,batch_size)\n\n#model.save_weights('final_weights_step1.h5')\nmodel.load_weights('final_weights_step1.hdf5')","metadata":{"id":"5W6-Twl9Z6TR","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Result","metadata":{"id":"-rpgdUdOSXtC"}},{"cell_type":"code","source":"predict = model.predict_generator(Datagen_sizecheck_model(cv_list,batch_size, is_train=False,random_crop=False),\n                                  steps=len(cv_list) // batch_size)\ntarget=[cv[1] for cv in cv_list]\nplt.scatter(predict,target[:len(predict)])\nplt.title('---letter_size/picture_size--- estimated vs target ',loc='center',fontsize=10)\nplt.show()","metadata":{"id":"DNonHXQ7fEC8","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Umm... not so good. Training is not enough. But I might as well use it.\n\nイマイチですが、とりあえず使っておきます。なお、学習をもう少し続ければもう少しマトモにはなります。","metadata":{"id":"Q2J5SM-QS88c"}},{"cell_type":"code","source":"batch_size=1\npredict_train = model.predict_generator(Datagen_sizecheck_model(train_input_for_size_estimate,batch_size, is_train=False,random_crop=False, ),\n                                  steps=len(train_input_for_size_estimate)//batch_size)","metadata":{"id":"pTyR0fO1oZuz","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the detector's size, split numbers are determined for each picture. In the next section I will make the CenterNet whose output shape is 128x128. Assuming the letters are detected every 5 pixcels, this detector can find the letters 25x25 at most.\n\n後述するCenterNetでは出力を128x128のヒートマップにします。検出できるのはせいぜい25x25くらいだと考えて、元の画像データの分割数を適当に決めます。","metadata":{"id":"vvTtQpu3UzR_"}},{"cell_type":"code","source":"base_detect_num_h,base_detect_num_w=25,25\nannotation_list_train_w_split=[]\nfor i, predicted_size in enumerate(predict_train):\n  detect_num_h=aspect_ratio_pic_all[i]*np.exp(-predicted_size/2)\n  detect_num_w=detect_num_h/aspect_ratio_pic_all[i]\n  h_split_recommend=np.maximum(1,detect_num_h/base_detect_num_h)\n  w_split_recommend=np.maximum(1,detect_num_w/base_detect_num_w)\n  annotation_list_train_w_split.append([annotation_list_train[i][0],annotation_list_train[i][1],h_split_recommend,w_split_recommend])\nfor i in np.arange(0,1):\n  print(\"recommended height split:{}, recommended width_split:{}\".format(annotation_list_train_w_split[i][2],annotation_list_train_w_split[i][3]))\n  img = np.asarray(Image.open(annotation_list_train_w_split[i][0]).convert('RGB'))\n  plt.imshow(img)\n  plt.show()","metadata":{"id":"uiW8NoOL8kV4","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## STEP 2: Detection by CenterNet","metadata":{"id":"Fuh4HivsWq98"}},{"cell_type":"markdown","source":"**Goal of stage2 is to detect the letters by CenterNet.**","metadata":{"id":"4P6dGIVNWrOL"}},{"cell_type":"markdown","source":"I'd like to change the generator and model. The differences from step_1 are as follows:\n\n\n*   Add random cropping to generator\n*  Make target data for CenterNet\n*   Add decoder to the network (similar to U-Net)\n\n**Input of CenterNet:** \n\n>Cropped image resized into 512x512x3\n\n\n**Output of CenterNet:** \n\n>Heatmap of center point 128x128x1\n\n>x-offset and y-offset of the center point inside the detected cell 128x128x2\n\n>width and height of the the detected object 128x128x2\n\nIt should be noted that this network can cope with multiple categories by increasing the number of output layers of center point. \n\n","metadata":{"id":"mk0t0a6-XGmY"}},{"cell_type":"markdown","source":"ここからCenterNetになります。step1とのモデルの違いは\n* ランダムクロッピングの追加\n* トレーニングのターゲットデータをCenterNet用に加工\n* CNNにデコーダ部分を追加(U-Netみたいな感じ)\n\n入力：512x512x3のイメージ、\n\n出力：128x128x1の中心位置ヒートマップ、x,y中心オフセット128x128x2、物体幅と高さ128x128x2の合計5層になります\n","metadata":{}},{"cell_type":"markdown","source":"### Create Model & Training Set","metadata":{"id":"Z3wxB7tzUd3V"}},{"cell_type":"markdown","source":"Before creating the model, I'd like to show the example image of target heatmap showing center points of objects.","metadata":{"id":"YxqBA229TQJy"}},{"cell_type":"code","source":"category_n=1\noutput_layer_n=category_n+4\noutput_height,output_width=128,128\n\ni=0\n\nh_split=annotation_list_train_w_split[i][2]\nw_split=annotation_list_train_w_split[i][3]\nmax_crop_ratio_h=1/h_split\nmax_crop_ratio_w=1/w_split\ncrop_ratio=np.random.uniform(0.5,1)\ncrop_ratio_h=max_crop_ratio_h*crop_ratio\ncrop_ratio_w=max_crop_ratio_w*crop_ratio\n\nwith Image.open(annotation_list_train_w_split[i][0]) as f:\n        \n        #random crop\n        pic_width,pic_height=f.size\n        f=np.asarray(f.convert('RGB'),dtype=np.uint8)\n        top_offset=np.random.randint(0,pic_height-int(crop_ratio_h*pic_height))\n        left_offset=np.random.randint(0,pic_width-int(crop_ratio_w*pic_width))\n        bottom_offset=top_offset+int(crop_ratio_h*pic_height)\n        right_offset=left_offset+int(crop_ratio_w*pic_width)\n        img=cv2.resize(f[top_offset:bottom_offset,left_offset:right_offset,:],(input_height,input_width))\n\n      \n      \noutput_layer=np.zeros((output_height,output_width,(output_layer_n+category_n)))\nfor annotation in annotation_list_train_w_split[i][1]:\n\n          x_c=(annotation[1]-left_offset)*(output_width/int(crop_ratio_w*pic_width))\n          y_c=(annotation[2]-top_offset)*(output_height/int(crop_ratio_h*pic_height))\n          width=annotation[3]*(output_width/int(crop_ratio_w*pic_width))\n          height=annotation[4]*(output_height/int(crop_ratio_h*pic_height))\n          \n          top=np.maximum(0,y_c-height/2)\n          left=np.maximum(0,x_c-width/2)\n          bottom=np.minimum(output_height,y_c+height/2)\n          right=np.minimum(output_width,x_c+width/2)\n          \n          if top>=output_height or left>=output_width or bottom<=0 or right<=0:#random crop(エリア外の除去)\n            continue\n          width=right-left\n          height=bottom-top\n          x_c=(right+left)/2\n          y_c=(top+bottom)/2\n          \n        \n        \n          category=0#not classify\n          heatmap=((np.exp(-(((np.arange(output_width)-x_c)/(width/10))**2)/2)).reshape(1,-1)\n                            *(np.exp(-(((np.arange(output_height)-y_c)/(height/10))**2)/2)).reshape(-1,1))\n          output_layer[:,:,category]=np.maximum(output_layer[:,:,category],heatmap[:,:])\n          output_layer[int(y_c//1),int(x_c//1),category_n+category]=1\n          output_layer[int(y_c//1),int(x_c//1),2*category_n]=y_c%1#height offset\n          output_layer[int(y_c//1),int(x_c//1),2*category_n+1]=x_c%1\n          output_layer[int(y_c//1),int(x_c//1),2*category_n+2]=height/output_height\n          output_layer[int(y_c//1),int(x_c//1),2*category_n+3]=width/output_width\n\nfig, axes = plt.subplots(1, 3,figsize=(15,15))\naxes[0].set_axis_off()\naxes[0].imshow(img)\naxes[1].set_axis_off()\naxes[1].imshow(output_layer[:,:,1])\naxes[2].set_axis_off()\naxes[2].imshow(output_layer[:,:,0])\nplt.show()","metadata":{"id":"hAB45SxiS4Sg","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Left image is the original picture(cropped).\n\n- Middle one is the target image in which center points are 1, other pixcels are 0.\n\n- Right one shows center points with gaussian distributions.\n\nRight image is necessary to reduce the training loss when the model detect the points near the exact center. \n\n真ん中が1-0の訓練用ターゲットデータです。右側がそれにガウス分布をあてたもので、検出器が中心の近傍点を検出した場合のロスを低減するために使用します。","metadata":{"id":"HtFsORI-E3CB"}},{"cell_type":"code","source":"category_n=1\noutput_layer_n=category_n+4\noutput_height,output_width=128,128\n\ndef Datagen_centernet(filenames, batch_size):\n  x=[]\n  y=[]\n  \n  count=0\n\n  while True:\n    for i in range(len(filenames)):\n      h_split=filenames[i][2]\n      w_split=filenames[i][3]\n      max_crop_ratio_h=1/h_split\n      max_crop_ratio_w=1/w_split\n      crop_ratio=np.random.uniform(0.5,1)\n      crop_ratio_h=max_crop_ratio_h*crop_ratio\n      crop_ratio_w=max_crop_ratio_w*crop_ratio\n      \n      with Image.open(filenames[i][0]) as f:\n        \n        #random crop\n        \n        pic_width,pic_height=f.size\n        f=np.asarray(f.convert('RGB'),dtype=np.uint8)\n        top_offset=np.random.randint(0,pic_height-int(crop_ratio_h*pic_height))\n        left_offset=np.random.randint(0,pic_width-int(crop_ratio_w*pic_width))\n        bottom_offset=top_offset+int(crop_ratio_h*pic_height)\n        right_offset=left_offset+int(crop_ratio_w*pic_width)\n        f=cv2.resize(f[top_offset:bottom_offset,left_offset:right_offset,:],(input_height,input_width))\n        x.append(f)      \n\n      output_layer=np.zeros((output_height,output_width,(output_layer_n+category_n)))\n      for annotation in filenames[i][1]:\n        x_c=(annotation[1]-left_offset)*(output_width/int(crop_ratio_w*pic_width))\n        y_c=(annotation[2]-top_offset)*(output_height/int(crop_ratio_h*pic_height))\n        width=annotation[3]*(output_width/int(crop_ratio_w*pic_width))\n        height=annotation[4]*(output_height/int(crop_ratio_h*pic_height))\n        top=np.maximum(0,y_c-height/2)\n        left=np.maximum(0,x_c-width/2)\n        bottom=np.minimum(output_height,y_c+height/2)\n        right=np.minimum(output_width,x_c+width/2)\n          \n        if top>=(output_height-0.1) or left>=(output_width-0.1) or bottom<=0.1 or right<=0.1:#random crop(out of picture)\n          continue\n        width=right-left\n        height=bottom-top\n        x_c=(right+left)/2\n        y_c=(top+bottom)/2\n\n        \n        category=0#not classify, just detect\n        heatmap=((np.exp(-(((np.arange(output_width)-x_c)/(width/10))**2)/2)).reshape(1,-1)\n                            *(np.exp(-(((np.arange(output_height)-y_c)/(height/10))**2)/2)).reshape(-1,1))\n        output_layer[:,:,category]=np.maximum(output_layer[:,:,category],heatmap[:,:])\n        output_layer[int(y_c//1),int(x_c//1),category_n+category]=1\n        output_layer[int(y_c//1),int(x_c//1),2*category_n]=y_c%1#height offset\n        output_layer[int(y_c//1),int(x_c//1),2*category_n+1]=x_c%1\n        output_layer[int(y_c//1),int(x_c//1),2*category_n+2]=height/output_height\n        output_layer[int(y_c//1),int(x_c//1),2*category_n+3]=width/output_width\n      y.append(output_layer)  \n    \n      count+=1\n      if count==batch_size:\n        x=np.array(x, dtype=np.float32)\n        y=np.array(y, dtype=np.float32)\n\n        inputs=x/255\n        targets=y       \n        x=[]\n        y=[]\n        count=0\n        yield inputs, targets\n\ndef all_loss(y_true, y_pred):\n    mask=K.sign(y_true[...,2*category_n+2])\n    N=K.sum(mask)\n    alpha=2.\n    beta=4.\n\n    heatmap_true_rate = K.flatten(y_true[...,:category_n])\n    heatmap_true = K.flatten(y_true[...,category_n:(2*category_n)])\n    heatmap_pred = K.flatten(y_pred[...,:category_n])\n    heatloss=-K.sum(heatmap_true*((1-heatmap_pred)**alpha)*K.log(heatmap_pred+1e-6)+(1-heatmap_true)*((1-heatmap_true_rate)**beta)*(heatmap_pred**alpha)*K.log(1-heatmap_pred+1e-6))\n    offsetloss=K.sum(K.abs(y_true[...,2*category_n]-y_pred[...,category_n]*mask)+K.abs(y_true[...,2*category_n+1]-y_pred[...,category_n+1]*mask))\n    sizeloss=K.sum(K.abs(y_true[...,2*category_n+2]-y_pred[...,category_n+2]*mask)+K.abs(y_true[...,2*category_n+3]-y_pred[...,category_n+3]*mask))\n    \n    all_loss=(heatloss+1.0*offsetloss+5.0*sizeloss)/N\n    return all_loss\n\ndef size_loss(y_true, y_pred):\n    mask=K.sign(y_true[...,2*category_n+2])\n    N=K.sum(mask)\n    sizeloss=K.sum(K.abs(y_true[...,2*category_n+2]-y_pred[...,category_n+2]*mask)+K.abs(y_true[...,2*category_n+3]-y_pred[...,category_n+3]*mask))\n    return (5*sizeloss)/N\n\ndef offset_loss(y_true, y_pred):\n    mask=K.sign(y_true[...,2*category_n+2])\n    N=K.sum(mask)\n    offsetloss=K.sum(K.abs(y_true[...,2*category_n]-y_pred[...,category_n]*mask)+K.abs(y_true[...,2*category_n+1]-y_pred[...,category_n+1]*mask))\n    return (offsetloss)/N\n  \ndef heatmap_loss(y_true, y_pred):\n    mask=K.sign(y_true[...,2*category_n+2])\n    N=K.sum(mask)\n    alpha=2.\n    beta=4.\n\n    heatmap_true_rate = K.flatten(y_true[...,:category_n])\n    heatmap_true = K.flatten(y_true[...,category_n:(2*category_n)])\n    heatmap_pred = K.flatten(y_pred[...,:category_n])\n    heatloss=-K.sum(heatmap_true*((1-heatmap_pred)**alpha)*K.log(heatmap_pred+1e-6)+(1-heatmap_true)*((1-heatmap_true_rate)**beta)*(heatmap_pred**alpha)*K.log(1-heatmap_pred+1e-6))\n    return heatloss/N\n\n  \ndef model_fit_centernet(model,train_list,cv_list,n_epoch,batch_size=32):\n    hist = model.fit_generator(\n        Datagen_centernet(train_list,batch_size),\n        steps_per_epoch = len(train_list) // batch_size,\n        epochs = n_epoch,\n        validation_data=Datagen_centernet(cv_list,batch_size),\n        validation_steps = len(cv_list) // batch_size,\n        callbacks = [lr_schedule],#early_stopping, reduce_lr, model_checkpoint],\n        shuffle = True,\n        verbose = 1\n    )\n    return hist","metadata":{"id":"30EkxAvQ-XkH","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"K.clear_session()\nmodel=create_model(input_shape=(input_height,input_width,3),size_detection_mode=False)\n\ndef lrs(epoch):\n    lr = 0.001\n    if epoch >= 20: lr = 0.0002\n    return lr\n\nlr_schedule = LearningRateScheduler(lrs)\n\n\"\"\"\n\n# EarlyStopping\nearly_stopping = EarlyStopping(monitor = 'val_loss', min_delta=0, patience = 60, verbose = 1)\n# ModelCheckpoint\nweights_dir = '/model_2/'\n\nif os.path.exists(weights_dir) == False:os.mkdir(weights_dir)\nmodel_checkpoint = ModelCheckpoint(weights_dir + \"val_loss{val_loss:.3f}.hdf5\", monitor = 'val_loss', verbose = 1,\n                                      save_best_only = True, save_weights_only = True, period = 3)\n# reduce learning rate\nreduce_lr = ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, patience = 10, verbose = 1)\n\"\"\"\nmodel.load_weights('final_weights_step1.h5',by_name=True, skip_mismatch=True)\n\nprint(model.summary())","metadata":{"id":"bJsSWwLCMEua","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training","metadata":{"id":"oxjOmJgKUyCQ"}},{"cell_type":"markdown","source":"Start training. This kernel runs 30 epochs. Longer training can improve the accuracy.\n\nとりあえず30epochほど計算しますが、多分もっと長い方がよいです。","metadata":{}},{"cell_type":"code","source":"train_list, cv_list = train_test_split(annotation_list_train_w_split, random_state = 111,test_size = 0.2)#stratified split is better\n\nlearning_rate=0.001\nn_epoch=30\nbatch_size=32\nmodel.compile(loss=all_loss, optimizer=Adam(lr=learning_rate), metrics=[heatmap_loss,size_loss,offset_loss])\nhist = model_fit_centernet(model,train_list,cv_list,n_epoch,batch_size)\n\nmodel.save_weights('final_weights_step2.h5')","metadata":{"id":"xjVS-w10MMfZ","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Result","metadata":{"id":"EaB7Y5aUVM1w"}},{"cell_type":"markdown","source":"Let's check a result of heatmap with validation data. You can see the centers of letters are detected.\n\n中心点のヒートマップの出力例です。なんとなく文字の中心が色濃くなっています。","metadata":{"id":"uwzWlysgVT7U"}},{"cell_type":"code","source":"pred_in_h=512\npred_in_w=512\npred_out_h=int(pred_in_h/4)\npred_out_w=int(pred_in_w/4)\n\nfor i in np.arange(0,1):\n  img = np.asarray(Image.open(cv_list[i][0]).resize((pred_in_w,pred_in_h)).convert('RGB'))\n  predict=model.predict((img.reshape(1,pred_in_h,pred_in_w,3))/255).reshape(pred_out_h,pred_out_w,(category_n+4))\n  heatmap=predict[:,:,0]\n\n  fig, axes = plt.subplots(1, 2,figsize=(15,15))\n  axes[0].set_axis_off()\n  axes[0].imshow(img)\n  axes[1].set_axis_off()\n  axes[1].imshow(heatmap)\n  plt.show()","metadata":{"id":"mexdIy9KMQjY","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, let's convert the outputs into bounding boxes. I use NMS (Non Maximum Suppression) to find the best boxes. \n(The original paper says that the max pooling also works well instead of NMS.)\n\n出力をバウンディングボックスにします。128x128個のバウンディングボックスがあることになるので、NMSで処理します。\n(元の論文では周囲8マスをmax poolingで処理する方法も提案されています)","metadata":{"id":"RZ4C1UM-Wfug"}},{"cell_type":"code","source":"from PIL import Image, ImageDraw\n\ndef NMS_all(predicts,category_n,score_thresh,iou_thresh):\n  y_c=predicts[...,category_n]+np.arange(pred_out_h).reshape(-1,1)\n  x_c=predicts[...,category_n+1]+np.arange(pred_out_w).reshape(1,-1)\n  height=predicts[...,category_n+2]*pred_out_h\n  width=predicts[...,category_n+3]*pred_out_w\n\n  count=0\n  for category in range(category_n):\n    predict=predicts[...,category]\n    mask=(predict>score_thresh)\n    #print(\"box_num\",np.sum(mask))\n    if mask.all==False:\n      continue\n    box_and_score=NMS(predict[mask],y_c[mask],x_c[mask],height[mask],width[mask],iou_thresh)\n    box_and_score=np.insert(box_and_score,0,category,axis=1)#category,score,top,left,bottom,right\n    if count==0:\n      box_and_score_all=box_and_score\n    else:\n      box_and_score_all=np.concatenate((box_and_score_all,box_and_score),axis=0)\n    count+=1\n  score_sort=np.argsort(box_and_score_all[:,1])[::-1]\n  box_and_score_all=box_and_score_all[score_sort]\n  #print(box_and_score_all)\n\n \n  _,unique_idx=np.unique(box_and_score_all[:,2],return_index=True)\n  #print(unique_idx)\n  return box_and_score_all[sorted(unique_idx)]\n  \ndef NMS(score,y_c,x_c,height,width,iou_thresh,merge_mode=False):\n  if merge_mode:\n    score=score\n    top=y_c\n    left=x_c\n    bottom=height\n    right=width\n  else:\n    #flatten\n    score=score.reshape(-1)\n    y_c=y_c.reshape(-1)\n    x_c=x_c.reshape(-1)\n    height=height.reshape(-1)\n    width=width.reshape(-1)\n    size=height*width\n    \n    \n    top=y_c-height/2\n    left=x_c-width/2\n    bottom=y_c+height/2\n    right=x_c+width/2\n    \n    inside_pic=(top>0)*(left>0)*(bottom<pred_out_h)*(right<pred_out_w)\n    outside_pic=len(inside_pic)-np.sum(inside_pic)\n    #if outside_pic>0:\n    #  print(\"{} boxes are out of picture\".format(outside_pic))\n    normal_size=(size<(np.mean(size)*10))*(size>(np.mean(size)/10))\n    score=score[inside_pic*normal_size]\n    top=top[inside_pic*normal_size]\n    left=left[inside_pic*normal_size]\n    bottom=bottom[inside_pic*normal_size]\n    right=right[inside_pic*normal_size]\n  \n\n    \n\n  #sort  \n  score_sort=np.argsort(score)[::-1]\n  score=score[score_sort]  \n  top=top[score_sort]\n  left=left[score_sort]\n  bottom=bottom[score_sort]\n  right=right[score_sort]\n  \n  area=((bottom-top)*(right-left))\n  \n  boxes=np.concatenate((score.reshape(-1,1),top.reshape(-1,1),left.reshape(-1,1),bottom.reshape(-1,1),right.reshape(-1,1)),axis=1)\n  \n  box_idx=np.arange(len(top))\n  alive_box=[]\n  while len(box_idx)>0:\n  \n    alive_box.append(box_idx[0])\n    \n    y1=np.maximum(top[0],top)\n    x1=np.maximum(left[0],left)\n    y2=np.minimum(bottom[0],bottom)\n    x2=np.minimum(right[0],right)\n    \n    cross_h=np.maximum(0,y2-y1)\n    cross_w=np.maximum(0,x2-x1)\n    still_alive=(((cross_h*cross_w)/area[0])<iou_thresh)\n    if np.sum(still_alive)==len(box_idx):\n      print(\"error\")\n      print(np.max((cross_h*cross_w)),area[0])\n    top=top[still_alive]\n    left=left[still_alive]\n    bottom=bottom[still_alive]\n    right=right[still_alive]\n    area=area[still_alive]\n    box_idx=box_idx[still_alive]\n  return boxes[alive_box]#score,top,left,bottom,right\n\n\n\ndef draw_rectangle(box_and_score,img,color):\n  number_of_rect=np.minimum(500,len(box_and_score))\n  \n  for i in reversed(list(range(number_of_rect))):\n    top, left, bottom, right = box_and_score[i,:]\n\n    \n    top = np.floor(top + 0.5).astype('int32')\n    left = np.floor(left + 0.5).astype('int32')\n    bottom = np.floor(bottom + 0.5).astype('int32')\n    right = np.floor(right + 0.5).astype('int32')\n    #label = '{} {:.2f}'.format(predicted_class, score)\n    #print(label)\n    #rectangle=np.array([[left,top],[left,bottom],[right,bottom],[right,top]])\n\n    draw = ImageDraw.Draw(img)\n    #label_size = draw.textsize(label)\n    #print(label_size)\n    \n    #if top - label_size[1] >= 0:\n    #  text_origin = np.array([left, top - label_size[1]])\n    #else:\n    #  text_origin = np.array([left, top + 1])\n    \n    thickness=4\n    if color==\"red\":\n      rect_color=(255, 0, 0)\n    elif color==\"blue\":\n      rect_color=(0, 0, 255)\n    else:\n      rect_color=(0, 0, 0)\n      \n    \n    if i==0:\n      thickness=4\n    for j in range(2*thickness):#薄いから何重にか描く\n      draw.rectangle([left + j, top + j, right - j, bottom - j],\n                    outline=rect_color)\n    #draw.rectangle(\n    #            [tuple(text_origin), tuple(text_origin + label_size)],\n    #            fill=(0, 0, 255))\n    #draw.text(text_origin, label, fill=(0, 0, 0))\n    \n  del draw\n  return img\n            \n  \ndef check_iou_score(true_boxes,detected_boxes,iou_thresh):\n  iou_all=[]\n  for detected_box in detected_boxes:\n    y1=np.maximum(detected_box[0],true_boxes[:,0])\n    x1=np.maximum(detected_box[1],true_boxes[:,1])\n    y2=np.minimum(detected_box[2],true_boxes[:,2])\n    x2=np.minimum(detected_box[3],true_boxes[:,3])\n    \n    cross_section=np.maximum(0,y2-y1)*np.maximum(0,x2-x1)\n    all_area=(detected_box[2]-detected_box[0])*(detected_box[3]-detected_box[1])+(true_boxes[:,2]-true_boxes[:,0])*(true_boxes[:,3]-true_boxes[:,1])\n    iou=np.max(cross_section/(all_area-cross_section))\n    #argmax=np.argmax(cross_section/(all_area-cross_section))\n    iou_all.append(iou)\n  score=2*np.sum(iou_all)/(len(detected_boxes)+len(true_boxes))\n  print(\"score:{}\".format(np.round(score,3)))\n  return score\n\n                \n\n\n\nfor i in np.arange(0,5):\n  #print(cv_list[i][2:])\n  img=Image.open(cv_list[i][0]).convert(\"RGB\")\n  width,height=img.size\n  predict=model.predict((np.asarray(img.resize((pred_in_w,pred_in_h))).reshape(1,pred_in_h,pred_in_w,3))/255).reshape(pred_out_h,pred_out_w,(category_n+4))\n  \n  box_and_score=NMS_all(predict,category_n,score_thresh=0.3,iou_thresh=0.4)\n\n  #print(\"after NMS\",len(box_and_score))\n  if len(box_and_score)==0:\n    continue\n\n  true_boxes=cv_list[i][1][:,1:]#c_x,c_y,width_height\n  top=true_boxes[:,1:2]-true_boxes[:,3:4]/2\n  left=true_boxes[:,0:1]-true_boxes[:,2:3]/2\n  bottom=top+true_boxes[:,3:4]\n  right=left+true_boxes[:,2:3]\n  true_boxes=np.concatenate((top,left,bottom,right),axis=1)\n    \n  heatmap=predict[:,:,0]\n \n  print_w, print_h = img.size\n  #resize predocted box to original size\n  box_and_score=box_and_score*[1,1,print_h/pred_out_h,print_w/pred_out_w,print_h/pred_out_h,print_w/pred_out_w]\n  check_iou_score(true_boxes,box_and_score[:,2:],iou_thresh=0.5)\n  img=draw_rectangle(box_and_score[:,2:],img,\"red\")\n  img=draw_rectangle(true_boxes,img,\"blue\")\n  \n  fig, axes = plt.subplots(1, 2,figsize=(15,15))\n  #axes[0].set_axis_off()\n  axes[0].imshow(img)\n  #axes[1].set_axis_off()\n  axes[1].imshow(heatmap)#, cmap='gray')\n  #axes[2].set_axis_off()\n  #axes[2].imshow(heatmap_1)#, cmap='gray')\n  plt.show()\n","metadata":{"id":"M9o_2oqHsZ2N","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not so bad? The letters in the pictures may be too small considering the output size(heatmap) of 128x128. \n\nSo let's split pictures into several parts using the results in step1, then run CenterNet for each splitted picture. After then, integrate them and run NMS.\n\nなんとなく検出できていますが、この時点では元画像を分割していないので、文字サイズが出力サイズ(128x128)に対して小さすぎます。\nStep1で得た分割数を適当に使って、分割した画像に対してそれぞれCenterNetにかけて、得られたデータをすべてひっくるめてNMSに投げることにします。","metadata":{"id":"GWqBJo4IYqro"}},{"cell_type":"code","source":"def split_and_detect(model,img,height_split_recommended,width_split_recommended,score_thresh=0.3,iou_thresh=0.4):\n  width,height=img.size\n  pred_in_w,pred_in_h=512,512\n  pred_out_w,pred_out_h=128,128\n  category_n=1\n  maxlap=0.5\n  height_split=int(-(-height_split_recommended//1)+1)\n  width_split=int(-(-width_split_recommended//1)+1)\n  height_lap=(height_split-height_split_recommended)/(height_split-1)\n  height_lap=np.minimum(maxlap,height_lap)\n  width_lap=(width_split-width_split_recommended)/(width_split-1)\n  width_lap=np.minimum(maxlap,width_lap)\n\n  if height>width:\n    crop_size=int((height)/(height_split-(height_split-1)*height_lap))#crop_height and width\n    if crop_size>=width:\n      crop_size=width\n      stride=int((crop_size*height_split-height)/(height_split-1))\n      top_list=[i*stride for i in range(height_split-1)]+[height-crop_size]\n      left_list=[0]\n    else:\n      stride=int((crop_size*height_split-height)/(height_split-1))\n      top_list=[i*stride for i in range(height_split-1)]+[height-crop_size]\n      width_split=-(-width//crop_size)\n      stride=int((crop_size*width_split-width)/(width_split-1))\n      left_list=[i*stride for i in range(width_split-1)]+[width-crop_size]\n\n  else:\n    crop_size=int((width)/(width_split-(width_split-1)*width_lap))#crop_height and width\n    if crop_size>=height:\n      crop_size=height\n      stride=int((crop_size*width_split-width)/(width_split-1))\n      left_list=[i*stride for i in range(width_split-1)]+[width-crop_size]\n      top_list=[0]\n    else:\n      stride=int((crop_size*width_split-width)/(width_split-1))\n      left_list=[i*stride for i in range(width_split-1)]+[width-crop_size]\n      height_split=-(-height//crop_size)\n      stride=int((crop_size*height_split-height)/(height_split-1))\n      top_list=[i*stride for i in range(height_split-1)]+[height-crop_size]\n  \n  count=0\n\n  for top_offset in top_list:\n    for left_offset in left_list:\n      img_crop = img.crop((left_offset, top_offset, left_offset+crop_size, top_offset+crop_size))\n      predict=model.predict((np.asarray(img_crop.resize((pred_in_w,pred_in_h))).reshape(1,pred_in_h,pred_in_w,3))/255).reshape(pred_out_h,pred_out_w,(category_n+4))\n  \n      box_and_score=NMS_all(predict,category_n,score_thresh,iou_thresh)#category,score,top,left,bottom,right\n      \n      #print(\"after NMS\",len(box_and_score))\n      if len(box_and_score)==0:\n        continue\n      #reshape and offset\n      box_and_score=box_and_score*[1,1,crop_size/pred_out_h,crop_size/pred_out_w,crop_size/pred_out_h,crop_size/pred_out_w]+np.array([0,0,top_offset,left_offset,top_offset,left_offset])\n      \n      if count==0:\n        box_and_score_all=box_and_score\n      else:\n        box_and_score_all=np.concatenate((box_and_score_all,box_and_score),axis=0)\n      count+=1\n  #print(\"all_box_num:\",len(box_and_score_all))\n  #print(box_and_score_all[:10,:],np.min(box_and_score_all[:,2:]))\n  if count==0:\n    box_and_score_all=[]\n  else:\n    score=box_and_score_all[:,1]\n    y_c=(box_and_score_all[:,2]+box_and_score_all[:,4])/2\n    x_c=(box_and_score_all[:,3]+box_and_score_all[:,5])/2\n    height=-box_and_score_all[:,2]+box_and_score_all[:,4]\n    width=-box_and_score_all[:,3]+box_and_score_all[:,5]\n    #print(np.min(height),np.min(width))\n    box_and_score_all=NMS(box_and_score_all[:,1],box_and_score_all[:,2],box_and_score_all[:,3],box_and_score_all[:,4],box_and_score_all[:,5],iou_thresh=0.5,merge_mode=True)\n  return box_and_score_all\n\n\nprint(\"test run. 5 image\")\nall_iou_score=[]\nfor i in np.arange(0,5):\n  img=Image.open(cv_list[i][0]).convert(\"RGB\")\n  box_and_score_all=split_and_detect(model,img,cv_list[i][2],cv_list[i][3],score_thresh=0.3,iou_thresh=0.4)\n  if len(box_and_score_all)==0:\n    print(\"no box found\")\n    continue\n  true_boxes=cv_list[i][1][:,1:]#c_x,c_y,width_height\n  top=true_boxes[:,1:2]-true_boxes[:,3:4]/2\n  left=true_boxes[:,0:1]-true_boxes[:,2:3]/2\n  bottom=top+true_boxes[:,3:4]\n  right=left+true_boxes[:,2:3]\n  true_boxes=np.concatenate((top,left,bottom,right),axis=1)\n\n  \n\n \n  print_w, print_h = img.size\n  iou_score=check_iou_score(true_boxes,box_and_score_all[:,1:],iou_thresh=0.5)\n  all_iou_score.append(iou_score)\n  \"\"\"\n  img=draw_rectangle(box_and_score_all[:,1:],img,\"red\")\n  img=draw_rectangle(true_boxes,img,\"blue\")\n  \n  fig, axes = plt.subplots(1, 2,figsize=(15,15))\n  #axes[0].set_axis_off()\n  axes[0].imshow(img)\n  #axes[1].set_axis_off()\n  axes[1].imshow(heatmap)#, cmap='gray')\n\n  plt.show()\n  \"\"\"\nprint(\"average_score:\",np.mean(all_iou_score))","metadata":{"id":"2mTV3cqWJXmR","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"OK. Score has improved.","metadata":{}},{"cell_type":"markdown","source":"## STEP 3: Classification","metadata":{"id":"E2sHIvRLHSp1"}},{"cell_type":"markdown","source":"**Goal of stage3 is to classify the letters.**\n\nSince this is not main topic of this kernel, I apply simple classification with CNN and skip any pre/postprocessing. I don't care inbalanced data as well.\n\n最後は文字の分類です。CenterNetの検出が本カーネル主題なので、かなりいい加減な感じです。ごめんなさい。","metadata":{"id":"wXgT65k4HSwN"}},{"cell_type":"markdown","source":"### Create Model & Training Set","metadata":{"id":"ePR1mzxkcQtb"}},{"cell_type":"markdown","source":"crop all letters in advance.\n\n文字部分を全部クロップしてしまいます。","metadata":{"id":"3h5Qn1dUcQ5x"}},{"cell_type":"code","source":"from tqdm import tqdm\ncount=0\ncrop_dir=\"/crop_letter/\"\nif os.path.exists(crop_dir) == False:os.mkdir(crop_dir)\n\ntrain_input=[]\npic_count=0\nfor ann_pic in tqdm(annotation_list_train):\n  pic_count+=1\n  with Image.open(ann_pic[0]) as img:\n    for ann in ann_pic[1]:#cat,center_x,center_y,width,height for each picture\n      cat=ann[0]\n      c_x=ann[1]\n      c_y=ann[2]\n      width=ann[3]\n      height=ann[4]\n      save_dir=crop_dir+str(count)+\".jpg\"\n      img.crop((int(c_x-width/2),int(c_y-height/2),int(c_x+width/2),int(c_y+height/2))).save(save_dir)\n      train_input.append([save_dir,cat])\n      count+=1\n                 ","metadata":{"id":"BBfwMtjMHTLr","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"show example of cropped picture.","metadata":{"id":"0PbF534toEmL"}},{"cell_type":"code","source":"df_unicode_translation=pd.read_csv(\"../input/unicode_translation.csv\")\nunicode=df_unicode_translation[\"Unicode\"].values\nchar=df_unicode_translation[\"char\"].values\ndict_translation={unicode[i]:char[i] for i in range(len(unicode))}\n\ni=0\nimg = np.asarray(Image.open(train_input[i][0]).resize((32,32)).convert('RGB'))\nname = dict_translation[inv_dict_cat[str(train_input[i][1])]]\nprint(name)\nplt.imshow(img)\nplt.show()\n  ","metadata":{"id":"9nMaUU_-HTTV","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"input_height,input_width=32,32\n\ndef Datagen_for_classification(filenames, batch_size, is_train=True,random_crop=True):\n  x=[]\n  y=[]\n  \n  count=0\n\n  while True:\n    for i in range(len(filenames)):\n      if random_crop:\n        crop_ratio=np.random.uniform(0.8,1)\n      else:\n        crop_ratio=1\n      with Image.open(filenames[i][0]) as f:\n        \n        #random crop\n        if random_crop and is_train:\n          pic_width,pic_height=f.size\n          f=np.asarray(f.convert('RGB'),dtype=np.uint8)\n          top_offset=np.random.randint(0,pic_height-int(crop_ratio*pic_height))\n          left_offset=np.random.randint(0,pic_width-int(crop_ratio*pic_width))\n          bottom_offset=top_offset+int(crop_ratio*pic_height)\n          right_offset=left_offset+int(crop_ratio*pic_width)\n          f=cv2.resize(f[top_offset:bottom_offset,left_offset:right_offset,:],(input_height,input_width))\n        else:\n          f=f.resize((input_width, input_height))\n          f=np.asarray(f.convert('RGB'),dtype=np.uint8)          \n        x.append(f)\n      \n        y.append(int(filenames[i][1]))\n      count+=1\n      if count==batch_size:\n        x=np.array(x, dtype=np.float32)\n        y=np.identity(len(dict_cat))[y].astype(np.float32)\n\n        inputs=x/255\n        targets=y       \n        x=[]\n        y=[]\n        count=0\n        yield inputs, targets\n        \ndef create_classification_model(input_shape, n_category):\n    input_layer = Input(input_shape)#32\n    x=cbr(input_layer,64,3,1)\n    x=resblock(x,64)\n    x=resblock(x,64)\n    x=cbr(input_layer,128,3,2)#16\n    x=resblock(x,128)\n    x=resblock(x,128)\n    x=cbr(input_layer,256,3,2)#8\n    x=resblock(x,256)\n    x=resblock(x,256)\n    x=GlobalAveragePooling2D()(x)\n    x=Dropout(0.2)(x)\n    out=Dense(n_category,activation=\"softmax\")(x)#sigmoid???catcrossていぎ\n    \n    classification_model=Model(input_layer, out)\n    \n    return classification_model\n      \ndef model_fit_classification(model,train_list,cv_list,n_epoch,batch_size=32):\n    hist = model.fit_generator(\n        Datagen_for_classification(train_list,batch_size, is_train=True,random_crop=True),\n        steps_per_epoch = len(train_list) // batch_size,\n        epochs = n_epoch,\n        validation_data=Datagen_for_classification(cv_list,batch_size, is_train=False,random_crop=False),\n        validation_steps = len(cv_list) // batch_size,\n        #callbacks = [early_stopping, reduce_lr, model_checkpoint],\n        shuffle = True,\n        verbose = 1\n    )\n    return hist","metadata":{"id":"OUBUl6dTHTWJ","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"K.clear_session()\ninput_height,input_width=32,32\nmodel=create_classification_model(input_shape=(input_height,input_width,3),n_category=len(dict_cat))\n\"\"\"\n\n# EarlyStopping\nearly_stopping = EarlyStopping(monitor = 'val_loss', min_delta=0, patience = 60, verbose = 1)\n# ModelCheckpoint\nweights_dir = './model_3/'\n\nif os.path.exists(weights_dir) == False:os.mkdir(weights_dir)\nmodel_checkpoint = ModelCheckpoint(weights_dir + \"val_loss{val_loss:.3f}.hdf5\", monitor = 'val_loss', verbose = 1,\n                                      save_best_only = True, save_weights_only = True, period = 1)\n# reduce learning rate\nreduce_lr = ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, patience = 10, verbose = 1)\n\"\"\"\n\nprint(model.summary())","metadata":{"id":"D1iT2UXIM36n","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training","metadata":{"id":"WnSIREedcvpC"}},{"cell_type":"markdown","source":"Only 10 epoch. Execution time is limited...\n\n10epochほどにしますが、学習率下げるなどしてもう少し頑張るほうがよいと思われます。","metadata":{}},{"cell_type":"code","source":"train_list, cv_list = train_test_split(train_input, random_state = 111,test_size = 0.2)\nlearning_rate=0.005\nn_epoch=10\nbatch_size=64\n\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=Adam(lr=learning_rate),metrics=[\"accuracy\"])\nhist = model_fit_classification(model,train_list,cv_list,n_epoch,batch_size)\n\nmodel.save_weights('final_weights_step3.h5')","metadata":{"id":"AZ_wbRYUM395","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Result","metadata":{"id":"whvk74IYczOc"}},{"cell_type":"markdown","source":"Let's check some results.","metadata":{"id":"WeHypI8Edoo0"}},{"cell_type":"code","source":"for i in range(3):\n  img = np.asarray(Image.open(train_input[i][0]).resize((32,32)).convert('RGB'))\n  predict=np.argmax(model.predict(img.reshape(1,32,32,3)/255),axis=1)[0]\n  name = dict_translation[inv_dict_cat[str(predict)]]\n  print(name)\n  plt.imshow(img)\n  plt.show()","metadata":{"id":"rC5N6PrpM333","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test & Submit","metadata":{"id":"42tLJg2q44Rk"}},{"cell_type":"code","source":"from tqdm import tqdm\n\nK.clear_session()\nprint(\"loading models...\")\nmodel_1=create_model(input_shape=(512,512,3),size_detection_mode=True)\n#model_1.load_weights('final_weights_step1.h5')\nmodel_1.load_weights('final_weights_step1.hdf5')\n\nmodel_2=create_model(input_shape=(512,512,3),size_detection_mode=False)\nmodel_2.load_weights('final_weights_step2.h5')\n\nmodel_3=create_classification_model(input_shape=(32,32,3),n_category=len(dict_cat))\nmodel_3.load_weights('final_weights_step3.h5')\n\n\ndef pipeline(i,print_img=False):\n  # model1: determine how to split image\n  if print_img: print(\"model 1\")\n  img = np.asarray(Image.open(id_test[i]).resize((512,512)).convert('RGB'))\n  predicted_size=model_1.predict(img.reshape(1,512,512,3)/255)\n  detect_num_h=aspect_ratio_pic_all_test[i]*np.exp(-predicted_size/2)\n  detect_num_w=detect_num_h/aspect_ratio_pic_all_test[i]\n  h_split_recommend=np.maximum(1,detect_num_h/base_detect_num_h)\n  w_split_recommend=np.maximum(1,detect_num_w/base_detect_num_w)\n  if print_img: print(\"recommended split_h:{}, split_w:{}\".format(h_split_recommend,w_split_recommend))\n\n  # model2: detection\n  if print_img: print(\"model 2\")\n  img=Image.open(id_test[i]).convert(\"RGB\")\n  box_and_score_all=split_and_detect(model_2,img,h_split_recommend,w_split_recommend,score_thresh=0.3,iou_thresh=0.4)#output:score,top,left,bottom,right\n  if print_img: print(\"find {} boxes\".format(len(box_and_score_all)))\n  print_w, print_h = img.size\n  if (len(box_and_score_all)>0) and print_img: \n    img=draw_rectangle(box_and_score_all[:,1:],img,\"red\")\n    plt.imshow(img)\n    plt.show()\n\n  # model3: classification\n  count=0\n  if (len(box_and_score_all)>0):\n    for box in box_and_score_all[:,1:]:\n      top,left,bottom,right=box\n      img_letter=img.crop((int(left),int(top),int(right),int(bottom))).resize((32,32))#大き目のピクセルのがいいか？\n      predict=(model_3.predict(np.asarray(img_letter).reshape(1,32,32,3)/255))\n      predict=np.argmax(predict,axis=1)[0]\n      code=inv_dict_cat[str(predict)]\n      c_x=int((left+right)/2)\n      c_y=int((top+bottom)/2)\n      if count==0:\n        ans=code+\" \"+str(c_x)+\" \"+str(c_y)\n      else:\n        ans=ans+\" \"+code+\" \"+str(c_x)+\" \"+str(c_y)\n      count+=1\n  else:\n    ans=\"\"\n  return ans\n\n_=pipeline(0,print_img=True)\n\n#I'm sorry. Not nice coding. Time consuming.\nfor i in tqdm(range(len(id_test))):\n  ans=pipeline(i,print_img=False)\n  df_submission.set_value(i, 'labels', ans)\n      \ndf_submission.to_csv(\"submission.csv\",index=False)\n","metadata":{"execution":{"iopub.status.busy":"2021-06-14T14:14:11.498554Z","iopub.execute_input":"2021-06-14T14:14:11.498855Z","iopub.status.idle":"2021-06-14T14:14:11.656173Z","shell.execute_reply.started":"2021-06-14T14:14:11.498801Z","shell.execute_reply":"2021-06-14T14:14:11.654487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thank you for reading!","metadata":{}}]}