{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction \n\nBonjour
\nIn [Bengali.AI Handwritten Grapheme Classification](https://www.kaggle.com/c/bengaliai-cv19/notebooks), we've all worked on the bengali character data set. In this competition we've to predict all the grapheme component properly. Though there are some other bengali character data sets and also numeric data sets out there. Some of the other datasets and some publications can be found [here](https://www.kaggle.com/c/bengaliai-cv19/discussion/122604#700071). Previously, I have made two notebooks for the bengali grapheme classification problem, namely\n\n- [[Keras]:Grapheme GridMask+AugMix & Ensemble](https://www.kaggle.com/ipythonx/keras-grapheme-gridmask-augmix-ensemble)\n- [[Keras]: Stratified Training & Ensemble](https://www.kaggle.com/ipythonx/keras-stratified-training-ensemble)\n\nNow, along with the character classification problem, bengali digits are also pretty challenging in bengali CV research. Bengali numbers are the units of the [numeral system](https://en.wikipedia.org/wiki/Numeral_system). In this notebook we will be exploring three different **bengali numerals dataset** and use a **Capsule Network** to classify those digits. We will be using the following three datasets in these notebook. \n\n- [Bengali.AI NumtaDB](https://bengali.ai/wp-content/uploads/datasets/assembled-bangla-handwritten.pdf)\n- [BanglaLekha](https://www.sciencedirect.com/science/article/pii/S2352340917301117)\n- [Ekush](https://shahariarrabby.github.io/ekush/#home)\n\nIt's quite like the [MNIST competition](https://www.kaggle.com/c/digit-recognizer). Here are the corresponding bn-eng numerals representation.\n\n> - ০ , ১ , ২ , ৩ , ৪ , ৫ , ৬ , ৭ , ৮ , ৯\n> - 0 , 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9","metadata":{}},{"cell_type":"markdown","source":"## Content\n- Preparing the Bangla Numerals Data Sets \n - NumtaDB \n - NumtaDB: Target Distribution \n - NumtaDB: Training Samples \n - BanglaLekha \n - BanglaLekha: Target Distribution \n - BanglaLekha: Training Samples \n - Ekush \n - Ekush: Target Distribution \n - Ekush: Training Samples \n- CapsNet Modeling \n - NumtaDB Capsule Training\n - NumtaDB Confusion Matrix\n - NumtaDB Classification Report\n - NumtaDB Display Some Prediction\n - NumtaDB Display Some Wrong Prediction\n - BanglaLekha Capsule Training \n - BanglaLekha Confusion Matrix\n - BanglaLekha Classification Report\n - BanglaLekha Display Some Prediction\n - BanglaLekha Display Some Wrong Prediction\n - Ekush Capsule Training \n - Ekush Confusion Matrix\n - Ekush Classification Report\n - Ekush Display Some Prediction\n - Ekush Display Some Wrong Prediction\n- Conclusion ","metadata":{}},{"cell_type":"code","source":"!pip install keras==2.2.4\n!pip install tensorflow==1.13.1\n\nimport cv2\nimport os\nimport glob\nimport gc\n\nimport numpy as np\nimport pandas as pd \nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport itertools \nfrom tqdm import tqdm\nfrom keras.models import Sequential\nfrom keras import callbacks\nfrom keras import backend as K\nfrom keras.optimizers import Adam\nfrom keras import initializers, layers\nfrom keras.utils import to_categorical\nfrom keras.models import Model, load_model\nfrom sklearn.metrics import confusion_matrix\nfrom keras.metrics import categorical_crossentropy\nfrom sklearn.model_selection import train_test_split\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\nfrom keras.layers import (Dense, Input, Conv2D, Flatten, MaxPooling2D, \n Activation, Dropout, Average)\n\nimport warnings\nwarnings.filterwarnings(\"ignore\",category=FutureWarning)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Functions\n> \n> - **def** utility: `use to extend the 2D image to 3D for model compatability`\n> - **def** preprocess_image: `few image processing using openCV`\n> - **def** plot_log: `plot the learning curve of the model`\n> - **def** plot_confusion_matrix: `plot the confusion matrix`\n> - **def** imshow_group: `display some predicted output of the trained model`\n> - **def** display_errors: `display some error prediction made by trained model`","metadata":{}},{"cell_type":"code","source":"sns.set(style=\"darkgrid\")\n\ndef utility(X, Y, size):\n indices = np.arange(X.shape[0])\n np.random.shuffle(indices)\n X = X[indices]\n Y = Y[indices]\n # extedn image axis for keras model and with normalization\n X = X.reshape(X.shape[0], size, size, 1).astype('float32')/255.\n return X, Y\n\n\ndef preprocess_image(image):\n img = cv2.bilateralFilter(image,9,75,75)\n \n # using sharpen kenel\n kernel_sharp = np.array([[0,-1,0], \n [-1,5,-1], \n [0,-1,0]])\n img = cv2.filter2D(img, -1, kernel_sharp)\n \n return img\n\n\ndef plot_log(filename, show=True):\n\n data = pd.read_csv(filename)\n\n fig = plt.figure(figsize=(8,10))\n fig.subplots_adjust(top=0.95, bottom=0.05, right=0.95)\n fig.add_subplot(211)\n for key in data.keys():\n if key.find('loss') >= 0: # training loss\n plt.plot(data['epoch'].values, data[key].values, label=key)\n plt.legend()\n plt.title('Training and Validtion Loss')\n\n fig.add_subplot(212)\n for key in data.keys():\n if key.find('acc') >= 0: # acc\n plt.plot(data['epoch'].values, data[key].values, label=key)\n plt.legend()\n plt.title('Training and Validation Accuracy')\n \n if show:\n plt.show()\n \n\n#Note, this code is taken straight from the SKLEARN website,\n# an nice way of viewing confusion matrix.\ndef plot_confusion_matrix(cm, classes,\n normalize=False,\n title='Confusion matrix',\n cmap=plt.cm.GnBu):\n \"\"\"\n This function prints and plots the confusion matrix.\n Normalization can be applied by setting `normalize=True`.\n \"\"\"\n plt.figure(figsize=(8,10))\n plt.grid(False)\n plt.imshow(cm, interpolation='nearest', cmap=cmap)\n plt.title(title)\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n if normalize:\n cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\n thresh = cm.max() / 2.\n for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n plt.text(j, i, cm[i, j],\n horizontalalignment=\"center\",\n color=\"white\" if cm[i, j] > thresh else \"black\")\n\n plt.tight_layout()\n plt.ylabel('True label')\n plt.xlabel('Predicted label')\n \n \n# isplay some predicted output of the trained model\ndef imshow_group(X,y,y_pred=None,n_per_row=10,phase='processed'):\n '''helper function to visualize a group of images along with their categorical true labels (y) and prediction probabilities.\n Args:\n X: images\n y: categorical true labels\n y_pred: predicted class probabilities\n n_per_row: number of images per row to be plotted\n phase: If the images are plotted after resizing, pass 'processed' to phase argument. \n It will plot the image and its true label. If the image is plotted after prediction \n phase, pass predicted class probabilities to y_pred and 'prediction' to the phase argument. \n It will plot the image, the true label, and it's top 3 predictions with highest probabilities.\n '''\n n_sample=len(X)\n img_dim = X.shape[1]\n j = np.ceil(n_sample/n_per_row)\n fig = plt.figure(figsize=(20,3*j))\n \n for i, img in enumerate(X):\n plt.subplot(j,n_per_row,i+1)\n img_sq = np.squeeze(img,axis = 2)\n plt.imshow(img_sq, cmap='gray')\n #plt.imshow(img)\n if phase=='processed':\n plt.title(np.argmax(y[i]))\n if phase=='prediction':\n top_n = 3 # top 3 predictions with highest probabilities\n ind_sorted = np.argsort(y_pred[i])[::-1]\n h = img_dim + 4\n for k in range(top_n):\n string = 'pred: {} ({:.0f}%)\\n'.format(ind_sorted[k],y_pred[i,ind_sorted[k]]*100)\n plt.text(img_dim/2, h, string, horizontalalignment='center',\n verticalalignment='center')\n h += 4\n if y is not None:\n plt.text(img_dim/2, -4, 'true label: {}'.format(np.argmax(y[i])), \n horizontalalignment='center',verticalalignment='center')\n plt.axis('off')\n plt.show()\n \n \n# display some error prediction made by trained model \ndef display_errors(errors_index, img_errors, pred_errors, obs_errors):\n \"\"\" This function shows 6 images with their predicted and real labels\"\"\"\n n = 0\n nrows = 2\n ncols = 3\n fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)\n for row in range(nrows):\n for col in range(ncols):\n error = errors_index[n]\n img_sq = np.squeeze(img_errors[error], axis = 2)\n ax[row,col].grid(False)\n ax[row,col].imshow(img_sq , cmap='gray')\n ax[row,col].set_title(\"Predicted label:{}\\nTrue label :{}\".format(pred_errors[error],\n obs_errors[error]))\n n += 1","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing the Bangla Numerals Data Sets \n\nFirst we will load the data set and do some simple EDA. \n\n## NumtaDB \n\nIt is the same [Bengali.AI](https://github.com/BengaliAI) who provided [NumtaDB](https://github.com/BengaliAI/Numta) and organized an [InClass competition](https://www.kaggle.com/BengaliAI/numta) almost 2 years ago. In my experiment, I found it more well-organized dataset compare to other bengali-numerals datasets. However, the training set consists of 5 separate training-sets (namely, `'a', 'b', 'c', 'd', 'e'`); and numerals in dataset `'e'` are collected and curated version of [BanglaLekha-Isolated](https://www.sciencedirect.com/science/article/pii/S2352340917301117).","metadata":{}},{"cell_type":"code","source":"# most of the data-loading code for numta-db from: https://www.kaggle.com/sharifamit19/data-augmentation-cross-validation-ensemble#Data-Concatenation\nnumta_data_dir=os.path.join('..','input/numta/')\npaths_train_all=[]\npath_label_train_all=[]\n\narr_train = ['a','b','c','d'] # I exclude part 'e' purposely, feel free to use it.\niterator_train = len(arr_train)\n\nfor i in range(iterator_train):\n #print (arr_train[i])\n dirx= 'training-'+arr_train[i]\n paths_train_x=glob.glob(os.path.join(numta_data_dir, dirx,'*.png'))\n paths_train_all=paths_train_all+paths_train_x\n\nfor i in range(iterator_train):\n dirx= 'training-'+arr_train[i] + '.csv'\n paths_label_train = glob.glob(os.path.join(numta_data_dir,dirx))\n path_label_train_all= path_label_train_all + paths_label_train","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_key(path):\n # seperates the key of an image from the filepath\n key=path.split(sep=os.sep)[-1]\n return key\n\ndef get_data(paths_img,path_label=None,resize_dim=None):\n '''reads images from the filepaths, resizes them (if given), and returns them in a numpy array\n Args:\n paths_img: image filepaths\n path_label: pass image label filepaths while processing training data, defaults to None while processing testing data\n resize_dim: if given, the image is resized to resize_dim x resize_dim (optional)\n Returns:\n X: group of images\n y: categorical true labels\n '''\n X=[] # initialize empty list for resized images\n for i,path in enumerate(paths_img):\n img=cv2.imread(path, cv2.IMREAD_GRAYSCALE) # images loaded in grayscale\n \n # calling the preprocess_image method\n img=preprocess_image(img)\n \n # some few more preprocessing\n ret, thresh = cv2.threshold(img, 0, 20, cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)\n kernel_dilation = np.ones((2,2), np.uint8)\n img = cv2.dilate(thresh, kernel_dilation, iterations = 1)\n\n # resize\n if resize_dim is not None:\n img=cv2.resize(img,(resize_dim,resize_dim),interpolation=cv2.INTER_AREA) \n \n X.append(img) # expand image to n x n x 1 and append to the list\n \n # display progress\n if i==len(paths_img)-1:\n end='\\n'\n else: end='\\r'\n print('processed {}/{}'.format(i+1,len(paths_img)),end=end)\n \n X=np.array(X) # tranform list to numpy array\n if path_label is None:\n return X\n else:\n # Concatenate all data into one DataFrame\n df = pd.DataFrame()\n l = []\n for file_ in path_label:\n df_x = pd.read_csv(file_,index_col=None, header=0)\n l.append(df_x)\n df = pd.concat(l)\n \n #df = pd.read_csv(path_label[i]) # read labels\n df=df.set_index('filename') \n y_label=[df.loc[get_key(path)]['digit'] for path in paths_img] # get the labels corresponding to the images\n y=to_categorical(y_label,10) # transfrom integer value to categorical variable\n\n return X, y","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_size = 32\nnumtaX_train, numtaY_train = get_data(paths_train_all, path_label_train_all,\n resize_dim=img_size)\n\nprint (numtaX_train.shape)\nprint (numtaY_train.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NumtaDB: Target Distribution ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nnumtaY_label = np.argmax(numtaY_train, axis=1) \n\n# plot how many images there are in each class\nsns.countplot(numtaY_label)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NumtaDB: Training Samples ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nfor i in range(0,6): # how many imgs will show from the 3x3 grid\n plt.subplot(330 + (i+1)) # open next subplot\n plt.grid(False)\n plt.imshow(numtaX_train[i], cmap=plt.get_cmap('gray'))\n plt.title(numtaY_label[i])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BanglaLekha-Isolated \n\nOne can find the details about the dataset from [here](https://www.sciencedirect.com/science/article/pii/S2352340917301117). However, I should say, I have removed manually few samples from the datasets because of **miss-located** and **empty-image**, such as: where a image should be numeric, it was character. Again some totally black images. And also few more which appears that someone have used **grid-mask augmentation** while preparing the dataset 😂","metadata":{}},{"cell_type":"code","source":"bangalalekha_data_dir = '../input/banglalekhaisolatednumerals/'\n\nfolders = []\n[folders.append('/'+i) for i in sorted(os.listdir(bangalalekha_data_dir))]\n\nX = []\nfor folder in tqdm(folders):\n # get images in list\n images = os.listdir((bangalalekha_data_dir + folder))\n \n # preprocess each image using opencv\n for image in images:\n # load the image\n img = cv2.imread(bangalalekha_data_dir+folder+'/'+image, cv2.IMREAD_GRAYSCALE)\n img = cv2.resize(img, (32, 32), interpolation=cv2.INTER_AREA)\n \n # calling the preprocess_image method\n img=preprocess_image(img)\n X.append(img) # append this image in empty list X = []\n \nbanglalekhaX_train = np.array(X) # convert image list to numpy array\n\n# label encoding \nbanglalekhaY_train = []\n[banglalekhaY_train.extend([bc]*1940) for bc in range(0,10)] \nbanglalekhaY_train = np.stack(banglalekhaY_train)\nbanglalekhaY_train = to_categorical(banglalekhaY_train, len(np.unique(banglalekhaY_train)))\n\n# little shuffling and extend image axis for keras model compatability\nbanglalekhaX_train, banglalekhaY_train = utility(banglalekhaX_train, banglalekhaY_train, 32)\n\nprint(banglalekhaX_train.shape, banglalekhaY_train.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BanglaLekha-Isolated: Target Distribution ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nbanglalekhaY_label = np.argmax(banglalekhaY_train, axis=1) \n\n# plot how many images there are in each class\nsns.countplot(banglalekhaY_label)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BanglaLekha-Isolated: Training Samples ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 10))\n\nfor i in range(0,6): # how many imgs will show from the 3x3 grid\n plt.subplot(330 + (i+1)) # open next subplot\n plt.grid(False)\n plt.imshow(banglalekhaX_train[i][:,:,0], cmap=plt.get_cmap('gray'))\n plt.title(banglalekhaY_label[i])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ekush-Numerals \n\nRocky! A Multipurpose and Multitype Comprehensive Database for Online Off-line Bangla Handwritten Characters. Some features we've on Ekush Dataset:\n\n- Characters Recognition\n- Recognition in context\n- Gender Identification\n- Forensic Investigation\n- 673,482 character instances\n- 242 character class\n- 4 captions per image\n- 340,243 Female data\n- 333,239 Male data\n\nKudus to the [Ekush](https://shahariarrabby.github.io/ekush/#home) team, specially [AKM Shahariar Azad Rabby](https://www.kaggle.com/shahariar) for such great contribution. Their contributin awarded as [best paper award in 2018](https://shahariarrabby.github.io/ekush/#format-data).","metadata":{}},{"cell_type":"code","source":"ekush_data_dir = '/kaggle/input/ekush-bangla-handwritten-data-numerals/'\n\nfolders = []\n[folders.append('/'+i) for i in sorted(os.listdir(ekush_data_dir))]\n\nX = []\nfor folder in tqdm(folders):\n # get images in list\n images = os.listdir((ekush_data_dir + folder))\n \n # preprocess each image using opencv\n for image in images:\n # load the image\n img = cv2.imread(ekush_data_dir+folder+'/'+image, cv2.IMREAD_GRAYSCALE)\n img = cv2.resize(img, (28, 28), interpolation=cv2.INTER_AREA) # resize image to 28x28\n \n # calling the preprocess_image method\n img=preprocess_image(img)\n X.append(img) # append this image in empty list X = []\n \nekushX_train = np.array(X) # convert image to numpy array\n\n# label encoding \nekushY_train = []\n[ekushY_train.extend([bc]*3069) for bc in range(0,10)] \nekushY_train = np.stack(ekushY_train)\nekushY_train = to_categorical(ekushY_train, len(np.unique(ekushY_train)))\n\n# little shuffling and extend image axis for keras model compatability\nekushX_train, ekushY_train = utility(ekushX_train, ekushY_train, 28)\n\nprint(ekushX_train.shape, ekushY_train.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ekush: Target Distribution ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 7))\nekushY_label = np.argmax(ekushY_train, axis=1) \n\n# plot how many images there are in each class\nsns.countplot(ekushY_label)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ekush: Training Samples ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 10))\n\nfor i in range(0,6): # how many imgs will show from the 3x3 grid\n plt.subplot(330 + (i+1)) # open next subplot\n plt.grid(False)\n plt.imshow(ekushX_train[i][:,:,0], cmap=plt.get_cmap('gray'))\n plt.title(ekushY_label[i])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling \n\nThere are some plenty of reason why I choose Capsule network over Convolutinal network. Rather than explaining why I choose and what is it actualy (how's it work), I would like to simply mention some good sources to keep my notebook clean and noise free.\n\n**Capsule Network**\n- [Dynamic Routing Between Capsules](https://arxiv.org/abs/1710.09829)\n- [Awesome Capsule](https://github.com/sekwiatkowski/awesome-capsule-networks)\n- [Aurélien Géron](https://www.youtube.com/watch?v=pPN8d0E3900&t=356s)\n\nAnd Most of the implementation of the Capsule Network are mostly taken from [Xifeng Guo, PhD](https://github.com/XifengGuo/CapsNet-Keras). ","metadata":{}},{"cell_type":"code","source":"class Length(layers.Layer):\n \"\"\"\n Compute the length of vectors. This is used to compute a Tensor that has the same shape with y_true in margin_loss.\n Using this layer as model's output can directly predict labels by using `y_pred = np.argmax(model.predict(x), 1)`\n inputs: shape=[None, num_vectors, dim_vector]\n output: shape=[None, num_vectors]\n \"\"\"\n def call(self, inputs, **kwargs):\n return K.sqrt(K.sum(K.square(inputs), -1) + K.epsilon())\n\n def compute_output_shape(self, input_shape):\n return input_shape[:-1]\n\n def get_config(self):\n config = super(Length, self).get_config()\n return config\n\n\ndef squash(vectors, axis=-1):\n \"\"\"\n The non-linear activation used in Capsule. \n It drives the length of a large vector to near 1 and small vector to 0\n \n :param vectors: some vectors to be squashed, N-dim tensor\n :param axis: the axis to squash\n :return: a Tensor with same shape as input vectors\n \"\"\"\n s_squared_norm = K.sum(K.square(vectors), axis, keepdims=True)\n scale = s_squared_norm / (1 + s_squared_norm) / K.sqrt(s_squared_norm + K.epsilon())\n return scale * vectors\n\n\nclass CapsuleLayer(layers.Layer):\n \"\"\"\n The capsule layer. It is similar to Dense layer. Dense layer has `in_num` inputs, each is a scalar, the output of the \n neuron from the former layer, and it has `out_num` output neurons. CapsuleLayer just expand the output of the neuron\n from scalar to vector. So its input shape = [None, input_num_capsule, input_dim_capsule] and output shape = \\\n [None, num_capsule, dim_capsule]. For Dense Layer, input_dim_capsule = dim_capsule = 1.\n \n :param num_capsule: number of capsules in this layer\n :param dim_capsule: dimension of the output vectors of the capsules in this layer\n :param routings: number of iterations for the routing algorithm\n \"\"\"\n def __init__(self, num_capsule, dim_capsule, routings=3,\n kernel_initializer='glorot_uniform',\n **kwargs):\n super(CapsuleLayer, self).__init__(**kwargs)\n self.num_capsule = num_capsule\n self.dim_capsule = dim_capsule\n self.routings = routings\n self.kernel_initializer = initializers.get(kernel_initializer)\n\n def build(self, input_shape):\n assert len(input_shape) >= 3, \"The input Tensor should have shape=[None, input_num_capsule, input_dim_capsule]\"\n self.input_num_capsule = input_shape[1]\n self.input_dim_capsule = input_shape[2]\n\n # Transform matrix\n self.W = self.add_weight(shape=[self.num_capsule, self.input_num_capsule,\n self.dim_capsule, self.input_dim_capsule],\n initializer=self.kernel_initializer,\n name='W')\n\n self.built = True\n\n def call(self, inputs, training=None):\n # inputs.shape=[None, input_num_capsule, input_dim_capsule]\n # inputs_expand.shape=[None, 1, input_num_capsule, input_dim_capsule]\n inputs_expand = K.expand_dims(inputs, 1)\n\n # Replicate num_capsule dimension to prepare being multiplied by W\n # inputs_tiled.shape=[None, num_capsule, input_num_capsule, input_dim_capsule]\n inputs_tiled = K.tile(inputs_expand, [1, self.num_capsule, 1, 1])\n\n # Compute `inputs * W` by scanning inputs_tiled on dimension 0.\n # x.shape=[num_capsule, input_num_capsule, input_dim_capsule]\n # W.shape=[num_capsule, input_num_capsule, dim_capsule, input_dim_capsule]\n # Regard the first two dimensions as `batch` dimension,\n # then matmul: [input_dim_capsule] x [dim_capsule, input_dim_capsule]^T -> [dim_capsule].\n # inputs_hat.shape = [None, num_capsule, input_num_capsule, dim_capsule]\n inputs_hat = K.map_fn(lambda x: K.batch_dot(x, self.W, [2, 3]), elems=inputs_tiled)\n\n # Begin: Routing algorithm ---------------------------------------------------------------------#\n # The prior for coupling coefficient, initialized as zeros.\n # b.shape = [None, self.num_capsule, self.input_num_capsule].\n b = tf.zeros(shape=[K.shape(inputs_hat)[0], self.num_capsule, \n self.input_num_capsule])\n\n assert self.routings > 0, 'The routings should be > 0.'\n for i in range(self.routings):\n # c.shape=[batch_size, num_capsule, input_num_capsule]\n c = tf.nn.softmax(b, dim=1)\n\n # c.shape = [batch_size, num_capsule, input_num_capsule]\n # inputs_hat.shape=[None, num_capsule, input_num_capsule, dim_capsule]\n # The first two dimensions as `batch` dimension,\n # then matmal: [input_num_capsule] x [input_num_capsule, dim_capsule] -> [dim_capsule].\n # outputs.shape=[None, num_capsule, dim_capsule]\n outputs = squash(K.batch_dot(c, inputs_hat, [2, 2])) # [None, 10, 16]\n\n if i < self.routings - 1:\n # outputs.shape = [None, num_capsule, dim_capsule]\n # inputs_hat.shape=[None, num_capsule, input_num_capsule, dim_capsule]\n # The first two dimensions as `batch` dimension,\n # then matmal: [dim_capsule] x [input_num_capsule, dim_capsule]^T -> [input_num_capsule].\n # b.shape=[batch_size, num_capsule, input_num_capsule]\n b += K.batch_dot(outputs, inputs_hat, [2, 3])\n # End: Routing algorithm -----------------------------------------------------------------------#\n\n return outputs\n\n def compute_output_shape(self, input_shape):\n return tuple([None, self.num_capsule, self.dim_capsule])\n\n def get_config(self):\n config = {\n 'num_capsule': self.num_capsule,\n 'dim_capsule': self.dim_capsule,\n 'routings': self.routings\n }\n base_config = super(CapsuleLayer, self).get_config()\n return dict(list(base_config.items()) + list(config.items()))\n\n\ndef PrimaryCap(inputs, dim_capsule, n_channels, kernel_size, strides, padding):\n \"\"\"\n Apply Conv2D `n_channels` times and concatenate all capsules\n :param inputs: 4D tensor, shape=[None, width, height, channels]\n :param dim_capsule: the dim of the output vector of capsule\n :param n_channels: the number of types of capsules\n :return: output tensor, shape=[None, num_capsule, dim_capsule]\n \"\"\"\n output = layers.Conv2D(filters=dim_capsule*n_channels, kernel_size=kernel_size, \n strides=strides, padding=padding,name='primarycap_conv2d')(inputs)\n \n outputs = layers.Reshape(target_shape=[-1, dim_capsule], \n name='primarycap_reshape')(output)\n \n return layers.Lambda(squash, name='primarycap_squash')(outputs)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras import layers, models\nfrom keras import backend as K\nfrom keras.utils import to_categorical\nimport keras\nimport tensorflow as tf\nfrom keras.layers import (Dense, Input, Conv2D, Flatten, MaxPooling2D, \n Activation, Dropout, Average,BatchNormalization,\n GlobalAveragePooling2D, concatenate, Add)\n\n# ---------------- another way --------------\n# sound of your voice - pain in reverse though I love to hear \n\ndef CapsNet(input_shape, n_class, routings):\n x = layers.Input(shape=input_shape)\n\n # Layer 1: Just a conventional Conv2D layer\n conv1 = layers.Conv2D(filters=256, kernel_size=9, strides=1, padding='valid', \n activation='relu', name='conv1')(x)\n\n # ----------------------------- Primary CapsuleNet -----------------------------\n # Layer 2: Conv2D layer with `squash` activation, then reshape to [None, num_capsule, dim_capsule]\n primarycaps = PrimaryCap(conv1, dim_capsule=8, n_channels=32, \n kernel_size=9, strides=2, padding='valid')\n\n # Layer 3: Capsule layer. Routing algorithm works here.\n digitcaps = CapsuleLayer(num_capsule=n_class, dim_capsule=16, routings=routings,\n name='digitcaps_')(primarycaps)\n\n # Layer 4: This is an auxiliary layer to replace each capsule with its length. Just to match the true label's shape.\n # If using tensorflow, this will not be necessary. :)\n out_caps = Length(name='capsnet')(digitcaps)\n\n train_model = models.Model(inputs=[x], outputs=[out_caps]) # only encoder\n \n return train_model\n \n\ndef margin_loss(y_true, y_pred):\n \"\"\"\n Margin loss for Eq.(4). When y_true[i, :] contains not just one `1`, this loss should work too. Not test it.\n :param y_true: [None, n_classes]\n :param y_pred: [None, num_capsule]\n :return: a scalar loss value.\n \"\"\"\n L = y_true * K.square(K.maximum(0., 0.9 - y_pred)) + \\\n 0.5 * (1 - y_true) * K.square(K.maximum(0., y_pred - 0.1))\n\n return K.mean(K.sum(L, 1))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Callbacks**","metadata":{}},{"cell_type":"code","source":"def callback():\n cb = []\n\n \"\"\"\n Model-Checkpoint\n \"\"\"\n checkpoint = callbacks.ModelCheckpoint('Vanila CapsuleNet Weights Best.h5',\n save_best_only=True, \n mode='min',\n monitor='val_loss', # val_capsnet_loss : only for encoder + decoder model\n save_weights_only=True, verbose=1)\n\n cb.append(checkpoint)\n \n \"\"\"\n Early Stopping callback\n \"\"\"\n #Uncomment for usage\n early_stop = callbacks.EarlyStopping(monitor = 'val_loss', # val_capsnet_loss : only for encoder + decoder model\n min_delta=0, \n patience=20, verbose=1, \n mode = 'auto')\n cb.append(early_stop)\n \n # learning reate decay\n #lr_decay = callbacks.LearningRateScheduler(schedule = lambda epoch: 0.001 * np.exp(-epoch / 10.))\n lr_decay = callbacks.LearningRateScheduler(schedule=lambda epoch: 0.001 * (0.9 ** epoch))\n cb.append(lr_decay)\n \n # Callback that streams epoch results to a csv file.\n log = callbacks.CSVLogger('log.csv')\n cb.append(log)\n\n return cb","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training Functions**\n\n> - compile the model\n> - data augmentation\n> - fit generator","metadata":{}},{"cell_type":"code","source":"def train_caps(model, data, epoch_size_frac=1.0, training = False, wg = None):\n \"\"\"\n Training a CapsuleNet\n :param model: the CapsuleNet model\n :param args: arguments\n :return: The trained model\n \"\"\"\n # unpacking the data\n (x_train, y_train), (x_val, y_val) = data\n\n # callbacks functions\n cb = callback()\n \n # compile the model\n '''Only for Encoder builded Model'''\n # if we like to exclude decoder section - just testing \n model.compile(optimizer=Adam(lr=1e-3),\n loss=[margin_loss],\n loss_weights=[1.],\n metrics={'capsnet': 'accuracy'})\n\n \n # --------------Begin: Training with data augmentation -------\n def train_generator(x, y, batch_size, shift_fraction=0.):\n train_datagen = ImageDataGenerator(width_shift_range = shift_fraction, \n height_shift_range = shift_fraction, \n rotation_range=20, \n shear_range=0.1, \n zoom_range=0.1, \n horizontal_flip=False, \n vertical_flip=False, \n fill_mode='nearest') \n \n generator = train_datagen.flow(x, y, batch_size=batch_size)\n \n while 1:\n x_batch, y_batch = generator.next() \n yield ([x_batch, y_batch])\n \n if training: \n # Training with data augmentation. \n '''Model for only Encoder Output'''\n history = model.fit_generator(generator = train_generator(x_train, y_train, 64, 0.1),\n steps_per_epoch = int(y_train.shape[0] / 64),\n validation_steps=x_val.shape[0] // 64,\n epochs = 100, \n validation_data = train_generator(x_val, y_val, 128),\n callbacks = cb)\n else:\n model.load_weights(wg) \n # -----End: Training with data augmentation -------#\n \n return model","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## NumtaDB Capsule Training ","metadata":{}},{"cell_type":"code","source":"# Set the random seed\nrandom_seed = 2019\n\n# Randomly split the data sets\nfrom sklearn.model_selection import train_test_split\n\n### Reshape \nnumtaX_train = numtaX_train.reshape(numtaX_train.shape[0],32,32,1).astype('float32')/255.\n\n# Split the train and the validation set for the fitting\nX_train, X_val, Y_train, Y_val = train_test_split(numtaX_train,\n numtaY_train, \n test_size = 0.1,\n random_state=random_seed,\n shuffle=True)\n\nprint('90% for training ', X_train.shape, Y_train.shape)\nprint('10% for validation ', X_val.shape, Y_val.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# define model\nmodel = CapsNet(input_shape=[32, 32, 1],\n n_class=10,\n routings=3)\n\n# calls training functions\ntrain_caps(model = model, data = ((X_train, Y_train), (X_val, Y_val)), \n training = False, wg = '../input/weightabatches/numta.h5',\n epoch_size_frac = 0.5)\n\n# print summary\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learning Curv**","metadata":{}},{"cell_type":"code","source":"log_file = \"../input/logsbatches/numta.csv\"\nplot_log(log_file)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NumtaDB Confusion Matrix ","metadata":{}},{"cell_type":"code","source":"# Predict the values from the validation dataset\n# encoder \nY_pred = model.predict([X_val], batch_size=64, verbose = 1)\n\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred, axis = 1) \n\n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val, axis = 1) \n\n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10), normalize=False,\n title='Confusion Matrix')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NumtaDB Classification Report","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report\ntarget_names = [\"Class {}\".format(i) for i in range(10)]\n\nscores = model.evaluate(X_val, Y_val, # model for encoder \n verbose = 1, batch_size=64) \n\nprint(classification_report(Y_true, Y_pred_classes, target_names = target_names))\nprint(scores)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NumtaDB Display Some Prediction","metadata":{}},{"cell_type":"code","source":"# encoder \npredictions_prob = model.predict([X_val], \n batch_size = 32, \n verbose = True)\n\nn_sample = 10\nnp.random.seed(42)\nind = np.random.randint(0, len(X_val), size = n_sample)\n\nimshow_group(X = X_val[ind],y = None, y_pred = predictions_prob[ind], phase='prediction')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### NumtaDB Some Wrong Prediction","metadata":{}},{"cell_type":"code","source":"# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors, axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## BanglaLekha Capsule Training ","metadata":{}},{"cell_type":"code","source":"del X_train, X_val, Y_train, Y_val\ndel numtaX_train, numtaY_train\ndel model\n\n'''\nX_train , Y_train : Actual training\nX_val , Y_val : Evaluationg\n'''\nX_train, X_val, Y_train, Y_val = train_test_split(banglalekhaX_train, \n banglalekhaY_train, \n test_size=0.20, \n random_state=random_seed,\n shuffle = True)\n\nprint('80% for training ', X_train.shape, Y_train.shape)\nprint('20% for validation ', X_val.shape, Y_val.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# define model\nmodel = CapsNet(input_shape=[32, 32, 1],\n n_class=10,\n routings=3)\n\n# calls training functions\ntrain_caps(model = model, data = ((X_train, Y_train), (X_val, Y_val)), \n training = False, wg = '../input/weightabatches/banglalekha.h5',\n epoch_size_frac = 0.5)\n\n# print summary\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learning Curve**","metadata":{}},{"cell_type":"code","source":"log_file = \"../input/logsbatches/banglalekha.csv\"\nplot_log(log_file)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BanglaLekha Confusion Matrix","metadata":{}},{"cell_type":"code","source":"# Predict the values from the validation dataset\n# encoder \nY_pred = model.predict([X_val], batch_size=64, verbose = 1)\n\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred, axis = 1) \n\n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val, axis = 1) \n\n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10), normalize=False,\n title='Confusion Matrix')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BanglaLekha Classification Report","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report\ntarget_names = [\"Class {}\".format(i) for i in range(10)]\n\nscores = model.evaluate(X_val, Y_val, # model for encoder \n verbose = 1, batch_size=64) \n\nprint(classification_report(Y_true, Y_pred_classes, target_names = target_names))\nprint(scores)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BanglaLekha Some Prediction","metadata":{}},{"cell_type":"code","source":"# encoder \npredictions_prob = model.predict([X_val], \n batch_size = 32, \n verbose = True)\n\nn_sample = 10\nnp.random.seed(42)\nind = np.random.randint(0, len(X_val), size = n_sample)\n\nimshow_group(X = X_val[ind],y = None, y_pred = predictions_prob[ind], phase='prediction')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BanglaLekha Some Wrong Prediction","metadata":{}},{"cell_type":"code","source":"# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors, axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ekush Capsule Training ","metadata":{}},{"cell_type":"code","source":"del X_train, X_val, Y_train, Y_val\ndel banglalekhaX_train, banglalekhaY_train\ndel model\n\n'''\nX_train , Y_train : Actual training\nX_val , Y_val : Evaluationg\n'''\nX_train, X_val, Y_train, Y_val = train_test_split(ekushX_train, \n ekushY_train,\n test_size=0.20, \n random_state=random_seed, \n shuffle = True)\n\nprint('80% for training ', X_train.shape, Y_train.shape)\nprint('20% for validation ', X_val.shape, Y_val.shape)","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# define model\nmodel = CapsNet(input_shape=[28, 28, 1],\n n_class=10,\n routings=3)\n\n# calls training functions\ntrain_caps(model = model, data = ((X_train, Y_train), (X_val, Y_val)), \n training = False, wg = '../input/weightabatches/ekush.h5',\n epoch_size_frac = 0.5)\n\n# print summary\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learning Curve**","metadata":{}},{"cell_type":"code","source":"log_file = \"../input/logsbatches/ekush.csv\"\nplot_log(log_file)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ekush Confusion Matrix","metadata":{}},{"cell_type":"code","source":"# Predict the values from the validation dataset\n# encoder \nY_pred = model.predict([X_val], batch_size=64, verbose = 1)\n\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred, axis = 1) \n\n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val, axis = 1) \n\n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n\n# plot the confusion matrix\nplot_confusion_matrix(confusion_mtx, classes = range(10), normalize=False,\n title='Confusion Matrix')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ekush Classification Report","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report\ntarget_names = [\"Class {}\".format(i) for i in range(10)]\n\nscores = model.evaluate(X_val, Y_val, # model for encoder \n verbose = 1, batch_size=64) \n\nprint(classification_report(Y_true, Y_pred_classes, target_names = target_names))\nprint(scores)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ekush Some Prediction","metadata":{}},{"cell_type":"code","source":"# encoder \npredictions_prob = model.predict([X_val], \n batch_size = 32, \n verbose = True)\n\nn_sample = 10\nnp.random.seed(42)\nind = np.random.randint(0, len(X_val), size = n_sample)\n\nimshow_group(X = X_val[ind],y = None, y_pred = predictions_prob[ind], phase='prediction')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ekush Some Wrong Prediction","metadata":{}},{"cell_type":"code","source":"# Errors are difference between predicted labels and true labels\nerrors = (Y_pred_classes - Y_true != 0)\n\nY_pred_classes_errors = Y_pred_classes[errors]\nY_pred_errors = Y_pred[errors]\nY_true_errors = Y_true[errors]\nX_val_errors = X_val[errors]\n\n# Probabilities of the wrong predicted numbers\nY_pred_errors_prob = np.max(Y_pred_errors, axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = Y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 6 errors \nmost_important_errors = sorted_dela_errors[6:]\n\n# Show the top 6 errors\ndisplay_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n\n# Conclusion\n\nThat's it for the numerals part. I hope you found this kernel useful or interesting. Here we've used **CapsNet** to classify bengali digits on three different numerals datasets. Though we've also worked on the character datasets as well (almost 50 classes) and the results were pretty promising. So, we've studied on the **CapsNet** for better understanding. We've proposed many **features-blocks** during our experiments with **CapsNets** and successfully we've achieved SOTA results [compare to the published works](https://www.kaggle.com/c/bengaliai-cv19/discussion/122604#700071) until now (2020). \n\n\nAfter getting promising results, we wanted to use the vanila **CapsNet** to the [Bengali.AI Handwritten Grapheme Classification](https://www.kaggle.com/c/bengaliai-cv19/notebooks) competition, but it appeared that the training parameter was too high, so we ended up that time. But if we can optimize the model parameter and efficient as well, definitely it would be great 💥","metadata":{}}]}