{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\n### This is a step-by-step commented kernel to approach simple EDA and baseline modeling on this competition and other NLP competitions.\n\n\n### Special thanks to Dieter and his work on this competition. I would also like to thank Andrew Lukyanenko for his [advice](https://www.kaggle.com/general/89512) on writing good kernels.\n\n\n### PLEASE UPVOTE IF YOU LIKE THIS KERNEL"},{"metadata":{},"cell_type":"markdown","source":"<center><img src=\"https://i.imgur.com/CtyQ8Ag.png\" width=\"250px\"></center>"},{"metadata":{},"cell_type":"markdown","source":"### Import necessary libraries."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport json\nfrom tqdm import tqdm\ntqdm.pandas()\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nimport gc\n\nfrom textblob import TextBlob\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\nimport gensim\n\nfrom sklearn.model_selection import KFold\n\nfrom keras.layers import *\nfrom keras.initializers import *\nfrom keras.constraints import *\nfrom keras.regularizers import *\nfrom keras.activations import *\nfrom keras.optimizers import *\nimport keras.backend as K\nfrom keras.models import Model\nfrom keras.utils import plot_model\nfrom keras.utils.vis_utils import model_to_dot\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\n\nfrom IPython.display import SVG\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Download data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/jigsaw-unintended-bias-in-toxicity-classification/train.csv')\ntest_df = pd.read_csv('../input/jigsaw-unintended-bias-in-toxicity-classification/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Initialize constants for data extraction and training"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEXT_COL = 'comment_text'\nEMB_PATH = '../input/fasttext-crawl-300d-2m/crawl-300d-2M.vec'\nMAXLEN = 128\nENDLEN = 32\nMAX_FEATURES = 100000\nEMBED_SIZE = 300\nBATCH_SIZE = 2048\nNUM_EPOCHS = 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Basic EDA"},{"metadata":{},"cell_type":"markdown","source":"### Number of characters in the sentence"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"lengths = train_df[TEXT_COL].apply(len)\ntrain_df['lengths'] = lengths\nlengths = train_df.loc[train_df['lengths']<1125]['lengths']\nsns.distplot(lengths, color='r')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we seem to have a bimodal distribution of character length in the data. Although the lengths seem to be heavily skewed to the lower lengths, we see another clear peak around the 1000 character mark."},{"metadata":{},"cell_type":"markdown","source":"### Number of words in the sentence"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"words = train_df[TEXT_COL].apply(lambda x: len(x) - len(''.join(x.split())) + 1)\ntrain_df['words'] = words\nwords = train_df.loc[train_df['words']<200]['words']\nsns.distplot(words, color='g')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks like we have a clear unimodal left-skewed distribution of the number of words in the data."},{"metadata":{},"cell_type":"markdown","source":"### Average Word Length"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"avg_word_len = train_df[TEXT_COL].apply(lambda x: 1.0*len(''.join(x.split()))/(len(x) - len(''.join(x.split())) + 1))\ntrain_df['avg_word_len'] = avg_word_len\navg_word_len = train_df.loc[train_df['avg_word_len']<10]['avg_word_len']\nsns.distplot(avg_word_len, color='b')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have a simple bell-shaped normal distribution of the average word length with a mean of around 4.5"},{"metadata":{},"cell_type":"markdown","source":"### Sentiment (negativity)\n#### The orange distribution is label 0 (non-toxic) and the purple distribution is label 1 (toxic)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# SIA = SentimentIntensityAnalyzer()\n# polarity_0 = train_df.loc[train_df.target<0.5][TEXT_COL].apply(lambda x: SIA.polarity_scores(x))\n# polarity_1 = train_df.loc[train_df.target>0.5][TEXT_COL].apply(lambda x: SIA.polarity_scores(x))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# sns.distplot([polarity['neg'] for polarity in polarity_0], color='darkorange')\n# sns.distplot([polarity['neg'] for polarity in polarity_1], color='purple')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nClearly, the purple distribution has a higher mean than the orange distribution. Both distributions have a somewhat equal spread (standard deviation). Although both distributions are skewed leftwards, the orange distribution has a stronger leftward skew."},{"metadata":{},"cell_type":"markdown","source":"This shows that the toxic comments generally tend to be more negative on average. This is probably because most comments considered \"toxic\" generally spread negative emotions like hate, anger or insult against certain people, beliefs or opinions."},{"metadata":{},"cell_type":"markdown","source":"### Sentiment (positivity)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# sns.distplot([polarity['pos'] for polarity in polarity_0], color='darkorange')\n# sns.distplot([polarity['pos'] for polarity in polarity_1], color='purple')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both the orange and purple distributions are very similar in every way : bimodality, shape, skew, mean etc."},{"metadata":{},"cell_type":"markdown","source":"This shows that there is no significant difference between positivity in toxic and non-toxic comments."},{"metadata":{},"cell_type":"markdown","source":"### Sentiment (neutrality)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# sns.distplot([polarity['neu'] for polarity in polarity_0], color='darkorange')\n# sns.distplot([polarity['neu'] for polarity in polarity_1], color='purple')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The orange distribution clearly has a higher mean than the purple distribution. Both distributions have a somewhat equal spread (standard deviation). Although both distributions are skewed rightwards, the orange distribution has a stronger rightward skew."},{"metadata":{},"cell_type":"markdown","source":"This shows that the non-toxic samples generally tend to be more neutral on average. This is probably because non-toxic comments generally do not have extreme emotions (positive or negative), but many toxic comments do."},{"metadata":{},"cell_type":"markdown","source":"### Sentiment (compoundness / complexity of comment)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# sns.distplot([polarity['compound'] for polarity in polarity_0], color='darkorange')\n# sns.distplot([polarity['compound'] for polarity in polarity_1], color='purple')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both distributions have a somewhat similar shape : evenly spread out with little skew in one direction and a small insignificant peak in the center (at around 0). But, the orange distribution seems to be slightly more skewed to the right and the purple distribution skewed to the left."},{"metadata":{},"cell_type":"markdown","source":"This shows that toxic comments are generally less gramatically complex than non-toxic comments. Maybe, this is because toxic comments are generally more blunt and short. They often attack/insult people at a personal level. On the other hand, non-toxic comments generally try to share a perspective or make a point, and thus, they tend to be more gramatically complex or compounded on average."},{"metadata":{"_kg_hide-input":false},"cell_type":"markdown","source":"**I tried to use a spelling corrector, but it did not improve the model's performance. The code for spelling correction is below.**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(''' # https://www.kaggle.com/cpmpml/spell-checker-using-word2vec\nspell_model = gensim.models.KeyedVectors.load_word2vec_format('../input/quora-insincere-questions-classification/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec')\nwords = spell_model.index2word\nw_rank = {}\n\nfor i,word in enumerate(words):\n    w_rank[word] = i\nWORDS = w_rank\n\n# Use fast text as vocabulary\n\n\ndef words(text): return re.findall(r'\\w+', text.lower())\n\ndef P(word): \n    \"Probability of `word`.\"\n    # use inverse of rank as proxy\n    # returns 0 if the word isn't in the dictionary\n    return - WORDS.get(word, 0)\n\ndef correction(word): \n    \"Most probable spelling correction for word.\"\n    return max(candidates(word), key=P)\n\ndef candidates(word): \n    \"Generate possible spelling corrections for word.\"\n    return (known([word]) or known(edits1(word)) or [word])\n\ndef known(words): \n    \"The subset of `words` that appear in the dictionary of WORDS.\"\n    return set(w for w in words if w in WORDS)\n\ndef edits1(word):\n    \"All edits that are one edit away from `word`.\"\n    letters    = 'abcdefghijklmnopqrstuvwxyz'\n    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]\n    deletes    = [L + R[1:]               for L, R in splits if R]\n    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]\n    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]\n    inserts    = [L + c + R               for L, R in splits for c in letters]\n    return set(deletes + transposes + replaces + inserts)\n\ndef edits2(word): \n    \"All edits that are two edits away from `word`.\"\n    return (e2 for e1 in edits1(word) for e2 in edits1(e1))\n\ndef singlify(word):\n    return \"\".join([letter for i,letter in enumerate(word) if i == 0 or letter != word[i-1]]) ''')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Functions for creating the embedding matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_coefs(word,*arr):\n    return word, np.asarray(arr, dtype='float32')\n\ndef load_embeddings(embed_dir=EMB_PATH):\n    embedding_index = dict(get_coefs(*o.strip().split(\" \")) for o in tqdm(open(embed_dir)))\n    return embedding_index\n\ndef build_embedding_matrix(word_index, embeddings_index, MAX_FEATURES, lower = True, verbose = True):\n    embedding_matrix = np.zeros((MAX_FEATURES, EMBED_SIZE))\n    for word, i in tqdm(word_index.items(),disable = not verbose):\n        if lower:\n            word = word.lower()\n        if i >= MAX_FEATURES: continue\n        try:\n            embedding_vector = embeddings_index[word]\n        except:\n            embedding_vector = embeddings_index[\"unknown\"]\n        if embedding_vector is not None:\n            # words not found in embedding index will be all-zeros.\n            embedding_matrix[i] = embedding_vector\n    return embedding_matrix\n\ndef build_matrix(word_index, embeddings_index):\n    embedding_matrix = np.zeros((len(word_index) + 1, EMBED_SIZE))\n    for word, i in word_index.items():\n        try:\n            embedding_matrix[i] = embeddings_index[word]\n        except:\n            embedding_matrix[i] = embeddings_index[\"unknown\"]\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Tokenize and pad the sentences"},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = Tokenizer(num_words=MAX_FEATURES, lower=True)\ntokenizer.fit_on_texts(list(train_df[TEXT_COL]) + list(test_df[TEXT_COL]))\nword_index = tokenizer.word_index\n\n# X_train = tokenizer.texts_to_sequences(list(train_df[TEXT_COL]))\n# X_test = tokenizer.texts_to_sequences(list(test_df[TEXT_COL]))\n# X_train = pad_sequences(X_train, maxlen=MAXLEN)\n# X_test = pad_sequences(X_test, maxlen=MAXLEN)\n# y_train = train_df['target'].values\n\n#del tokenizer\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the embeddings"},{"metadata":{"trusted":true},"cell_type":"code","source":"# embeddings_index = load_embeddings()\n# embedding_matrix = build_matrix(word_index, embeddings_index)\n# del embeddings_index\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling"},{"metadata":{},"cell_type":"markdown","source":"### Create the Weighted Average Attention layer\nThis layer computes a weighted average of the channels of the 3D input (from the recurrent layer) across timesteps. It performs better than the Attention layer most kernels are using"},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://github.com/bfelbo/DeepMoji/blob/master/deepmoji/attlayer.py\nclass AttentionWeightedAverage(Layer):\n    \"\"\"\n    Computes a weighted average of the different channels across timesteps.\n    Uses 1 parameter pr. channel to compute the attention value for a single timestep.\n    \"\"\"\n\n    def __init__(self, return_attention=False, **kwargs):\n        self.init = initializers.get('uniform')\n        self.supports_masking = True\n        self.return_attention = return_attention\n        super(AttentionWeightedAverage, self).__init__(** kwargs)\n\n    def build(self, input_shape):\n        self.input_spec = [InputSpec(ndim=3)]\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight(shape=(input_shape[2], 1),\n                                 name='{}_W'.format(self.name),\n                                 initializer=self.init)\n        self.trainable_weights = [self.W]\n        super(AttentionWeightedAverage, self).build(input_shape)\n\n    def call(self, x, mask=None):\n        # computes a probability distribution over the timesteps\n        # uses 'max trick' for numerical stability\n        # reshape is done to avoid issue with Tensorflow\n        # and 1-dimensional weights\n        logits = K.dot(x, self.W)\n        x_shape = K.shape(x)\n        logits = K.reshape(logits, (x_shape[0], x_shape[1]))\n        ai = K.exp(logits - K.max(logits, axis=-1, keepdims=True))\n\n        # masked timesteps have zero weight\n        if mask is not None:\n            mask = K.cast(mask, K.floatx())\n            ai = ai * mask\n        att_weights = ai / (K.sum(ai, axis=1, keepdims=True) + K.epsilon())\n        weighted_input = x * K.expand_dims(att_weights)\n        result = K.sum(weighted_input, axis=1)\n        if self.return_attention:\n            return [result, att_weights]\n        return result\n\n    def get_output_shape_for(self, input_shape):\n        return self.compute_output_shape(input_shape)\n\n    def compute_output_shape(self, input_shape):\n        output_len = input_shape[2]\n        if self.return_attention:\n            return [(input_shape[0], output_len), (input_shape[0], input_shape[1])]\n        return (input_shape[0], output_len)\n\n    def compute_mask(self, input, input_mask=None):\n        if isinstance(input_mask, list):\n            return [None] * len(input_mask)\n        else:\n            return None","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Create the Capsule layer\nThe classic Capsule layer as proposed by Geoffrey Hinton in 2017. This implementation is taken from [here](https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/layers/capsule.py) on keras-contrib."},{"metadata":{"trusted":true},"cell_type":"code","source":"class Capsule(Layer):\n    \"\"\"Capsule Layer implementation in Keras\n       This implementation is based on Dynamic Routing of Capsules,\n       Geoffrey Hinton et. al.\n       The Capsule Layer is a Neural Network Layer which helps\n       modeling relationships in image and sequential data better\n       than just CNNs or RNNs. It achieves this by understanding\n       the spatial relationships between objects (in images)\n       or words (in text) by encoding additional information\n       about the image or text, such as angle of rotation,\n       thickness and brightness, relative proportions etc.\n       This layer can be used instead of pooling layers to\n       lower dimensions and still capture important information\n       about the relationships and structures within the data.\n       A normal pooling layer would lose a lot of\n       this information.\n       This layer can be used on the output of any layer\n       which has a 3-D output (including batch_size). For example,\n       in image classification, it can be used on the output of a\n       Conv2D layer for Computer Vision applications. Also,\n       it can be used on the output of a GRU or LSTM Layer\n       (Bidirectional or Unidirectional) for NLP applications.\n       The default activation function is 'linear'. But, this layer\n       is generally used with the 'squash' activation function\n       (recommended). To use the squash activation function, do :\n       from keras_contrib.activations import squash\n       capsule = Capsule(num_capsule=10,\n                         dim_capsule=10,\n                         routings=3,\n                         share_weights=True,\n                         activation=squash)\n       # Example usage :\n           1). COMPUTER VISION\n           input_image = Input(shape=(None, None, 3))\n           conv_2d = Conv2D(64,\n                            (3, 3),\n                            activation='relu')(input_image)\n           capsule = Capsule(num_capsule=10,\n                             dim_capsule=16,\n                             routings=3,\n                             activation='relu',\n                             share_weights=True)(conv_2d)\n           2). NLP\n           maxlen = 72\n           max_features = 120000\n           input_text = Input(shape=(maxlen,))\n           embedding = Embedding(max_features,\n                                 embed_size,\n                                 weights=[embedding_matrix],\n                                 trainable=False)(input_text)\n           bi_gru = Bidirectional(GRU(64,\n                                      return_seqeunces=True))(embedding)\n           capsule = Capsule(num_capsule=5,\n                             dim_capsule=5,\n                             routings=4,\n                             activation='sigmoid',\n                             share_weights=True)(bi_gru)\n       # Arguments\n           num_capsule : Number of Capsules (int)\n           dim_capsules : Dimensions of the vector output of each Capsule (int)\n           routings : Number of dynamic routings in the Capsule Layer (int)\n           share_weights : Whether to share weights between Capsules or not\n           (boolean)\n           activation : Activation function for the Capsules\n           regularizer : Regularizer for the weights of the Capsules\n           initializer : Initializer for the weights of the Caspules\n           constraint : Constraint for the weights of the Capsules\n       # Input shape\n            3D tensor with shape:\n            (batch_size, input_num_capsule, input_dim_capsule)\n            [any 3-D Tensor with the first dimension as batch_size]\n       # Output shape\n            3D tensor with shape:\n            (batch_size, num_capsule, dim_capsule)\n       # References\n        - [Dynamic-Routing-Between-Capsules]\n          (https://arxiv.org/pdf/1710.09829.pdf)\n        - [Keras-Examples-CIFAR10-CNN-Capsule]\"\"\"\n\n    def __init__(self,\n                 num_capsule,\n                 dim_capsule,\n                 routings=3,\n                 share_weights=True,\n                 initializer='glorot_uniform',\n                 activation=None,\n                 regularizer=None,\n                 constraint=None,\n                 **kwargs):\n        super(Capsule, self).__init__(**kwargs)\n        self.num_capsule = num_capsule\n        self.dim_capsule = dim_capsule\n        self.routings = routings\n        self.share_weights = share_weights\n\n        self.activation = activations.get(activation)\n        self.regularizer = regularizers.get(regularizer)\n        self.initializer = initializers.get(initializer)\n        self.constraint = constraints.get(constraint)\n\n    def build(self, input_shape):\n        input_dim_capsule = input_shape[-1]\n        if self.share_weights:\n            self.W = self.add_weight(name='capsule_kernel',\n                                     shape=(1,\n                                            input_dim_capsule,\n                                            self.num_capsule *\n                                            self.dim_capsule),\n                                     initializer=self.initializer,\n                                     regularizer=self.regularizer,\n                                     constraint=self.constraint,\n                                     trainable=True)\n        else:\n            input_num_capsule = input_shape[-2]\n            self.W = self.add_weight(name='capsule_kernel',\n                                     shape=(input_num_capsule,\n                                            input_dim_capsule,\n                                            self.num_capsule *\n                                            self.dim_capsule),\n                                     initializer=self.initializer,\n                                     regularizer=self.regularizer,\n                                     constraint=self.constraint,\n                                     trainable=True)\n\n        self.build = True\n\n    def call(self, inputs):\n        if self.share_weights:\n            u_hat_vectors = K.conv1d(inputs, self.W)\n        else:\n            u_hat_vectors = K.local_conv1d(inputs, self.W, [1], [1])\n\n        # u_hat_vectors : The spatially transformed input vectors (with local_conv_1d)\n\n        batch_size = K.shape(inputs)[0]\n        input_num_capsule = K.shape(inputs)[1]\n        u_hat_vectors = K.reshape(u_hat_vectors, (batch_size,\n                                                  input_num_capsule,\n                                                  self.num_capsule,\n                                                  self.dim_capsule))\n\n        u_hat_vectors = K.permute_dimensions(u_hat_vectors, (0, 2, 1, 3))\n        routing_weights = K.zeros_like(u_hat_vectors[:, :, :, 0])\n\n        for i in range(self.routings):\n            capsule_weights = K.softmax(routing_weights, 1)\n            outputs = K.batch_dot(capsule_weights, u_hat_vectors, [2, 2])\n            if K.ndim(outputs) == 4:\n                outputs = K.sum(outputs, axis=1)\n            if i < self.routings - 1:\n                outputs = K.l2_normalize(outputs, -1)\n                routing_weights = K.batch_dot(outputs, u_hat_vectors, [2, 3])\n                if K.ndim(routing_weights) == 4:\n                    routing_weights = K.sum(routing_weights, axis=1)\n\n        return self.activation(outputs)\n\n    def compute_output_shape(self, input_shape):\n        return (None, self.num_capsule, self.dim_capsule)\n\n    def get_config(self):\n        config = {'num_capsule': self.num_capsule,\n                  'dim_capsule': self.dim_capsule,\n                  'routings': self.routings,\n                  'share_weights': self.share_weights,\n                  'activation': activations.serialize(self.activation),\n                  'regularizer': regularizers.serialize(self.regularizer),\n                  'initializer': initializers.serialize(self.initializer),\n                  'constraint': constraints.serialize(self.constraint)}\n\n        base_config = super(Capsule, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The squash activation function to use with the Capsule layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"def squash(x, axis=-1):\n    \"\"\"\n    Squash activation function (generally used in Capsule layers).\n    \"\"\"\n    s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()\n    scale = K.sqrt(s_squared_norm) / (0.5 + s_squared_norm)\n    return scale * x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Create a model that:\n1). Takes embedded sentences as input and passes it through spatial dropout\n\n2). Passes the output through a Bidirectional LSTM layer\n\n3). The LSTM output is passed through Max Pooling and Weighted Average Attention\n\n4). The same LSTM output also passed through a Capsule layer (+ Flatten)\n\n5). The outputs of Pooling, Attention and Capsule are then concatenated\n\n6). They are then passed through a dense layer with one neuron\n\n7). Finally, a sigmoid activation is applied to the output to get probabilities"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model():\n    inp = Input(shape=(MAXLEN,))\n    \n    embed_inp = Embedding(3303 + 1, EMBED_SIZE, input_length=MAXLEN, trainable=False)(inp)\n    drop_inp = SpatialDropout1D(0.3)(embed_inp)\n    \n    bi_lstm = Bidirectional(CuDNNLSTM(64, return_sequences=True))(drop_inp)\n    \n    max_pool_lstm = GlobalMaxPooling1D()(bi_lstm)\n    attention_lstm = AttentionWeightedAverage()(bi_lstm)\n    capsule = Capsule(num_capsule=5, dim_capsule=5, routings=4, activation=squash)(bi_lstm)\n    capsule = Flatten()(capsule)\n    \n    x = concatenate([max_pool_lstm, attention_lstm, capsule], axis=1)\n    outp = Dense(1, activation='sigmoid')(x)\n    \n    model = Model(inp, outp)\n    model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.005, decay=0.001), metrics=['acc'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"warnings.filterwarnings('ignore')\nsample_model  = get_model()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model summary"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model plot visualization"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(model_to_dot(sample_model).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train model with KFold Cross Validation and Early Stopping Callback (and generate predictions)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# warnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ckpt = ModelCheckpoint(f'model.h5', save_best_only = True)\n# es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=3)\n# model = get_model()\n# split = np.int32(0.8*len(X_train))\n\n# model.fit(X_train[:split],\n          # y_train[:split]>0.5,\n          # batch_size=BATCH_SIZE,\n          # epochs=NUM_EPOCHS//2,\n          # validation_data=(X_train[split:], y_train[split:]>0.5),\n          # callbacks = [es,ckpt])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save model weights and architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"# model_json = model.to_json()\n# with open(\"model.json\", \"w\") as json_file:\n    # json_file.write(model_json)\n    \nwith open('word_index.json', 'w') as f:\n    json.dump(word_index, f)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.4"}},"nbformat":4,"nbformat_minor":1}