{"cells":[{"metadata":{},"cell_type":"markdown","source":"## LSTM RNN - FastText and GloVe embeddings"},{"metadata":{},"cell_type":"markdown","source":"Even if the best-performing models are multilingual (Multilingual BERT, XLM-Roberta...), classifying translated text is a worth-trying alternative. In this kernel, highly inspired from kernels from the previous Jigsaw competition, I experiment a LSTM RNN equipped with an Attention module. \n\nFor future research, I stumbled upon this technique that might be worth exploring: https://arxiv.org/pdf/1804.07983.pdf\n\nAn other idea might be to try a weighted loss to offset the imbalance of the training set.\n\nI did not have time to further explore fine-tuning, the model is clearly overfitting. You might want to decrease the number of epochs, learning rate and add regularization...\n\n### References:\n\n- https://www.kaggle.com/qqgeogor/keras-lstm-attention-glove840b-lb-0-043\n- https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n- https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part2-usage\n- https://www.kaggle.com/christofhenkel/bert-embeddings-lstm"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os, warnings, pickle, gc, re, string\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nimport tensorflow as tf\n\nfrom tensorflow.keras import initializers, regularizers, constraints, optimizers, layers\n\nfrom tensorflow.keras.layers import Layer, Dense, Input, Activation, Embedding, SpatialDropout1D, Bidirectional, LSTM, GRU, GlobalMaxPooling1D, GlobalAveragePooling1D, Dropout\nfrom tensorflow.keras.layers import concatenate, add\n\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, LearningRateScheduler, EarlyStopping\nfrom tensorflow.keras import backend as K\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\n\nfrom sklearn.metrics import roc_auc_score\n\nwarnings.simplefilter('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# HYPERPARAMETERS\n\nMAX_LEN = 220\nMAX_FEATURES = 100000\nEMBED_SIZE = 600\n\nBATCH_SIZE = 128\nN_EPOCHS = 5\n\nLEARNING_RATE = 8e-4\n\n# We will concatenate Crawl and GloVe embeddings\n\nCRAWL_EMB_PATH = '../input/pickled-glove840b300d-for-10sec-loading/glove.840B.300d.pkl'\nGLOVE_EMB_PATH = '../input/pickled-crawl300d2m-for-kernel-competitions/crawl-300d-2M.pkl'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_training_curves(training, validation, title, subplot):\n    '''\n    Quickly display training curves\n    '''\n    if subplot % 10 == 1:\n        plt.subplots(figsize=(10, 10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    \n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model' + title)\n    ax.set_ylabel(title)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_coeffs(word, *arr):\n    return word, np.asarray(arr, dtype='float32')\n\n\ndef load_embeddings(embed_dir):\n    with open(embed_dir, 'rb') as  infile:\n        embeddings = pickle.load(infile)\n        return embeddings","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_embedding_matrix(word_index, embeddings_index, max_features, lower = True, verbose = True):\n    embedding_matrix = np.zeros((max_features, 300))\n    for word, i in tqdm(word_index.items(), len=(word_index.items())):\n        if lower:\n            word = word.lower()\n        if i >= max_features: continue\n        try:\n            embedding_vector = embeddings_index[word]\n        except:\n            embedding_vector = embeddings_index[\"unknown\"]\n        if embedding_vector is not None:\n            # words not found in embedding index will be all-zeros.\n            embedding_matrix[i] = embedding_vector\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_matrix(word_index, embeddings_index):\n    embedding_matrix = np.zeros((len(word_index) + 1,300))\n    for word, i in word_index.items():\n        try:\n            embedding_matrix[i] = embeddings_index[word]\n        except:\n            embedding_matrix[i] = embeddings_index[\"unknown\"]\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Attention(Layer):\n    \"\"\"\n    Custom Keras attention layer\n    Reference: https://www.kaggle.com/qqgeogor/keras-lstm-attention-glove840b-lb-0-043\n    \"\"\"\n    def __init__(self, step_dim, W_regularizer=None, b_regularizer=None, \n                 W_constraint=None, b_constraint=None, bias=True, **kwargs):\n\n        self.supports_masking = True\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = None\n        super(Attention, self).__init__(**kwargs)\n\n        self.param_W = {\n            'initializer': initializers.get('glorot_uniform'),\n            'name': '{}_W'.format(self.name),\n            'regularizer': regularizers.get(W_regularizer),\n            'constraint': constraints.get(W_constraint)\n        }\n        self.W = None\n\n        self.param_b = {\n            'initializer': 'zero',\n            'name': '{}_b'.format(self.name),\n            'regularizer': regularizers.get(b_regularizer),\n            'constraint': constraints.get(b_constraint)\n        }\n        self.b = None\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.features_dim = input_shape[-1]\n        self.W = self.add_weight(shape=(input_shape[-1],), \n                                 **self.param_W)\n\n        if self.bias:\n            self.b = self.add_weight(shape=(input_shape[1],), \n                                     **self.param_b)\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        step_dim = self.step_dim\n        features_dim = self.features_dim\n\n        eij = K.reshape(\n            K.dot(K.reshape(x, (-1, features_dim)), K.reshape(self.W, (features_dim, 1))),\n            (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n        eij = K.tanh(eij)\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0], self.features_dim","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load text data into memory"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We create a balanced\n\nprint('Loading train sets...')\ntrain1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\n\ntrain = pd.concat([\n    train1[['comment_text', 'toxic']],\n    train2[['comment_text', 'toxic']].query('toxic==1'),\n    train2[['comment_text', 'toxic']].query('toxic==0').sample(n=100000, random_state=0)\n])\n\ndel train1, train2\n\nprint('Loading validation sets...')\nvalid = pd.read_csv('/kaggle/input/val-en-df/validation_en.csv')\n\nprint('Loading test sets...')\ntest = pd.read_csv('/kaggle/input/test-en-df/test_en.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cleaning data"},{"metadata":{},"cell_type":"markdown","source":"I clean the data using Christoph Henkel's kernels: \n- https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part2-usage\n- https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part2-usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"misspell_dict = {\"aren't\": \"are not\", \"can't\": \"cannot\", \"couldn't\": \"could not\",\n                 \"didn't\": \"did not\", \"doesn't\": \"does not\", \"don't\": \"do not\",\n                 \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n                 \"he'd\": \"he would\", \"he'll\": \"he will\", \"he's\": \"he is\",\n                 \"i'd\": \"I had\", \"i'll\": \"I will\", \"i'm\": \"I am\", \"isn't\": \"is not\",\n                 \"it's\": \"it is\", \"it'll\": \"it will\", \"i've\": \"I have\", \"let's\": \"let us\",\n                 \"mightn't\": \"might not\", \"mustn't\": \"must not\", \"shan't\": \"shall not\",\n                 \"she'd\": \"she would\", \"she'll\": \"she will\", \"she's\": \"she is\",\n                 \"shouldn't\": \"should not\", \"that's\": \"that is\", \"there's\": \"there is\",\n                 \"they'd\": \"they would\", \"they'll\": \"they will\", \"they're\": \"they are\",\n                 \"they've\": \"they have\", \"we'd\": \"we would\", \"we're\": \"we are\",\n                 \"weren't\": \"were not\", \"we've\": \"we have\", \"what'll\": \"what will\",\n                 \"what're\": \"what are\", \"what's\": \"what is\", \"what've\": \"what have\",\n                 \"where's\": \"where is\", \"who'd\": \"who would\", \"who'll\": \"who will\",\n                 \"who're\": \"who are\", \"who's\": \"who is\", \"who've\": \"who have\",\n                 \"won't\": \"will not\", \"wouldn't\": \"would not\", \"you'd\": \"you would\",\n                 \"you'll\": \"you will\", \"you're\": \"you are\", \"you've\": \"you have\",\n                 \"'re\": \" are\", \"wasn't\": \"was not\", \"we'll\": \" will\", \"tryin'\": \"trying\"}\n\n\ndef _get_misspell(misspell_dict):\n    misspell_re = re.compile('(%s)' % '|'.join(misspell_dict.keys()))\n    return misspell_dict, misspell_re\n\n\ndef replace_typical_misspell(text):\n    misspellings, misspellings_re = _get_misspell(misspell_dict)\n\n    def replace(match):\n        return misspellings[match.group(0)]\n\n    return misspellings_re.sub(replace, text)\n    \n\npuncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']',\n          '>', '%', '=', '#', '*', '+', '\\\\', '•', '~', '@', '£', '·', '_', '{', '}', '©', '^',\n          '®', '`', '<', '→', '°', '€', '™', '›', '♥', '←', '×', '§', '″', '′', 'Â', '█',\n          '½', 'à', '…', '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶',\n          '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', '▒', '：', '¼',\n          '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲',\n          'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', '∙', '）', '↓', '、', '│', '（', '»', '，', '♪',\n          '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√']\n\n\ndef clean_text(x):\n    x = str(x)\n    for punct in puncts + list(string.punctuation):\n        if punct in x:\n            x = x.replace(punct, f' {punct} ')\n    return x\n\n\ndef clean_numbers(x):\n    return re.sub(r'\\d+', ' ', x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess(train, valid, test, tfms):\n    for tfm in tfms:\n        print(tfm.__name__)\n        train['comment_text'] = train['comment_text'].progress_apply(tfm)\n        valid['comment_text_en'] = valid['comment_text_en'].progress_apply(tfm)\n        test['content'] = test['content'].progress_apply(tfm)\n    \n    return train, valid, test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = [replace_typical_misspell, clean_text, clean_numbers]\ntrain, valid, test = preprocess(train, valid, test, tfms)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Encoding datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = Tokenizer(num_words=MAX_FEATURES, filters='', lower=False)\n\nprint('Fitting tokenizer...')\ntokenizer.fit_on_texts(list(train['comment_text']) + list(valid['comment_text_en']) + list(test['content_en']))\nword_index = tokenizer.word_index\n\nprint('Building training set...')\nX_train = tokenizer.texts_to_sequences(list(train['comment_text']))\ny_train = train['toxic'].values\n\nprint('Building validation set...')\nX_valid = tokenizer.texts_to_sequences(list(valid['comment_text_en']))\ny_valid = valid['toxic'].values\n\nprint('Building test set ...')\nX_test = tokenizer.texts_to_sequences(list(test['content_en']))\n\nprint('Padding sequences...')\nX_train = pad_sequences(X_train, maxlen=MAX_LEN)\nX_valid = pad_sequences(X_valid, maxlen=MAX_LEN)\nX_test = pad_sequences(X_test, maxlen=MAX_LEN)\n\ny_train = train.toxic.values\ny_valid = valid.toxic.values\n\ndel tokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build embedding matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Loading Crawl embeddings...')\ncrawl_embeddings = load_embeddings(CRAWL_EMB_PATH)\n\nprint('Loading GloVe embeddings...')\nglove_embeddings = load_embeddings(GLOVE_EMB_PATH)\n\nprint('Building matrices...')\nembedding_matrix_1 = build_matrix(word_index, crawl_embeddings)\nembedding_matrix_2 = build_matrix(word_index, glove_embeddings)\n\nprint('Concatenating embedding matrices...')\nembedding_matrix = np.concatenate([embedding_matrix_1, embedding_matrix_2], axis=1)\n\ndel embedding_matrix_1, embedding_matrix_2\ndel crawl_embeddings, glove_embeddings\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build dataset objects"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((X_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((X_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(X_test)\n    .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(word_index, embedding_matrix, verbose=True):\n    '''\n    credits go to: https://www.kaggle.com/thousandvoices/simple-lstm/\n    '''\n    sequence_input = Input(shape=(MAX_LEN,), dtype=tf.int32)\n    \n    embedding_layer = Embedding(*embedding_matrix.shape,\n                                weights=[embedding_matrix],\n                                trainable=False)\n    \n    x = embedding_layer(sequence_input)\n    x = SpatialDropout1D(0.3)(x)\n    x = Bidirectional(LSTM(256, return_sequences=True))(x)\n    x = Bidirectional(LSTM(128, return_sequences=True))(x)\n    \n    att = Attention(MAX_LEN)(x)\n    avg_pool1 = GlobalAveragePooling1D()(x)\n    max_pool1 = GlobalMaxPooling1D()(x)\n    hidden = concatenate([att, avg_pool1, max_pool1])\n    \n    hidden = Dense(512, activation='relu')(hidden)\n    hideen = Dense(128, activation='relu')(hidden)\n\n    out = Dense(1, activation='sigmoid')(hidden)\n    \n    model = Model(sequence_input, out)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = build_model(word_index, embedding_matrix)\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=[tf.keras.metrics.AUC()])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"file_weights = 'best_model.h5'\n#cb1 = ModelCheckpoint(file_weights, save_best_only=True)\n\ncb2 = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=3)\n\ncb3 = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=2, verbose=1, cooldown=0, min_lr=0.0001)\n\ncb4 = LearningRateScheduler(lambda epoch: LEARNING_RATE * (0.6 ** epoch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### First stage"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = X_train.shape[0] // BATCH_SIZE\n\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    callbacks=[cb4],\n    epochs=N_EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_training_curves(\n    train_history.history['loss'],\n    train_history.history['val_loss'],\n    'loss',\n    211)\n\ndisplay_training_curves(\n    train_history.history['auc'],\n    train_history.history['val_auc'],\n    'AUC',\n    212)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fitting on valid set"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = X_valid.shape[0] // BATCH_SIZE\n\ntrain_history = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    callbacks=[cb4],\n    epochs=N_EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = model.predict(test_dataset, verbose=1)\nsub['toxic'] = preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}