{"cells":[{"metadata":{"_uuid":"4f6e2eb2c89ade43e6aaaae53cc11a1de8d49d35"},"cell_type":"markdown","source":"# 🚀 Multi-Task and Transfer Learning\n\nRecently in Deep NLP, pretraining, multi-task and transfer learning is all the hype! The general idea is to leverage additional data / targets before or during your training. This is one of the reasons why BERT (the current state of the art) is so great:\n\n1. You can do the pretraining on a large corpus of unlabeled text\n2. With additional objectives they make sure the model generalizes well\n\nOf course, we've been doing something similar by using pretrained word embeddings, though nowhere near as cool as BERT does. Unfortunately, we don't have additional data sources in this competition, so can't exactly build much on 1. But is there a possibility to extend on 2? Can we add additional (auxiliary) objectives to our training?\n\nLet me give you some examples of what BERT tries to optimize in parallel and why this may help. For example, they don't just look at how words relate to each other, as is done in Word2Vec and friends. They also teach the model to decide whether two sentences follow each other in the text. Basically, they add this objective to the training: \"Hey BERT, given these two sentences, do you think they are following each other in real text, or not?\". This is a genius way to give the model some additional understanding of text, while keeping the problem relatively simple. It can't just stop at understanding how the single words interact. It also needs to grog to some extent what makes a text flow and what are some causal relationships across sentences. And it comes at no cost, since the data can be extracted from the text corpus in an unsupervised fashion.\n\nBtw, here's a great article on the matter of Multi-Task Learning in NLP by Sebastian Ruder: http://ruder.io/multi-task-learning-nlp/"},{"metadata":{"_uuid":"6026fd218267099c5fb55e712cd04667e5116c2c"},"cell_type":"markdown","source":"# Our Auxiliary Task\n\nHow can we achieve something similar in this competition? Of course, we could try the sentence trick, but I assume most if not all questions only have a single sentence in their title. This may also lead to overfitting on the training set.\n\nSo, here's my idea: Sentiment is probably somehow correlated to insincerity, but not equivalent. We are allowed to use NLTK, and NLTK comes with a sentiment analysis tool. It's simple, but better than nothing. How about we analyze each sentence, get a sentiment score, and add that as a secondary target for our neural network to estimate?\n\nSo instead of asking this question \"Hey model, given this question, give me how insincere it is\" we ask: \"Hey model, given this question, tell me both how insincere it is and what the sentiment is like\".\n\nIf this works, this may have two effects:\n1. The model may learn **more general features**, reducing overfitting\n2. The model may receive **more information** relevant for the insincerity decision, increasing it's performance"},{"metadata":{"_uuid":"c9d2abe74ece202d48a8ba97f2503d9501491366"},"cell_type":"markdown","source":"# The Experiment\n\nTo explore the effect of this idea, we run three experiments using the same model architecture:\n\n1. **BASELINE**: Train on the sincerity target only\n2. **MULTI-TASK LEARNING**: Train on both targets at the same time\n3. **TRANSFER LEARNING**: Pretrain on sentiment, then fine-tune on insincerity\n\nWe compare their performance in terms of F1 on a hold-out validation set.\n\n**Note**: As we have all had to learn, the variance of outcomes is pretty high in this competition. That's why I ran each of the experiments several times and I encourage you to do the same.\n\nI've taken some of the standard boilerplate code from other kernels. Thanks to the original authors!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm, tqdm_notebook\nimport math\nfrom sklearn.model_selection import KFold\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, CuDNNLSTM, Conv1D, Add\nfrom keras.layers import Bidirectional, GlobalMaxPooling1D, GlobalAveragePooling1D, SpatialDropout1D, Lambda, Concatenate\nfrom keras.optimizers import Nadam, Adam\nfrom keras.models import Model, Sequential\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\ntrain_n = len(train_df)\ntest_n = len(test_df)\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d429e25d135801ba2e0a0aebc42d713612564a87"},"cell_type":"markdown","source":"## Getting the sentiments\n\nNow for each question, let's ask NLTK what it thinks about the sentiment. The `compound` score is ranged from -1 to 1, so we normalize it to be in [0, 1] and call it `polarity`."},{"metadata":{"trusted":true,"_uuid":"179fce3323698c57901aeeaae42e761f73ba52a7"},"cell_type":"code","source":"# NLTK sentiment cell\nprint('\\nGetting sentiments...')\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA\nsia = SIA()\nsentiments = np.zeros(train_n)\n# for _, row in train_df.sample(10).iterrows():\n#     print(row.question_text, sia.polarity_scores(row.question_text))\n\nfor i, (_, row) in tqdm_notebook(enumerate(train_df.iterrows()), total=train_n):\n    sentiments[i] = sia.polarity_scores(row.question_text)['compound']\n\ntrain_df['sentiment'] = pd.Series(sentiments)\ntrain_df['sentiment_target'] = (train_df['sentiment'] + 1) / 2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31b0ef4379fca2aa131cd6420c6a676178e8ae93"},"cell_type":"markdown","source":"Alright, let's see if there is any correlation at all between the polarity score and the binary insincerity label:"},{"metadata":{"trusted":true,"_uuid":"982bb0891f6bb4cd3a8f0cf6eabacd2d252233d3"},"cell_type":"code","source":"# Get correlation between strong polarity and insincerity\nprint('\\nCorrelations to polarity:')\n# train_df['strong_polarity'] = pd.Series((train_df['sentiment'] >= 0.5) | (train_df['sentiment'] <= -0.5)).astype(int)\ntrain_df['polarity'] = train_df['sentiment'].abs()\nprint('Pearson ', train_df['target'].corr(train_df['polarity'], method='pearson'))\nprint('Kendall ', train_df['target'].corr(train_df['polarity'], method='kendall'))\nprint('Spearman', train_df['target'].corr(train_df['polarity'], method='spearman'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae2c593b76dad4ef09149b2961f06482a063a550"},"cell_type":"markdown","source":"Only minimally! But that's fine, if they were equivalent, we wouldn't need this additional target at all.\n\nLet's look at some datapoints:"},{"metadata":{"trusted":true,"_uuid":"537f1bc843cee8525431d33d5a7d9157bae8329b"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b86a09f93adbd82b37eece13a8900cb27becc048"},"cell_type":"code","source":"print(train_df.describe())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f38deb74944ff4f6cec6ba42b32c6f20d35ef0fd"},"cell_type":"markdown","source":"As you can see, the polarity is not very evenly distributed: Most of the entries are below the 0.5 mark. But it definitely should contain more information than the binary, unbalanced insincerity tag."},{"metadata":{"_uuid":"eb415eed076111b4668c6a3e2b53469c435e68b8"},"cell_type":"markdown","source":"## Preparing the data and the model\n\nI'm going for a very simple data loading and model here."},{"metadata":{"trusted":true,"_uuid":"cb4066f1aece9928d195b24a2af114c53aabf3ca"},"cell_type":"code","source":"print('\\nPreparing data...')\n\n# some config values \nembed_size = 300 # how big is each word vector\nmax_features = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 50 # max number of words in a question to use\n\n# fill up the missing values\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\n\n# Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X = tokenizer.texts_to_sequences(train_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n# Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\n\n# Get the target values\ntrain_y = train_df[['target', 'sentiment_target']].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9894c71bf0c6b5c9e3139c87a00d5f2deda8399d"},"cell_type":"code","source":"print('\\nGetting embeddings...')\nEMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n\nfor word, i in tqdm_notebook(word_index.items(), total=len(word_index)):\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"30aac2cf06e356e57e714b0cbe655ee71028d663"},"cell_type":"code","source":"def f1(y_true, y_pred):\n    '''\n    metric from here \n    https://stackoverflow.com/questions/43547402/how-to-calculate-f1-macro-in-keras\n    '''\n    def recall(y_true, y_pred):\n        \"\"\"Recall metric.\n\n        Only computes a batch-wise average of recall.\n\n        Computes the recall, a metric for multi-label classification of\n        how many relevant items are selected.\n        \"\"\"\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        \"\"\"Precision metric.\n\n        Only computes a batch-wise average of precision.\n\n        Computes the precision, a metric for multi-label classification of\n        how many selected items are relevant.\n        \"\"\"\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives / (predicted_positives + K.epsilon())\n        return precision\n    \n    # So we only measure F1 on the target y value:\n    y_true = y_true[:, 0]\n    y_pred = y_pred[:, 0]\n    \n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"da189cb63e8af50140b61ef3334cf1e6940af62a","scrolled":true},"cell_type":"code","source":"# Create a simple 1-layer LSTM model\ndef create_model(embedding_trainable=False, dropout=0.1, size=32, n_outputs=2, lr=0.003):\n    model = Sequential()\n    model.add(Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=embedding_trainable))\n    model.add(SpatialDropout1D(dropout))\n    model.add(Bidirectional(CuDNNLSTM(size, return_sequences=True)))\n    model.add(GlobalMaxPooling1D())\n    model.add(Dropout(dropout))\n    model.add(Dense(n_outputs, activation=\"sigmoid\"))\n\n    model.compile(\n        loss='binary_crossentropy',\n        optimizer=Adam(lr=lr),\n        metrics=[f1]\n    )\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf53a0052f9235c99fbb4f4de6b64c0e368bf3b0"},"cell_type":"markdown","source":"## Training on just the target label (BASELINE)\n\nLet's see how well our model performs if we train it just on the target label. The most important metric we should be watching is `val_f1`! For statistical validity, we will run each experiment on three splits and then average the F1 score we've reached."},{"metadata":{"trusted":true,"_uuid":"6293c2d2c6a7dbf5ed4679ba88b1b8b303b88a80"},"cell_type":"code","source":"scores = []\nfor train_idx, val_idx in KFold(n_splits=5, shuffle=True, random_state=123).split(train_X, train_y):\n    model = create_model(n_outputs=1)\n\n    model.fit(\n        train_X[train_idx], train_y[train_idx, 0],\n        validation_data=(train_X[val_idx], train_y[val_idx, 0]),\n        batch_size=1024,\n        epochs=3,\n        verbose=2,\n    )\n\n    scores.append(model.evaluate(train_X[val_idx], train_y[val_idx, 0], batch_size=1024, verbose=0)[1])\n    print()\n\nprint('Average F1:', np.mean(scores))\nprint('Standard Deviation:', np.std(scores))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4c61fb9435c55be4ac3e0986e29695d096e7ccb4"},"cell_type":"markdown","source":"Not bad, not very good (because our model is very small for the sake of this experiment). This is our baseline."},{"metadata":{"_uuid":"23cc52f4eb0907e81ca700e7d31335587d1d6904"},"cell_type":"markdown","source":"## Training on both targets (MULTI-TASK LEARNING)\n\nNow let's repeat the training, but this time our model gets two outputs: One for the insincerity, one for polarity.\n\n**Note** that I designed the F1 metric to measure only the F1 on the insincerity target, so the scores are directly comparable to the baseline."},{"metadata":{"trusted":true,"_uuid":"aba861065caa982ca75ab6e964c7e22e6a41f9c4"},"cell_type":"code","source":"scores = []\nfor train_idx, val_idx in KFold(n_splits=5, shuffle=True, random_state=123).split(train_X, train_y):\n    model = create_model(n_outputs=2)\n\n    model.fit(\n        train_X[train_idx], train_y[train_idx],\n        validation_data=(train_X[val_idx], train_y[val_idx]),\n        batch_size=1024,\n        epochs=3,\n        verbose=2,\n    )\n\n    scores.append(model.evaluate(train_X[val_idx], train_y[val_idx], batch_size=1024)[1])\n    print()\n\nprint('Average F1:', np.mean(scores))\nprint('Standard Deviation:', np.std(scores))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66ae14d2014f3b09b5f0e954138218b435d27d50"},"cell_type":"markdown","source":"Very interesting! We get an improved result with less overfitting!"},{"metadata":{"_uuid":"c9e1b43f7efdec8137c43fe36f9c69a819bb44da"},"cell_type":"markdown","source":"## Using polarity for pretraining (TRANSFER LEARNING)\n\nNow let's split the training into two parts: First, we train on the polarity for a few epochs, then we change targets and train for our actual insincerity target. We also reduce the learning rate for fine-tuning to avoid catastrophic forgetting.\n\nI tried several configurations here, but could not find any that worked better than this:"},{"metadata":{"trusted":true,"_uuid":"940895f98ba907bc97a72eb4361576c22efd1905","scrolled":true},"cell_type":"code","source":"scores = []\nfor train_idx, val_idx in KFold(n_splits=5, shuffle=True, random_state=123).split(train_X, train_y):\n    model = create_model(n_outputs=1)\n\n    model.fit(\n        train_X[train_idx], train_y[train_idx, 1],\n        validation_data=(train_X[val_idx], train_y[val_idx, 1]),\n        batch_size=1024,\n        epochs=2,\n        verbose=2,\n    )\n\n    # Reduce the model's learning rate:\n    model.compile(\n        loss=model.loss,\n        metrics=model.metrics,\n        optimizer=Adam(lr=0.001),\n    )\n    \n    # Fine-tuning:\n    model.fit(\n        train_X[train_idx], train_y[train_idx, 0],\n        validation_data=(train_X[val_idx], train_y[val_idx, 0]),\n        batch_size=512,\n        epochs=2,\n        verbose=2,\n    )\n    \n    scores.append(model.evaluate(train_X[val_idx], train_y[val_idx, 0], batch_size=1024)[1])\n    print()\n\nprint('Average F1:', np.mean(scores))\nprint('Standard Deviation:', np.std(scores))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac92c5322f5b5c3ef16901eceec26a748cfac8da"},"cell_type":"markdown","source":"It seems that pretraining does not help much, though maybe we just need to explore more training configurations to make it work."},{"metadata":{"_uuid":"73008359e7f4cd4a87c7d57b24aefc58b5680183"},"cell_type":"markdown","source":"## Discussion\n\nOver several repetitions of my experiment, I could confirm a positive effect of adding the auxiliary target:\n\n**Higher validation F1 performance at less overfitting!**\n\nThis is exactly what we were hoping for, though the effect is not large.\n\nNote that statistical validity is no simple matter. I gave it my best given the limitations of these kernels, but this is no real proof that adding auxiliary targets will help you with your score. E.g. it could behave different in larger models, with longer training times, etc. Let's find find out 😊\n\nIf you guys have more ideas for auxiliary targets, please try them out and share them with the community, or comment down below! 😊"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}