{"cells":[{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://i.imgur.com/I08M6KP.png\" width=\"600px\">","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this project, I will try to finetune a roBERTa base transformer model to predict whether a Quora question is sincere or insincere. An insincere question is defined as a question intended to make a statement rather than look for helpful answers. Some characteristics that can signify that a question is insincere:\n\n* Has a non-neutral tone\n  * Has an exaggerated tone to underscore a point about a group of people\n  * Is rhetorical and meant to imply a statement about a group of people\n  \n\n* Is disparaging or inflammatory\n  * Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype\n  * Makes disparaging attacks/insults against a specific person or group of people\n  * Based on an outlandish premise about a group of people\n  * Disparages against a characteristic that is not fixable and not measurable\n  \n\n* Isn't grounded in reality\n  * Based on false information, or contains absurd assumptions\n  \n\n* Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers\n\nSolving this problem will help combat online trolls at scale and help Quora uphold their policy of “Be Nice, Be Respectful” and continue to be a place for sharing and growing the world’s knowledge. I will use **PyTorch XLA** (PyTorch for TPUs) and **huggingface transformers** for this project.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Set up PyTorch-XLA\n\n* These few lines of code sets up PyTorch XLA for us.\n* We need PyTorch XLA to help us train PyTorch models on TPU.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!export XLA_USE_BF16=1\n!pip install -q colored","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Install and import libraries\n\n* We will import several different packages and libraries required for different parts of the project. For example, we import numpy and pandas for data manipulation, torch and torch_xla for modeling, and plotly for visualization.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport time\nimport colored\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom colored import fg, bg, attr\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport torch\nimport torch.nn as nn\nfrom torch.optim import Adam\nfrom torch import LongTensor, FloatTensor, DoubleTensor\nfrom torch.utils.data import Dataset, DataLoader, sampler\nfrom torch.utils.data.distributed import DistributedSampler\n\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.parallel_loader as pl\nimport torch_xla.distributed.xla_multiprocessing as xmp\n\nfrom tqdm.notebook import tqdm\nfrom sklearn.utils import shuffle\nfrom transformers import RobertaModel, RobertaTokenizer\n\nfrom keras.utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences as pad","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define hyperparameters and load data\n\n* Here, we define the required hyperparameters such as the training batch size, learning rate, training/validation split percentage, etc.\n* We also load the training and tessting data required for the project using the read_csv function from the pandas library.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"EPOCHS = 5\nMAXLEN = 64\nSPLIT = 0.8\nDROP_RATE = 0.3\nLR = (4e-5, 1e-2)\nBATCH_SIZE = 256\nVAL_BATCH_SIZE = 8192\nMODEL_SAVE_PATH = 'insincerity_model.pt'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)\ntorch.manual_seed(42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_df = pd.read_csv('../input/quora-insincere-questions-classification/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define PyTorch Dataset\n\n* Now we define a PyTorch Dataset which will help us feed data to the roBERTa model for training and inference.\n* We remove leading and trailing whitespaces using .strip(), tokenize the values using huggingface, and pad the tokens using keras.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class QuoraDataset(Dataset):\n    def __init__(self, data, tokenizer):\n        self.text = data.question_text\n        self.data, self.tokenizer = data, tokenizer\n        self.target = data.target if \"target\" in data.columns else [-1]*len(data)\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, i):\n        pg, tg = 'post', 'post'\n        target = [self.target[i]]\n        question = str(self.text[i])\n        quest_ids = self.tokenizer.encode(question.strip())\n\n        attention_mask_idx = len(quest_ids) - 1\n        if 0 not in quest_ids: quest_ids = 0 + quest_ids\n        quest_ids = pad([quest_ids], maxlen=MAXLEN, value=1, padding=pg, truncating=tg)\n\n        attention_mask = np.zeros(MAXLEN)\n        attention_mask[1:attention_mask_idx] = 1\n        attention_mask = attention_mask.reshape((1, -1))\n        if 2 not in quest_ids: quest_ids[-1], attention_mask[-1] = 2, 0\n        return FloatTensor(target), LongTensor(quest_ids), LongTensor(attention_mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define tokenizer\n\n* Here we simply define the RobertaTokenizer from huggingface which we use in the Dataset to generate tokens from words.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = 'roberta-base'\ntokenizer = RobertaTokenizer.from_pretrained(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define roBERTa-base model\n\n* Now, we get to the interesting part: training roBERTa base! roBERTa base is a pretrained language model developed by Facebook AI.\n* We will use roBERTa with its pretrained weights and add a custom (Dropout + Dense) head at the top to turn it into a binary text classifier.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Roberta(nn.Module):\n    def __init__(self):\n        super(Roberta, self).__init__()\n        self.dropout = nn.Dropout(DROP_RATE)\n        self.dense_output = nn.Linear(768, 1)\n        self.roberta = RobertaModel.from_pretrained(model)\n\n    def forward(self, inp, att):\n        inp = inp.view(-1, MAXLEN)\n        _, self.feat = self.roberta(inp, att)\n        return self.dense_output(self.dropout(self.feat))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"m = Roberta()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"print(m); del m; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define binary cross entropy and F1 score in PyTorch\n\n* Here we implement binary cross entropy and F1 score functions in PyTorch.\n* BCE is the loss function which is commonly used in binary classification tasks and helps us finetune roBERTa's weights.\n* F1 Score is an evaluation metric ranging from 0 to 1, and we use it instead of accuracy because it is immune to class imabalance.\n* Our dataset is almost 93% sincere and 7% insincere. This extreme imbalance makes accuracy an unreliable metric and thus, F1 Score.\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def bce(y_pred, y_true):\n    return nn.BCEWithLogitsLoss()(y_pred, y_true)*len(y_pred)\n\ndef f1_score(y_pred, y_true):\n    y_true = y_true.squeeze()\n    y_pred = torch.round(nn.Sigmoid()(y_pred)).squeeze()\n    \n    tp = (y_true * y_pred).sum().to(torch.float32)\n    fp = ((1 - y_true) * y_pred).sum().to(torch.float32)\n    fn = (y_true * (1 - y_pred)).sum().to(torch.float32)\n    tn = ((1 - y_true) * (1 - y_pred)).sum().to(torch.float32)\n    \n    epsilon = 1e-7\n    recall = tp / (tp + fn + epsilon)\n    precision = tp / (tp + fp + epsilon)\n    return 2*(precision*recall) / (precision + recall + epsilon)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split data and train model on all 8 TPU cores\n\n* Now, we will train the roBERTa base model to classify tweet sentiments.\n* We define a simple training loop in PyTorch to train the model and validate it after each epoch.\n* We parallelize the training on all 8 TPU cores using <code>xmp.spawn</code> from PyTorch XLA (distributes training).\n* We aslo use <code>DistributedSampler</code> and <code>ParallelLoader</code> to parallelize data sampling and model training.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def print_metric(data, batch, epoch, start, end, metric, typ):\n    t = typ, metric, \"%s\", data, \"%s\"\n    if typ == \"Train\": pre = \"BATCH %s\" + str(batch-1) + \"%s  \"\n    if typ == \"Val\": pre = \"\\nEPOCH %s\" + str(epoch+1) + \"%s  \"\n    time = np.round(end - start, 1); time = \"Time: %s{}%s s\".format(time)\n    fonts = [(fg(211), attr('reset')), (fg(212), attr('reset')), (fg(213), attr('reset'))]\n    xm.master_print(pre % fonts[0] + \"{} {}: {}{}{}\".format(*t) % fonts[1] + \"  \" + time % fonts[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"global val_f1s; global train_f1s\nglobal val_losses; global train_losses\n\ndef train_fn(df):\n    split = np.int32(SPLIT*len(df))\n    val_df, train_df = df[split:], df[:split]\n\n    val_df = val_df.reset_index(drop=True)\n    val_dataset = QuoraDataset(val_df, tokenizer)\n    val_sampler = DistributedSampler(val_dataset, num_replicas=8,\n                                     rank=xm.get_ordinal(), shuffle=True)\n    \n    val_loader = DataLoader(dataset=val_dataset, batch_size=VAL_BATCH_SIZE,\n                            sampler=val_sampler, num_workers=0, drop_last=True)\n\n    train_df = train_df.reset_index(drop=True)\n    train_dataset = QuoraDataset(train_df, tokenizer)\n    train_sampler = DistributedSampler(train_dataset, num_replicas=8,\n                                       rank=xm.get_ordinal(), shuffle=True)\n\n    train_loader = DataLoader(dataset=train_dataset, batch_size=BATCH_SIZE,\n                              sampler=train_sampler, num_workers=0, drop_last=True)\n\n    device = xm.xla_device()\n    network = Roberta().to(device)\n    optimizer = Adam([{'params': network.roberta.parameters(), 'lr': LR[0]},\n                      {'params': network.dense_output.parameters(), 'lr': LR[1]}])\n\n    val_losses, val_f1s = [], []\n    train_losses, train_f1s = [], []\n    \n    start = time.time()\n    xm.master_print(\"STARTING TRAINING ...\\n\")\n\n    for epoch in range(EPOCHS):\n        fonts = (fg(48), attr('reset'))\n        xm.master_print((\"EPOCH %s\" + str(epoch+1) + \"%s\") % fonts)\n\n        val_parallel = pl.ParallelLoader(val_loader, [device]).per_device_loader(device)\n        train_parallel = pl.ParallelLoader(train_loader, [device]).per_device_loader(device)\n        \n        batch = 1\n        network.train()\n        for train_batch in train_parallel:\n            train_targ, train_in, train_att = train_batch\n\n            train_in = train_in.to(device)\n            train_att = train_att.to(device)\n            train_targ = train_targ.to(device)\n            train_preds = network.forward(train_in, train_att)\n\n            train_loss = bce(train_preds, train_targ)/len(train_preds)\n            train_f1 = f1_score(train_preds, train_targ.squeeze(dim=1))\n\n            optimizer.zero_grad()\n            train_loss.backward()\n            xm.optimizer_step(optimizer)\n            \n            end = time.time()\n            batch = batch + 1\n            is_print = batch % 10 == 1\n            f1 = np.round(train_f1.item(), 3)\n            if is_print: print_metric(f1, batch, None, start, end, \"F1\", \"Train\")\n\n        val_loss, val_f1, val_points = 0, 0, 0\n\n        network.eval()\n        with torch.no_grad():\n            for val_batch in val_parallel:\n                targ, val_in, val_att = val_batch\n                \n                targ = targ.to(device)\n                val_in = val_in.to(device)\n                val_att = val_att.to(device)\n                pred = network.forward(val_in, val_att)\n\n                val_points += len(targ)\n                val_loss += bce(pred, targ).item()\n                val_f1 += f1_score(pred, targ.squeeze(dim=1)).item()*len(pred)\n        \n        end = time.time()\n        val_f1 /= val_points\n        val_loss /= val_points\n        f1 = xm.mesh_reduce('f1', val_f1, lambda x: sum(x)/len(x))\n        loss = xm.mesh_reduce('loss', val_loss, lambda x: sum(x)/len(x))\n        print_metric(np.round(f1, 3), None, epoch, start, end, \"F1\", \"Val\")\n    \n        xm.master_print(\"\")\n        val_f1s.append(f1); train_f1s.append(train_f1.item())\n        val_losses.append(loss); train_losses.append(train_loss.item())\n\n    xm.master_print(\"ENDING TRAINING ...\")\n    xm.save(network.state_dict(), MODEL_SAVE_PATH); del network; gc.collect()\n    \n    metric_lists = [val_losses, train_losses, val_f1s, train_f1s]\n    metric_names = ['val_loss_', 'train_loss_', 'val_f1_', 'train_f1_']\n    \n    for i, metric_list in enumerate(metric_lists):\n        for j, metric_value in enumerate(metric_list):\n            torch.save(metric_value, metric_names[i] + str(j) + '.pt')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FLAGS = {}\ntrain_df = shuffle(train_df)\ntrain_df = train_df.reset_index(drop=True)\n\ndef _mp_fn(rank, flags): train_fn(df=train_df)\nxmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize loss and F1 score over time\n\n* We now visualize how the loss and F1 score of the model change over time.\n* We can see that the model eventually converges to around 0.7 F1 Score at the end.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"val_f1s = [0] + [torch.load('val_f1_{}.pt'.format(i)) for i in range(EPOCHS)]\ntrain_f1s = [0] + [torch.load('train_f1_{}.pt'.format(i)) for i in range(EPOCHS)]\nval_losses = [0.25] + [torch.load('val_loss_{}.pt'.format(i)) for i in range(EPOCHS)]\ntrain_losses = [0.25] + [torch.load('train_loss_{}.pt'.format(i)) for i in range(EPOCHS)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = go.Figure()\n\nfig.add_trace(go.Scatter(x=np.arange(1, len(val_losses)+1),\n                         y=val_losses, mode=\"lines+markers\", name=\"val\",\n                         marker=dict(color=\"indianred\", line=dict(width=.5,\n                                                                  color='rgb(0, 0, 0)'))))\n\nfig.add_trace(go.Scatter(x=np.arange(1, len(train_losses)+1),\n                         y=train_losses, mode=\"lines+markers\", name=\"train\",\n                         marker=dict(color=\"darkorange\", line=dict(width=.5,\n                                                                   color='rgb(0, 0, 0)'))))\n\nfig.update_layout(xaxis_title=\"Epochs\", yaxis_title=\"Binary Cross Entropy\",\n                  title_text=\"Binary Cross Entropy vs. Epochs\", template=\"plotly_white\", paper_bgcolor=\"#f0f0f0\")\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = go.Figure()\n\nfig.add_trace(go.Scatter(x=np.arange(1, len(val_f1s)+1),\n                         y=val_f1s, mode=\"lines+markers\", name=\"val\",\n                         marker=dict(color=\"indianred\", line=dict(width=.5,\n                                                                  color='rgb(0, 0, 0)'))))\n\nfig.add_trace(go.Scatter(x=np.arange(1, len(train_f1s)+1),\n                         y=train_f1s, mode=\"lines+markers\", name=\"train\",\n                         marker=dict(color=\"darkorange\", line=dict(width=.5,\n                                                                   color='rgb(0, 0, 0)'))))\n\nfig.update_layout(xaxis_title=\"Epochs\", yaxis_title=\"F1 Score\",\n                  title_text=\"F1 Score vs. Epochs\", template=\"plotly_white\", paper_bgcolor=\"#f0f0f0\")\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load model and check sample performance\n\n* We first load the model to check its performance.\n* We will now see how the model performs on sample questions.\n* It appears to classify insincerity pretty accurately in these simple examples.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"network = Roberta()\nnetwork.load_state_dict(torch.load('insincerity_model.pt'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"device = xm.xla_device()\nnetwork = network.to(device); network = network.eval()\n\ndef predict_insincerity(question):\n    pg, tg = 'post', 'post'\n    ins = {0: 'sincere', 1: 'insincere'}\n    quest_ids = tokenizer.encode(question.strip())\n\n    attention_mask_idx = len(quest_ids) - 1\n    if 0 not in quest_ids: quest_ids = 0 + quest_ids\n    quest_ids = pad([quest_ids], maxlen=MAXLEN, value=1, padding=pg, truncating=tg)\n\n    att_mask = np.zeros(MAXLEN)\n    att_mask[1:attention_mask_idx] = 1\n    att_mask = att_mask.reshape((1, -1))\n    if 2 not in quest_ids: quest_ids[-1], attention_mask[-1] = 2, 0\n    quest_ids, att_mask = torch.LongTensor(quest_ids), torch.LongTensor(att_mask)\n    \n    output = network.forward(quest_ids.to(device), att_mask.to(device))\n    return ins[int(np.round(nn.Sigmoid()(output.detach().cpu()).item()))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_insincerity(\"How can I train roBERTa base on TPUs?\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_insincerity(\"Why is that stupid man the biggest dictator in the world?\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Run inference on the test data\n\n* Next I will run inference on the test data and store the test predictions in a list.\n* These predictions are logits and will be converted to probabilities later using <code>sigmoid</code>.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def sigmoid(x):\n    return 1/(1 + np.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"network.eval()\ntest_preds = []\n\ntest_dataset = QuoraDataset(test_df, tokenizer)\ntest_loader = tqdm(DataLoader(test_dataset, batch_size=VAL_BATCH_SIZE))\n\nwith torch.no_grad():\n    for batch in test_loader:\n        _, test_in, test_att = batch\n        test_in = test_in.to(device)\n        test_att = test_att.to(device)\n        test_pred = network.forward(test_in, test_att)\n        test_preds.extend(test_pred.squeeze().detach().cpu().numpy())\n\ntest_preds = np.int32(np.round(sigmoid(np.array(test_preds))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/quora-insincere-questions-classification/'\nsample_submission = pd.read_csv(path + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.prediction = test_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Takeaways\n\n* Training is very fast on TPUs and should be used when possible.\n* Pretrained models like roBERTa-base can generalize effectively to new language tasks.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}