{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Jigsaw Multilingual Toxic Comment Classification - Bert \n#### Members: 資科四 劉上銓 105703030 資科四 邱顯安 105703012 資科四 林瀚軒 105703004","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# TPU\n1. 安裝新版的 pip，去除 warning\n2. 安裝為了使用 TPU 的相依套件","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!python -m pip install --upgrade pip\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nimport pandas as pd\nimport numpy as np\nfrom tqdm.notebook import tqdm\nfrom sklearn.metrics import roc_auc_score\nimport IPython\nimport sys\n\nimport torch\nfrom torch.utils.data import DataLoader\nfrom torch.utils.data import Dataset\nimport transformers\nfrom transformers import BertForSequenceClassification, BertPreTrainedModel, BertConfig, BertModel\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nimport torch_xla\nimport torch_xla.core.xla_model as xm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bert 基礎參數設定","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class config:\n    EPOCHS = 1\n    BATCH_SIZE = 32\n    VAL_BATCH_SIZE = 128\n    TEST_BATCH_SIZE = 128\n    LR = 3e-5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 讀取資料\n1. 從競賽中取出經預處理的訓練資料 `jigsaw-toxic-comment-train-processed-seqlen128.csv` --> 句子的最大長度是 128\n2. 也取出處理過的驗證資料 `validation-processed-seqlen128.csv` 和測試資料 `test-processed-seqlen128.csv`\n3. 保留 `[id, comment_text, input_word_ids, input_mask, all_segment_id, toxic]`，comment_text 代表 twitter 的留言，toxic 是 1 表示惡意，0 表示安全\n4. 與前面兩個模型相同使用 20000 筆資料當作我們最後的訓練資料","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"valid = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation-processed-seqlen128.csv\")\ntrain = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train-processed-seqlen128.csv\")\ntest = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test-processed-seqlen128.csv\")\nsubmit = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv\")\ntrain = train[['id', 'comment_text', 'input_word_ids', 'input_mask','all_segment_id', 'toxic']].iloc[:20000]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset\n1. 因為放進 Bert 的資料需要三樣東西，分別是 (token_tensor, segment_tensor, mask_tensor)\n    - token tensor 就是每一個句子轉換為 id 之後的句子\n    - segment tensor 是因為 Bert 的應用中某一些情況是兩個句子的輸入，必須告訴模型哪一些是第一句，哪一些是第二句\n    - mask tensor 主要跟 Bert 兩個主要的任務有關，Bert 會把單字遮罩起來當作克漏字來預測 (另一個任務是 Next Sentence Prediction)\n2. 將那三樣東西從 train 拿出來，還有正確答案 label_tensor","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class TweetDataset(Dataset):\n    def __init__(self, mode, df):\n        self.mode = mode\n        self.df = df\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        token, segment, mask = self.df.loc[idx, [\"input_word_ids\", \"all_segment_id\", \"input_mask\"]].values\n        if self.mode==\"train\" or self.mode == \"valid\":\n            label_tensor = torch.tensor(self.df.loc[idx, \"toxic\"])\n        else:\n            label_tensor = torch.tensor(-1)\n        tokens_tensor = torch.tensor([int(i) for i in token[1:-1].split(\",\")])\n        segments_tensor = torch.tensor([int(i) for i in segment[1:-1].split(\",\")])\n        masks_tensor = torch.tensor([int(i) for i in mask[1:-1].split(\",\")])\n           \n        return tokens_tensor, segments_tensor, masks_tensor, label_tensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataloader\n1. 將從 csv 讀出來的資料經過剛剛的 Dataset 進行處理\n2. 將不同的語言從 valid 分離出來，分別存放，為了觀察各語言的情況\n3. 之後切割出對應的 batch size ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lang = {'Spanish': 'es', 'Italian': 'it', 'Turkish': 'tr'}\n\nvalidsets = {}\nfor i, k in lang.items():\n    validsets[i] = TweetDataset(\"valid\", valid[valid[\"lang\"] == k].reset_index(drop=True))\ntrainset = TweetDataset(\"train\", train)\nvalidset = TweetDataset(\"valid\", valid)\ntestset = TweetDataset(\"test\", test)\n\nvalidloaders = {}\nfor i, k in validsets.items():\n    validloaders[i] = DataLoader(k, batch_size=config.VAL_BATCH_SIZE, num_workers=4, shuffle=False)\ntrainloader = DataLoader(trainset, batch_size=config.BATCH_SIZE, num_workers=4, shuffle=False)\nvalidloader = DataLoader(validset, batch_size=config.VAL_BATCH_SIZE, num_workers=4, shuffle=False)\ntestloader = DataLoader(testset, batch_size=config.TEST_BATCH_SIZE, num_workers=4, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bert 架構\n1. 讀取 bert-base-multilingual-cased 的預訓練模型\n2. 取出最後一層的 hidden states 而不是 CLS (自首分類的特殊字元) 的資訊 --> 經過嘗試使用 CLS 的成績比較低落\n3. 將取出來的 hidden states 做一次 average pooling，接著做 max pooling，將兩個的結果串接起來，因此維度變為原來的兩倍 ($2\\times768$)\n4. 接著 dropout(0.3) --> linear(tanh) --> output\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Model(nn.Module):\n    \n    def __init__(self, labels=1):\n        \n        super().__init__()\n        \n        self.bert = BertModel.from_pretrained(\"bert-base-multilingual-cased\")\n        self.num_features = self.bert.pooler.dense.out_features\n        self.labels = labels\n        \n        self.drop = nn.Dropout(0.3)\n        self.fc1 = nn.Linear(self.num_features * 2, self.num_features)\n        self.logit = nn.Linear(self.num_features, self.labels)\n        \n    def forward(self, tokens_tensors, segments_tensors, masks_tensors):\n        \n        hidden_states, cls = self.bert(input_ids=tokens_tensors, token_type_ids=segments_tensors, attention_mask=masks_tensors)\n        avgpool = torch.mean(hidden_states, 1)\n        maxpool, _ = torch.max(hidden_states, 1)\n        cat = torch.cat((avgpool, maxpool), 1)\n        x = self.drop(cat)\n        x = torch.tanh(self.fc1(x))\n        output = self.logit(x)\n\n        return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Model()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"確認模型的輸出跟所設計的架構是否相同","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Device\n- 將整個模型放入 TPU 當中","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = xm.xla_device()\nmodel.to(device)\nprint(f\"Now we use {device}\\n\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train - function\n1. optimizer: Adam, scheduler: lr 隨著步驟下降, loss function: BCEWithLogitsLoss\n2. 將每一個 input 分別放進 tensor 以利計算\n3. loss --> backward --> optimize --> schedule\n4. 計算一個 epoch 結束後 valid 的成績","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def training(model, warmup_prop=0.1):\n\n    optimizer = torch.optim.AdamW(model.parameters(), lr=config.LR)\n    num_warmup_steps = int(warmup_prop * config.EPOCHS * len(trainloader))\n    num_training_steps = config.EPOCHS * len(trainloader)\n    scheduler = transformers.get_linear_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps)\n    loss_fun = torch.nn.BCEWithLogitsLoss(reduction='mean').to(device)    \n\n    for epoch in range(config.EPOCHS):\n        model.train()\n        \n        optimizer.zero_grad()\n        avg_loss = 0\n        \n        for data in tqdm(trainloader):             \n            tokens_tensor, segments_tensor, masks_tensor, labels_tensor = [k.to(device) for k in data if k is not None]\n            output = model(tokens_tensor, segments_tensor, masks_tensor)\n            loss = loss_fun(output.view(-1).float(), labels_tensor.float().to(device))\n            loss.backward()\n            avg_loss += loss.item() / len(trainloader)\n\n            xm.optimizer_step(optimizer, barrier=True)\n            scheduler.step()\n            model.zero_grad()\n            optimizer.zero_grad()\n                \n        model.eval()\n        preds = []\n        truths = []\n        avg_val_loss = 0.\n\n        with torch.no_grad():\n            for data in validloader:\n                tokens_tensor, segments_tensor, masks_tensor, labels_tensor = [k.to(device) for k in data if k is not None]\n                output = model(tokens_tensor, segments_tensor, masks_tensor)\n                loss = loss_fun(output.detach().view(-1).float(), labels_tensor.float().to(device))\n                avg_val_loss += loss.item() / len(validloader)\n                \n                probs = torch.sigmoid(output).detach().cpu().numpy()\n                preds += list(probs.flatten())\n                truths += list(labels_tensor.detach().cpu().numpy().flatten())\n            score = roc_auc_score(truths, preds)\n        \n        lr = scheduler.get_last_lr()[0]\n        print(f'[Epoch {epoch + 1}] lr={lr:.1e} loss={avg_loss:.4f} val_loss={avg_val_loss:.4f} val_auc={score:.4f}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict\n1. 將超過 0.5 定為有惡意的評論\n2. 輸入模型、切割好 batch 的 dataloader、跟原來的 dataframe 算出預測的答案","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"threshold = lambda x: 1 if x>=0.5 else 0\n\ndef predict(model, dataloader, df, isAccuracy=True):\n \n    model.eval().to(device)\n    preds = np.empty((0, 1))\n    accuracy = None\n\n    with torch.no_grad():\n        for data in tqdm(dataloader):\n            tokens_tensor, segments_tensor, masks_tensor, labels_tensor = [k.to(device) for k in data if k is not None]\n            probs = torch.sigmoid(model(tokens_tensor, segments_tensor, masks_tensor)).detach().cpu().numpy()\n            preds = np.concatenate([preds, probs])\n            \n    preds = preds.reshape(len(preds))        \n    predicts = np.array([threshold(i) for i in preds])\n    if isAccuracy:\n        accuracy = (df[\"toxic\"].values == predicts).sum() / len(df)\n\n    return preds, predicts, accuracy ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bert 在訓練之前的成績\n- 分別是查看 train、各個語言的 valid 在訓練之前的成果\n- 從這裡我們可以看出，不管是 train 還是 valid 的各種語言在沒有訓練之前的 auc 跟 accuracy 的表現都是非常不好的","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# before training model accuracy\npre, pre_class, accuracy = predict(model, trainloader, train)\nauc = roc_auc_score(train[\"toxic\"].values, pre_class)\nprint(\"Train: \")\nprint(f\"Model before fine-tune accuracy: {accuracy * 100:.3f}%\\nModel before fine-tune AUC: {auc:.3f}\")\n\nfor key, value in validloaders.items():\n    pre, pre_class, accuracy = predict(model, value, valid[valid[\"lang\"] == lang[key]].reset_index(drop=True))\n    auc = roc_auc_score(valid[valid[\"lang\"] == lang[key]].reset_index(drop=True)[\"toxic\"].values, pre_class)\n    print(f\"{key} Valid: \")\n    print(f\"Model before fine-tune accuracy: {accuracy * 100:.2f}%\\nModel before fine-tune AUC: {auc:.3f}\")\n\npre, pre_class, accuracy = predict(model, validloader, valid)\nauc = roc_auc_score(valid[\"toxic\"].values, pre_class)\nprint(f\"Combined Valid: \")\nprint(f\"Model before fine-tune accuracy: {accuracy * 100:.2f}%\\nModel before fine-tune AUC: {auc:.3f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train\n1. 訓練階段只訓練了一個 epoch，因為之前嘗試使用更多 epoch 時，loss 不斷的上升\n2. 在 5 分鐘以內能完成了 20000 筆資料的訓練 (曾嘗試使用較大量的資料但 performance 沒有太大的進步)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n\ntraining(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bert 在訓練之後的成績\n1. 經過參數的微調讓 Bert 更符合現在這個任務\n2. 結果可以看出進步非常地顯著，AUC 跟 accuracy 在所有的資料集上幾乎都有達到 0.8 以上\n3. 透過 fine-tune 能讓 transfer learning 產生最大的效果\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# After training model accuracy\npre, pre_class, accuracy = predict(model, trainloader, train)\nauc = roc_auc_score(train[\"toxic\"].values, pre)\nprint(\"Train: \")\nprint(f\"Model before fine-tune accuracy: {accuracy * 100:.3f}%\\nModel before fine-tune AUC: {auc:.3f}\")\n\nfor key, value in validloaders.items():\n    pre, pre_class, accuracy = predict(model, value, valid[valid[\"lang\"] == lang[key]].reset_index(drop=True))\n    auc = roc_auc_score(valid[valid[\"lang\"] == lang[key]].reset_index(drop=True)[\"toxic\"].values, pre)\n    print(f\"{key} Valid: \")\n    print(f\"Model before fine-tune accuracy: {accuracy * 100:.2f}%\\nModel before fine-tune AUC: {auc:.3f}\")\n\npre, pre_class, accuracy = predict(model, validloader, valid)\nauc = roc_auc_score(valid[\"toxic\"].values, pre)\nprint(f\"Combined Valid: \")\nprint(f\"Model before fine-tune accuracy: {accuracy * 100:.2f}%\\nModel before fine-tune AUC: {auc:.3f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"將 model 的權重儲存起來","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), \"./model.bin\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 預測測試資料並輸出","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pre, pre_class, accuracy = predict(model, testloader, test, False)\nsubmit['toxic'] = pre\nsubmit.to_csv('submission.csv', index=False)\nsubmit.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 視覺化 Attention\n- 從這個視覺話工具中我們可以看出，在某一些 Bert 的某些層是能知道『他 - 阿明』之間的關係，或是『給 - 阿明』這個動作的關係\n- 不同的 head 能監控不一樣的特徵","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!test -d bertviz_repo || git clone https://github.com/jessevig/bertviz bertviz_repo\nif not 'bertviz_repo' in sys.path:\n    sys.path += ['bertviz_repo']\n\nfrom transformers import BertTokenizer, BertModel\nfrom bertviz import head_view\n\ndef call_html():\n    display(IPython.core.display.HTML('''\n        <script src=\"/static/components/requirejs/require.js\"></script>\n        <script>\n          requirejs.config({\n            paths: {\n              base: '/static/base',\n              \"d3\": \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.8/d3.min\",\n              jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',\n            },\n          });\n        </script>\n        '''))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_version = 'bert-base-chinese'\nmodel = BertModel.from_pretrained(model_version, output_attentions=True)\ntokenizer = BertTokenizer.from_pretrained(model_version)\n\nsentence_a = \"阿明去買東西，\"\nsentence_b = \"回來的時候要給他錢。\"\n\ninputs = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', add_special_tokens=True)\ntoken_type_ids = inputs['token_type_ids']\ninput_ids = inputs['input_ids']\nattention = model(input_ids, token_type_ids=token_type_ids)[-1]\ninput_id_list = input_ids[0].tolist() # Batch index 0\ntokens = tokenizer.convert_ids_to_tokens(input_id_list)\ncall_html()\n\nhead_view(attention, tokens)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}