{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## What is BERT?\n\nBERT (introduced in [this paper](https://arxiv.org/abs/1810.04805)) stands for Bidirectional Encoder Representations from Transformers. If you don't know what most of that means - you've come to the right place! Let's unpack the main ideas:\n\n- Bidirectional - to understand the text  you're looking you'll have to look back (at the previous words) and forward (at the next words)\n- Transformers - The [Attention Is All You Need](https://arxiv.org/abs/1706.03762) paper presented the Transformer model. The Transformer reads entire sequences of tokens at once. In a sense, the model is non-directional, while LSTMs read sequentially (left-to-right or right-to-left). The attention mechanism allows for learning contextual relations between words (e.g. `his` in a sentence refers to Jim).\n- (Pre-trained) contextualized word embeddings - [The ELMO paper](https://arxiv.org/abs/1802.05365v2) introduced a way to encode words based on their meaning/context. Nails has multiple meanings - fingernails and metal nails.\n\nBERT was trained by masking 15% of the tokens with the goal to guess them. An additional objective was to predict the next sentence. Let's look at examples of these tasks:\n\n### Masked Language Modeling (Masked LM)\n\nThe objective of this task is to guess the masked tokens. Let's look at an example, and try to not make it harder than it has to be:\n\nThat's `[mask]` she `[mask]` -> That's what she said\n\n### Next Sentence Prediction (NSP)\n\nGiven a pair of two sentences, the task is to say whether or not the second follows the first (binary classification). Let's continue with the example:\n\n*Input* = `[CLS]` That's `[mask]` she `[mask]`. [SEP] Hahaha, nice! [SEP]\n\n*Label* = *IsNext*\n\n*Input* = `[CLS]` That's `[mask]` she `[mask]`. [SEP] Dwight, you ignorant `[mask]`! [SEP]\n\n*Label* = *NotNext*\n\nThe training corpus was comprised of two entries: [Toronto Book Corpus](https://arxiv.org/abs/1506.06724) (800M words) and English Wikipedia (2,500M words). While the original Transformer has an encoder (for reading the input) and a decoder (that makes the prediction), BERT uses only the decoder.\n\nBERT is simply a pre-trained stack of Transformer Encoders. How many Encoders? We have two versions - with 12 (BERT base) and 24 (BERT Large).\n\n### Is This Thing Useful in Practice?\n\nThe BERT paper was released along with [the source code](https://github.com/google-research/bert) and pre-trained models.\n\nThe best part is that you can do Transfer Learning (thanks to the ideas from OpenAI Transformer) with BERT for many NLP tasks - Classification, Question Answering, Entity Recognition, etc. You can train with small amounts of data and achieve great performance!","metadata":{"id":"Tbodro8Fpmwr"}},{"cell_type":"markdown","source":"## Setup\n\nWe'll need [the Transformers library](https://huggingface.co/transformers/) by Hugging Face:","metadata":{"id":"wmj22-TcZMef"}},{"cell_type":"code","source":"!pip install -q -U watermark","metadata":{"id":"Kj_7Tz0-pK69","execution":{"iopub.status.busy":"2022-08-25T02:03:54.141614Z","iopub.execute_input":"2022-08-25T02:03:54.142568Z","iopub.status.idle":"2022-08-25T02:04:08.120995Z","shell.execute_reply.started":"2022-08-25T02:03:54.142467Z","shell.execute_reply":"2022-08-25T02:04:08.119729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -qq transformers","metadata":{"id":"Jjsbi1u3QFEM","execution":{"iopub.status.busy":"2022-08-25T02:05:07.511493Z","iopub.execute_input":"2022-08-25T02:05:07.512012Z","iopub.status.idle":"2022-08-25T02:05:18.633139Z","shell.execute_reply.started":"2022-08-25T02:05:07.511971Z","shell.execute_reply":"2022-08-25T02:05:18.631289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%reload_ext watermark\n%watermark -v -p numpy,pandas,torch,transformers","metadata":{"id":"AJqoaFpVpoM8","outputId":"fce4532a-f6f4-45fb-dc15-2893ea9bf32b","execution":{"iopub.status.busy":"2022-08-25T02:05:18.636198Z","iopub.execute_input":"2022-08-25T02:05:18.636640Z","iopub.status.idle":"2022-08-25T02:05:20.780063Z","shell.execute_reply.started":"2022-08-25T02:05:18.636601Z","shell.execute_reply":"2022-08-25T02:05:20.779084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#@title Setup & Config\nimport transformers\nfrom transformers import BertModel, BertTokenizer, AdamW, get_linear_schedule_with_warmup\nimport torch\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom pylab import rcParams\nimport matplotlib.pyplot as plt\nfrom matplotlib import rc\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, classification_report\nfrom collections import defaultdict\nfrom textwrap import wrap\n\nfrom torch import nn, optim\nfrom torch.utils.data import Dataset, DataLoader\nimport torch.nn.functional as F\n\nfrom tqdm import tqdm, tqdm_notebook # log writer\n\n%matplotlib inline\n%config InlineBackend.figure_format='retina'\n\nsns.set(style='whitegrid', palette='muted', font_scale=1.2)\n\nHAPPY_COLORS_PALETTE = [\"#01BEFE\", \"#FFDD00\", \"#FF7D00\", \"#FF006D\", \"#ADFF02\", \"#8F00FF\"]\n\nsns.set_palette(sns.color_palette(HAPPY_COLORS_PALETTE))\n\nrcParams['figure.figsize'] = 12, 8\n\nRANDOM_SEED = 42\nnp.random.seed(RANDOM_SEED)\ntorch.manual_seed(RANDOM_SEED)\n\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ndevice","metadata":{"id":"w68CZpOwFoly","outputId":"61814b6e-796f-416d-f2da-9ef92faeba00","execution":{"iopub.status.busy":"2022-08-25T02:05:20.781200Z","iopub.execute_input":"2022-08-25T02:05:20.782073Z","iopub.status.idle":"2022-08-25T02:05:28.564831Z","shell.execute_reply.started":"2022-08-25T02:05:20.782035Z","shell.execute_reply":"2022-08-25T02:05:28.563457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Exploration","metadata":{}},{"cell_type":"code","source":"train_path = \"/kaggle/input/quora-insincere-questions-classification/train.csv\"\ntest_path = \"/kaggle/input/quora-insincere-questions-classification/test.csv\"\ndf_train = pd.read_csv(\"/kaggle/input/quora-insincere-questions-classification/train.csv\")\ndf_test = pd.read_csv(\"/kaggle/input/quora-insincere-questions-classification/test.csv\")\nsample = pd.read_csv(\"/kaggle/input/quora-insincere-questions-classification/sample_submission.csv\")","metadata":{"id":"KSPAUnkDKHxI","execution":{"iopub.status.busy":"2022-08-25T02:06:05.222093Z","iopub.execute_input":"2022-08-25T02:06:05.222589Z","iopub.status.idle":"2022-08-25T02:06:12.277500Z","shell.execute_reply.started":"2022-08-25T02:06:05.222545Z","shell.execute_reply":"2022-08-25T02:06:12.276336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df_train\ndf.head()","metadata":{"id":"mUKLyKc7I6Qp","outputId":"c394025c-98ba-45db-e728-48def4d6b6ce","execution":{"iopub.status.busy":"2022-08-25T02:06:12.280129Z","iopub.execute_input":"2022-08-25T02:06:12.280642Z","iopub.status.idle":"2022-08-25T02:06:12.304239Z","shell.execute_reply.started":"2022-08-25T02:06:12.280592Z","shell.execute_reply":"2022-08-25T02:06:12.302955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"id":"dB2jE6am7Dpo","outputId":"04501edd-a827-450a-f330-4e42828e697e","execution":{"iopub.status.busy":"2022-08-25T02:06:12.308857Z","iopub.execute_input":"2022-08-25T02:06:12.309240Z","iopub.status.idle":"2022-08-25T02:06:12.318043Z","shell.execute_reply.started":"2022-08-25T02:06:12.309208Z","shell.execute_reply":"2022-08-25T02:06:12.316673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have about 16k examples. Let's check for missing values:","metadata":{"id":"TWqVNHJbn10l"}},{"cell_type":"code","source":"df.info()","metadata":{"id":"VA_wGSLQLKCh","outputId":"c15a597b-6d53-4741-9412-319e9e3564be","execution":{"iopub.status.busy":"2022-08-25T02:06:14.532775Z","iopub.execute_input":"2022-08-25T02:06:14.533308Z","iopub.status.idle":"2022-08-25T02:06:14.679530Z","shell.execute_reply.started":"2022-08-25T02:06:14.533262Z","shell.execute_reply":"2022-08-25T02:06:14.678106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Preprocessing\n\nYou might already know that Machine Learning models don't work with raw text. You need to convert text to numbers (of some sort). BERT requires even more attention (good one, right?). Here are the requirements: \n\n- Add special tokens to separate sentences and do classification\n- Pass sequences of constant length (introduce padding)\n- Create array of 0s (pad token) and 1s (real token) called *attention mask*\n\nThe Transformers library provides (you've guessed it) a wide variety of Transformer models (including BERT). It works with TensorFlow and PyTorch! It also includes prebuild tokenizers that do the heavy lifting for us!\n","metadata":{"id":"9aHyGuTFgyPO"}},{"cell_type":"code","source":"PRE_TRAINED_MODEL_NAME = 'bert-base-cased'","metadata":{"id":"E7Mj-0ne--5t","execution":{"iopub.status.busy":"2022-08-25T02:06:17.882922Z","iopub.execute_input":"2022-08-25T02:06:17.883383Z","iopub.status.idle":"2022-08-25T02:06:17.888847Z","shell.execute_reply.started":"2022-08-25T02:06:17.883321Z","shell.execute_reply":"2022-08-25T02:06:17.887595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> You can use a cased and uncased version of BERT and tokenizer. I've experimented with both. The cased version works better. Intuitively, that makes sense, since \"BAD\" might convey more sentiment than \"bad\".","metadata":{"id":"fMSr7C-F_sey"}},{"cell_type":"markdown","source":"Let's load a pre-trained [BertTokenizer](https://huggingface.co/transformers/model_doc/bert.html#berttokenizer):","metadata":{"id":"NiLb-ltM-ZRz"}},{"cell_type":"code","source":"tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME)","metadata":{"id":"H3AfJSZ8NNLF","execution":{"iopub.status.busy":"2022-08-25T02:06:19.741159Z","iopub.execute_input":"2022-08-25T02:06:19.741663Z","iopub.status.idle":"2022-08-25T02:06:20.803233Z","shell.execute_reply.started":"2022-08-25T02:06:19.741623Z","shell.execute_reply":"2022-08-25T02:06:20.802029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll use this text to understand the tokenization process:","metadata":{"id":"CfrSbwTQ-wi_"}},{"cell_type":"code","source":"# sample_txt = 'When was I last outside? I am stuck at home for 2 weeks.'","metadata":{"id":"HZMitwrqm2eb","execution":{"iopub.status.busy":"2022-08-04T00:00:42.205037Z","iopub.status.idle":"2022-08-04T00:00:42.205500Z","shell.execute_reply.started":"2022-08-04T00:00:42.205295Z","shell.execute_reply":"2022-08-04T00:00:42.205316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some basic operations can convert the text to tokens and tokens to unique integers (ids):","metadata":{"id":"yO2qBTVl_KPs"}},{"cell_type":"code","source":"# tokens = tokenizer.tokenize(sample_txt)\n# token_ids = tokenizer.convert_tokens_to_ids(tokens)\n# # \n# print(f' Sentence: {sample_txt}')\n# print(f'   Tokens: {tokens}')\n# print(f'Token IDs: {token_ids}')","metadata":{"id":"iTFhpHpsoWO7","execution":{"iopub.status.busy":"2022-08-04T00:00:42.207451Z","iopub.status.idle":"2022-08-04T00:00:42.207947Z","shell.execute_reply.started":"2022-08-04T00:00:42.207723Z","shell.execute_reply":"2022-08-04T00:00:42.207745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Special Tokens\n\n`[SEP]` - marker for ending of a sentence\n","metadata":{"id":"bzbbKLR8lZbu"}},{"cell_type":"code","source":"# tokenizer.sep_token, tokenizer.sep_token_id","metadata":{"id":"EXwz47bQvCbc","execution":{"iopub.status.busy":"2022-08-04T00:00:42.209361Z","iopub.status.idle":"2022-08-04T00:00:42.210546Z","shell.execute_reply.started":"2022-08-04T00:00:42.210318Z","shell.execute_reply":"2022-08-04T00:00:42.210341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`[CLS]` - we must add this token to the start of each sentence, so BERT knows we're doing classification","metadata":{"id":"Mip_eGeXwLFF"}},{"cell_type":"code","source":"# tokenizer.cls_token, tokenizer.cls_token_id","metadata":{"id":"_6K4it5HwE6l","execution":{"iopub.status.busy":"2022-08-04T00:00:42.212298Z","iopub.status.idle":"2022-08-04T00:00:42.212728Z","shell.execute_reply.started":"2022-08-04T00:00:42.212519Z","shell.execute_reply":"2022-08-04T00:00:42.212539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is also a special token for padding:","metadata":{"id":"Qi6O-yEY09gl"}},{"cell_type":"code","source":"# tokenizer.pad_token, tokenizer.pad_token_id","metadata":{"id":"Vx7gD5xf1AFK","execution":{"iopub.status.busy":"2022-08-04T00:00:42.213933Z","iopub.status.idle":"2022-08-04T00:00:42.214377Z","shell.execute_reply.started":"2022-08-04T00:00:42.214144Z","shell.execute_reply":"2022-08-04T00:00:42.214192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"BERT understands tokens that were in the training set. Everything else can be encoded using the `[UNK]` (unknown) token:","metadata":{"id":"6GWCfijM0TWB"}},{"cell_type":"code","source":"# tokenizer.unk_token, tokenizer.unk_token_id","metadata":{"id":"4cmfFsbEKQDT","execution":{"iopub.status.busy":"2022-08-04T00:00:42.215674Z","iopub.status.idle":"2022-08-04T00:00:42.216092Z","shell.execute_reply.started":"2022-08-04T00:00:42.215896Z","shell.execute_reply":"2022-08-04T00:00:42.215916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All of that work can be done using the [`encode_plus()`](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode_plus) method:","metadata":{"id":"W9ap7jdL0LYU"}},{"cell_type":"code","source":"# encoding = tokenizer.encode_plus(\n#   sample_txt,\n#   max_length=32,\n#   add_special_tokens=True, # Add '[CLS]' and '[SEP]'\n#   return_token_type_ids=False,\n#   pad_to_max_length=True,\n#   return_attention_mask=True,\n#   return_tensors='pt',  # Return PyTorch tensors\n# )\n\n# encoding.keys()","metadata":{"id":"Vea9edaaxSPO","execution":{"iopub.status.busy":"2022-08-04T00:00:42.218029Z","iopub.status.idle":"2022-08-04T00:00:42.219088Z","shell.execute_reply.started":"2022-08-04T00:00:42.218853Z","shell.execute_reply":"2022-08-04T00:00:42.218878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The token ids are now stored in a Tensor and padded to a length of 32:","metadata":{"id":"sS69c8WvdOED"}},{"cell_type":"code","source":"# print(len(encoding['input_ids'][0]))\n# encoding['input_ids'][0]","metadata":{"id":"YzBmcOla0yQR","execution":{"iopub.status.busy":"2022-08-04T00:00:42.221499Z","iopub.status.idle":"2022-08-04T00:00:42.221939Z","shell.execute_reply.started":"2022-08-04T00:00:42.221733Z","shell.execute_reply":"2022-08-04T00:00:42.221755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The attention mask has the same length:","metadata":{"id":"itAyVPsNdyc1"}},{"cell_type":"code","source":"# print(len(encoding['attention_mask'][0]))\n# encoding['attention_mask']","metadata":{"id":"Wiv5LLiw03Ox","execution":{"iopub.status.busy":"2022-08-04T00:00:42.224006Z","iopub.status.idle":"2022-08-04T00:00:42.224488Z","shell.execute_reply.started":"2022-08-04T00:00:42.224279Z","shell.execute_reply":"2022-08-04T00:00:42.224301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can inverse the tokenization to have a look at the special tokens:","metadata":{"id":"m1RvhC4jNHHy"}},{"cell_type":"code","source":"# tokenizer.convert_ids_to_tokens(encoding['input_ids'][0])","metadata":{"id":"IagGoafKLUwW","execution":{"iopub.status.busy":"2022-08-04T00:00:42.229458Z","iopub.status.idle":"2022-08-04T00:00:42.229947Z","shell.execute_reply.started":"2022-08-04T00:00:42.229717Z","shell.execute_reply":"2022-08-04T00:00:42.229740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Choosing Sequence Length\n\nBERT works with fixed-length sequences. We'll use a simple strategy to choose the max length. Let's store the token length of each review:","metadata":{"id":"waKjYxTDuaWt"}},{"cell_type":"code","source":"# token_lens = []\n\n# for txt in tqdm(df.question_text):\n#   tokens = tokenizer.encode(txt, max_length=512)\n#   token_lens.append(len(tokens))","metadata":{"id":"BUnE5CT9hbeZ","execution":{"iopub.status.busy":"2022-08-04T00:00:42.232024Z","iopub.status.idle":"2022-08-04T00:00:42.232510Z","shell.execute_reply.started":"2022-08-04T00:00:42.232288Z","shell.execute_reply":"2022-08-04T00:00:42.232311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"and plot the distribution:","metadata":{"id":"tI4goUrHf6da"}},{"cell_type":"code","source":"# sns.distplot(token_lens)\n# plt.xlim([0, 256]);\n# plt.xlabel('Token count');","metadata":{"id":"SzE1j4jxmUtd","execution":{"iopub.status.busy":"2022-08-04T00:00:42.233729Z","iopub.status.idle":"2022-08-04T00:00:42.234200Z","shell.execute_reply.started":"2022-08-04T00:00:42.233973Z","shell.execute_reply":"2022-08-04T00:00:42.233993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the reviews seem to contain less than 128 tokens, but we'll be on the safe side and choose a maximum length of 160.","metadata":{"id":"oW6ajl30t6du"}},{"cell_type":"code","source":"MAX_LEN = 50","metadata":{"id":"t7xSmJtLuoxW","execution":{"iopub.status.busy":"2022-08-25T02:15:13.902890Z","iopub.execute_input":"2022-08-25T02:15:13.903502Z","iopub.status.idle":"2022-08-25T02:15:13.910159Z","shell.execute_reply.started":"2022-08-25T02:15:13.903435Z","shell.execute_reply":"2022-08-25T02:15:13.908742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have all building blocks required to create a PyTorch dataset. Let's do it:","metadata":{"id":"XvvcoU6nurHy"}},{"cell_type":"code","source":"class BERTDataset(Dataset):\n\n    def __init__(self, question_text, targets, tokenizer, max_len):\n        self.question_text = question_text\n        self.targets = targets\n        self.tokenizer = tokenizer\n        self.max_len = max_len\n\n    def __len__(self):\n        return len(self.question_text)\n  \n    def __getitem__(self, item):\n        text = str(self.question_text[item])\n        target = self.targets[item]\n\n        encoding = self.tokenizer.encode_plus(\n          text,\n          add_special_tokens=True,\n          max_length=self.max_len,\n          return_token_type_ids=False,\n          padding='max_length',\n          truncation=True,\n          return_attention_mask=True,\n          return_tensors='pt',\n        )\n\n        return {\n          'question_text': text,\n          'input_ids': encoding['input_ids'].flatten(),\n          'attention_mask': encoding['attention_mask'].flatten(),\n          'targets': torch.tensor(target, dtype=torch.long)\n        }","metadata":{"id":"E2BPgRJ7YBK0","execution":{"iopub.status.busy":"2022-08-25T02:15:19.473962Z","iopub.execute_input":"2022-08-25T02:15:19.474503Z","iopub.status.idle":"2022-08-25T02:15:19.485230Z","shell.execute_reply.started":"2022-08-25T02:15:19.474462Z","shell.execute_reply":"2022-08-25T02:15:19.483805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The tokenizer is doing most of the heavy lifting for us. We also return the review texts, so it'll be easier to evaluate the predictions from our model. Let's split the data:","metadata":{"id":"x2uwsvCYqDJK"}},{"cell_type":"code","source":"df_train, df_test = train_test_split(df, test_size=0.1, random_state=RANDOM_SEED)\ndf_val, df_test = train_test_split(df_test, test_size=0.5, random_state=RANDOM_SEED)","metadata":{"id":"B-vWzoo81dvO","execution":{"iopub.status.busy":"2022-08-25T02:16:46.228596Z","iopub.execute_input":"2022-08-25T02:16:46.229127Z","iopub.status.idle":"2022-08-25T02:16:47.020143Z","shell.execute_reply.started":"2022-08-25T02:16:46.229084Z","shell.execute_reply":"2022-08-25T02:16:47.018265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.shape, df_val.shape, df_test.shape","metadata":{"id":"xz3ZOQXVPCwh","outputId":"97f5531f-abce-48d3-f29e-32b10eee9ae9","execution":{"iopub.status.busy":"2022-08-25T02:16:48.020779Z","iopub.execute_input":"2022-08-25T02:16:48.021298Z","iopub.status.idle":"2022-08-25T02:16:48.029180Z","shell.execute_reply.started":"2022-08-25T02:16:48.021257Z","shell.execute_reply":"2022-08-25T02:16:48.027947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We also need to create a couple of data loaders. Here's a helper function to do it:","metadata":{"id":"J4tQ1x-vqNab"}},{"cell_type":"code","source":"def create_data_loader(df, tokenizer, max_len, batch_size):\n    ds = BERTDataset(\n    question_text=df.question_text.to_numpy(),\n    targets=df.target.to_numpy(),\n    tokenizer=tokenizer,\n    max_len=max_len\n    )\n\n    return DataLoader(\n    ds,\n    batch_size=batch_size,\n    num_workers=4\n    )","metadata":{"id":"KEGqcvkuOuTX","execution":{"iopub.status.busy":"2022-08-25T02:16:51.995796Z","iopub.execute_input":"2022-08-25T02:16:51.996332Z","iopub.status.idle":"2022-08-25T02:16:52.004123Z","shell.execute_reply.started":"2022-08-25T02:16:51.996288Z","shell.execute_reply":"2022-08-25T02:16:52.002882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE = 256\n\ntrain_data_loader = create_data_loader(df_train, tokenizer, MAX_LEN, BATCH_SIZE)\nval_data_loader = create_data_loader(df_val, tokenizer, MAX_LEN, BATCH_SIZE)\ntest_data_loader = create_data_loader(df_test, tokenizer, MAX_LEN, BATCH_SIZE)","metadata":{"id":"vODDxMKsPHqI","outputId":"49c4502b-f5ce-49c6-e65b-dd42c19a5906","execution":{"iopub.status.busy":"2022-08-25T02:16:54.304680Z","iopub.execute_input":"2022-08-25T02:16:54.305222Z","iopub.status.idle":"2022-08-25T02:16:54.314205Z","shell.execute_reply.started":"2022-08-25T02:16:54.305174Z","shell.execute_reply":"2022-08-25T02:16:54.312830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at an example batch from our training data loader:","metadata":{"id":"A6dlOptwqlhF"}},{"cell_type":"code","source":"data = next(iter(train_data_loader))\ndata.keys()","metadata":{"id":"Y93ldSN47FeT","outputId":"1f190c4e-fc25-4f43-f53f-0bf7d120011f","execution":{"iopub.status.busy":"2022-08-25T02:16:58.158391Z","iopub.execute_input":"2022-08-25T02:16:58.158909Z","iopub.status.idle":"2022-08-25T02:16:58.865101Z","shell.execute_reply.started":"2022-08-25T02:16:58.158872Z","shell.execute_reply":"2022-08-25T02:16:58.863729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(data['input_ids'].shape)\nprint(data['attention_mask'].shape)\nprint(data['targets'].shape)","metadata":{"id":"IdU4YVqb7N8M","outputId":"5c0d6433-fec8-402b-ce77-41fc87cde51b","execution":{"iopub.status.busy":"2022-08-25T02:17:00.839156Z","iopub.execute_input":"2022-08-25T02:17:00.840072Z","iopub.status.idle":"2022-08-25T02:17:00.847442Z","shell.execute_reply.started":"2022-08-25T02:17:00.840014Z","shell.execute_reply":"2022-08-25T02:17:00.846446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Sentiment Classification with BERT and Hugging Face","metadata":{"id":"H63Y-TjyRC7S"}},{"cell_type":"markdown","source":"There are a lot of helpers that make using BERT easy with the Transformers library. Depending on the task you might want to use [BertForSequenceClassification](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification), [BertForQuestionAnswering](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering) or something else. \n\nBut who cares, right? We're *hardcore*! We'll use the basic [BertModel](https://huggingface.co/transformers/model_doc/bert.html#bertmodel) and build our sentiment classifier on top of it. Let's load the model:","metadata":{"id":"440Nd31VTHER"}},{"cell_type":"code","source":"bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)","metadata":{"id":"0P41FayISNRI","outputId":"ba674ffd-8786-4fc1-ced2-4922e3efeb2a","execution":{"iopub.status.busy":"2022-08-25T02:17:11.364725Z","iopub.execute_input":"2022-08-25T02:17:11.365624Z","iopub.status.idle":"2022-08-25T02:17:43.092799Z","shell.execute_reply.started":"2022-08-25T02:17:11.365571Z","shell.execute_reply":"2022-08-25T02:17:43.091452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And try to use it on the encoding of our sample text:","metadata":{"id":"aFE7YSbFdY4t"}},{"cell_type":"code","source":"# out = bert_model(\n#   input_ids=encoding['input_ids'], \n#   attention_mask=encoding['attention_mask']\n# )","metadata":{"id":"s1aoFxbQSn15","execution":{"iopub.status.busy":"2022-08-04T00:00:42.254700Z","iopub.status.idle":"2022-08-04T00:00:42.255399Z","shell.execute_reply.started":"2022-08-04T00:00:42.255167Z","shell.execute_reply":"2022-08-04T00:00:42.255197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `last_hidden_state` is a sequence of hidden states of the last layer of the model. Obtaining the `pooled_output` is done by applying the [BertPooler](https://github.com/huggingface/transformers/blob/edf0582c0be87b60f94f41c659ea779876efc7be/src/transformers/modeling_bert.py#L426) on `last_hidden_state`:","metadata":{"id":"mLLu8zmqbaHV"}},{"cell_type":"markdown","source":"We have the hidden state for each of our 32 tokens (the length of our example sequence). But why 768? This is the number of hidden units in the feedforward-networks. We can verify that by checking the config:","metadata":{"id":"Q4dAot4zbz8k"}},{"cell_type":"code","source":"bert_model.config.hidden_size","metadata":{"id":"nsxB7Qy7b5YN","outputId":"9833969b-f5a5-4933-fe5e-c88ca81d977b","execution":{"iopub.status.busy":"2022-08-25T02:18:00.560179Z","iopub.execute_input":"2022-08-25T02:18:00.560849Z","iopub.status.idle":"2022-08-25T02:18:00.570241Z","shell.execute_reply.started":"2022-08-25T02:18:00.560787Z","shell.execute_reply":"2022-08-25T02:18:00.568843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nYou can think of the `pooled_output` as a summary of the content, according to BERT. Albeit, you might try and do better. Let's look at the shape of the output:","metadata":{"id":"wTKi8-rTd_j4"}},{"cell_type":"markdown","source":"We can use all of this knowledge to create a classifier that uses the BERT model:","metadata":{"id":"0o_NiS3WgOFf"}},{"cell_type":"code","source":"class SentimentClassifier(nn.Module):\n\n    def __init__(self, n_classes):\n        super(SentimentClassifier, self).__init__()\n        self.bert = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)\n        self.drop = nn.Dropout(p=0.3)\n        self.out = nn.Linear(self.bert.config.hidden_size, n_classes)\n\n    def forward(self, input_ids, attention_mask):\n        out = self.bert(\n          input_ids=input_ids,\n          attention_mask=attention_mask\n        )\n        output = self.drop(out['pooler_output'])\n        return self.out(output)","metadata":{"id":"m_mRflxPl32F","execution":{"iopub.status.busy":"2022-08-25T02:18:02.557360Z","iopub.execute_input":"2022-08-25T02:18:02.558670Z","iopub.status.idle":"2022-08-25T02:18:02.567553Z","shell.execute_reply.started":"2022-08-25T02:18:02.558616Z","shell.execute_reply":"2022-08-25T02:18:02.566362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our classifier delegates most of the heavy lifting to the BertModel. We use a dropout layer for some regularization and a fully-connected layer for our output. Note that we're returning the raw output of the last layer since that is required for the cross-entropy loss function in PyTorch to work.\n\nThis should work like any other PyTorch model. Let's create an instance and move it to the GPU:","metadata":{"id":"UJg8m3NQJahc"}},{"cell_type":"code","source":"model = SentimentClassifier(n_classes=2)\nmodel.bert.requires_grad_(False)\nmodel = model.to(device)","metadata":{"id":"i0yQnuSFsjDp","outputId":"bf0d5f52-5f4b-4650-bddb-7868b530a2dc","execution":{"iopub.status.busy":"2022-08-25T02:18:05.032036Z","iopub.execute_input":"2022-08-25T02:18:05.032440Z","iopub.status.idle":"2022-08-25T02:18:07.076784Z","shell.execute_reply.started":"2022-08-25T02:18:05.032406Z","shell.execute_reply":"2022-08-25T02:18:07.075213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll move the example batch of our training data to the GPU:","metadata":{"id":"VCPCFDLlKIQd"}},{"cell_type":"code","source":"input_ids = data['input_ids'].to(device)\nattention_mask = data['attention_mask'].to(device)\n\nprint(input_ids.shape) # batch size x seq length\nprint(attention_mask.shape) # batch size x seq length","metadata":{"id":"mz7p__CqdaMO","outputId":"a01872fd-7f0a-4d9a-d8c1-de240f1403d7","execution":{"iopub.status.busy":"2022-08-25T02:19:21.859178Z","iopub.execute_input":"2022-08-25T02:19:21.859659Z","iopub.status.idle":"2022-08-25T02:19:21.866947Z","shell.execute_reply.started":"2022-08-25T02:19:21.859620Z","shell.execute_reply":"2022-08-25T02:19:21.865659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To get the predicted probabilities from our trained model, we'll apply the softmax function to the outputs:","metadata":{"id":"Hr1EgkEtKOIB"}},{"cell_type":"markdown","source":"### Training","metadata":{"id":"g9xikRdtRN1N"}},{"cell_type":"markdown","source":"To reproduce the training procedure from the BERT paper, we'll use the [AdamW](https://huggingface.co/transformers/main_classes/optimizer_schedules.html#adamw) optimizer provided by Hugging Face. It corrects weight decay, so it's similar to the original paper. We'll also use a linear scheduler with no warmup steps:","metadata":{"id":"76g7FV85H-T8"}},{"cell_type":"code","source":"EPOCHS = 10\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)\ntotal_steps = len(train_data_loader) * EPOCHS\n\nscheduler = get_linear_schedule_with_warmup(\n  optimizer,\n  num_warmup_steps=0,\n  num_training_steps=total_steps\n)\n\nloss_fn = nn.CrossEntropyLoss().to(device)","metadata":{"id":"5v-ArJ2fCCcU","execution":{"iopub.status.busy":"2022-08-25T02:19:23.457820Z","iopub.execute_input":"2022-08-25T02:19:23.458316Z","iopub.status.idle":"2022-08-25T02:19:23.469197Z","shell.execute_reply.started":"2022-08-25T02:19:23.458277Z","shell.execute_reply":"2022-08-25T02:19:23.467773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How do we come up with all hyperparameters? The BERT authors have some recommendations for fine-tuning:\n\n- Batch size: 16, 32\n- Learning rate (Adam): 5e-5, 3e-5, 2e-5\n- Number of epochs: 2, 3, 4\n\nWe're going to ignore the number of epochs recommendation but stick with the rest. Note that increasing the batch size reduces the training time significantly, but gives you lower accuracy.\n\nLet's continue with writing a helper function for training our model for one epoch:","metadata":{"id":"A8522g7JIu5J"}},{"cell_type":"code","source":"from tqdm import notebook\ndef train_epoch(\n  model, \n  data_loader, \n  loss_fn, \n  optimizer, \n  device, \n  scheduler, \n  n_examples\n):\n    model = model.train()\n\n    losses = []\n    correct_predictions = 0\n\n    progress_bar = notebook.tqdm(data_loader, leave=False)\n\n#     for d in tqdm(data_loader, position=0, leave=True):\n    for d in progress_bar:\n        input_ids = d[\"input_ids\"].to(device)\n        attention_mask = d[\"attention_mask\"].to(device)\n        targets = d[\"targets\"].to(device)\n\n\n    outputs = model(\n      input_ids=input_ids,\n      attention_mask=attention_mask\n    )\n\n    _, preds = torch.max(outputs, dim=1)\n    loss = loss_fn(outputs, targets)\n\n#     progress_bar.set_description(f'Loss: {loss.item():.3f}')\n\n    correct_predictions += torch.sum(preds == targets)\n    losses.append(loss.item())\n\n    loss.backward()\n    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n    optimizer.step()\n    scheduler.step()\n    optimizer.zero_grad()\n\n    return correct_predictions.double() / n_examples, np.mean(losses)","metadata":{"id":"bzl9UhuNx1_Q","execution":{"iopub.status.busy":"2022-08-25T02:19:26.421201Z","iopub.execute_input":"2022-08-25T02:19:26.422453Z","iopub.status.idle":"2022-08-25T02:19:26.432422Z","shell.execute_reply.started":"2022-08-25T02:19:26.422401Z","shell.execute_reply":"2022-08-25T02:19:26.431134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Training the model should look familiar, except for two things. The scheduler gets called every time a batch is fed to the model. We're avoiding exploding gradients by clipping the gradients of the model using [clip_grad_norm_](https://pytorch.org/docs/stable/nn.html#clip-grad-norm).\n\nLet's write another one that helps us evaluate the model on a given data loader:","metadata":{"id":"E4PniYIte0fr"}},{"cell_type":"code","source":"def eval_model(model, data_loader, loss_fn, device, n_examples):\n    model = model.eval()\n\n    losses = []\n    correct_predictions = 0\n\n    with torch.no_grad():\n        for d in data_loader:\n            input_ids = d[\"input_ids\"].to(device)\n            attention_mask = d[\"attention_mask\"].to(device)\n            targets = d[\"targets\"].to(device)\n\n            outputs = model(\n            input_ids=input_ids,\n            attention_mask=attention_mask\n            )\n            _, preds = torch.max(outputs, dim=1)\n\n            loss = loss_fn(outputs, targets)\n\n            correct_predictions += torch.sum(preds == targets)\n            losses.append(loss.item())\n\n    return correct_predictions.double() / n_examples, np.mean(losses)","metadata":{"id":"CXeRorVGIKre","execution":{"iopub.status.busy":"2022-08-25T02:19:31.218527Z","iopub.execute_input":"2022-08-25T02:19:31.218981Z","iopub.status.idle":"2022-08-25T02:19:31.228555Z","shell.execute_reply.started":"2022-08-25T02:19:31.218947Z","shell.execute_reply":"2022-08-25T02:19:31.227508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using those two, we can write our training loop. We'll also store the training history:","metadata":{"id":"a_rdSDBHhhCh"}},{"cell_type":"code","source":"%%time\n\nhistory = defaultdict(list)\nbest_accuracy = 0\n\nfor epoch in range(EPOCHS):\n\n    print(f'Epoch {epoch + 1}/{EPOCHS}')\n    print('-' * 10)\n\n    train_acc, train_loss = train_epoch(\n        model,\n        train_data_loader,    \n        loss_fn, \n        optimizer, \n        device, \n        scheduler, \n        len(df_train)\n    )\n\n    print(f'Train loss {train_loss} accuracy {train_acc}')\n\n    val_acc, val_loss = eval_model(\n    model,\n    val_data_loader,\n    loss_fn, \n    device, \n    len(df_val)\n    )\n\n    print(f'Val   loss {val_loss} accuracy {val_acc}')\n    print()\n\n    history['train_acc'].append(train_acc)\n    history['train_loss'].append(train_loss)\n    history['val_acc'].append(val_acc)\n    history['val_loss'].append(val_loss)\n\n    if val_acc > best_accuracy:\n#         torch.save(model.state_dict(), 'best_model_state.bin')\n        best_accuracy = val_acc","metadata":{"id":"1zhHoFNsxufs","outputId":"a10050e2-d50e-4832-b88e-0b7cf5729b02","execution":{"iopub.status.busy":"2022-08-25T02:36:42.719576Z","iopub.execute_input":"2022-08-25T02:36:42.720143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that we're storing the state of the best model, indicated by the highest validation accuracy.","metadata":{"id":"4r8-5zWsiVur"}},{"cell_type":"markdown","source":"Whoo, this took some time! We can look at the training vs validation accuracy:","metadata":{"id":"wLQf52c7fbzr"}},{"cell_type":"code","source":"print(type(history['train_acc']))\n\nplt.plot(history['train_acc'], label='train accuracy')\n# plt.plot(history['val_acc'], label='validation accuracy')\n\nplt.title('Training history')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend()\nplt.ylim([0, 1]);","metadata":{"id":"-FWG7kBm372V","execution":{"iopub.status.busy":"2022-08-04T00:00:42.275700Z","iopub.status.idle":"2022-08-04T00:00:42.276100Z","shell.execute_reply.started":"2022-08-04T00:00:42.275908Z","shell.execute_reply":"2022-08-04T00:00:42.275927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The training accuracy starts to approach 100% after 10 epochs or so. You might try to fine-tune the parameters a bit more, but this will be good enough for us.\n\nDon't want to wait? Uncomment the next cell to download my pre-trained model:","metadata":{"id":"ZsHqkLAuf8pv"}},{"cell_type":"code","source":"# !gdown --id 1V8itWtowCYnb2Bc9KlK9SxGff9WwmogA\n\n# model = SentimentClassifier(len(class_names))\n# model.load_state_dict(torch.load('best_model_state.bin'))\n# model = model.to(device)","metadata":{"id":"zoGUH8VZ-pPQ","execution":{"iopub.status.busy":"2022-08-04T00:00:42.278353Z","iopub.status.idle":"2022-08-04T00:00:42.278795Z","shell.execute_reply.started":"2022-08-04T00:00:42.278606Z","shell.execute_reply":"2022-08-04T00:00:42.278625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluation\n\nSo how good is our model on predicting sentiment? Let's start by calculating the accuracy on the test data:","metadata":{"id":"U3HZb3NWFtFf"}},{"cell_type":"code","source":"test_acc, _ = eval_model(\n  model,\n  test_data_loader,\n  loss_fn,\n  device,\n  len(df_test)\n)\n\ntest_acc.item()","metadata":{"id":"jS3gJ_qBEljD","execution":{"iopub.status.busy":"2022-08-04T00:00:42.280379Z","iopub.status.idle":"2022-08-04T00:00:42.280758Z","shell.execute_reply.started":"2022-08-04T00:00:42.280575Z","shell.execute_reply":"2022-08-04T00:00:42.280592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The accuracy is about 1% lower on the test set. Our model seems to generalize well.\n\nWe'll define a helper function to get the predictions from our model:","metadata":{"id":"mdQ7-ylCj8Gd"}},{"cell_type":"code","source":"def get_predictions(model, data_loader):\n    model = model.eval()\n\n    review_texts = []\n    predictions = []\n    prediction_probs = []\n    real_values = []\n\n    with torch.no_grad():\n        for d in data_loader:\n\n            texts = d[\"review_text\"]\n            input_ids = d[\"input_ids\"].to(device)\n            attention_mask = d[\"attention_mask\"].to(device)\n            targets = d[\"targets\"].to(device)\n\n            outputs = model(\n            input_ids=input_ids,\n            attention_mask=attention_mask\n            )\n            _, preds = torch.max(outputs, dim=1)\n\n            probs = F.softmax(outputs, dim=1)\n\n            review_texts.extend(texts)\n            predictions.extend(preds)\n            prediction_probs.extend(probs)\n            real_values.extend(targets)\n\n    predictions = torch.stack(predictions).cpu()\n    prediction_probs = torch.stack(prediction_probs).cpu()\n    real_values = torch.stack(real_values).cpu()\n    return review_texts, predictions, prediction_probs, real_values","metadata":{"id":"EgR6MuNS8jr_","execution":{"iopub.status.busy":"2022-08-04T00:00:42.282005Z","iopub.status.idle":"2022-08-04T00:00:42.282424Z","shell.execute_reply.started":"2022-08-04T00:00:42.282222Z","shell.execute_reply":"2022-08-04T00:00:42.282242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is similar to the evaluation function, except that we're storing the text of the reviews and the predicted probabilities (by applying the softmax on the model outputs):","metadata":{"id":"dkbnBTI7kd_y"}},{"cell_type":"code","source":"y_review_texts, y_pred, y_pred_probs, y_test = get_predictions(\n  model,\n  test_data_loader\n)","metadata":{"id":"zHdPZr60-0c_","execution":{"iopub.status.busy":"2022-08-04T00:00:42.283771Z","iopub.status.idle":"2022-08-04T00:00:42.284175Z","shell.execute_reply.started":"2022-08-04T00:00:42.283960Z","shell.execute_reply":"2022-08-04T00:00:42.283978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at the classification report","metadata":{"id":"gVwoVij2lC7F"}},{"cell_type":"code","source":"print(classification_report(y_test, y_pred, target_names=class_names))","metadata":{"id":"L8a9_8-ND3Is","execution":{"iopub.status.busy":"2022-08-04T00:00:42.285546Z","iopub.status.idle":"2022-08-04T00:00:42.285928Z","shell.execute_reply.started":"2022-08-04T00:00:42.285740Z","shell.execute_reply":"2022-08-04T00:00:42.285760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like it is really hard to classify neutral (3 stars) reviews. And I can tell you from experience, looking at many reviews, those are hard to classify.\n\nWe'll continue with the confusion matrix:","metadata":{"id":"rFAekw3mmWUi"}},{"cell_type":"code","source":"def show_confusion_matrix(confusion_matrix):\n    hmap = sns.heatmap(confusion_matrix, annot=True, fmt=\"d\", cmap=\"Blues\")\n    hmap.yaxis.set_ticklabels(hmap.yaxis.get_ticklabels(), rotation=0, ha='right')\n    hmap.xaxis.set_ticklabels(hmap.xaxis.get_ticklabels(), rotation=30, ha='right')\n    plt.ylabel('True sentiment')\n    plt.xlabel('Predicted sentiment');\n\ncm = confusion_matrix(y_test, y_pred)\ndf_cm = pd.DataFrame(cm, index=class_names, columns=class_names)\nshow_confusion_matrix(df_cm)","metadata":{"id":"6d1qxsc__DTh","execution":{"iopub.status.busy":"2022-08-04T00:00:42.287231Z","iopub.status.idle":"2022-08-04T00:00:42.287629Z","shell.execute_reply.started":"2022-08-04T00:00:42.287442Z","shell.execute_reply":"2022-08-04T00:00:42.287461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This confirms that our model is having difficulty classifying neutral reviews. It mistakes those for negative and positive at a roughly equal frequency.\n\nThat's a good overview of the performance of our model. But let's have a look at an example from our test data:","metadata":{"id":"wx0U7oNsnZ3A"}},{"cell_type":"code","source":"idx = 2\n\nreview_text = y_review_texts[idx]\ntrue_sentiment = y_test[idx]\npred_df = pd.DataFrame({\n  'class_names': class_names,\n  'values': y_pred_probs[idx]\n})","metadata":{"id":"iANBiY3sLo-K","execution":{"iopub.status.busy":"2022-08-04T00:00:42.288820Z","iopub.status.idle":"2022-08-04T00:00:42.289219Z","shell.execute_reply.started":"2022-08-04T00:00:42.289014Z","shell.execute_reply":"2022-08-04T00:00:42.289032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n\".join(wrap(review_text)))\nprint()\nprint(f'True sentiment: {class_names[true_sentiment]}')","metadata":{"id":"-8D0rb1yfnv4","execution":{"iopub.status.busy":"2022-08-04T00:00:42.291880Z","iopub.status.idle":"2022-08-04T00:00:42.292298Z","shell.execute_reply.started":"2022-08-04T00:00:42.292083Z","shell.execute_reply":"2022-08-04T00:00:42.292101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can look at the confidence of each sentiment of our model:","metadata":{"id":"f7hj_IZFnn2X"}},{"cell_type":"code","source":"sns.barplot(x='values', y='class_names', data=pred_df, orient='h')\nplt.ylabel('sentiment')\nplt.xlabel('probability')\nplt.xlim([0, 1]);","metadata":{"id":"qj4d8lZyMkhf","execution":{"iopub.status.busy":"2022-08-04T00:00:42.293697Z","iopub.status.idle":"2022-08-04T00:00:42.294129Z","shell.execute_reply.started":"2022-08-04T00:00:42.293897Z","shell.execute_reply":"2022-08-04T00:00:42.293915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting on Raw Text\n\nLet's use our model to predict the sentiment of some raw text:","metadata":{"id":"7WL5pDmvFyaU"}},{"cell_type":"code","source":"review_text = \"I love completing my todos! Best app ever!!!\"","metadata":{"id":"QEPi7zQRsDhH","execution":{"iopub.status.busy":"2022-08-04T00:00:42.295275Z","iopub.status.idle":"2022-08-04T00:00:42.295657Z","shell.execute_reply.started":"2022-08-04T00:00:42.295477Z","shell.execute_reply":"2022-08-04T00:00:42.295494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have to use the tokenizer to encode the text:","metadata":{"id":"GaN4RnqMnxYw"}},{"cell_type":"code","source":"encoded_review = tokenizer.encode_plus(\n  review_text,\n  max_length=MAX_LEN,\n  add_special_tokens=True,\n  return_token_type_ids=False,\n  pad_to_max_length=True,\n  return_attention_mask=True,\n  return_tensors='pt',\n)","metadata":{"id":"zA5Or4D2sLc9","execution":{"iopub.status.busy":"2022-08-04T00:00:42.297119Z","iopub.status.idle":"2022-08-04T00:00:42.297541Z","shell.execute_reply.started":"2022-08-04T00:00:42.297350Z","shell.execute_reply":"2022-08-04T00:00:42.297369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's get the predictions from our model:","metadata":{"id":"et8xlDrKpH60"}},{"cell_type":"code","source":"input_ids = encoded_review['input_ids'].to(device)\nattention_mask = encoded_review['attention_mask'].to(device)\n\noutput = model(input_ids, attention_mask)\n_, prediction = torch.max(output, dim=1)\n\nprint(f'Review text: {review_text}')\nprint(f'Sentiment  : {class_names[prediction]}')","metadata":{"id":"Qr_t3rUksumr","execution":{"iopub.status.busy":"2022-08-04T00:00:42.299524Z","iopub.status.idle":"2022-08-04T00:00:42.299964Z","shell.execute_reply.started":"2022-08-04T00:00:42.299753Z","shell.execute_reply":"2022-08-04T00:00:42.299774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summary\n\nNice job! You learned how to use BERT for sentiment analysis. You built a custom classifier using the Hugging Face library and trained it on our app reviews dataset!\n\n- [Read the tutorial](https://www.curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/)\n- [Run the notebook in your browser (Google Colab)](https://colab.research.google.com/drive/1PHv-IRLPCtv7oTcIGbsgZHqrB5LPvB7S)\n- [Read the `Getting Things Done with Pytorch` book](https://github.com/curiousily/Getting-Things-Done-with-Pytorch)\n\nYou learned how to:\n\n- Intuitively understand what BERT is\n- Preprocess text data for BERT and build PyTorch Dataset (tokenization, attention masks, and padding)\n- Use Transfer Learning to build Sentiment Classifier using the Transformers library by Hugging Face\n- Evaluate the model on test data\n- Predict sentiment on raw text\n\nNext, we'll learn how to deploy our trained model behind a REST API and build a simple web app to access it.","metadata":{"id":"PVhwzq7bpPRl"}},{"cell_type":"markdown","source":"## References\n\n- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)\n- [L11 Language Models - Alec Radford (OpenAI)](https://www.youtube.com/watch?v=BnpB3GrpsfM)\n- [The Illustrated BERT, ELMo, and co.](https://jalammar.github.io/illustrated-bert/)\n- [BERT Fine-Tuning Tutorial with PyTorch](https://mccormickml.com/2019/07/22/BERT-fine-tuning/)\n- [How to Fine-Tune BERT for Text Classification?](https://arxiv.org/pdf/1905.05583.pdf)\n- [Huggingface Transformers](https://huggingface.co/transformers/)\n- [BERT Explained: State of the art language model for NLP](https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270)","metadata":{"id":"Wf39tauBa2V2"}}]}