{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

🧩Jigsaw🧩: All Competition EDA at One Place

\n \n
","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"

Contents

","metadata":{}},{"cell_type":"markdown","source":"1. [Competition Overview](#competition-overview) \n2. [Libraries](#libraries) \n3. [Weights and Biases](#weights-and-biases) \n4. [Global Config](#global-config) \n5. [Load Datasets](#load-datasets) \n6. [Tabular Exploration](#tabular-exploration)\n7. [Standard NLP Exploration](#standard-nlp-exploration)\n8. [Toxicity and Polarity](#toxicity-and-polarity)\n9. [Model Config](#model-config)\n10. [Utilities](#utilities)\n11. [Dataset Class](#dataset-class)\n12. [Model](#model)\n13. [Engine](#engine)\n14. [Run](#run)\n14. [References](#references)","metadata":{}},{"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation. This notebook will be updated frequently so keep checking for furthur developments.

","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Competition Overview

","metadata":{}},{"cell_type":"markdown","source":"## **Description**\n\nIn this competition, we will be asking you to score a set of about fourteen thousand comments. Pairs of comments were presented to expert raters, who marked one of two comments more harmful — each according to their own notion of toxicity. In this contest, when you provide scores for comments, they will be compared with several hundred thousand rankings. Your average agreement with the raters will determine your individual score. In this way, we hope to focus on ranking the severity of comment toxicity from innocuous to outrageous, where the middle matters as much as the extremes.\n\n## **Evaluation Criteria**\n\nSubmissions are evaluated on Average Agreement with Annotators. For the ground truth, annotators were shown two comments and asked to identify which of the two was more toxic. Pairs of comments can be, and often are, rated by more than one annotator, and may have been ordered differently by different annotators.\n \nFor each of the approximately 200,000 pair ratings in the ground truth test data, we use your predicted toxicity score to rank the comment pair. The pair receives a 1 if this ranking matches the annotator ranking, or 0 if it does not match.\n \nThe final score is the average across all the pair evaluations.\n \nPlease note the following:\n \n- score is not constrained to any numeric range (e.g., you can predict [0, 1] or [-999, 999]).\n- There is no tie breaking; tied comment scores will always be evaluated as 0. You could consider using something like scipy.stats.rankdata to force unique value.","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Libraries

","metadata":{}},{"cell_type":"code","source":"import gc\nimport os\nimport pdb\nimport time\nimport glob\nimport sys\nimport math\nimport random\nimport wandb\nimport math\n\nimport numpy as np\nimport pandas as pd\n\nimport plotly.express as px\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\n\nplt.rcParams.update({'font.size': 18})\nplt.style.use('fivethirtyeight')\n\nimport seaborn as sns\nimport matplotlib\nfrom termcolor import colored\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport re\nimport folium\n# import textstat\nfrom scipy import stats\nfrom colorama import Fore, Back, Style, init\nfrom bs4 import BeautifulSoup\n\nimport scipy as sp\n\nimport random\nimport networkx as nx\nfrom pandas import Timestamp\nfrom collections import defaultdict\n\nfrom PIL import Image\nfrom IPython.display import SVG\n\nimport requests\nfrom IPython.display import HTML\n\nimport matplotlib.cm as cm\n\nfrom tqdm import tqdm\ntqdm.pandas()\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport transformers\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom tokenizers import BertWordPieceTokenizer\n\nfrom sklearn import metrics\nfrom sklearn.utils import shuffle\nfrom gensim.models import Word2Vec\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.feature_extraction.text import TfidfVectorizer,\\\n CountVectorizer,\\\n HashingVectorizer\n\nfrom nltk.stem.wordnet import WordNetLemmatizer \nfrom nltk.tokenize import word_tokenize\nfrom nltk.tokenize import TweetTokenizer \n\nimport nltk\nfrom textblob import TextBlob\n\nfrom nltk.corpus import wordnet\nfrom nltk.corpus import stopwords\nfrom nltk import WordNetLemmatizer\n# from polyglot.detect import Detector\nfrom nltk.stem import WordNetLemmatizer\nfrom wordcloud import WordCloud, STOPWORDS\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.optim.lr_scheduler import OneCycleLR\n \n# NLP\nfrom transformers import AutoTokenizer, AutoModel\n\nstopword=set(STOPWORDS)\n\nlem = WordNetLemmatizer()\ntokenizer=TweetTokenizer()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:57:40.268689Z","iopub.execute_input":"2021-11-24T07:57:40.269184Z","iopub.status.idle":"2021-11-24T07:57:50.764111Z","shell.execute_reply.started":"2021-11-24T07:57:40.269076Z","shell.execute_reply":"2021-11-24T07:57:50.763357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T08:00:41.579626Z","iopub.execute_input":"2021-11-24T08:00:41.580303Z","iopub.status.idle":"2021-11-24T08:00:51.199798Z","shell.execute_reply.started":"2021-11-24T08:00:41.580264Z","shell.execute_reply":"2021-11-24T08:00:51.199077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Weights and Biases (W&B)

","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"**Weights & Biases** is the machine learning platform for developers to build better models faster. \n\nYou can use W&B's lightweight, interoperable tools to \n- quickly track experiments, \n- version and iterate on datasets, \n- evaluate model performance, \n- reproduce models, \n- visualize results and spot regressions, \n- and share findings with colleagues. \n\nSet up W&B in 5 minutes, then quickly iterate on your machine learning pipeline with the confidence that your datasets and models are tracked and versioned in a reliable system of record.\n\nIn this notebook I will use Weights and Biases's amazing features to perform wonderful visualizations and logging seamlessly. ","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Global Config

","metadata":{}},{"cell_type":"code","source":"class config:\n DIRECTORY_PATH = \"/kaggle/input/jigsaw-multilingual-toxic-comment-classification\"\n TRAIN_PATH = DIRECTORY_PATH + \"/jigsaw-toxic-comment-train.csv\"\n \n# Device Optimization\nif torch.cuda.is_available():\n device = torch.device('cuda')\nelse:\n device = torch.device('cpu')","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:57:50.765773Z","iopub.execute_input":"2021-11-24T07:57:50.766007Z","iopub.status.idle":"2021-11-24T07:57:50.819893Z","shell.execute_reply.started":"2021-11-24T07:57:50.765969Z","shell.execute_reply":"2021-11-24T07:57:50.819153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# wandb config\nWANDB_CONFIG = {\n 'competition': 'Jigsaw', \n '_wandb_kernel': 'neuracort'\n }","metadata":{"execution":{"iopub.status.busy":"2021-11-24T08:03:05.693004Z","iopub.execute_input":"2021-11-24T08:03:05.693507Z","iopub.status.idle":"2021-11-24T08:03:05.697278Z","shell.execute_reply.started":"2021-11-24T08:03:05.69347Z","shell.execute_reply":"2021-11-24T08:03:05.696576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Load Datasets

","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(config.TRAIN_PATH)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:18:32.329599Z","iopub.execute_input":"2021-11-24T07:18:32.329891Z","iopub.status.idle":"2021-11-24T07:18:35.492338Z","shell.execute_reply.started":"2021-11-24T07:18:32.329862Z","shell.execute_reply":"2021-11-24T07:18:35.491643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Tabular Exploration

","metadata":{}},{"cell_type":"markdown","source":"## **Dataset Head and Info**","metadata":{}},{"cell_type":"markdown","source":"Let us quickly have a brief look upon the dataset. Hope to not encounter any NULL values!","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:18:39.562462Z","iopub.execute_input":"2021-11-24T07:18:39.563516Z","iopub.status.idle":"2021-11-24T07:18:39.586449Z","shell.execute_reply.started":"2021-11-24T07:18:39.563467Z","shell.execute_reply":"2021-11-24T07:18:39.585781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:18:40.325092Z","iopub.execute_input":"2021-11-24T07:18:40.325991Z","iopub.status.idle":"2021-11-24T07:18:40.413764Z","shell.execute_reply.started":"2021-11-24T07:18:40.325945Z","shell.execute_reply":"2021-11-24T07:18:40.412842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aha! There are no NULL values, so we are good to go.","metadata":{}},{"cell_type":"markdown","source":"## **Dataset Size**","metadata":{}},{"cell_type":"code","source":"print(f\"Training Dataset Shape: {colored(train.shape, 'yellow')}\")","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:18:44.285564Z","iopub.execute_input":"2021-11-24T07:18:44.285863Z","iopub.status.idle":"2021-11-24T07:18:44.291288Z","shell.execute_reply.started":"2021-11-24T07:18:44.285829Z","shell.execute_reply":"2021-11-24T07:18:44.290438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Column-wise Unique Values**","metadata":{}},{"cell_type":"code","source":"for col in train.columns:\n print(col + \":\" + colored(str(len(train[col].unique())), 'yellow'))","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:18:47.249946Z","iopub.execute_input":"2021-11-24T07:18:47.250492Z","iopub.status.idle":"2021-11-24T07:18:47.624272Z","shell.execute_reply.started":"2021-11-24T07:18:47.250439Z","shell.execute_reply":"2021-11-24T07:18:47.623419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Standard NLP Exploration

","metadata":{}},{"cell_type":"markdown","source":"## **Wordcloud**","metadata":{}},{"cell_type":"code","source":"def nonan(x):\n if type(x) == str:\n return x.replace(\"\\n\", \"\")\n else:\n return \"\"\n\ntext = ' '.join([nonan(abstract) for abstract in train[\"comment_text\"]])\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n width=1200, height=1000).generate(text)\nfig = px.imshow(wordcloud)\nfig.update_layout(title_text='Common words in comments')","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:18:49.944213Z","iopub.execute_input":"2021-11-24T07:18:49.944584Z","iopub.status.idle":"2021-11-24T07:19:17.318207Z","shell.execute_reply.started":"2021-11-24T07:18:49.944536Z","shell.execute_reply":"2021-11-24T07:19:17.317385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **English vs Non English Comments**","metadata":{}},{"cell_type":"code","source":"def get_language(text):\n return Detector(\"\".join(x for x in text if x.isprintable()), quiet=True).languages[0].name\n\ntrain[\"lang\"] = train[\"comment_text\"].progress_apply(get_language)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:19:17.319786Z","iopub.execute_input":"2021-11-24T07:19:17.320083Z","iopub.status.idle":"2021-11-24T07:19:40.577563Z","shell.execute_reply.started":"2021-11-24T07:19:17.320042Z","shell.execute_reply":"2021-11-24T07:19:40.576625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lang_list = sorted(list(set(train[\"lang\"])))\ncounts = [list(train[\"lang\"]).count(cont) for cont in lang_list]\ndf = pd.DataFrame(np.transpose([lang_list, counts]))\ndf.columns = [\"Language\", \"Count\"]\ndf[\"Count\"] = df[\"Count\"].apply(int)\n\ndf_en = pd.DataFrame(np.transpose([[\"English\", \"Non-English\"], [max(counts), sum(counts) - max(counts)]]))\ndf_en.columns = [\"Language\", \"Count\"]\n\nfig = px.bar(df_en, x=\"Language\", y=\"Count\", title=\"Language of comments\", color=\"Language\", text=\"Count\")\nfig.update_layout(template=\"plotly_white\")\nfig.data[0].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[0].marker.line.width = 0.5\nfig.data[1].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[1].marker.line.width = 0.5\nfig.data[0].textfont.color = \"black\"\nfig.data[0].textposition = \"outside\"\nfig.data[1].textfont.color = \"black\"\nfig.data[1].textposition = \"outside\"\nfig","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:19:40.579742Z","iopub.execute_input":"2021-11-24T07:19:40.580018Z","iopub.status.idle":"2021-11-24T07:19:45.882523Z","shell.execute_reply.started":"2021-11-24T07:19:40.579945Z","shell.execute_reply":"2021-11-24T07:19:45.881599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Bar chart of non-English languages**","metadata":{}},{"cell_type":"code","source":"fig = px.bar(df.query(\"Language != 'English' and Language != 'un'\").query(\"Count >= 50\"),\n y=\"Language\", x=\"Count\", title=\"Language of non-English comments\", template=\"plotly_white\", color=\"Language\", text=\"Count\", orientation=\"h\")\nfig.update_traces(marker=dict(line=dict(width=0.75,\n color='black')), textposition=\"outside\")\nfig.update_layout(showlegend=False)\nfig","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:19:45.885252Z","iopub.execute_input":"2021-11-24T07:19:45.885617Z","iopub.status.idle":"2021-11-24T07:19:46.027699Z","shell.execute_reply.started":"2021-11-24T07:19:45.885571Z","shell.execute_reply":"2021-11-24T07:19:46.026883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Comment Word Distribution**","metadata":{}},{"cell_type":"code","source":"def new_len(x):\n if type(x) is str:\n return len(x.split())\n else:\n return 0\n\ntrain[\"comment_words\"] = train[\"comment_text\"].apply(new_len)\nnums = train.query(\"comment_words != 0 and comment_words < 200\").sample(frac=0.1)[\"comment_words\"]\nfig = ff.create_distplot(hist_data=[nums],\n group_labels=[\"All comments\"],\n colors=[\"coral\"])\n\nfig.update_layout(title_text=\"Comment words\", xaxis_title=\"Comment words\", template=\"simple_white\", showlegend=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:19:46.029023Z","iopub.execute_input":"2021-11-24T07:19:46.029303Z","iopub.status.idle":"2021-11-24T07:19:47.90053Z","shell.execute_reply.started":"2021-11-24T07:19:46.029273Z","shell.execute_reply":"2021-11-24T07:19:47.899273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Average Comment Words vs Language**","metadata":{}},{"cell_type":"code","source":"df = pd.DataFrame(np.transpose([lang_list, train.groupby(\"lang\").mean()[\"comment_words\"]]))\ndf.columns = [\"Language\", \"Average_comment_words\"]\ndf[\"Average_comment_words\"] = df[\"Average_comment_words\"].apply(float)\ndf = df.query(\"Average_comment_words < 500\")\nfig = go.Figure(go.Bar(x=df[\"Language\"], y=df[\"Average_comment_words\"]))\n\nfig.update_layout(xaxis_title=\"Language\", yaxis_title=\"Average comment words\", title_text=\"Average comment words vs. language\", template=\"plotly_white\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:19:47.902074Z","iopub.execute_input":"2021-11-24T07:19:47.902928Z","iopub.status.idle":"2021-11-24T07:19:48.003434Z","shell.execute_reply.started":"2021-11-24T07:19:47.902882Z","shell.execute_reply":"2021-11-24T07:19:48.002663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Toxicity and Polarity

","metadata":{}},{"cell_type":"code","source":"def polarity(x):\n if type(x) == str:\n return SIA.polarity_scores(x)\n else:\n return 1000\n \nSIA = SentimentIntensityAnalyzer()\ntrain[\"polarity\"] = train[\"comment_text\"].progress_apply(polarity)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:19:48.004477Z","iopub.execute_input":"2021-11-24T07:19:48.005243Z","iopub.status.idle":"2021-11-24T07:23:16.060452Z","shell.execute_reply.started":"2021-11-24T07:19:48.005203Z","shell.execute_reply":"2021-11-24T07:23:16.059083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neg\"] for pols in train[\"polarity\"] if pols[\"neg\"] != 0], marker=dict(\n color='seagreen')\n ))\n\nfig.update_layout(xaxis_title=\"Negativity sentiment\", title_text=\"Negativity sentiment\", template=\"simple_white\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:23:16.06306Z","iopub.execute_input":"2021-11-24T07:23:16.06416Z","iopub.status.idle":"2021-11-24T07:23:17.501531Z","shell.execute_reply.started":"2021-11-24T07:23:16.064108Z","shell.execute_reply":"2021-11-24T07:23:17.500551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Model Config

","metadata":{}},{"cell_type":"code","source":"data_dir = '../input/jrstc-train-folds'\ntrain_file_path = os.path.join(data_dir, 'validation_data_5_folds.csv')\ntrain_df = pd.read_csv(train_file_path)\n\nparams = {\n 'device': device,\n 'debug': False,\n 'checkpoint': 'roberta-base',\n 'output_logits': 768,\n 'max_len': 256,\n 'num_folds': 1,\n 'batch_size': 16,\n 'dropout': 0.2,\n 'num_workers': 2,\n 'epochs': 1,\n 'lr': 2e-5,\n 'margin': 0.7,\n 'scheduler_name': 'OneCycleLR',\n 'max_lr': 5e-5, # OneCycleLR\n 'pct_start': 0.1, # OneCycleLR\n 'anneal_strategy': 'cos', # OneCycleLR\n 'div_factor': 1e3, # OneCycleLR\n 'final_div_factor': 1e3, # OneCycleLR\n 'no_decay': True\n}\n\nif params['debug']:\n train_df = train_df.sample(frac=0.01)\n print('Reduced training Data Size for Debugging purposes')","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:25.014895Z","iopub.execute_input":"2021-11-24T07:58:25.015144Z","iopub.status.idle":"2021-11-24T07:58:25.239167Z","shell.execute_reply.started":"2021-11-24T07:58:25.01511Z","shell.execute_reply":"2021-11-24T07:58:25.238431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Utilities

","metadata":{}},{"cell_type":"markdown","source":"## **Text Preprocessing**","metadata":{}},{"cell_type":"code","source":"def text_cleaning(text):\n '''\n Cleans text into a basic form for NLP. Operations include the following:-\n 1. Remove special charecters like &, #, etc\n 2. Removes extra spaces\n 3. Removes embedded URL links\n 4. Removes HTML tags\n 5. Removes emojis\n \n text - Text piece to be cleaned.\n '''\n template = re.compile(r'https?://\\S+|www\\.\\S+') #Removes website links\n text = template.sub(r'', text)\n \n soup = BeautifulSoup(text, 'lxml') #Removes HTML tags\n only_text = soup.get_text()\n text = only_text\n \n emoji_pattern = re.compile(\"[\"\n u\"\\U0001F600-\\U0001F64F\" # emoticons\n u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n u\"\\U00002702-\\U000027B0\"\n u\"\\U000024C2-\\U0001F251\"\n \"]+\", flags=re.UNICODE)\n text = emoji_pattern.sub(r'', text)\n \n text = re.sub(r\"[^a-zA-Z\\d]\", \" \", text) #Remove special Charecters\n text = re.sub(' +', ' ', text) #Remove Extra Spaces\n text = text.strip() # remove spaces at the beginning and at the end of string\n\n return text\n\ntqdm.pandas()\ntrain_df['less_toxic'] = train_df['less_toxic'].progress_apply(text_cleaning)\ntrain_df['more_toxic'] = train_df['more_toxic'].progress_apply(text_cleaning)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:28.359647Z","iopub.execute_input":"2021-11-24T07:58:28.359901Z","iopub.status.idle":"2021-11-24T07:58:48.146457Z","shell.execute_reply.started":"2021-11-24T07:58:28.359872Z","shell.execute_reply":"2021-11-24T07:58:48.145711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Metric Monitor**","metadata":{}},{"cell_type":"code","source":"class MetricMonitor:\n def __init__(self, float_precision=4):\n self.float_precision = float_precision\n self.reset()\n\n def reset(self):\n self.metrics = defaultdict(lambda: {\"val\": 0, \"count\": 0, \"avg\": 0})\n\n def update(self, metric_name, val):\n metric = self.metrics[metric_name]\n\n metric[\"val\"] += val\n metric[\"count\"] += 1\n metric[\"avg\"] = metric[\"val\"] / metric[\"count\"]\n\n def __str__(self):\n return \" | \".join(\n [\n \"{metric_name}: {avg:.{float_precision}f}\".format(\n metric_name=metric_name, avg=metric[\"avg\"],\n float_precision=self.float_precision\n )\n for (metric_name, metric) in self.metrics.items()\n ]\n )","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.151859Z","iopub.execute_input":"2021-11-24T07:58:48.152446Z","iopub.status.idle":"2021-11-24T07:58:48.165868Z","shell.execute_reply.started":"2021-11-24T07:58:48.152403Z","shell.execute_reply":"2021-11-24T07:58:48.16506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Scheduler**","metadata":{}},{"cell_type":"code","source":"def get_scheduler(optimizer, scheduler_params=params):\n if scheduler_params['scheduler_name'] == 'CosineAnnealingWarmRestarts':\n scheduler = CosineAnnealingWarmRestarts(\n optimizer,\n T_0=scheduler_params['T_0'],\n eta_min=scheduler_params['min_lr'],\n last_epoch=-1\n )\n elif scheduler_params['scheduler_name'] == 'OneCycleLR':\n scheduler = OneCycleLR(\n optimizer,\n max_lr=scheduler_params['max_lr'],\n steps_per_epoch=int(df_train.shape[0] / params['batch_size']) + 1,\n epochs=scheduler_params['epochs'],\n pct_start=scheduler_params['pct_start'],\n anneal_strategy=scheduler_params['anneal_strategy'],\n div_factor=scheduler_params['div_factor'],\n final_div_factor=scheduler_params['final_div_factor'],\n )\n return scheduler","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.171722Z","iopub.execute_input":"2021-11-24T07:58:48.172381Z","iopub.status.idle":"2021-11-24T07:58:48.18183Z","shell.execute_reply.started":"2021-11-24T07:58:48.172341Z","shell.execute_reply":"2021-11-24T07:58:48.181082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Dataset Class

","metadata":{}},{"cell_type":"code","source":"class BERTDataset:\n def __init__(self, more_toxic, less_toxic, max_len=params['max_len'], checkpoint=params['checkpoint']):\n self.more_toxic = more_toxic\n self.less_toxic = less_toxic\n self.max_len = max_len\n self.checkpoint = checkpoint\n self.tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n self.num_examples = len(self.more_toxic)\n\n def __len__(self):\n return self.num_examples\n\n def __getitem__(self, idx):\n more_toxic = str(self.more_toxic[idx])\n less_toxic = str(self.less_toxic[idx])\n\n tokenized_more_toxic = self.tokenizer(\n more_toxic,\n add_special_tokens=True,\n truncation=True,\n padding='max_length',\n max_length=self.max_len,\n return_attention_mask=True,\n return_token_type_ids=True,\n )\n\n tokenized_less_toxic = self.tokenizer(\n less_toxic,\n add_special_tokens=True,\n truncation=True,\n padding='max_length',\n max_length=self.max_len,\n return_attention_mask=True,\n return_token_type_ids=True,\n )\n\n ids_more_toxic = tokenized_more_toxic['input_ids']\n mask_more_toxic = tokenized_more_toxic['attention_mask']\n token_type_ids_more_toxic = tokenized_more_toxic['token_type_ids']\n\n ids_less_toxic = tokenized_less_toxic['input_ids']\n mask_less_toxic = tokenized_less_toxic['attention_mask']\n token_type_ids_less_toxic = tokenized_less_toxic['token_type_ids']\n\n return {'ids_more_toxic': torch.tensor(ids_more_toxic, dtype=torch.long),\n 'mask_more_toxic': torch.tensor(mask_more_toxic, dtype=torch.long),\n 'token_type_ids_more_toxic': torch.tensor(token_type_ids_more_toxic, dtype=torch.long),\n 'ids_less_toxic': torch.tensor(ids_less_toxic, dtype=torch.long),\n 'mask_less_toxic': torch.tensor(mask_less_toxic, dtype=torch.long),\n 'token_type_ids_less_toxic': torch.tensor(token_type_ids_less_toxic, dtype=torch.long),\n 'target': torch.tensor(1, dtype=torch.float)}","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.184301Z","iopub.execute_input":"2021-11-24T07:58:48.184812Z","iopub.status.idle":"2021-11-24T07:58:48.20368Z","shell.execute_reply.started":"2021-11-24T07:58:48.184776Z","shell.execute_reply":"2021-11-24T07:58:48.20254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Model

","metadata":{}},{"cell_type":"code","source":"class ToxicityModel(nn.Module):\n def __init__(self, checkpoint=params['checkpoint'], params=params):\n super(ToxicityModel, self).__init__()\n self.checkpoint = checkpoint\n self.bert = AutoModel.from_pretrained(checkpoint, return_dict=False)\n self.layer_norm = nn.LayerNorm(params['output_logits'])\n self.dropout = nn.Dropout(params['dropout'])\n self.dense = nn.Sequential(\n nn.Linear(params['output_logits'], 128),\n nn.SiLU(),\n nn.Dropout(params['dropout']),\n nn.Linear(128, 1)\n )\n\n def forward(self, input_ids, token_type_ids, attention_mask):\n _, pooled_output = self.bert(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)\n pooled_output = self.layer_norm(pooled_output)\n pooled_output = self.dropout(pooled_output)\n preds = self.dense(pooled_output)\n return preds","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.205069Z","iopub.execute_input":"2021-11-24T07:58:48.205559Z","iopub.status.idle":"2021-11-24T07:58:48.217927Z","shell.execute_reply.started":"2021-11-24T07:58:48.205525Z","shell.execute_reply":"2021-11-24T07:58:48.216985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Engine

","metadata":{}},{"cell_type":"markdown","source":"## **Training Function**","metadata":{}},{"cell_type":"code","source":"def train_fn(train_loader, model, criterion, optimizer, epoch, params, scheduler=None):\n metric_monitor = MetricMonitor()\n model.train()\n stream = tqdm(train_loader)\n \n for i, batch in enumerate(stream, start=1):\n ids_more_toxic = batch['ids_more_toxic'].to(device)\n mask_more_toxic = batch['mask_more_toxic'].to(device)\n token_type_ids_more_toxic = batch['token_type_ids_more_toxic'].to(device)\n \n ids_less_toxic = batch['ids_less_toxic'].to(device)\n mask_less_toxic = batch['mask_less_toxic'].to(device)\n token_type_ids_less_toxic = batch['token_type_ids_less_toxic'].to(device)\n \n target = batch['target'].to(device)\n\n logits_more_toxic = model(ids_more_toxic, token_type_ids_more_toxic, mask_more_toxic)\n logits_less_toxic = model(ids_less_toxic, token_type_ids_less_toxic, mask_less_toxic)\n \n loss = criterion(logits_more_toxic, logits_less_toxic, target)\n \n metric_monitor.update('Loss', loss.item())\n \n loss.backward()\n optimizer.step()\n \n if scheduler is not None:\n scheduler.step()\n \n optimizer.zero_grad()\n stream.set_description(f\"Epoch: {epoch:02}. Train. {metric_monitor}\")","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.21926Z","iopub.execute_input":"2021-11-24T07:58:48.219776Z","iopub.status.idle":"2021-11-24T07:58:48.233141Z","shell.execute_reply.started":"2021-11-24T07:58:48.219741Z","shell.execute_reply":"2021-11-24T07:58:48.232432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Validation Function**","metadata":{}},{"cell_type":"code","source":"def validate_fn(val_loader, model, criterion, epoch, params):\n metric_monitor = MetricMonitor()\n model.eval()\n stream = tqdm(val_loader)\n all_loss = []\n with torch.no_grad():\n for i, batch in enumerate(stream, start=1):\n ids_more_toxic = batch['ids_more_toxic'].to(device)\n mask_more_toxic = batch['mask_more_toxic'].to(device)\n token_type_ids_more_toxic = batch['token_type_ids_more_toxic'].to(device)\n \n ids_less_toxic = batch['ids_less_toxic'].to(device)\n mask_less_toxic = batch['mask_less_toxic'].to(device)\n token_type_ids_less_toxic = batch['token_type_ids_less_toxic'].to(device)\n \n target = batch['target'].to(device)\n\n logits_more_toxic = model(ids_more_toxic, token_type_ids_more_toxic, mask_more_toxic)\n logits_less_toxic = model(ids_less_toxic, token_type_ids_less_toxic, mask_less_toxic)\n \n loss = criterion(logits_more_toxic, logits_less_toxic, target)\n all_loss.append(loss.item())\n \n metric_monitor.update('Loss', loss.item())\n \n stream.set_description(f\"Epoch: {epoch:02}. Valid. {metric_monitor}\")\n \n return np.mean(all_loss)","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.234687Z","iopub.execute_input":"2021-11-24T07:58:48.235238Z","iopub.status.idle":"2021-11-24T07:58:48.249084Z","shell.execute_reply.started":"2021-11-24T07:58:48.235201Z","shell.execute_reply":"2021-11-24T07:58:48.248162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Run

","metadata":{}},{"cell_type":"code","source":"best_models_of_each_fold = []","metadata":{"execution":{"iopub.status.busy":"2021-11-24T07:58:48.250251Z","iopub.execute_input":"2021-11-24T07:58:48.250752Z","iopub.status.idle":"2021-11-24T07:58:48.259873Z","shell.execute_reply.started":"2021-11-24T07:58:48.250716Z","shell.execute_reply":"2021-11-24T07:58:48.259055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()\nfor fold in range(params['num_folds']):\n print(f'******************** Training Fold: {fold+1} ********************')\n current_fold = fold\n df_train = train_df[train_df['kfold'] != current_fold].copy()\n df_valid = train_df[train_df['kfold'] == current_fold].copy()\n\n train_dataset = BERTDataset(\n df_train.more_toxic.values,\n df_train.less_toxic.values\n )\n valid_dataset = BERTDataset(\n df_valid.more_toxic.values,\n df_valid.less_toxic.values\n )\n\n train_dataloader = DataLoader(\n train_dataset, batch_size=params['batch_size'], shuffle=True,\n num_workers=params['num_workers'], pin_memory=True\n )\n valid_dataloader = DataLoader(\n valid_dataset, batch_size=params['batch_size']*2, shuffle=False,\n num_workers=params['num_workers'], pin_memory=True\n )\n \n model = ToxicityModel()\n model = model.to(params['device'])\n criterion = nn.MarginRankingLoss(margin=params['margin'])\n if params['no_decay']:\n param_optimizer = list(model.named_parameters())\n no_decay = ['bias', 'LayerNorm.weight', 'LayerNorm.bias']\n optimizer_grouped_parameters = [\n {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},\n {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n ]\n optimizer = optim.AdamW(optimizer_grouped_parameters, lr=params['lr'])\n else:\n optimizer = optim.AdamW(model.parameters(), lr=params['lr'])\n scheduler = get_scheduler(optimizer)\n\n # Training and Validation Loop\n best_loss = np.inf\n best_epoch = 0\n best_model_name = None\n \n # Initialize W&B\n run = wandb.init(project='Jigsaw-RoBerta', config= WANDB_CONFIG)\n \n for epoch in range(1, params['epochs'] + 1):\n train_fn(train_dataloader, model, criterion, optimizer, epoch, params, scheduler)\n valid_loss = validate_fn(valid_dataloader, model, criterion, epoch, params)\n if valid_loss <= best_loss:\n best_loss = valid_loss\n best_epoch = epoch\n if best_model_name is not None:\n os.remove(best_model_name)\n torch.save(model.state_dict(), f\"{params['checkpoint']}_{epoch}_epoch_f{fold+1}.pth\")\n best_model_name = f\"{params['checkpoint']}_{epoch}_epoch_f{fold+1}.pth\"\n \n wandb.log({'valid_loss': valid_loss})\n \n # Close W&B run\n wandb.finish()\n\n # Print summary of this fold\n print('')\n print(f'The best LOSS: {best_loss} for fold {fold+1} was achieved on epoch: {best_epoch}.')\n print(f'The Best saved model is: {best_model_name}')\n best_models_of_each_fold.append(best_model_name)\n del df_train, df_valid, train_dataset, valid_dataset, train_dataloader, valid_dataloader, model\n _ = gc.collect()\n torch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2021-11-24T08:03:10.020765Z","iopub.execute_input":"2021-11-24T08:03:10.021454Z","iopub.status.idle":"2021-11-24T08:25:08.897319Z","shell.execute_reply.started":"2021-11-24T08:03:10.021418Z","shell.execute_reply":"2021-11-24T08:25:08.896523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

References

","metadata":{}},{"cell_type":"markdown","source":"> - [Jigsaw Multilingual Toxicity : EDA + Models](https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models)\n> - [Pytorch RoBERTa Ranking Baseline JRSTC [Train]](https://www.kaggle.com/manabendrarout/pytorch-roberta-ranking-baseline-jrstc-train)\n>\n>---","metadata":{}},{"cell_type":"markdown","source":"

More Plots and Models coming soon!

\n\n
","metadata":{}},{"cell_type":"markdown","source":"--- \n\n## **Let's have a Talk!**\n> ### Reach out to me on [LinkedIn](https://www.linkedin.com/in/ishandutta0098)\n\n---","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}