{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":19018,"databundleVersionId":1053282,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# Installations\n!pip install -q nltk\n!pip install -q WordCloud\n!pip install -q plotly\n!pip install -q transformers\n!pip install -q twython","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\n\n#sklearn\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.model_selection import train_test_split\n\n#NLP\nimport string\nimport re    #for regex\nimport nltk\nfrom nltk.corpus import stopwords\nfrom wordcloud  import WordCloud, STOPWORDS\nfrom nltk import pos_tag\nfrom nltk.stem.wordnet import WordNetLemmatizer \nfrom nltk.tokenize import word_tokenize\n# Tweet tokenizer does not split at apostophes which is what we want\nfrom nltk.tokenize import TweetTokenizer \nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\n\n\nnltk.download('stopwords')\nnltk.download('punkt')\nnltk.download('vader_lexicon')\n\n#Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\n#Modelling\n\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nimport transformers\nfrom transformers import TFAutoModel, AutoTokenizer\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors\n\nfrom kaggle_datasets import KaggleDatasets","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.listdir('/kaggle/working')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"###### Code Header\n- **Notebook Name**: Multilingual Toxicity Scoring\n- **Author(s)**: Vallabh Reddy\n- **Date**: 13th May 2020\n- **Edits to be made**:\n- **Additions Planned**:\n- **Workflow Plan**: \n    - Import data and preliminary setup\n    - Inspect datsets\n    - Visualize dataset properties\n    - Follow text preprocessing steps such as stemming, lemmatization, case generalization\n    - Wrangle the text datasets to extract unigrams, bigrams and trigrams ( Does \n    - Visualize the top used n-grams for toxicity and non-toxicity through word clouds and other means\n    - Investigate need for any other text representations required like tfidf, word vectors etc\n    - Pick models, train models. Should I train them only in English? Or would translating to other languages and then training models on that data help? Instead I could just translate test to english and then pass into model\n    - Use validation dataset to tune hyperparameters\n    - Test models on test dataset after translating\n    - Investigate value of ensembling\n- **Notes to Self**:\n    - How do I deal with spelling mistakes? Is there a way to coerce words to the right spelling using sentence context? Explore existing text analysis models for this.\n    ","metadata":{}},{"cell_type":"markdown","source":"### Acknowledgments\n\n1. [VADER(Valence Aware Dictionary and sEntiment Reasoner)](https://pypi.org/project/vaderSentiment/) - The NLTK package contains the VADER tool which allows us to score the sentiment \n2. [HuggingFace's Transformers and Tokenizers](https://huggingface.co/transformers/) - HuggingFace has a collection of pretrained NLP models to pick from including Facebook's RoBERTa and Google's BERT. The same package comes with tokenizers to preprocess the text for these models","metadata":{}},{"cell_type":"markdown","source":"## Contents:\n- [About This Project](#Problem-Space)\n- [About The Datasets](#about-datasets)\n- [Setup](#Setup)\n    - [Wrangling](#Wrangling)\n    - [EDA](#eda)\n- [Modelling](#Modelling)\n    - [1. XLM-RoBERTa](#Roberta)","metadata":{}},{"cell_type":"markdown","source":"## About this project <a class=\"anchor\" id=\"Problem-Space\"></a>\n\nThe Conversation AI team, a research initiative founded by Google and Jigsaw, is tasked with improving the vigilance against online toxicity in conversation. The goal of [this competition](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/overview) is to be able to predict the toxicity of multilingual comments using only English comments as our training data. \n\nExcerpts from the competition are given below.\n\n>It only takes one toxic comment to sour an online discussion. The Conversation AI team, a research initiative founded by Jigsaw and Google, builds technology to protect voices in conversation. A main area of focus is machine learning models that can identify toxicity in online conversations, where toxicity is defined as anything rude, disrespectful or otherwise likely to make someone leave a discussion. If these toxic contributions can be identified, we could have a safer, more collaborative internet.\n>\n>In the previous 2018 Toxic Comment Classification Challenge, Kagglers built multi-headed models to recognize toxicity and several subtypes of toxicity. In 2019, in the Unintended Bias in Toxicity Classification Challenge, you worked to build toxicity models that operate fairly across a diverse range of conversations. This year, we're taking advantage of Kaggle's new TPU support and challenging you to build multilingual models with English-only training data.\n>\n>Jigsaw's API, Perspective, serves toxicity models and others in a growing set of languages (see our documentation for the full list). Over the past year, the field has seen impressive multilingual capabilities from the latest model innovations, including few- and zero-shot learning. We're excited to learn whether these results \"translate\" (pun intended!) to toxicity classification. Your training data will be the English data provided for our previous two competitions and your test data will be Wikipedia talk page comments in several different languages.\n>\n>As our computing resources and modeling capabilities grow, so does our potential to support healthy conversations across the globe. Develop strategies to build effective multilingual models and you'll help Conversation AI and the entire industry realize that potential.\n>\n>*Disclaimer: The dataset for this competition contains text that may be considered profane, vulgar, or offensive.*","metadata":{}},{"cell_type":"markdown","source":"### About the datasets <a class = 'anchor' id = 'about-datasets'></a>\nWe are given the following datasets.\n\n**Training set 1**\n- *Comment_text*: Contains the string that stores the comment.\n- *Toxic*: A boolean value, 1 = toxic, 0 = non-toxic.\n\n**Training set 2**: Has 'Comment_text' and 'toxic' similar to Training Set 1, but the 'Toxic' column is a probability. Also has several other descriptor probabilities.\n\n**Validation Set**\n- *Comment_text*: Same as Training Set 1.\n- *Toxic*: Same as Training Set 1.\n- *Lang*: Two letter representation of the language of the comment. 'es'= Espaniol, 'it' = Italian etc.\n\n**Test Set**\n- *Comment_text*: Same as Training Set 1.\n- *Lang*: same as Validation Set.\n- Does not have a 'Toxic' flag and we are tasked with predicting it.\n","metadata":{}},{"cell_type":"markdown","source":"## Setup <a class=\"anchor\" id=\"Setup\"></a>","metadata":{}},{"cell_type":"code","source":"# Importing all the required datasets\ndir_path = '/kaggle/input/jigsaw-multilingual-toxic-comment-classification/'\n\ntrain_data_1 = pd.read_csv(dir_path + \"/jigsaw-toxic-comment-train.csv\")\ntrain_data_2 = pd.read_csv(dir_path + \"/jigsaw-unintended-bias-train.csv\")\nvalidation_data = pd.read_csv(dir_path + \"/validation.csv\")\ntest_data = pd.read_csv(dir_path + \"/test.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_1.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_1.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_2.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_2.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"validation_data.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"validation_data.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.shape","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Wrangling <a class=\"anchor\" id=\"Wrangling\"></a>","metadata":{}},{"cell_type":"code","source":"# In the second dataset, the toxicity is not 1 or 0 but instead a probability, we will round it to convert to a 1/0 column\ntrain_data_2.toxic = train_data_2.toxic.round().astype(int)\n\n# We combined the entire training set 1 with all the toxic comments of training set 2 and 200k non-toxic comments from set 2\ntrain_data = pd.concat([\n                train_data_1[['comment_text','toxic']]\n                , train_data_2[['comment_text','toxic']].query('toxic == 1')\n                , train_data_2[['comment_text', 'toxic']].query('toxic == 0').sample(n = 200000, random_state = 1993)\n                ])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(train_data.toxic)","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(validation_data.toxic)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(train_data.comment_text[1])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#count_vectorizer = CountVectorizer(stop_words = 'english', ngram_range=(1,3))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#trial = count_vectorizer.fit_transform(train_data.comment_text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#trial.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Splitting up the comment into single words\ntext_words = word_tokenize(train_data.comment_text[1])\n# Converting to lower case\ntext_words = [word.lower() for word in text_words]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modified_stopwords = stopwords.words('english')\nmodified_stopwords.remove('not')\n#Removing stopwords and sumbols\ntext_words = [word for  word in text_words if not word in modified_stopwords and word.isalpha()]\nlen(text_words)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"text_words","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sample = train_data.sample(n = 10000, random_state = 1993)\ntrain_sample = train_sample.reset_index(drop = True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### EDA <a class = 'anchor'  id ='eda'></a>\n","metadata":{}},{"cell_type":"code","source":"# Filtering comment text column, removing newline characters and filtering out unexpected data types from the column\ndef nan_filter(x):\n    if type(x) == str:\n        return (x.replace(\"\\n\", \"\")).lower()\n    else:\n        return \"\"\n\nnontoxic_text = ' '.join([nan_filter(comment) for comment in train_sample.query('toxic==0')['comment_text']])\ntoxic_text = ' '.join([nan_filter(comment) for comment in train_sample.query('toxic == 1')['comment_text']])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordcloud = WordCloud(max_font_size=300\n                      , background_color='white'\n                      , stopwords = modified_stopwords\n                      , collocations=True\n                      , max_words = 100\n                      , width=1200\n                      , height=1000).generate(nontoxic_text)\n\nfig = px.imshow(wordcloud)\n\nfig.update_layout(title_text='Non-Toxic Word Cloud(with bigrams)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordcloud = WordCloud(max_font_size=300\n                      , background_color='white'\n                      , stopwords = modified_stopwords\n                      , collocations=False\n                      , max_words = 100\n                      , width=1200\n                      , height=1000).generate(nontoxic_text)\n\nfig = px.imshow(wordcloud)\n\nfig.update_layout(title_text='Non-Toxic Word Cloud(unigrams)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordcloud = WordCloud(max_font_size=300\n                      , background_color='white'\n                      , stopwords = modified_stopwords\n                      , collocations=True\n                      , width=1200\n                      , max_words = 100\n                      , height=1000).generate(toxic_text)\n\nfig = px.imshow(wordcloud)\n\nfig.update_layout(title_text='Toxic Word Cloud(with bigrams)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordcloud = WordCloud(max_font_size=300\n                      , background_color='white'\n                      , stopwords = modified_stopwords\n                      , collocations=False\n                      , max_words = 100\n                      , width=1200\n                      , height=1000).generate(toxic_text)\n\nfig = px.imshow(wordcloud)\n\nfig.update_layout(title_text='Toxic Word Cloud(unigrams)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's interesting to note that in the toxic word cloud we see both 'hate' and 'like' as high frequency unigrams. On further inspection, I realized 'like' is used more often to compare the subject to something derogatory. \"You're acting like a ...\" , \" You're just like ..\" etc and less often in the sense \"I like ...\".","metadata":{}},{"cell_type":"code","source":"train_sample.iloc[1,0]","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Comment size visualizations\n\ndef text_len(x):\n    if type(x) is str:\n        return len(x.split())\n    else:\n        return 0\n    \n\ntrain_sample['comment_size'] = train_sample.comment_text.apply(text_len)\n\ntoxic_text_lengths = train_sample.query('toxic == 1 and comment_size < 200') ['comment_size'].sample(frac = 1, random_state = 1993)\nnontoxic_text_lengths = train_sample.query('toxic == 0 and comment_size < 200')['comment_size'].sample(frac = 1, random_state = 1993)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(13,5))\nax = sns.distplot(toxic_text_lengths)\nplt.title('Toxic Comment Lengths')\nplt.xlabel('Comment Length')\nplt.xticks(np.arange(0,210,10))\nplt.yticks(np.arange(0,0.025,0.0025));","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(13,5))\nax = sns.distplot(nontoxic_text_lengths)\nplt.title('Non-Toxic Comment Lengths')\nplt.xlabel('Comment Length')\nplt.xticks(np.arange(0,210,10))\nplt.yticks(np.arange(0,0.025,0.0025));","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sentiment Scores using VADER\nWe'll try scoring the sentiment of the comments using the VADER component of NLTK. Here is an [article](http://t-redactyl.io/blog/2017/04/using-vader-to-handle-sentiment-analysis-with-social-media-text.html) that expands on the procedure, it works better with social media content than general approaches. Here's a link to the original team's [paper](http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf).\n\nNote that the negative sentiment here is not the same as the toxicity we are looking for. Negativity might simply be portrayal of discontent, which is not toxic.","metadata":{}},{"cell_type":"code","source":"def sentiment(x):\n    if type(x) is str:\n        return SIA.polarity_scores(x)\n    else:\n        return 1000\n\nSIA = SentimentIntensityAnalyzer()\ntrain_sample['polarity'] = train_sample.comment_text.apply(sentiment)\n# Vader outputs 4 scores, Negative, Neutral, Positive and Compound\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sample.query('toxic == 0').head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sample.query('toxic==1').head(10)","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This comment has a negative score of 0 despite clearly being toxic.\ntrain_sample.comment_text[22]","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This comment has a negative score of 0 despite clearly being toxic.\n\nOn first look, it appears that VADER does not recognize negative terms when the writer masks characters with \\*.","metadata":{}},{"cell_type":"code","source":"train_sample['negativity'] = train_sample.polarity.apply(lambda x: x['neg'])\ntrain_sample['positivity'] = train_sample.polarity.apply(lambda x: x['pos'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Comparing the Negativity Score with Toxicity","metadata":{}},{"cell_type":"code","source":"nontoxic_negativity = train_sample.query('toxic == 0').sample(frac = 1, random_state = 1993)['negativity']\ntoxic_negativity = train_sample.query('toxic == 1').sample(frac = 1, random_state = 1993)['negativity']\n\nplot = ff.create_distplot([nontoxic_negativity, toxic_negativity]\n                           , group_labels = ['Non-Toxic', 'Toxic']\n                           , colors = ['Green', 'Red']\n                           , show_hist= False)\nplot.update_layout(title_text = 'Negativity vs Toxicity'\n                   , xaxis_title = 'Negativity'\n                   , xaxis = dict(tickmode = 'linear', tick0 = 0, dtick = 0.1))\n\nplot.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The negativity score seems to be able to differentiate the toxic comments from the non toxic to a certain extent. With a greater share of non-toxic comments having lower negativity and many toxic comments having at least a slight negative sentiment around 0.1-0.3","metadata":{}},{"cell_type":"markdown","source":"### Comparing the Positivity Score with Toxicity","metadata":{}},{"cell_type":"code","source":"nontoxic_positivity = train_sample.query('toxic == 0').sample(frac = 1, random_state = 1993)['positivity']\ntoxic_positivity = train_sample.query('toxic == 1').sample(frac = 1, random_state = 1993)['positivity']\n\nplot = ff.create_distplot([nontoxic_positivity, toxic_positivity]\n                          , group_labels=['Non-Toxic', 'Toxic']\n                          , colors = ['Green', 'Red']\n                          , show_hist= False)\n\nplot.update_layout( title_text = 'Positivity vs Toxicity'\n                    , xaxis_title = 'Positivity'\n                    , xaxis = dict(tickmode = 'linear', tick0 = 0, dtick = 0.1))\nplot.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Though we see that non-toxic comments have more observations at higher positivity levels, the positivity score does seems to be able differentiate toxic from non-toxic very well.","metadata":{}},{"cell_type":"markdown","source":"### Modelling <a class=anchor id='Modelling'></a>","metadata":{}},{"cell_type":"markdown","source":"### Model 1: XLM RoBERTa <a class = 'anchor' id = 'Roberta'></a>","metadata":{}},{"cell_type":"code","source":"roberta_string = 'jplu/tf-xlm-roberta-large'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Encoding\nWe must first encode our data before we feed it to the model. The reason we must encode it is because ML algorithms cannot directly interpret text. They are designed to work on numbers. So we must convert our text input to numbers in a manner such that the algorithm can interpret it and we also maintain the sequence of the text as it is integral to the interpretation of the text as a whole. \n\nHere is a brief [article](https://towardsdatascience.com/text-encoding-a-review-7c929514cccf) to read on encoding.","metadata":{"trusted":true}},{"cell_type":"code","source":"tokenizer = AutoTokenizer.from_pretrained(roberta_string)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's take a look at the first two comments and their encoded values","metadata":{}},{"cell_type":"code","source":"print('Comment 1: - ' + '\\n\\n' + \n      train_sample.comment_text.values[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Comment 2: - ' + '\\n\\n' + \n      train_sample.comment_text.values[1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below is an array which contains the encoded versions of both these comments","metadata":{"trusted":true}},{"cell_type":"code","source":"sample_encoded = tokenizer.batch_encode_plus(train_sample.comment_text.values[0:2]\n                                    , return_attention_masks=False\n                                   , return_token_type_ids=False\n                                   , pad_to_max_length=True\n                                   , max_length = 512)\nsample_encoded","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The result is a dictionary with a single key because we disabled the rest by setting the return parameters to False. The value of this key is a list of 2 lists. One for each comment in our input.","metadata":{"trusted":true}},{"cell_type":"markdown","source":"We see that each comment is now a sequence of numeric values followed by a long list of 1's. These numbers are all actually indices in the transformer's vocabulary. The 1 actually represents '0' in the dictionary and the reason of the long trailing zeroes is something called 'zero-padding' which is done to set all input text to an equal size as the model requires the input to be so. This size is the 'max_length' value of 512 we set in the tokenizer. No string can be larger than this length.","metadata":{}},{"cell_type":"markdown","source":"We only need the numeric values but get a dictionary, let's create a function the encodes, and returns the values in our desired format as we need to repeat this a few times.","metadata":{}},{"cell_type":"code","source":"def encode(text, max_len = 512):\n    encoded_dict = tokenizer.batch_encode_plus(text\n                               , return_attention_masks=False\n                               , return_token_type_ids=False\n                               , pad_to_max_length=True\n                               , max_length = max_len)\n    return np.array(encoded_dict['input_ids'])\n\nMAX_LEN = 192","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's setup the TPU config as the model training operations can get compute intensive","metadata":{}},{"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n# Configuration\nEPOCHS = 2\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us continue with the encoding","metadata":{}},{"cell_type":"code","source":"# We split up the datasets into X and y as we will train the model to predict target y's using feature sets X's\nX_train = encode(train_data.comment_text.values, MAX_LEN)\nX_valid = encode(validation_data.comment_text.values, MAX_LEN)\nX_test = encode(test_data.content.values, MAX_LEN)\n\n# target datasets don't need to be encoded since these are toxicity flag values of 0 and 1 for each comment\ny_train = train_data.toxic.values\ny_valid = validation_data.toxic.values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((X_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((X_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(X_test)\n    .batch(BATCH_SIZE)\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = TFAutoModel.from_pretrained(roberta_string)\n    \n    input_word_ids = Input(shape=(MAX_LEN,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer_layer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n    \nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_steps = X_train.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_steps = X_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    epochs=EPOCHS\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')\nsubmission['toxic'] = model.predict(test_dataset, verbose = 1)\nsubmission.to_csv('/kaggle/working/submission.csv', index = False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}