{"cells":[{"metadata":{},"cell_type":"markdown","source":"Over the past year, the field has seen impressive multilingual capabilities from the latest model innovations, including few- and zero-shot learning. We're excited to learn whether these results \"translate\" (pun intended!) to toxicity classification. Our training data will be the English data provided for our previous two competitions and your test data will be Wikipedia talk page comments in several different languages.\nWe will be using DistilBERT as it is 2 times faster and 25% lighter than multilingual BERT base, all while retaining 92% of its performance. This model let you quickly experiments with different ideas, and when you are ready for the real thing, just change two lines of code to use bert-base-multilingual-cased.\n* You can try other models like BERT large,Xlnet,RoBERTa,etc and compare the performances.\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Importing all the necessory Libraries","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install -q googletrans\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nimport re\n\nimport transformers\nimport tensorflow as tf\nfrom tqdm.notebook import tqdm\nfrom wordcloud import WordCloud, STOPWORDS\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nfrom tokenizers import BertWordPieceTokenizer\n\n\nsns.set(style=\"darkgrid\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get the training datas of both the previous competetions and concatenate to be used as a single training dataset. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dir = '/kaggle/input/jigsaw-multilingual-toxic-comment-classification'\n\ntrain_set1 = pd.read_csv(os.path.join(dir, 'jigsaw-toxic-comment-train.csv'))\ntrain_set2 = pd.read_csv(os.path.join(dir, 'jigsaw-unintended-bias-train.csv'))\ntrain_set2.toxic = train_set2.toxic.round().astype(int)\n\nvalid = pd.read_csv(os.path.join(dir, 'validation.csv'))\ntest = pd.read_csv(os.path.join(dir, 'test.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.concat([\n    train_set1[['comment_text', 'toxic']],\n    train_set2[['comment_text', 'toxic']].query('toxic==1'),\n    train_set2[['comment_text', 'toxic']].query('toxic==0').sample(n=100000, random_state=0)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(valid.shape)\nvalid.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid.lang.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.lang.unique()\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA on textual data and toxic distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.toxic.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train.toxic)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nrow_train=train.shape[0]\nnrow_test=test.shape[0]\nsum=nrow_train+nrow_test\nprint(\"       : train : test\")\nprint(\"rows   :\",nrow_train,\":\",nrow_test)\nprint(\"perc   :\",round(nrow_train*100/sum),\"   :\",round(nrow_test*100/sum))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x=train.iloc[:,2:].sum()\n#marking comments without any tags as \"clean\"\nrowsums=train.iloc[:,2:].sum(axis=1)\ntrain['clean']=(rowsums==0)\n#count number of clean entries\ntrain['clean'].sum()\nprint(\"Total comments = \",len(train))\nprint(\"Total clean comments = \",train['clean'].sum())\nprint(\"Total tags =\",x.sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Check for missing values in Train dataset\")\nnull_check=train.isnull().sum()\nprint(null_check)\nprint(\"Check for missing values in Test dataset\")\nnull_check=test.isnull().sum()\nprint(null_check)\nprint(\"filling NA with \\\"unknown\\\"\")\ntrain[\"comment_text\"].fillna(\"unknown\", inplace=True)\n# test[\"comment_text\"].fillna(\"unknown\", inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#plot\nplt.figure(figsize=(8,4))\nax= sns.countplot(valid.lang, alpha=0.8)\nplt.title(\"# per class\")\nplt.ylabel('# of Occurrences', fontsize=12)\nplt.xlabel('Type of Language', fontsize=12)\n#adding the text labels\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#plot\nplt.figure(figsize=(8,4))\nax= sns.countplot(test.lang,alpha=0.8)\nplt.title(\"# per class\")\nplt.ylabel('# of Occurrences', fontsize=12)\nplt.xlabel('Type of Language', fontsize=12)\n#adding the text labels\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#plot\nplt.figure(figsize=(8,4))\nax= sns.countplot(valid.toxic, alpha=0.8)\nplt.title(\"# per class\")\nplt.ylabel('# of Occurrences', fontsize=12)\nplt.xlabel('Type ', fontsize=12)\n#adding the text labels\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_ax(rows = 1,cols = 2,size = 7):\n    fig, ax = plt.subplots(rows, cols, figsize=(size*cols, size*rows))\n    return fig,ax","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = get_ax()\nsns.distplot(train[train[\"toxic\"]==0][\"comment_text\"].str.len(),ax = ax[0])\nsns.distplot(train[train[\"toxic\"]==1][\"comment_text\"].str.len(),ax = ax[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def wordcloud(data):\n    wordcloud = WordCloud(background_color = 'Black',\n                         max_words = 50,\n                         max_font_size = 40,\n                         scale = 5,\n                         random_state = 5).generate(str(data))\n    fig = plt.figure(1, figsize=(10,10))\n    plt.imshow(wordcloud)\n    plt.axis(\"off\")\n    plt.show()\nwordcloud(train[\"comment_text\"])    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wordcloud(valid[\"comment_text\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wordcloud(test[\"content\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Setting a fixed size of encoding i.e tokenizing and padding each input string","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n    \"\"\"\n    https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n    \"\"\"\n    tokenizer.enable_truncation(max_length=maxlen)\n    tokenizer.enable_padding(max_length=maxlen)\n    all_ids = []\n    \n    for i in tqdm(range(0, len(texts), chunk_size)):\n        text_chunk = texts[i:i+chunk_size].tolist()\n        encs = tokenizer.encode_batch(text_chunk)\n        all_ids.extend([enc.ids for enc in encs])\n    \n    return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Building model layers with a input layer with encoded string ,transformer layer for processing and final dense layer to get predictions.Since its a binary classification we are using binary crossentropy.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, max_len=512):\n    \"\"\"\n    https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n    \"\"\"\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Configuring TPU\n* [Read the TPU documentation](http://https://www.kaggle.com/docs/tpu) one-pager","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n# Configuration\nEPOCHS = 3\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nMAX_LEN = 192","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Instancing the tokenizer from DistilBERT model and then applying WordPeice Tokenizer ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = fast_encode(train.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)\nx_valid = fast_encode(valid.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)\nx_test = fast_encode(test.content.astype(str), fast_tokenizer, maxlen=MAX_LEN)\n\ny_train = train.toxic.values\ny_valid = valid.toxic.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Converting data into Tensordata for TPU processing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model initialization and fitting on train and valid sets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = (\n        transformers.TFDistilBertModel\n        .from_pretrained('distilbert-base-multilingual-cased')\n    )\n    model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = x_train.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = x_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    epochs=EPOCHS*2\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"    preds = model.predict(test_dataset,verbose = 1)\n    # # final = pd.DataFrame({\"test_content\":test.content,\"Preds\":preds})\n    # # final.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# sub['toxic'] = model.predict(test_dataset, verbose=1)\n# sub.to_csv('submission.csv', index=False)\nsub = pd.DataFrame(preds,index = [i for i in range(len(preds))])\nsub.to_csv(\"submiss.csv\",index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thank You for visiting this Notebook!!If you like it Please UPVOTE!!It motivates me to learn more.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}