{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Intoduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Being anonymous over the internet can sometimes make people say nasty things that they normally would not in real life. Let's filter out the hate from our platforms one comment at a time.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Objective\n ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"To create an EDA/ feature-engineering starter notebook for toxic comment classification.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Overview","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The dataset here is from wiki corpus dataset which was rated by human raters for toxicity. The corpus contains 63M comments from discussions relating to user pages and articles dating from 2004-2015.\n\nDifferent platforms/sites can have different standards for their toxic screening process. Hence the comments are tagged in the following five categories","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**EDA**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> I will visulaize and analysis the comments in the training data before moving on to the modeling section.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Install and import necessary packages**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade pyicu","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q polyglot\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade Word2Vec","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install  --upgrade smart-open","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade utils","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade googletrans","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade pycld2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os\nimport gc\nimport re\nimport folium\nfrom scipy import stats\nfrom colorama import Fore, Back, Style, init\nimport string\nimport math\nimport numpy as np\nimport scipy as sp\nimport pandas as pd\nimport scipy.stats as ss\n\nimport random\nimport networkx as nx\nfrom pandas import Timestamp\n\nfrom PIL import Image\nfrom IPython.display import SVG\nfrom keras.utils import model_to_dot\nimport string\nimport requests\nfrom IPython.display import HTML\n\nimport seaborn as sns\nfrom tqdm import tqdm\nimport matplotlib.cm as cm\nimport matplotlib.pyplot as plt\n\ntqdm.pandas()\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport transformers\nimport tensorflow as tf\n\nfrom tensorflow.keras.callbacks import Callback\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, CSVLogger\n\nfrom tensorflow.keras.models import Model\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow.keras.optimizers import Adam\nfrom tokenizers import BertWordPieceTokenizer\nfrom tensorflow.keras.layers import Dense, Input, Dropout, Embedding\nfrom tensorflow.keras.layers import LSTM, GRU, Conv1D, SpatialDropout1D\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras import activations\nfrom tensorflow.keras import constraints\nfrom tensorflow.keras import initializers\nfrom tensorflow.keras import regularizers\n\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.optimizers import *\nfrom tensorflow.keras.activations import *\nfrom tensorflow.keras.constraints import *\nfrom tensorflow.keras.initializers import *\nfrom tensorflow.keras.regularizers import *\n\n\n\nfrom sklearn import metrics\nfrom sklearn.utils import shuffle\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn.base import BaseEstimator,ClassifierMixin\nfrom sklearn.metrics import log_loss\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer,HashingVectorizer\n\nfrom nltk.stem.wordnet import WordNetLemmatizer \nfrom nltk.tokenize import word_tokenize\nfrom nltk.tokenize import TweetTokenizer  \n\nimport nltk\nfrom textblob import TextBlob\nfrom nltk.corpus import stopwords\nfrom nltk.corpus import wordnet\nfrom nltk.corpus import stopwords\nfrom googletrans import Translator\nfrom nltk import WordNetLemmatizer\nfrom polyglot.detect import Detector\nfrom nltk.stem import WordNetLemmatizer\nfrom wordcloud import WordCloud, STOPWORDS\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\nimport time\nstopword=set(STOPWORDS)\n#settings\nstart_time=time.time()\ncolor = sns.color_palette()\nsns.set_style(\"dark\")\neng_stopwords = set(stopwords.words(\"english\"))\nwarnings.filterwarnings(\"ignore\")\n\nlem = WordNetLemmatizer()\ntokenizer=TweetTokenizer()\n\nlem = WordNetLemmatizer()\ntokenizer=TweetTokenizer()\n\nnp.random.seed(0)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid1=pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-test-translated/jigsaw_miltilingual_valid_translated.csv')\nvalid1.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"validate=pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation-processed-seqlen128.csv')\nvalidate.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"comment=pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train-processed-seqlen128.csv\")\ndisplay(comment.head())\nprint(\"Shape:\",comment.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation=pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv\")\ndisplay(validation.head())\nprint('shape:',validation.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_processed=pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test-processed-seqlen128.csv\")\ndisplay(test_processed.head())\nprint('shape:',test_processed.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train=pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ndisplay(train.head())\nprint(\"Shape:\",train.shape)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test=pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\ndisplay(test.head())\nprint(\"shape:\",test.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test['lang'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(test['lang'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_bias=pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train-processed-seqlen128.csv\")\ndisplay(train_bias.head())\nprint(\"shape:\",train_bias.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_bias.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission=pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv\")\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nrow_train=train.shape[0]\nnrow_test=test.shape[0]\nsum=nrow_train+nrow_test\nprint(\"       : train : test\")\nprint(\"rows   :\",nrow_train,\":\",nrow_test)\nprint(\"perc   :\",round(nrow_train*100/sum),\"   :\",round(nrow_test*100/sum))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is 70:30 train:test split and test might be changed in future .\nbut in this case, train:test 78:22.it indicates that class is balanced in dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x=train.iloc[:,2:].sum()\n#marking comments without any tags as \"clean\"\nrowsums=train.iloc[:,2:].sum(axis=1)\ntrain['clean']=(rowsums==0)\n#count number of clean entries\ntrain['clean'].sum()\nprint(\"Total comments = \",len(train))\nprint(\"Total clean comments = \",train['clean'].sum())\nprint(\"Total tags =\",x.sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Check for missing values in Train dataset\")\nnull_check=train.isnull().sum()\nprint(null_check)\n\nprint(\"filling NA with \\\"unknown\\\"\")\ntrain[\"comment_text\"].fillna(\"unknown\", inplace=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Check for missing values in Test dataset\")\nnull_check=test.isnull().sum()\nprint(null_check)\nprint(\"filling NA with \\\"unknown\\\"\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_cols = ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']\ntrain['none'] = 1-train[label_cols].max(axis=1)\ntrain.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Example of Commen**ts:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"toxic:\")\nprint(train[train.severe_toxic==1].iloc[3,1])\n#print(train[train.severe_toxic==1].iloc[5,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"severe_toxic:\")\nprint(train[train.severe_toxic==1].iloc[4,1])\n#print(train[train.severe_toxic==1].iloc[4,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Threat:\")\nprint(train[train.threat==1].iloc[1,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Obscene:\")\nprint(train[train.obscene==1].iloc[1,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"identity_hate:\")\nprint(train[train.identity_hate==1].iloc[4,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x=train.iloc[:,2:].sum()\n#plot\nplt.figure(figsize=(8,4))\nax= sns.barplot(x.index, x.values, alpha=0.8)\nplt.title(\"# per class\")\nplt.ylabel('# of Occurrences', fontsize=12)\nplt.xlabel('Type ', fontsize=12)\n#adding the text labels\nrects = ax.patches\nlabels = x.values\nfor rect, label in zip(rects, labels):\n    height = rect.get_height()\n    ax.text(rect.get_x() + rect.get_width()/2, height + 5, label, ha='center', va='bottom')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x=rowsums.value_counts()\n\n#plot\nplt.figure(figsize=(8,4))\nax = sns.barplot(x.index, x.values, alpha=0.8)\nplt.title(\"Multiple tags per comment\")\nplt.ylabel('# of Occurrences', fontsize=12)\nplt.xlabel('# of tags ', fontsize=12)\n\n#adding the text labels\nrects = ax.patches\nlabels = x.values\nfor rect, label in zip(rects, labels):\n    height = rect.get_height()\n    ax.text(rect.get_x() + rect.get_width()/2, height + 5, label, ha='center', va='bottom')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure(data=[\n    go.Pie(labels=train.columns[2:7],\n           values=train.iloc[:, 2:7].sum().values, marker=dict(colors=px.colors.qualitative.Plotly))\n])\nfig.update_traces(textposition='outside', textfont=dict(color=\"black\"))\nfig.update_layout(title_text=\"Pie chart of labels\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The toxicity is  highly spread out across classes.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df=train.iloc[:,2:-1]\n# filter temp by removing clean comments\n# temp_df=temp_df[~train.clean]\n\ncorr=temp_df.corr()\nplt.figure(figsize=(10,8))\nsns.heatmap(corr,\n            xticklabels=corr.columns.values,\n            yticklabels=corr.columns.values, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://pandas.pydata.org/pandas-docs/stable/style.html\ndef highlight_min(data, color='orange'):\n    '''\n    highlight the maximum in a Series or DataFrame\n    '''\n    attr = 'background-color: {}'.format(color)\n    if data.ndim == 1:  # Series from .apply(axis=0) or axis=1\n        is_min = data == data.min()\n        return [attr if v else '' for v in is_min]\n    else:  # from .apply(axis=None)\n        is_max = data == data.min().min()\n        return pd.DataFrame(np.where(is_min, attr, ''),index=data.index, columns=data.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Crosstab\n# Since technically a crosstab between all 6 classes is impossible to vizualize, lets take a \n# look at toxic with other tags\nmain_col=\"toxic\"\ncorr_mats=[]\nfor other_col in temp_df.columns[1:]:\n    confusion_matrix = pd.crosstab(temp_df[main_col], temp_df[other_col])\n    corr_mats.append(confusion_matrix)\nout = pd.concat(corr_mats,axis=1,keys=temp_df.columns[1:])\n\n#cell highlighting\nout = out.style.apply(highlight_min,axis=0)\nout","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above table represents the Crosstab/ consufion matix of Toxic comments with the other classes.\n\nSome interesting observations:\n\nA Severe toxic comment is always toxic\n\nOther classes seem to be a subset of toxic barring a few exceptions","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Wordcloud of all comments**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def nonan(x):\n    if type(x)==str:\n        return x.replace(\"\\n\",\"\")\n    else:\n        return \"\"\ntext=''.join([nonan(abstract)for abstract in train[\"comment_text\"]])\nwordcloud=WordCloud(max_font_size=None,background_color=\"black\",collocations=False,width=1240,height=800).generate(text)\nfig=px.imshow(wordcloud)\nfig.update_layout(title=\"Common words in comments\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the wordcloud above, we can see the most common words in the comments. These words include \"wikipedia\", \"page\",\"Thank\", and \"article\" among other words. More offensive words like \"f**k\" seem to occur less often, indicating that toxic, insulting comments are seen less frequently than non-toxic comments","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Wordclouds-Frequent words:**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now,let's take a look a t words that are associated with these classes.\n\nChart Desc:The visuals here are word clouds (ie0 more frequent words appear bigger\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"A simpler way would be to create a new kaggle dataset and import images from there.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!du -l ../input/*","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/imagesforkernal/\nstopword=set(STOPWORDS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#clean comments\nclean_mask=np.array(Image.open(\"../input/imagesforkernal/safe-zone.png\"))\nclean_mask=clean_mask[:,:,1]\n#wordcloud for clean comments\nsubset=train[train.clean==True]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=clean_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(20,10))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Clean Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"toxic_mask=np.array(Image.open(\"../input/imagesforkernal/toxic-sign.png\"))\ntoxic_mask=toxic_mask[:,:,1]\n#wordcloud for clean comments\nsubset=train[train.toxic==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=4000,mask=toxic_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(20,20))\nplt.subplot(221)\nplt.axis(\"off\")\nplt.title(\"Words frequented in Toxic Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'gist_earth' , random_state=244), alpha=0.98)\n\n#Severely toxic comments\nplt.subplot(222)\nsevere_toxic_mask=np.array(Image.open(\"../input/imagesforkernal/bomb.png\"))\nsevere_toxic_mask=severe_toxic_mask[:,:,1]\nsubset=train[train.severe_toxic==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=severe_toxic_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Severe Toxic Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Reds' , random_state=244), alpha=0.98)\n\n#Threat comments\nplt.subplot(223)\nthreat_mask=np.array(Image.open(\"../input/imagesforkernal/anger.png\"))\nthreat_mask=threat_mask[:,:,1]\nsubset=train[train.threat==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=threat_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Threatening Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'summer' , random_state=2534), alpha=0.98)\n\n#insult\nplt.subplot(224)\ninsult_mask=np.array(Image.open(\"../input/imagesforkernal/swords.png\"))\ninsult_mask=insult_mask[:,:,\n                        1]\nsubset=train[train.insult==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=insult_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in insult Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Paired_r' , random_state=244), alpha=0.98)\n\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#identity_hate\nsubset=train[train.toxic==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=4000,mask=toxic_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.subplot(111)\nidentity_hate=np.array(Image.open(\"../input/imagesforkernal/megaphone.png\"))\nidentity_hate=identity_hate[:,:,1]\nsubset=train[train.identity_hate==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=identity_hate,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in identity_hate Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Paired_r' , random_state=244), alpha=0.98)\n#obsence\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#obsence\nplt.subplot(122)\nobscene=np.array(Image.open(\"../input/imagesforkernal/biohazard-symbol.png\"))\nobscene=obscene[:,:,1]\nsubset=train[train.obscene==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=obscene,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in obscene Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Paired_r' , random_state=244), alpha=0.98)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Languages**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now,We will go for analysis of language in the dataset and detect the language present in the comments.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_language(text):\n    return Detector(\"\".join(x for x in text if x.isprintable()), quiet=True).languages[0].name\n\ntrain[\"lang\"] = train[\"comment_text\"].progress_apply(get_language)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**English Vs Non English**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lang_list=sorted(list(set(train[\"lang\"])))\ncounts=[list(train[\"lang\"]).count(cont) for cont in lang_list]\ndf=pd.DataFrame(np.transpose([lang_list,counts]))\ndf.columns=[\"Language\",\"Count\"]\ndf[\"Count\"]=df[\"Count\"].apply(int)\n\ndisplay(df[\"Language\"].unique())\ndisplay(sns.countplot(df[\"Language\"]))\n\nfig=px.bar(df,x=\"Language\",y=\"Count\",title=\"Language of comments\",color='Language',text=\"Count\",width=700)\ndisplay(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df[\"Count\"].sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.bar(df.query(\" Language!='un' & Language != 'en'\").query(\"Count >= 50\"),y=\"Language\", x=\"Count\", title=\"Language of non-English comments\",template=\"plotly_white\", color=\"Language\", text=\"Count\", orientation=\"h\")\nfig.update_traces(marker=dict(line=dict(width=0.75,color='black')),  textposition=\"outside\")\nfig.update_layout(showlegend=False)\nfig","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see in bar chart .German(de),Scots(sc) and Danish(da) are the most common non-english languages featuring in the dataset,with more than 100 comments in each language.remaining non english language are not far behind.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure([go.Pie(labels=df.query(\"Language!='un' & Language != 'en'\").query(\"Count >= 50\")[\"Language\"],\nvalues=df.query(\" Language!='un' & Language !='en'\").query(\"Count >= 50\")[\"Count\"])])\nfig.update_layout(title_text=\"Pie chart of non-English languages\", template=\"plotly_white\")\nfig.data[0].marker.colors = [px.colors.qualitative.Plotly[2:]]\nfig.data[0].textfont.color = \"black\"\nfig.data[0].textposition = \"outside\"\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"from the Pie chart above, we can once see that German,Danish ,and Scots with more than 15% of the pie belonging to each of these three languages.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_country(Language):\n    if Language==\"de\":\n        return \"Germany\"\n    if Language==\"sco\":\n        return \"Scotland\"\n    if Language==\"da\":\n        return \"Denmark\"\n    if Language==\"ar\":\n        return \"Saudi Arabia\"\n    if Language==\"es\":\n        return \"Spain\"\n    if Language==\"fa\":\n        return \"Iran\"\n    if Language==\"el\":\n        return \"Greece\"\n    if Language==\"pt\":\n        return \"Portugal\"\n    \n    if Language==\"en\":\n        return \"United Kingdom\"\n    if Language==\"ht\":\n        return \"India\"\n    if Language==\"aa\":\n        return \"Albania\"\n    if Language==\"bn\":\n        return \"Bosnia and Herzegovina\"\n    if Language==\"crs\":\n        return \"Croatia\"\n    if Language==\"de\":\n        return\"Netherlands\"\n    if Language==\"sr\":\n        return \"Russia\"\n    if Language=='vi':\n        return \"Vietnam\"\n    if Language=='sm':\n        return \"Somalia\"\n    if Language==\"sr\":\n        return \"Serbia\"\n    if Language==\"ie\":\n        return \"Indonesia\"\n    if Language==\"mk\":\n        return \"Ireland\"\n    if Language==\"iv\":\n        return \"Holy See (Vatican City State)\"\n    if Language==\"af\":\n        return \"South Africa\"\n    return \"None\"\ndf[\"country\"]=df[\"Language\"].progress_apply(get_country)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**World plot of non-English languages**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfig = px.choropleth(df.query(\"Language != 'en'& Language != 'un' & country != 'None'\").query(\"Count >= 5\"), locations=\"country\", hover_name=\"country\",projection=\"natural earth\", locationmode=\"country names\", title=\"Countries of non-English languages\", color=\"Count\",template=\"plotly\", color_continuous_scale=\"agsunset\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"from the world plot above .we can see that western Europe and the middle-east are the most represented regions in the dataset .Africa,Asia and eastern Europe are relatively under-represented.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Non-English European**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'en' & Language != 'un' & country != 'None'\"), locations=\"country\", hover_name=\"country\", projection=\"natural earth\", locationmode=\"country names\", title=\"Non-English European countries\", color=\"Count\",template=\"plotly\", color_continuous_scale=\"aggrnyl\", scope=\"europe\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that German and English are the most common European languages to feature in the dataset,although Spanish and Greek are not far behind.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'en' & Language != 'un' \"), locations=\"country\", hover_name=\"country\",\n                     projection=\"natural earth\", locationmode=\"country names\", title=\"Asian countries\", color=\"Count\",\n                     template=\"plotly\", color_continuous_scale=\"spectral\", scope=\"asia\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This plot shows that middle-eastern languages such as Arabic and Persian are represented more than languages from the Indian Subcontinent or south-east Asia,such as Hindi,Vietnamese and Indonesian.There is not a single Comment In amndarin,Korean or Japanese!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'English' & Language != 'un' & country != 'None'\").query(\"Count >= 5\"), locations=\"country\", hover_name=\"country\",\n                     projection=\"natural earth\", locationmode=\"country names\", title=\"African countries\", color=\"Count\",\n                     template=\"plotly\", color_continuous_scale=\"agsunset\", scope=\"africa\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that Africa is not as well as other continents in the dataset.The two most common African languages in the dataset are Afrikans and Somali","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Feature engineering","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I've broadly classified my feature engineering ideas into the following three groups.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Direct Features**\n\nfeatures which are a directly due to word/content .We would exploring the following techniques.\n* Word frequency features\n    1. Count features\n    1. Bigrams\n    1. Trigrams\n* Vector distance mapping of words(Word2Vec)\n* Sentiment scores\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Indirect features**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Some more experimental features.\n* count of sentences\n* count of words\n* count of unique words\n* count of letters\n* count of punctuations\n* count of uppercase words/letters\n* count of stop words\n* Avg length of each word\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Leaky features**\nfrom the example ,we know that the comments contain identifier information (eg:IP,username etc.).We can create features out of them but ,it will certainly lead to overfitting to this specific wikipedia use-case.\n* toxic IP scores\n* toxic users\n**Note:**Creating the indirect and leaky features first .There are two reasons for this,\n* Count features(Direct features)are usefu; only if they are created from a clean corpus\n* Also the indirect features help compensate for the loss of information when cleaning the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"merge=pd.concat([train.iloc[:,0:2],test.iloc[:,0:2]])\ndf=merge.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Indirect features\n# sentence count in each comment:\n#'\\n 'can be used to count the number of sentences in each comment\ndf['count_sent']=df[\"comment_text\"].apply(lambda x:len(re.findall(\"\\n\",str(x)))+1)\n#word count in each comment\ndf['count_word']=df[\"comment_text\"].apply(lambda x:len(str(x).split()))\n#Unique word count\ndf[\"count_unique_word\"]=df[\"comment_text\"].apply(lambda x:len(set(str(x).split())))\n#Letter count\ndf['count_letters']=df[\"comment_text\"].apply(lambda x:len(str(x)))\n#punctuation count\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#upper case words count\n\ndf[\"count_words_upper\"] = df[\"comment_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\n\ndf[\"count_word_title\"]=df[\"comment_text\"].apply(lambda x:len([w for w in str(x).split() if w.istitle()]))\n#Number of stopwords\ndf[\"count_stopwords\"]=df[\"comment_text\"].apply(lambda x:len([w for w in str(x).lower().split() if w in stopword])) \n#Average length of the words\ndf[\"mean_word_len\"]=df[\"comment_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#derived features\n#Word count percent in each comment:\ndf['word_unique_percent']=df['count_unique_word']*100/df['count_word']\n#derived features\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#serperate train and test features\ntrain_feats=df.iloc[0:len(train),]\ntest_feats=df.iloc[len(train):,]\n#join the tags\ntrain_tags=train.iloc[:,2:]\ntrain_feats=pd.concat([train_feats,train_tags],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_feats['count_sent'].loc[train_feats['count_sent']>10] = 10 \nplt.figure(figsize=(12,6))\n## sentenses\nplt.subplot(121)\nplt.suptitle(\"Are longer comments more toxic?\",fontsize=20)\nsns.violinplot(y='count_sent',x='clean', data=train_feats,split=True)\nplt.xlabel('Clean?', fontsize=12)\nplt.ylabel('# of sentences', fontsize=12)\nplt.title(\"Number of sentences in each comment\", fontsize=15)\n# words\ntrain_feats['count_word'].loc[train_feats['count_word']>200] = 200\nplt.subplot(122)\nsns.violinplot(y='count_word',x='clean', data=train_feats,split=True,inner=\"quart\")\nplt.xlabel('Clean?', fontsize=12)\nplt.ylabel('# of words', fontsize=12)\nplt.title(\"Number of words in each comment\", fontsize=15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Long sentences or more words do not seem to be a significant indicator of toxicity.\n\nChart desc:Violin plot is an alternative to the traditional box plot.The inner markings show the percentiles whi;e the width of the \"violin\" shows the volume of comments at that level","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_feats['count_unique_word'].loc[train_feats['count_unique_word']>200] = 200\n#prep for split violin plots\n#For the desired plots , the data must be in long format\ntemp_df = pd.melt(train_feats, value_vars=['count_word', 'count_unique_word'], id_vars='clean')\n#spammers - comments with less than 40% unique words\nspammers=train_feats[train_feats['word_unique_percent']<30]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.gridspec as gridspec \nplt.figure(figsize=(16,12))\nplt.suptitle(\"What's so unique ?\",fontsize=20)\ngridspec.GridSpec(2,2)\nplt.subplot2grid((2,2),(0,0))\nsns.violinplot(x='variable', y='value', hue='clean', data=temp_df,split=True,inner='quartile')\nplt.title(\"Absolute wordcount and unique words count\")\nplt.xlabel('Feature', fontsize=12)\nplt.ylabel('Count', fontsize=12)\n\nplt.subplot2grid((2,2),(0,1))\nplt.title(\"Percentage of unique words of total words in comment\")\n#sns.boxplot(x='clean', y='word_unique_percent', data=train_feats)\nax=sns.kdeplot(train_feats[train_feats.clean == 0].word_unique_percent, label=\"Bad\",shade=True,color='r')\nax=sns.kdeplot(train_feats[train_feats.clean == 1].word_unique_percent, label=\"Clean\")\nplt.legend()\nplt.ylabel('Number of occurances', fontsize=12)\nplt.xlabel('Percent unique words', fontsize=12)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Corpus cleaning**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Its important to use a clean dataset before creating count features.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"corpus=merge.comment_text\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://drive.google.com/file/d/0B1yuv8YaUVlZZ1RzMFJmc1ZsQmM/view\n# Aphost lookup dict\nAPPO = {\n\"aren't\" : \"are not\",\n\"can't\" : \"cannot\",\n\"couldn't\" : \"could not\",\n\"didn't\" : \"did not\",\n\"doesn't\" : \"does not\",\n\"don't\" : \"do not\",\n\"hadn't\" : \"had not\",\n\"hasn't\" : \"has not\",\n\"haven't\" : \"have not\",\n\"he'd\" : \"he would\",\n\"he'll\" : \"he will\",\n\"he's\" : \"he is\",\n\"i'd\" : \"I would\",\n\"i'd\" : \"I had\",\n\"i'll\" : \"I will\",\n\"i'm\" : \"I am\",\n\"isn't\" : \"is not\",\n\"it's\" : \"it is\",\n\"it'll\":\"it will\",\n\"i've\" : \"I have\",\n\"let's\" : \"let us\",\n\"mightn't\" : \"might not\",\n\"mustn't\" : \"must not\",\n\"shan't\" : \"shall not\",\n\"she'd\" : \"she would\",\n\"she'll\" : \"she will\",\n\"she's\" : \"she is\",\n\"shouldn't\" : \"should not\",\n\"that's\" : \"that is\",\n\"there's\" : \"there is\",\n\"they'd\" : \"they would\",\n\"they'll\" : \"they will\",\n\"they're\" : \"they are\",\n\"they've\" : \"they have\",\n\"we'd\" : \"we would\",\n\"we're\" : \"we are\",\n\"weren't\" : \"were not\",\n\"we've\" : \"we have\",\n\"what'll\" : \"what will\",\n\"what're\" : \"what are\",\n\"what's\" : \"what is\",\n\"what've\" : \"what have\",\n\"where's\" : \"where is\",\n\"who'd\" : \"who would\",\n\"who'll\" : \"who will\",\n\"who're\" : \"who are\",\n\"who's\" : \"who is\",\n\"who've\" : \"who have\",\n\"won't\" : \"will not\",\n\"wouldn't\" : \"would not\",\n\"you'd\" : \"you would\",\n\"you'll\" : \"you will\",\n\"you're\" : \"you are\",\n\"you've\" : \"you have\",\n\"'re\": \" are\",\n\"wasn't\": \"was not\",\n\"we'll\":\" will\",\n\"didn't\": \"did not\",\n\"tryin'\":\"trying\"\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def clean(comment):\n    \"\"\"\n    This function receives comments and returns clean word-list\n    \"\"\"\n    #Convert to lower case , so that Hi and hi are the same\n    \n    \n    comment=re.sub(\"\\\\n\",\"\",comment)\n    # remove leaky elements like ip,user\n    comment=re.sub(\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\",\"\",comment)\n    #removing usernames\n    comment=re.sub(\"\\[\\[.*\\]\",\"\",comment)\n    \n    #Split the sentences into words\n    words=tokenizer.tokenize(comment)\n    \n    # (')aphostophe  replacement (ie)   you're --> you are  \n    # ( basic dictionary lookup : master dictionary present in a hidden block of code)\n    words=[APPO[word] if word in APPO else word for word in words]\n    words=[lem.lemmatize(word, \"v\") for word in words]\n    words = [w for w in words if not w in eng_stopwords]\n    \n    clean_sent=\" \".join(words)\n    # remove any non alphanum,digit character\n    #clean_sent=re.sub(\"\\W+\",\" \",clean_sent)\n    #clean_sent=re.sub(\"  \",\" \",clean_sent)\n    return(clean_sent)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corpus.iloc[12235]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clean(corpus.iloc[12235])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Building Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n    tokenizer.enable_truncation(max_length=maxlen)\n    tokenizer.enable_padding(max_length=maxlen)\n    all_ids = []\n    \n    for i in tqdm(range(0, len(texts), chunk_size)):\n        text_chunk = texts[i:i+chunk_size].tolist()\n        encs = tokenizer.encode_batch(text_chunk)\n        all_ids.extend([enc.ids for enc in encs])\n    \n    return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def regular_encode(texts,tokenizer,maxlen=512):\n    enc_di=tokenizer.batch_encode_plus(texts,return_attention_masks=False,return_token_type_ids=False,pad_to_max_length=True,max_langth=maxlen)\n    return np.array(enc_di['input_ids'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, loss='binary_crossentropy', max_len=512):\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    x = tf.keras.layers.Dropout(0.3)(cls_token)\n    out = Dense(1, activation='sigmoid')(x)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss=loss, metrics=[tf.keras.metrics.AUC()])\n    \n    return model\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU Configs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport transformers\nfrom transformers import TFAutoModel, AutoTokenizer\nfrom tqdm.notebook import tqdm\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n\n\n# Configuration\nEPOCHS = 2\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nMAX_LEN = 192\nMODEL = 'jplu/tf-xlm-roberta-large'\n#GCS_DS_PATH = KaggleDatasets().get_gcs_path('kaggle/input/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Create fast tokenizer**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = AutoTokenizer.from_pretrained(MODEL)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\ntrain2.toxic=train2.toxic.round().astype(int)\nvalid = pd.read_csv('/kaggle/input/val-en-df/validation_en.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Combine train1 with a subset of train2\ntrain = pd.concat([\n    train[['comment_text', 'toxic']],\n    train2[['comment_text', 'toxic']].query('toxic==1'),\n    train2[['comment_text', 'toxic']].query('toxic==0').sample(n=100000, random_state=0)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n\nx_train = regular_encode(train.comment_text.values, tokenizer, maxlen=MAX_LEN)\nx_valid = regular_encode(valid.comment_text.values, tokenizer, maxlen=MAX_LEN)\nx_test = regular_encode(test.content.values, tokenizer, maxlen=MAX_LEN)\n\ny_train = train.toxic.values\ny_valid = valid.toxic.values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = [(\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"****","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = transformers.TFBertModel.from_pretrained('bert-base-uncased')\n    model = build_model(transformer_layer, loss=focal_loss(gamma=1.5), max_len=512)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Train Model**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First,we train in the subset of taining set,which is completely in English.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = x_train.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have pretty much saturated the learning potential of the model on english only data,we train it one more epoch on the validation set,which is significantly smaller but contains a mixture of diffferent languages.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = x_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally, create the submission file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['toxic'] = model.predict(test_dataset, verbose=1)\nsub.to_csv('submission.csv', index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"* # Please UPVOTE me and comments for any suggestion regarding this notebook","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}