{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/imageseda/live_chat_anim_2.gif\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"It only takes one toxic comment to sour an online discussion.
\nToday the social media has grown to be a necessary tool for communicating on a personal and practical level but this has consciousness In social platforms it’s possible to be hidden behind fake Identity and can't be recognized. As a result, the internet community becomes more comfortable turning any conversation into a negative one because they know that they can getaway easily from the causes of their behavior.
\nOur capstone project focus on a model that can identify the toxicity of the conversation on the internet. Using natural Language Processor, we will experience with three models RNN, Bert and XLM to classify the toxicity of different languages comments like a rude, obscene, insult, threat, identity attack, and normal comments","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Problem Statement","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Nowadays social media companies started to used machine learning technology to help them identify and ban violent extremism to grow more friendly, thriving environment and they will continue to enhance their algorithms from the huge data that already exist in media platform. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Dictionary","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Notebook contents:\n \n1. Import Libraries\n2. Import Datasets\n3. Exploratory Data Analysis: Data Observation
\n 3.1 Search for Null Values
\n 3.2 Find The Unique Values in Train Dataframe
\n 3.3 Display Dataframe General Information
\n 3.4 Print Selective Rows from Toxic Comments
\n 3.5 Print Selective Rows from Non-Toxic Comments\n4. Exploratory Data Analysis: Data Visualization
\n 4.1 Plot Data Distribution for Most Common Words in Train
\n 4.2 Plot Data Distribution for Most Common toxic Words only in Train
\n 4.3 Plot pie charts to show the percentages between toxic types
\n 4.4 The Number of Occurrences of comments in each columns
\n 4.5 plot worldCloud for Training Dataframe
\n5. Data Cleaning
\n 5.1 Clean The Data From Emoji
\n 5.2 Clean The Data From Punctuations
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Import Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom scipy import stats\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\nfrom plotly.offline import init_notebook_mode, iplot \nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\nfrom PIL import Image\nfrom collections import Counter\nimport nltk\nimport emoji\n\nstopwords = set(STOPWORDS)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Import Datasets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_data=pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/validation.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data=pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"testing_data=pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Exploratory Data Analysis: Data Observation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('traning data, validation data, test data ')\ntraining_data.shape, validation_data.shape, testing_data.shape\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_data.head()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"testing_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.1 search for null values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_data.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"testing_data.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"all the three datasets does not contains nulls","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Find The Unique Values in Train Dataframe","execution_count":null},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"# find how many unique values in the training dataframe\ntraining_data.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#validation data has 3 languages non of them english\nvalidation_data.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#vtest data has 6 languages non of them english\ntesting_data.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(validation_data.lang.unique())\n#replace column values with the language name instead of the language code\nvalidation_data.lang.replace('es','Spanish',inplace=True)\nvalidation_data.lang.replace('it','Italian',inplace=True)\nvalidation_data.lang.replace('tr','Turkish',inplace=True)\n\nprint(validation_data.lang.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(testing_data.lang.unique())\n#replace column values with the language name instead of the language code\ntesting_data.lang.replace('es','Spanish',inplace=True)\ntesting_data.lang.replace('it','Italian',inplace=True)\ntesting_data.lang.replace('tr','Turkish',inplace=True)\ntesting_data.lang.replace('ru','Russian',inplace=True)\ntesting_data.lang.replace('fr','French',inplace=True)\ntesting_data.lang.replace('pt','Portuguese',inplace=True)\n\nprint(testing_data.lang.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 Display Dataframe General Information","execution_count":null},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"training_data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 Print Selective Rows from Toxic Comments","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# select toxic comments from data\nprint(\"toxic comments:\")\nprint(training_data[training_data.toxic==1].iloc[10,1],'\\n')\nprint(training_data[training_data.toxic==1].iloc[500,1],'\\n')\nprint(training_data[training_data.toxic==1].iloc[1573,1],'\\n')\nprint(training_data[training_data.toxic==1].iloc[4310,1],'\\n')\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.5 Print Selective Rows from Non-Toxic Comments","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"# select non-toxic comments from data\nprint(\"non-toxic comments:\")\nprint(training_data[training_data.toxic==0].iloc[10,1],'\\n')\nprint(training_data[training_data.toxic==0].iloc[90,1],'\\n')\nprint(training_data[training_data.toxic==0].iloc[210,1],'\\n')\nprint(training_data[training_data.toxic==0].iloc[4311,1],'\\n')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nFrom these comments we can see that the data has varies kind of comments, short, long, upper case, lowercase comment with symbol and later we will see comments with emoji also\nThere're also some comments have really bad words and can be really aggressive,and even some of them attack very aggressively on the person identity ","execution_count":null},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"training_data.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" all the columns have the two values only 0 and 1","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 4. Exploratory Data Analysis:Data visualization","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Plot Data Distribution for Most Common Words in Train ","execution_count":null},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"# find the top words distribution in the training data\n\n#take the column comment_text and split each word in a column\ntoxic_words = training_data['comment_text'].str.split(expand=True).unstack().value_counts()\n\n#plot bar chart with 100 value in x and y\ndata = [go.Bar(x = toxic_words.index.values[:100],y = toxic_words.values[:100],\nmarker= dict(colorscale='Viridis',color = toxic_words.values[:100]),text='Word counts')]\n\nlayout = go.Layout(title='Top 100 Word frequencies in the training dataset without stopword')\n\nfig = go.Figure(data=data, layout=layout)\niplot(fig, filename='Word frequencies bar chart')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We notice from the plot most the word frequancies are the common word. and that's will not help us to understand the concept or what most words repeated in training data.\nTo fix this issue we will use stopwords with the next plot\n","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"#use stopword to remove unessesry common words in english\nstopwords = nltk.corpus.stopwords.words('english')\n\n# RegEx for stopwords\nRE_stopwords = r'\\b(?:{})\\b'.format('|'.join(stopwords))\n\n# replace characters with ' ' and drop all stopwords\nwords = (training_data['comment_text'].str.lower().replace([r\"[\\.\\'\\,\\-\\\"\\?\\()\\==]\", \nRE_stopwords], [' ', ''], regex=True).str.cat(sep=' ').split())\n\n\n# add the new words frequances in dataframe\nrslt = pd.DataFrame(Counter(words).most_common(100),\n columns=['Word', 'Frequency']).set_index('Word')\n\n\n#plot bar chart with the most frequancies words\ndata = [go.Bar(x = rslt.index.values,y = rslt.Frequency.values,\nmarker= dict(colorscale='Viridis',color = rslt.Frequency.values[:100]),text='Word counts')]\n\nlayout = go.Layout(title='Top 100 Word frequencies in the training dataset with stopword')\n\nfig = go.Figure(data=data, layout=layout)\niplot(fig, filename='Word frequencies bar chart')\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Plot Data Distribution for Most Common toxic Words only in Train ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"#after we saw the most common words in all the train data \n#let's plot only the most common toxic words \n\ntoxity_data=training_data[training_data.toxic==1]\n\n#use stopword to remove unessesry common words in english\nstopwords = nltk.corpus.stopwords.words('english')\n\n# RegEx for stopwords\nRE_stopwords = r'\\b(?:{})\\b'.format('|'.join(stopwords))\n\n# replace characters with ' ' and drop all stopwords\nwords = (toxity_data['comment_text'].str.lower().replace([r\"[\\.\\'\\,\\-\\\"\\?\\()\\==\\!]\", \nRE_stopwords], [' ', ''], regex=True).str.cat(sep=' ').split())\n\n\n# add the new words frequances in dataframe\nrslt_T = pd.DataFrame(Counter(words).most_common(100),\n columns=['Word', 'Frequency']).set_index('Word')\n\n\n#plot bar chart with the most frequancies words\ndata = [go.Bar(x = rslt_T.index.values[:25],y = rslt_T.Frequency.values[:25],\nmarker= dict(colorscale='Viridis',color = rslt_T.Frequency.values[:20]),text='Word counts')]\n\nlayout = go.Layout(title='Top 100 Toxic Word frequencies in the training dataset with stopword')\n\nfig = go.Figure(data=data, layout=layout)\niplot(fig, filename='Word frequencies bar chart')\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To find the most common toxic words in the dataset, we used stopword function to remove common words like the, a, I, me, you..., etc. then use the function replace to replace special characters [., ‘, -,? , =] from the texts. These steps will help to plot only the distribution of the toxic words without unnecessary words or characters. In the bar chart, we choose to plot only the most common 25 words from top 100 toxic words. The highest word was repeated 15950 time. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Plot pie charts to show the percentages between toxic types","execution_count":null},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"#plot pie chart to display the toxic words percentage between different toxics words types\n\n#select colors for the pie plot\ncolors=['gold','darkslateblue','mediumturquoise','lightcoral','lightskyblue','lightseagreen']\n\n#plotting pie with 5 columns \nfig = go.Figure(data=[go.Pie(labels=training_data.columns[[3,4,5,6,7]],\nvalues=training_data.iloc[:,[3,4,5,6,7]].sum().values, marker=dict(colors=colors))])\n\n# choose to display the percentage outside the circle with color black\nfig.update_traces(textposition='outside', textfont=dict(color=\"black\"))\nfig.update_layout(title_text=\"toxic comments types\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The toxic word comes in different shapes, and in our data the comments categorize the toxicity based on 5 categories obscene, insult, identity, and severe toxic. From the pie chart the category obscene got the highest percentage of the toxic comments from the data came after that the insult comments.\n\nIn the events we see every day around the world. One can see the racism and bullies that happened to someone is not just because of races like African American or white, but also on religions and the gender of the person. The category identity hate describes this type of toxic comments.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.4 The Number of Occurrences of comments & languages in each columns","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"#select columns for the first row plot to show the Occurrences of toxic words in these columns\nx_data_1=training_data.iloc[:,[3,4,5,6,7]].sum()\n\n#plot\nplt.figure(1,figsize=(24,17))\nplt.subplot(211)\nax= sns.barplot(x_data_1.index, x_data_1.values, alpha=0.8,palette='mako')\nplt.title(\"toxic comment in each column\",fontsize=30)\nplt.ylabel('number of Occurrences', fontsize=23,labelpad=20)\nplt.xlabel('Type of the toxic comment', fontsize=23,labelpad=20)\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#plot the number of Occurrences for the languages in validation data\nplt.figure(1,figsize=(10,6))\nsns.countplot(validation_data.lang,alpha=0.8,palette='mako')\nplt.title(\"Appearance of languages in validation data \",fontsize=20)\nplt.ylabel('number of Occurrences', fontsize=15,labelpad=20)\nplt.xlabel('Languages', fontsize=15,labelpad=20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#plot the number of Occurrences for the languages in validation data\nplt.figure(1,figsize=(10,6))\nsns.countplot(testing_data.lang,alpha=0.8,palette='mako')\nplt.title(\"Appearance of languages in validation data \",fontsize=20)\nplt.ylabel('number of Occurrences', fontsize=15,labelpad=20)\nplt.xlabel('Languages', fontsize=15,labelpad=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.figure(1,figsize=(10,6))\nsns.countplot(training_data.toxic,alpha=0.8,palette='mako')\nplt.title(\"Appearance of languages in validation data \",fontsize=20)\nplt.ylabel('number of Occurrences', fontsize=15,labelpad=20)\nplt.xlabel('Languages', fontsize=15,labelpad=20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we are building a model that operates on a diverse range of conversation from different languages. We looked at how many languages we have in both the validation and test datasets. First in the training dataset we only have English conversation which means our models will be training on only one language. Then in the validation and testing dataset we will test our models on different languages other than the English. In the bar chart on the left the validation data has only 3 languages Spanish, Italian, and Turkish. while the bar chart on the right the test dataset has 6 languages Spanish, Russian, Italian, French, Portuguese, and Turkish","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#using the wordCloud to display the text on comment text column \n#first we will use stopword to remove unnessecery common word in english \ncomment_words = '' \nstopwords = set(STOPWORDS) \nfor word in training_data['comment_text']: \n \n word = str(word) \n \n # split the value \n tokens = word.split() \n \n # Converts each token into lowercase \n for i in range(len(tokens)): \n tokens[i] = tokens[i].lower() \n #add token to a list \n comment_words += \" \".join(tokens)+\" \"\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.5 Plot WorldCloud for Training Dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#define a wordcloud with the most common word in comment text column\nwordcloud = WordCloud(background_color='black', collocations=False,\n width=1400, height=1200,stopwords=stopwords).generate(comment_words)\nfig = px.imshow(wordcloud)\nfig.update_layout(title_text='Common words in comment text column')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#define a figure for the four wordcloud plot\nfig= plt.figure(figsize=(30,30))\n\nax = fig.add_subplot(221)\n#choose the image as the mask for the plot\nthreat_mask = np.array(Image.open(\"../input/imageseda/bomb.jpg\"))\n# choose the threat comments only from comment_text column\nsubset=training_data[training_data.threat==1]\ntext=subset.comment_text.values\n\n#plot the data with wordcloud \nwordcloud = WordCloud(mask=threat_mask,background_color='black',\n stopwords=stopwords).generate(\"\".join(text))\n\nplt.axis('off')\nplt.imshow( wordcloud.recolor(colormap= 'Paired', random_state=24))\nplt.title('Common threat words',fontsize=23)\n\n############################\nax2 = fig.add_subplot(222)\n#choose the image as the mask for the plot\ninsult_mask = np.array(Image.open(\"../input/imageseda/bullying.jpg\"))\n# choose the insult comments only from comment_text column\nsubset=training_data[training_data.insult==1]\ntext=subset.comment_text.values\n\n#plot the data with wordcloud \nwordcloud = WordCloud(mask=insult_mask,background_color='black',\n stopwords=stopwords).generate(\"\".join(text))\nplt.axis('off')\nplt.imshow( wordcloud.recolor(colormap= 'Paired', random_state=24))\nplt.title('Common insult words',fontsize=23)\n\n#########################\nax3 = fig.add_subplot(223)\n#choose the image as the mask for the plot\ntoxic_mask = np.array(Image.open(\"../input/imageseda/spider3.jpg\"))\n\n# choose the toxic comment only from comment_text column\nsubset=training_data[training_data.toxic==1]\ntext=subset.comment_text.values\n\n#plot the data with wordcloud \nwordcloud = WordCloud(mask=toxic_mask,background_color='black',\n stopwords=stopwords).generate(\"\".join(text))\nplt.axis('off')\nplt.imshow( wordcloud.recolor(colormap= 'Paired', random_state=24))\nplt.title('Common toxic words',fontsize=23)\n\n########################\nax4 = fig.add_subplot(224)\n#choose the image as the mask for the plot\nattack_mask = np.array(Image.open(\"../input/imageseda/stop.jpg\"))\n\n# choose the identity comment only from comment_text column\nsubset=training_data[training_data.identity_hate==1]\ntext=subset.comment_text.values\n\n#plot the data with wordcloud \nwordcloud = WordCloud(mask=attack_mask,background_color='black',\n stopwords=stopwords).generate(\"\".join(text))\n\nplt.axis('off')\nplt.imshow( wordcloud.recolor(colormap= 'Paired', random_state=24))\nplt.title('Common identity attack words',fontsize=23);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Data Cleaning","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Clean The Data From Emoji","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def remove_emoji(text):\n # decode the text from UTF-8 source format\n allchars = [str for str in text.decode('utf-8')]\n \n #define a list of emoji from the library\"emoji\"\n emoji_list = [c for c in allchars if c in emoji.UNICODE_EMOJI]\n \n #for each word in the text, if the word are not an emoji in the list, split and add the word in (clean_text)\n clean_text = ' '.join([str for str in text.decode('utf-8').split() if not any(i in str for i in emoji_list)])\n return clean_text","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#the comment before removing emoji\ntraining_data[\"comment_text\"][3250]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"text_no_emoji=[]\nfor text in training_data[\"comment_text\"]:\n text_no_emoji.append(remove_emoji(text.encode('utf8')))\n\ntraining_data[\"comment_text\"]=text_no_emoji\n#the comment after removing emoji\ntraining_data[\"comment_text\"][3250]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#comment before removing emoji\nvalidation_data[\"comment_text\"][197]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"text_no_emoji_valid=[]\nfor text in validation_data[\"comment_text\"]:\n text_no_emoji_valid.append(remove_emoji(text.encode('utf8')))\nvalidation_data[\"comment_text\"]=text_no_emoji_valid\n#comment after removing emoji\nvalidation_data[\"comment_text\"][197]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#comment before removing emoji\ntesting_data[\"content\"][48657]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"text_no_emoji_test=[]\nfor text in testing_data[\"content\"]:\n text_no_emoji_test.append(remove_emoji(text.encode('utf8')))\n \ntesting_data[\"content\"]=text_no_emoji_test\n\n#comment after removing emoji\ntesting_data[\"content\"][48657]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Clean The Data From Punctuations","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def remove_punctuations(text):\n # define a list with repeated symbol in the dataset\n puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']',\n '>', '%', '=', '#', '*', '+', '\\\\', '•', '~', '@', '£', '·', '_', '{', '}', '©', '^',\n '®', '`', '<', '→', '°', '€', '™', '›', '♥', '←', '×', '§', '″', '′', 'Â', '█',\n '½', 'à', '…', '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶',\n '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', '▒', '：', '¼',\n '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲',\n 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', '∙', '）', '↓', '、', '│', '（', '»', '，', '♪',\n '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√',\n '\\t','\\n','❖','«','✉','❽','♪♫','☆','ψ']\n\n #in for loop replace each symbol with space\n for punctuation in puncts:\n text = text.replace(punctuation, ' ')\n return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#the comment before removing punctuations\ntraining_data[\"comment_text\"] [2508]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"training_data[\"comment_text\"] = training_data['comment_text'].apply(remove_punctuations)\n#the comment after removing punctuations\ntraining_data[\"comment_text\"] [2508]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#the comment before removing punctuations\nvalidation_data[\"comment_text\"] [361]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"validation_data[\"comment_text\"] = validation_data[\"comment_text\"].apply(remove_punctuations)\n#the comment after removing punctuations\nvalidation_data[\"comment_text\"][361]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#the comment before removing punctuations\ntesting_data[\"content\"][76]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"testing_data[\"content\"] = testing_data[\"content\"].apply(remove_punctuations)\n#the comment after removing punctuations\ntesting_data[\"content\"][76]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inights From Data Cleaning:\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Deleting the emoji from the comments didn't improve our models (Bert,XLM), it cause our prediction to give less accurate results.We found after some research that Bert tokenizer was updated on 2019 to new version that understand the emoji. the older version was considering the emoji as UNK token which means (unknown word that doesn't exist in the vocabulary set).the new version improved this issue and know now the method works with emoji. the bottom line we don't need to delete the emoji from XLM and Bert models because it's already exist in the vocabulary set.
\n\n\n* The same results happened when we removed the punctuations. our models(Bert,XLM) become less accurate around (-2%). We thought that one of the reasons for this is because a lot of conversion on the internet especially that's contains rude words they used symbols like($, ** *) with the rude words for example (shi***y,a$$) to decive the program and avoids getting blocked\n\n* The model LSTM has build in parameters in it's tokenizer for cleaning the data from punctuations","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}