{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <font face=\"verdana\" color=\"dodgerblue\">Introduction:</font>\nBeing anonymous over the internet can sometimes make people say nasty things that they normally would not in real life. Let's filter out the hate from our platforms one comment at a time.\n\n# <font face=\"verdana\" color=\"dodgerblue\">Objective:</font>\nTo create an EDA/ feature-engineering starter notebook for toxic comment classification.\n\n# <font face=\"verdana\" color=\"dodgerblue\">Data Description</font>\n\n### <font face=\"verdana\" color=\"darkred\">What should I expect the data format to be?</font>\nThe primary data for the competition is, in each provided file, the comment_text column. This contains the text of a comment which has been classified as toxic or non-toxic (0...1 in the toxic column). The train set’s comments are entirely in english and come either from Civil Comments or Wikipedia talk page edits. The test data's comment_text columns are composed of multiple non-English languages.\n\nThe *-train.csv files and validation.csv file also contain a toxic column that is the target to be trained on.\n\nThe jigsaw-toxic-comment-train.csv and jigsaw-unintended-bias-train.csv contain training data (comment_text and toxic) from the two previous Jigsaw competitions, as well as additional columns that you may find useful.\n\n*-seqlen128.csv files contain training, validation, and test data that has been processed for input into BERT.\n\n### <font face=\"verdana\" color=\"darkred\">What am I predicting?</font>\nYou are predicting the probability that a comment is toxic. A toxic comment would receive a 1.0. A benign, non-toxic comment would receive a 0.0. In the test set, all comments are classified as either a 1.0 or a 0.0.\n\n### <font face=\"verdana\" color=\"darkred\">Files</font>.\n* **jigsaw-toxic-comment-train.csv** - data from our first competition. The dataset is made up of English comments from Wikipedia’s talk page edits.\n* **jigsaw-unintended-bias-train.csv** - data from our second competition. This is an expanded version of the Civil Comments dataset with a range of additional labels.\n* **sample_submission.csv** - a sample submission file in the correct format\n* **test.csv** - comments from Wikipedia talk pages in different non-English languages.\n* **validation.csv** - comments from Wikipedia talk pages in different non-English languages.\n* **jigsaw-toxic-comment-train-processed-seqlen128.csv** - training data preprocessed for BERT\n* **jigsaw-unintended-bias-train-processed-seqlen128.csv** - training data preprocessed for BERT\n* **validation-processed-seqlen128.csv** - validation data preprocessed for BERT\n* **test-processed-seqlen128.csv** - test data preprocessed for BERT\n\n### <font face=\"verdana\" color=\"darkred\">Columns</font>\n* **id** - identifier within each file.\n* **comment_text** - the text of the comment to be classified.\n* **lang** - the language of the comment.\n* **toxic** - whether or not the comment is classified as toxic. (Does not exist in test.csv.)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nplt.style.use('fivethirtyeight')\n%matplotlib inline\n\nimport plotly.offline as py\nfrom plotly.offline import iplot, init_notebook_mode\nimport plotly.graph_objs as go\ninit_notebook_mode(connected = True)\n\nimport urllib.request\nfrom PIL import Image\nfrom wordcloud import WordCloud ,STOPWORDS\nfrom IPython.display import Markdown\ndef bold(string):\n    display(Markdown(string))\n\nfrom tqdm import tqdm\nimport time\nimport re\nimport string\nimport nltk\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import word_tokenize\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom scipy.sparse import csr_matrix, hstack\nfrom sklearn.preprocessing import MinMaxScaler, StandardScaler\nfrom sklearn.linear_model import LogisticRegression\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold, StratifiedKFold\nfrom sklearn.metrics import roc_auc_score as auc\nimport gc\nfrom collections import defaultdict\nimport os\nimport psutil","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# read the data file\ndf_train = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv')\ndf_train1 = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv')\ndf_valid = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ndf_test = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# preview the date file\nbold('**TRAINING DATA**')\ndisplay(df_train.head(3))\nbold('**BAIS TRAINING DATA**')\ndisplay(df_train1.head(3))\nbold('**VALIDATION DATA**')\ndisplay(df_valid.head(3))\nbold('**TEST DATA**')\ndisplay(df_test.head(3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# shape of the datasets\nprint('Shape of training data:', df_train.shape)\nprint('Shape of validation data:', df_valid.shape)\nprint('Shape of test data:', df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the missing values\nprint(\"Check for missing values in Train dataset\")\nnull_check=df_train.isnull().sum()\nprint(null_check)\nprint('\\n')\nprint(\"Check for missing values in Validation dataset\")\nnull_check=df_valid.isnull().sum()\nprint(null_check)\nprint('\\n')\nprint(\"Check for missing values in Test dataset\")\nnull_check=df_test.isnull().sum()\nprint(null_check)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Target Distribution"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(ncols=2, figsize=(16, 7), dpi=100)\n\ntemp = df_train.toxic.value_counts()\nsns.barplot(temp.index, temp, ax=axes[0], palette='Dark2')\n\ntemp = df_valid.toxic.value_counts()\nsns.barplot(temp.index, temp, ax=axes[1], palette='Dark2')\n\n\naxes[0].set_ylabel('Count ')\naxes[1].set_ylabel(' ')\naxes[0].set_xticklabels([\"Non-toxic (90.4%) [0's]\", \"toxic (9.6%) [1's]\"])\naxes[1].set_xticklabels([\"Non-toxic (84.6%) [0's]\", \"toxic (15.4%) [1's]\"])\n\naxes[0].set_title('Target Distribution of Train Dataset', fontsize=13)\naxes[1].set_title('Target Distribution of Valid Dataset', fontsize=13)\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Target Distribution of Train Dataset have **93.7% for 0 (Non-toxic comment)** and **6.3% for 1 (Toxic comment)** and Validation Dataset have **84.6% for 0 (Non-toxic comment)** and **15.4% for 1 (Toxic comment).** It is show that taget variable is highly imbalanced."},{"metadata":{},"cell_type":"markdown","source":"## <font face=\"verdana\" color=\"dodgerblue\">Example Comments</font>"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Toxic Comment')\nprint('\\n')\nprint(df_train[df_train.toxic==1].iloc[3,1])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Non-toxic Comment')\nprint('\\n')\nprint(df_train[df_train.toxic==0].iloc[7,1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <font face=\"verdana\" color=\"dodgerblue\">Language distribution"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(ncols=3, figsize=(17, 7), dpi=100)\n\ntemp = df_valid.lang.value_counts()\nsns.barplot(temp.index, temp, ax=axes[0], palette='Set1')\n\ntemp = df_test.lang.value_counts()\nsns.barplot(temp.index, temp, ax=axes[1], palette='Set1')\n\nsns.countplot(data=df_valid, x=\"lang\", hue=\"toxic\" ,ax=axes[2], palette='Set1')\n\naxes[0].set_ylabel(' Count ')\naxes[1].set_ylabel(' ')\naxes[2].set_ylabel(' ')\naxes[2].set_xlabel(' ')\n\naxes[0].set_title('Language Distribution of Valid Dataset', fontsize=13)\naxes[1].set_title('Language Distribution of Test Dataset', fontsize=13)\naxes[2].set_title('Language Distribution by Taget of Valid dataset', fontsize=13)\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">Wordclouds - Frequent words:</font>\n**Now, let's take a look at words that are associated with these classes. The visuals here are word clouds (ie) more frequent words appear bigger.** "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"non_toxic_mask=np.array(Image.open(urllib.request.urlopen(url='https://image.flaticon.com/icons/png/512/99/99665.png')))\n#wordcloud for non-toxic comments\nsubset=df_train[df_train.toxic==0]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=non_toxic_mask,stopwords=STOPWORDS)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(20,10))\nplt.axis(\"off\")\nplt.title(\"Words frequented in non-toxic Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"stopwords = STOPWORDS\ntoxic_mask=np.array(Image.open(urllib.request.urlopen(url='https://cdn4.vectorstock.com/i/1000x1000/81/98/radiation-hazard-symbol-vector-23088198.jpg')))\n#wordcloud for toxic comments\nsubset=df_train[df_train.toxic==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=toxic_mask,stopwords=stopwords)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(20,10))\nplt.axis(\"off\")\nplt.title(\"Words frequented in toxic Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Reds' , random_state=17), alpha=0.98)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">Ngrams Analysis</font>\nMost common unigrams exist in both classes are mostly punctuations, stop words or numbers. It is better to clean them before modelling since they don't give much information about target.\n\nIn the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. The n-grams typically are collected from a text or speech corpus [source](https://en.wikipedia.org/wiki/N-gram).\n\nNgrams can be a very useful tool when trying to figure out which words and phrases are used in English. They can help show when certain phrases entered into the vernacular, and when they fell out of favor. But they have their limitations."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"#source: https://www.kaggle.com/shahules/basic-eda-cleaning-and-glove\ndef get_top_bigrams(corpus, n=None):\n    vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)\n    bag_of_words = vec.transform(corpus)\n    sum_words = bag_of_words.sum(axis=0) \n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n    return words_freq[:n]\n\ndef get_top_threegrams(corpus, n=None):\n    vec = CountVectorizer(ngram_range=(3, 3)).fit(corpus)\n    bag_of_words = vec.transform(corpus)\n    sum_words = bag_of_words.sum(axis=0) \n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n    return words_freq[:n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#plt.style.use('ggplot')\nnt_comment = df_train[df_train.toxic==0][\"comment_text\"]\nt_comment = df_train[df_train.toxic==1][\"comment_text\"]\nfig, axes = plt.subplots(2, 2, figsize=(18, 20), dpi=100)\n           \ntop_unigrams=get_top_bigrams(nt_comment)[:20]\nx,y=map(list,zip(*top_unigrams))\nsns.barplot(x=y,y=x, ax=axes[0,0], color='c')\n\n\ntop_bigrams=get_top_bigrams(t_comment)[:20]\nx,y=map(list,zip(*top_bigrams))\nsns.barplot(x=y,y=x, ax=axes[0,1], color='red')\n\ntop_threegrams=get_top_threegrams(nt_comment)[:20]\nx,y=map(list,zip(*top_threegrams))\nsns.barplot(x=y,y=x, ax=axes[1, 0], color='c')\n\ntop_fourgrams=get_top_threegrams(t_comment)[:20]\nx,y=map(list,zip(*top_fourgrams))\nsns.barplot(x=y,y=x, ax=axes[1, 1], color='red')\n\n\naxes[0, 0].set_ylabel(' ')\naxes[0, 1].set_ylabel(' ')\naxes[1, 0].set_ylabel(' ')\naxes[1, 1].set_ylabel(' ')\n\naxes[0, 0].yaxis.set_tick_params(labelsize=15)\naxes[0, 1].yaxis.set_tick_params(labelsize=15)\naxes[1, 0].yaxis.set_tick_params(labelsize=15)\naxes[1, 1].yaxis.set_tick_params(labelsize=15)\n\naxes[0, 0].set_title('Top 20 most common bigrams in Non-toxic', fontsize=15)\naxes[0, 1].set_title('Top 20 most common bigrams in toxic', fontsize=15)\naxes[1, 0].set_title('Top 20 most common threegrams in Non-toxic', fontsize=15)\naxes[1, 1].set_title('Top 20 most common threegrams in toxic', fontsize=15)\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <font face=\"verdana\" color=\"dodgerblue\">Feature engineering:</font>\nLet’s look at some popular and effective strategies for handling text data and extracting meaningful features from the same which can be used in downstream machine learning systems. Some of the  below code has be taken @olivier's [notebook](https://www.kaggle.com/ogrellier/lgbm-with-words-and-chars-n-gram)."},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">Cleaning The Text</font>\nLet's create the funtion for the cleaning the text. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Contraction replacement patterns\n#https://www.pythonforbeginners.com/regex/regular-expressions-in-python\ncont_patterns = [\n    (b'(W|w)on\\'t', b'will not'),\n    (b'(C|c)an\\'t', b'can not'),\n    (b'(I|i)\\'m', b'i am'),\n    (b'(A|a)in\\'t', b'is not'),\n    (b'(\\w+)\\'ll', b'\\g<1> will'),\n    (b'(\\w+)n\\'t', b'\\g<1> not'),\n    (b'(\\w+)\\'ve', b'\\g<1> have'),\n    (b'(\\w+)\\'s', b'\\g<1> is'),\n    (b'(\\w+)\\'re', b'\\g<1> are'),\n    (b'(\\w+)\\'d', b'\\g<1> would'),\n]\npatterns = [(re.compile(regex), repl) for (regex, repl) in cont_patterns]\n\ndef count_regexp_occ(regexp=\"\", text=None):\n    \"\"\" Simple way to get the number of occurence of a regex\"\"\"\n    return len(re.findall(regexp, text))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_for_char_n_gram(text):\n    \"\"\" Simple text clean up process\"\"\"\n    # 1. Go to lower case (only good for english)\n    # Go to bytes_strings as I had issues removing all \\n in r\"\"\n    clean = bytes(text.lower(), encoding=\"utf-8\")\n    # 2. Drop \\n and  \\t\n    clean = clean.replace(b\"\\n\", b\" \")\n    clean = clean.replace(b\"\\t\", b\" \")\n    clean = clean.replace(b\"\\b\", b\" \")\n    clean = clean.replace(b\"\\r\", b\" \")\n    # 3. Replace english contractions\n    for (pattern, repl) in patterns:\n        clean = re.sub(pattern, repl, clean)\n    # 4. Drop puntuation\n    # I could have used regex package with regex.sub(b\"\\p{P}\", \" \")\n    exclude = re.compile(b'[%s]' % re.escape(bytes(string.punctuation, encoding='utf-8')))\n    clean = b\" \".join([exclude.sub(b'', token) for token in clean.split()])\n    # 5. Drop numbers - as a scientist I don't think numbers are toxic ;-)\n    clean = re.sub(b\"\\d+\", b\" \", clean)\n    # 6. Remove extra spaces - At the end of previous operations we multiplied space accurences\n    clean = re.sub(b'\\s+', b' ', clean)\n    # Remove ending space if any\n    clean = re.sub(b'\\s+$', b'', clean)\n    # 7. Now replace words by words surrounded by # signs\n    # e.g. my name is bond would become #my# #name# #is# #bond#\n    #clean = re.sub(b\"([a-z]+)\", b\"#\\g<1>#\", clean)\n    clean = re.sub(b\" \", b\"# #\", clean)  # Replace space\n    clean = b\"#\" + clean + b\"#\"  # add leading and trailing #\n\n    return str(clean, 'utf-8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">Extracting Feature</font>\nCreating the function for extracting feature from text data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_indicators_and_clean_comments(df):\n    \"\"\"\n    Check all sorts of content as it may help find toxic comment\n    Though I'm not sure all of them improve scores\n    \"\"\"\n    # Count number of \\n\n    df[\"ant_slash_n\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\n\", x))\n    # Get length in words and characters\n    df[\"raw_word_len\"] = df[\"comment_text\"].apply(lambda x: len(x.split()))\n    df[\"raw_char_len\"] = df[\"comment_text\"].apply(lambda x: len(x))\n    # Check number of upper case, if you're angry you may write in upper case\n    df[\"nb_upper\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"[A-Z]\", x))\n    # Number of F words - f..k contains folk, fork,\n    df[\"nb_fk\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"[Ff]\\S{2}[Kk]\", x))\n    # Number of S word\n    df[\"nb_sk\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"[Ss]\\S{2}[Kk]\", x))\n    # Number of D words\n    df[\"nb_dk\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"[dD]ick\", x))\n    # Number of occurence of You, insulting someone usually needs someone called : you\n    df[\"nb_you\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\W[Yy]ou\\W\", x))\n    # Just to check you really refered to my mother ;-)\n    df[\"nb_mother\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\Wmother\\W\", x))\n    # Just checking for toxic 19th century vocabulary\n    df[\"nb_ng\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\Wnigger\\W\", x))\n    # Some Sentences start with a <:> so it may help\n    df[\"start_with_columns\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"^\\:+\", x))\n    # Check for time stamp\n    df[\"has_timestamp\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\d{2}|:\\d{2}\", x))\n    # Check for dates 18:44, 8 December 2010\n    df[\"has_date_long\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\D\\d{2}:\\d{2}, \\d{1,2} \\w+ \\d{4}\", x))\n    # Check for date short 8 December 2010\n    df[\"has_date_short\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\D\\d{1,2} \\w+ \\d{4}\", x))\n    # Check for http links\n    df[\"has_http\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"http[s]{0,1}://\\S+\", x))\n    # check for mail\n    df[\"has_mail\"] = df[\"comment_text\"].apply(\n        lambda x: count_regexp_occ(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+', x)\n    )\n    # Looking for words surrounded by == word == or \"\"\"\" word \"\"\"\"\n    df[\"has_emphasize_equal\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\={2}.+\\={2}\", x))\n    df[\"has_emphasize_quotes\"] = df[\"comment_text\"].apply(lambda x: count_regexp_occ(r\"\\\"{4}\\S+\\\"{4}\", x))\n\n    # Now clean comments\n    df[\"clean_comment\"] = df[\"comment_text\"].apply(lambda x: prepare_for_char_n_gram(x))\n\n    # Get the new length in words and characters\n    df[\"clean_word_len\"] = df[\"clean_comment\"].apply(lambda x: len(x.split()))\n    df[\"clean_char_len\"] = df[\"clean_comment\"].apply(lambda x: len(x))\n    # Number of different characters used in a comment\n    # Using the f word only will reduce the number of letters required in the comment\n    df[\"clean_chars\"] = df[\"clean_comment\"].apply(lambda x: len(set(x)))\n    df[\"clean_chars_ratio\"] = df[\"clean_comment\"].apply(lambda x: len(set(x))) / df[\"clean_comment\"].apply(\n        lambda x: 1 + min(99, len(x)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dropinf the unwanted columns and rename text column\nkeep_cols = ['id', 'comment_text', 'toxic']\ndf_train = df_train[keep_cols]\ndf_test.rename(columns = {\"content\": \"comment_text\"}, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# performing the function\nget_indicators_and_clean_comments(df_train)\nget_indicators_and_clean_comments(df_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">Can we characterize non-toxic and toxic comment through meta feature?</font>\nCheck all sorts of content as it may help find toxic comment. Though I'm not sure all of them improve scores"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Function to plot histogram'''\ndef histogram_plot(x1, x2, title, end,size):\n    trace1 = go.Histogram(x = x1,\n                        name ='Non-Toxic', \n                        xbins = dict(end=end,size=size),\n                        marker = dict(color = '#1bd902'))\n    trace2 = go.Histogram(x = x2,\n                        name='Toxic',\n                        xbins = dict(end=end,size=size),\n                        marker = dict(color = '#ff0307'))\n    layout = go.Layout(barmode='stack', \n                       title = title, \n                       width=600, height=500,\n                       template=\"ggplot2\",\n                       yaxis = dict(title = \"Count\"),\n                       xaxis = dict(title = \"Value\"),\n                       font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",\n                                 color='black'))\n    fig = go.Figure(data = [trace1, trace2], layout = layout,)\n    return iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['ant_slash_n']\nt  = df_train[df_train.toxic==1]['ant_slash_n']\nhistogram_plot(nt, t, \"Count of Newline\", 30, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['raw_word_len']\nt  = df_train[df_train.toxic==1]['raw_word_len']\nhistogram_plot(nt, t, \"Count of Word length\", 400, 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['raw_char_len']\nt  = df_train[df_train.toxic==1]['raw_char_len']\nhistogram_plot(nt, t, \"Count of Character length\", 1000, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_upper']\nt  = df_train[df_train.toxic==1]['nb_upper']\nhistogram_plot(nt, t, \"Count of Upper Case Length\", 100, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_fk']\nt  = df_train[df_train.toxic==1]['nb_fk']\nhistogram_plot(nt, t, \"Count of F**K word contains folk, fork\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_sk']\nt  = df_train[df_train.toxic==1]['nb_sk']\nhistogram_plot(nt, t, \"Count of S**K word\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_dk']\nt  = df_train[df_train.toxic==1]['nb_dk']\nhistogram_plot(nt, t, \"Count of D**K word\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_you']\nt  = df_train[df_train.toxic==1]['nb_you']\nhistogram_plot(nt, t, \"Number of Occurence of You\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_mother']\nt  = df_train[df_train.toxic==1]['nb_mother']\nhistogram_plot(nt, t, \"Number of Occurence of Mother\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['nb_ng']\nt  = df_train[df_train.toxic==1]['nb_ng']\nhistogram_plot(nt, t, \"Number of Occurence of Nigger\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['has_timestamp']\nt  = df_train[df_train.toxic==1]['has_timestamp']\nhistogram_plot(nt, t, \"Number of Occurence of Time Stamp\", 15, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['has_date_long']\nt  = df_train[df_train.toxic==1]['has_date_long']\nhistogram_plot(nt, t, \"Number of Occurence of Date Long\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['has_http']\nt  = df_train[df_train.toxic==1]['has_http']\nhistogram_plot(nt, t, \"Number of Occurence of http Links\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['has_mail']\nt  = df_train[df_train.toxic==1]['has_mail']\nhistogram_plot(nt, t, \"Number of Occurence of Mail\", 10, 1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['clean_word_len']\nt  = df_train[df_train.toxic==1]['clean_word_len']\nhistogram_plot(nt, t, \"Word Lenght After Cleaning\", 800, 10)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"nt = df_train[df_train.toxic==0]['clean_char_len']\nt  = df_train[df_train.toxic==1]['clean_char_len']\nhistogram_plot(nt, t, \"Character Lenght After Cleaning\", 3000, 30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Meta feature do not seem to be a significant indicator of toxicity. Some of them like count of fk, st, ng can classify the toxicity.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# correlation between meta featrue\nnum_features = [f for f in df_train.columns\n                        if f not in [\"comment_text\", \"clean_comment\", \"id\", \"toxic\"]]\ncorr = df_train[num_features].corr()\nsns.set_style(\"white\")\nplt.rcParams['figure.figsize'] = (20,15)\nmask = np.zeros_like(corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\nsns.heatmap(corr, mask=mask, cmap='RdYlGn',square=True, linewidths=.5,annot=True)\nplt.title(\"Correlation Between Meta Featrue\", fontsize=25)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">TF-IDF Vectorizer</font>"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#Scaling numerical features\nskl = MinMaxScaler()\ntrain_num_features = csr_matrix(skl.fit_transform(df_train[num_features]))\ntest_num_features = csr_matrix(skl.fit_transform(df_test[num_features]))\n\n#Get TF-IDF features\ntrain_text = df_train['comment_text']\ntest_text = df_test['comment_text']\nall_text = pd.concat([train_text, test_text])\n\n #  On real words\nword_vectorizer = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents='unicode',\n    analyzer='word',\n    token_pattern=r'\\w{1,}',\n    stop_words='english',\n    ngram_range=(1, 2),\n    max_features=20000)\nword_vectorizer.fit(all_text)\ntrain_word_features = word_vectorizer.transform(train_text)\ntest_word_features = word_vectorizer.transform(test_text)\n\ndel word_vectorizer\ngc.collect()\n\n# On character\nchar_vectorizer = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents='unicode',\n    analyzer='char',\n    stop_words='english',\n    ngram_range=(2, 6),\n    max_features=50000)\nchar_vectorizer.fit(all_text)\ntrain_char_features = char_vectorizer.transform(train_text)\ntest_char_features = char_vectorizer.transform(test_text)\n\ndel train_text\ndel test_text\ndel all_text\ndel char_vectorizer\ngc.collect()\n\n\n# Now stack TF IDF matrices\ntrain_features = hstack([train_char_features, \n                         train_word_features, \n                         train_num_features]).tocsr()\n\ndel train_char_features\ndel train_word_features\ngc.collect()\n\ntest_features = hstack([test_char_features, \n                        test_word_features, \n                        test_num_features]).tocsr()\ndel test_char_features \ndel test_word_features,\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <font face=\"verdana\" color=\"darkred\">Baseline Model</font>"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntarget =  df_train[\"toxic\"]\n\n# Model\ndef run_cv_model(train, test, target, model_fn, params={}, eval_fn=None, label='model'):\n    # Stratified k-fold \n    kf = StratifiedKFold(n_splits=5)\n    fold_splits = kf.split(train, target)\n    cv_scores = []\n    pred_full_test = 0\n    pred_train = np.zeros((train.shape[0]))\n    i = 1\n    for dev_index, val_index in fold_splits:\n        print('Started ' + label + ' fold ' + str(i) + '/5')\n        dev_X, val_X = train[dev_index], train[val_index]\n        dev_y, val_y = target[dev_index], target[val_index]\n        params2 = params.copy()\n        pred_val_y, pred_test_y = model_fn(dev_X, dev_y, val_X, val_y, test, params2)\n        pred_full_test = pred_full_test + pred_test_y\n        pred_train[val_index] = pred_val_y\n        if eval_fn is not None:\n            cv_score = eval_fn(val_y, pred_val_y)\n            cv_scores.append(cv_score)\n            print(label + ' cv score {}: {}'.format(i, cv_score))\n        i += 1\n    print('{} cv scores : {}'.format(label, cv_scores))\n    print('{} cv mean score : {}'.format(label, np.mean(cv_scores)))\n    print('{} cv std score : {}'.format(label, np.std(cv_scores)))\n    pred_full_test = pred_full_test / 5.0\n    results = {'label': label,\n              'train': pred_train, 'test': pred_full_test,\n              'cv': cv_scores}\n    return results\n\n\ndef runLR(train_X, train_y, test_X, test_y, test_X2, params):\n    print('Train LR')\n    model = LogisticRegression(**params)\n    model.fit(train_X, train_y)\n    print('Predict 1/2')\n    pred_test_y = model.predict_proba(test_X)[:, 1]\n    print('Predict 2/2')\n    pred_test_y2 = model.predict_proba(test_X2)[:, 1]\n    return pred_test_y, pred_test_y2\n\n\nlr_params = {'solver': 'sag', 'C':  0.1, 'max_iter': 1000}\nresults = run_cv_model(train_features, test_features, target, runLR, lr_params, auc, 'lr')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsub_id = sub['id']\nsubmission = pd.DataFrame({'id': sub_id, 'toxic': results['test']})\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Give me your feedback and if you find my kernel helpful please UPVOTE will be appreciated."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}