{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Quora Insincerity Detection\n---\nStarted: 30 Jan 2022","metadata":{}},{"cell_type":"markdown","source":"## Step Zero\n#### Aim: Import necessary libraries, download objects and prepare constants (if any), Download data\n---","metadata":{}},{"cell_type":"code","source":"#!pip install pyspellchecker\n# import all necessary libraries\n\n# For dataframes\nimport pandas as pd \n\n# For numerical arrays\nimport numpy as np \n\n# For stemming/Lemmatisation/POS tagging\nimport spacy\n\n# For getting stopwords\nfrom spacy.lang.en.stop_words import STOP_WORDS\n\n# For K-Fold cross validation\nfrom sklearn.model_selection import KFold\n\n# For visualizations\nimport matplotlib.pyplot as plt\n\n# For regular expressions\nimport re\n\n# For handling string\nimport string\n\n# For all torch-supported actions\nimport torch\n\n# For spell-check\n# from spellchecker import SpellChecker\n\n# For performing mathematical operations\nimport math\n\n# For dictionary related activites\nfrom collections import defaultdict\n\n# For counting actions (EDA)\nfrom collections import  Counter\n\n# For count vectorisation (EDA)\nfrom sklearn.feature_extraction.text import CountVectorizer\n\n# For one-hot encoding\nfrom tensorflow.keras.utils import to_categorical\n\n# For DL model\nfrom tensorflow.keras.layers import Dense, Input, GlobalMaxPooling1D\nfrom tensorflow.keras.layers import Conv1D, MaxPooling1D, Embedding, LSTM\nfrom tensorflow.keras.models import Model, Sequential\n\n# For generating random integers\nfrom random import randint\n\n#For making wordclouds\nfrom wordcloud import WordCloud \n\n# For TF-IDF vectorisation\nfrom sklearn.feature_extraction.text import TfidfVectorizer\n\n# For padding\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\n\n# For tokenization\nfrom tensorflow.keras.preprocessing.text import Tokenizer\n\n# For plotting\nimport seaborn as sns\n\nprint(\"Necessary libraries imported\")\n\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:24:41.766185Z","iopub.execute_input":"2022-02-07T18:24:41.766820Z","iopub.status.idle":"2022-02-07T18:24:52.149667Z","shell.execute_reply.started":"2022-02-07T18:24:41.766728Z","shell.execute_reply":"2022-02-07T18:24:52.148781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ndf.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:24:56.317048Z","iopub.execute_input":"2022-02-07T18:24:56.317396Z","iopub.status.idle":"2022-02-07T18:25:00.770251Z","shell.execute_reply.started":"2022-02-07T18:24:56.317342Z","shell.execute_reply":"2022-02-07T18:25:00.769345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step One\n#### Aim: Time to do some EDA baby!\n---\nThis phase involves complete understanding of what is there in the dataset, and the key nuances that needs to be understood before framing the Input-Output ML pipeline. We perform the following:\n\n- Dataset description (to know what's presented and what's not available)","metadata":{}},{"cell_type":"markdown","source":"#### 1.1 Dataset Description\nStudying the basic statistics of the dataset, which covers the following aspects:\n\n- Analysing columns\n- Null-Value statistics\n- Overall column-wise stats\n- Highest and lowest word length, input length\n- Target types and frequency","metadata":{}},{"cell_type":"code","source":"print('Total rows in dataset: ',len(df),'Rows\\n')\nprint('Dataset columns: ')\nprint(df.columns)\nprint('\\nNull Statistics (in %): ')\nprint(df.isnull().sum()* 100 / len(df))\nprint('\\nDataset description: ')\nprint(df.describe())\nprint('\\nEssay prompt frequency: ')\nprint(df.groupby(\"target\").describe().loc[:,[('question_text',  'count'),('question_text', 'unique')]])\n\nprint('\\nMax and Min statistics for word/char count of question_text: ')\nprint('MAX\\t',max(df.question_text.apply(lambda x: len(x))),'characters,',\n     max(df.question_text.apply(lambda x: len(x.split()))), 'words')\nprint('MIN\\t',min(df.question_text.apply(lambda x: len(x))),'character(s),',\n     min(df.question_text.apply(lambda x: len(x.split()))), 'word(s)')\n\nprint('Pie plot against Target')\ndf.target.value_counts().plot(title='Target categories',kind='pie')\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:27:00.134433Z","iopub.execute_input":"2022-02-07T18:27:00.134724Z","iopub.status.idle":"2022-02-07T18:27:07.269220Z","shell.execute_reply.started":"2022-02-07T18:27:00.134692Z","shell.execute_reply":"2022-02-07T18:27:07.268331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n**Inference:**\n1. There are `1306122` rows, all of which are non-null and unique *(Thats a lot!)*\n2. We only have 3 columns: one column is the Q-ID (won't contribute to decision making), the other column is the Question Text *(the main input)*, and the last column is the Target *(the expected output)*\n3. The data is highly biased, with 1225k sincere questions `target=0` and only 80k insincere questions `target=1` *(Which means we've got to be careful while training our models cuz the dataset itself is partial towards sincerity)*\n4. There can be 100+ words or 1000+ characters in a `question_text` *(ie, our model should be scalable enough to handle bigger sentences)*","metadata":{}},{"cell_type":"markdown","source":"#### 1.2 Effect of essay-word-lengths over score\nHere, we observe the trend of distribution of targets across different essay sets, capturing the following trends:\n\n- Total words vs target\n- Word length vs target","metadata":{}},{"cell_type":"code","source":"def get_avg_length(essay):\n    summ=0\n    for word in essay.split():\n        summ+=len(word)\n    return round(summ/len(essay.split()),2)\n\ndf['avg_word_length']=df.question_text.apply(lambda x: get_avg_length(x))\nprint(\"ScatterPlot for Target vs average word length\")\nplt.plot(figsize=(10,10)) \nplt.title('Target versus average word length')\n\nsns.stripplot(data=df,\n    x=\"target\", y=\"avg_word_length\")\n\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:27:55.559115Z","iopub.execute_input":"2022-02-07T18:27:55.559432Z","iopub.status.idle":"2022-02-07T18:28:06.400063Z","shell.execute_reply.started":"2022-02-07T18:27:55.559397Z","shell.execute_reply":"2022-02-07T18:28:06.399309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['total_words']=df.question_text.apply(lambda x: len(x.split()))\nprint(\"Boxplot for Target vs total words\")\nplt.plot(figsize=(8,15)) \nplt.title('Target versus total words')\n\nsns.boxplot(data=df,\n    x=\"target\", y=\"total_words\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:28:06.401441Z","iopub.execute_input":"2022-02-07T18:28:06.401677Z","iopub.status.idle":"2022-02-07T18:28:08.215276Z","shell.execute_reply.started":"2022-02-07T18:28:06.401646Z","shell.execute_reply":"2022-02-07T18:28:08.214425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference:**\n1. The average wordlength for both the categories is scattered around the 5-15 mark, but there are a few outliers whose average wordlength crosses 60. <br>*(we have noise in our data: we can expect sentences that aren't grammatically/syntactically/spellingly correct.)*\n2. Although the maximum of total words used could cross 100, majority of the questions are based around the 0-50 region, with exceptions of outliers. <br>*(We're gonna have to use a dataset which has majority of sentences within a respectable range, and a minority of sentences that could pose a problem due to unusually high number of words.)*","metadata":{}},{"cell_type":"markdown","source":"#### 1.3 Unigram analysis\nThe primary goal here is to see what words are most frequently used. This will be done in the following ways:\n\n- Frequency of stop-words used\n- Most commonly occuring non-stop-words in each target category","metadata":{}},{"cell_type":"code","source":"print(\"Stop-word freuqency\")\n\nfig, axes = plt.subplots(1,len(df.target.unique()), figsize=(20,8))\nfig.suptitle('Stop-word frequency')\n\nfor index,target in enumerate(df.target.unique()):\n    dct=defaultdict(int) \n    curdf=df[df['target']==target]  \n    \n    allwordsarr=curdf.question_text.str.cat().split() #First, we join all strings from the question_text column, then we split em all so that we get an array of all words, which could be counted\n    counter=Counter(allwordsarr)\n    most=counter.most_common()\n    x=[]\n    y=[]\n    for word,count in most[:30]:\n        if (word in STOP_WORDS) :\n            x.append(word)\n            y.append(count)\n    sns.barplot(ax=axes[index],x=y,y=x)\n    axes[index].set_title(\"Target: \"+str(target))\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:28:08.216382Z","iopub.execute_input":"2022-02-07T18:28:08.216575Z","iopub.status.idle":"2022-02-07T18:28:15.129548Z","shell.execute_reply.started":"2022-02-07T18:28:08.216552Z","shell.execute_reply":"2022-02-07T18:28:15.128677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Wordcloud\")\n\nfig, axes = plt.subplots(1,len(df.target.unique()), figsize=(20,8))\n\n    \nfor index,target in enumerate(df.target.unique()):\n    dct=defaultdict(int) \n    curdf=df[df['target']==target]  \n    \n    df_fullstring=\" \".join(curdf.question_text.str.cat().split()) #first, we join all strings of column, then split by space, then join again because we want a full string here\n    wordcloud = WordCloud(background_color='white',max_words=100,\n                      max_font_size=40,\n                      scale=3,\n                      random_state=1).generate(df_fullstring)\n\n    axes[index].imshow(wordcloud)\n    axes[index].set_title('Target: '+str(target))\n    axes[index].axis('off')\n    \nfig.suptitle('Wordcloud')\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:28:15.131752Z","iopub.execute_input":"2022-02-07T18:28:15.131962Z","iopub.status.idle":"2022-02-07T18:29:06.598744Z","shell.execute_reply.started":"2022-02-07T18:28:15.131937Z","shell.execute_reply":"2022-02-07T18:29:06.586998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Most commonly occcuring words in all categories\")\n\nfig, axes = plt.subplots(1,len(df.target.unique()), figsize=(20,8))\nfig.suptitle('Common-word frequency')\n\nfor index,target in enumerate(df.target.unique()):\n    dct=defaultdict(int) \n    curdf=df[df['target']==target]  \n    allwordsarr=curdf.question_text.str.cat().split()\n    counter=Counter(allwordsarr)\n    most=counter.most_common()\n    x=[]\n    y=[]\n    for word,count in most[:100]:\n        if (word.lower() not in STOP_WORDS):\n            x.append(word)\n            y.append(count)\n    sns.barplot(ax=axes[index],x=y,y=x)\n    axes[index].set_title(\"Target: \"+str(target))\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:29:06.600049Z","iopub.execute_input":"2022-02-07T18:29:06.600437Z","iopub.status.idle":"2022-02-07T18:29:13.179615Z","shell.execute_reply.started":"2022-02-07T18:29:06.600406Z","shell.execute_reply":"2022-02-07T18:29:13.178877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference**\n\nWe get the following information from these two series of barcharts:\n\n- The frequency of stop-words in each word is heavy enough *(raising the need to do stopword-removal during data cleaning and feature-formatting)*\n- There seems to be no explainable reason to have a word heavily associated with a category. For instance, the popular word \"India\" appears in both sides ","metadata":{}},{"cell_type":"markdown","source":"#### 1.4 Bigram analysis\nThe primary goal here is to analyse the trend of bigrams used in each category","metadata":{}},{"cell_type":"code","source":"def get_top_bigrams(corpus, n=None):\n    vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)\n    bag_of_words = vec.transform(corpus)\n    sum_words = bag_of_words.sum(axis=0) \n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n    return words_freq[:n]\n\nprint(\"Bigram analysis\")\n\nfig, axes = plt.subplots(1,len(df.target.unique()), figsize=(20,8))\nfig.suptitle('Bigram analysis')\n\nfor index,target in enumerate(df.target.unique()):\n    dct=defaultdict(int) \n    top_bigrams=get_top_bigrams(df[df['target']==target].question_text)[:50]\n    x,y=map(list,zip(*top_bigrams))\n    sns.barplot(ax=axes[index],x=y,y=x)\n    axes[index].set_title(\"Target: \"+str(target))\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:29:13.180970Z","iopub.execute_input":"2022-02-07T18:29:13.181172Z","iopub.status.idle":"2022-02-07T18:30:26.004020Z","shell.execute_reply.started":"2022-02-07T18:29:13.181148Z","shell.execute_reply":"2022-02-07T18:30:26.003179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference**\n- The data is highly populated with stop words, almost all of the top occurences have stop-words in them\n- Its surprising to see a good amount of region-specific, religion-specific and community-specific words. For instance, the word \"Donald Trump\" is very popular in `target=1` class *(ie, there is geographical bias; and we have to ensure that the model we make WILL NOT blindly imbibe this trait)*","metadata":{}},{"cell_type":"markdown","source":"#### 1.5 Other stuff\n\nHere, we analyse the following\n- Presence of HTML tags\n- Presence of URLs\n- Presence of emojis\n- Capitalisation\n- Punctuation-statistics","metadata":{}},{"cell_type":"code","source":"from emoji import UNICODE_EMOJI\n\ndef count_emojis(s):\n    count = 0\n    for emoji in UNICODE_EMOJI['en']:\n        count += s.count(emoji)\n    return count\n\nallvalues=[]\nplt.figure(figsize=(8,8))\nfor index,target in enumerate(df.target.unique()):\n    cur_dict=dict()\n    curdf=df[df['target']==target]  \n    \n    curdf.question_text=curdf.question_text.apply(lambda x: str(x))\n    df_caps=\" \".join(curdf.question_text.str.cat().split())\n    curdf.question_text=curdf.question_text.apply(lambda x: x.lower())\n    df_fullstring=curdf.question_text.str.cat()\n    cur_dict['HTML tags']=len(re.findall(\"<.*>\",df_fullstring))\n    cur_dict['URL']=len(re.findall(\"http\",df_fullstring))\n    cur_dict['Capitalised']=len(re.findall(\"[^\\.!\\?]\\s[A-Z]\\w+[\\W\\?:\\.!-_]\",df_caps))\n    cur_dict['Emojis']=count_emojis(df_caps)\n    \n    x_keys = list(cur_dict.keys())\n    y_values = list(cur_dict.values())\n    x_axis = np.arange(len(x_keys))\n    bars=plt.bar(x_axis - 0.2+0.4*index, y_values,0.4, label = 'Target: '+str(target))\n    allvalues+=y_values\n    for bar in bars:\n        yval = bar.get_height()\n        plt.text(bar.get_x()+0.15, yval+5, yval)\n\n\n    \nplt.xticks(x_axis, x_keys)\nplt.xlabel(\"Elements\")\nplt.ylabel(\"Quantity of occurence\")\nplt.legend()\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:30:26.005435Z","iopub.execute_input":"2022-02-07T18:30:26.005882Z","iopub.status.idle":"2022-02-07T18:30:48.253097Z","shell.execute_reply.started":"2022-02-07T18:30:26.005837Z","shell.execute_reply":"2022-02-07T18:30:48.252520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndef count_punctuations(s):\n    arr=list()\n    for punct in string.punctuation:\n        count = s.count(punct)\n        arr.append(count)\n    return arr\n\nallvalues=[]\nplt.figure(figsize=(20,20))\nfor index,target in enumerate(df.target.unique()):\n    cur_dict=dict()\n    curdf=df[df['target']==target]  \n    \n    curdf.question_text=curdf.question_text.apply(lambda x: str(x))\n    df_fullstring=curdf.question_text.str.cat()\n    \n    x_keys = string.punctuation\n    y_values = count_punctuations(df_fullstring)\n    x_axis = np.arange(len(x_keys))\n    bars=plt.bar(x_axis - 0.2+0.4*index, y_values,0.4, label = 'Target: '+str(target))\n    \n\n\n    \nplt.xticks(x_axis, x_keys)\nplt.xlabel(\"Punctuation\")\nplt.ylabel(\"Quantity of occurence\")\nplt.legend()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:30:48.254202Z","iopub.execute_input":"2022-02-07T18:30:48.254973Z","iopub.status.idle":"2022-02-07T18:30:51.148186Z","shell.execute_reply.started":"2022-02-07T18:30:48.254926Z","shell.execute_reply":"2022-02-07T18:30:51.147142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference**\n- HTML tags and HTTP URLs do occur in our dataset *(ignoring them might result in loss of significant data, hence we need to find out a way to treat them)*\n- There are A LOT of capitalised words, **indicating** the fact that there could be a huge number of proper nouns in the dataset *(Proper nouns like names of people, cities, etc are unseen entities with respect to pre-trained transformers like BERT. We may or may not lose information by lowercasing them)*\n- There are emojis in the dataset, but almost all of them are either of the following: `©`, `™`, `®` *(hence, we don't need to care about cleaning emoticons)*\n- Punctuations.... there's a huge load of punctuations. Its not at all surprising to see the Question mark symbol top the charts, because after all... Quora is for Questions!","metadata":{}},{"cell_type":"markdown","source":"### **EDA: the conclusion**\n\nWe gained the following knowledge by doing Exploratory Data Knowledge:\n\n- There are no null rows, and no duplicate rows\n- Oh boy, We have hell-a-lot of data imbalance! (~93% of `target=0` and ~7% of `target=1`)\n- There's bias, ie, we can see community specific and location specific terms skewed towards a category\n- Both unigram and bigram analysis shows the abundance of Stop Words *(Spoiler alert: we're not doing anything to treat it😉)*\n- We are unable to draw a clear relation between question length, average word length and target class *(they come in all shapes and sizes 🏈⚽)*\n- We have a few HTML tags, and a few HTTP URLs\n- We only have ©orporate emoticons™ in our dataset®\n- The dataset is enriched with punctuations, keeping them could hopefully contribute to the knowledge mining process❕‼","metadata":{}},{"cell_type":"markdown","source":"### STEPS 2 and above\n\nFor data pre-processing, feature engineering and model development pipeline of this problem statement, hop on to [the **Part two** of this notebook](https://www.kaggle.com/kodoorakiller/a-bertilicious-way)\n\n\n","metadata":{}},{"cell_type":"code","source":"# ---------------------------------------------------------------------------------------------------------\n# Copy/Edit if you find this notebook useful. Reach out via comments or email if you wish to!\n# Author: kodooraKILLER \n# ---------------------------------------------------------------------------------------------------------","metadata":{"execution":{"iopub.status.busy":"2022-02-07T18:30:51.149470Z","iopub.execute_input":"2022-02-07T18:30:51.149951Z","iopub.status.idle":"2022-02-07T18:30:51.155241Z","shell.execute_reply.started":"2022-02-07T18:30:51.149916Z","shell.execute_reply":"2022-02-07T18:30:51.154443Z"},"trusted":true},"execution_count":null,"outputs":[]}]}