{"cells":[{"metadata":{"id":"B8QW9LLwFmz1","colab_type":"text"},"cell_type":"markdown","source":"# Quora Insincere Questions Classification\n[Quora](https://www.quora.com/) is a platform that empowers people to learn from each other. On Quora, people can ask questions and connect with others who contribute unique insights and quality answers. A key challenge is to weed out insincere questions -- those founded upon false premises, or that intend to make a statement rather than look for helpful answers.\n[Competition: Quora Insincere Questions Classification](https://https://www.kaggle.com/c/quora-insincere-questions-classification)"},{"metadata":{"id":"6zRp4sMEROhu","colab_type":"text"},"cell_type":"markdown","source":"## Problem statement:\n  Build a model to classify whether a question asked on Quora is sincere or not.\n\nTo help Quora uphold their policy of “Be Nice, Be Respectful” and continue to be a place for sharing and growing the world’s knowledge."},{"metadata":{"id":"Kllm8423Tf-Z","colab_type":"text"},"cell_type":"markdown","source":"## Overview of the data:\n\nQuora provided a good amount of training and test data to identify the insincere questions. Train data consists of 1.3 million rows and 3 features in it. And the Test data consists of 300K rows and 2 features. "},{"metadata":{"id":"4JTIgvDWRpHY","colab_type":"text"},"cell_type":"markdown","source":"## Evaluation Metrics :\nMetric is F1 Score between the predicted and the observed targets. There are just two classes, but the positive class makes just over 6% of the total. So the target is highly imbalanced, which is why a metric such as F1 seems appropriate for this kind of problem as it considers both precision and recall of the test to compute the score.\n"},{"metadata":{"id":"dUc95v4WGvzA","colab_type":"text"},"cell_type":"markdown","source":"# 1. Data loading and exploration:"},{"metadata":{"id":"1K-0-sx2usif","colab_type":"code","outputId":"1f62b4a9-c401-46ba-e33b-80d866fd7028","colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport pandas as pd \n","execution_count":null,"outputs":[]},{"metadata":{"id":"iJN87LDFYBjh","colab_type":"text"},"cell_type":"markdown","source":"## Load the data from CSV files into a pandas dataframe"},{"metadata":{"id":"EYgWMi_9szrH","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_data = pd.read_csv('../input/quora-insincere-questions-classification/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"id":"y6swZ-chnR5Y","colab_type":"code","outputId":"eaea186b-670f-4440-bf3e-2bc3c5908c39","colab":{"base_uri":"https://localhost:8080/","height":359},"trusted":true},"cell_type":"code","source":"train_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"id":"255HIixQnUNO","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"#test_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"id":"Dvxz4oCs-21d","colab_type":"code","outputId":"b965e8e0-bb81-4a8e-822d-73de43961f9c","colab":{"base_uri":"https://localhost:8080/","height":51},"trusted":true},"cell_type":"code","source":"print(\"Train shape : \", train_data.shape)\nprint(\"Test shape : \", test_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"aao8vD19ywxD","colab_type":"code","outputId":"a7658452-f3a5-4f36-faad-e036579851a7","colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"train_data.columns","execution_count":null,"outputs":[]},{"metadata":{"id":"YuU7XRC5wR0E","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"train_data= train_data.drop(['qid'], axis=1)\ntest_data= test_data.drop(['qid'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"id":"YAK9L3E5xz2C","colab_type":"code","outputId":"d72a8aa3-77d3-4513-d39e-5ba40e349c21","colab":{"base_uri":"https://localhost:8080/","height":359},"trusted":true},"cell_type":"code","source":"test_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"id":"cMvImAvuzP0l","colab_type":"code","outputId":"4a5f18c9-9d48-432a-fbf5-1d197a60ef08","colab":{"base_uri":"https://localhost:8080/","height":68},"trusted":true},"cell_type":"code","source":"train_data.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"id":"W_ZRqXBlnmxM","colab_type":"code","outputId":"1d6e17c8-000f-40d3-8301-e80f791f7271","colab":{"base_uri":"https://localhost:8080/","height":51},"trusted":true},"cell_type":"code","source":"test_data.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"id":"9jK818zsHX9Y","colab_type":"text"},"cell_type":"markdown","source":"# 2. Data Visualization:"},{"metadata":{"id":"B0rFfvJ_W5-m","colab_type":"text"},"cell_type":"markdown","source":"## Target Count:"},{"metadata":{"id":"QFoNsKnWYfS-","colab_type":"code","outputId":"ff9a553a-0994-4acb-c51b-499a513468d9","colab":{"base_uri":"https://localhost:8080/","height":296},"trusted":true},"cell_type":"code","source":"sns.countplot(train_data['target'])","execution_count":null,"outputs":[]},{"metadata":{"id":"0M7NpjAhXXzg","colab_type":"code","outputId":"677756e1-e778-41e3-a374-72022cdb1845","colab":{"base_uri":"https://localhost:8080/","height":68},"trusted":true},"cell_type":"code","source":"train_data['target'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"id":"2jQdTPtGfVYQ","colab_type":"text"},"cell_type":"markdown","source":"## Target distribution:\n"},{"metadata":{"id":"NQ8NjlQquzLN","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"sincere_percent= (len(train_data.question_text[train_data['target'] == 0]) /  len(train_data['question_text']) * 100)\ninsincere_percent= (len(train_data.question_text[train_data['target'] == 1]) / len(train_data['question_text']) * 100)","execution_count":null,"outputs":[]},{"metadata":{"id":"gd9xigmVbp4S","colab_type":"code","outputId":"1635573a-d564-47b3-f109-a2f2eba62020","colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":true},"cell_type":"code","source":"print(sincere_percent, insincere_percent)","execution_count":null,"outputs":[]},{"metadata":{"id":"U0CItoGxat-h","colab_type":"code","outputId":"477fc7e4-3277-4329-b985-146918e63e04","colab":{"base_uri":"https://localhost:8080/","height":248},"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n# Data to plot\nlabels = 'Sincere', 'Insincere'\nsizes = [sincere_percent, insincere_percent]\ncolors = ['lightskyblue', 'lightcoral']\nexplode = (0.1, 0)  # explode 1st slice\n\n# Plot\nplt.pie(sizes, explode=explode, labels=labels, colors=colors,\nautopct='%1.1f%%', shadow=True, startangle=140)\n\nplt.axis('equal')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"l7IbQaM1fNL_","colab_type":"text"},"cell_type":"markdown","source":"## Word Frequency plot of sincere & insincere questions:\nLet us look at the frequently occuring words in the data by creating a word cloud on the 'question_text' column."},{"metadata":{"id":"77SKZQRAOcE1","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"import nltk\nfrom wordcloud import WordCloud, STOPWORDS","execution_count":null,"outputs":[]},{"metadata":{"id":"K0EZFHVYH2Xp","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"from collections import defaultdict\ntrain1_data = train_data[train_data[\"target\"]==1]\ntrain0_data = train_data[train_data[\"target\"]==0]","execution_count":null,"outputs":[]},{"metadata":{"id":"u-ckD1zPFHKQ","colab_type":"code","outputId":"43bc6fcf-9a09-493d-b645-19a30dec2eaa","colab":{"base_uri":"https://localhost:8080/","height":461},"trusted":true},"cell_type":"code","source":"def cloud(text, title, size = (10,7)):\n    # Processing Text\n    wordcloud = WordCloud(width=800, height=400, background_color ='white',\n                          collocations=False\n                         ).generate(\" \".join(text))\n    \n    # Output Visualization\n    fig = plt.figure(figsize=size, dpi=80)\n    plt.imshow(wordcloud,interpolation='bilinear')\n    plt.axis('off')\n    plt.title(title, fontsize=25,color='k')\n    plt.tight_layout(pad=0)\n    plt.show()\ncloud(train_data['question_text'], title=\"Word Cloud of Questions\")","execution_count":null,"outputs":[]},{"metadata":{"id":"LgmU78ZGFn7f","colab_type":"code","outputId":"bc45822c-9d4f-4bb8-f7e6-ebb8fb864236","colab":{"base_uri":"https://localhost:8080/","height":461},"trusted":true},"cell_type":"code","source":"cloud(train0_data[\"question_text\"], title=\"Word Cloud of sincere Questions\")","execution_count":null,"outputs":[]},{"metadata":{"id":"bxHAyWd_FyXR","colab_type":"code","outputId":"f7005226-26a6-4649-ee8c-879a08b25c4d","colab":{"base_uri":"https://localhost:8080/","height":461},"trusted":true},"cell_type":"code","source":"cloud(train1_data[\"question_text\"], title=\"Word Cloud of insincere Questions\")","execution_count":null,"outputs":[]},{"metadata":{"id":"JWvyStL_9Rja","colab_type":"text"},"cell_type":"markdown","source":"## Build language model:"},{"metadata":{"id":"Pzb9yMTUHzTr","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"## custom function for ngram generation ##\ndef generate_ngrams(text, n_gram=1):\n    token = [token for token in text.lower().split(\" \") if token != \"\" if token not in STOPWORDS]\n    ngrams = zip(*[token[i:] for i in range(n_gram)])\n    return [\" \".join(ngram) for ngram in ngrams]\n","execution_count":null,"outputs":[]},{"metadata":{"id":"mDBTQq6ie26J","colab_type":"text"},"cell_type":"markdown","source":"## Unigram model:"},{"metadata":{"id":"gZvNBbVm6IH0","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"## Get the bar chart from sincere questions ##\nfreq_dict = defaultdict(int)\nfor sent in train0_data[\"question_text\"]:\n    for word in generate_ngrams(sent):\n        freq_dict[word] += 1\nfd_sorted0 = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted0.columns = [\"word\", \"wordcount\"]\n\n## Get the bar chart from insincere questions ##\nfreq_dict = defaultdict(int)\nfor sent in train1_data[\"question_text\"]:\n    for word in generate_ngrams(sent):\n        freq_dict[word] += 1\nfd_sorted1 = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted1.columns = [\"word\", \"wordcount\"]\n","execution_count":null,"outputs":[]},{"metadata":{"id":"_gxawpX-H41y","colab_type":"code","outputId":"932018a5-0a74-4aea-a34f-e36e2cffc480","colab":{"base_uri":"https://localhost:8080/","height":638},"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(11,10))\nplt.title(\"Frequent words of sincere question\")\nfd_sorted0_head= fd_sorted0.head(40)\nsns.barplot(x=fd_sorted0_head['wordcount'], y=fd_sorted0_head['word'])","execution_count":null,"outputs":[]},{"metadata":{"id":"lb5LIIKPVaEq","colab_type":"code","outputId":"4d0b53fb-5913-448c-fc3f-e46dd317f974","colab":{"base_uri":"https://localhost:8080/","height":638},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(11,10))\nplt.title(\"Frequent words of insincere question\")\nfd_sorted1_head= fd_sorted1.head(40)\nsns.barplot(x=fd_sorted1_head['wordcount'], y=fd_sorted1_head['word'])","execution_count":null,"outputs":[]},{"metadata":{"id":"C71aR3l1fBXc","colab_type":"code","colab":{}},"cell_type":"markdown","source":"## Bigram model:"},{"metadata":{"id":"_3EdiYZpZZxA","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"freq_dict = defaultdict(int)\nfor sent in train0_data[\"question_text\"]:\n    for word in generate_ngrams(sent,2):\n        freq_dict[word] += 1\nfd_sorted0 = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted0.columns = [\"word\", \"wordcount\"]\n\n## Get the bar chart from insincere questions ##\nfreq_dict = defaultdict(int)\nfor sent in train1_data[\"question_text\"]:\n    for word in generate_ngrams(sent,2):\n        freq_dict[word] += 1\nfd_sorted1 = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted1.columns = [\"word\", \"wordcount\"]","execution_count":null,"outputs":[]},{"metadata":{"id":"IhP-EBJ8b5_R","colab_type":"code","outputId":"609cfafa-2a42-4dea-c717-3ba3c82a2d5e","colab":{"base_uri":"https://localhost:8080/","height":638},"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(11,10))\nplt.title(\"Frequent words of sincere question\")\nfd_sorted0_head= fd_sorted0.head(40)\nsns.barplot(x=fd_sorted0_head['wordcount'], y=fd_sorted0_head['word'])","execution_count":null,"outputs":[]},{"metadata":{"id":"B_gkeBTXb7Mi","colab_type":"code","outputId":"cf963401-8fe4-44dd-87b9-b9ba16b94b5b","colab":{"base_uri":"https://localhost:8080/","height":638},"trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(11,10))\nplt.title(\"Frequent words of insincere question\")\nfd_sorted1_head= fd_sorted1.head(40)\nsns.barplot(x=fd_sorted1_head['wordcount'], y=fd_sorted1_head['word'])","execution_count":null,"outputs":[]},{"metadata":{"id":"6eo14rpVlp9s","colab_type":"code","colab":{},"trusted":false},"cell_type":"code","source":"","execution_count":0,"outputs":[]}],"metadata":{"colab":{"name":"Quora Insincere Questions Classification_Visualization_V1.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"nbformat":4,"nbformat_minor":1}