An existential problem for any major website today is how to handle toxic and divisive content. Quora wants to tackle this problem head-on to keep their platform a place where users can feel safe sharing their knowledge with the world.Quora is a platform that empowers people to learn from each other. On Quora, people can ask questions and connect with others who contribute unique insights and quality answers. A key challenge is to weed out insincere questions -- those founded upon false premises, or that intend to make a statement rather than look for helpful answers.
\n
In this competition, Kagglers will develop models that identify and flag insincere questions. To date, Quora has employed both machine learning and manual review to address this problem. With your help, they can develop more scalable methods to detect toxic and misleading content.
\n
Here's your chance to combat online trolls at scale. Help Quora uphold their policy of “Be Nice, Be Respectful” and continue to be a place for sharing and growing the world’s knowledge.
\n
\n\n
Outline of the Notebook
\n\n
Objective : Predict whether a question asked on Quora is sincere or not. This is a kernels only comeptition.
\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-qiqc\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n# https://www.kaggle.com/arunkumarramanan/market-data-nn-baseline\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nfrom vowpalwabbit import pyvw\nimport sklearn.metrics as metrics\nfrom gensim.parsing import preprocessing as prep\nimport plotly.tools as tls\nimport warnings\n# from plotly.tools import FigureFactory as FF \nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nplt.style.use(\"fivethirtyeight\")\n%matplotlib inline\nimport seaborn as sns\nimport numpy as np\nimport plotly.figure_factory as ff\n\n\n######### Function\ndef mis_value_graph(data):\n# data.isnull().sum().plot(kind=\"bar\", figsize = (20,10), fontsize = 20)\n# plt.xlabel(\"Columns\", fontsize = 20)\n# plt.ylabel(\"Value Count\", fontsize = 20)\n# plt.title(\"Total Missing Value By Column\", fontsize = 20)\n# for i in range(len(data)):\n# colors.append(generate_color())\n \n data = [\n go.Bar(\n x = data.columns,\n y = data.isnull().sum(),\n name = 'Unknown Assets',\n textfont=dict(size=20),\n marker=dict(\n# color= colors,\n line=dict(\n color=generate_color(),\n width=2,\n ), opacity = 0.45\n )\n ),\n ]\n layout= go.Layout(\n title= '\"Total Missing Value By Column\"',\n xaxis= dict(title='Columns', ticklen=5, zeroline=False, gridwidth=2),\n yaxis=dict(title='Value Count', ticklen=5, gridwidth=2),\n showlegend=True\n )\n fig= go.Figure(data=data, layout=layout)\n py.iplot(fig, filename='skin')\n \n\ndef mis_impute(data):\n for i in data.columns:\n if data[i].dtype == \"object\":\n data[i] = data[i].fillna(\"other\")\n elif (data[i].dtype == \"int64\" or data[i].dtype == \"float64\"):\n data[i] = data[i].fillna(data[i].mean())\n else:\n pass\n return data\n\n\nimport random\n\ndef generate_color():\n color = '#{:02x}{:02x}{:02x}'.format(*map(lambda x: random.randint(0, 255), range(3)))\n return color","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0fd001524ee68affb1fbc49d31226333cdb7686"},"cell_type":"markdown","source":"
1.Data Reading and Investigation
"},{"metadata":{"trusted":true,"_uuid":"a1766d07c2c1d5cd9c401b9e3a24d664020b3e90"},"cell_type":"code","source":"train = pd.read_csv(\"../input/quora-insincere-questions-classification/train.csv\")\nmis_value_graph(train)\nprint(\"Train Shape:\",train.shape)\ndisplay(train.isna().sum().to_frame())\nprint(\"=====Train Data Column Types=====\")\ndisplay(train.dtypes)\nprint(\"=====Train Data=====\")\ndisplay(train.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"30eed18ebe0e0a572ec6d151b8883a860d96aaf4","_kg_hide-input":false},"cell_type":"code","source":"test = pd.read_csv(\"../input/quora-insincere-questions-classification/test.csv\")\nmis_value_graph(train)\nprint(\"Test Shape:\",test.shape)\ndisplay(test.isna().sum().to_frame())\nprint(\"=====Test Data Column Types=====\")\ndisplay(test.dtypes)\nprint(\"=====Test Data=====\")\ndisplay(train.head())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f84e8824d30b21ea1f7ee6c453a0d3f39667de31"},"cell_type":"markdown","source":"
"},{"metadata":{"trusted":true,"_uuid":"058613d6d390f8b279e118d55be0e134823c1421","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"# https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-qiqc\nfrom collections import defaultdict\ntrain1_df = train[train[\"target\"]==1]\ntrain0_df = train[train[\"target\"]==0]\n\n## custom function for ngram generation ##\ndef generate_ngrams(text, n_gram=1):\n token = [token for token in text.lower().split(\" \") if token != \"\" if token not in STOPWORDS]\n ngrams = zip(*[token[i:] for i in range(n_gram)])\n return [\" \".join(ngram) for ngram in ngrams]\n\n## custom function for horizontal bar chart ##\ndef horizontal_bar_chart(df, color):\n trace = go.Bar(\n y=df[\"word\"].values[::-1],\n x=df[\"wordcount\"].values[::-1],\n showlegend=False,\n orientation = 'h',\n marker=dict(\n color=[i for j in range(100) for i in ['#aa2200','#2b6dad',generate_color(),generate_color(),generate_color()]],\n ),\n )\n return trace\n\n## Get the bar chart from sincere questions ##\nfreq_dict = defaultdict(int)\nfor sent in train0_df[\"question_text\"]:\n for word in generate_ngrams(sent):\n freq_dict[word] += 1\nfd_sorted = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted.columns = [\"word\", \"wordcount\"]\nprint(\"Frequency Trigram for Sincere Question\")\ndisplay(fd_sorted.head(10))\ntrace0 = horizontal_bar_chart(fd_sorted.head(30), 'blue')\n\n## Get the bar chart from insincere questions ##\nfreq_dict = defaultdict(int)\nfor sent in train1_df[\"question_text\"]:\n for word in generate_ngrams(sent):\n freq_dict[word] += 1\nfd_sorted = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted.columns = [\"word\", \"wordcount\"]\nprint(\"Frequency Trigram for Insincere Question\")\ndisplay(fd_sorted.head(10))\ntrace1 = horizontal_bar_chart(fd_sorted.head(50), color = 'blue')\n\n# Creating two subplots\nfig = tls.make_subplots(rows=1, cols=2, vertical_spacing=0.04,subplot_titles=[\"Frequent words of sincere questions\",\"Frequent words of insincere questions\"])\nfig.append_trace(trace0, 1, 1)\nfig.append_trace(trace1, 1, 2)\nfig['layout'].update(height=1200, width=900,title=\"Word Count Plots\")\npy.iplot(fig, filename='word-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b6569e2d860965753dc56967dbdce4ad345e8ef1"},"cell_type":"markdown","source":"
Words Counts Insights
\n\n
\n
We can see that their are so many comman words people use to write their question here.
\n
In Insincere Graph you can check insincere bar plot `people,trump, women, will, think, many, white, men, indian. muslims` are most used words more than 4k times.
\n
In Sincere Graphyou can check Sincere bar plot `Best, will, people, good, one, make, think, many, much, someone etc.` are most used words more than 30k times.
\n
\n
5. Bi-gram by Sincere Vs Insincere
"},{"metadata":{"trusted":true,"_uuid":"f7aaed035a056c0d4c0c002380c467b91d90cba6","_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"freq_dict = defaultdict(int)\nfor sent in train0_df[\"question_text\"]:\n for word in generate_ngrams(sent,2):\n freq_dict[word] += 1\nfd_sorted = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted.columns = [\"word\", \"wordcount\"]\nprint(\"Frequency Trigram for Sincere Question\")\ndisplay(fd_sorted.head(10))\ntrace0 = horizontal_bar_chart(fd_sorted.head(30), generate_color())\n\n\nfreq_dict = defaultdict(int)\nfor sent in train1_df[\"question_text\"]:\n for word in generate_ngrams(sent,2):\n freq_dict[word] += 1\nfd_sorted = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted.columns = [\"word\", \"wordcount\"]\nprint(\"Frequency Trigram for Insincere Question\")\ndisplay(fd_sorted.head(10))\ntrace1 = horizontal_bar_chart(fd_sorted.head(30), generate_color())\n\n# Creating two subplots\nfig = tls.make_subplots(rows=1, cols=2, vertical_spacing=0.04,horizontal_spacing=0.15,\n subplot_titles=[\"Frequent bigrams of sincere questions\", \n \"Frequent bigrams of insincere questions\"])\nfig.append_trace(trace0, 1, 1)\nfig.append_trace(trace1, 1, 2)\nfig['layout'].update(height=1200, width=900,title=\"Bigram Word pair Count Plots\")\npy.iplot(fig, filename='word-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97162b7a1257eb0808be3e2fecdefb4ca2b8a4c5"},"cell_type":"markdown","source":"
Bigram Insights
\n
\n
A bigram is a sequence of two adjacent elements from a string of tokens, which are typically letters, syllables, or words. A bigram is an n-gram for n=2
\n
In Sincere Graph you can check Sincere bar plot `Best way, Year old, will happen` are most used words more than 2000 times.
\n
In Insincere Graph you can check Insincere bar plot `Donald Trump, White Perople, black people, many people etc.` are most used words more than 500 times.
\n\n
6. Tri-gram by Sincere Vs Insincere
"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"ccb54a49fef1d82c24b07cee1bbaf98aefc79bd6"},"cell_type":"code","source":"freq_dict = defaultdict(int)\nfor sent in train0_df[\"question_text\"]:\n for word in generate_ngrams(sent,3):\n freq_dict[word] += 1\nfd_sorted = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted.columns = [\"word\", \"wordcount\"]\nprint(\"Frequency Trigram for Sincere Question\")\ndisplay(fd_sorted.head(10))\ntrace0 = horizontal_bar_chart(fd_sorted.head(30), 'green')\n\n\nfreq_dict = defaultdict(int)\nfor sent in train1_df[\"question_text\"]:\n for word in generate_ngrams(sent,3):\n freq_dict[word] += 1\nfd_sorted = pd.DataFrame(sorted(freq_dict.items(), key=lambda x: x[1])[::-1])\nfd_sorted.columns = [\"word\", \"wordcount\"]\nprint(\"Frequency Trigram for Insincere Question\")\ndisplay(fd_sorted.head(10))\ntrace1 = horizontal_bar_chart(fd_sorted.head(30), 'green')\n\n# Creating two subplots\nfig = tls.make_subplots(rows=1, cols=2, vertical_spacing=0.5, horizontal_spacing=0.2,\n subplot_titles=[\"Frequent trigrams of sincere questions\", \n \"Frequent trigrams of insincere questions\"])\nfig.append_trace(trace0, 1, 1)\nfig.append_trace(trace1, 1, 2)\nfig['layout'].update(height=1200, width=900, title=\"Trigram Count Plots\")\npy.iplot(fig, filename='word-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"441344e9a090ffa006318a924e3c7cf9506a2fc0"},"cell_type":"markdown","source":"
Trigram Insights
\n
\n
A Trigra is a sequence of three adjacent elementsfrom a string of tokens, which are typically letters, syllables, or words. A Trigramis an n-gram for n=3
\n
In Sincere Graph you can check Sincere bar plot `tips someone starting, someone starting work, useful tips someone, advice give someone,short-term business travelers, hotels short-term business, good hotels short-term, give someone moving, good bad neighborhoods, best known for? etc.`*are most used words more than 400 times.
\n
In Insincere Graph you can check Sincere bar plot `will donald trump, black lives matter, long will take, kim jong un,12 year old, people still believe,14 year old,united states america.`are most used words more than 30 times.
Now let us create some meta features and then look at how they are distributed between the classes. The ones that we will create are
\n
Number of words in the text
\n
Number of unique words in the text
\n
Number of characters in the text
\n
Number of stopwords
\n
Number of punctuations
\n
Number of upper case words
\n
Number of title case words
\n
Average length of the words
\n
"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"589fc7d529cb3678ec7b08f48452d887a230fa2e"},"cell_type":"code","source":"import string\n## Number of words in the text ##\ntrain[\"num_words\"] = train[\"question_text\"].apply(lambda x: len(str(x).split()))\ntest[\"num_words\"] = test[\"question_text\"].apply(lambda x: len(str(x).split()))\n\n## Number of unique words in the text ##\ntrain[\"num_unique_words\"] = train[\"question_text\"].apply(lambda x: len(set(str(x).split())))\ntest[\"num_unique_words\"] = test[\"question_text\"].apply(lambda x: len(set(str(x).split())))\n\n## Number of characters in the text ##\ntrain[\"num_chars\"] = train[\"question_text\"].apply(lambda x: len(str(x)))\ntest[\"num_chars\"] = test[\"question_text\"].apply(lambda x: len(str(x)))\n\n## Number of stopwords in the text ##\ntrain[\"num_stopwords\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).lower().split() if w in STOPWORDS]))\ntest[\"num_stopwords\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).lower().split() if w in STOPWORDS]))\n\n## Number of punctuations in the text ##\ntrain[\"num_punctuations\"] =train['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]) )\ntest[\"num_punctuations\"] =test['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]) )\n\n## Number of title case words in the text ##\ntrain[\"num_words_upper\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\ntest[\"num_words_upper\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\n\n## Number of title case words in the text ##\ntrain[\"num_words_title\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\ntest[\"num_words_title\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\n\n## Average length of the words in the text ##\ntrain[\"mean_word_len\"] = train[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))\ntest[\"mean_word_len\"] = test[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))\n\nprint(\"Train shape:\",train.shape)\nprint(\"Test Shape:\",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ea814ca651a746e7e01493f890236d0e6ce1adb"},"cell_type":"markdown","source":"
8. Model Training using Vowpal Wabbit Algorithm
\n
\n
Vowpal Wabbit. Vowpal Wabbit (also known as \"VW\") is an open source fast out-of-core machine learning system library and program developed originally at Yahoo! Research, and currently at Microsoft Research. It was started and is led by John Langford.
\n
The Vowpal Wabbit (VW) is a project started at Yahoo! Research and now sponsored by Microsoft Research. Started and led by John Langford, VW focuses on fast learning by building an intrinsically fast learning algorithm. John gave two guest lectures to us on AllReduce and Bandits during NYU Big Data class this semester. From what I see, he is a reputed researcher and really passionate about online learning algorithm.
\n
The Vowpal Wabbit name is oddly pronounced and strange. Langford explained that Vowpal Wabbit is how Elmer Fudd would pronounce “Vorpal Rabbit”. As for “Vorpal”, if you Google it, you will find “Vorpal Bunny”, which is also known as a “killer rabbit” in a popular computer game. Maybe this is exactly what he wants VW to be – cute but also powerful and fast.
\n
VW supports a number of machine learning problems, importance weighting, a selection of loss functions and optimization algorithms, like SGD (Stochastic Gradient Descent), BFGS (a a popular algorithm for parameter estimation), conjugate gradient etc. It has been used to learn a sparse terafeature (i.e. 1012 sparse features) dataset on 1000 nodes in one hour, which beats all current machine linear learning algorithms. According to its tutorial on John Langford’s GitHub, VW is about a factor of 3 faster than svmsgd on the RCV1 example, which is a collection for text categorization.
\n
For More Reading : https://www.zinkov.com/posts/2013-08-13-vowpal-tutorial/
"},{"metadata":{"trusted":true,"_uuid":"6b5643f65a74ea1d4fa72324ad81225845f2a492"},"cell_type":"code","source":"%%time\nfor fit_iter in range(7):\n for num, feature in enumerate(make_vw_corpus(X_train['question_text'], X_train['target'])):\n ex = vw.example(feature)\n vw.learn(ex)\n ex.finish()\n \n print('pass num {} done'.format(fit_iter))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"60373a7176e45de4f53ed838637058c007e09888"},"cell_type":"code","source":"pred = np.array([get_pred(x) for x in make_vw_corpus(X_test['question_text'], X_test['target'])])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea073fe5bc58085256c423fcaed69660774bd6ff"},"cell_type":"code","source":"thresholds = np.linspace(0, 1, 100)\nf1_scores = [metrics.f1_score(X_test['target'], pred > threshold) for threshold in thresholds]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"504cf62e6c94bbc148d35a06a48533ea74e78fca"},"cell_type":"markdown","source":"
10. F1-Score Graph
"},{"metadata":{"trusted":true,"_uuid":"30ccf97d9c16259bcafdbe77c858621b03e410a7"},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nplt.plot(thresholds, f1_scores)\nplt.grid(True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7d87bd71e38f6535af1bb366f2f714edf6337f91"},"cell_type":"code","source":"print('best f1 score is {} with threshold {}'.format(np.max(f1_scores), thresholds[np.argmax(f1_scores)]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"494437d0f469c992edd01669a43c841918749bd6"},"cell_type":"markdown","source":"
11. Final Submission
"},{"metadata":{"trusted":true,"_uuid":"f02644ef1d47aea6d38ecf5083e109dab4744077"},"cell_type":"code","source":"test['question_text'] = test['question_text'].apply(tokenizer)\ntest['question_text'] = test['question_text'].apply(filter_words)\n\npred = np.array([get_pred(x) for x in make_vw_corpus(test['question_text'], [1] * len(test))])\n\nexample = pd.read_csv('../input/quora-insincere-questions-classification/sample_submission.csv')\nexample['prediction'] = (pred > thresholds[np.argmax(f1_scores)]).astype(int)\nexample.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08ad46d430858d4239ace6e4b60bf88e01e85cc0"},"cell_type":"markdown","source":"### Suggestions are welcome...I want to improve this kernel more...Upvote if you like it!!!\n### Thanks for reading🙏🙏🙏"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}