{"cells":[{"metadata":{"_uuid":"9f8d043cdaae45df19c6993f0e04897fa88fbfc9"},"cell_type":"markdown","source":"# Quora insincere questions"},{"metadata":{"_uuid":"67b686f96821b15891a6f27a45b3e29ee36b9d4a"},"cell_type":"markdown","source":"### Objective of the competition : predict if a question is insincere (1) or not (0) to help the moderation system of Quora. \n**Parameters to evaluate the insincerity:**\n* non-neutral tone (exagerated or rhetorical)\n* disparaging or inflamatory questions (suggest discriminatory idea, hateful, based on prejudices)\n* not grounded in reality (absurd or false)\n* sexual content for shock value + illicite sexual content"},{"metadata":{"_uuid":"f1222da4c522678f79e0e29b29bb8db2d43d4007"},"cell_type":"markdown","source":"I'm new to machine learning so i began by doing a topic modeling with gensim to see if i can identify recurent topics differencing sincere and insincere question.\n\nThe machine learning part is a more concise cause i wanted to focus on topic modeling. "},{"metadata":{"_uuid":"b3e771a30125f304f8cde60f9966326fe4b3ff17"},"cell_type":"markdown","source":"# I. Imports and loading the data"},{"metadata":{"_uuid":"577a8fe33d17fcea54a0ec894bcbaf330daa614c","trusted":true},"cell_type":"code","source":"import os\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e5700fd6179086268b1e14b8ca5a3b8ea4f86b2","trusted":true},"cell_type":"code","source":"#from vivadata.datasets.common import get_path_for_dataset\n#base_path = get_path_for_dataset('quora')\nX_train_filepath = os.path.join('..', 'input', 'train.csv')\nX_test_filepath = os.path.join('..', 'input', 'test.csv')\nsample_filepath = os.path.join('..', 'input', 'sample_submission.csv')\nX_train_filepath, X_test_filepath, sample_filepath","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b7358f5b361b068c1599bdb279da32d95819e68","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(X_train_filepath)\ndf_test = pd.read_csv(X_test_filepath)\nsample = pd.read_csv(sample_filepath)\ndf_train.shape, df_test.shape, sample.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"79a4101d4cdbdbb5a971b413e0f7eaeb6eece0fb"},"cell_type":"markdown","source":"Now that the data is loaded, let's take a quick look. \n![](https://media.giphy.com/media/94hqi5hBHu5W/giphy.gif)"},{"metadata":{"_uuid":"5ee5841bcb36c99667bae629b4003cc794c32d15"},"cell_type":"markdown","source":"#  II.EDA"},{"metadata":{"_uuid":"501206bff3ac1dffebab97152a3fe0afda23c790","trusted":true,"scrolled":true},"cell_type":"code","source":"df_train.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c6df2de49f99794c190e045a3f38f5c214992710"},"cell_type":"markdown","source":"- **qid** = unique question identifier\n- **question_text** = text of the question\n- **target** = 1: insincere question, 0:not insincere question"},{"metadata":{"_uuid":"2b29963e5eb52e85d83dbd9237843e3d1b208a78","scrolled":true,"trusted":true},"cell_type":"code","source":"df_test.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3f7bf37785af749359209cb18bbd5ca240a3271"},"cell_type":"markdown","source":"**qid** = unique question identifier\n**question_text** = text of the question"},{"metadata":{"_uuid":"4032f753b58f5eb4ee1189d5ea79302141de19dd","trusted":true},"cell_type":"code","source":"df_train.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e5c2307fef25939036b586432009eb9d8da2b4d","trusted":true},"cell_type":"code","source":"df_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7da2279d132afba499ca994360c2421188446407","trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(10, 7))\nsns.countplot(x='target', data=df_train)\nplt.title('Reparition of question by insincerity');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"663b61c6f71fcb951fc9f63d7b600253205cd9db"},"cell_type":"markdown","source":"There is 80 810 insincere questions on 1 306 122, so around 6% (0.0618). This would be a problem for the machin learning process cause it's an important disparity with 1 225 312 sincere ones. "},{"metadata":{"_uuid":"a36adf7ade1ba2427a7ceb518be2eac3c3b46b9a"},"cell_type":"markdown","source":"**_Now, we need to transform the data from text to something the computer could understand in order to study it more precisely._**"},{"metadata":{"_uuid":"e556d7a2327e43a2acabfc125684edbaaae4a403"},"cell_type":"markdown","source":"# III.Topic modelisation"},{"metadata":{"_uuid":"d8c1af96d3f57eafbd5591b2d75165e489f8e923"},"cell_type":"markdown","source":"I'm gonna prepare the data,  then use gensim and lda modeling to create topics for sincere and insincere questions. "},{"metadata":{"_uuid":"4652a1c388aa3a8c8d8c91dc36a3451f33bad00f","trusted":true},"cell_type":"code","source":"import gensim\nimport nltk\nfrom nltk.corpus import stopwords","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b83a3c1f52bb59188bf5872e9657170594f38ed0","trusted":true},"cell_type":"code","source":"# Define the target and the variable.\ny_train = df_train.loc[:, 'target']\nX_train = df_train.loc[:, 'question_text']\nX_train.shape, y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"157c4d713f540ced023046b8b096aca17767a8c0","trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4a087f1198e4394d1a79eb3974d1a036a783eea","scrolled":true,"trusted":true},"cell_type":"code","source":"# Create a variable with all the sincere questions.\nX_train_sincere = X_train[df_train['target'] == 0]\nX_train_sincere[:5]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd1d2b1742345340428b655f7af9ddb5b316c0ec","scrolled":true,"trusted":true},"cell_type":"code","source":"# Create a variable with all insincere questions.\nX_train_insincere = X_train[df_train['target'] == 1]\nX_train_insincere[:5]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"914b3cc960bb6608df0233c24ceae86b26440770"},"cell_type":"markdown","source":"## III-A. Topic modeling with 10 topics"},{"metadata":{"_uuid":"0999d94e0bec5352a3201f39a25da7ac2edcf2d7"},"cell_type":"markdown","source":"### III-A. 1. Tokenisation"},{"metadata":{"_uuid":"115305e95e861f1038833d578dade91aa4c29151"},"cell_type":"markdown","source":"We need to transform the text data in token to be able to compute it. "},{"metadata":{"_uuid":"f9c149e7adee6b4edeec9fe6f88ee34432733cf2","trusted":true},"cell_type":"code","source":"# First i load the list of stopwords, the words which aren't useful to understand the meaning.\nstop_words = stopwords.words('english')\nstop_words[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"20b79966634510ca2de90a0507c92e908626833b","scrolled":true,"trusted":true},"cell_type":"code","source":"# I transform the sincere questions in a list of words.\nsincere_prepro_questions = [gensim.utils.simple_preprocess(question) \n                            for question in X_train_sincere]\nsincere_prepro_questions[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cac7c930067848b035ba4bd0ffecd91bc2236667","trusted":true},"cell_type":"code","source":"# I transform the insincere questions in a list of words.\ninsincere_prepro_questions = [gensim.utils.simple_preprocess(question) \n                              for question in X_train_insincere]\ninsincere_prepro_questions[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6bccf903b7db840a6946ba9be130cc640d1a0c4","trusted":true},"cell_type":"code","source":"# Verifying the length of the both lists.\nlen(sincere_prepro_questions), len(insincere_prepro_questions)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"47bb008071e10b6f0cbb0bb79b1ff23c0ed5ab3f","trusted":true},"cell_type":"code","source":"# I remove the stopwords from the sincere questions list of words.\nclear_sincere_questions = [[word for word in question if word not in stop_words] \n                             for question in sincere_prepro_questions]\nclear_sincere_questions[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0cd2135f9bf8c56224e80d1f954bc6db29506691","trusted":true},"cell_type":"code","source":"# I do the same for insincere one.\nclear_insincere_questions = [[word for word in question if word not in stop_words] \n                             for question in insincere_prepro_questions]\nclear_insincere_questions[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c6fc8c66f22064970372aa26df1ea4aa93c4eb2"},"cell_type":"markdown","source":"### III-A.2. Creation of gensim dictionnaries"},{"metadata":{"_uuid":"975ad3662530be084a500edeac33ffa4d3cdc506","trusted":true},"cell_type":"code","source":"# Creation of a dictionnary of the words and their id for sincere questions\nsincere_questions_dictionary = gensim.corpora.Dictionary(clear_sincere_questions)\nsincere_token = sincere_questions_dictionary.token2id","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12d1ee97d13307701bc352fa0c97eef3650d8a9e","trusted":true},"cell_type":"code","source":"# Association of the dictonary words and their frequency for sincere questions\nsincere_dict_frequency = {sincere_questions_dictionary[k]: v for k,v in sincere_questions_dictionary.dfs.items()}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87e11d7abf983d0f8edc396926dfbbd5ba3f3f40","trusted":true},"cell_type":"code","source":"# Creation of a dictionnary of the words and their id for insincere questions\ninsincere_questions_dictionary = gensim.corpora.Dictionary(clear_insincere_questions)\ninsincere_token = insincere_questions_dictionary.token2id","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c207320d9210d9831b949aaa953fed9bb403590b","trusted":true},"cell_type":"code","source":"# Association of the dictonary words and their frequency for insincere questions\ninsincere_dict_frequency = {sincere_questions_dictionary[k]: v for k,v in sincere_questions_dictionary.dfs.items()}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd019436f022eaefd05ecb321add84b895f4f787"},"cell_type":"markdown","source":"### III-A.3. Vectorization"},{"metadata":{"_uuid":"7946402dc8bc6163c8e6f5febba1ca6a822a040c","trusted":true},"cell_type":"code","source":"# I transform the elements of the sincere dictionary in vectors. \nsincere_corpus = [sincere_questions_dictionary.doc2bow(question) \n                  for question in clear_sincere_questions]\nsincere_corpus[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80e1b12bfda53e1fe576b1cdc15c308cec5605d4","scrolled":true,"trusted":true},"cell_type":"code","source":"# I do the same for the insincere questions. \ninsincere_corpus = [insincere_questions_dictionary.doc2bow(question) \n                    for question in clear_insincere_questions]\ninsincere_corpus[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8494cc3162710b5ff8f8854922ee47a15ffa2f5e"},"cell_type":"markdown","source":"### III-A.4. Lda Modeling"},{"metadata":{"_uuid":"b8db83be80027f90fac5bfda44c56cc8aa1b39fd"},"cell_type":"markdown","source":"Lda modeling is a statistical modeling which discovers abstract \"topics\" in documents. Here Lda modeling will help me to see if sincere and insincere questions have some majors topics. \n\nThis topics could be used as a variable later, for the machine learning."},{"metadata":{"_uuid":"7512625e7742c9fea90e22467535bbbf8dc858dd","scrolled":true,"trusted":true},"cell_type":"code","source":"# Creating a lda model, which create 10 topics from the sincere questions\nlda_model_sincere = gensim.models.ldamodel.LdaModel(\n    corpus=sincere_corpus, num_topics=10, id2word=sincere_questions_dictionary,\n    random_state=25, passes=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"48c57666ba81ea5e43c7e23806bc10fc0ec4c747","trusted":true},"cell_type":"code","source":"# Creating a lda model, which create 10 topics from the insincere questions\nlda_model_insincere = gensim.models.ldamodel.LdaModel(\n    corpus=insincere_corpus, num_topics=10, id2word=insincere_questions_dictionary,\n    random_state=25, passes=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4fbecf59674ac58d475b378753165e74110fdad8","scrolled":false,"trusted":true},"cell_type":"code","source":"from pprint import pprint","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8027790e4cd246ca563da6b042a435ad6162a31","trusted":true},"cell_type":"code","source":"# Printing the result of the modeling for sincere questions.\npprint(lda_model_sincere.print_topics()[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2fcf8b751e48b29fa8edea9b3d215b9059b2db4","trusted":true},"cell_type":"code","source":"# Same for insincere ones. \npprint(lda_model_insincere.print_topics()[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2d1be01809367bf8177af4acae6e344d655c307"},"cell_type":"markdown","source":"### III-A.5. Visualization\n\nI will now measure how good the topic modeling is and create a visual of the topics. \n\nMeasurement of how good a topic model is. \n* Coherence score : \n    * intrinsic measure: compare a word with the preceding & sucseeding ones. Calculate a log probability\n    * extrinsic measure: every single word is paired with every others. Use pointwise mutual information. "},{"metadata":{"_uuid":"620b9008db74b11971fe3e30decfc42c3c5a2696","trusted":true},"cell_type":"code","source":"import pyLDAvis.gensim\npyLDAvis.enable_notebook()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab8ceeffd47484137210d821caae8c8463e0350e","trusted":true},"cell_type":"code","source":"pyLDAvis.gensim.prepare(lda_model_sincere, sincere_corpus, sincere_questions_dictionary)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f071f77b707bb2e7416997856a1c8122afabfac"},"cell_type":"markdown","source":"Some topics overlapt each other (1 &4, 9&7, 2&3). The topic are less easily recognizable than for insincere questions. We can \nrecognize word showing interogation, some vocabulary about computer... but the vocabulary is larger and more miscellaneous. "},{"metadata":{"_uuid":"5d9eae1cebca57474211cac2f7b52a62d3b772ed","trusted":true},"cell_type":"code","source":"pyLDAvis.gensim.prepare(lda_model_insincere, insincere_corpus, \n                        insincere_questions_dictionary)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee31e751e2fc59907b736c9d822519afaab0687b"},"cell_type":"markdown","source":"The number of topics are not optimal, we can see that some topics (1, 3, 7 and 2, 5) overlapt. Maybe those subjects could\ncould have be reunited in a same topics. \n\nThe 1, 3, 7 seems to be about politic:\n    * the first about Donald Trump;\n    * the third about international politic;\n    * the seventh about political parties. \n We also see that the 30 most relevant words are sometimes repeated in the 3 topics ('trump', 'liberals', 'president', 'supporters'). \n\nThe 2, 5 topics seems to be about racial and religious subject:\n    * the second refers to race ('white', 'black', 'chinese');\n    * and the second seems to be about Islam ('muslims', 'muslim', 'pakistant'). \n   \nTopics 8 and 6, even if they are not close to the 1st cluster contain a lot of similar words and seems to also be about politics:\n       * the 8th about Donald Trump's election also ('trump', 'president', 'obama')\n       * the 6th about interior politic and social subjects ('gun', 'clinton', 'rape')\n       \nThe 9th topic is about religion ('jews', 'christians', 'atheist')\n\nThe 10th and 4th seems to be about sex and gender:\n    * the 4th more about sex ('girl', 'sex', 'sexual');\n    * the 10th is less easy to classify but seems to be more about gender ('girl', 'gender', 'bible')\n\nAll those subjects has in common to be really polemics and suitable to trolls. This criteria would be easily used by a human \ncontrol but we need to find how to train your model to try to recognize it. \n\nThe insincere questions seems easier to distinct than the sincere ones, cause they are centred on a less various spectrum of words. "},{"metadata":{"_uuid":"c5827b3999ddf9ff1934de5358fc4d5e4b7b2b91"},"cell_type":"markdown","source":"---"},{"metadata":{"_uuid":"ca1170b3ffd04161bb5716c77ecdee6ea4c1745f"},"cell_type":"markdown","source":"**Now we do the same process with bigrams!**!"},{"metadata":{"_uuid":"ab3b8eea4b67733a0f540a97e1a653e05360111c"},"cell_type":"markdown","source":"## III-B. Topics modeling with bigrams (10 topics)"},{"metadata":{"_uuid":"ec22df4b463eb479fd2063f668ad9f57cb6fc327"},"cell_type":"markdown","source":"### III-B.1. Creation of the bigrams (from preprocessed data)"},{"metadata":{"_uuid":"7d4113dbc53d9cd685edb53a319c2ac80220be7e","trusted":true},"cell_type":"code","source":"from gensim.models import Phrases\nfrom gensim.models.phrases import Phraser","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18776ed18ed023493e3944f525f0742107fbc06d","trusted":true},"cell_type":"code","source":"# Create the bigram for sincere questions.\nsincere_bigram_model = Phrases(sincere_prepro_questions, min_count=1, threshold=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c79d824ab235a5ce21d261ff16a1497026cc334","trusted":true},"cell_type":"code","source":"# Same for insincere ones. \ninsincere_bigram_model = Phrases(insincere_prepro_questions, min_count=1, threshold=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbaa02d7e15e4d6f06afd6a10605203c5dc84681","trusted":true},"cell_type":"code","source":"# Creation of the phrasers, which is needed to use the bigrams. First for sincere questions...\nsincere_bigram_phraser = Phraser(sincere_bigram_model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10324baec7fedd4f956f56207d0a3fc3b157d6f2","trusted":true},"cell_type":"code","source":"# Then for the insincere. \ninsincere_bigram_phraser = Phraser(sincere_bigram_model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8bee51b6678ca254bef378d58c54e256826cef0f"},"cell_type":"markdown","source":"### III-B.2.Creation of the bigram dictionary"},{"metadata":{"_uuid":"c73ab5f0828ea80fd303c5f9a11d5a6015c8f5f0","scrolled":false,"trusted":true},"cell_type":"code","source":"sincere_bigrams = [sincere_bigram_model[question] for question in sincere_prepro_questions]\nsincere_bigrams[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3fb510db651dae94b5e3963dc4b23580d2485247","trusted":true},"cell_type":"code","source":"insincere_bigrams = [insincere_bigram_model[question] for question in insincere_prepro_questions]\ninsincere_bigrams[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b42bc729a87eb4ff37925ab24a43e276ee60743c","trusted":true},"cell_type":"code","source":"# Creation of a dictionary with the words and their id for sincere bigrams\nsincere_bigrams_dictionary = gensim.corpora.Dictionary(sincere_bigrams)\nsincere_bigrams_token = sincere_bigrams_dictionary.token2id","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d9e4a17a6302b615b0bf1874eb36fc50acd28c6","trusted":true},"cell_type":"code","source":"# Association of the dictonary words and their frequency for sincere bigrams\nsincere_bigrams_frequency = {sincere_bigrams_dictionary[k]: \n                             v for k,v in sincere_bigrams_dictionary.dfs.items()}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72787ff0db0649d335fd749066c05a044149d892","trusted":true},"cell_type":"code","source":"# Creation of a dictionary with the words and their id for insincere bigrams\ninsincere_bigrams_dictionary = gensim.corpora.Dictionary(insincere_bigrams)\ninsincere_bigrams_token = insincere_bigrams_dictionary.token2id","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68825eeda0e1e6270bc8ba55f480c227a9110116","trusted":true},"cell_type":"code","source":"# Association of the dictonary words and their frequency for insincere bigrams\ninsincere_bigrams_frequency = {insincere_bigrams_dictionary[k]: \n                               v for k,v in insincere_bigrams_dictionary.dfs.items()}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"545652d2cf6e6333f368fc3dce9271495d7b7610"},"cell_type":"markdown","source":"### III-B.3. Vectorization"},{"metadata":{"_uuid":"fb890995d2d48ac0f9e66a49f57cdc477d5fd354","trusted":true},"cell_type":"code","source":"sincere_bigrams_corpus = [sincere_bigrams_dictionary.doc2bow(question) \n                  for question in clear_sincere_questions]\nsincere_bigrams_corpus[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92ec5a1b67271360f884ee282a3d9bf6ed61db42","trusted":true},"cell_type":"code","source":"insincere_bigrams_corpus = [insincere_bigrams_dictionary.doc2bow(question) \n                  for question in clear_insincere_questions]\ninsincere_bigrams_corpus[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5cdb535fdc30d17b73bc13f79659186fd86f2fc7"},"cell_type":"markdown","source":" ### III-B.4. Lda Modeling"},{"metadata":{"_uuid":"34e995fa804764ce7afd43d78f16bd8a12a13de6","trusted":true},"cell_type":"code","source":"# Creating a lda model, which create 10 topics from the sincere bigrams\nlda_model_sincere_bigrams = gensim.models.ldamodel.LdaModel(\n    corpus=sincere_bigrams_corpus, num_topics=10, id2word=sincere_bigrams_dictionary,\n    random_state=25, passes=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cafcaf4fc454c958531d22f3a51ff3613c9c05f","trusted":true},"cell_type":"code","source":"# Creating a lda model, which create 10 topics from the insincere bigrams\nimport warnings\nwarnings.filterwarnings('ignore')\n\nlda_model_insincere_bigrams = gensim.models.ldamodel.LdaModel(\n    corpus=insincere_bigrams_corpus, num_topics=10, id2word=insincere_bigrams_dictionary,\n    random_state=25, passes=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e58bbc1a3b17089860a2b74db6124c2160b903fd","trusted":true},"cell_type":"code","source":"pprint(lda_model_sincere_bigrams.print_topics()[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4a902fc33b9a25a17f894868c4f536264e3f8d52","trusted":true},"cell_type":"code","source":"pprint(lda_model_insincere_bigrams.print_topics()[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2894db30bd4ec4a4f0cb111cb903f2105b7802b2"},"cell_type":"markdown","source":"### III-B.5. Visualization"},{"metadata":{"_uuid":"d7921a83394527b1b2448b3edcc885fa463f6b41","scrolled":false,"trusted":true},"cell_type":"code","source":"pyLDAvis.gensim.prepare(lda_model_sincere_bigrams, sincere_bigrams_corpus, \n                        sincere_bigrams_dictionary)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8c402bd109dca9fbb6902c628d3d535422d58fe"},"cell_type":"markdown","source":"Even if the graph shows unique word, the circles take into account the bigrams. \n\nThe repartition of the topics is quite different and some emerge more clearly. \n\nThe first topic, which is one of the 2 biggest, seems to be about education and job. \n\nThe 2nd one, which is also quite big, gather word indicating interrogation. It seems to be less a topic and more markers of real question (opposed to affirmation disguise in question)\n\nThe others topics are more mixed and once again some of them overlap each other, indicating that 10 topics is probably too much. Some secondary subjects appears but they're not as recognizable as insincere questions. \n"},{"metadata":{"_uuid":"72fd76e2b5b16d65c04f6fd738eec8ee801356ed","trusted":true},"cell_type":"code","source":"pyLDAvis.gensim.prepare(lda_model_insincere_bigrams, insincere_bigrams_corpus, \n                        insincere_bigrams_dictionary)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88bade77990dfeaa1e70b2965962a90a0ffa7155"},"cell_type":"markdown","source":"The result are quite similar to the first one. Even if some subjects are analogous we can recognize most of the topics by reading the \nmost relevant words. \n* Topics 1, 4 and 8 are about politics.\n*  Topics 2, 3 and  6, are about race and religions.\n* Topics 5, 7 and 9 are sex questions\n*  Topic 10 is more contrasted and we couldn't recognize a clear subject, it seems to gather different hot social subjects (gender, guns, criminals)"},{"metadata":{"_uuid":"c8a0e77e646a61666d4944b9a44bad3eed3e453e"},"cell_type":"markdown","source":"_**The clear separation in 4 group of subject let think we should try a lda with 4 topics. **_"},{"metadata":{"_uuid":"f3d8a3226fe59cc60ceedef658d5e99211a916da"},"cell_type":"markdown","source":"## III-C. Recreate new Lda with 4 topics"},{"metadata":{"_uuid":"8f29efe4c51cd91e6d9b4b3d75b84c9b969bdef4"},"cell_type":"markdown","source":"We already have preprocessed data, wo we will just recreate the Lda in itself. "},{"metadata":{"_uuid":"bf00f08d575be2f922ffc8895fa579b3fe9a6c52"},"cell_type":"markdown","source":"### III-C.1. Creating lda model"},{"metadata":{"trusted":true,"_uuid":"d05c7084f54bf62087fe23c83b47ffb912c384c8"},"cell_type":"code","source":"# Creating a lda model, which create 4 topics from the sincere questions\nimport warnings\nwarnings.filterwarnings('ignore')\nlda_model_sincere_4 = gensim.models.ldamodel.LdaModel(\n    corpus=sincere_corpus, num_topics=4, id2word=sincere_questions_dictionary,\n    random_state=25, passes=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"57301e4b852c66f6c2e15d2f577cf24e01ac8903"},"cell_type":"code","source":"# Creating a lda model, which create 4 topics from the insincere questions\nlda_model_insincere_4 = gensim.models.ldamodel.LdaModel(\n    corpus=insincere_corpus, num_topics=4, id2word=insincere_questions_dictionary,\n    random_state=25, passes=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3be2c08c1531d5a4da5525c34ce774ce0688f2c0"},"cell_type":"code","source":"# Printing the result of the modeling for sincere questions.\npprint(lda_model_sincere_4.print_topics()[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d82560ba0c2287ef923ad7e0560bf54ef96e1b7"},"cell_type":"code","source":"# Printing the result of the modeling for insincere questions.\npprint(lda_model_insincere_4.print_topics()[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ebfafc544d6554764fde787195ed3876b0134b9"},"cell_type":"markdown","source":"### III-C.2. Visualization"},{"metadata":{"trusted":true,"_uuid":"32f0ac10d1d34310a91dd919cb64a1452fa16e8e"},"cell_type":"code","source":"# Display the topic modeling for sincere questions.\npyLDAvis.gensim.prepare(lda_model_sincere_4, sincere_corpus, sincere_questions_dictionary)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b01440979eda6791a9231b680d5e7e0e3794cbe4"},"cell_type":"markdown","source":"The 4 topics created by the lda model are very distinct, this is better than the test with 10 topics. \nWe still can't differenciate clearly the subjects of the topics. Some words come back a lot \"best\", \"guess\", \"world\", \"people\".\nThis tend to show that the sincere question have various subject and there is no common theme around them. "},{"metadata":{"trusted":true,"_uuid":"c16978717b1b80202d63153096fc65a3bb501747"},"cell_type":"code","source":"# # Display the topic modeling for insincere questions.\npyLDAvis.gensim.prepare(lda_model_insincere_4, insincere_corpus, insincere_questions_dictionary)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"30cb0f608e902483810c6837a2a987247d98d8e9"},"cell_type":"markdown","source":"We could see 4 distincts topics: \n- topic 1 is about politic with top-3 keyword being \"Trump\", \"liberals\", \"presidents\". It's also the bigger topic;\n- topic 2 is about race and nationality with keywords \"white\", \"black\", \"people\";\n- topic 3 is about gender and sex, \"women\", \"men\", \"gay\";\n- topic 4 is about religion it seems, with keywords \"christians\", \"muslims\".\n\nThis topics could be use cause they rejoign the subjects given by Quora as sensible. This topic could be used as a variable \nto characteristic insincere question and be used for machine learning. "},{"metadata":{"_uuid":"e0c0ba1a4fbb143f38e032196ff0147a9ac058bd"},"cell_type":"markdown","source":"---"},{"metadata":{"_uuid":"02c09fa4cd105a1263758543f2832812f8693d15"},"cell_type":"markdown","source":"# IV. Quick machine learning model"},{"metadata":{"_uuid":"78afaaade3fd165ef38ccc92d25624d46be9645b"},"cell_type":"markdown","source":"This kernel is mostly center around the topic modeling but let's try a quick and simple model just to see how it works. This model doesn't use the topic modeling, which maybe could help to improve the final score. "},{"metadata":{"_uuid":"2122e9767e084ff68d25ccaf750cad0fa897304b","trusted":true},"cell_type":"code","source":"# I import all the functions from sklearn.\nfrom sklearn.metrics import classification_report\nfrom sklearn.naive_bayes import MultinomialNB\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2460705db26e3cf4e8437f1b0acba9c40efe52ce"},"cell_type":"markdown","source":"Train_test_split is not the most precise method of validation but I just want to quickly test the model as a first try. "},{"metadata":{"_uuid":"f750ac8625dc7cc006d93b9c1227f580ac2fd0b8","trusted":true},"cell_type":"code","source":"# I split the date into train and test, variable and target. \nX_train, X_test, y_train, y_test = train_test_split(X_train, y_train)\nX_train.shape, y_train.shape, X_test.shape, y_test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32ae03a5cfcbbe2394bc4b2a830afa0ba361e28f"},"cell_type":"markdown","source":"Now i have my training and testing sample for train my model. I will do a pipeline with a transformer, to turn the text value to vectors and an estimator, to predict. "},{"metadata":{"_uuid":"6bb71a93c19a41664841da68341c92f894dbc9d4","trusted":true},"cell_type":"code","source":"# I create of the transformer and the estimator and insertion in a pipeline.\ntfidv = TfidfVectorizer(lowercase=True, stop_words='english')\nmultinomialnb = MultinomialNB()\npipe = make_pipeline(tfidv, multinomialnb)\npipe","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"610c9f39785f635a27f08300c6fc1a8758517dc1","trusted":true},"cell_type":"code","source":"# I fit of the pipeline on X_train and y_train,\n# then predict on the y_test and estimate of the predictions. \npipe.fit(X_train, y_train)\ny_pred = pipe.predict(X_test)\nprint(classification_report(y_test, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ca85c39d9f5b42ee1d1008e1b713e3811a0d0f5"},"cell_type":"markdown","source":"**As expected for a naive model and without any ponderation on a disbalance datasets, the F1 score is quite good for 0 and really bad for 1.**\n\n**Enginere featuring and ponderation could let us have a better score. **"},{"metadata":{"_uuid":"58c77e7f809415752739c93a1a16587e493ba80f","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}