{"cells":[{"metadata":{"_uuid":"4031bb53fd2934badd10fa5b88aa35ea9bd980ed"},"cell_type":"markdown","source":"# Quora Insincere Questions Classification\n\n## By Koby Nimni, Noam Frank, and Ziv Bar Nahum"},{"metadata":{"_uuid":"5178da3568d0374f223db60ac58b01e3d9470c17"},"cell_type":"markdown","source":"## Project Outline:\n#### 1. Import Libraries and Data\n#### 2. Raw Data “Descriptive Statistics” \n#### 3. Cleaning the Data and Pre-Processing\n#### 4.\tVisualization of the Clean Data\n#### 5.\tVectorising the Data\n#### 6.\tUnder Sampling and clustering\n#### 7.\tModelling and Results\n#### 8.\tConcluding Remarks and Work in Progress "},{"metadata":{"_uuid":"f240579692270e31505ffcfa426f237f43c0d828"},"cell_type":"markdown","source":"## 1. Import Libraries and Data\n\n### 1.1 Import DS Basics and NLP Specifics:"},{"metadata":{"trusted":true,"_uuid":"ca3fed43891a9d5ceec3e696705ce0557e083516"},"cell_type":"code","source":"################# GENERAL IMPORTS\nimport os\nimport string\nfrom pprint import pprint\nfrom operator import itemgetter\nimport matplotlib.pyplot as plt\nfrom sklearn.svm import LinearSVC\nfrom sklearn.preprocessing import MultiLabelBinarizer\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.metrics import f1_score, precision_score, recall_score,accuracy_score\n##################################### NLP SPECIFIC IMPORTS\nfrom nltk.tokenize import word_tokenize\nfrom nltk.corpus import stopwords\nfrom nltk import ngrams\nfrom nltk.stem import PorterStemmer\nfrom nltk.corpus import reuters\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom wordcloud import WordCloud\nfrom collections import Counter\nreuters.fileids()\nstopwords.words('english')\n\nimport numpy as np\nimport pandas as pd\nimport warnings\nfrom sys import modules\n\nwarnings.filterwarnings('ignore')\n%matplotlib inline\n\nfrom gensim.models import word2vec\nimport logging\n\nfrom keras.preprocessing import sequence\nfrom keras.models import Sequential\nfrom keras.layers.core import Dense, Activation, Flatten\nfrom keras.layers.wrappers import TimeDistributed\nfrom keras.layers.embeddings import Embedding\nfrom keras.layers.recurrent import LSTM\nfrom keras.layers import Dropout\n\nimport seaborn as sns\nsns.set(style = 'darkgrid')\nprint(os.listdir(\"../input\"))\nimport re\npd.set_option('max_colwidth', 800)\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\n\nnp.random.seed(1234)\n%matplotlib inline\n\nprint('all set')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8879c22208a49de84c33bb9b35d7d053cce11d61"},"cell_type":"markdown","source":"### 1.2 Load the Text Data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"quora_train=pd.read_csv(\"../input/train.csv\")\nquora_test=pd.read_csv(\"../input/test.csv\")\nprint(\"Train size =\" ,quora_train.shape)\nprint(\"Test size =\" ,quora_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3276cb401e5354d36ea07ff390786c5f50b2b59c"},"cell_type":"code","source":"# quora_train=quora_train[0:5000]\n# quora_test=quora_test[0:1000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f2b9b56842ee58ad7a4533924a2f56dc5327c12"},"cell_type":"code","source":"quora_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"356fe003419bc14127b221ebea93bb7e02990f14"},"cell_type":"markdown","source":"## 2.\tRaw Data “Descriptive Statistics”"},{"metadata":{"trusted":true,"_uuid":"b87c788afc7b53632101c1a779a7b0491d1efaa6"},"cell_type":"code","source":"quora_train.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b3471dd07a372b9a4cc1f5b3ddb3cd10895666f"},"cell_type":"markdown","source":"* #### Dataset has 3 columns and 1,306,122 questions.\n* #### No missing data."},{"metadata":{"_uuid":"cd87a0976203e2944e9af97bc77b8d1c9213564f"},"cell_type":"markdown","source":"> ### Sincere question examples: "},{"metadata":{"trusted":true,"_uuid":"dfbbb0105561159705958c557ed8d97474b3fc47"},"cell_type":"code","source":"sincere = quora_train[quora_train.target==0]\ninsincere = quora_train[quora_train.target==1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"970b4ca9c84b31330dcabae62a34504297a695fb"},"cell_type":"code","source":"[print(q,'\\n') for q in sincere['question_text'][[1,6,40,300,120]]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5c9d1bd2fe3d3933f1d66fd8358464c911ce5d6c"},"cell_type":"markdown","source":"### 2.1 Target Distribution"},{"metadata":{"trusted":true,"_uuid":"d7cd5db8a60de221b7ffdf04bb7c048536abf246"},"cell_type":"code","source":"quora_train['target'].value_counts().plot(kind='bar', title='Target distribution')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"467ae861251ad37dd66c0cb816963a8ff3c7b5d4"},"cell_type":"code","source":"round(quora_train['target'].value_counts(normalize =True),3)*100","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6e0c14457acf999330e37064b0a5cad9450e744"},"cell_type":"markdown","source":"### => Data is Imbalanced ! \n### Target is only 6.2% of the data"},{"metadata":{"_uuid":"4e1aa987a5bd931d56004226b725b874c06756dd"},"cell_type":"markdown","source":"### 2.2 Word and Character Count "},{"metadata":{"trusted":true,"_uuid":"1df11d878c54394a7089181ea266b4b28283b1b4"},"cell_type":"code","source":"quora_train['words'] = quora_train.question_text.apply(lambda x: len(x.split()))\nquora_train['characters'] = quora_train.question_text.apply(lambda x: len(x))\nquora_test['words'] = quora_test.question_text.apply(lambda x: len(x.split()))\nquora_test['characters'] = quora_test.question_text.apply(lambda x: len(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2e7cf9271f8fa2818ffcf273f21e9dcbdea6ff53","_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"quora_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"908f89a1864588a040dc179561d605fe0badf254"},"cell_type":"markdown","source":"### 2.2.1 Average Word and Character Count by Target"},{"metadata":{"trusted":true,"_uuid":"50e8d591a73bfbadc84cbea78835c2c7917fa2c9"},"cell_type":"code","source":"fig = plt.figure(figsize=(18, 7))\n\nplt.subplot(1, 2, 1)\nquora_train.groupby('target')['words'].mean().plot(kind='bar', ylim=(0,20), title= 'Average word count by target')\n\nplt.subplot(1, 2, 2)\nquora_train.groupby('target')['characters'].mean().plot(kind='bar', ylim=(0,105), title= 'Average character count by target')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92feee61a9911995f131fe4893659646a464532d"},"cell_type":"markdown","source":"* ### Insincere questions have on average longer word counts and character counts.\n* ### We will use these as features in our model."},{"metadata":{"trusted":true,"_uuid":"ccd90db80045ced86c5edca1b07ee9d1cccad662"},"cell_type":"code","source":"fig = plt.figure(figsize=(18, 8))\nfont = {'size': 16, 'weight': 'bold'}\n\nplt.subplot(2, 1, 1)\nplt.title('Word count - by outcome',fontdict=font)\nplt.xlim(0,70)\nax = sns.boxplot(x=\"words\", y=\"target\", data=quora_train, orient=\"h\")\n\nplt.subplot(2, 1, 2)\nplt.title('character count - by outcome',fontdict=font)\nplt.xlim(0,350)\nax = sns.boxplot(x=\"characters\", y=\"target\", data=quora_train, orient=\"h\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e5ac25c4266c829e92e36a6e355efb888ee8198"},"cell_type":"markdown","source":"* ### Insincere questions have larger variance and median for both word and character counts."},{"metadata":{"_uuid":"be7eee1dbe3ff1c21f3782242bfc48a8165ca4d7"},"cell_type":"markdown","source":"### 2.3 Additional Features - Verb and Noun Count "},{"metadata":{"trusted":true,"_uuid":"8b629f93ecdaf7a4246a04f8d260e9437ba37f82"},"cell_type":"code","source":"from nltk import pos_tag\n\ndef verb_count(text):\n token_text= word_tokenize(text)\n tagged_text = pos_tag(token_text)\n counter=0\n for w,t in tagged_text:\n t = t[:2]\n if t in ['VB']:\n counter+=1\n return counter\n\ndef noun_count(text):\n token_text= word_tokenize(text)\n tagged_text = pos_tag(token_text)\n counter=0\n for w,t in tagged_text:\n t = t[:2]\n if t in ['NN']:\n counter+=1\n return counter\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21ab9a360b0f353db5d933959209f22ac28bde10"},"cell_type":"code","source":"quora_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5cc0d8571a01e4ef217eefc1e04b9741d7671865"},"cell_type":"markdown","source":" # 3. Cleaning the Data and Pre-Processing"},{"metadata":{"_uuid":"f991cd8bb0a7473d0aefa371033f3078d050ac5c"},"cell_type":"markdown","source":" ### 3.1 Lower-casing"},{"metadata":{"trusted":true,"_uuid":"a3478298c71b9db06f9ffb1b9570b4ba24cd0fb3"},"cell_type":"code","source":"quora_train['question_text_prep'] = quora_train['question_text'].apply(lambda x: x.lower())\nquora_test['question_text_prep'] = quora_test['question_text'].apply(lambda x: x.lower())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c16c6093ec0bc95e4306bc25dba05d3c2af70b37"},"cell_type":"markdown","source":"### 3.2 Punctuations and Stop-words"},{"metadata":{"trusted":true,"_uuid":"45032606b07505f92d3a0232a498ce9adca7e5cf"},"cell_type":"code","source":"def pad_punctuation_w_space(string):\n s = re.sub('([:;\"*.,!?()/\\=-])', r' \\1 ', string)\n s=re.sub('[^a-zA-Z]',' ',s)\n s = re.sub('\\s{2,}', ' ', s)\n s = re.sub(r\"\\b[a-zA-Z]\\b\", \"\", s) #code for removing single characters\n return s\nquora_train['question_text_prep'] = quora_train['question_text_prep'].apply(lambda x: pad_punctuation_w_space(x))\nquora_test['question_text_prep'] = quora_test['question_text_prep'].apply(lambda x: pad_punctuation_w_space(x))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12df12de7c47eddf8a69bc58db6a2b9de0c7fc72"},"cell_type":"markdown","source":"### 3.3\tQuestions as List of Strings"},{"metadata":{"trusted":true,"_uuid":"19eaa92755daa9c9866c8dfd0aaa54bd05649d04"},"cell_type":"code","source":"quora_train['question_text_prep'] = quora_train['question_text_prep'].apply(lambda x: x.split())\nquora_test['question_text_prep'] = quora_test['question_text_prep'].apply(lambda x: x.split())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7a508aa7f1640d253fe74697e827d929d8fcc980"},"cell_type":"code","source":"stop_list = stopwords.words('english') + list(string.punctuation)\nquora_train['question_text_prep'] = quora_train['question_text_prep'].apply(lambda x: [i for i in x if i not in stop_list])\nquora_test['question_text_prep'] = quora_test['question_text_prep'].apply(lambda x: [i for i in x if i not in stop_list]) \n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87cdb7a314c6b622ff5233ba3906d7d5f034421b"},"cell_type":"markdown","source":"#### ** this was done because word embedding needs the text to be represented as list of strings."},{"metadata":{"_uuid":"73f01241dbc977d31a68e6eefb762f209ad86707"},"cell_type":"markdown","source":"## 4. Visualization of the Clean Data"},{"metadata":{"_uuid":"c3f58940f4c4753031c7913be23f816d4e5ede04"},"cell_type":"markdown","source":"### 4.1\tUnigram Distribution – Sincere vs. Insincere Questions"},{"metadata":{"trusted":true,"_uuid":"eeaa1401a35014f3c2a8d894689934ba0b97818b"},"cell_type":"code","source":"quora_train['question_text_prep_string'] = quora_train['question_text_prep'].str.join(\" \")\nquora_test['question_text_prep_string'] = quora_test['question_text_prep'].str.join(\" \")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be7c38d56f97f3d1fcb510d5bdf5724e6a770e16"},"cell_type":"code","source":"# quora_insincere = quora_train[quora_train.target==1]\n# quora_sincere = quora_train[quora_train.target==0]\n\n# # make \"all in one\" corpuses for the 2 classes in the target\n# insincere_all_in_one = ' '.join([q for q in quora_insincere.question_text_prep_string])\n# sincere_all_in_one = ' '.join([q for q in quora_sincere.question_text_prep_string]) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43995eb9979cc814a631a08b9b9277c9c3c68eaa"},"cell_type":"markdown","source":"### 4.1.1 Word Cloud for Sincere and Insincere Questions"},{"metadata":{"trusted":true,"_uuid":"89c66f2cc2ff0cb0137175e974d25c7369b53909"},"cell_type":"code","source":"# fig = plt.figure(figsize=(30, 12))\n# font = {'size': 20, 'weight': 'bold'}\n\n# plt.subplot(1, 2, 1)\n# plt.title('Insincere questions',fontdict=font)\n# cloud1 = WordCloud(max_words=100,width=480, height=480, background_color='grey')\n# cloud1.generate_from_text(insincere_all_in_one)\n# plt.imshow(cloud1)\n# plt.axis('off')\n\n# plt.subplot(1, 2, 2)\n# plt.title('Sincere questions',fontdict=font)\n# cloud = WordCloud(max_words=100,width=480, height=480, background_color='skyblue')\n# cloud.generate_from_text(sincere_all_in_one)\n# plt.imshow(cloud)\n# plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6b7ed4f3cb5a38b3c231caca36b988ba672d943"},"cell_type":"markdown","source":"### 4.2\tBigram and Trigram Distribution - Sincere vs. Insincere Questions"},{"metadata":{"trusted":true,"_uuid":"6a2c0f11c8c97478023096fd1759ac9049d64684"},"cell_type":"code","source":"# insincere_tokens = [t for t in word_tokenize(insincere_all_in_one)]\n# sincere_tokens = [t for t in word_tokenize(sincere_all_in_one)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dcbf49662e7c9fd10db1141c54452723a8ec41cb"},"cell_type":"code","source":"# from collections import Counter\n# from nltk import ngrams\n# bi_gram_insincere = Counter(ngrams(insincere_tokens, 2))\n# tri_gram_insincere = Counter(ngrams(insincere_tokens, 3))\n# bi_gram_sincere = Counter(ngrams(sincere_tokens, 2))\n# tri_gram_sincere = Counter(ngrams(sincere_tokens, 3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"51972d195212c75ab7d9d39ca735e11ae93f7693"},"cell_type":"code","source":"# bi_gram_ins = pd.DataFrame(bi_gram_insincere.most_common(20), columns=['bi_gram_ins','frequency'])\n# bi_gram_sin = pd.DataFrame(bi_gram_sincere.most_common(20), columns=['bi_gram_sin','frequency'])\n# tri_gram_ins = pd.DataFrame(tri_gram_insincere.most_common(20), columns=['tri_gram_ins','frequency'])\n# tri_gram_sin = pd.DataFrame(tri_gram_sincere.most_common(20), columns=['tri_gram_sin','frequency'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9846c107c72c6fddc40b788335b74789313d2ff8"},"cell_type":"markdown","source":"### 4.2.1 Bigram Distribution - Sincere vs. Insincere Questions"},{"metadata":{"trusted":true,"_uuid":"3d76fcabf229acf0332d1cc677d25454f015d98d"},"cell_type":"code","source":"# import matplotlib.pyplot as plt\n# fig, (ax, ax2) = plt.subplots(ncols=2, sharex=False)\n# fig.subplots_adjust(wspace =0.6)\n# ax.invert_xaxis()\n\n# bi_gram_ins.sort_values(by='frequency').plot(kind='barh', x='bi_gram_ins', legend=True, ax=ax, figsize=(18,9), fontsize =16)\n# bi_gram_sin.sort_values(by='frequency').plot(kind='barh', x='bi_gram_sin',ax=ax2, figsize=(18,9),fontsize =16)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"163ce831f57cb0c57bf02df7e925572d9324788d"},"cell_type":"markdown","source":"* ### The most common bigram among sincere questions is \"best way\" (e.g., what is the best way...?).\n* ### Not surprisingly, \"Donald Trump\" is the most common bigram among insincere questions (obviously all of these instances are statements/opinions on \"Donald Trump\")."},{"metadata":{"_uuid":"d4c591be5d6a725f1141e36fb806ab47e0d96ad7"},"cell_type":"markdown","source":"### 4.2.2 Trigram Distribution - Sincere vs. Insincere Questions"},{"metadata":{"trusted":true,"_uuid":"bdd0c34420474145475894f622e166276c00d2cd"},"cell_type":"code","source":"# fig, (ax, ax2) = plt.subplots(ncols=2, sharex=False)\n# fig.subplots_adjust(wspace =0.8)\n# ax.invert_xaxis()\n\n# tri_gram_ins.sort_values(by='frequency').plot(kind='barh', x='tri_gram_ins', legend=True, ax=ax, figsize=(18,9), fontsize =16)\n# tri_gram_sin.sort_values(by='frequency').plot(kind='barh', x='tri_gram_sin',ax=ax2, figsize=(18,9), fontsize =16)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9417abfd00aa1c7f92f2ea3451038bf4e346223a"},"cell_type":"markdown","source":"* ### The most common trigram among sincere questions is \"advice would give\" (e.g., what advise would you give for ..?).\n* ### \"Kim Jong Un\" is the most common trigram among insincere questions (obviously all of these instances are statements/opinions on \"Kim Jong Un\").\n* ### It seems that the most common bigrams and trigrams are revolved around different topics and therefore cluster analysis might be beneficial. "},{"metadata":{"_uuid":"62b05dfda6743ae291a7714c2dd88ebfc3d2fc86"},"cell_type":"markdown","source":"## 5. Vectorising the Data"},{"metadata":{"trusted":true,"_uuid":"5a4983b3c730efe239517ad46d9eca34cc3cef5e"},"cell_type":"code","source":"sents = list(quora_train.question_text_prep.values) \nsents[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5c78e3c89c4c9e6348a9eaa2c018ab9f522611ce"},"cell_type":"markdown","source":"### 5.1\tUsing the \"Glove\" Word Embedding"},{"metadata":{"trusted":true,"_uuid":"1194f188689998096e7b89d248139fdfa1251699"},"cell_type":"code","source":"min_num = 3 # minimum number of occurrences in text\nEMBEDDING_FILE= \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78286b07a6aeed2ac369059c4786434508427dfa"},"cell_type":"code","source":"import numpy as np\ndef loadGloveModel(gloveFile):\n print (\"Loading Glove Model\")\n f = open(gloveFile,'r', encoding='utf8')\n model = {}\n for line in f:\n splitLine = line.split(' ')\n word = splitLine[0]\n embedding = np.asarray(splitLine[1:], dtype='float32')\n model[word] = embedding\n print (\"Done.\",len(model),\" words loaded!\")\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b26b84b85f4ce2321027a12b512827c9941eb3c2"},"cell_type":"code","source":"word_model= loadGloveModel(EMBEDDING_FILE) \n# print (word_model['hello']) # if we want to see an example for a vector","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7138f9c77f3311f98bb68ebdedf56a48832a7e05"},"cell_type":"code","source":"print('Loaded %s word vectors.' % len(word_model))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dabcec6c21fc253828355a87f3c343142d59d6ef"},"cell_type":"markdown","source":"### 5.2\tHandling Unknown Words"},{"metadata":{"_uuid":"09d96f8052fcb8ac6d1e6b96c47f34217ea24146"},"cell_type":"markdown","source":"### 5.2.1 Identify and Count Unknown Words"},{"metadata":{"trusted":true,"_uuid":"32dd4c7322327c5c0064df0f14dbef3258a9f9d4"},"cell_type":"code","source":"unknown_words = []\nfor question in quora_train.question_text_prep:\n for word in question:\n if word not in word_model:\n unknown_words.append(word)\n else: pass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4173997219a40dd52f011ce29d18758f56903576"},"cell_type":"code","source":"len(unknown_words)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c56d071580de9c2aece7e6602f710d32e18dbf39"},"cell_type":"code","source":"unknown_words[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5813efe81e3ada462e977f6b0b4d451f2134b6c"},"cell_type":"markdown","source":"### 5.2.2 The Frequency of the Unknown Words"},{"metadata":{"trusted":true,"_uuid":"ca569cccfaa0e884b4abdcab71f979d5e5fe3746"},"cell_type":"code","source":"total_term_frequency = Counter(unknown_words)\n\nfor word, freq in total_term_frequency.most_common(20):\n print(\"{}\\t{}\".format(word, freq))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84c50e5525c46001c60adc9892e58c5f95f2bd71"},"cell_type":"markdown","source":"### 5.3\tVectorising the Data Using the “word model”"},{"metadata":{"trusted":true,"_uuid":"f9d9a93ca48e3ac0e54c5c62c0622cd8f2c4588d"},"cell_type":"code","source":"def get_vector(DataFrame):\n vec_X = []\n i = 0\n for item in DataFrame.question_text_prep_string: \n \n sentence = pad_punctuation_w_space(item)\n s = np.array([])\n s = []\n if len(sentence)==0:\n s = np.array(word_model['UNK'])\n vec_X.append(s) \n i += 1\n else:\n for word in sentence.split():\n if len(s) == 0:\n try:\n s = np.array(word_model[word])\n except: \n s = np.array(word_model['UNK'])\n else:\n try:\n s += np.array(word_model[word])\n except: \n s += np.array(word_model['UNK']) \n vec_X.append(s) \n i += 1\n\n return vec_X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"824681a6943ce5197d10c71f81329efec90a8497"},"cell_type":"code","source":"vec_X_train=get_vector(quora_train)\nvec_X_test=get_vector(quora_test)\nquora_train[\"vector\"]=vec_X_train\nquora_test[\"vector\"]=vec_X_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5986e9e541a5882ed40f4769aaf127cf36a1e430"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d040ab60cb5907ad749aaa3dfb3fc1936df3896b"},"cell_type":"markdown","source":"## 6. Under Sampling\nData is very imballanced. Without Under sampling the classifier will see the target as **noise** "},{"metadata":{"_uuid":"dec625c8af2200ed20e84368258e71bbbb51652f"},"cell_type":"markdown","source":"### 6.1\tProceed with the Under Sample Data "},{"metadata":{"trusted":true,"_uuid":"69c79f7fde28b98fbcf7508db1461ae220a80e10"},"cell_type":"code","source":"from imblearn.under_sampling import RandomUnderSampler","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"537802a84e6b63549c56a515e982e145610af02e"},"cell_type":"markdown","source":"### Two general techniques for working with imballanced data:\n
\n\n\n### We chose random undersampling"},{"metadata":{"trusted":true,"_uuid":"eab8ef7bcd66881eb5341afe16cd457cb6103a63"},"cell_type":"code","source":"X = quora_train[['words','characters','vector']] #,'noun_count'\ny = quora_train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b9b04990d11ff438d31fc528e757d31dbc0c5167"},"cell_type":"code","source":"rus = RandomUnderSampler(return_indices=True, ratio = 0.42)\nX_rus, y_rus, id_rus = rus.fit_sample(X, y)\n\nprint('indexes:', id_rus)\nprint(len(id_rus))\nprint(quora_train.target.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"136b85f5e9c138fecea2acd0c1c4fb691dfc0c4b"},"cell_type":"code","source":"quora_undr=quora_train.loc[id_rus]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40190554af13592b5d2d943c867d6715a245351d"},"cell_type":"code","source":"quora_undr['target'].value_counts(ascending=True).plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0120ecacfdbae76c9438b86dd692e5803f5234a2"},"cell_type":"code","source":"quora_undr['target'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65d58bf13a2c185e6da8029fbf96b3e164d88740"},"cell_type":"markdown","source":"### Important note: Undersampling was carried out while preserving a bias in the data."},{"metadata":{"trusted":true,"_uuid":"6f2fbe36de3e6e73357000053e0ca0db3fe7a296"},"cell_type":"code","source":"quora_under_prep = quora_undr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3974db7e77b9879a41f6422ddd854a9c7210727"},"cell_type":"code","source":"quora_under_prep[\"characters\"].head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"512eff4f5725589182183495a9be2e30921c77bd"},"cell_type":"markdown","source":"**### 6.3\tCheck Zero-Size Vectors"},{"metadata":{"_uuid":"86461aca6fa694d6346a993c1cbad8b8493b5da9"},"cell_type":"markdown","source":"### important notice- cleaning data using stopwords, removing numbers etc. may leave the line\\vector empty and cause later errors. that is why we check if there are lines with zero length vector"},{"metadata":{"trusted":true,"_uuid":"52ed2df34d9e7cb35c01c1ab3eff4140a10d9290"},"cell_type":"code","source":"quora_under_prep['noun_count'] = quora_under_prep.question_text.apply(lambda x: noun_count(x))\nquora_test['noun_count'] = quora_test.question_text.apply(lambda x: noun_count(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e838ea4b520428b0fdfbb2442389f9ac68ea243f"},"cell_type":"code","source":"quora_under_prep['vector_length']= quora_under_prep['vector'].apply(lambda x: len(x))\nquora_test['vector_length']= quora_test['vector'].apply(lambda x: len(x))\nquora_test['vector_length'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c6fb8b926bb40b050cc63a0def52387d7569f5b"},"cell_type":"code","source":"quora_best=quora_under_prep","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3befb1601eec1d731b1098706ebd0c8e9fe5dc19"},"cell_type":"markdown","source":"### We believe that the direction of the vector is affected from the variables we have found earlier. For this reason we decided to join all those variables with the word embedding vector and create a new, more reliable vector. from now on this vector will be our point of reference"},{"metadata":{"trusted":true,"_uuid":"e74174528ff9892a24d80b7e2aa3893ec942735b"},"cell_type":"code","source":"import numpy as np\nquora_best[\"joinvector\"]=[np.concatenate((np.array([quora_best[\"characters\"].iloc[i]]),quora_best[\"vector\"].iloc[i]), axis=None) for i in range(len(quora_best))]\nquora_best[\"joinvector_2\"]=[np.concatenate((np.array([quora_best[\"words\"].iloc[i]]),quora_best[\"joinvector\"].iloc[i]), axis=None) for i in range(len(quora_best))]\nquora_best[\"joinvector_all\"]=[np.concatenate((np.array([quora_best[\"noun_count\"].iloc[i]]),quora_best[\"joinvector_2\"].iloc[i]), axis=None) for i in range(len(quora_best))]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b98dd776d04a88fa93b0bbdcb83ffaf327d8144f"},"cell_type":"markdown","source":"## Doing the same for the test data"},{"metadata":{"trusted":true,"_uuid":"5e664927ecd446014188224b33a33059eb221a42"},"cell_type":"code","source":"quora_test[\"joinvector\"]=[np.concatenate((np.array([quora_test[\"characters\"].iloc[i]]),quora_test[\"vector\"].iloc[i]), axis=None) for i in range(len(quora_test))]\nquora_test[\"joinvector_2\"]=[np.concatenate((np.array([quora_test[\"words\"].iloc[i]]),quora_test[\"joinvector\"].iloc[i]), axis=None) for i in range(len(quora_test))]\nquora_test[\"joinvector_all\"]=[np.concatenate((np.array([quora_test[\"noun_count\"].iloc[i]]),quora_test[\"joinvector_2\"].iloc[i]), axis=None) for i in range(len(quora_test))]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8988cfe806480231439c495a77deedca9ecd0a4a"},"cell_type":"markdown","source":"### 6.5\tClustering the data"},{"metadata":{"_uuid":"e612bdb4385fb0b52e45cadd8f5d35f2e315511d"},"cell_type":"markdown","source":"#### We believe that certain categories affect the feelings of the users more than others, that is why we want to understand the differenct clusters within the questions"},{"metadata":{"_uuid":"4af5e04e635dd39f8aa1dfd0b7b2bc39af270e2c"},"cell_type":"markdown","source":"#### we will use our predefined joined vector in order to cluster our data"},{"metadata":{"trusted":true,"_uuid":"93e0f81317adf4325a7f574469de48cf9a96cdc3"},"cell_type":"code","source":"# quora_test.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d0a6846d76e9c3c244c2a968f0e8c25a2c9caba1"},"cell_type":"code","source":"X_joinvec=quora_best[\"joinvector_all\"].tolist()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fefc877239d696b2b2b3198ff91929e865fb330c"},"cell_type":"markdown","source":"### performing DBSCAN"},{"metadata":{"trusted":true,"_uuid":"09858548bd26392d431b11b42b6e56b9ac0d663e"},"cell_type":"code","source":"# from sklearn.cluster import DBSCAN\n# fit_model_joinvec=DBSCAN(eps=0.25,min_samples=10).fit(X_joinvec)\n# fitted_model_joinvec=fit_model_joinvec.labels_\n# quora_best['DBSCAN_Cluster_joinvec']=fitted_model_joinvec\n\n# quora_best.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4950aa46bdac3b27d82ee4f2d6832a5c1a6d882"},"cell_type":"code","source":"#quora_best['DBSCAN_Cluster_joinvec'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"718e23be7a61059125acee9f4de56c564c243130"},"cell_type":"markdown","source":"* #### running the code above we saw that DBSCAN cluster did not cluster efficiently"},{"metadata":{"_uuid":"6595c67bafdccac1960d6fcb0ec7932d08552b29"},"cell_type":"markdown","source":"#### let's try KMEANS"},{"metadata":{"trusted":true,"_uuid":"188ae627bdab9420b580073c35a824cc3d3abbcd"},"cell_type":"code","source":"# from sklearn.cluster import KMeans, MiniBatchKMeans\n# def calc_inertia(k):\n# model_kmeans= KMeans(n_clusters=k, init='k-means++',verbose=1,random_state=42).fit(X_joinvec)\n# return model_kmeans.inertia_,model_kmeans.labels_\n\n# inertias,labels_kmeans = [(k, calc_inertia(k)) for k in range(1, 21)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6395c7eafa4af3218e710c56ed93b3315cebde67"},"cell_type":"markdown","source":"#### Trying KMEAN both with gererating dynamic n_clusters and using a fixed number returned a memory issue that is why we decided to work with a ligther version of KMEANS- called MiniBatchKMeans"},{"metadata":{"trusted":true,"_uuid":"00aec201f002b0c2ebeaade240689b7204121cb3"},"cell_type":"code","source":"# from sklearn.cluster import KMeans, MiniBatchKMeans\n# n_clusters=50\n# model_MiniBatch = MiniBatchKMeans(n_clusters=n_clusters, init='k-means++', n_init=1,\n# init_size=500,\n# batch_size=500, verbose=1)\n\n# print (\"Clustering sparse data with %s\" % model_MiniBatch)\n# model_MiniBatch.fit(X_joinvec)\n# labels_MiniBatch = model_MiniBatch.labels_\n# print(\"done\")\n# # print(\"labels\", labels_MiniBatch)\n# # print(\"intertia:\", model_MiniBatch.inertia_)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22b9f0a6ef1e0733afd71b3e995e90de3bc24bad"},"cell_type":"code","source":"# quora_best['MiniBatch_Cluster_joinvec']=labels_MiniBatch\n# # quora_best.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"36a9475c0338f8651a9370ef0b3ede7f879ac02b"},"cell_type":"code","source":"# quora_best['MiniBatch_Cluster_joinvec'].value_counts().head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75426c23c8d7f09e3488a242fa057517076508c9"},"cell_type":"code","source":"# import matplotlib.pylab as plt\n# prob_groups = quora_best.groupby(\"MiniBatch_Cluster_joinvec\").MiniBatch_Cluster_joinvec.count()\n# prob_groups.plot(kind='hist')\n# plt.figure(figsize=(30,100))\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5316832d969dc2118efccb6f5ebefcd1933538eb"},"cell_type":"markdown","source":"### let's inspect some clusters that have both 1 and 0 values"},{"metadata":{"trusted":true,"_uuid":"1f7bc16c87061c24ce502bf0e127bbf32732eafa"},"cell_type":"code","source":"# pd.set_option('max_colwidth', 800)\n# cluster_columns=quora_best[[\"question_text\",\"target\",\"MiniBatch_Cluster_joinvec\"]]\n# random_cluster=cluster_columns[cluster_columns[\"MiniBatch_Cluster_joinvec\"]==38].sort_values(by='target',ascending=False)\n# random_cluster.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d84d7c506e2d0f7cf57cec896745015d41fb1f98"},"cell_type":"code","source":"# random_cluster.target.value_counts(normalize=True).plot(kind='bar', title='target', figsize=(10,5))\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72aeb4327dfc9cdec0139673e775feb0ed2b90c0"},"cell_type":"markdown","source":"#### We also tried clustering based on tf-idf which did not perform that well"},{"metadata":{"_uuid":"5dd6a65d09a8d7bb6f4dca5767b5858b7676380e"},"cell_type":"markdown","source":"### One option for future processing would be generating a combined vector which includes the cluster dummies"},{"metadata":{"_uuid":"2d505dc04295db354f9b77a30732cfb3dca45713"},"cell_type":"markdown","source":"## 7. Modelling and Results "},{"metadata":{"_uuid":"a1f5d7e15424df013910751a5beeb76afc919be6"},"cell_type":"markdown","source":"## 7.1 Train test split"},{"metadata":{"trusted":true,"_uuid":"d62c144a3e1bdfa0304cdfe91c0b4bf9f848babd"},"cell_type":"code","source":"Features = quora_best['joinvector_all']\n# Features2=quora_best['vector']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ebc103baa029d54ba7408726b33b173fb60dbf7"},"cell_type":"code","source":"# FF=Features.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b54f484afdad7b5b76dd9c3bc35bfbd117815d4e"},"cell_type":"code","source":"# Features.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"173542b8bccb33000049265462763fd07af6352e"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n\n\nX_train, X_val, y_train, y_val = train_test_split(Features,quora_best['target'],\n train_size=0.7, random_state = 143, stratify=quora_best['target'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c2001d51a419fbfeab380442db3babac9e45e76"},"cell_type":"markdown","source":"## 7.2 Base line classifier"},{"metadata":{"trusted":true,"_uuid":"ec607d792ff20c847c30adf2863e464e7989ddbb"},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import GradientBoostingClassifier\n\n#evaluators:\nfrom sklearn.metrics import confusion_matrix, classification_report, accuracy_score, f1_score\nfrom sklearn.model_selection import cross_val_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"303e1c6f473a456e76cd177bd71e4588d909f33e"},"cell_type":"code","source":"# # Lr_clf = LogisticRegression()\n# # X = X_train.tolist()\n# # y = y_train\n\n# # Lr_clf.fit(X,y)\n# GB_clf=GradientBoostingClassifier()\n# X = X_train.tolist()\n# y = y_train\n\n# GB_clf.fit(X,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"084bdc5122bc08c8142196fb4512ab3ed7a97ac0"},"cell_type":"code","source":"# f1 = cross_val_score(GB_clf, X, y, scoring='f1', cv=5)\n# accuracy = cross_val_score(GB_clf, X, y, cv=5, scoring='accuracy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c29442361f87c9a2c18839624f582869d3708c49"},"cell_type":"code","source":"print('f1 score:{}\\nacurracy: {}'.format(round(f1.mean(),2),round(accuracy.mean(),2)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9488ad23b1ab5d4838c4d6a9fb2632237a3e55aa"},"cell_type":"markdown","source":"## 7.3 GridSearchCV for logistic regression classifier"},{"metadata":{"trusted":true,"_uuid":"e455712a090af2ddf2d81edf16775c579481d12e"},"cell_type":"code","source":"#weights = [2,3]\n# param_grid = {'C': [0.5,10],\n# 'class_weight':[{0:1, 1:w} for w in weights]}\n# param_grid ={\"learning_rate\":(0.1,0.5),\n# 'max_depth' : range(2,5,10),\n# 'min_samples_split': range(2,5,10),\n# 'min_samples_leaf' : range(2,5,10),\n# #'max_features':range(4,8),\n# \"subsample\":(0.5,0.8)} \nparam_grid ={\"learning_rate\":(0.5),\n 'max_depth' : range(2),\n 'min_samples_split': range(2),\n 'min_samples_leaf' : range(2),\n #'max_features':range(4,8),\n \"subsample\":(0.8)} ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2ae10ce017faf2573c5bdefe6bd06fee2c7ffc00"},"cell_type":"code","source":"X_grid5 = X_train.tolist()\ny_grid5 = y_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4943ab8921fde17dad595b9aab8a1cbb104739c1"},"cell_type":"code","source":"#Lr_clf = LogisticRegression()\nGB_clf=GradientBoostingClassifier()\n# gs= GridSearchCV(estimator=Lr_clf, param_grid=param_grid, cv=2,scoring='f1') # verbose=15, n_jobs=-1\ngs= GridSearchCV(estimator=GB_clf, param_grid=param_grid, cv=2,scoring='f1') # verbose=15, n_jobs=-1\ngs.fit(X_grid5, y_grid5)\nbest_model=gs.best_estimator_ ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22c6539af1a02ee4e2a10a28a0891491b53b419e"},"cell_type":"code","source":"best_model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84e4c05efd706001257cee11c204d82a00c13151"},"cell_type":"markdown","source":"### 7.4 Results After Grid Search"},{"metadata":{"trusted":true,"_uuid":"da98d52e4d82d8770f8c51178a8c965915898514"},"cell_type":"code","source":"y_pred= best_model.predict(X_grid5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"844e44ddd6ff2f803babbd09e3d3f6635f3341fb"},"cell_type":"code","source":"f1 = cross_val_score(best_model, X_grid5, y_grid5, scoring='f1', cv=2)\naccuracy = cross_val_score(best_model, X_grid5, y_grid5, cv=2, scoring='accuracy')\ncm = confusion_matrix(y_true=y_grid5, y_pred=y_pred)\n\nprint('f1-score:',round(f1.mean(),2),'\\naccuracy:',round(accuracy.mean(),2),'\\n______________\\n')\ncm = cm\nprint (classification_report(y_grid5, y_pred))\n\nprint('confusion matrix:')\npd.DataFrame(cm, \n index=best_model.classes_, \n columns=best_model.classes_)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9d10b979c4b366983cc8c154501c975055efa3f"},"cell_type":"markdown","source":"## 7.5 Evaluate the model"},{"metadata":{"trusted":true,"_uuid":"0a32f3a16efba2b34bebe6f141a4518a4b395635"},"cell_type":"code","source":"X1_val=X_val.tolist()\ny1_val=y_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65151e24901be8ae15309a2393f6b706240a025a"},"cell_type":"code","source":"best_model.fit(X1_val,y1_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c50d3a026b03cc4d4e62d01e0aca506fff5f6105"},"cell_type":"code","source":"f1 = cross_val_score(best_model, X1_val,y1_val, scoring='f1', cv=3)\nprint('f1 score of \"val\":', round(f1.mean(),2))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d1f39f82ac9ffb010687e39b5a1d7bf8ccd5f0d"},"cell_type":"markdown","source":"# Using NNclalification model"},{"metadata":{"trusted":true,"_uuid":"cd5359a087ef87ebd05ecfcbe2a8e040879b8e9a"},"cell_type":"code","source":"# X_NN = X_train.tolist()\n# y_NN = y_train\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85b9792cc01ca50872d0cf1bdca91557249b6e3b"},"cell_type":"code","source":"# from keras.preprocessing import sequence\n# from keras.models import Sequential\n# from keras.layers import Dense, Embedding\n# from keras.layers import LSTM","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"708bd1120e75f5cf01ac04aeabffade58ad4cb08"},"cell_type":"code","source":"# max_features=50000\n# MAX_SEQUENCE_LENGTH=303","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"806e66c1008e996c8c764e6351cc6addac01a977"},"cell_type":"code","source":"from keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0bd11547b69c5489cf49a61c41a1d7914fcddf1"},"cell_type":"code","source":"# def f1(y_true, y_pred):\n# '''\n# metric from here \n# https://stackoverflow.com/questions/43547402/how-to-calculate-f1-macro-in-keras\n# '''\n# def recall(y_true, y_pred):\n# \"\"\"Recall metric.\n\n# Only computes a batch-wise average of recall.\n\n# Computes the recall, a metric for multi-label classification of\n# how many relevant items are selected.\n# \"\"\"\n# true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n# possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n# recall = true_positives / (possible_positives + K.epsilon())\n# return recall\n\n# def precision(y_true, y_pred):\n# \"\"\"Precision metric.\n\n# Only computes a batch-wise average of precision.\n\n# Computes the precision, a metric for multi-label classification of\n# how many selected items are relevant.\n# \"\"\"\n# true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n# predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n# precision = true_positives / (predicted_positives + K.epsilon())\n# return precision\n# precision = precision(y_true, y_pred)\n# recall = recall(y_true, y_pred)\n# return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8bc8c72c6836eb5a2e3df9809559a901f6ccbc70"},"cell_type":"code","source":"from keras.layers import LSTM, Bidirectional, GlobalMaxPool1D, Dropout\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"add0218d94a4329c430c0c2ecdd66becb3998a70"},"cell_type":"code","source":"# TT=np.array(FF)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10e1f0c5bb321435795ec711a7fa99aeb95f487a"},"cell_type":"code","source":"# # num_words = min(max_features, len(TT))\n# num_words=len(TT)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5f2bb06f68a2d876e97599c75214e036e0a9444"},"cell_type":"code","source":"# from keras.preprocessing.text import Tokenizer\n\n# t = Tokenizer()\n# t.fit_on_texts(quora_best[\"question_text_prep\"])\n\n# vocab_size = len(t.word_index) + 1\n# #vocab_size\n\n# encoded_docs = t.texts_to_sequences(quora_best[\"question_text_prep\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd12c06b7d18a53effb159af789474aab1e07bda"},"cell_type":"code","source":"# from numpy import zeros\n# embedding_matrix = zeros((vocab_size, 300))\n# for word, i in t.word_index.items():\n# embedding_vector = word_model.get(word)\n# if embedding_vector is not None:\n# embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a09cc1e4aa710c371903d02182b57e182841d80"},"cell_type":"code","source":"# from keras.preprocessing.sequence import pad_sequences\n# max_length = 300\n# padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\n# # print(padded_docs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35fcfd4e2779b78c68fa766a24da2f9613b97389"},"cell_type":"code","source":"# embedding_matrix.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ddc3739f807550e45b1b3a4b39410b23984df2e"},"cell_type":"code","source":"# from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n\n\n# X_train, X_val, y_train, y_val = train_test_split(padded_docs,quora_best['target'],\n# train_size=0.7, random_state = 143, stratify=quora_best['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3606326e7c0939597df92c07568750251519377e"},"cell_type":"code","source":"# model = Sequential()\n# #model.add(Embedding(num_words,input_length=MAX_SEQUENCE_LENGTH,weights=[TT],trainable=False,output_dim=MAX_SEQUENCE_LENGTH))\n\n# model.add(Embedding(vocab_size, 300, weights=[embedding_matrix], input_length=max_length, trainable=False))\n# model.add(Bidirectional(LSTM(128,dropout=0.2, recurrent_dropout=0.2, return_sequences=True))) #\n# model.add(Bidirectional(LSTM(64,return_sequences=True))) # dropout=0.2, recurrent_dropout=0.2,\n# #model.add(Attention(MAX_SEQUENCE_LENGTH))\n# model.add(GlobalMaxPool1D())\n# model.add(Dense(16, activation='relu'))\n# model.add(Dropout(0.1))\n# model.add(Dense(1, activation='sigmoid'))\n\n# model.compile(loss='binary_crossentropy',\n# optimizer='adam',\n# metrics=[f1])\n\n# print(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd90267b4856c4f0927b218e5f06206c99482341"},"cell_type":"code","source":"# batch_size = 512 #32","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffa6883f38b28a36d783f6ea782f9148413fcbc3"},"cell_type":"code","source":"# X_train_list=X_train.tolist()\n# X_val_list=X_val.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23f32413f5c9c32e5083a9742553af6dc92e30ff"},"cell_type":"code","source":"# observation_train = np.asarray(X_train_list)\n# observation_val = np.asarray(X_val_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35a95d1b6514baf791a1742b95a69cc6c865a176"},"cell_type":"code","source":"# #weight={0:0.4,1:1}\n# weight={0:1,1:2}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5cd746abb975b9bfa9f50da15789e59c8aab862"},"cell_type":"code","source":"# model.fit(X_train, y_train,\n# batch_size=batch_size,\n# epochs=2,\n# validation_data=(X_val, y_val),\n# class_weight=weight)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c45d070f6333f2f2720d5556bb13eed6317cc55"},"cell_type":"code","source":"# model.fit(observation_train, y_train,\n# batch_size=batch_size,\n# epochs=10,\n# validation_data=(observation_val, y_val),\n# class_weight=weight)\n\n# score, f1_calc = model.evaluate(X_val, y_val,\n# batch_size=batch_size)\n# print('Test score:', score)\n# print('Test f1:', f1_calc)\n# #X_train, X_val, y_train, y_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dffff90e94f2b0288f5921137f23dc44a3e64fdd"},"cell_type":"code","source":"# y_pred_train = model.predict(X_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fda30870b97d226a2d995c21fd4e6e2ef8cda576"},"cell_type":"code","source":"# y_pred_val = model.predict(observation_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3cdb932581a5db01f795066947fbe347212c5749"},"cell_type":"code","source":"from tqdm.auto import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f45a6c8429b2d153af79c8932ba6d3f1dc3c521e"},"cell_type":"code","source":"# def bestThresshold(y_train,y_pred_train):\n# tmp = [0,0,0] # idx, cur, max\n# delta = 0\n# for tmp[0] in tqdm(np.arange(0.1, 0.501, 0.01)):\n# tmp[1] = f1_score(y_train, np.array(y_pred_train)>tmp[0])\n# if tmp[1] > tmp[2]:\n# delta = tmp[0]\n# tmp[2] = tmp[1]\n# print('best threshold is {:.4f} with F1 score: {:.4f}'.format(delta, tmp[2]))\n# return delta\n# delta = bestThresshold(y_train,y_pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a8fe4432978d2f37dcadfcc25e5781be74558df"},"cell_type":"code","source":"# t_test = Tokenizer()\n# t_test.fit_on_texts(quora_test[\"question_text_prep\"])\n\n# #vocab_size = len(t.word_index) + 1\n# #vocab_size\n\n# encoded_docs_test = t_test.texts_to_sequences(quora_test[\"question_text_prep\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26c90659a8d88c47631c090f192a78c5abe2eadc"},"cell_type":"code","source":"# max_length = 300\n# padded_docs_test = pad_sequences(encoded_docs_test, maxlen=max_length, padding='post')\n# #print(padded_docs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"248ed7af2e865c03959a845393e8145c04d94aa5"},"cell_type":"code","source":"# # test_Features = quora_test['vector'] #quora_test['joinvector_all']\n# # X_test_original=test_Features.tolist()\n# # observation_test = np.asarray(X_test_original)\n# # y_test_pred= model.predict(observation_test)\n# y_test_pred= model.predict(padded_docs_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10ab1fffd4afcb745606fea94c001608c7b6531d"},"cell_type":"code","source":"test_Features = quora_test['joinvector_all']\nX_test_original=test_Features.tolist()\ny_test_pred= best_model.predict(X_test_original)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9780a9505200d6f13af1895df850965ae5a2691"},"cell_type":"code","source":"# y_test_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa68c774bc2e5fbd74f9b5ab90dadafc16c6effd"},"cell_type":"code","source":"quora_test_tmp=quora_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a07cfb7509fb1a9f88bb40ca08501e3d64eac28"},"cell_type":"code","source":"# delta=0.4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2cb1ce9423e937167812e73457233018db4f687"},"cell_type":"code","source":"quora_test_tmp[\"pred\"]=y_test_pred #(y_test_pred > delta).astype(int) \nquora_test_tmp1 = quora_test_tmp[['qid','question_text','pred']]\nquora_test_tmp1[quora_test_tmp1['pred']==1].sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e879c83a1426a37eec5e12ebd49de5d45b199001"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5537cfa4ac03659acccff06035351ff1d9a75c6"},"cell_type":"code","source":"sub = pd.read_csv('../input/sample_submission.csv')\nout_df = pd.DataFrame({\"qid\":sub[\"qid\"].values})\nout_df['prediction'] = y_test_pred\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40c61dbcec89344c984b6ac121d80f5d62b6d58e"},"cell_type":"code","source":"# sub = pd.read_csv('../input/sample_submission.csv')\n# out_df = pd.DataFrame({\"qid\":sub[\"qid\"].values})\n# out_df['prediction'] =y_test_pred# (y_test_pred > delta).astype(int) #y_test_pred\n# out_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"95f80da4d6f97cd73da89c9825a3a5b898978de6"},"cell_type":"markdown","source":"### Let's check the proportions of the submission file"},{"metadata":{"trusted":true,"_uuid":"2c427c03cd3215d20214608e5f907a7a48f5810f"},"cell_type":"code","source":"round(out_df['prediction'].value_counts(normalize =True),3)*100","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37d543a957b046212d9c389cf6df1d5ec9736f67"},"cell_type":"markdown","source":"### We can see that the proportion of the insincire questions is similar and a little bit more than the original train file"},{"metadata":{"trusted":true,"_uuid":"1fbef1a6a9fe984b561788781cd7fba31a70e53f"},"cell_type":"markdown","source":"## 8. Concluding Remarks and Work in Progress "},{"metadata":{"_uuid":"8fd8a2ae1da8f845d7396526e65ce905a019aa21"},"cell_type":"markdown","source":"* ### We used Logistic Classifier as our classification model. \n* ### We have reached the following results: f1_score: 0.8 vs. 0.79 , in the training and testing (val) sets, respectively. \n\n* ### The results indicate on the model's robustness as well as its ability to distiguish between insencere and sincere questions. \n### Later on we have used NN algorithems to predict the same data\n### Points for further exploration:\n### 1. Joining the Clustering feature to the combined vector\n### 2. Trying different Classification based on Neural Net algorithems with the combined vector"},{"metadata":{"trusted":true,"_uuid":"c2e530c3313e9e80db9729ddd4af85e546b4fda7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}