{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"\n# Trying Different ML Techniques for Classification.\n\n### Techniques i'll be using:\n* Naive Bayes\n* SVM\n* RNN\n* GRU\n* LSTM\n* Attention model\n\n> For Embedding and Preprocessing, i've referenced the amazing kernels listed here , Do Check Them Out: \n* <a href='https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings'>A Look At Different Embeddings</a>\n* <a href='https://www.kaggle.com/theoviel/improve-your-score-with-some-text-preprocessing'>Improve Your Score With Some Preprocessing</a>"},{"metadata":{"_uuid":"9a81a21ae16c77e4c3a0ed1e0bb9e354e394c776"},"cell_type":"markdown","source":"### First Let's Import The Libraries Needed.\n"},{"metadata":{"_uuid":"16ba7c5bbabaa639e4f2d4b85b148d6cfc8b1886"},"cell_type":"markdown","source":"### A Note To Remember : Sincere = 0, Insincere = 1"},{"metadata":{"trusted":true,"_uuid":"b4e16173c738d2c5229f24b4df3cb1499dd431fb"},"cell_type":"code","source":"# the obvious\nimport numpy as np\nimport pandas as pd\n\n# core utility modules\nfrom os import listdir, path\nimport string\nfrom collections import Counter\nimport time\nimport gc\n\n# for visualization\nimport matplotlib.pyplot as plt\nfrom wordcloud import WordCloud\n%matplotlib inline\n\n# for preprocessing and feature extraction\nimport keras.preprocessing.text as text\nimport keras.preprocessing.sequence as seq \nfrom keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split, RandomizedSearchCV\n# from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\nfrom nltk.tokenize import word_tokenize\n# from nltk.corpus import stopwords\n# from nltk.stem import PorterStemmer\n\n# for logging and early stopping and learning rate scheduling\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau, TensorBoard\n\n# for the metric (F1 Score)\nfrom sklearn.metrics import f1_score\n\n# for model creation and training\nfrom keras.models import Sequential, Model\nfrom keras.layers import Layer, Dense, Input, LSTM, Dropout, Bidirectional, CuDNNLSTM, CuDNNGRU, SimpleRNN, Embedding, GlobalMaxPool1D\nimport keras.backend as K\nfrom sklearn.svm import SVC\nfrom keras.optimizers import Adam\nfrom keras import initializers\n\n# other imports\nimport operator \nimport re","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4c81e3d41bd32b6658ceb0a56323fd6b6b0a072"},"cell_type":"markdown","source":"### Let's Declare Some Globals"},{"metadata":{"trusted":true,"_uuid":"2455c66a63648dc1c089d75d0ffe798e1f9ee2e9"},"cell_type":"code","source":"max_seq_len = 60 # The Max Length Of The Text Sequence\nembed_size = 300 # The Number Of Features In The Embedding For A Single Word\nmax_features = 50000 # The Maximum Number Of Words In The Vocab\nEMBEDDING = 'glove.840B.300d' # Learned Embedding To Be Used, Change This For Using Different Embeddings\nMODEL = 'attention' # The Model To Use To Classify The Insincere/Sincere Questions, Other Possible Vals Are : 'nb', 'svm', 'rnn', 'gru' and 'lstm'\nembedding_matrix = 'None' # The Embeddigns Matrix\nembeddings_idx = 'None' # The Mappings From Embedding Index To The Embedding\ncheckpoint = ModelCheckpoint('./checkpoints/', monitor='val_acc', verbose=0, save_best_only=True)\nearlystop = EarlyStopping(monitor='val_acc', min_delta=0, patience=1, verbose=0)\ntensorboard = TensorBoard(log_dir='./logs', histogram_freq=0, batch_size=32, write_graph=True, write_grads=True, write_images=True)\nreducelr = ReduceLROnPlateau(monitor='val_acc', factor=0.1, patience=3, verbose=0, mode='auto', min_delta=0.0001, cooldown=0, min_lr=0)\nthresh = 0.4","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0d42d4c57f301ad33392b2d9f1b51298c24eac4"},"cell_type":"markdown","source":"### Now Let's Get The Data"},{"metadata":{"trusted":true,"_uuid":"f7a990764f0fdd5f82a5ae3d56c561db0e3041fa"},"cell_type":"code","source":"listdir('../input')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"589f3a7b90fa9dff10f01976ed8bc3e2f0a70cd8"},"cell_type":"code","source":"train_set = pd.read_csv('../input/train.csv')\ntest_set = pd.read_csv('../input/test.csv')\ntrain_set.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3a3c7f8a8f3cd4eb2bc64d2d099615d27b2ff15"},"cell_type":"code","source":"test_set.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66a621562e0d7e2caec75d21f58a5242c871ee0e"},"cell_type":"markdown","source":"### Let's see the distribution"},{"metadata":{"trusted":true,"_uuid":"687d0b5cd252bbdc76c219bf5808a3f675638a3e"},"cell_type":"code","source":"x = train_set['target'].value_counts(dropna=False)\nprint(x)\nsincere_examples = x[0]\ninsincere_examples = x[1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54c1f9d6e578c09fa65202b89bc42e63326e9e7c"},"cell_type":"markdown","source":"So there are no missing values in the dataset, but, there is a large difference in data distribution as #examples[i==0] >> #examples[i==1]\n\n### Visualizing The Data Separation"},{"metadata":{"trusted":true,"_uuid":"b96f9539692f294a95f5c1d59b2c9869ae2da097"},"cell_type":"code","source":"plt.hist(train_set['target'], bins=range(0,6), align='left', rwidth=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bde15bd2b6f616715d542a2cf7fe764909aa852b"},"cell_type":"markdown","source":"### Let's Do some EDA\nFirstly, Let's See The Word Length Distributions."},{"metadata":{"trusted":true,"_uuid":"6bc4ac5e43c1c5674d6623313cd233d5bf64ac9b"},"cell_type":"code","source":"# max and min question lengths\n# to remove punctuations : translate(str.maketrans('','',string.punctuation))\nlengths_without_puncs = [len(i.translate(str.maketrans('','',string.punctuation)).split()) for i in train_set['question_text']]\nlengths = [len(i.split()) for i in train_set['question_text']]\nprint('With Punctuations: ')\nprint('Max Length Of Questions: {}'.format(np.max(lengths)))\nprint('Min Length Of Questions: {}'.format(np.min(lengths)))\nprint('Without Punctuations: ')\nprint('Max Length Of Questions: {}'.format(np.max(lengths_without_puncs)))\nprint('Min Length Of Questions: {}'.format(np.min(lengths_without_puncs)))\n# print(len(lengths))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"156a63445227592dce3e64499ba7c7a969d61567"},"cell_type":"markdown","source":"So We Can See That the length of questions range from 0 to 132. We've to remove the empty length questions as they'll not contribute anything to learning. First Let's see how many 0 length questions are there."},{"metadata":{"trusted":true,"_uuid":"3ed97196c37196f8acd424da8c0ac57f1964d248"},"cell_type":"code","source":"print(len(lengths_without_puncs) - np.count_nonzero(lengths_without_puncs)) # Will remove them or use fillna to overcome this","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"858552df5329770c2002fe01305ecd0916e2659a"},"cell_type":"code","source":"plt.hist(lengths)\nplt.yscale('log')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6fc9507c643a85ef9075b094d5d232eb57615aa"},"cell_type":"markdown","source":"So Most Questions Range In Length From 0 to 60."},{"metadata":{"trusted":true,"_uuid":"d541e04c38638429694ca76b38fed97bab5cb4c5"},"cell_type":"markdown","source":"### Let's Do Some Preprocessing on the data"},{"metadata":{"trusted":true,"_uuid":"01804b679adb5ee1756f337a622014a01fc74fa5"},"cell_type":"code","source":"# Code from https://www.kaggle.com/theoviel/improve-your-score-with-some-text-preprocessing\n\ncontraction_mapping = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",  \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\" }\n\npunct = \"/-'?!.,#$%\\'()*+-/:;<=>@[\\\\]^_`{|}~\" + '\"\"“”’' + '∞θ÷α•à−β∅³π‘₹´°£€\\×™√²—–&'\npunct_mapping = {\"‘\": \"'\", \"₹\": \"e\", \"´\": \"'\", \"°\": \"\", \"€\": \"e\", \"™\": \"tm\", \"√\": \" sqrt \", \"×\": \"x\", \"²\": \"2\", \"—\": \"-\", \"–\": \"-\", \"’\": \"'\", \"_\": \"-\", \"`\": \"'\", '“': '\"', '”': '\"', '“': '\"', \"£\": \"e\", '∞': 'infinity', 'θ': 'theta', '÷': '/', 'α': 'alpha', '•': '.', 'à': 'a', '−': '-', 'β': 'beta', '∅': '', '³': '3', 'π': 'pi', }\n\nmispell_dict = {'colour': 'color', 'centre': 'center', 'favourite': 'favorite', 'travelling': 'traveling', 'counselling': 'counseling', 'theatre': 'theater', 'cancelled': 'canceled', 'labour': 'labor', 'organisation': 'organization', 'wwii': 'world war 2', 'citicise': 'criticize', 'youtu ': 'youtube ', 'Qoura': 'Quora', 'sallary': 'salary', 'Whta': 'What', 'narcisist': 'narcissist', 'howdo': 'how do', 'whatare': 'what are', 'howcan': 'how can', 'howmuch': 'how much', 'howmany': 'how many', 'whydo': 'why do', 'doI': 'do I', 'theBest': 'the best', 'howdoes': 'how does', 'mastrubation': 'masturbation', 'mastrubate': 'masturbate', \"mastrubating\": 'masturbating', 'pennis': 'penis', 'Etherium': 'Ethereum', 'narcissit': 'narcissist', 'bigdata': 'big data', '2k17': '2017', '2k18': '2018', 'qouta': 'quota', 'exboyfriend': 'ex boyfriend', 'airhostess': 'air hostess', \"whst\": 'what', 'watsapp': 'whatsapp', 'demonitisation': 'demonetization', 'demonitization': 'demonetization', 'demonetisation': 'demonetization'}\n\ndef clean_contractions(text, mapping=contraction_mapping):\n    specials = [\"’\", \"‘\", \"´\", \"`\"]\n    for s in specials:\n        text = text.replace(s, \"'\")\n    text = ' '.join([mapping[t] if t in mapping else t for t in text.split(\" \")])\n    return text\n\ndef clean_special_chars(text, punct=punct, mapping=punct_mapping):\n    for p in mapping:\n        text = text.replace(p, mapping[p])\n    \n    for p in punct:\n        text = text.replace(p, f' {p} ')\n    \n    specials = {'\\u200b': ' ', '…': ' ... ', '\\ufeff': '', 'करना': '', 'है': ''}  # Other special characters that I have to deal with in last\n    for s in specials:\n        text = text.replace(s, specials[s])\n    \n    return text\n\ndef correct_spelling(x, dictionary=mispell_dict):\n    for word in dictionary.keys():\n        x = x.replace(word, dictionary[word])\n    return x\n\n\ndef clean(text):\n    text = text.lower()\n    text = clean_contractions(text)\n    text = clean_special_chars(text)\n    text = correct_spelling(text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c1c8f986f919b51afd0e56bc8fd64943294123b"},"cell_type":"code","source":"sincere_counts = Counter()\ninsincere_counts = Counter()\nword_dict = Counter()\nsincere_to_insincere_ratio = Counter()\n\ndef prepare_dicts():\n    qs = [clean(i) for i in train_set['question_text']]\n    lbl = [j for j in train_set['target']]\n    for i,j in zip(qs,lbl):\n        words = i.split()\n        # making the dictionaries\n        for word in words:\n            word_dict[word] += 1\n            if j == 0:\n                sincere_counts[word] += 1\n            elif j == 1:\n                insincere_counts[word] += 1\n    \n    tst_qs = [clean(i) for i in test_set['question_text']]\n    \n    for i in tst_qs:\n        i = i.split()\n        for j in i:\n            word_dict[j] += 1\n    \n    print('Words in sincere Questions: {}'.format(len(sincere_counts)))\n    print('Words in insincere Questions: {}'.format(len(insincere_counts)))\n    print('Total Words in corpus: {}'.format(len(word_dict)))\n\n    print('Most Common Words in Sincere Questions : ')\n    print(sincere_counts.most_common()[:10])\n    print('Most Common Words in Insincere Questions : ')\n    print(insincere_counts.most_common()[:10])\n\n    for i in sincere_counts:\n        if sincere_counts[i] >= 100:\n            sincere_to_insincere_ratio[i] = np.log(sincere_counts[i]/(insincere_counts[i] + 1))\n\n    print('The Most Sincere Words : ')\n    print(sincere_to_insincere_ratio.most_common()[:10])\n    print('The Most Insincere Words : ')\n    print(list(reversed(sincere_to_insincere_ratio.most_common()))[:10])\n    \n#     return sincere_counts, insincere_counts, word_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a92c62fd7a7d16d5b680f94e3032a56b87062bfb"},"cell_type":"code","source":"prepare_dicts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a5d57e4d77b8cf138977f5349b73c5ddb9db028"},"cell_type":"markdown","source":"### Aaannddd a wordcloud for fun"},{"metadata":{"trusted":true,"_uuid":"000654ec215bd6c3640eabfc7c9f99a516a3dab8"},"cell_type":"code","source":"wordCloud = WordCloud().generate(\" \".join([key[0] for key in sincere_to_insincere_ratio.most_common()[:10]]))\nfig = plt.figure()\nplt.imshow(wordCloud, interpolation=\"bilinear\")\nplt.axis(\"off\")\nplt.margins(x=0, y=0)\nfig.suptitle('Most Common Words In Sincere Questions', fontsize=14, fontweight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d09d16c8b6f4f36885338f34ab918b63313fba1e"},"cell_type":"code","source":"wordCloud = WordCloud().generate(\" \".join([key[0] for key in list(reversed(sincere_to_insincere_ratio.most_common()))[:10]]))\nfig = plt.figure()\nplt.imshow(wordCloud, interpolation=\"bilinear\")\nplt.axis(\"off\")\nplt.margins(x=0, y=0)\nfig.suptitle('Most Common Words In Insincere Questions', fontsize=14, fontweight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf88178a007ff76d6e8ab3d1c584854221b9b77d"},"cell_type":"code","source":"# Creating The Datasets First\ntrain_x = list(train_set['question_text'].fillna(\"_na_\").values)\ntrain_y = train_set['target'].values\n\ntest_x = list(test_set['question_text'].fillna(\"_na_\").values)\n\ntrain_x, val_x, train_y, val_y = train_test_split(train_x, train_y, test_size=0.2)\n\n# Cleaning Up The Data (train + test)\ntrain_x = [clean(i) for i in train_x]\nval_x = [clean(i) for i in val_x]\ntest_x = [clean(i) for i in test_x]\n\n# An Example From Train Set\nprint('An Example From Train Set: ')\nprint(train_x[0])\n\n## Tokenize the sentences\ntokenizer = text.Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_x))\ntrain_X = tokenizer.texts_to_sequences(train_x)\nval_X = tokenizer.texts_to_sequences(val_x)\ntest_X = tokenizer.texts_to_sequences(test_x)\n\n# After Tokenizing\nprint('After Tokenizing: ')\nprint(train_X[0])\n\n## Pad the sentences \ntrain_X = seq.pad_sequences(train_X, maxlen=max_seq_len)\nval_X = seq.pad_sequences(val_X, maxlen=max_seq_len)\ntest_X = seq.pad_sequences(test_X, maxlen=max_seq_len)\n\n# After Padding\nprint('After Padding: ')\nprint(train_X[0])\n\nprint(np.shape(train_X), np.shape(train_y), np.shape(val_X), np.shape(val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"773ab1a797d93fe72d76a8f3fbd0dd65eea4987b"},"cell_type":"markdown","source":"### Let's Get Those Embeddings"},{"metadata":{"trusted":true,"_uuid":"a80100836a46cf604d0978dae973843ed755690c"},"cell_type":"code","source":"# Thanks to https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n\ndef get_embeddings(embedding_name, mode='new'):\n    # Getting The File\n    filePath = '../input/embeddings/{0}/{0}.txt'.format(embedding_name)\n    \n    # Creating a Dictionary of format {word : Embedding}\n    if mode == 'new':\n        embeddings_idx = dict(get_coefs(*i.split(\" \")) for i in open(filePath))\n        # All Embeddings\n        all_embs = np.stack(embeddings_idx.values())\n\n        # Creating The Embedding Matrix with distribution, for if there is a missing word in the embeddings, it'll have\n        # the embedding vector with the same distribution\n        emb_mean,emb_std = all_embs.mean(), all_embs.std()\n        embed_size = all_embs.shape[1]\n\n        word_index = tokenizer.word_index\n        nb_words = min(max_features, len(word_index))\n        embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n\n        # Filling in the given learned embeddings in the embedding matrix\n        for word, i in word_index.items():\n            if i >= max_features: continue\n            embedding_vector = embeddings_idx.get(word)\n            if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n                \n    return embeddings_idx, embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fb0ed96ada1784bfe6c2b24f679767b3821f28c2"},"cell_type":"code","source":"# Checking OOV words (Out Of Vocab words)\ndef check_coverage(vocab, embeddings_index):\n    known_words = {}\n    unknown_words = {}\n    nb_known_words = 0\n    nb_unknown_words = 0\n    for word in vocab.keys():\n        try:\n            known_words[word] = embeddings_index[word]\n            nb_known_words += vocab[word]\n        except:\n            unknown_words[word] = vocab[word]\n            nb_unknown_words += vocab[word]\n            pass\n\n    print('Found embeddings for {:.2%} of vocab'.format(len(known_words) / len(vocab)))\n    print('Found embeddings for  {:.2%} of all text'.format(nb_known_words / (nb_known_words + nb_unknown_words)))\n    unknown_words = sorted(unknown_words.items(), key=operator.itemgetter(1))[::-1]\n\n    return unknown_words","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f88f32f555f42ca503f15c8cee394fe808674cd"},"cell_type":"code","source":"embedding_idxs, embedding_mtx = get_embeddings(EMBEDDING, 'new')\nunk_wrds = check_coverage(word_dict, embedding_idxs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bdb9df3e9182a8c98dd345f33cb34d8bc5aff804"},"cell_type":"code","source":"print(unk_wrds[:10])\nprint(np.shape(embedding_mtx))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"231e3ff982cff1d26478b79a15ef84949997ca16"},"cell_type":"markdown","source":"### Enough Chit-Chat, Let's Start Making Some Models And Getting Some Results"},{"metadata":{"trusted":true,"_uuid":"49a75fa36de35b2ec7a5f6844289f961c53d29f9"},"cell_type":"code","source":"# Create The Model\ndef get_model(model_type):\n    if model_type == 'nb':\n        # create naivebayes model\n        model = NaiveBayes()\n\n    elif model_type == 'svm':\n        # create svm model\n        model = __SVC__()\n\n    elif model_type == 'rnn':\n        inp = Input(shape=(max_seq_len,))\n        layer = Embedding(max_features, embed_size, weights=[embedding_mtx], trainable=False)(inp)\n#         layer = SimpleRNN(128, return_sequences=True)(layer)\n        layer = SimpleRNN(32, return_sequences=True)(layer)\n        layer = GlobalMaxPool1D()(layer)\n        layer = Dense(16, activation='relu')(layer)\n        layer = Dropout(0.1)(layer)\n        layer = Dense(1, activation='sigmoid')(layer)\n        model = Model(inputs=inp, outputs=layer)\n\n    elif model_type == 'lstm':\n        # create lstm model\n        inp = Input(shape=(max_seq_len,))\n        layer = Embedding(max_features, embed_size, weights=[embedding_mtx], trainable=False)(inp)\n#         layer = Bidirectional(CuDNNLSTM(128, return_sequences=True))(layer)\n        layer = Bidirectional(CuDNNLSTM(32, return_sequences=True))(layer)\n        layer = GlobalMaxPool1D()(layer)\n        layer = Dense(16, activation='relu')(layer)\n        layer = Dropout(0.1)(layer)\n        layer = Dense(1, activation='sigmoid')(layer)\n        model = Model(inputs=inp, outputs=layer)\n\n    elif model_type == 'gru':\n        # create attention model\n        inp = Input(shape=(max_seq_len,))\n        layer = Embedding(max_features, embed_size, weights=[embedding_mtx], trainable=False)(inp)\n#         layer = Bidirectional(CuDNNGRU(128, return_sequences=True))(layer)\n        layer = Bidirectional(CuDNNGRU(32, return_sequences=True))(layer)\n        layer = GlobalMaxPool1D()(layer)\n        layer = Dense(16, activation='relu')(layer)\n        layer = Dropout(0.1)(layer)\n        layer = Dense(1, activation='sigmoid')(layer)\n        model = Model(inputs=inp, outputs=layer)\n\n    elif model_type == 'attention':\n        inp = Input(shape=(max_seq_len,))\n        layer = Embedding(max_features, embed_size, weights=[embedding_mtx], trainable=False)(inp)\n#         layer = Bidirectional(CuDNNLSTM(128, return_sequences=True))(layer)\n        layer = Bidirectional(CuDNNLSTM(32, return_sequences=True))(layer)\n        layer = Attention(max_seq_len)(layer)\n        layer = Dense(16, activation='relu')(layer)\n        layer = Dropout(0.1)(layer)\n        layer = Dense(1, activation='sigmoid')(layer)\n        model = Model(inputs=inp, outputs=layer)\n\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a883e0d205853cb97ab7968216efe869a9e7bf5"},"cell_type":"markdown","source":"Defining Naive Bayes, SVM and Attention Layer"},{"metadata":{"trusted":true,"_uuid":"b76c0f69201351c0f5ea6bd723cc9829238f6f79"},"cell_type":"code","source":"# Defining The NaiveBayes Class\nclass NaiveBayes():\n    def __init__(self):\n        self.sincere_example_count = sincere_examples\n        self.insincere_example_count = insincere_examples\n        self.total_examples = x[0]+x[1]\n        self.sincere_dict = sincere_counts\n        self.insincere_dict = insincere_counts\n        self.word_dict= word_dict\n        self.sincere_word_count = np.sum(list(sincere_counts.values()))\n        self.insincere_word_count = np.sum(list(insincere_counts.values()))\n        self.sincere_prob = self.sincere_example_count / self.total_examples\n        self.insincere_prob = self.insincere_example_count / self.total_examples\n    \n    def summary(self):\n        print('Positive Examples : {}, Negative Examples : {}, Total Examples : {}'.format(self.sincere_example_count, self.insincere_example_count, self.total_examples))\n    \n    def predict(self, x_test):\n        # The NB Prediction with Laplace Smoothing\n        print('Predicting...')\n        predictions = []\n        for example in x_test:\n            p_words = np.prod([word_dict[j]/np.sum(list(word_dict.values())) for j in example.split()])\n            p_words += 2\n            sincere_prob_num = np.prod([sincere_counts[j]/self.sincere_word_count for j in example.split()]) * self.sincere_prob\n            insincere_prob_num = np.prod([insincere_counts[j]/self.insincere_word_count for j in example.split()]) * self.insincere_prob\n\n            sincere_prob = sincere_prob_num/p_words\n            insincere_prob = insincere_prob_num/p_words\n\n#             print('Sincere_prob: {}, Insincere_prob: {}'.format(sincere_prob, insincere_prob))\n            predictions.append(np.argmax([sincere_prob, insincere_prob]))\n#             print('predicted Class : {}'.format(np.argmax([sincere_prob, insincere_prob])))\n        return predictions\n\n# The SVM Class\nclass __SVC__(SVC):\n    def __init__(self):\n        super(__SVC__,self).__init__(verbose=True)\n        print('initializing...')\n    \n    def summary(self):\n        print(self.__dict__)\n        \n    def prepare_data(self):\n        self.X_train = [embedding_mtx[i] for example in train_X for i in example]\n        self.X_val = [embedding_mtx[i] for example in val_X for i in example]\n        self.X_test = [embedding_mtx[i] for example in test_X for i in example]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a8ee132ec25c3ae87d6dfde80d73c7b09746d8bf"},"cell_type":"code","source":"class Attention(Layer):\n    def __init__(self, step_dim, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n        self.features_dim = 0\n        self.step_dim = step_dim\n        self.bias = True\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name))\n        \n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     )\n        else:\n            self.b = None\n\n        self.built = True\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b11f7d083284a1258ef9ae6e41f45b0bdad95bb"},"cell_type":"markdown","source":"## That's it. Let's Create The Submission Files From Each Model.\n🤞"},{"metadata":{"trusted":true,"_uuid":"7322bf9f9d92354f0c02fcc84ef5ab1275145dbe"},"cell_type":"code","source":"def print_f1s(predictions):\n    for threshold in np.arange(0.1, 0.501, 0.01):\n        threshold = np.round(threshold, 2)\n        print(\"F1 score at threshold {0} is {1}\".format(threshold, f1_score(val_y, (predictions>threshold).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4db12f1d97b48e06d59585d8100c44510d607bce"},"cell_type":"markdown","source":"\n## Naive Bayes, Working Without Embeddings"},{"metadata":{"trusted":true,"_uuid":"0e9aa8a49ce6f9911a1ac23e7979020566313b1e"},"cell_type":"code","source":"# nb = get_model('nb')\n# nb.summary()\n# predictions = nb.predict(val_x)\n# predictions_nb = nb.predict(test_x)\n# print('Done!')\n\n# print_f1s(predictions)\n    \n# predictions_nb = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\n# predictions_nb['prediction'] = predictions_nb\n# predictions_nb.to_csv(\"submission_nb.csv\", index=False)\n\n# #freeing up some memory\n# del nb, word_dict, sincere_counts, insincere_counts, sincere_to_insincere_ratio\n\n# gc.collect()\n# time.sleep(10)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0a222a26e88748783f0617298640aba798a1cd1"},"cell_type":"markdown","source":"## SVM, Simple ML Classifier"},{"metadata":{"trusted":true,"_uuid":"6db056a1ab3870da1d5fb7097f9aeb7f52014fc1"},"cell_type":"code","source":"# svm = get_model('svm')\n# svm.summary()\n# svm.prepare_data()\n# svm.fit(svm.X_train, train_y)\n\n# predictions = svm.predict(svm.X_val)\n\n# predictions_svm = svm.predict(svm.X_test)\n\n# print_f1s(predictions)\n    \n# # predictions_svm = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\n# # predictions_svm['prediction'] = predictions_svm\n# # predictions_svm.to_csv(\"submission_svm.csv\", index=False)\n\n# del svm\n# gc.collect()\n# time.sleep(10)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e81d2ff21379037b35f0da6bf5b92731ed54e4c7"},"cell_type":"markdown","source":"## Simple RNN, Let's get reccurring..."},{"metadata":{"trusted":true,"_uuid":"a0547779f5bdd207263e42dea516506b34e11f1a"},"cell_type":"code","source":"rnn = get_model('rnn')\nrnn.summary()\nrnn.compile(loss='binary_crossentropy', optimizer=Adam(lr=1e-3), metrics=['accuracy'])\nrnn.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y), callbacks=[earlystop, reducelr])\n\n\npredictions_rnn_real = rnn.predict(test_X)\npredictions_rnn = (predictions_rnn_real >= thresh).astype(int)\npredictions_val_rnn = rnn.predict(val_X, batch_size=1024)\n\nprint_f1s(predictions_val_rnn)\n\n# prediction_rnn = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\n# prediction_rnn['prediction'] = predictions_rnn\n# prediction_rnn.to_csv(\"submission.csv\", index=False)\n\ndel rnn\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c07713bcfc069d07e30aea270eade3f9fb21feb2"},"cell_type":"markdown","source":"## GRU, I'll Remember This."},{"metadata":{"trusted":true,"_uuid":"5b57c02f0545ce438070e34da9c2bae0fed9d66e"},"cell_type":"code","source":"gru = get_model('gru')\ngru.summary()\ngru.compile(loss='binary_crossentropy', optimizer=Adam(lr=1e-3), metrics=['accuracy'])\ngru.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y), callbacks=[earlystop, reducelr])\n\npredictions_gru_real = gru.predict(test_X)\npredictions_gru = (predictions_gru_real >= thresh).astype(int)\npredictions_val_gru = gru.predict(val_X, batch_size=1024)\n\nprint_f1s(predictions_val_gru)\n\nprediction_gru = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\nprediction_gru['prediction'] = predictions_gru\nprediction_gru.to_csv(\"submission.csv\", index=False)\n\ndel gru\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a16a62e144084f5d701a3cc2ef75659252a512a"},"cell_type":"markdown","source":"## LSTM it is."},{"metadata":{"trusted":true,"_uuid":"aa588bd84b8f1e1524fcfdd898ee387fc8f88352"},"cell_type":"code","source":"lstm = get_model('lstm')\nlstm.summary()\nlstm.compile(loss='binary_crossentropy', optimizer=Adam(lr=1e-3), metrics=['accuracy'])\nlstm.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y), callbacks=[earlystop, reducelr])\n\npredictions_lstm_real = lstm.predict(test_X)\npredictions_lstm = (predictions_lstm_real >= thresh).astype(int)\npredictions_val_lstm = lstm.predict(val_X, batch_size=1024)\n\nprint_f1s(predictions_val_lstm)\n\n# prediction_lstm = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\n# prediction_lstm['prediction'] = predictions_lstm\n# prediction_lstm.to_csv(\"submission.csv\", index=False)\n\ndel lstm\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8469bcfc0789bb8dee76d72eecfb05ba4d0dd68"},"cell_type":"markdown","source":"## Attention Folks!"},{"metadata":{"trusted":true,"_uuid":"f20e04511507f135ea0fa496a170e5e72ee32262"},"cell_type":"code","source":"attention = get_model('attention')\nattention.summary()\nattention.compile(loss='binary_crossentropy', optimizer=Adam(lr=1e-3), metrics=['accuracy'])\nattention.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y), callbacks=[earlystop, reducelr])\n\npredictions_attention_real = attention.predict(test_X)\npredictions_attention = (predictions_attention_real >= thresh).astype(int)\npredictions_val_attention = attention.predict(val_X, batch_size=1024)\n\nprint_f1s(predictions_val_attention)\n\n# prediction_attention = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\n# prediction_attention['prediction'] = predictions_attention\n# prediction_attention.to_csv(\"submission.csv\", index=False)\n\ndel attention\ngc.collect()\ntime.sleep(10)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab71fd5e5dc466b61e1a150d20bd06ca55646a33"},"cell_type":"markdown","source":"Let's See The Combined Performance"},{"metadata":{"trusted":true,"_uuid":"491b175822bbc2a11c65eb221f303d98d012c2f7"},"cell_type":"code","source":"val_preds = 0.50*predictions_val_gru + 0.25*predictions_val_lstm + 0.25*predictions_val_attention\nval_preds = (val_preds > thresh).astype(int)\nprint_f1s(val_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f26542fb19fad353ebf002704db8ad82355f318"},"cell_type":"code","source":"final_preds = 0.50*predictions_gru_real + 0.25*predictions_lstm_real + 0.25*predictions_attention_real\nfinal_preds = (final_preds > thresh).astype(int)\n\nfinal_prediction = pd.DataFrame({\"qid\":test_set[\"qid\"].values})\nfinal_prediction['prediction'] = final_preds\nfinal_prediction.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df89a80cc460316a8eed38439808fa1b3bdab4fd"},"cell_type":"markdown","source":"## Comparing All Performances"},{"metadata":{"trusted":true,"_uuid":"d1e7038168cb0b205235b7d97c8fb2d59e3a13ce"},"cell_type":"markdown","source":"Score Obtained From Naive Bayes : <br />\nScore Obtained From SVM : <br />\nScore Obtained From Simple RNN : <b>0.653</b><br />\nScore Obtained From Bidirectional GRU : <b>0.662</b><br />\nScore Obtained From Bidirectional LSTM : <b>0.656</b><br />\nScore Obtained From Attention Model : <b>0.656</b><br />\nScore Obtained From Combining (RNN, GRU, LSTM and Attention) models : 0.649<br />"},{"metadata":{"trusted":true,"_uuid":"b0d63dc9e0aae84a3571a76c2f7116a84fe1ebe7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}