{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Quora Insicere Questions Classification\n\n**Experiment 1:**\n- Without pretrained + LSTM\n\n**Experiment 2:**\n- With pretrained(Google news pretrained Embeddings ) + LSTM"},{"metadata":{"trusted":true},"cell_type":"code","source":"### importing the libraries \nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nfrom gensim.models import KeyedVectors\nimport re ## Regular expresssions\nfrom nltk import word_tokenize\nfrom sklearn import metrics\nfrom gensim.models import KeyedVectors\nimport operator\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### reading the data files\ntest_data=pd.read_csv(\"../input/quora-insincere-questions-classification/test.csv\")\ntrain_data=pd.read_csv(\"../input/quora-insincere-questions-classification/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### splitting the dataset in to the training set and validation set \ntrain,val=train_test_split(train_data,test_size=0.2,stratify=train_data.target,random_state=123)\nprint(\"Shape of the Training set :\",train.shape)\nprint(\"Shape of the Validation set :\",val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n### unzipping all the pretrained embeddings\n!unzip ../input/quora-insincere-questions-classification/embeddings.zip","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Experiment 1: With out Pretrained Embeddings + LSTM\n- here we will train the embeddings From scratch \n\n**steps to do this**\n        - firstly preprocess the data\n        - Get vocabulary of training corpus\n        - Convert the text to sequence (one hot formate)\n        - pad sequence to maxlength this has to be fixed length\n        - we will input the padded sequence to the Embedding layer in the network."},{"metadata":{},"cell_type":"markdown","source":"#### step:1 Preprocess the data\n**Experiment 1.1 : with out preprocess**\n        - here i am not going to preprocess much here\n        - becz i am just want the sequence as it was \n        - here we do just normalization i.e converting from uppercase to lowercase letters\n\n**Experiment 1.2 : With Preprocess**\n\n    - converting the number to unique code\n    - Applying the contractions \n    - removing the punctuation marks\n    - Normalizing the text\n    "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.question_text[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"contractions={\"I'm\": 'I am',\n \"I'm'a\": 'I am about to',\n \"I'm'o\": 'I am going to',\n \"I've\": 'I have',\n \"I'll\": 'I will',\n \"I'll've\": 'I will have',\n \"I'd\": 'I would',\n \"I'd've\": 'I would have',\n 'Whatcha': 'What are you',\n \"amn't\": 'am not',\n \"ain't\": 'are not',\n \"aren't\": 'are not',\n \"'cause\": 'because',\n \"can't\": 'can not',\n \"can't've\": 'can not have',\n \"could've\": 'could have',\n \"couldn't\": 'could not',\n \"couldn't've\": 'could not have',\n \"daren't\": 'dare not',\n \"daresn't\": 'dare not',\n \"dasn't\": 'dare not',\n \"didn't\": 'did not',\n 'didn’t': 'did not',\n \"don't\": 'do not',\n 'don’t': 'do not',\n \"doesn't\": 'does not',\n \"e'er\": 'ever',\n \"everyone's\": 'everyone is',\n 'finna': 'fixing to',\n 'gimme': 'give me',\n \"gon't\": 'go not',\n 'gonna': 'going to',\n 'gotta': 'got to',\n \"hadn't\": 'had not',\n \"hadn't've\": 'had not have',\n \"hasn't\": 'has not',\n \"haven't\": 'have not',\n \"he've\": 'he have',\n \"he's\": 'he is',\n \"he'll\": 'he will',\n \"he'll've\": 'he will have',\n \"he'd\": 'he would',\n \"he'd've\": 'he would have',\n \"here's\": 'here is',\n \"how're\": 'how are',\n \"how'd\": 'how did',\n \"how'd'y\": 'how do you',\n \"how's\": 'how is',\n \"how'll\": 'how will',\n \"isn't\": 'is not',\n \"it's\": 'it is',\n \"'tis\": 'it is',\n \"'twas\": 'it was',\n \"it'll\": 'it will',\n \"it'll've\": 'it will have',\n \"it'd\": 'it would',\n \"it'd've\": 'it would have',\n 'kinda': 'kind of',\n \"let's\": 'let us',\n 'luv': 'love',\n \"ma'am\": 'madam',\n \"may've\": 'may have',\n \"mayn't\": 'may not',\n \"might've\": 'might have',\n \"mightn't\": 'might not',\n \"mightn't've\": 'might not have',\n \"must've\": 'must have',\n \"mustn't\": 'must not',\n \"mustn't've\": 'must not have',\n \"needn't\": 'need not',\n \"needn't've\": 'need not have',\n \"ne'er\": 'never',\n \"o'\": 'of',\n \"o'clock\": 'of the clock',\n \"ol'\": 'old',\n \"oughtn't\": 'ought not',\n \"oughtn't've\": 'ought not have',\n \"o'er\": 'over',\n \"shan't\": 'shall not',\n \"sha'n't\": 'shall not',\n \"shalln't\": 'shall not',\n \"shan't've\": 'shall not have',\n \"she's\": 'she is',\n \"she'll\": 'she will',\n \"she'd\": 'she would',\n \"she'd've\": 'she would have',\n \"should've\": 'should have',\n \"shouldn't\": 'should not',\n \"shouldn't've\": 'should not have',\n \"so've\": 'so have',\n \"so's\": 'so is',\n \"somebody's\": 'somebody is',\n \"someone's\": 'someone is',\n \"something's\": 'something is',\n 'sux': 'sucks',\n \"that're\": 'that are',\n \"that's\": 'that is',\n \"that'll\": 'that will',\n \"that'd\": 'that would',\n \"that'd've\": 'that would have',\n 'em': 'them',\n \"there're\": 'there are',\n \"there's\": 'there is',\n \"there'll\": 'there will',\n \"there'd\": 'there would',\n \"there'd've\": 'there would have',\n \"these're\": 'these are',\n \"they're\": 'they are',\n \"they've\": 'they have',\n \"they'll\": 'they will',\n \"they'll've\": 'they will have',\n \"they'd\": 'they would',\n \"they'd've\": 'they would have',\n \"this's\": 'this is',\n \"those're\": 'those are',\n \"to've\": 'to have',\n 'wanna': 'want to',\n \"wasn't\": 'was not',\n \"we're\": 'we are',\n \"we've\": 'we have',\n \"we'll\": 'we will',\n \"we'll've\": 'we will have',\n \"we'd\": 'we would',\n \"we'd've\": 'we would have',\n \"weren't\": 'were not',\n \"what're\": 'what are',\n \"what'd\": 'what did',\n \"what've\": 'what have',\n \"what's\": 'what is',\n \"what'll\": 'what will',\n \"what'll've\": 'what will have',\n \"when've\": 'when have',\n \"when's\": 'when is',\n \"where're\": 'where are',\n \"where'd\": 'where did',\n \"where've\": 'where have',\n \"where's\": 'where is',\n \"which's\": 'which is',\n \"who're\": 'who are',\n \"who've\": 'who have',\n \"who's\": 'who is',\n \"who'll\": 'who will',\n \"who'll've\": 'who will have',\n \"who'd\": 'who would',\n \"who'd've\": 'who would have',\n \"why're\": 'why are',\n \"why'd\": 'why did',\n \"why've\": 'why have',\n \"why's\": 'why is',\n \"will've\": 'will have',\n \"won't\": 'will not',\n \"won't've\": 'will not have',\n \"would've\": 'would have',\n \"wouldn't\": 'would not',\n \"wouldn't've\": 'would not have',\n \"y'all\": 'you all',\n \"y'all're\": 'you all are',\n \"y'all've\": 'you all have',\n \"y'all'd\": 'you all would',\n \"y'all'd've\": 'you all would have',\n \"you're\": 'you are',\n \"you've\": 'you have',\n \"you'll've\": 'you shall have',\n \"you'll\": 'you will',\n \"you'd\": 'you would',\n \"you'd've\": 'you would have',\n 'jan.': 'january',\n 'feb.': 'february',\n 'mar.': 'march',\n 'apr.': 'april',\n 'jun.': 'june',\n 'jul.': 'july',\n 'aug.': 'august',\n 'sep.': 'september',\n 'oct.': 'october',\n 'nov.': 'november',\n 'dec.': 'december',\n 'I’m': 'I am',\n 'I’m’a': 'I am about to',\n 'I’m’o': 'I am going to',\n 'I’ve': 'I have',\n 'I’ll': 'I will',\n 'I’ll’ve': 'I will have',\n 'I’d': 'I would',\n 'I’d’ve': 'I would have',\n 'amn’t': 'am not',\n 'ain’t': 'are not',\n 'aren’t': 'are not',\n '’cause': 'because',\n 'can’t': 'can not',\n 'can’t’ve': 'can not have',\n 'could’ve': 'could have',\n 'couldn’t': 'could not',\n 'couldn’t’ve': 'could not have',\n 'daren’t': 'dare not',\n 'daresn’t': 'dare not',\n 'dasn’t': 'dare not',\n 'doesn’t': 'does not',\n 'e’er': 'ever',\n 'everyone’s': 'everyone is',\n 'gon’t': 'go not',\n 'hadn’t': 'had not',\n 'hadn’t’ve': 'had not have',\n 'hasn’t': 'has not',\n 'haven’t': 'have not',\n 'he’ve': 'he have',\n 'he’s': 'he is',\n 'he’ll': 'he will',\n 'he’ll’ve': 'he will have',\n 'he’d': 'he would',\n 'he’d’ve': 'he would have',\n 'here’s': 'here is',\n 'how’re': 'how are',\n 'how’d': 'how did',\n 'how’d’y': 'how do you',\n 'how’s': 'how is',\n 'how’ll': 'how will',\n 'isn’t': 'is not',\n 'it’s': 'it is',\n '’tis': 'it is',\n '’twas': 'it was',\n 'it’ll': 'it will',\n 'it’ll’ve': 'it will have',\n 'it’d': 'it would',\n 'it’d’ve': 'it would have',\n 'let’s': 'let us',\n 'ma’am': 'madam',\n 'may’ve': 'may have',\n 'mayn’t': 'may not',\n 'might’ve': 'might have',\n 'mightn’t': 'might not',\n 'mightn’t’ve': 'might not have',\n 'must’ve': 'must have',\n 'mustn’t': 'must not',\n 'mustn’t’ve': 'must not have',\n 'needn’t': 'need not',\n 'needn’t’ve': 'need not have',\n 'ne’er': 'never',\n 'o’': 'of',\n 'o’clock': 'of the clock',\n 'ol’': 'old',\n 'oughtn’t': 'ought not',\n 'oughtn’t’ve': 'ought not have',\n 'o’er': 'over',\n 'shan’t': 'shall not',\n 'sha’n’t': 'shall not',\n 'shalln’t': 'shall not',\n 'shan’t’ve': 'shall not have',\n 'she’s': 'she is',\n 'she’ll': 'she will',\n 'she’d': 'she would',\n 'she’d’ve': 'she would have',\n 'should’ve': 'should have',\n 'shouldn’t': 'should not',\n 'shouldn’t’ve': 'should not have',\n 'so’ve': 'so have',\n 'so’s': 'so is',\n 'somebody’s': 'somebody is',\n 'someone’s': 'someone is',\n 'something’s': 'something is',\n 'that’re': 'that are',\n 'that’s': 'that is',\n 'that’ll': 'that will',\n 'that’d': 'that would',\n 'that’d’ve': 'that would have',\n 'there’re': 'there are',\n 'there’s': 'there is',\n 'there’ll': 'there will',\n 'there’d': 'there would',\n 'there’d’ve': 'there would have',\n 'these’re': 'these are',\n 'they’re': 'they are',\n 'they’ve': 'they have',\n 'they’ll': 'they will',\n 'they’ll’ve': 'they will have',\n 'they’d': 'they would',\n 'they’d’ve': 'they would have',\n 'this’s': 'this is',\n 'those’re': 'those are',\n 'to’ve': 'to have',\n 'wasn’t': 'was not',\n 'we’re': 'we are',\n 'we’ve': 'we have',\n 'we’ll': 'we will',\n 'we’ll’ve': 'we will have',\n 'we’d': 'we would',\n 'we’d’ve': 'we would have',\n 'weren’t': 'were not',\n 'what’re': 'what are',\n 'what’d': 'what did',\n 'what’ve': 'what have',\n 'what’s': 'what is',\n 'what’ll': 'what will',\n 'what’ll’ve': 'what will have',\n 'when’ve': 'when have',\n 'when’s': 'when is',\n 'where’re': 'where are',\n 'where’d': 'where did',\n 'where’ve': 'where have',\n 'where’s': 'where is',\n 'which’s': 'which is',\n 'who’re': 'who are',\n 'who’ve': 'who have',\n 'who’s': 'who is',\n 'who’ll': 'who will',\n 'who’ll’ve': 'who will have',\n 'who’d': 'who would',\n 'who’d’ve': 'who would have',\n 'why’re': 'why are',\n 'why’d': 'why did',\n 'why’ve': 'why have',\n 'why’s': 'why is',\n 'will’ve': 'will have',\n 'won’t': 'will not',\n 'won’t’ve': 'will not have',\n 'would’ve': 'would have',\n 'wouldn’t': 'would not',\n 'wouldn’t’ve': 'would not have',\n 'y’all': 'you all',\n 'y’all’re': 'you all are',\n 'y’all’ve': 'you all have',\n 'y’all’d': 'you all would',\n 'y’all’d’ve': 'you all would have',\n 'you’re': 'you are',\n 'you’ve': 'you have',\n 'you’ll’ve': 'you shall have',\n 'you’ll': 'you will',\n 'you’d': 'you would',\n 'you’d’ve': 'you would have'}\n\ndef contraction_fix(word):\n    try:\n        a=contractions[word]\n    except KeyError:\n        a=word\n    return a","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def Preprocess(doc):\n    corpus=[]\n    for text in tqdm(doc):\n        text=\" \".join([contraction_fix(w) for w in text.split()])\n        text=re.sub(r'[^a-z0-9A-Z]',\" \",text)\n        text=re.sub(r'[0-9]{1}',\"#\",text)\n        text=re.sub(r'[0-9]{2}','##',text)\n        text=re.sub(r'[0-9]{3}','###',text)\n        text=re.sub(r'[0-9]{4}','####',text)\n        text=re.sub(r'[0-9]{5,}','#####',text)\n        corpus.append(text)\n    return corpus","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## defining the vocabular function\ndef get_vocab(corpus):\n    vocab={}\n    for text in tqdm(corpus):\n        for word in text.split():\n            try:\n                vocab[word]+=1\n            except KeyError:\n                vocab[word]=1\n    vocab=dict(sorted(vocab.items(),reverse=True ,key=lambda item: item[1]))\n    return vocab","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Expo 1.1 : without preprocessing + without pretrained Embeddings\n\n- here we are not going to do any preproess on text . just using the raw text.\n- Learning Embeddings during Trainig phase."},{"metadata":{"trusted":true},"cell_type":"code","source":"### importing the libraire\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, Conv1D,GRU\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model,load_model\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.utils import plot_model\nfrom keras.models import Sequential\nfrom keras.callbacks import ModelCheckpoint,EarlyStopping\nfrom keras.optimizers import Adam\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### defining the parameter\nmax_feat=30000\nmax_len=40\nfeat_vec=300","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### gettting the index for the each word in vocabulary\ndef get_word_index(vocab):\n    word_index=dict((w,i+1) for i,w in enumerate(vocab.keys()))\n    return word_index\ndef fit_one_hot(word_index,corpus):\n    sent=[]\n    for text in tqdm(corpus):\n        li=[]\n        for word in text.split():\n            try:\n                li.append(word_index[word])\n            except KeyError:\n                li.append(0)\n        sent.append(li)\n    return sent","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vocab=get_vocab(train.question_text)\ntop_feat=dict(list(vocab.items())[:max_feat])\nword_index=get_word_index(top_feat)\nencoded_docs=fit_one_hot(word_index,train.question_text)\npadded_doc=pad_sequences(encoded_docs,maxlen=max_len,padding=\"post\")\nvocab_size=len(word_index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### getting the Validation data\nval_encodes=fit_one_hot(word_index,val.question_text)\nval_padded_doc=pad_sequences(val_encodes,maxlen=max_len,padding=\"post\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"### defining the parameter\nmax_feat=30000\nmax_len=40\nfeat_vec=300","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n### preprocessing the text\ntrain_text=Preprocess(train.question_text)\nval_text=Preprocess(val.question_text)\n\n\nvocab=get_vocab(train_text)\ntop_feat=dict(list(vocab.items())[:max_feat])\nword_index=get_word_index(top_feat)\n\n### encoding the training set\nencoded_docs=fit_one_hot(word_index,train_text)\npadded_doc=pad_sequences(encoded_docs,maxlen=max_len,padding=\"post\")\n\n### encoding the Validation set\nencoded_docs=fit_one_hot(word_index,val_text)\nval_padded_doc=pad_sequences(encoded_docs,maxlen=max_len,padding=\"post\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Building the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"inp = Input(shape=(max_len,))\nx = Embedding(max_feat+1, feat_vec)(inp)\nx = Bidirectional(GRU(128, return_sequences=True))(x)\nx = Conv1D(64,5,activation=\"relu\")(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(128, activation=\"relu\")(x)\nx = Dropout(0.2)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\n\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### defining some callbacks\nopt=Adam(learning_rate=0.002)\nbin_loss=tf.keras.losses.BinaryCrossentropy(\n                                            from_logits=False, \n                                            label_smoothing=0.2,\n                                            name='binary_crossentropy'\n                                        )\n\n## defining the call backs\nearly_stopping=tf.keras.callbacks.EarlyStopping(\n                                                monitor=\"val_loss\",\n                                                patience=3,\n                                                mode=\"min\",\n                                                restore_best_weights=True\n                                              )\n### Now reducing the learning rate when the model is not improvinig \nreduce_lr=tf.keras.callbacks.ReduceLROnPlateau(\n                                                monitor=\"val_loss\",\n                                                factor=0.2,\n                                                patience=2,\n                                                verbose=1,\n                                                mode=\"auto\"\n                                            )\n\nmy_callbacks=[early_stopping,reduce_lr]\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Experiment 2: Pretrained Embeddings + LSTM\n- Here I am using the GOOGle News as pretrained Embeddings"},{"metadata":{},"cell_type":"markdown","source":"### HOw much percentage that covered of Entire Corpus\n- Here we check for the each pretrained Embeddings \n- Do processing that will conver much embeddings\n"},{"metadata":{},"cell_type":"markdown","source":"#### 2.1 LOading and Checking the Pretrained Embeddings (GOOGLE NEWS DATA) "},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n### Loading the Google News Pretrained Embeddings\nfile_name=\"./GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin\"\nmodel_embed=KeyedVectors.load_word2vec_format(file_name,binary=True)\n\n### Building the Vocabulary \ndef vocab_build(corpus):\n    vocab={}\n    for text in tqdm(corpus):\n        for word in text.split():\n            try:\n                vocab[word]+=1\n            except KeyError:\n                vocab[word]=1\n    return vocab\n\n\n### Checking the Vocabulary That how much percentage of Vocabulary was covered \ndef check_voc(vocab,model):\n    embed_words=[]\n    out_vocab={}\n    total_words=0\n    total_text=0\n    for i in tqdm(vocab):\n        try:\n            vec=model[i]\n            embed_words.append(vec)\n            total_words+=vocab[i]\n        except KeyError:\n            out_vocab[i]=vocab[i]\n            total_text+=vocab[i]\n    print(\"The {:.2f}% of vocabularies have Covered of corpus\".format(100*len(embed_words)/len(vocab)))\n    print(\"The {:.2f}% of total text had coverded \".format((100*total_words/(total_words+total_text))))\n    return out_vocab","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### here we are concatenating the both the training and validation dataset\n### building the vocabulary and checking coverage of vocabulary\ntotal_text=pd.concat([train_data.question_text,test_data.question_text])\nvocabulary=vocab_build(total_text)\noov=check_voc(vocabulary,model_embed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Lets Analyze\n- We observed that there will less amount of vocabulary was covered\n- now we wiil analyze the out of vocabulary.\n- to increase the coverage we will do preprocess the data "},{"metadata":{"trusted":true},"cell_type":"code","source":"sort_oov=dict(sorted(oov.items(), key=operator.itemgetter(1),reverse=True))\ndict(list(sort_oov.items())[:50])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observations**\n- some of the puncutation marks are not in pretrained embeedings\n- May be the raw numbers are in the form of other type\n- Question mark is followed by the word so we need to seperate that and that too question mark doesn't have a embedding.\n- Some preprosition too doesn't have the embeddings like (a,to,of and)"},{"metadata":{"trusted":true},"cell_type":"code","source":"del oov,vocabulary,sort_oov\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### lets try to resolve above problems by preprocessing it \npre_text=Preprocess(total_text)\nvocabulary=vocab_build(pre_text)\noov=check_voc(vocabulary,model_embed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sort_oov=dict(sorted(oov.items(), key=operator.itemgetter(1),reverse=True))\ndict(list(sort_oov.items())[:50])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- here i am ending this here .\n\n**Furthur Preprocessing is Required**\n- Spell correction mainly \n- Some word doesn't have a embeddings in a pretrained models"},{"metadata":{"trusted":true},"cell_type":"code","source":"del oov,pre_text,sort_oov,total_text\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vocab_size=len(vocabulary)+1\nmax_len=40\n\nword_index=get_word_index(vocabulary)\n### preprocess the data\ntrain_text=Preprocess(train.question_text)\nval_text=Preprocess(val.question_text)\ntest_text=Preprocess(test_data.question_text)\n\n### encoding the training data\nencodes=fit_one_hot(word_index,train_text)\ntrain_padded=pad_sequences(encodes,maxlen=max_len,padding=\"post\")\n\n### encoding the validation_data\nencodes_=fit_one_hot(word_index,val_text)\nval_padded=pad_sequences(encodes_,maxlen=max_len,padding=\"post\")\n\n### encoding the testing_data\nencodes__=fit_one_hot(word_index,test_text)\ntest_padded=pad_sequences(encodes__,maxlen=max_len,padding=\"post\")\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Constructing the Embedding vector"},{"metadata":{"trusted":true},"cell_type":"code","source":"## step 3: constructinng embedding matrix for the corpus vocabulary using the pretrained embeddings:\n## here each row will have the emedding vector for each unique word\ncount=0\nembedding_mat=np.zeros((vocab_size,300))\nfor word,i in tqdm(word_index.items()):\n    try:\n        vec=model_embed[word]\n        embedding_mat[i]=vec\n    except KeyError:\n        count+=1\n        continue\n\nprint(\"Number of Out of Vocabulary\",count)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inp = Input(shape=(max_len,))\nx = Embedding(vocab_size,300,weights=[embedding_mat],input_length=max_len,trainable=False)(inp)\nx = Bidirectional(LSTM(128, return_sequences=True))(x)\nx = Conv1D(64,3,activation=\"relu\")(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(128, activation=\"relu\")(x)\nx = Dropout(0.2)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\n\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"opt=Adam(learning_rate=0.001)\nbin_loss=tf.keras.losses.BinaryCrossentropy(\n                                            from_logits=False, \n                                            label_smoothing=0,\n                                            name='binary_crossentropy'\n                                        )\n\n## defining the call backs\nearly_stopping=tf.keras.callbacks.EarlyStopping(\n                                                monitor=\"val_loss\",\n                                                patience=3,\n                                                mode=\"min\",\n                                                restore_best_weights=True\n                                              )\n### Now reducing the learning rate when the model is not improvinig \nreduce_lr=tf.keras.callbacks.ReduceLROnPlateau(\n                                                monitor=\"val_loss\",\n                                                factor=0.2,\n                                                patience=2,\n                                                verbose=1,\n                                                mode=\"auto\"\n                                            )\n\nmy_callbacks=[early_stopping,reduce_lr]\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss=bin_loss, optimizer=opt, metrics=['accuracy'])\nhistory=model.fit(train_padded, train.target, batch_size=512, epochs=30, validation_data=(val_padded, val.target),callbacks=my_callbacks)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pre=model.predict(val_padded)\nfor thresh in np.arange(0.1,0.5,0.01):\n    print(\"threshold {0:2.2f} f1 score:{1:2.3f}\".format(thresh,metrics.f1_score(val.target,(y_pre>thresh).astype(int))))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### INference the Test data\n\nthreshold=0.36\ny_test_pre=model.predict(test_padded)\ny_test_pre=(y_test_pre>thresh).astype(int)\n\n### Creating the submission File\nsubmit=pd.DataFrame()\nsubmit[\"qid\"]=test_data.qid\nsubmit[\"prediction\"]=y_test_pre\nsubmit.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('hamza.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}