{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Quora insincere question classification\n\n- In part 1:\n            EDA\n            Preprocessing the data\n            used TFIDF for feature extraction\n            Experiment-1 : only derived Features + Logistic Regression and Trees\n            Experiment-2 : TFIDF vectorizer + Logistic Regression\n            Experiment-3 : TFIDF vectorizer + Feature Engineering + Logistic Regression.\n        \n    Here is the link https://www.kaggle.com/sai24kumar/qiq-classification-logistic-regression\n            \n            \n- Part 2:\n           Experiment 1: Pretrained Word Embeddings + Logistic Regression\n           Experiment 2: Pretrained Word Embeddings + Multi_layer perceptron \n           Experiment 3: Pretrained Word Embeddings + Convolutional networks"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"##importing the modules\nimport pandas as pd\nimport numpy as np\nfrom gensim.models import KeyedVectors\nfrom nltk.tokenize import word_tokenize\nimport re\nimport gc\nimport seaborn as sbn\n## for the progress bar we are importing the tqdm\nfrom tqdm import tqdm\n\nfrom keras.preprocessing.text import Tokenizer\n### splitting the trainset and test set\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Loading the dataset\ntrain_data=pd.read_csv(\"../input/quora-insincere-questions-classification/train.csv\")\ntest_data=pd.read_csv(\"../input/quora-insincere-questions-classification/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## displaying the dataset\ndisplay(train_data.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Printing the missing values in the dataset\ndisplay(train_data.isnull().sum())\ndisplay(test_data.isnull().sum())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### To preprocess we are anlyzing with the google news word2vec embedding pretrained models."},{"metadata":{},"cell_type":"markdown","source":"### Preprocessing\n\n- When we are using the preptrained embeddings then no need to normalize the words like converting from uppercase to lower case.\n- No need to remove stopwords.\n- Here we need to preprocess according to the embedding we use."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n!unzip ../input/quora-insincere-questions-classification/embeddings.zip","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Experiment 1: Pretrained Word Embedding + tranditional Machinelearning models\n\n-  Here I am using the Google's word2vec pretrained model\n-  This prerained model will return the embedding vector for the each word but we want the vector for the whole sentence.\n- So we will average the Embedding vectors of words in a documnet"},{"metadata":{},"cell_type":"markdown","source":"### Extracting the Features using the Pretrained word Embeddings"},{"metadata":{},"cell_type":"markdown","source":"- Here we will load the google word2vec model with the help of gensim\n- By this word to vec we will represent the words in to vector and we will average the word to vec for the whole sentence."},{"metadata":{"trusted":true},"cell_type":"code","source":"from gensim.models import KeyedVectors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfile_name=\"./GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin\"\nmodel=KeyedVectors.load_word2vec_format(file_name,binary=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### checking the Vocabulary that the number of words are presenved by the w2v.\ndef vocab_build(corpus):\n    vocab={}\n    for text in tqdm(corpus):\n        for word in text.split():\n            try:\n                vocab[word]+=1\n            except KeyError:\n                vocab[word]=1\n    return vocab\n\ndef check_voc(vocab,model):\n    embed_words=[]\n    out_vocab={}\n    total_words=0\n    total_text=0\n    for i in tqdm(vocab):\n        try:\n            vec=model[i]\n            embed_words.append(vec)\n            total_words+=vocab[i]\n        except KeyError:\n            out_vocab[i]=vocab[i]\n            total_text+=vocab[i]\n    print(\"The {:.2f}% of vocabularies have Covered of corpus\".format(100*len(embed_words)/len(vocab)))\n    print(\"The {:.2f}% of total text had coverded \".format((100*total_words/(total_words+total_text))))\n    return out_vocab","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vocabulary=vocab_build(train_data.question_text)\noov=check_voc(vocabulary,model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Here it covered only few vocabulary. so only for 34% of vocabulary got the embeddings.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import operator\nsort_oov=dict(sorted(oov.items(), key=operator.itemgetter(1),reverse=True))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(sort_oov.keys())[:40])\ndel sort_oov,oov,vocabulary\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In this dataset have the non english characters which are have in high frequency.\n- There are special characters were associated with the numbers and characters."},{"metadata":{"trusted":true},"cell_type":"code","source":"contractions={\"I'm\": 'I am',\n \"I'm'a\": 'I am about to',\n \"I'm'o\": 'I am going to',\n \"I've\": 'I have',\n \"I'll\": 'I will',\n \"I'll've\": 'I will have',\n \"I'd\": 'I would',\n \"I'd've\": 'I would have',\n 'Whatcha': 'What are you',\n \"amn't\": 'am not',\n \"ain't\": 'are not',\n \"aren't\": 'are not',\n \"'cause\": 'because',\n \"can't\": 'can not',\n \"can't've\": 'can not have',\n \"could've\": 'could have',\n \"couldn't\": 'could not',\n \"couldn't've\": 'could not have',\n \"daren't\": 'dare not',\n \"daresn't\": 'dare not',\n \"dasn't\": 'dare not',\n \"didn't\": 'did not',\n 'didn’t': 'did not',\n \"don't\": 'do not',\n 'don’t': 'do not',\n \"doesn't\": 'does not',\n \"e'er\": 'ever',\n \"everyone's\": 'everyone is',\n 'finna': 'fixing to',\n 'gimme': 'give me',\n \"gon't\": 'go not',\n 'gonna': 'going to',\n 'gotta': 'got to',\n \"hadn't\": 'had not',\n \"hadn't've\": 'had not have',\n \"hasn't\": 'has not',\n \"haven't\": 'have not',\n \"he've\": 'he have',\n \"he's\": 'he is',\n \"he'll\": 'he will',\n \"he'll've\": 'he will have',\n \"he'd\": 'he would',\n \"he'd've\": 'he would have',\n \"here's\": 'here is',\n \"how're\": 'how are',\n \"how'd\": 'how did',\n \"how'd'y\": 'how do you',\n \"how's\": 'how is',\n \"how'll\": 'how will',\n \"isn't\": 'is not',\n \"it's\": 'it is',\n \"'tis\": 'it is',\n \"'twas\": 'it was',\n \"it'll\": 'it will',\n \"it'll've\": 'it will have',\n \"it'd\": 'it would',\n \"it'd've\": 'it would have',\n 'kinda': 'kind of',\n \"let's\": 'let us',\n 'luv': 'love',\n \"ma'am\": 'madam',\n \"may've\": 'may have',\n \"mayn't\": 'may not',\n \"might've\": 'might have',\n \"mightn't\": 'might not',\n \"mightn't've\": 'might not have',\n \"must've\": 'must have',\n \"mustn't\": 'must not',\n \"mustn't've\": 'must not have',\n \"needn't\": 'need not',\n \"needn't've\": 'need not have',\n \"ne'er\": 'never',\n \"o'\": 'of',\n \"o'clock\": 'of the clock',\n \"ol'\": 'old',\n \"oughtn't\": 'ought not',\n \"oughtn't've\": 'ought not have',\n \"o'er\": 'over',\n \"shan't\": 'shall not',\n \"sha'n't\": 'shall not',\n \"shalln't\": 'shall not',\n \"shan't've\": 'shall not have',\n \"she's\": 'she is',\n \"she'll\": 'she will',\n \"she'd\": 'she would',\n \"she'd've\": 'she would have',\n \"should've\": 'should have',\n \"shouldn't\": 'should not',\n \"shouldn't've\": 'should not have',\n \"so've\": 'so have',\n \"so's\": 'so is',\n \"somebody's\": 'somebody is',\n \"someone's\": 'someone is',\n \"something's\": 'something is',\n 'sux': 'sucks',\n \"that're\": 'that are',\n \"that's\": 'that is',\n \"that'll\": 'that will',\n \"that'd\": 'that would',\n \"that'd've\": 'that would have',\n 'em': 'them',\n \"there're\": 'there are',\n \"there's\": 'there is',\n \"there'll\": 'there will',\n \"there'd\": 'there would',\n \"there'd've\": 'there would have',\n \"these're\": 'these are',\n \"they're\": 'they are',\n \"they've\": 'they have',\n \"they'll\": 'they will',\n \"they'll've\": 'they will have',\n \"they'd\": 'they would',\n \"they'd've\": 'they would have',\n \"this's\": 'this is',\n \"those're\": 'those are',\n \"to've\": 'to have',\n 'wanna': 'want to',\n \"wasn't\": 'was not',\n \"we're\": 'we are',\n \"we've\": 'we have',\n \"we'll\": 'we will',\n \"we'll've\": 'we will have',\n \"we'd\": 'we would',\n \"we'd've\": 'we would have',\n \"weren't\": 'were not',\n \"what're\": 'what are',\n \"what'd\": 'what did',\n \"what've\": 'what have',\n \"what's\": 'what is',\n \"what'll\": 'what will',\n \"what'll've\": 'what will have',\n \"when've\": 'when have',\n \"when's\": 'when is',\n \"where're\": 'where are',\n \"where'd\": 'where did',\n \"where've\": 'where have',\n \"where's\": 'where is',\n \"which's\": 'which is',\n \"who're\": 'who are',\n \"who've\": 'who have',\n \"who's\": 'who is',\n \"who'll\": 'who will',\n \"who'll've\": 'who will have',\n \"who'd\": 'who would',\n \"who'd've\": 'who would have',\n \"why're\": 'why are',\n \"why'd\": 'why did',\n \"why've\": 'why have',\n \"why's\": 'why is',\n \"will've\": 'will have',\n \"won't\": 'will not',\n \"won't've\": 'will not have',\n \"would've\": 'would have',\n \"wouldn't\": 'would not',\n \"wouldn't've\": 'would not have',\n \"y'all\": 'you all',\n \"y'all're\": 'you all are',\n \"y'all've\": 'you all have',\n \"y'all'd\": 'you all would',\n \"y'all'd've\": 'you all would have',\n \"you're\": 'you are',\n \"you've\": 'you have',\n \"you'll've\": 'you shall have',\n \"you'll\": 'you will',\n \"you'd\": 'you would',\n \"you'd've\": 'you would have',\n 'jan.': 'january',\n 'feb.': 'february',\n 'mar.': 'march',\n 'apr.': 'april',\n 'jun.': 'june',\n 'jul.': 'july',\n 'aug.': 'august',\n 'sep.': 'september',\n 'oct.': 'october',\n 'nov.': 'november',\n 'dec.': 'december',\n 'I’m': 'I am',\n 'I’m’a': 'I am about to',\n 'I’m’o': 'I am going to',\n 'I’ve': 'I have',\n 'I’ll': 'I will',\n 'I’ll’ve': 'I will have',\n 'I’d': 'I would',\n 'I’d’ve': 'I would have',\n 'amn’t': 'am not',\n 'ain’t': 'are not',\n 'aren’t': 'are not',\n '’cause': 'because',\n 'can’t': 'can not',\n 'can’t’ve': 'can not have',\n 'could’ve': 'could have',\n 'couldn’t': 'could not',\n 'couldn’t’ve': 'could not have',\n 'daren’t': 'dare not',\n 'daresn’t': 'dare not',\n 'dasn’t': 'dare not',\n 'doesn’t': 'does not',\n 'e’er': 'ever',\n 'everyone’s': 'everyone is',\n 'gon’t': 'go not',\n 'hadn’t': 'had not',\n 'hadn’t’ve': 'had not have',\n 'hasn’t': 'has not',\n 'haven’t': 'have not',\n 'he’ve': 'he have',\n 'he’s': 'he is',\n 'he’ll': 'he will',\n 'he’ll’ve': 'he will have',\n 'he’d': 'he would',\n 'he’d’ve': 'he would have',\n 'here’s': 'here is',\n 'how’re': 'how are',\n 'how’d': 'how did',\n 'how’d’y': 'how do you',\n 'how’s': 'how is',\n 'how’ll': 'how will',\n 'isn’t': 'is not',\n 'it’s': 'it is',\n '’tis': 'it is',\n '’twas': 'it was',\n 'it’ll': 'it will',\n 'it’ll’ve': 'it will have',\n 'it’d': 'it would',\n 'it’d’ve': 'it would have',\n 'let’s': 'let us',\n 'ma’am': 'madam',\n 'may’ve': 'may have',\n 'mayn’t': 'may not',\n 'might’ve': 'might have',\n 'mightn’t': 'might not',\n 'mightn’t’ve': 'might not have',\n 'must’ve': 'must have',\n 'mustn’t': 'must not',\n 'mustn’t’ve': 'must not have',\n 'needn’t': 'need not',\n 'needn’t’ve': 'need not have',\n 'ne’er': 'never',\n 'o’': 'of',\n 'o’clock': 'of the clock',\n 'ol’': 'old',\n 'oughtn’t': 'ought not',\n 'oughtn’t’ve': 'ought not have',\n 'o’er': 'over',\n 'shan’t': 'shall not',\n 'sha’n’t': 'shall not',\n 'shalln’t': 'shall not',\n 'shan’t’ve': 'shall not have',\n 'she’s': 'she is',\n 'she’ll': 'she will',\n 'she’d': 'she would',\n 'she’d’ve': 'she would have',\n 'should’ve': 'should have',\n 'shouldn’t': 'should not',\n 'shouldn’t’ve': 'should not have',\n 'so’ve': 'so have',\n 'so’s': 'so is',\n 'somebody’s': 'somebody is',\n 'someone’s': 'someone is',\n 'something’s': 'something is',\n 'that’re': 'that are',\n 'that’s': 'that is',\n 'that’ll': 'that will',\n 'that’d': 'that would',\n 'that’d’ve': 'that would have',\n 'there’re': 'there are',\n 'there’s': 'there is',\n 'there’ll': 'there will',\n 'there’d': 'there would',\n 'there’d’ve': 'there would have',\n 'these’re': 'these are',\n 'they’re': 'they are',\n 'they’ve': 'they have',\n 'they’ll': 'they will',\n 'they’ll’ve': 'they will have',\n 'they’d': 'they would',\n 'they’d’ve': 'they would have',\n 'this’s': 'this is',\n 'those’re': 'those are',\n 'to’ve': 'to have',\n 'wasn’t': 'was not',\n 'we’re': 'we are',\n 'we’ve': 'we have',\n 'we’ll': 'we will',\n 'we’ll’ve': 'we will have',\n 'we’d': 'we would',\n 'we’d’ve': 'we would have',\n 'weren’t': 'were not',\n 'what’re': 'what are',\n 'what’d': 'what did',\n 'what’ve': 'what have',\n 'what’s': 'what is',\n 'what’ll': 'what will',\n 'what’ll’ve': 'what will have',\n 'when’ve': 'when have',\n 'when’s': 'when is',\n 'where’re': 'where are',\n 'where’d': 'where did',\n 'where’ve': 'where have',\n 'where’s': 'where is',\n 'which’s': 'which is',\n 'who’re': 'who are',\n 'who’ve': 'who have',\n 'who’s': 'who is',\n 'who’ll': 'who will',\n 'who’ll’ve': 'who will have',\n 'who’d': 'who would',\n 'who’d’ve': 'who would have',\n 'why’re': 'why are',\n 'why’d': 'why did',\n 'why’ve': 'why have',\n 'why’s': 'why is',\n 'will’ve': 'will have',\n 'won’t': 'will not',\n 'won’t’ve': 'will not have',\n 'would’ve': 'would have',\n 'wouldn’t': 'would not',\n 'wouldn’t’ve': 'would not have',\n 'y’all': 'you all',\n 'y’all’re': 'you all are',\n 'y’all’ve': 'you all have',\n 'y’all’d': 'you all would',\n 'y’all’d’ve': 'you all would have',\n 'you’re': 'you are',\n 'you’ve': 'you have',\n 'you’ll’ve': 'you shall have',\n 'you’ll': 'you will',\n 'you’d': 'you would',\n 'you’d’ve': 'you would have'}\n\ndef contraction_fix(word):\n    try:\n        a=contractions[word]\n    except KeyError:\n        a=word\n    return a","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\n### Preprocess the train dataset \ndef Preprocess(x):\n    remove_words=[\"to\",\"a\",\"and\",\"of\"]\n    \n    x=\" \".join([contraction_fix(w) for w in x.split() if w not in remove_words])        \n    x=re.sub(r\"[^a-zA-Z0-9]\",\" \",x)\n    x=re.sub(r'[0-9]{5,}','##### ',x)\n    x=re.sub(r'[0-9]{4}',\"#### \",x)\n    x=re.sub(r'[0-9]{3}','### ',x)\n    x=re.sub(r'[0-9]{2}',\"## \",x)\n    x=re.sub(r'[0-9]{1}',\"# \",x)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"text_data=train_data.question_text.apply(Preprocess)\nvocab=vocab_build(text_data)\noov=check_voc(vocab,model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sort_oov=dict(sorted(oov.items(), key=operator.itemgetter(1),reverse=True))\nprint(list(sort_oov.keys())[:20])\n### analysis over we are deleting the unnecessary variables and dealocate those variables \ndel text_data,vocab,oov,sort_oov\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Furthur Preprocess do spell checking and correct those spellings"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Preprocessing will be doing according to the Google word2vec\ntrain_data.question_text=train_data.question_text.apply(Preprocess)\ntrain_data[\"num_words\"]=train_data.question_text.apply(lambda x:len(x.split()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Getting Sentence vector from a word2vec Pretrained Embedding models\n\n**Some of the Ideas:**\n\nIdea 1:\n        - we will average the all the word embeddings for a each doc .\n        - So that we can get a fixed 300 dimensional vector representation for a document\n        \nIdea 2:\n        - concatenate the embedded words in a each document.\n        - But the problem is length of the vector varies with the length of the document.\n        "},{"metadata":{},"cell_type":"markdown","source":"### Using Average word2vec"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n## defing the function for the getting embedding vectores for the doc i.e Doc2Vec\ndef Doc2Vec(corpus):\n    vector_sent=[]\n    for i in tqdm(corpus):\n        words=i.split()\n        count=0\n        avg=np.zeros((300,))\n        for word in words:\n            try:\n                avg=avg+model[word]\n                count+=1\n            except KeyError:\n                continue\n        if(count!=0):\n            avg=avg/count\n        else:\n            avg=np.zeros((300,))\n        vector_sent.append(avg)\n        del avg\n        \n        \n    return np.array(vector_sent)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train,x_val,y_train,y_val=train_test_split(train_data.question_text,train_data.target.values,stratify=train_data.target.values)\n## printing the shapes of the datasets\nprint(\"The shape of the train dataset :\",x_train.shape,y_train.shape)\nprint(\"The shape of the Test dataset :\",x_val.shape,y_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train=Doc2Vec(x_train)\nx_val=Doc2Vec(x_val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_log=LogisticRegression(C=15,max_iter=300)\nmodel_log.fit(x_train,y_train)\ntrain_pre=model_log.predict(x_train)\nval_pre=model_log.predict(x_val)\n\nprint(\"-------------Evalution Scores------------------\")\nprint(\"F1-score of Trainset :\",metrics.f1_score(y_train,train_pre))\nprint(\"F1-score of Valset :\",metrics.f1_score(y_val,val_pre))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In previous notebook i used a bag of words representation + Logistic regression i got 55% f1-score but here is getting very low even though it has semantic representation.\n- This is because here we lost lot of information . In bag of words there can be words which are relevent and irrelevent so logistic regression will deffirentiate beween those words . but here both the relevent and non relevent words are to averaged ."},{"metadata":{"trusted":true},"cell_type":"code","source":"del model_log,x_train,y_train,y_val,x_val\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Paragraph embeddings\n\n- There are some sentence techniques \n    They are :\n            1.Average word2vec (discussed in the above)\n            2.Doc2vec also called as paragraph embedding\n            3.SentenceBert\n            4.InferSent\n            5.Universal Sentence Encoder.\n Ref: https://www.analyticsvidhya.com/blog/2020/08/top-4-sentence-embedding-techniques-using-python/ \n \n In this experiment i am using only doc2vec method"},{"metadata":{},"cell_type":"markdown","source":"### Doc2vec\n- step 1: preprocess and tokenize the text data.\n- step 2: prepare the tagged document \n- step 3: Train with the PV-DM or PV-DBOW\n- step 4: get the Paragraph embeddings by infering the sentences\n\nNOte : I am not goind to use this . beacause it is taking lot of time to train and embede the documents."},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\n## importing the neceesary libraries \nfrom gensim.models.doc2vec import Doc2Vec,TaggedDocument\nimport re \ndef Preprocess_doc(corpus):\n    doc=[]\n    for text in tqdm(corpus):\n        text=text.lower()\n        text=text.strip()\n        text=re.sub(r\"[^a-zA-Z]\",\" \",text)\n        word=[i for i in text.split() if(len(i)>1 and i.isalpha() and (i not in stop_words))]\n        text=[ps.stem(i) for i in word]\n        doc.append(text)\n    return doc\ncorpus=Preprocess_doc(train_data.question_text)\n#converting the tokinezed documents to a tagged Documents\ntagged_data =[]\nfor i,doc in tqdm(enumerate(corpus)):\n    if(len(doc)==0):\n        doc=[\"unknown\"]\n    tagged_data.append(TaggedDocument(doc,[i]))\n    %%time\n### Train the model\nmodel = Doc2Vec(tagged_data, vector_size=30, window=2, min_count=1, workers=-1, epochs = 100)\n# Save trained doc2vec model\nmodel.save(\"test_doc2vec.model\")\n\n'''\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Experiment :2 Pretrained Embeddings + MLP"},{"metadata":{"trusted":true},"cell_type":"code","source":"### importing the libraire\nfrom keras.layers import Conv1D,Flatten,Dense,BatchNormalization,Embedding,MaxPooling1D,Input\nfrom keras.layers import SpatialDropout1D,Dropout\nfrom keras.layers.merge import concatenate\nfrom keras.models import Model,Sequential\nfrom keras.activations import relu\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.utils import plot_model\nfrom keras.callbacks import ModelCheckpoint,EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## we need to prepare the input to the Embedding layer that it was accepted\nmax_lenth=30\nvec_size=300\n# get a word to index dictonary and and encoded sentence\n## step 1: defining a fit_on_text functions it will return the vecabular to index dictonary and \n## encoded sentence\n\ndef get_word_index(corpus):\n    vocab=vocab_build(corpus)\n    word_index=dict((w,i+1) for i,w in enumerate(list(vocab.keys())))\n    return word_index\n\n    \ndef fit_on_text(corpus,word_index):\n    sent=[]\n    for text in tqdm(corpus):\n        li=[word_index[w] for w in text.split()]\n        sent.append(li)\n    return sent\n\n# word_index and encoded sentence \nword_index=get_word_index(train_data.question_text)\nencoded_docs=fit_on_text(train_data.question_text,word_index)\n\n## vocabulary size\nvocab_size=len(word_index)+1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The Number of Vocabularies are in the dataset is :\",vocab_size)\n\n## step 2: padding the sequence to a maximum length\n## the pad_to_seqeucbe method wiill expect the input of encoded and max leght, and where to pad this is with the zeros\n## i.e post or pre \npadded_doc=pad_sequences(encoded_docs,maxlen=max_lenth,padding=\"post\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## step 3: constructinng embedding matrix for the corpus vocabulary using the pretrained embeddings:\n## here each row will have the emedding vector for each unique word\ncount=0\nembedding_mat=np.zeros((vocab_size,300))\nfor word,i in tqdm(word_index.items()):\n    try:\n        vec=model[word]\n        embedding_mat[i]=vec\n    except KeyError:\n        count+=1\n        continue\n\nprint(\"Number of Out of Vocabulary\",count)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### splitting dataset in to train_set and test_set\nfrom sklearn.model_selection import train_test_split\nx_train,x_val,y_train,y_val=train_test_split(np.array(padded_doc),train_data.target,test_size=0.2,stratify=train_data.target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"### defing the model\nclassfy=Sequential()\nclassfy.add(Embedding(vocab_size,300,weights=[embedding_mat],input_length=max_lenth,trainable=False))\nclassfy.add(Flatten())\nclassfy.add(Dense(256,activation=\"relu\"))\nclassfy.add(Dense(128,activation=\"relu\"))\nclassfy.add(Dense(1,activation=\"sigmoid\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classfy.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_callbacks=[EarlyStopping(patience=2,monitor=\"val_loss\")]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classfy.compile(loss=\"binary_crossentropy\",optimizer=\"adam\",metrics=[\"accuracy\"])\nclassfy.fit(x_train,y_train,epochs=5,validation_data=(x_val,y_val),callbacks=my_callbacks,batch_size=128)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evalution of Validation data \n\n- here the Metric we using is f1_score and roc_auc_score "},{"metadata":{"trusted":true},"cell_type":"code","source":"val_pre=classfy.predict(x_val)\npredicts=[]\nthreshold=0.3\nfor i in val_pre:\n    if(i<threshold):\n        predicts.append(0)\n    else:\n        predicts.append(1)\n\nprint(\"AUC and ROC score is :\",metrics.roc_auc_score(y_val,val_pre))\nprint(\"F1_score :\",metrics.f1_score(y_val,predicts))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating the Submission file from the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"clean_test_data=test_data.question_text.apply(Preprocess)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit_on_test_text(corpus,word_index):\n    sent=[]\n    for text in tqdm(corpus):\n       \n        li=[]\n        for w in text.split():\n            try:\n                li.append(word_index[w])\n            except KeyError:\n                li.append(0)\n        sent.append(li)\n    return sent","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def Submission_file(model,name):\n    test_encode=fit_on_test_text(clean_test_data,word_index)\n    test_pad=pad_sequences(test_encode,maxlen=max_lenth,padding=\"post\")\n    test_predict=model.predict(test_pad)\n    predicts=[]\n    threshold=0.3\n    for i in test_predict:\n        if(i<threshold):\n            predicts.append(0)\n        else:\n            predicts.append(1)\n    submit=pd.DataFrame()\n    submit[\"qid\"]=test_data.qid\n    submit[\"prediction\"]=predicts\n    submit.to_csv(name,index=False)\n    return submit","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_mlp=Submission_file(classfy,\"submit_mlp.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Experiment 3: Pretrained Embedding + Convolutional"},{"metadata":{},"cell_type":"markdown","source":"### Model 1:\n- here i take the filter size of 3 (like tri-gram)\n- here i used the 2 convolutions "},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel_cnn=Sequential()\nmodel_cnn.add(Embedding(vocab_size,300,weights=[embedding_mat],input_length=max_lenth,trainable=False))\nmodel_cnn.add(Conv1D(64,3,activation=\"relu\",input_shape=(max_lenth,vec_size)))\nmodel_cnn.add(MaxPooling1D())\nmodel_cnn.add(Conv1D(32,3,activation=\"relu\"))\nmodel_cnn.add(Flatten())\nmodel_cnn.add(Dense(256,activation=\"relu\"))\nmodel_cnn.add(Dense(128,activation=\"relu\"))\nmodel_cnn.add(Dense(1,activation=\"sigmoid\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn.compile(loss=\"binary_crossentropy\",optimizer=\"adam\",metrics=[\"accuracy\"])\nmodel_cnn.fit(x_train,y_train,epochs=5,validation_data=(x_val,y_val),callbacks=my_callbacks,batch_size=128)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_pre=model_cnn.predict(x_val)\npredicts=[]\nthreshold=0.3\nfor i in val_pre:\n    if(i<threshold):\n        predicts.append(0)\n    else:\n        predicts.append(1)\n\nprint(\"AUC and ROC score is :\",metrics.roc_auc_score(y_val,val_pre))\nprint(\"F1_score :\",metrics.f1_score(y_val,predicts))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## creating the submission File for convolutional networks\ndf_mlp=Submission_file(model_cnn,\"submit_model_cnn.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### model 2:\n- Here i use n_gram filters 3,5,7 size of filters and i will combine those "},{"metadata":{"trusted":true},"cell_type":"code","source":"input_=Input((max_lenth,))\nemb_vec=Embedding(vocab_size,300,weights=[embedding_mat],input_length=max_lenth,trainable=False)(input_)\n## 3-gram convolution\nout_3g=Conv1D(64,3,activation=\"relu\",input_shape=(max_lenth,vec_size))(emb_vec)\nout_3g=MaxPooling1D()(out_3g)\nout_3g=SpatialDropout1D(0.5)(out_3g)\nout_3g=Conv1D(64,3,activation=\"relu\")(out_3g)\nout_3g=MaxPooling1D()(out_3g)\nout_3g=SpatialDropout1D(0.5)(out_3g)\nout_3g=Flatten()(out_3g)\n## 5-gram convolutin\nout_5g=Conv1D(64,5,activation=\"relu\",input_shape=(max_lenth,vec_size))(emb_vec)\nout_5g=MaxPooling1D()(out_5g)\nout_5g=SpatialDropout1D(0.5)(out_5g)\nout_5g=Conv1D(64,5,activation=\"relu\")(out_5g)\nout_5g=MaxPooling1D()(out_5g)\nout_5g=SpatialDropout1D(0.5)(out_5g)\nout_5g=Flatten()(out_5g)\n## 7-gram convolution\nout_7g=Conv1D(64,7,activation=\"relu\",input_shape=(max_lenth,vec_size))(emb_vec)\nout_7g=MaxPooling1D()(out_7g)\nout_7g=SpatialDropout1D(0.5)(out_7g)\nout_7g=Conv1D(64,7,activation=\"relu\")(out_7g)\nout_7g=MaxPooling1D()(out_7g)\nout_7g=SpatialDropout1D(0.5)(out_7g)\nout_7g=Flatten()(out_7g)\n\n#concatenate all those outputs\ncom_grams=concatenate([out_3g,out_5g,out_7g],axis=-1)\noutput=Dense(256,activation=\"relu\")(com_grams)\n\noutput=Dense(1,activation=\"sigmoid\")(output)\n\nmodel_cnn_2=Model(inputs=input_,outputs=output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn_2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## plotting the model\nplot_model(model_cnn_2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_cnn_2.compile(loss=\"binary_crossentropy\",optimizer=\"adam\",metrics=[\"accuracy\"])\nmodel_cnn_2.fit(x_train,y_train,epochs=5,validation_data=(x_val,y_val),callbacks=my_callbacks,batch_size=256)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_pre=model_cnn_2.predict(x_val)\npredicts=[]\nthreshold=0.3\nfor i in val_pre:\n    if(i<threshold):\n        predicts.append(0)\n    else:\n        predicts.append(1)\n\nprint(\"AUC and ROC score is :\",metrics.roc_auc_score(y_val,val_pre))\nprint(\"F1_score :\",metrics.f1_score(y_val,predicts))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df=Submission_file(model_cnn_2,\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}