{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Objective:**\n \n This kernel is made as a walkthrough for basic NLP analysis and modelling.\n \n**Table Of Contents:**\n* Importing packages and modules\n* Text data preprocessing\n* Basic EDA with WordCloud and Plotly\n* Machine Learning:\n    * Logistic Regression\n    * Support Vector Machine\n    * Multinomial Naive Bayes\n    * Decision Tree Classifier\n    * AdaBoost Classifier   \n    * Random Forest Classifier\n    * XGBoost Classifier\n    * XGBClassifier Hyperparameter tuning\n* Deep Learning:\n    * Text data preprocessing\n    * Bidirectional LSTM with own Embeddings\n    * Bidirectional LSTM with GloVe"},{"metadata":{},"cell_type":"markdown","source":"**If you like the kernel, please upvote it**. Your upvotes will motivate me to code more. Happy Kaggling !"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport plotly.plotly as py\nfrom plotly.offline import init_notebook_mode,iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nfrom nltk.tokenize import word_tokenize\nfrom nltk.corpus import stopwords\nfrom nltk.stem.snowball import SnowballStemmer\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom wordcloud import WordCloud\nimport string\nfrom sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\nfrom sklearn.model_selection import train_test_split,GridSearchCV\nfrom sklearn.metrics import confusion_matrix,f1_score,roc_curve,make_scorer\nfrom sklearn.svm import SVC\nfrom sklearn.naive_bayes import GaussianNB,MultinomialNB\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier,AdaBoostClassifier\nfrom xgboost import XGBClassifier\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn.manifold import TSNE\nimport os\nimport scikitplot as skplt\nimport seaborn as sns\nimport time\nprint(os.listdir(\"../input\"))\nstopwords=set(stopwords.words('english'))\nstemmer=SnowballStemmer('english')\nseed=5","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"data=pd.read_csv('../input/train.csv')\ntest=pd.read_csv('../input/test.csv')\nsub=pd.read_csv('../input/sample_submission.csv')\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_count=data.target.value_counts()\ntrace1=go.Bar(x=target_count.index,\n             y=target_count.values,\n             name='Target Counts',\n             marker=dict(color='rgba(0,255,255,0.5)',\n                         line=dict(color='rgb(0,0,0)',width=0.5)),\n             text=['Sincere Questions','Insincere Questions'])\nlayout=go.Layout(title='Bar plot of target counts',\n                xaxis=dict(title='Target'),\n                yaxis=dict(title='Number of questions'))\nplt_data=[trace1]\nfig=dict(data=plt_data,layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data is highly imbalanced. For the baseline model, even if predict always 0, we would be correct (1225312/1306122)*100 = 93.81 percent of time.\n\nLets clean the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"target1=data[data['target']==1]\ntarget0=data[data['target']==0]\nsampled_size=target1.shape[0]\nsampled_target0=target0.sample(sampled_size,random_state=seed)\nnew_data=pd.concat([target1,sampled_target0],axis=0)\n#Shuffling the data\nnew_data=new_data.sample(frac=1,random_state=seed).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def filter_text(text):\n    tokenized_words=word_tokenize(text)\n    filtered_words=[word.lower() for word in tokenized_words if ((word.lower() not in string.punctuation) &\n                                                                 (word.lower() not in stopwords))]\n    stemmed_words=[stemmer.stem(word) for word in filtered_words]\n    return ' '.join(filtered_words)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Preprocessing for traditional NLP: This includes removing stop words and punctuation, followed by stemming."},{"metadata":{"trusted":true},"cell_type":"code","source":"trad_data=new_data.copy()\ntrad_data['question_text']=trad_data['question_text'].apply(lambda x: filter_text(x))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_wordcloud(text,max_font_size=40,max_words=100):\n    plt.figure(figsize=(10,5))\n    wordcloud=WordCloud(max_font_size=max_font_size,max_words=max_words,random_state=seed)\n    plot=wordcloud.generate(text)\n    plt.imshow(plot)\n    plt.axis('off')\n    plt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target0=trad_data[trad_data['target']==0].reset_index(drop=True)\ntarget1=trad_data[trad_data['target']==1].reset_index(drop=True)\n\ntarget0_text=''\ntarget1_text=''\nfor i in range(target0.shape[0]):\n    target0_text+=target0.question_text[i]\nfor i in range(target1.shape[0]):\n    target1_text+=target1.question_text[i]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_wordcloud(target0_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_wordcloud(target1_text,max_words=200)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_top_ngrams(text,ngrams=(1,1),top=10,max_features=10000,color='rgba(0,255,255,0.5)'):\n    cv=CountVectorizer(ngram_range=ngrams,max_features=max_features)\n    trans_text=cv.fit_transform(text)\n    col_sum=trans_text.sum(axis=0)\n    word_index=[(word,col_sum[0,idx]) for word,idx in cv.vocabulary_.items()]\n    sorted_word_index=sorted(word_index,key=lambda x:x[1],reverse=True)\n    top_words_index=sorted_word_index[:top]\n    top_words=[element[0] for element in top_words_index]\n    counts=[element[1] for element in top_words_index]\n    trace1=go.Bar(x=top_words,\n                 y=counts,\n                 marker=dict(color=color,\n                             line=dict(color='rgb(0,0,0)',width=0.5)))\n    layout=go.Layout(title='Top ngrams',\n                    xaxis=dict(title='Ngrams'),\n                    yaxis=dict(title='Counts of words'))\n    plot_data=[trace1]\n    fig=dict(data=plot_data,layout=layout)\n    iplot(fig)\n    \n    \n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_top_ngrams(target1.question_text,ngrams=(1,1),top=30,color='rgba(128,0,0,0.5)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that most of the insincere questions start with 'Why'. "},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_top_ngrams(target1.question_text,ngrams=(2,2),top=30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_top_ngrams(target1.question_text,ngrams=(3,3),top=30,color='rgba(128,128,128,0.5)')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_top_ngrams(target0.question_text,ngrams=(1,1),top=30,color='rgba(128,0,0,0.5)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that most of the sincere questions start with 'What'."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_top_ngrams(target0.question_text,ngrams=(2,2),top=30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_top_ngrams(target0.question_text,ngrams=(3,3),top=30,color='rgba(128,128,128,0.5)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mini_df=trad_data.sample(25000,random_state=seed)\nX=mini_df['question_text']\nY=mini_df['target']\n\ntrain_X,val_X,train_y,val_y=train_test_split(X,Y,test_size=0.2,random_state=seed)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cv=CountVectorizer(ngram_range=(1,3),analyzer='word')\ntrain_X_cv=cv.fit_transform(train_X.values)\nval_X_cv=cv.transform(val_X.values)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tsvd=TruncatedSVD(n_components=50,random_state=seed)\ntrain_X_svd=tsvd.fit_transform(train_X_cv)\nval_X_svd=tsvd.transform(val_X_cv)\ntsne=TSNE(n_components=2,random_state=seed)\ntrain_X_tsne=tsne.fit_transform(train_X_svd)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df=pd.DataFrame()\ndf['tsne1']=pd.Series(train_X_tsne[:,0])\ndf['tsne2']=pd.Series(train_X_tsne[:,1])\ndf['target']=train_y\nsns.scatterplot(df['tsne1'],df['tsne2'],hue='target',data=df)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Evidently, we can say from TSNE plot that the two classes are not trivially differentiaable."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(model,train_X,train_y,val_X):\n    model.fit(train_X,train_y)\n    pred_probs=model.predict_proba(val_X)\n    pred_train=model.predict(train_X)\n    pred_val=model.predict(val_X)\n    score_train=f1_score(train_y,pred_train)\n    score_val=f1_score(val_y,pred_val)\n    return pred_probs,pred_train,pred_val,score_train,score_val\n\ndef get_confusion_matrix(val_y,pred,title):\n    cm=confusion_matrix(val_y,pred)\n    plt.figure(figsize=(10,5))\n    sns.heatmap(cm,annot=True)\n    plt.title(title)\n    plt.ylabel('True labels')\n    plt.xlabel('Predicted labels')\n    plt.show()\n    \ndef get_roc_curve(val_y,pred_probs,title):\n    plt.title(title)\n    skplt.metrics.plot_roc(val_y,pred_probs)\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"models=[LogisticRegression(random_state=seed),MultinomialNB(),DecisionTreeClassifier(random_state=seed),\n        AdaBoostClassifier(DecisionTreeClassifier(max_depth=3),n_estimators=100,learning_rate=0.1,random_state=seed),\n        RandomForestClassifier(n_estimators=100,max_depth=3,random_state=seed),\n        XGBClassifier(random_state=seed)]\nmodel_names=['LR','Multinomial NB','DTC','ABC','RFC','XGBC']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_probs={}\npred_train={}\npred_val={}\nscore_train={}\nscore_val={}\n\nfor i in range(len(models)):\n    pred_probs[model_names[i]],pred_train[model_names[i]],pred_val[model_names[i]],\\\n    score_train[model_names[i]],score_val[model_names[i]]=get_model(models[i],train_X_cv,train_y,val_X_cv)\n    \n\n\nscl=StandardScaler()                                                     \ntrain_X_scl_cv=scl.fit_transform(train_X_svd)\nval_X_scl_cv=scl.transform(val_X_svd)\npred_probs['SVC'],pred_train['SVC'],pred_val['SVC'],\\\nscore_train['SVC'],score_val['SVC']=get_model(SVC(probability=True,random_state=seed),\n                                                      train_X_scl_cv,train_y,val_X_scl_cv)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trace1=go.Bar(x=list(score_train.keys()),\n              y=list(score_train.values()),\n             name='Training Score with CV',\n             marker=dict(color='rgba(0,255,0,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\ntrace2=go.Bar(x=list(score_val.keys()),\n              y=list(score_val.values()),\n             name='Validation Score with CV',\n             marker=dict(color='rgba(255,255,0,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\n\nlayout=go.Layout(barmode='group',\n                title='Scores of Different Models')\nplot_data=[trace1,trace2]\nfig=dict(data=plot_data,layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Clearly, LogisticRegression, Multinomial NB and DecisionTree Classifier overfit the data. AdaBoost, RandomForest and XGB Classifiers do not overfit but their scores are low compared to the former 3 models. Regularization and Hyperparameter tuning can surely help to not overfit in case of initial 3 models and can help the latter 3 models to improve their scores,respectively."},{"metadata":{"trusted":true},"cell_type":"code","source":"for model,probs in pred_probs.items():\n    get_roc_curve(val_y,probs,model)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I don't know what is the problem with roc curve for LR. It is printed at the bottom with title ROC Curves. Multinomial NB and Decision Tree Classifier seem to perform best. "},{"metadata":{"trusted":true},"cell_type":"code","source":"for model,pred in pred_val.items():\n    get_confusion_matrix(val_y,pred,model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the confusion matrix of Multinomial NB and Decision Tree Classifier, we see that MNB has higher misclassification of 'Sincere Questions' and lower misclassification of 'Insincere Questions' as compared to DTC.\nDTC has large misclassification of 'Insincere Questions'. This is not at all desired. Practically speaking, it is better to misclassify true 'Sincere Question' than to misclassify true 'Insincere Question'. In case of classification of true 'Insincere Question', MNB performs best while RFC performs worst. "},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"tfv=TfidfVectorizer(ngram_range=(1,3),analyzer='word',min_df=3)\ntrain_X_tfv=tfv.fit_transform(train_X.values)\nval_X_tfv=tfv.transform(val_X.values)\ntsvd_tfv=TruncatedSVD(n_components=50,random_state=seed)\ntrain_X_svd_tfv=tsvd_tfv.fit_transform(train_X_tfv)\nval_X_svd_tfv=tsvd_tfv.transform(val_X_tfv)\ntsne_tfv=TSNE(n_components=2,random_state=seed)\ntrain_X_tsne_tfv=tsne_tfv.fit_transform(train_X_svd_tfv)\n\n\ndf=pd.DataFrame()\ndf['tsne1']=pd.Series(train_X_tsne_tfv[:,0])\ndf['tsne2']=pd.Series(train_X_tsne_tfv[:,1])\ndf['target']=train_y\nsns.scatterplot(df['tsne1'],df['tsne2'],hue='target',data=df)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_probs_tfv={}\npred_train_tfv={}\npred_val_tfv={}\nscore_train_tfv={}\nscore_val_tfv={}\n\nfor i in range(len(models)):\n    pred_probs_tfv[model_names[i]],pred_train_tfv[model_names[i]],pred_val_tfv[model_names[i]],\\\n    score_train_tfv[model_names[i]],score_val_tfv[model_names[i]]=get_model(models[i],train_X_tfv,train_y,val_X_tfv)\n\nscl=StandardScaler()\ntrain_X_scl_tfv=scl.fit_transform(train_X_svd_tfv)\nval_X_scl_tfv=scl.transform(val_X_svd_tfv)\npred_probs_tfv['SVC'],pred_train_tfv['SVC'],pred_val_tfv['SVC'],\\\nscore_train_tfv['SVC'],score_val_tfv['SVC']=get_model(SVC(probability=True,random_state=seed),\n                                                      train_X_scl_tfv,train_y,val_X_scl_tfv)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrace1=go.Bar(x=list(score_train_tfv.keys()),\n              y=list(score_train_tfv.values()),\n             name='Training Score with TFV',\n             marker=dict(color='rgba(0,255,0,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\ntrace2=go.Bar(x=list(score_val_tfv.keys()),\n              y=list(score_val_tfv.values()),\n             name='Validation Score with TFV',\n             marker=dict(color='rgba(255,255,0,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\n\nlayout=go.Layout(barmode='group',\n                title='Scores of Different Models')\nplot_data=[trace1,trace2]\nfig=dict(data=plot_data,layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the use of TfidfVectorizer, overfitting is not observed in LR and Multinomial NB models unlike during usage of CountVectorizer. Overfitting of DTC still remains an issue. Other models aren't overfitting. It looks like Multinomial NB is the best model. But we haven't yet tuned hyperparameters of tree based models. In the following graphs, we compare the training scores of models with CountVectorizer vs with TfidfVectorizer. Same is done for validation scores as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"trace1=go.Bar(x=list(score_train.keys()),\n              y=list(score_train.values()),\n             name='Training Score with CV',\n             marker=dict(color='rgba(0,0,255,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\ntrace2=go.Bar(x=list(score_train_tfv.keys()),\n              y=list(score_train_tfv.values()),\n             name='Training Score with TFV',\n             marker=dict(color='rgba(255,0,0,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\n\nlayout=go.Layout(barmode='group',\n                title='Training Scores of Different Models')\nplot_data=[trace1,trace2]\nfig=dict(data=plot_data,layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When switched from CountVectorizer to TfidfVectorizer, the training scores of LR and Multinomial NB have decreased, while that of other models it has increased. "},{"metadata":{"trusted":true},"cell_type":"code","source":"trace1=go.Bar(x=list(score_val.keys()),\n              y=list(score_val.values()),\n             name='Validation Score with CV',\n             marker=dict(color='rgba(0,0,255,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\ntrace2=go.Bar(x=list(score_val_tfv.keys()),\n              y=list(score_val_tfv.values()),\n             name='Validation Score with TFV',\n             marker=dict(color='rgba(255,0,0,0.5)',\n                        line=dict(color='rgb(0,0,0)',width=1.5)))\n\nlayout=go.Layout(barmode='group',\n                title='Validation Scores of Different Models')\nplot_data=[trace1,trace2]\nfig=dict(data=plot_data,layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When switched from CountVectorizer to TfidfVectorizer, there is either a slight decrease or increase in validation scores. Increase in the validation score of RFC is huge.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"for model,probs in pred_probs_tfv.items():\n    get_roc_curve(val_y,probs,model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From ROC Curves, it looks like Multinomial NB and DTC are best models. Let us check their confusion matrices."},{"metadata":{"trusted":true},"cell_type":"code","source":"for model,pred in pred_val_tfv.items():\n    get_confusion_matrix(val_y,pred,model)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Classification of true 'Insincere Questions' is highest in Multinomial NB and least in RFC. Again, Multinomial NB with TfidfVectorizer seems to be the best model."},{"metadata":{},"cell_type":"markdown","source":"Let us try to tune the hyperparameters of XGBClassifier. Following codes were computationally expensive, so I just commented them."},{"metadata":{"trusted":true},"cell_type":"code","source":"#start=time.time()\n#params={'n_estimators':[100,300,500],\n#       'learning_rate':[0.01,0.05,0.1],\n#       'subsample':[0.8]}\n\n#model=XGBClassifier(random_state=seed)\n#score=make_scorer(f1_score)\n#grid=GridSearchCV(model,params,cv=3,scoring=score)\n#grid.fit(train_X_tfv,train_y)\n\n#end=time.time()\n#print('Total time taken: ' + str(end-start))\n\n#Total time taken: 371.8829896450043","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(grid.best_params_)\n#print(grid.best_score_)\n#xgb1=grid.best_estimator_\n#xgb1.fit(train_X_tfv,train_y)\n#pred1=xgb1.predict(val_X_tfv)\n#score1=f1_score(val_y,pred1)\n#print(score1)\n\n#{'learning_rate': 0.1, 'n_estimators': 500, 'subsample': 0.8}\n#0.8230178094880998\n#0.8253164556962025","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Little bit of tuning has increased our score by 4.5%. The best part is, it is not even overfitting."},{"metadata":{"trusted":true},"cell_type":"code","source":"#start=time.time()\n#params={'n_estimators':[500,800,1000],\n#       'learning_rate':[0.1,0.15,0.2],\n#       'subsample':[0.8],\n#       'max_depth':[3,5,7],\n#       'gamma':[0,10]}\n\n#model=XGBClassifier(random_state=seed)\n#score=make_scorer(f1_score)\n#grid2=GridSearchCV(model,params,cv=3,scoring=score)\n#grid2.fit(train_X_tfv,train_y)\n\n#end=time.time()\n#print('Total time taken: ' + str(end-start))\n\n#0.8397122528906257\n#{'gamma': 0, 'learning_rate': 0.1, 'max_depth': 7, 'n_estimators': 800, 'subsample': 0.8}\n#Total time taken: 8327.329408407211","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"#print(grid2.best_score_)\n#print(grid2.best_params_)\n#xgb2=grid2.best_estimator_\n#xgb2.fit(train_X_tfv,train_y)\n#pred2=xgb2.predict(val_X_tfv)\n#score2=f1_score(val_y,pred2)\n#print(score2)\n\n#0.8397122528906257\n#{'gamma': 0, 'learning_rate': 0.1, 'max_depth': 7, 'n_estimators': 800, 'subsample': 0.8}\n#0.8515138946495231","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thus, by further parameter tuning we have increased XGBClassifier score by 7%. You can try out more out of it. "},{"metadata":{},"cell_type":"markdown","source":"**NLP with Deep Learning**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense,Embedding,SpatialDropout1D,Dropout,CuDNNLSTM,Bidirectional\nfrom keras.utils import to_categorical\nfrom keras import backend\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.callbacks import Callback,EarlyStopping,ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the pre-processing for deeplearning, I am removing only the punctuations. Stop words might be helpful. You can even try the model by keeping punctuations. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def filter_for_nn(text):\n    tokenized_word=word_tokenize(text)\n    filtered_words=[word.lower() for word in tokenized_word if word not in string.punctuation]\n    return ' '.join(filtered_words)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_data['question_text']=new_data['question_text'].apply(lambda x: filter_for_nn(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nX=new_data.question_text\nY=new_data.target\n\ntrain_X,val_X,train_y,val_y=train_test_split(X,Y,test_size=0.01,random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Converting text to sequences"},{"metadata":{"trusted":true},"cell_type":"code","source":"token=Tokenizer()\ntoken.fit_on_texts(train_X.values)\ntrain_seq=token.texts_to_sequences(train_X.values)\nval_seq=token.texts_to_sequences(val_X.values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Padding the sequences"},{"metadata":{"trusted":true},"cell_type":"code","source":"maxlen=100\ntrain_pad=pad_sequences(train_seq,maxlen=maxlen)\nval_pad=pad_sequences(val_seq,maxlen=maxlen)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y=to_categorical(train_y.values)\nval_y=to_categorical(val_y.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vocabulary=token.word_index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Getting our own Embeddings."},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_len=10\nmodel=Sequential()\nmodel.add(Embedding(len(vocabulary)+1,embedding_len,input_length=maxlen))\nmodel.add(SpatialDropout1D(0.3))\nmodel.add(Bidirectional(CuDNNLSTM(100)))\nmodel.add(Dense(512,activation='relu'))\nmodel.add(Dropout(0.8,seed=seed))\nmodel.add(Dense(512,activation='relu'))\nmodel.add(Dropout(0.8,seed=seed))\nmodel.add(Dense(2,activation='softmax'))\n\nmodel.compile(loss='binary_crossentropy',optimizer='adam')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using Callbacks. Due to computation cost, I am running the model only for 5 epochs. So, callbacks are rendered ineffective. For better performance, run for about 100 epochs"},{"metadata":{"trusted":true},"cell_type":"code","source":"est=EarlyStopping(monitor='val_loss',patience=10,restore_best_weights=True)\nrlr=ReduceLROnPlateau(monitor='val_loss',patience=10,min_lr=0.0001,factor=0.1)\ncall_back=[est,rlr]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result1=model.fit(train_pad,train_y,epochs=5,batch_size=64,validation_data=[val_pad,val_y],callbacks=call_back)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred1_proba=model.predict(val_pad)\npred1=np.argmax(pred1_proba,axis=1)\nscore1=f1_score(np.argmax(val_y,axis=1),pred1)\nprint(score1)\n\ntrain_pred1_proba=model.predict(train_pad)\ntrain_pred1=np.argmax(train_pred1_proba,axis=1)\ntrain_score1=f1_score(np.argmax(train_y,axis=1),train_pred1)\nprint(train_score1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Simple Bidirectional LSTM with own embeddings with only 5 epochs has worked better than XGBClassifier! But this model used significantly more data than XGBClassifier. But we can see that this model is overfitting. You can train it for around 100 epochs. We have set patience=10 in callbacks and so they are definitely not effective in 5 epochs. During 100 epochs, they will help in not overfitting. You can reduce the learning rate of Adam optimizer to reduce overfitting aswell."},{"metadata":{},"cell_type":"markdown","source":"Taking Embeddings from GloVe"},{"metadata":{"trusted":true},"cell_type":"code","source":"Embedding_file='../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr):return word,np.asarray(arr,dtype='float32')\nembeddings_index=dict(get_coefs(*o.split(\" \")) for o in open(Embedding_file))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_stack=np.stack(embeddings_index.values())\nemb_mean,emb_std=embedding_stack.mean(),embedding_stack.std()\nembedding_matrix=np.random.normal(emb_mean,emb_std,(len(vocabulary)+1,embedding_stack.shape[1]))\nfor word,i in vocabulary.items():\n    embedding_vector=embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i]=embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel2=Sequential()\nmodel2.add(Embedding(embedding_matrix.shape[0],embedding_matrix.shape[1],weights=[embedding_matrix],input_length=maxlen))\nmodel2.add(SpatialDropout1D(0.3))\nmodel2.add(Bidirectional(CuDNNLSTM(100)))\nmodel2.add(Dense(512,activation='relu'))\nmodel2.add(Dropout(0.8,seed=seed))\nmodel2.add(Dense(512,activation='relu'))\nmodel2.add(Dropout(0.8,seed=seed))\nmodel2.add(Dense(2,activation='softmax'))\n\nmodel2.compile(loss='binary_crossentropy',optimizer='adam')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result2=model2.fit(train_pad,train_y,epochs=5,batch_size=64,validation_data=[val_pad,val_y],callbacks=call_back)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred2_proba=model2.predict(val_pad)\npred2=np.argmax(pred2_proba,axis=1)\nscore2=f1_score(np.argmax(val_y,axis=1),pred2)\nprint(score2)\n\ntrain_pred2_proba=model2.predict(train_pad)\ntrain_pred2=np.argmax(train_pred2_proba,axis=1)\ntrain_score2=f1_score(np.argmax(train_y,axis=1),train_pred2)\nprint(train_score2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using GloVe Embeddings on the same model has increased the score significantly. Here again, the model is overfitting but you can train for higher epochs with callbacks and reduced learning rate and playing around with dropout values to ensure that the model doesn't overfit. Thus, we see that neural networks are extremely powerful. **If you liked the kernel, please upvote it**\n\nThe important thing you can try out is to optimize the hyperparameters of neural network to get incredible scores without overfitting. You can do it for traditional models as well. Happy Kaggling !!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}