{"cells":[{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"raw","source":""},{"metadata":{},"cell_type":"markdown","source":"# NLP Course-Quora Binary Classification\n\n\nThis is a binary classification (traditional)/semantic classification approach by applying different techniques -EDA(feature engineering,gram analysis),embeddings,deep neural networks,transformer architectures and evaluation.\n\n\n## Important Points\n\n1. EDA- Data Analysis, Statistical Analysis, Tensor Decomposition Strategies, non semantic inference, gram analysis\n2. Embeddings- Static Embeddings - word2vec, glove, wikitext,keras Embeddings\n3. Preliminary Deep Models- using Keras, TF, important aspects on LSTM,Convolution Networks.\n4. Extensive Embedding Architectures- Transformers - Variants of Bert, GPT\n5. Evaluation- Checking accuracy and predictions\n\n## Salient resources\n\nResources for each part will be shared along with very important kernels and other notebooks which are relevant pertaining to this course.\n\n"},{"metadata":{},"cell_type":"markdown","source":"# EDA- Analysis and Benchmarking\n\n\nThe theme of the kernel is to abstract and segregate potentially insincere Quora questions. This module will be used as a starting point for initial analysis of semantics,grams. For an extended implication and requirements of the kernel,please refer to : [Quora Insincere Question Classification](https://www.kaggle.com/c/quora-insincere-questions-classification/overview). \n\nIn this module we will be looking into the following aspects:\n\n1. The training and test data \n2. Simplified Statistics and Analysis-vectorization, gram analysis\n3. Classic Reduction Techniques- PCA,SVD, TSNE\n4. Creating a benchmark with classic statistical models"},{"metadata":{},"cell_type":"markdown","source":"## Evaluation of the Dataset\n\nThis submodule will assist us in understanding the data as well as have an inital idea of the same.\nSome resources for this include:\n\n1. [PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html),[TSNE](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html),[SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html)\n2. [tfidfvectorization](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html),[CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)\n3. [WordCloud](https://pypi.org/project/wordcloud/)\n4. [Gram-Analysis](https://www.nltk.org/)\n5. [Eda-notebook](https://www.kaggle.com/abhilash1910/tweet-analysis-eda-cleaning-tsne-glove-tf)\n\n\nPCA works by eigenvector decomposition strategy as shown in the sample:\n\n\n\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#see the input\n\n!ls ../input\n!ls ../input/quora-insincere-questions-classification","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#import libraries useful for the entire module\nimport pandas as pd \nimport numpy as np\nimport xgboost as xgb\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as mpatches\nimport matplotlib\nfrom tqdm import tqdm\nfrom sklearn.svm import SVC\nfrom keras.models import Sequential\nfrom keras.layers.recurrent import LSTM, GRU\nfrom keras.layers.core import Dense, Activation, Dropout\nfrom keras.layers.embeddings import Embedding\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.utils import np_utils\nfrom sklearn import preprocessing, decomposition, model_selection, metrics, pipeline\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.naive_bayes import MultinomialNB\nfrom keras.layers import GlobalMaxPooling1D, Conv1D, MaxPooling1D, Flatten, Bidirectional, SpatialDropout1D\nfrom keras.preprocessing import sequence, text\nfrom keras.callbacks import EarlyStopping\nfrom nltk import word_tokenize\nfrom nltk.corpus import stopwords\nstop_words = stopwords.words('english')\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as mpatches\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.decomposition import PCA, TruncatedSVD,SparsePCA\nfrom sklearn.metrics import classification_report,confusion_matrix\nfrom nltk.tokenize import word_tokenize\nfrom collections import defaultdict\nfrom collections import Counter\nimport seaborn as sns\nfrom wordcloud import WordCloud,STOPWORDS\nimport nltk\nfrom nltk.corpus import stopwords\nimport string\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import preprocessing,metrics,manifold\nfrom sklearn.manifold import TSNE\nfrom sklearn.model_selection import train_test_split,cross_val_score,GridSearchCV,cross_val_predict\nfrom imblearn.over_sampling import ADASYN,SMOTE\nfrom imblearn.under_sampling import NearMiss\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.svm import SVC\nimport collections\nimport keras as k\nimport matplotlib.patches as mpatches\nfrom sklearn.metrics import accuracy_score\n%matplotlib inline\nfrom sklearn.preprocessing import RobustScaler\nimport xgboost\nfrom imblearn.metrics import classification_report_imbalanced\nfrom sklearn.metrics import classification_report,roc_auc_score,roc_curve,r2_score,recall_score,confusion_matrix,precision_recall_curve\nfrom collections import Counter\nfrom sklearn.model_selection import StratifiedKFold,KFold,StratifiedShuffleSplit\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\nprint(train_df.head())\nprint(test_df.head())\nprint(\"===========\")\nprint(\"Training Shape\".format(),train_df.shape)\nprint(\"Testing Shape\".format(),test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The type of columns in the dataset\")\nprint(\"Columns\".format(),train_df.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#Insincere question test\ntrain_ext=train_df[train_df['target']==1]['question_text']\nprint(train_ext)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysis 1- Word statistics\n\nThis analysis would be done on both the training and testing datasets.\nA fundamental part of any NLP pipeline before applying transformations"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Training data statistics\n\n#Estimate the value counts\ncount_types=train_df['target'].value_counts()\nprint(\"Extracting counts\".format(),count_types)\n#Count targets with value 1\n\ncount_ones=train_df[train_df['target']==1].shape[0]\nprint(count_ones)\ncount_zeros=train_df[train_df['target']==0].shape[0]\nprint(count_zeros)\n\n#Matplot to plot the amount of questions from either types\ndef plot_counts(count_ones,count_zeros):\n plt.rcParams['figure.figsize']=(6,6)\n plt.bar(0,count_ones,width=0.6,label='InSincere Questions',color='Red')\n plt.legend()\n plt.bar(2,count_zeros,width=0.6,label='Sincere Questions',color='Green')\n plt.legend()\n plt.ylabel('Count of Questions (in M)')\n plt.xlabel('Types of Questions')\n plt.show()\n\n \n#Seaborn Dist plot for analysing the length of a question sentence \ndef plot_wordcount(count_ones_length,count_zeros_length):\n fig,(ax1,ax2)=plt.subplots(1,2,figsize=(15,5))\n sns.distplot(count_zeros_length,ax=ax1,color='Blue')\n ax1.set_title('Sincere Question Length')\n sns.distplot(count_ones_length,ax=ax2,color='Red')\n ax2.set_title('Insincere Question Length')\n fig.suptitle('Average Length of Words in Question')\n plt.show() \n\n#Generic Plotter\ndef plot_count(count_punct_ones,count_punct_zeros,title_1,title_2,subtitle):\n fig,(ax1,ax2)=plt.subplots(1,2,figsize=(15,5))\n sns.distplot(count_punct_zeros,ax=ax1,color='Blue')\n ax1.set_title(title_1)\n sns.distplot(count_punct_ones,ax=ax2,color='Red')\n ax2.set_title(title_2)\n fig.suptitle(subtitle)\n plt.show() \n\n\n#preliminary word cloud statistics\ndef display_cloud(data):\n stopwords=set(STOPWORDS)\n wordcloud=WordCloud(stopwords=stopwords,max_font_size=120,max_words=300,width=800,height=400,background_color='white',min_font_size=5).generate(str(data))\n plt.figure(figsize=(24,8))\n plt.imshow(wordcloud)\n plt.axis(\"off\")\n plt.title(\"Word Cloud of the questions\")\n plt.show()\n\n\n \n \n#Extract the length of the insincere-sincere questions\ndef word_length(x):\n return len(x)\n\n\ncount_ones_length=train_df[train_df['target']==1]['question_text'].str.split().apply(lambda z:word_length(z))\nprint(\"Length of each insincere questions\".format(),count_ones_length[:5])\ncount_zeros_length=train_df[train_df['target']==0]['question_text'].str.split().apply(lambda z: word_length(z))\nprint(\"Length of each sincere questions\".format(),count_zeros_length[:5])\n\n#Plots\nplot_counts(count_ones,count_zeros)\nplot_wordcount(count_ones_length,count_zeros_length)\ndisplay_cloud(train_df['question_text'])\ndisplay_cloud(train_df[train_df['target']==1]['question_text'])\ndisplay_cloud(train_df[train_df['target']==0]['question_text'])\n#Other analysis\nstops=set(stopwords.words('english'))\ncount_punct_ones=train_df[train_df['target']==1]['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]))\ncount_punct_zeros=train_df[train_df['target']==0]['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]))\ntitle_1='Sincere Question Punctuations'\ntitle_2='Insincere Question Punctuations'\nsubtitle='Punctuations in Questions'\nplot_count(count_punct_ones,count_punct_zeros,title_1,title_2,subtitle)\n\ncount_avg_ones=train_df[train_df['target']==1]['question_text'].apply(lambda z : np.mean([len(z) for w in str(z).split()]))\ncount_avg_zeros=train_df[train_df['target']==0]['question_text'].apply(lambda z: np.mean([len(z) for w in str(z).split()]))\ntitle_1='Insincere Question Average Length'\ntitle_2='Sincere Question Average Length'\nsubtitle='Average Length'\nplot_count(count_avg_ones,count_avg_zeros,title_1,title_2,subtitle)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Test dataset Word Statistics\n\ndef plot_testcount(count_test,title1,subtitle):\n fig,(ax1)=plt.subplots(1,figsize=(15,5))\n sns.distplot(count_test,ax=ax1,color='Orange')\n ax1.set_title(title_1)\n fig.suptitle(subtitle)\n plt.show() \n\ncount_test=test_df['question_text'].str.split().apply(lambda z:word_length(z))\nprint(\"Length of each test questions\".format(),count_test[:5])\n\n#Plots\nplot_testcount(count_test,'Questions in test','Total Questions')\ndisplay_cloud(test_df['question_text'])\n#Other analysis\nstops=set(stopwords.words('english'))\ncount_puncttest=test_df['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]))\ntitle_1='Test Question Punctuations'\nsubtitle='Punctuations in Questions'\nplot_testcount(count_puncttest,title_1,subtitle)\n\ncount_avg_test=test_df['question_text'].apply(lambda z : np.mean([len(z) for w in str(z).split()]))\ntitle_1='Test Question Average Length'\nsubtitle='Average Length'\nplot_testcount(count_avg_test,title_1,subtitle)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysis 2- Gram Statistics\n\nThis will be useful for analysing the gram (bi,tri) in the words in the questions. It is an important step before cleaning.\nResources:\n\n1. [Seaborn](https://seaborn.pydata.org/index.html)\n2. [Gram Analysis](https://albertauyeung.github.io/2018/06/03/generating-ngrams.html)\n\nA pictorial representation of gram analysis is provided here:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Gram analysis on Training set\nstopword=set(stopwords.words('english'))\ndef gram_analysis(data,gram):\n tokens=[t for t in data.lower().split(\" \") if t!=\"\" if t not in stopword]\n ngrams=zip(*[tokens[i:] for i in range(gram)])\n final_tokens=[\" \".join(z) for z in ngrams]\n return final_tokens\n\n#analyse most common Sentences\ndef mostcommon_words(data):\n counter=Counter(data)\n commonwords=counter.most_common()\n x_coord,y_coord=[],[]\n for words,occ in commonwords[:20]:\n if words not in stopword:\n x_coord.append(occ)\n y_coord.append(words)\n \n sns.barplot(x=x_coord,y=y_coord,saturation=1,orient=\"h\")\n\n#Create frequency grams for analysis\n \ndef create_dict(data,grams):\n freq_dict=defaultdict(int)\n for sentence in data:\n for tokens in gram_analysis(sentence,grams):\n freq_dict[tokens]+=1\n return freq_dict\n\ndef horizontal_bar_chart(df, color):\n trace = go.Bar(\n y=df[\"n_gram_words\"].values[::-1],\n x=df[\"n_gram_frequency\"].values[::-1],\n showlegend=False,\n orientation = 'h',\n marker=dict(\n color=color,\n ),\n )\n return trace\n\n\n\ndef create_new_df(freq_dict,):\n freq_df=pd.DataFrame(sorted(freq_dict.items(),key=lambda z:z[1])[::-1])\n freq_df.columns=['n_gram_words','n_gram_frequency']\n #print(freq_df.head())\n #plt.barh(freq_df['n_gram_words'][:20],freq_df['n_gram_frequency'][:20],linewidth=0.3)\n #plt.show()\n trace=horizontal_bar_chart(freq_df[:20],'blue')\n return trace\n \ndef plot_grams(trace_zero,trace_one):\n fig = tools.make_subplots(rows=1, cols=2, vertical_spacing=0.04,\n subplot_titles=[\"Frequent words of sincere questions\", \n \"Frequent words of insincere questions\"])\n fig.append_trace(trace_zero, 1, 1)\n fig.append_trace(trace_ones, 1, 2)\n fig['layout'].update(height=1200, width=900, paper_bgcolor='rgb(233,233,233)', title=\"Word Count Plots\")\n py.iplot(fig, filename='word-plots')\n \n \ntrain_df_zero=train_df[train_df['target']==0]['question_text']\ntrain_df_ones=train_df[train_df['target']==1]['question_text']\n\n#mostcommon_words(train_df_zero)\nprint(\"Bi-gram analysis\")\nfreq_train_df_zero=create_dict(train_df_zero[:200],2)\n#print(freq_train_df_zero)\ntrace_zero=create_new_df(freq_train_df_zero)\nfreq_train_df_ones=create_dict(train_df_ones[:200],2)\n#print(freq_train_df_zero)\ntrace_ones=create_new_df(freq_train_df_ones)\nplot_grams(trace_zero,trace_ones)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Tri-gram analysis\")\nfreq_train_df_zero=create_dict(train_df_zero[:200],3)\n#print(freq_train_df_zero)\ntrace_zero=create_new_df(freq_train_df_zero)\nfreq_train_df_ones=create_dict(train_df_ones[:200],3)\n#print(freq_train_df_zero)\ntrace_ones=create_new_df(freq_train_df_ones)\nplot_grams(trace_zero,trace_ones)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Sentence Analysis\nmostcommon_words(train_df_zero)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Sentence Analysis\nmostcommon_words(train_df_ones)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Similar gram analysis using nltk after regexing\n\nimport re\nfrom nltk.util import ngrams\ndef gram_analyse(s):\n \n s = s.lower()\n s = re.sub(r'[^a-zA-Z0-9\\s]', ' ', s)\n tokens = [token for token in s.split(\" \") if token != \"\"]\n output = list(ngrams(tokens, 5))\n return output\n\nfor j in range(0,10):\n\n out=gram_analyse(train_df_ones.iloc[j])\n print(\"Output grams\",out)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference From Initial Word Analysis\n\nFrom the analysis, we see Sincere questions are in a larger quantity as compared to the Insincere questions. This may lead us to think the dataset is imbalanced. The gram statistics shows that there is a good correlation in trigram analysis of Insincere questions as compared to the Sincere ones. The number of words in insincere questions is larger as compared to the sincere ones, and the same can be deduced from the average sentence length of them as well.\n\nFor balancing certain strategies are important such as synthetic oversampling\n\n1. [SMOTE](https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html)\n2. [ADASYN](https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.ADASYN.html)\n\n\n\nThere are other undersampling and oversampling methods as well:\n\n1. [UnderSampling](https://imbalanced-learn.readthedocs.io/en/stable/under_sampling.html)\n2. [OverSampling](https://imbalanced-learn.readthedocs.io/en/stable/over_sampling.html)\n\nThis cannot be directly applied to textual data as it can be applied to vectorized(float) data.\n\n\nSMOTE for imbalanced data can be previewed as follows (with orange samples being the synthetically generated class):\n\n\n\nSynthetic Data points generated by density functions in Adasyn is represented below:\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Theoretical Diagram of SMOTE/ADASYN operation\n\nSmote and Adasyn employ distance metrics of KNN/Kmeans for generating synthetic points. The difference lies in how the points are generated- Smote employs weighted distance metrics whereas Adasyn performs probabilistic density distributions for generating new points.\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Some cleaning\n\nThis is mainly used for removing certain punctuations, stopwords from the corpus. Though the data is quite clean, we will clean this further with regex. \n\n1. [Regex](https://docs.python.org/3/library/re.html)\n\nWe can also stem the corpus but that will not be helpful in the long run, though lemmatizing may be helpful to a certain extent.\n\n1. [Stem](https://www.nltk.org/howto/stem.html)\n2. [Lemmatize](http://www.nltk.org/api/nltk.stem.html?highlight=lemmatizer)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Regex cleaning\n\ndef remove_punctuations(data):\n punct_tag=re.compile(r'[^\\w\\s]')\n data=punct_tag.sub(r'',data)\n return data\ndef remove_html(data):\n html_tag=re.compile(r'<.*?>')\n data=html_tag.sub(r'',data)\n return data\n\ndef remove_url(data):\n url_clean= re.compile(r\"https://\\S+|www\\.\\S+\")\n data=url_clean.sub(r'',data)\n return data\ndef clean_data(data):\n emoji_clean= re.compile(\"[\"\n u\"\\U0001F600-\\U0001F64F\" # emoticons\n u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n u\"\\U00002702-\\U000027B0\"\n u\"\\U000024C2-\\U0001F251\"\n \"]+\", flags=re.UNICODE)\n data=emoji_clean.sub(r'',data)\n url_clean= re.compile(r\"https://\\S+|www\\.\\S+\")\n data=url_clean.sub(r'',data)\n return data\n\n\ntrain_df['question_text']=train_df['question_text'].apply(lambda z : remove_url(z))\ntrain_df['question_text']=train_df['question_text'].apply(lambda z: clean_data(z))\ntrain_df['question_text']=train_df['question_text'].apply(lambda z: remove_html(z))\ntrain_df['question_text']=train_df['question_text'].apply(lambda z: remove_punctuations(z))\n\nprint(\"Cleaned Train Insincere Question Set\")\nprint(train_df[train_df['target']==1]['question_text'].head())\nprint(\"Cleaned Train Sincere Question Set\")\nprint(train_df[train_df['target']==0]['question_text'].head())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Lemmatizing the corpus as a backup\n\nfrom nltk.stem import WordNetLemmatizer\n\n\ndef lemma_traincorpus(data):\n lemmatizer=WordNetLemmatizer()\n out_data=\"\"\n for words in data:\n out_data+= lemmatizer.lemmatize(words)\n return out_data\n\ntrain_df['question_text']=train_df['question_text'].apply(lambda z: lemma_traincorpus(z))\nprint(train_df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Vectorization and Benchmarking\n\nIn this contextt, we will look into vectorizing the training set with tfidf vectorizer and count vectorizer. Post that we will be fitting a statistical model from the sklearn library to get an initial benchmark . This is a very important step as an initial statistical benchmark should always be done before going in to deep models and advanced architectures.\n\nIn this section we will be vectorizing the train corpus first and then we will be using the following models:\n\n1. [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)\n2. [Naive Bayes](https://scikit-learn.org/stable/modules/naive_bayes.html)\n3. [SupportVectorMachines](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)\n4. [LinearDiscriminantAnalysis](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html)\n5. [DecisionTrees](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)\n6. [RandomForest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n7. [XGBoost](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html)\n8. [LightGBM](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html) \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#tfidf vectorization\n\ntfidf_vect=TfidfVectorizer(stop_words='english',ngram_range=(1,3))\n#tfidf_vect.fit_transform(train_df['question_text'].values.tolist() + test_df['question_text'].values.tolist())\ntrain_tfidf=tfidf_vect.fit_transform(train_df['question_text'].values.tolist())\ntest_tfidf=tfidf_vect.fit_transform(test_df['question_text'].values.tolist())\nprint(train_tfidf)\n\n#count vectorization\ndef count_vectorize(train_data,test_data):\n count_vectorize=CountVectorizer(stop_words='english',ngram_range=(1,3),analyzer='word',token_pattern='r\\w{1,}')\n count_vectorize.fit_transform(train_data['question_text'].values.tolist() + test_data['question_text'].values.tolist())\n train_count=count_vectorize.transform(train_data['question_text'].values.tolist())\n test_count=count_vectorize.transform(test_data['question_text'].values.tolist())\n return train_count,test_count\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Statistical Models \n\nIn this section, we will be analysing the outcomes of different models mentioned above using k fold statistics from sklearn.\n\n1. [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html)\n2. [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html)\n3. [Kernel Using KFold](https://www.kaggle.com/abhilash1910/credit-card-fraud)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y=train_df['target']\n\ntrain_x=train_df['question_text']\nprint(train_y)\nprint(train_x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Some preliminary models for sequential evaluation on the training set\n\n#train_y=train_df['target'].values\n\n# models=[]\n# models.append(('LR',LogisticRegression()))\n# models.append(('KNN',KNeighborsClassifier()))\n# models.append(('LDA',LinearDiscriminantAnalysis()))\n# models.append(('DT',DecisionTreeClassifier()))\n# #models.append(('SVC',SVC()))\n# model_result=[]\n# scoring='accuracy'\n# for name,model in models:\n# kfold=KFold(n_splits=10,random_state=7)\n# results=cross_val_score(model,train_tfidf,train_y,cv=kfold,scoring=scoring)\n# print(\"Classifiers: \",name, \"Has a training score of\", round(results.mean(), 2) * 100, \"% accuracy score\")\n# model_result.append(results.mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Partitioning the Set\n\nSince there is no validation set to benchmark against , we create an internal partition of training and testing sets from the tfidf transformed data.\n\n1.[TrainTestSplit](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#split the training set into train and test evaluation sets\nprint(train_tfidf.shape)\n\ntrain_x,test_x,train_y,test_y=train_test_split(train_tfidf,train_y,test_size=0.2,random_state=42)\nprint(train_x.shape)\nprint(train_y.shape)\nprint(test_x.shape)\nprint(test_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# models=[]\n# #models.append(('LR',LogisticRegression()))\n# models.append(('KNN',KNeighborsClassifier()))\n# models.append(('LDA',LinearDiscriminantAnalysis()))\n# models.append(('DT',DecisionTreeClassifier()))\n\n# #Now trying out different stats models\n# def model_training(model,train_x,test_x,train_y,test_y):\n# model.fit(train_x,train_y)\n# pred=model.predict(test_x)\n# print(\"Evaluate confusion matrix\")\n# print(confusion_matrix(test_y,pred))\n# print(accuracy_score(test_y,pred))\n# return accuracy_score(test_y,pred)\n \n# for name,mods in models:\n# accuracy_score=model_training(mods,train_x,test_x,train_y,test_y)\n# print(accuracy_score)\n \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Logistic Regression\n\nThe logistic regression relies on Sigmoid function for classifications:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Logistic Regression\nmodel=LogisticRegression()\nmodel.fit(train_x,train_y)\npred=model.predict(test_x)\nprint(\"Evaluate confusion matrix\")\nprint(confusion_matrix(test_y,pred))\nprint(accuracy_score(test_y,pred))\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"#Naive Bayes\n\nmodel=MultinomialNB()\nmodel.fit(train_x,train_y)\npred=model.predict(test_x)\nprint(\"Evaluate confusion matrix\")\nprint(confusion_matrix(test_y,pred))\nprint(accuracy_score(test_y,pred))\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LDA OverView:\n\nA Generic LDA, can be viewed as follows:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#LDA- Linear Disciminant Analysis\n\nmodel=LinearDiscriminantAnalysis()\nx_train=train_x.toarray()\nmodel.fit(x_train,train_y)\npred=model.predict(test_x)\nprint(\"Evaluate confusion matrix\")\nprint(confusion_matrix(test_y,pred))\nprint(accuracy_score(test_y,pred))\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Idea on Boosted Trees Architecture\n\nThis [link](https://xgboost.readthedocs.io/en/latest/tutorials/model.html) provides a good material for boosted trees , which can be shown below as :\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#XGBoost\nfrom xgboost import XGBClassifier as xg\nmodel_xgb= xg(n_estimators=100,random_state=42)\nmodel_xgb.fit(train_x,train_y)\ny_pred_xgb=model_xgb.predict(test_x)\nprint(\"Confusion matrix\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Boosting Architecture Summary\n\nThis provides a proper outline of a generic boosting model:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(confusion_matrix(test_y,y_pred_xgb))\nprint(accuracy_score(test_y,y_pred_xgb.round()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#LightGBM\nfrom lightgbm import LGBMClassifier as lg\nmodel_lgbm= lg(n_estimators=100,random_state=42)\nmodel_lgbm.fit(train_x,train_y)\ny_pred_lgbm=model_lgbm.predict(test_x)\nprint(\"Confusion matrix\")\nprint(confusion_matrix(test_y,y_pred_lgbm))\nprint(accuracy_score(test_y,y_pred_lgbm.round()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Decision Trees\nmodel_dt=DecisionTreeClassifier(random_state=42)\nmodel_dt.fit(train_x,train_y)\ny_pred=model_dt.predict(test_x)\nprint(\"Confusion matrix\")\nprint(confusion_matrix(test_y,y_pred))\nprint(accuracy_score(test_y,y_pred.round()))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Random Forest Architecture\n\nA random forest architecture with majority voting mechanism appears as follows:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Random Forest\nmodel_dt=RandomForestClassifier(random_state=42)\nmodel_dt.fit(train_x,train_y)\ny_pred=model_dt.predict(test_x)\nprint(\"Confusion matrix\")\nprint(confusion_matrix(test_y,y_pred))\nprint(accuracy_score(test_y,y_pred.round()))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some facts on SVM\n\nSVM uses a kernel(basis) for marginalizing the classes and hence it can take up a lot of time based on certain kernels. An overview of certain terms in SVM classification can be viewed as follows:\n\n"},{"metadata":{},"cell_type":"markdown","source":"### SVM takes a lot of time to train and evaluate\n\nSince SVM takes time for vectorized datasets, we will be seeing it after the reduction/decomposition techniques- PCA/TSNE/SVD."},{"metadata":{},"cell_type":"markdown","source":"## Inference From Statistical Models\n\nWe see that statistical models provide a good benchmark in terms of performance. Hyperparamater tuning using gridsearch,random search can also help to increase the performance. \n\n1. [GridSearch](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)\n2. [RandomizedSearch](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n\nThere are other models which are important in this context, namely SVM.SVM generally takes a lot of time to train and hence dimensionality reduction is very important before using SVM for statistical modelling. \n"},{"metadata":{},"cell_type":"markdown","source":"## Dimensionality Reduction Techniques\n\nIn this context , we will get into PCA, SVD and TSNE transformation of the input vectors to generate finite central points . These techniques involve statistical tensor/matrix decomposition algorithms which try to preserve the features based on the importance of the eigen vectors of the matrix."},{"metadata":{"trusted":true},"cell_type":"code","source":"def dimen_reduc_plot(test_data,test_label,option):\n tsvd= TruncatedSVD(n_components=2,algorithm=\"randomized\",random_state=42)\n tsne=TSNE(n_components=2,random_state=42) #not recommended instead use PCA\n pca=SparsePCA(n_components=2,random_state=42)\n if(option==1):\n tsvd_result=tsvd.fit_transform(test_data)\n plt.figure(figsize=(10,8))\n colors=['orange','red']\n \n sns.scatterplot(x=tsvd_result[:,0],y=tsvd_result[:,1],hue=test_label )\n \n plt.show()\n plt.figure(figsize=(10,10))\n plt.scatter(tsvd_result[:,0],tsvd_result[:,1],c=test_label,cmap=matplotlib.colors.ListedColormap(colors))\n color_red=mpatches.Patch(color='red',label='False_Tweet')\n color_orange=mpatches.Patch(color='orange',label='Real_Tweet')\n plt.legend(handles=[color_orange,color_red])\n plt.title(\"TSVD\")\n plt.show()\n if(option==2):\n tsne_result=tsne.fit_transform(test_data)\n plt.figure(figsize=(10,8))\n colors=['orange','red']\n sns.scatterplot(x=tsne_result[:,0],y=tsne_result[:,1],hue=test_label)\n plt.show()\n plt.figure(figsize=(10,10))\n plt.scatter(x=tsne_result[:,0],y=tsne_result[:,1],c=test_label,cmap=matplotlib.colors.ListedColormap(colors))\n color_red=mpatches.Patch(color='red',label='False_tweet')\n color_orange=mpatches.Patch(color='orange',label='Real_Tweet')\n plt.legend(handles=[color_orange,color_red])\n plt.title(\"PCA\")\n plt.show() \n if(option==3):\n pca_result=pca.fit_transform(test_data.toarray())\n plt.figure(figsize=(10,8))\n colors=['orange','red']\n sns.scatterplot(x=pca_result[:,0],y=pca_result[:,1],hue=test_label)\n plt.show()\n plt.figure(figsize=(10,10))\n plt.scatter(x=pca_result[:,0],y=pca_result[:,1],c=test_label,cmap=matplotlib.colors.ListedColormap(colors))\n color_red=mpatches.Patch(color='red',label='False_tweet')\n color_orange=mpatches.Patch(color='orange',label='Real_Tweet')\n plt.legend(handles=[color_orange,color_red])\n plt.title(\"TSNE\")\n plt.show()\n\ndimen_reduc_plot(train_x,train_y,1)\ndimen_reduc_plot(train_x,train_y,3)\ndimen_reduc_plot(train_x,train_y,2)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dimen_reduc_plot(test_data,test_label,option):\n tsvd= TruncatedSVD(n_components=2,algorithm=\"randomized\",random_state=42)\n tsne=TSNE(n_components=2,random_state=42) #not recommended instead use PCA\n pca=SparsePCA(n_components=2,random_state=42)\n if(option==1):\n tsvd_result=tsvd.fit_transform(test_data)\n plt.figure(figsize=(10,8))\n colors=['orange','red']\n \n sns.scatterplot(x=tsvd_result[:,0],y=tsvd_result[:,1],hue=test_label )\n \n plt.show()\n plt.figure(figsize=(10,10))\n plt.scatter(tsvd_result[:,0],tsvd_result[:,1],c=test_label,cmap=matplotlib.colors.ListedColormap(colors))\n color_red=mpatches.Patch(color='red',label='False_Tweet')\n color_orange=mpatches.Patch(color='orange',label='Real_Tweet')\n plt.legend(handles=[color_orange,color_red])\n plt.title(\"TSVD\")\n plt.show()\n if(option==2):\n tsne_result=tsne.fit_transform(test_data)\n plt.figure(figsize=(10,8))\n colors=['orange','red']\n sns.scatterplot(x=tsne_result[:,0],y=tsne_result[:,1],hue=test_label)\n plt.show()\n plt.figure(figsize=(10,10))\n plt.scatter(x=tsne_result[:,0],y=tsne_result[:,1],c=test_label,cmap=matplotlib.colors.ListedColormap(colors))\n color_red=mpatches.Patch(color='red',label='False_tweet')\n color_orange=mpatches.Patch(color='orange',label='Real_Tweet')\n plt.legend(handles=[color_orange,color_red])\n plt.title(\"PCA\")\n plt.show() \n if(option==3):\n pca_result=pca.fit_transform(test_data.toarray())\n plt.figure(figsize=(10,8))\n colors=['orange','red']\n sns.scatterplot(x=pca_result[:,0],y=pca_result[:,1],hue=test_label)\n plt.show()\n plt.figure(figsize=(10,10))\n plt.scatter(x=pca_result[:,0],y=pca_result[:,1],c=test_label,cmap=matplotlib.colors.ListedColormap(colors))\n color_red=mpatches.Patch(color='red',label='False_tweet')\n color_orange=mpatches.Patch(color='orange',label='Real_Tweet')\n plt.legend(handles=[color_orange,color_red])\n plt.title(\"TSNE\")\n plt.show()\n\ndimen_reduc_plot(test_x,test_y,1)\ndimen_reduc_plot(test_x,test_y,3)\ndimen_reduc_plot(test_x,test_y,2)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Statistical Learning and Inference - Conclusion\n\nThrough these methods, we have evaluated the importance of statistical models in nlp classification tasks just by sampling words, tokens .While these approaches are often overlooked upon , it is very important to mention that these benchmarks are important to validate against deep learning models. [Abhishek Thakur's Kernel](https://www.kaggle.com/abhishek/approaching-almost-any-nlp-problem-on-kaggle) provides a good idea on this, and can be used for reference. "},{"metadata":{},"cell_type":"markdown","source":"# Neural Networks\n\nIn this context, we will be building a preliminary deep model using sophisticated neural networks and variants of RNNs.\nWe will be building a simple LSTM model for validating the influence of deep models with respect to the statistical ones. In the first case, we will be using the Keras Embedding layer and visualize the results before using the embedding models.\n\n1. [Keras LSTM](https://keras.io/api/layers/recurrent_layers/lstm/)\n2. [Keras](https://keras.io/about/)\n3. [Keras Starter Guides](https://keras.io/examples/nlp/)\n4. [Tensorflow Starter](https://www.tensorflow.org/tutorials/keras/text_classification)\n5. [Tensorflow Hub](https://www.tensorflow.org/tutorials/keras/text_classification_with_hub)\n6. [Jason's Blog-Best practises](https://machinelearningmastery.com/best-practices-document-classification-deep-learning/)\n7. [Jason's Blog-Convolution Networks](https://machinelearningmastery.com/develop-word-embedding-model-predicting-movie-review-sentiment/)\n\nThese are some starter resources for creating preliminary networks for sentiment analysis, text/intent classifications.\nThere will be some advanced architectures which will be focussed later.\n\n\n\nLSTMs are a modified variant of RNNs having 4 gates (i,f,c,o) with the task of alleviating exploding and vanishing gradients.LSTM architecture can be represented as follows:\n\n\n\n\nThe following are the mathematical formulations of a LSTM cell:\n\n\n\n\nGRUs are another variant of LSTM cells with reduced gates and can be represented as follows:\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Convolution Architecture for Image Processing\n\nA classical Convolution 2D architecture for image processing and recognition is shown below:\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Simple Dense Model with Embedding Architecture\n\nA simplified architecture of a neural network model with Embeddings and a MLP Dense Neuron is shown below:\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Creating a Preliminary LSTM/CNN model \n\nIn this stage, let us create a preliminary model without any sophisticated embeddings or architectures. We will be using the Keras Sequential API for creating our models. Let us follow the pipeline for creating any network for text classification:\n\n1. Tokenize the input features- This implies converting the input data into tokens (by using one hot encoding/tokenizing) . \n2. Tokenize the targets- This can be done with the help of Label Encoder(sklearn) or using \"values()\" from python\n3. Padd the tokenized features- To ensure that the length of the tokenized feature is same across all the entries(post padding)\n4. Create a simple model- Build a Sequential Network with Keras Embedding layer as the starting point\n5. Add Layers to the model- Add either Conv/LSTM/Bi-LSTM/RNN/GRU layers with different activations- relu is recommended\n6. Add the Dense layer to the model- At the end , we have to add the Dense layer by flattening the output of the previous layer\n7. Add necessary activation functions- Sigmoid for Binary Classification , Softmax for Multi-class classification\n8. Launch Tensorboard- Launch the Tensorboard and visualize the training parameters-loss,accuracy etc.\n\nThis forms the fundamental steps to build a basic but fundamental pipeline for any language modelling task. Sophistications include adding custom embeddings before the keras Embedding layer and then adding certain other layers(transformer architectures) before the LSTM.\n"},{"metadata":{},"cell_type":"markdown","source":"## Sigmoid LSTM/Embedding Architecture\n\nThis diagram provides an outline how the processing of words are done in LSTM - Embedding layers with Sigmoid activation:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.layers import LSTM, Dense,Flatten,Conv2D,Conv1D,GlobalMaxPooling1D\nfrom keras.optimizers import Adam\n#import MiniAttention.MiniAttention as ma\nimport numpy as np \nimport pandas as pd \nimport re \nfrom bs4 import BeautifulSoup \nfrom keras.preprocessing.sequence import pad_sequences\nfrom nltk.corpus import stopwords \nfrom tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate, TimeDistributed, Bidirectional\nfrom tensorflow.keras.models import Model,Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.model_selection import train_test_split\nfrom keras.utils import to_categorical\nimport numpy as np\nimport pandas as pd\nfrom sklearn.preprocessing import LabelBinarizer, LabelEncoder","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the input features\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tokenize the Input Features\n\nIn this stage we will be tokenizing the inputs using Keras Tokenizers."},{"metadata":{"trusted":true},"cell_type":"code","source":"maxlen=1000\nmax_features=5000 \nembed_size=300\n\n#clean some null words or use the previously cleaned & lemmatized corpus\n\ntrain_x=train_set['question_text'].fillna('_na_').values\nval_x=test_set['question_text'].fillna('_na_').values\n\n#Tokenizing steps- must be remembered\ntokenizer=Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_x))\ntrain_x=tokenizer.texts_to_sequences(train_x)\nval_x=tokenizer.texts_to_sequences(val_x)\n\n#Pad the sequence- To allow same length for all vectorized words\ntrain_x=pad_sequences(train_x,maxlen=maxlen)\nval_x=pad_sequences(val_x,maxlen=maxlen)\n\n\n#get the target values - either using values or using Label Encoder\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Padded and Tokenized Training Sequence\".format(),train_x.shape)\nprint(\"Target Values Shape\".format(),train_y.shape)\nprint(\"Padded and Tokenized Training Sequence\".format(),val_x.shape)\nprint(\"Target Values Shape\".format(),val_y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_x[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating a basic Model \n\nIn this step, we create a basic model with help of Keras Embedding Layer. Here we are using hte sequential API from Keras. Some resources related to the activation functions and layers:\n\n1. [Sequential](https://keras.io/guides/sequential_model/)\n2. [Tensorflow Activations](https://www.tensorflow.org/api_docs/python/tf/keras/activations)\n3. [Keras Loss](https://keras.io/api/losses/)\n4. [Keras Optimizers](https://keras.io/api/optimizers/)\n5. [Keras Metrics](https://keras.io/api/metrics/)\n6. [Dense Layer](https://keras.io/api/layers/core_layers/dense/)\n7. [Core Layers](https://keras.io/api/layers/core_layers/)\n\nThese should be sufficient for getting started and creating models."},{"metadata":{},"cell_type":"markdown","source":"## Flow of layers in Sequential Models\n\nParticularly in NLP, the flow of the layers is as follows:\n\n1. Embedding Layer\n2. LSTM/CuDNNLSTM/ Bidirectional LSTM/CuDNNGRU/GRU layers\n2. Convolution/CNN\n3. GlobalMaxPooling Layer -Optional\n4. Dense Layer\n5. Final Dense Layer (with sigmoid activation for binary and softmax activation for multiclass)\n6. Model Compile - loss- binary_crossentropy (binary classification) or categorical_crossentropy(categorical classification)\n7. Optimizers for Model.compile - Adam ,Adagrad,Adadelta ,Nadam,Nesterov,SGD,RMSProp\n8. Metrics- accuracy\n9. Model fit - Input features\n\nThis is the general flow for building any network in NLP \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating a basic model- without pretrained embeddings\nmodel=Sequential()\nmodel.add(Embedding(max_features,embed_size,input_length=maxlen))\nmodel.add(LSTM(60))\nmodel.add(Dense(16,activation='relu'))\nmodel.add(Dense(1,activation='sigmoid'))\nmodel.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fit the model with the inputs\n\nmodel.fit(train_x,train_y,batch_size=512,epochs=2,verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fit and validate together\nmodel.fit(train_x,train_y,batch_size=128,epochs=3,verbose=2,validation_data=(val_x,val_y))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating the model without using Sequential API\n\nIn this case, we will be using a different technique to build models . We will specify the inputs explicitly at each layer of the models for this purpose.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nimport math\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade keras","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#CuDNN uses the Nvidia GPU for faster training\n\ninp=Input(shape=(maxlen,))\nz=Embedding(max_features,embed_size)(inp)\nz=Bidirectional(LSTM(60,return_sequences='True'))(z)\nz=GlobalMaxPool1D()(z)\nz=Dense(16,activation='relu')(z)\nz=Dense(1,activation='sigmoid')(z)\nmodel=Model(inputs=inp,outputs=z)\nmodel.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\nmodel.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fit and validate together\nmodel.fit(train_x,train_y,batch_size=512,epochs=1,verbose=2,validation_data=(val_x,val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Embeddings\n\nIn this context, we will explore certain embeddings which may increase the performance of the model. Pre-trained embeddings provide a better representation of word vectors, and we will be analysing the word2vec, glove embeddigns as our starting point\n\n1. [Glove](https://nlp.stanford.edu/projects/glove/)\n2. [Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html)\n\n\nWord2Vec employs cosine distance measure between word vectors:\n\n\n\n\nThis excellent resource by Jalammar will help in [understandability](http://jalammar.github.io/illustrated-word2vec/) of skipgram and common bag of words models\n\nGlobal Vectors or glove also employs distance metric for comparison of the word vectors in an unsupervised manner:\n\n\n\n\nWe will be exploring more as we progress"},{"metadata":{"trusted":true},"cell_type":"code","source":"#see the input\n\n!ls ../input\n!ls ../input/quora-insincere-questions-classification","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/wikinews300d1msubwordvec","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization of Embeddings\n\nWe can also visualize the embeddings which appears like a wave. \nA presentation of this is done in the notebook:\n\n1. [Twitter Analysis](https://www.kaggle.com/abhilash1910/tweet-analysis-eda-cleaning-tsne-glove-tf)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#visualize the embeddings\nplt.plot(embedding_matrix[10])\nplt.plot(embedding_matrix[5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Rebuilding the model with pre-trained embeddings from Glove\n\ninp=Input(shape=(maxlen,))\nz=Embedding(max_features,embed_size,weights=[embedding_matrix])(inp)\nz=Bidirectional(LSTM(60,return_sequences='True'))(z)\nz=GlobalMaxPool1D()(z)\nz=Dense(16,activation='relu')(z)\nz=Dense(1,activation='sigmoid')(z)\nmodel=Model(inputs=inp,outputs=z)\nmodel.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\nmodel.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fit and validate together- with glove embedding\nmodel.fit(train_x,train_y,batch_size=512,epochs=1,verbose=2,validation_data=(val_x,val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n\n#Fast Text Embeddings for training \nEMBEDDING_FILE = '../input/wikinews300d1msubwordvec/wiki-news-300d-1M-subword.vec'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n if i >= max_features: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#visualize the embeddings-FastText\nplt.plot(embedding_matrix[10])\nplt.plot(embedding_matrix[5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inp=Input(shape=(maxlen,))\nz=Embedding(max_features,embed_size,weights=[embedding_matrix])(inp)\nz=Bidirectional(LSTM(60,return_sequences='True'))(z)\nz=GlobalMaxPool1D()(z)\nz=Dense(16,activation='relu')(z)\nz=Dense(1,activation='sigmoid')(z)\nmodel=Model(inputs=inp,outputs=z)\nmodel.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\nmodel.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fit and validate together- with glove embedding\nmodel.fit(train_x,train_y,batch_size=512,epochs=1,verbose=2,validation_data=(val_x,val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! ls ../input\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/googlenewsvectorsnegative300","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/paragram-300-sl999","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! ls ../input/word2vec-sample","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n\n#Paragram Embeddings for training \nEMBEDDING_FILE = '../input/paragram-300-sl999/paragram_300_sl999.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n if i >= max_features: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_corpus(data):\n corpus=[]\n for i in tqdm(data):\n words=[word for word in (i)]\n corpus.append(words)\n return corpus\ndef create_embedding(data):\n embedding_map={}\n file=open('../input/paragram-300-sl999/paragram_300_sl999.txt','r')\n for f in file:\n values=f.split(' ')\n word=values[0]\n coef=np.asarray(values[1:],dtype='float32')\n embedding_map[word]=coef\n file.close()\n return embedding_map\ndef embedding_preprocess(data):\n #max_word_length=1000\n max_sequence_length=100\n tokenizer=Tokenizer()\n tokenizer.fit_on_texts(data)\n sequences=tokenizer.texts_to_sequences(data)\n \n word_idx=tokenizer.word_index\n data_pad=pad_sequences(sequences,padding=\"post\",maxlen=max_sequence_length)\n emb_dim=data.get('a').shape[0]\n num_length=len(word_idx)+1\n emb_mat=np.zeros((num_length,emb_dim))\n for word,idx in tqdm(word_idx.items()):\n if idx > num_length:\n continue\n elif idx < num_length:\n emb_vector=data.get(word)\n if emb_vector is not None: \n emb_mat[idx]=emb_vector\n \n return emb_mat,word_idx,data_pad,num_length\n \n \ncorpus_train_data=add_corpus(train_x)\nprint(\"corpus created\")\n\nembedding_map= create_embedding(corpus_train_data)\nprint(\"Embedding matrix created\")\nemb_mat,word_idx,pad_data,num_words=embedding_preprocess(embedding_map)\nprint(pad_data.shape)\n\nprint(\"Visualise embedded vectors\")\nplt.plot(emb_mat[10])\nplt.plot(emb_mat[20])\nplt.plot(emb_mat[50])\nplt.title(\"Embedding Vectors\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from gensim.models import KeyedVectors\nEMBEDDING_FILE = '../input/googlenewsvectorsnegative300/GoogleNews-vectors-negative300.bin'\nembeddings_index = KeyedVectors.load_word2vec_format(EMBEDDING_FILE, binary=True)\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = (np.random.rand(nb_words, embed_size) - 0.5) / 5.0\nfor word, i in word_index.items():\n if i >= max_features: continue\n if word in embeddings_index:\n embedding_vector = embeddings_index.get_vector(word)\n embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inp=Input(shape=(maxlen,))\nz=Embedding(max_features,embed_size,weights=[embedding_matrix])(inp)\nz=Bidirectional(LSTM(60,return_sequences='True'))(z)\nz=GlobalMaxPool1D()(z)\nz=Dense(16,activation='relu')(z)\nz=Dense(1,activation='sigmoid')(z)\nmodel=Model(inputs=inp,outputs=z)\nmodel.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\nmodel.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fit and validate together- with glove embedding\nmodel.fit(train_x,train_y,batch_size=512,epochs=1,verbose=2,validation_data=(val_x,val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Word2vec examples\n\nfrom gensim.models import Word2Vec\n\nmodel = Word2Vec.load_word2vec_format('../input/googlenewsvectorsnegative300/GoogleNews-vectors-negative300.bin', binary=True, norm_only=True)\n\n\ndog = model['dog']\n\nprint(model.most_similar(positive=['woman', 'king'], negative=['man']))\n\nprint(model.doesnt_match(\"breakfast cereal dinner lunch\".split()))\n\nprint(model.similarity('woman', 'man'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Embeddings Conclusion\n\nThere are many embedding libraries which are present in the Kaggle datasets. This provides a preview using some of the most popular and commonly used word embeddings and running the deep models using it.\nHere is an important kernel which may be of help:\n\n1. [SRK's kernel](https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings)\n\nThis concludes today's session -------\n\n\nThe next session on Transformers is [here](https://www.kaggle.com/abhilash1910/nlp-workshop-2-ml-india)\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}