{"cells":[{"metadata":{},"cell_type":"markdown","source":"# End-to-End Natural Language Processing-1\n\nIn this notebook, I am doing end-to-end word embeddings of Natural Languange Processing on this problem. There are 4 data sets are given. \n\n- Train data is collection of sincere and insincere questions with target (0=sincere; 1=insincere questions) (text data)\n\n- Test Data \n\n- Submission data \n\n- Embeddings\n\n## Detailed Steps to Build Non-Semantic Vectors & Semantic Embeddings\n\nIn this first Part of the \"Quora Insincere Questions\" Kaggle kernel, I am doing the exploratory analysis of the train data and build different static (word2vec) embeddings on the words of train corpus\n\nIn the Second Part of this problem, I'd discuss performance of the various statistical ML algorithms with Non-Semantic, Static Semantic and Dynamic embeddings.\n\nIn the third Part, I'd build the keras-TF neural networks to train BERT, GPT3, etc.\n\nIn this kernel, I am starting with the loading of the data with Pandas library of Python.\n\nSo, let's get started.\n\n> As always, I hope you find this kernel useful and your **UPVOTES** would be highly appreciated."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\nimport string\nimport collections\nfrom collections import defaultdict,Counter\nimport nltk\nfrom nltk import word_tokenize\nfrom nltk.corpus import stopwords\nstop_words = stopwords.words('english')\nfrom nltk.tokenize import word_tokenize,TreebankWordTokenizer,WordPunctTokenizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split,cross_val_score,GridSearchCV,cross_val_predict\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.metrics import classification_report,roc_auc_score,roc_curve,r2_score,recall_score,confusion_matrix,precision_recall_curve,accuracy_score\nfrom sklearn.model_selection import StratifiedKFold,KFold,StratifiedShuffleSplit\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing of text data\n\nCleaning and Normaling the text data is hard and time consuming. Visualisation of text data is done with and without cleaning the data.\n\nTo clean the data, the following items have to be removed -\n1. punctuations\n2. whitespaces\n3. numbers\n4. typos, single words, etc\n5. stopwords\n6. urls,emojis,other redundant words/letters, if any\n\n**Cleaning** is very important to filter the words which are not necessary to train ML model - as such words makes training very costly.\n\n### Visualisation of the text data \n\nI've done stat analysis & visualisation of the text data with the help of \n- Ngram analysis (onegram,bigram,trigram, fourgram)\n- Distribution plots of words\n- WordCloud\n\nwith and without cleaning the data\n\nand\n\nwith normalizing (stemming & lemmatizing) after cleaning the text data"},{"metadata":{},"cell_type":"markdown","source":"## Load train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv('../input/traindata/train.csv')\ntraining_data = train_data.copy()\n#check missing data\ntraining_data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## First 5 records of train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_colwidth',None)\ntraining_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nfrom seaborn import countplot\nfrom matplotlib.pyplot import suptitle\n%matplotlib inline\n\n#counts of sincere and insincere question texts:\ncount=training_data['target'].value_counts()\nprint('Total Counts of both sets\\n'.format(),count)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Counts of Classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.countplot(y=\"target\",palette =['maroon','blue'],data=training_data)\nplt.suptitle(\"Count values of Sincere Questions (0) & Insincere Questions (1)\")\nplt.xlabel(\"Number of Questions\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sincere_questions = training_data[training_data['target']== 0]['question_text']\nprint(\"Quora Sincere Question Content\\n\",sincere_questions[:2])\nprint(\"_______________________________________________________________________________________________\")\ninsincere_questions = training_data[training_data['target']== 1]['question_text']\nprint(\"Quora Insincere Question Content\\n\",insincere_questions[:2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ## Exploratory Data Analysis of Text Data - Part I"},{"metadata":{},"cell_type":"markdown","source":"## Word Counts in Each Document"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Analyse the count of words in each segment - both positive and negative questions\n#Function for checking word length\ndef cal_len(data):\n    return len(data)\n\ncount_sincere = sincere_questions.str.split().apply(lambda z:cal_len(z))\ncount_insincere = insincere_questions.str.split().apply(lambda z:cal_len(z))\n\nprint(\"Word Counts of Sincere Questions:\\n\" + str(count_sincere[:2]))\nprint(\"______________________________________________________________\")\nprint(\"Word Counts of Insincere Questions:\\n\" + str(count_insincere[:2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"max words in the sincere questions text   :%d \\nmax words in the insincere questions text :%d\" % (max(count_sincere),max(count_insincere)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Distribution Plot - Word Counts**"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,(ax1,ax2)= plt.subplots(1,2,figsize=(20,5))\nsns.histplot(count_insincere,ax=ax1,color='Blue')\nax1.set_title(\"counts of words in target=1 (Insincere)\")\nsns.histplot(count_sincere,ax=ax2,color='Red')\nax2.set_title(\"counts of words in target=0 (Sincere)\")\nfig.suptitle(\"Distribution of Words in the train Data\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Count Plots - Punctuations & Stopwords"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Count Punctuations/Stopwords and other semantic datatypes\n#Create generic plotter with Seaborn\ndef plot_count(count_ones,count_zeros,title_1,title_2,subtitle,figsize):\n    fig,(ax1,ax2)=plt.subplots(1,2,figsize=figsize)\n    sns.histplot(count_zeros,bins=50,ax=ax1,color='Blue')\n    ax1.set_title(title_1)\n    sns.histplot(count_ones,bins=50,ax=ax2,color='Red')\n    ax2.set_title(title_2)\n    fig.suptitle(subtitle)\n    plt.show()\n\nsincere_punct= sincere_questions.apply(lambda z: len([c for c in str(z) if c in string.punctuation]))\ninsincere_punct= insincere_questions.apply(lambda z:len([c for c in str(z) if c in string.punctuation]))\nplot_count(sincere_punct,insincere_punct,\"Sincere Questions-Count of Punctuations\\n\",\"Insincere Questions-Count of Punctuations\\n\",\"Punctuation Analysis\",figsize=(15,5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stopwords Counts"},{"metadata":{"trusted":true},"cell_type":"code","source":"stops=set(stopwords.words('english'))\nsincere_stops= sincere_questions.apply(lambda z : np.mean([len(z) for w in str(z).split()]))\ninsincere_stops= insincere_questions.apply(lambda z : np.mean([len(z) for w in str(z).split()]))\nplot_count(sincere_stops,insincere_stops,\"Sincere Questions-Stopwords\",\"Insincere Questions-Stopwords\",\"Stopwords Analysis\\n\",figsize=(20,5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## WordCloud Visualizations"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Method for creating wordclouds\n\nfrom wordcloud import WordCloud,ImageColorGenerator,STOPWORDS\nfrom PIL import Image\nimport urllib\nimport requests\n\n\n# combining the image with the dataset\nMask = np.array(Image.open('../input/quota-logo/quora-logo-2439.png'))\n\n# I've used the ImageColorGenerator library from Wordcloud \n# Here I took the color of the image and impose it over our wordcloud\nimage_colors = ImageColorGenerator(Mask)\n\n\ndef display_cloud(data,title,figsize):\n    wc = WordCloud(stopwords=STOPWORDS,background_color='white', max_words=2000, max_font_size=256,random_state=42,\n               height=1500, width=4000,mask=Mask)\n    wc.generate(' '.join(data))\n    plt.figure(figsize=figsize)\n    plt.imshow(wc.recolor(color_func=image_colors),interpolation=\"hamming\")\n    plt.title(title)\n    plt.axis('off')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Word Clouds for Sincere Questions & Insincere Questions\n(without removing punctuations, stopwords,etc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set_context(\"notebook\", font_scale=1.1)\ndisplay_cloud(sincere_questions[:10000],title='Word Cloud - Quora Sincere Qiuestions\\n',figsize=(10,5))\ndisplay_cloud(insincere_questions[:10000],title='Word Cloud - Quora Insincere Qiuestions\\n',figsize=(10,5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Most Frequent Words in the Corpus\n(Omitted question-words)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Simplified counter function\ndef create_corpus(x=0):\n    corpus=[]\n    for x in training_data[training_data['target']==x]['question_text'].str.split():\n        for i in x:\n            corpus.append(i)\n    return corpus\n\ncorpus=create_corpus(x=0)\ncounter=Counter(corpus)\nmost=counter.most_common()\nx=[]\ny=[]\nfor word,count in most[:100]:\n    q_words=['What','How','how','Where','where','Why','why','Is','Which','Are','Would','would','Will','will',\n             'Could','could','are','Am','am','Can','Can','if','If','Would','would','Will','will','could','Could',\n             'I','i','You','you','Do','do','did','Did','Does','does','he','she','It','it','Should','should',\n             'shall','Shall','Who','who','One','one','Two','two'\n            ]\n    if (word not in stops and word not in q_words):\n        x.append(word)\n        y.append(count)\n        \nplt.figure(figsize=(15,10))\nsns.barplot(x=y,y=x)\nplt.title(\"Most Frequent Words (in descending order)\\n\")\nplt.xlabel(\"Frequency of Words\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference from Part-I of Stat Analysis\n​\nThe following can be inferred from the data:\n​\n- The dataset is unbalanced\n- The dataset contains unequal number of semantics and punctuations\n- The dataset contains redundant words \n- Stopwords are present in a equal distribution in the dataset"},{"metadata":{},"cell_type":"markdown","source":"> ## Statistical Analysis - Part II\n\nAs it is clear from the above graphs and stats, train data contains lot of stopwords and redundant text. I am removing the whitespaces, numbers, punctuations, stopwords, other redundent text using regular expressions.\n\nGram analysis is an essential tool which forms the base of preparing a common bag of words model containing relevant data. This process implies that we are taking into consideration which words are present in conjunction with other words with a maximum frequency in the dataset."},{"metadata":{},"cell_type":"markdown","source":"### Gram analysis on Training set - OneGram, Bigram and Trigram"},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly \nfrom plotly import subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nstopword=set(stopwords.words('english'))\n\ndef gram_analysis(data,gram):\n    tokens=[t for t in data.lower().split(\" \") if t!=\"\" if t not in stopword]\n    ngrams=zip(*[tokens[i:] for i in range(gram)])\n    final_tokens=[\" \".join(z) for z in ngrams]\n    return final_tokens\n\n#Create frequency grams for analysis\ndef create_dict(data,grams):\n    freq_dict=defaultdict(int)\n    for sentence in data:\n        for tokens in gram_analysis(sentence,grams):\n            freq_dict[tokens]+=1\n    return freq_dict\n\ndef horizontal_bar_chart(df, color):\n    trace = go.Bar(\n        y=df[\"n_gram_words\"].values[::-1],\n        x=df[\"n_gram_frequency\"].values[::-1],\n        showlegend=False,\n        orientation = 'h',\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\ndef create_new_df(freq_dict,):\n    freq_df=pd.DataFrame(sorted(freq_dict.items(),key=lambda z:z[1])[::-1])\n    freq_df.columns=['n_gram_words','n_gram_frequency']\n    trace=horizontal_bar_chart(freq_df[:10],'orange')\n    return trace\n\ndef plot_grams(trace_zero,trace_one):\n    fig = subplots.make_subplots(rows=1, cols=2, vertical_spacing=0.04,\n                                 subplot_titles=[\"Frequent Ngrams of sincere questions\",\"Frequent Ngrams of insincere questions\"])\n    fig.append_trace(trace_zero, 1, 1)\n    fig.append_trace(trace_ones, 1, 2)\n    fig['layout'].update(height=900, width=1000, paper_bgcolor='rgb(233,233,233)', title=\"Top 10 Most Frequent Words of Ngrams\")\n    py.iplot(fig, filename='word-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Onegram analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_train_sin=create_dict(sincere_questions[:1000],1)\ntrace_zero=create_new_df(freq_train_sin)\n\nfreq_train_insin=create_dict(insincere_questions[:1000],1)\ntrace_ones=create_new_df(freq_train_insin)\n\n#plotting OneGrams\nplot_grams(trace_zero,trace_ones)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bigram analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_train_sin=create_dict(sincere_questions[:500],2)\ntrace_zero=create_new_df(freq_train_sin)\n\nfreq_train_insin=create_dict(insincere_questions[:500],2)\ntrace_ones=create_new_df(freq_train_insin)\n\n#plotting Bigrams\nplot_grams(trace_zero,trace_ones)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Trigram analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_train_sin=create_dict(sincere_questions[:500],3)\ntrace_zero=create_new_df(freq_train_sin)\n\nfreq_train_insin=create_dict(insincere_questions[:500],3)\ntrace_ones=create_new_df(freq_train_insin)\n\nplot_grams(trace_zero,trace_ones)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Transforming the uppercase,stopwords, common words, numbers, etc"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Converting uppercase letters to lowercase\ndef convert_2lowercase(data):\n    data =[string.lower() for string in data if string.isupper]\n    return ''.join(data)\n\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: convert_2lowercase(z))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nimport re\n\n#Removes Punctuations-----------1\ndef remove_punct(data):\n    punct_tag=re.compile(r'[^\\w\\s]')\n    data=punct_tag.sub(r'',data)\n    return data\n\n\n# Remove whitespace ----2\ndef remove_whitespace(data):\n    tag=re.compile(r'\\s+')\n    data=tag.sub(r' ',data)\n    return data\n\n#Removes Roman words----------3\ndef remove_roman(data):\n    en_tag =re.compile(r'^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$')\n    data=en_tag.sub(r'',data)\n    return data\n\n\n#Remove redundant words---------------4\ndef remove_redun(data):\n    red_tag=re.compile(r'[?<=(  )\\\\]|[&&|\\|\\|-]')\n    data=red_tag.sub(r' ',data)\n    return \"\".join(data)\n\n\n#Removes Numbers -------5\ndef remove_num(data):\n    tag=re.compile(r'[0-9]+')\n    data=tag.sub(r'',data)\n    return data\n\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: remove_punct(z))\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: remove_whitespace(z))\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: remove_roman(z))\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: remove_redun(z))\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: remove_num(z))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Word Clouds for Sincere Questions & Insincere Questions\n(with removing punctuations, stopwords,etc)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sincere_questions = training_data[training_data['target']== 0]['question_text']\nprint(\"Quora Sincere Question Content\\n\",sincere_questions[:2])\nprint(\"_______________________________________________________________________________________________\")\ninsincere_questions = training_data[training_data['target']== 1]['question_text']\nprint(\"Quora Insincere Question Content\\n\",insincere_questions[:2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_cloud(sincere_questions[:1000],'Quora Sincere Qiuestions\\n',figsize=(10,5))\ndisplay_cloud(insincere_questions[:1000],'Quora Insincere Qiuestions\\n',figsize=(10,5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Trigram Plot of Cleaned train corpus"},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_train_sin=create_dict(sincere_questions[:200],3)\ntrace_zero=create_new_df(freq_train_sin)\n\nfreq_train_insin=create_dict(insincere_questions[:200],3)\ntrace_ones=create_new_df(freq_train_insin)\n\nplot_grams(trace_zero,trace_ones)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The data is cleaned! Next step is to normalize the data."},{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://media.giphy.com/media/9rceHGVEDBuatVhTlS/giphy.gif\" align='left'>"},{"metadata":{},"cell_type":"markdown","source":"## Normalizing the text data"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#stemmimng the text\nfrom nltk.stem.porter import PorterStemmer\nfrom nltk.stem import *\n\ndef stem_traincorpus(data):\n    stemmer = PorterStemmer()\n    out_data=\"\"\n    for words in data:\n        out_data+= stemmer.stem(words)\n    return out_data\n\n\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: stem_traincorpus(z))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#Lemmatize the dataset\nfrom nltk.stem import WordNetLemmatizer\n\n\ndef lemma_traincorpus(data):\n    lemmatizer=WordNetLemmatizer()\n    out_data=\"\"\n    for words in data:\n        out_data+= lemmatizer.lemmatize(words)\n    return out_data\n\ntraining_data['question_text']=training_data['question_text'].apply(lambda z: lemma_traincorpus(z))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Let's have a look at the clean and preprocessed train data set (corpus)\ntraining_data.to_csv(\"clean_lem_stemmed_train_data.csv\",sep=\",\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_colwidth',None)\ntraining_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> # The data set is cleaned, stemmed and lemmatized!\n\n1. It's important to clean the data to create static/dynamic embeddings to analize the word vectors correctly.\n\n2. In the context of embeddings, if I donot remove these inconsistencies, the vectors will not be properly placed in vector space."},{"metadata":{"trusted":true},"cell_type":"code","source":"#copy of the data with the first column dropped\nnew_train_data = training_data.copy()\nnew_train_data.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#garbage collector\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Word Embeddings\n\nVectorizing is a technique to map the words of train corpus into higher dimensional space as the unique vectors. The vectorization strategies allow the word corpus to be properly suitable for advanced semantic analysis.\n\nThere are 2 variants of (vectorizing/embeddings) transforming the textual corpus to a numerical vector:\n1. Vectorize without semantics\n2. Vectorize with semantics (Retain Semantics)\n\n\n### Vectorize without semantics - vectorize to provide a co-occurence probabilistic distribution\n\nTF-IDF, Count vectorization,One hot vectorization, etc are few vectorization methods\n\nThese methods leverage statistical co-occurence probabilities and log likelihoods for determining the frequently occuring sentences or group of words in a corpus.\n\n### Vectorize with semantics (Retain Semantics) - relies on applying vectors with respect to semantic importance\n\n* Embeddings fall under this category. Embeddings are largely of 2 kinds:\n\n(i)  Static Embeddings: Word2Vec, Glove, Fasttext, Paragram\n\n(ii) Dynamic Embeddings: ELMO, BERT & its variants, XLNet/Transformer-XL\n\nAll of these embeddings rely on pretrained word vectors where a probabilistic score is attributed to each word in the corpus. These probabilities are plotted in a low dimensional plane and the \"meaning\" of the words are inferred from these vectors. Generally speaking cosine distance is taken as the major metric of similarity measurement between word and sentence vectors to infer similarity."},{"metadata":{},"cell_type":"markdown","source":"## TF-IDF Vectorization"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\n\ntfidf_vect=TfidfVectorizer(stop_words='english',ngram_range=(1,3))\ntrain_tfidf=tfidf_vect.fit_transform(new_train_data['question_text'].values.tolist())\ntrain_tfidf.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are more than 1lakh features (words) in the TF-IDF high-dimensional matrix. It's very sparse matrix."},{"metadata":{},"cell_type":"markdown","source":"## Visualizing the word vectors\n* As words and sentences are vectorized, the dimensions of the vector space becomes significantly large to be accomodated in a model\n* For any computation system it is recommended to keep the dimensions of a tensor (matrix) as small as possible and maintain its regularity\n* For tensors with larger dimensions and irregular shapes, it is difficult for the system to perform any operation (matrix/tensor multiplication)\n* Complex operations like tensor differentiation (Jacobian) or numerical approximation is another difficult thing to do for large matrices\n\nI've done dimension reduction of sparse matrix with PCA and t-SNE:\n\n1) **PCA** - Principal Component Analysis is a well known method and forms the base of all decomposition techniques\n\n2) **TSNE** - is a more sophisticated method which uses a non convex optimization along with gradient descent. \n\nThis is different than Eigen Vector (convex optimization) method of PCA and hence different results may be obtained in different iterations. It is a memory intensive method and is often powerful at the expense of longer execution time.\n\nThe reduced matrices are well-fitted to perform any numerical approximation tasks from differentiation to higher order non linear dynamics. "},{"metadata":{},"cell_type":"markdown","source":"# Visualisation of vectorised words with PCA,Truncated SVD and t-SNE Plots in 2D space"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Vectors of training data corpus    \ntext_vectors = new_train_data['question_text'].values\ntarget_vectors = new_train_data['target'].values\n\n#Vectors of sincere questions corpus\ncount_zero= new_train_data[new_train_data['target']==0]\ntext_vectors_sincere = count_zero['question_text'].values\ntarget_vectors_sincere = count_zero['target'].values\n\n#Vectors of  insincere questions corpus\ncount_one= new_train_data[new_train_data['target']==1]\ntext_vectors_insincere = count_one['question_text'].values\ntarget_vectors_insincere = count_one['target'].values\n\n#Using Count Vectorization - a simple count method to vectorize the tokens in text data\n\ndef vectorize(data):\n    c=CountVectorizer(lowercase=False)\n    cv_data= c.fit_transform(data)\n    return cv_data\n\ntrain_data_cv = vectorize(text_vectors)\nsincere_train_data_cv = vectorize(text_vectors_sincere)\ninsincere_train_data_cv = vectorize(text_vectors_insincere)\n\n#Using TF_IDF Count Vectorization function - TF-IDF values of tokens in the text data\ndef tfidf(data):\n    t=TfidfVectorizer()\n    tfidf_data =t.fit_transform(data)\n    return tfidf_data\n\ntrain_data_tfidf= tfidf(text_vectors)\nsincere_train_data_tfidf= tfidf(text_vectors_sincere)\ninsincere_train_data_tfidf= tfidf(text_vectors_insincere)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Principal Component Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfrom sklearn import preprocessing,metrics,manifold\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.manifold import TSNE\nfrom sklearn.decomposition import PCA, TruncatedSVD,SparsePCA\nfrom sklearn.svm import SVC\nfrom matplotlib import *\nimport matplotlib.pyplot as plt\nfrom matplotlib import cm\nimport matplotlib.colors\nfrom matplotlib.colors import ListedColormap\n\n##PCA\npca=SparsePCA(n_components=2,random_state=14)\npca_result=pca.fit_transform(train_data_cv[:2000].toarray())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##Plots of high dimensional word vectors on to the 2D surface with PCA, SVD & t-SNE\nimport matplotlib.pyplot as plt\n\n#PCA plot\nplt.figure(figsize=(15,10))\nsns.scatterplot(x=pca_result[:,0],y=pca_result[:,1],hue=target_vectors[:2000],style=target_vectors[:2000])\nplt.legend(loc='best')\nplt.title(\"PCA plot of train corpus\\n\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del pca_result\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Truncated SVD"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n##SVD\ntsvd= TruncatedSVD(n_components=2,algorithm=\"randomized\",random_state=14)\ntsvd_result=tsvd.fit_transform(train_data_cv[:5000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,12))\nsns.scatterplot(x=tsvd_result[:,0],y=tsvd_result[:,1],hue=target_vectors[:5000],style=target_vectors[:5000])\nplt.title(\"Truncated SVD Plot of train Corpus\\n\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### T-SNE"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n##TSNE\ntsne=TSNE(n_components=2,random_state=14)\ntsne_result=tsne.fit_transform(train_data_cv[:5000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#TSNE plot\nplt.figure(figsize=(15,10))\nsns.scatterplot(x=tsne_result[:,0],y=tsne_result[:,1],hue=target_vectors[:5000],style=target_vectors[:5000])\nplt.title(\"T-SNE Plot of train Corpus\\n\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del tsvd_result,tsne_result\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://media.giphy.com/media/VGVwLultLZjrrssAak/giphy.gif\" align=\"left\">"},{"metadata":{},"cell_type":"markdown","source":"# Semantic Embeddings\nSemantic embeddings include word embeddings which can either by ***static and dynamic***.\n\n- When words converted to vectors, several metrics can be applied like finding similarity, distance measurement between the vectors,etc\n\n- With word vectors, I am showing the semantic similarity between different words or collection of words\n\nPrimarily I am starting with Word2Vec algorithm."},{"metadata":{},"cell_type":"markdown","source":"## Static Word Embeddings\n\nThese embeddings are pre-trained on large corpuses like Wikipedia, News corpuses.etc. However, the base of these algorithms rely on 2 important techniques:\n\n1) ***CBOW approach*** tries to predict single correct target word given the context of words/phrases.\n\n2) ***The Skipgram approach*** tries to predict contextual words/phrases given a single target word"},{"metadata":{"trusted":true},"cell_type":"code","source":"import gensim\n\n#Convert Input DataFrame to a List\ncheck_df=list(new_train_data['question_text'].str.split())\n\n#Load word2vec algorithm from gensim\nfrom gensim.models import Word2Vec,KeyedVectors\n\nmodel=Word2Vec(check_df,min_count=1)\nword_list=list(model.wv.vocab)\nprint(word_list[:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize the word Vector\n\nLet's visualise the word \"best\" vector projection as well as its corresponding tensor."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#View the Tensor 'best'\nprint(model.wv['best'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#View the Embedding Word Vector\nplt.figure(figsize=(20,5))\nplt.plot(model.wv['best'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Save the modeled words produced from Word2Vec\nmodel.save('word2vec_model.bin')\nloaded_model=KeyedVectors.load('word2vec_model.bin')\nprint(loaded_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ## Dimension Reduction of the embedding vectors\n\n\nLet us try to visualize the compressed and decomposed embedding space based on first 200 entries in the dataset. When I plot using the Word2Vec -Skipgram/CBOW model, the relative positioning of the words close to each other could be found. \n\n\n***Cosine Distance*** measurement is a metric that determines closeness of any 2 word vectors, when plotted using matplotlib."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Measure Cosine distance\ndistance=model.wv.similarity('best','good')\nprint(distance)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del loaded_model\nimport gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Words 'best' and 'good' are placed close to each other as they are similar in meaning with respect to the context given.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#PCA transform in 2D for visualization of embedded words\n\nfrom matplotlib import pyplot\nplt.figure(figsize=(30,14))\npca = PCA(n_components=2)\ntransformation_model= model.wv[model.wv.vocab]\nresult = pca.fit_transform(transformation_model[:150])\n# create a scatter plot of the projection\npyplot.scatter(result[:, 0], result[:, 1])\nwords = list(model.wv.vocab)\nfor i, word in enumerate(words[:150]):\n    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))\n    plt.title(\"PCA Transformation of Embedded Words on to 2D-space\\n\")\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del model\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> # Variants of Word2Vec algorithms\n\nThese variants include:\n\n(1) FastText\n\n(2) Glove\n\n(3) Google News Vectors\n\nAll of these are traditional static embeddings . However they are very powerful on their own terms. These can be used with any neural network or classifier model with correct activations to suit our purpose. \n\n* Word2Vec is the first (State-of-the-Art) SOTA model which relies on Softmax and probabilistic log likelihood of softmax to generate the predicted outputs. \n* Word2vec has laid the foundation of all the algorithms which we saw here as well as variants in Node2Vec, Doc2Vec etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n#Using Google News Embeddings for the corpus\nfrom gensim.models import Word2Vec\n\n#download Google News Embeddings\ngoogle_news_embed=\"https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\"\n\ngoogle_loaded_model=KeyedVectors.load_word2vec_format(google_news_embed,binary=True)\nprint(google_loaded_model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Visualize the Word Vectors\n##Comparing the similarities of words graphically\n\nplt.figure(figsize=(30,10))\nplt.plot(google_loaded_model['best'])\nplt.plot(google_loaded_model['good'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# PCA transform in 2D for visualization of google news embedded words\nfrom matplotlib import pyplot\n\nplt.figure(figsize=(20,12))\npca = PCA(n_components=2)\ntransformation_model = google_loaded_model[google_loaded_model.vocab]\nresult = pca.fit_transform(transformation_model[:100])\n# create a scatter plot of the projection\npyplot.scatter(result[:, 0], result[:, 1])\nwords = list(google_loaded_model.wv.vocab)\nfor i, word in enumerate(words[:100]):\n    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))\n    plt.title(\"PCA Transformation of word vectors with Google News Embeddings on to 2D-space\\n\")\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del google_loaded_model\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Glove Embeddings\n\n\n- Glove embeddings rely on global vector representations mechanism, which is an unsupervised algorithm. \n\n- This captures both the global corpus statistics as well as local semantic information. \n\n- Glove vectors used here can be converted from \"txt\" format to Word2Vec format by using scripts provided in the Gensim library.\n\n- This allows us to manipulate the glove embeddings in a manner similar to Word2Vec and apply the similarity metric.\n\n- The loss function for the glove relies on logistic regression of the log co-occurence probabilities."},{"metadata":{"trusted":true},"cell_type":"code","source":"from gensim.scripts.glove2word2vec import glove2word2vec\nfrom gensim.test.utils import datapath, get_tmpfile\nfrom gensim.models import KeyedVectors\n\nglove_file = '../input/glovedata/glove.6B.100d.txt'\nword2vec_output_file = 'glove.6B.50d.txt.word2vec'\n\nglove_loaded=glove2word2vec(glove_file, word2vec_output_file)\nprint(glove_loaded)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Comparing the similarities of words graphically\nplt.figure(figsize=(20,8))\nglove_model = KeyedVectors.load_word2vec_format(word2vec_output_file, binary=False)\nplt.plot(glove_model['best'])\nplt.plot(glove_model['good'])\nplt.title(\"Visualize the Word Vectors with GloVe Embeddings\\n\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#PCA transform in 2D for visualization of glove embedded words\nfrom matplotlib import pyplot\n\nplt.figure(figsize=(30,15))\npca = PCA(n_components=2)\ntransformation_model = glove_model[glove_model.vocab]\nresult = pca.fit_transform(transformation_model[:100])\n# create a scatter plot of the projection\npyplot.scatter(result[:, 0], result[:, 1])\nwords = list(loaded_model.wv.vocab)\nfor i, word in enumerate(words[:100]):\n    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))\n    plt.title(\"PCA Transformation of word vectors with GloVe Embeddings on to 2D-space\\n\")\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fastext embeddings"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Using the fasttext word embeddding from crawl\n\nfrom gensim.models import Word2Vec,KeyedVectors\nfasttext_file=\"../input/fasttext-crawl-300d-2m/crawl-300d-2M.vec\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##I am not running this algo becuase time consumed is more than 20 min and memory also exhausted.\n\n#Comparing the similarities of words graphically\nplt.figure(figsize=(20,8))\nfasttext_model = KeyedVectors.load_word2vec_format(fasttext_file, binary=False)\nplt.plot(fasttext_model['best'])\nplt.plot(fasttext_model['like'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#PCA transform in 2D for visualization of glove embedded words\n\nfrom matplotlib import pyplot\npca = PCA(n_components=2)\ntransformation_model=fasttext_model[fasttext_model.vocab]\nresult = pca.fit_transform(transformation_model[:50])\n\nplt.figure(figsize=(20,12))\n#create a scatter plot of the projection\npyplot.scatter(result[:, 0], result[:, 1])\nwords = list(fasttext_model.vocab)\nfor i, word in enumerate(words[:50]):\n    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))\n    plt.title('PCA Transformation of word vectors with Fasttext Embeddings on to 2D-space\\n')\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Procedure to create Embedding Matrix\n\n- The importance of building a Embedding matrix is to have mutual co-occurence embedding probabilities (vectors) of all the words present in the corpus. This is the format in which neural network frameworks like Keras, Tensorflow uses. \n\n- These embedding vectors are then passed through deep learning layers. \n\n- The flexibility of these static embeddings is that ,they provide a good benchmark on the initial task (classification) in our case.\n\n- These also provide a very good approximation to the amount of percentage accuracy or loss which can be achieved with a coupled Encoder-Decoder/Transformer like architectures."},{"metadata":{},"cell_type":"markdown","source":"> ### End of Embeddings\n\nThe Word2Vec and its variant algorithms are very powerful for initial benchmarking for any classification tasks. These old SOTA models are hence very useful in NLP.\n\n### [Visit Next Part](https://www.kaggle.com/rizdelhi/end-to-end-natural-language-processing-2)\n\nI am building the statistical Machine Learning models in this notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image\nimport urllib\nimport requests\n\nImage.open('../input/mylion/hard_work.jpeg')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}