{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"data = pd.read_csv('../input/train.csv',usecols=['title','description'])","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"91fca3440362d92c839e91951b92d6a309059678"},"cell_type":"markdown","source":"![Word2Vec and FastText Word Embedding with Gensim](https://cdn-images-1.medium.com/max/2000/1*r_R38HJ9NkbPyb0Y7PkHRw.png)"},{"metadata":{"trusted":true,"_uuid":"6fbd488f5816166166e17f7a44a23d14f75fc21d"},"cell_type":"code","source":"data.head()","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"6107cf58fa315bcd08e8f23878720958727c552c"},"cell_type":"markdown","source":"## Word2Vec\nWord2Vec is an efficient solution to these problems, which leverages the context of the target words. Essentially, we want to use the surrounding words to represent the target words with a Neural Network whose hidden layer encodes the word representation.\n\nThere are two types of Word2Vec, Skip-gram and Continuous Bag of Words (CBOW). I will briefly describe how these two methods work in the following paragraphs.\n\n### Skip-gram\nFor skip-gram, the input is the target word, while the outputs are the words surrounding the target words. For instance, in the sentence “I have a cute dog”, the input would be “a”, whereas the output is “I”, “have”, “cute”, and “dog”, assuming the window size is 5. All the input and output data are of the same dimension and one-hot encoded. The network contains 1 hidden layer whose dimension is equal to the embedding size, which is smaller than the input/ output vector size. At the end of the output layer, a softmax activation function is applied so that each element of the output vector describes how likely a specific word will appear in the context. The graph below visualizes the network structure.\n\n![Skip-gram](https://cdn-images-1.medium.com/max/1600/1*TbjQNQLuyEW-cgsofyDioQ.png)\n\nThe word embedding for the target words can obtained by extracting the hidden layers after feeding the one-hot representation of that word into the network.\n\nWith skip-gram, the representation dimension decreases from the vocabulary size (V) to the length of the hidden layer (N). Furthermore, the vectors are more “meaningful” in terms of describing the relationship between words. The vectors obtained by subtracting two related words sometimes express a meaningful concept such as gender or verb tense, as shown in the following figure (dimensionality reduced).\n\n![Visualize Word Vectors](https://cdn-images-1.medium.com/max/1600/1*jpnKO5X0Ii8PVdQYFO2z1Q.png)\n\n### CBOW\nContinuous Bag of Words (CBOW) is very similar to skip-gram, except that it swaps the input and output. The idea is that given a context, we want to know which word is most likely to appear in it.\n\n![CBOW](https://cdn-images-1.medium.com/max/1600/1*UdLFo8hgsX0a1NKKuf_n9Q.png)\n\nThe biggest difference between Skip-gram and CBOW is that the way the word vectors are generated. For CBOW, all the examples with the target word as target are fed into the networks, and taking the average of the extracted hidden layer. For example, assume we only have two sentences, “He is a nice guy” and “She is a wise queen”. To compute the word representation for the word “a”, we need to feed in these two examples, “He is nice guy”, and “She is wise queen” into the Neural Network and take the average of the value in the hidden layer. Skip-gram only feed in the one and only one target word one-hot vector as input.\n\nIt is claimed that Skip-gram tends to do better in rare words. Nevertheless, the performance of Skip-gram and CBOW are generally similar."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"18268b0de4f476474baffb989ff8179e29755474"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport re\n\nimport pickle \n#import mglearn\nimport time\n\n\nfrom nltk.tokenize import TweetTokenizer # doesn't split at apostrophes\nimport nltk\nfrom nltk import Text\nfrom nltk.tokenize import regexp_tokenize\nfrom nltk.tokenize import word_tokenize  \nfrom nltk.tokenize import sent_tokenize \nfrom nltk.corpus import stopwords\nfrom nltk.stem import WordNetLemmatizer\nfrom nltk.stem import PorterStemmer\n\n\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.feature_extraction.text import TfidfVectorizer","execution_count":10,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca81e4c81af6ea2c9044443138225da89ed7adec"},"cell_type":"code","source":"data.description.values[:10]","execution_count":13,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"8e78d2987b0c48d8e0e8334968e36dc5cd1b68e3"},"cell_type":"code","source":"txt = ['Кокон для сна малыша,пользовались меньше месяца.цвет серый',\n       'Стойка для одежды, под вешалки. С бутика.',\n       'В хорошем состоянии, домашний кинотеатр с blu ray, USB. Если настроить, то работает смарт тв /\\nТорг',\n       'Продам кресло от0-25кг', 'Все вопросы по телефону.',\n       'В хорошем состоянии',\n       'Электро водонагреватель накопительный на 100 литров Термекс ID 100V, плоский, внутренний бак из нержавейки, 2 кВт, б/у 2 недели, на гарантии.',\n       'Бойфренды в хорошем состоянии.', '54 раз мер очень удобное',\n       'По стельке 15.5см мерить приокский район. Цвет темнее чем на фото']","execution_count":14,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b05b15ac56827efe432a2c599cea1938d05c3e4"},"cell_type":"code","source":"# Initialize a CountVectorizer object: count_vectorizer\ncount_vec = CountVectorizer(stop_words=stopwords.words('russian'), analyzer='word', encoding='KOI8-R',\n                            ngram_range=(1, 1), max_df=1.0, min_df=1, max_features=None)\n\n# Transforms the data into a bag of words\ncount_train = count_vec.fit(txt)\nbag_of_words = count_vec.transform(txt)\n\n# Print the first 10 features of the count_vec\nprint(\"Every feature:\\n{}\".format(count_vec.get_feature_names()))\nprint(\"\\nEvery 3rd feature:\\n{}\".format(count_vec.get_feature_names()[::3]))","execution_count":17,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10982ab6b77a7d3ab4d75bfc04df803f70f91332"},"cell_type":"code","source":"print(\"Vocabulary size: {}\".format(len(count_train.vocabulary_)))\nprint(\"Vocabulary content:\\n {}\".format(count_train.vocabulary_))","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"b437d9cbb3fe55e50538e10f5358e46d66eb5d5d"},"cell_type":"markdown","source":"### N-grams (sets of consecutive words)\n**N==2**"},{"metadata":{"trusted":true,"_uuid":"99bc833171e1395be612c4fd864f0ad5140f4591"},"cell_type":"code","source":"count_vec = CountVectorizer(stop_words=stopwords.words('russian'), analyzer='word', encoding='KOI8-R',\n                            ngram_range=(1, 2), max_df=1.0, min_df=1, max_features=None)\n\ncount_train = count_vec.fit(txt)\nbag_of_words = count_vec.transform(txt)\n\nprint(count_vec.get_feature_names())","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"3c2e81b4a6e9cd02ba42a37ae3abdccba68c1bf3"},"cell_type":"markdown","source":"**N==3**"},{"metadata":{"trusted":true,"_uuid":"a7114df7a6faae1649be8f7101dd86a7e96553ce"},"cell_type":"code","source":"count_vec = CountVectorizer(stop_words=stopwords.words('russian'), analyzer='word', encoding='KOI8-R',\n                            ngram_range=(1, 3), max_df=1.0, min_df=1, max_features=None)\n\ncount_train = count_vec.fit(txt)\nbag_of_words = count_vec.transform(txt)\n\nprint(count_vec.get_feature_names())","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"47d9274e535ae09374cd903e1c2ef268df7e1251"},"cell_type":"markdown","source":"### Min_df\n\n**Min_df ignores terms that have a document frequency (presence in % of documents) strictly lower than the given threshold. For example, Min_df=0.66 requires that a term appear in 66% of the docuemnts for it to be considered part of the vocabulary.**\n\nSometimes min_df is used to limit the vocabulary size, so it learns only those terms that appear in at least 10%, 20%, etc. of the documents."},{"metadata":{"trusted":true,"_uuid":"15650116b75157d4f21f275a0cc24f583d3b288d"},"cell_type":"code","source":"count_vec = CountVectorizer(stop_words=stopwords.words('russian'), analyzer='word', encoding='KOI8-R',\n                            ngram_range=(1, 1), max_df=1.0, min_df=0.3, max_features=None)\n\ncount_train = count_vec.fit(txt)\nbag_of_words = count_vec.transform(txt)\n\nprint(count_vec.get_feature_names())\nprint(\"\\nOnly 'park' becomes the vocabulary of the document term matrix (dtm) because it appears in 2 out of 3 documents, \\\nmeaning 0.66% of the time.\\\n      \\nThe rest of the words such as 'big' appear only in 1 out of 3 documents, meaning 0.33%. which is why they don't appear\")","execution_count":24,"outputs":[]},{"metadata":{"_uuid":"d55b65f414590ca990fdaa978c088cc35154668a"},"cell_type":"markdown","source":"### Max_df\nWhen building the vocabulary, it ignores terms that have a document frequency strictly higher than the given threshold. This could be used to exclude terms that are too frequent and are unlikely to help predict the label. For example, by analyzing reviews on the movie Lion King, the term 'Lion' might appear in 90% of the reviews (documents), in which case, we could consider establishing Max_df=0.89"},{"metadata":{"trusted":true,"_uuid":"e470d59a2b02474d3eb6b965c64092ba60954806"},"cell_type":"code","source":"count_vec = CountVectorizer(stop_words=stopwords.words('russian'), analyzer='word', encoding='KOI8-R',\n                            ngram_range=(1, 1), max_df=0.50, min_df=1, max_features=None)\n\ncount_train = count_vec.fit(txt)\nbag_of_words = count_vec.transform(txt)\n\nprint(count_vec.get_feature_names())\nprint(\"\\nOnly 'park' is ignored because it appears in 2 out of 3 documents, meaning 0.66% of the time.\")","execution_count":25,"outputs":[]},{"metadata":{"_uuid":"a2f36007251a5fe067ecc7fe1aafac53df3fb260"},"cell_type":"markdown","source":"### Max_features\nLimit the amount of features (vocabulary) that the vectorizer will learn"},{"metadata":{"trusted":true,"_uuid":"1588334d135088595cb5524413a95b5aa116e30b"},"cell_type":"code","source":"count_vec = CountVectorizer(stop_words=stopwords.words('russian'), analyzer='word', encoding='KOI8-R',\n                            ngram_range=(1, 1), max_df=1.0, min_df=1, max_features=4)\n\ncount_train = count_vec.fit(txt)\nbag_of_words = count_vec.transform(txt)\n\nprint(count_vec.get_feature_names())","execution_count":26,"outputs":[]},{"metadata":{"_uuid":"8e9148f73669f67b6b1bd75a1cc02938a8f4ae16"},"cell_type":"markdown","source":"### TfidfVectorizer -- Brief Tutorial\nThe goal of using tf-idf is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus. (https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/feature_extraction/text.py#L1365)"},{"metadata":{"trusted":true,"_uuid":"81be3814b530f746cadf9defb51c1be95dc63d3c"},"cell_type":"code","source":"data.description.values[10:30]","execution_count":27,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cced5ab402cff8b7796015a9afa443f757bca00a"},"cell_type":"code","source":"txt1 = ['Семейная пара из двух человек снимет коттедж с дизайнерским ремонтом, со стильной мебелью и бытовой техникой. Гараж на 2 машины, 2 санузла. Огороженная территория, сигнализация.  Рассмотрим варианты коттеджей, которые выставлены на продажу. Ежемесячная оплата до 100 тыс.руб. в месяц.  Агентства просьба не беспокоить.',\n       'Дом находиться внутри квартала./\\nПластиковые окна во всей квартире, балкон застеклен железом, обшит деревом, в комнате натяжной потолок, новый линолиум,/\\nванна и туалет- стеновые панели.Квартира подходит под ипотеку, 1 собственник, ЧП, небольшой торг возможен.']\ntf = TfidfVectorizer(smooth_idf=False, sublinear_tf=False, norm=None, analyzer='word',\n                     stop_words=stopwords.words('russian'), encoding='KOI8-R')\ntxt_fitted = tf.fit(txt1)\ntxt_transformed = txt_fitted.transform(txt1)\nprint (\"The text: \", txt1)\nprint (\"The txt_transformed: \", txt_transformed)","execution_count":31,"outputs":[]},{"metadata":{"_uuid":"4b06be4dbd4bc9486af03cb4f95717f894d42032"},"cell_type":"markdown","source":"The learned corpus vocabulary"},{"metadata":{"trusted":true,"_uuid":"cce119eb0fc204b49b6fb8eab9ff5db8dd1b3c74"},"cell_type":"code","source":"tf.vocabulary_","execution_count":32,"outputs":[]},{"metadata":{"_uuid":"a6762c76e4bb04abe57009a8b2127127c6043c36"},"cell_type":"markdown","source":"IDF: The inverse document frequency"},{"metadata":{"trusted":true,"_uuid":"7aae3bed68d0a45ec4996c032d3aa9e04a04e089"},"cell_type":"code","source":"idf = tf.idf_\nprint(dict(zip(txt_fitted.get_feature_names(), idf)))\nprint(\"\\nWe see that the tokens 'sang','she' have the most idf weight because \\\nthey are the only tokens that appear in one document only.\")\nprint(\"\\nThe token 'not' appears 6 times but it is also in all documents, so its idf is the lowest\")","execution_count":33,"outputs":[]},{"metadata":{"_uuid":"2dcd4fca34480737544b596bfeb3a3dc1f896929"},"cell_type":"markdown","source":"Graphing inverse document frequency"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"489c83a89ad09c9c8ac4eba0252f79b9cb157e45"},"cell_type":"code","source":"rr = dict(zip(txt_fitted.get_feature_names(), idf))","execution_count":34,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15c91456cf42cd44a6559c0d46131e71e550d8de"},"cell_type":"code","source":"token_weight = pd.DataFrame.from_dict(rr, orient='index').reset_index()\ntoken_weight.columns=('token','weight')\ntoken_weight = token_weight.sort_values(by='weight', ascending=False)[:10]\ntoken_weight\n\nsns.barplot(x='token', y='weight', data=token_weight)\nplt.title(\"Inverse Document Frequency(idf) per token\")\nfig=plt.gcf()\nfig.set_size_inches(10,5)\nfig.set_figwidth(10)\nplt.show()","execution_count":39,"outputs":[]},{"metadata":{"_uuid":"55b9d76cf8eee4ec94d7a540c29f55ed51a90ae4"},"cell_type":"markdown","source":"Listing (instead of graphing) inverse document frequency"},{"metadata":{"trusted":true,"_uuid":"d182cc48446fb917ce31c4010994e8be256b2e21"},"cell_type":"code","source":"# get feature names\nfeature_names = np.array(tf.get_feature_names())\nsorted_by_idf = np.argsort(tf.idf_)\nprint(\"Features with lowest idf:\\n{}\".format(\n       feature_names[sorted_by_idf[:3]]))\nprint(\"\\nFeatures with highest idf:\\n{}\".format(\n       feature_names[sorted_by_idf[-3:]]))","execution_count":40,"outputs":[]},{"metadata":{"_uuid":"cefd39c2fc7f60de755166939be7f73c9ec3fde3"},"cell_type":"markdown","source":"Weight of tokens per document"},{"metadata":{"trusted":true,"_uuid":"f4ba68e7b3b1d7149db653b165bf77bbc597e752"},"cell_type":"code","source":"data.description.values[132]","execution_count":47,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"df2d72e93c058b7e6a520c295d43a9d300350514"},"cell_type":"code","source":"print(\"The token 'not' has  the largest weight in document #2 because it appears 3 times there. But in document #1\\\n its weight is 0 because it does not appear there.\")\ntxt_transformed.toarray()","execution_count":48,"outputs":[]},{"metadata":{"_uuid":"2224b7c12c75b04d7cbc27d238da5723795d78be"},"cell_type":"markdown","source":"TF-IDF - Maximum token value throughout the whole dataset"},{"metadata":{"trusted":true,"_uuid":"7acc041cc02131cbb22a61c93886dc5dde891c71"},"cell_type":"code","source":"new1 = tf.transform(txt1)\n\n# find maximum value for each of the features over all of dataset:\nmax_val = new1.max(axis=0).toarray().ravel()\n\n#sort weights from smallest to biggest and extract their indices \nsort_by_tfidf = max_val.argsort()\n\nprint(\"Features with lowest tfidf:\\n{}\".format(\n      feature_names[sort_by_tfidf[:3]]))\n\nprint(\"\\nFeatures with highest tfidf: \\n{}\".format(\n      feature_names[sort_by_tfidf[-3:]]))","execution_count":49,"outputs":[]},{"metadata":{"_uuid":"ed54776a9acd517be8a2f2eef926cdf1ee79f751"},"cell_type":"markdown","source":"MORE IS COMING..."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}