{"cells":[{"metadata":{"_uuid":"01f361ddc47e0b386595316fe3d7f4dabbd260db"},"cell_type":"markdown","source":" # Using Word Embeddings for Data Augmentation\n\n###  In this kernel, I'm going to show you a way to do data augmentation for texts, when you have word embeddings.\n\nI will focus on augmenting texts labelled as 1, as this class is under-represented. Oversampling can help improve perfomances.\n\n\n#### Feel free to give any feedback, it is always appreciated.\n\n##### References :\n\nInspired by this kernel :\n> https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings#  by SRK\n\nContinuation of :\n> https://www.kaggle.com/theoviel/dealing-with-class-imbalance-with-smote"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom collections import Counter\nfrom wordcloud import WordCloud\n\nnp.random.seed(100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1ab2f6912fca0d9ff1f9682da249d30eaf9a12d"},"cell_type":"markdown","source":"## Step 1 : Loading Word Embeddings"},{"metadata":{"trusted":true,"_uuid":"f6af5a1deac4badb38248d0b407a47e549d08404"},"cell_type":"code","source":"def get_coefs(word,*arr): \n    return word, np.asarray(arr, dtype='float32')\n\ndef load_embedding(file):\n    if file == '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec':\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file) if len(o)>100)\n    else:\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file, encoding='latin'))\n    return embeddings_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d1c8d2172c5c0d4807b45cb1a8518ee44ee62dd"},"cell_type":"code","source":"def make_embedding_matrix(embedding, tokenizer, len_voc):\n    all_embs = np.stack(embedding.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n    word_index = tokenizer.word_index\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (len_voc, embed_size))\n    \n    for word, i in word_index.items():\n        if i >= len_voc:\n            continue\n        embedding_vector = embedding.get(word)\n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n    \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a1d10b0b04c85cac8fb1f0d3f85a66eaf016b73"},"cell_type":"code","source":"glove = load_embedding('../input/embeddings/glove.840B.300d/glove.840B.300d.txt')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63530632439404a85540565eb31c6390bddb33e9"},"cell_type":"markdown","source":"I'm using GloVe, because that's the embedding I got the best results with inbefore, but paragram is quite good as well.\n\n## Step 2 : Loading Data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.csv\")\nprint(\"Number of texts: \", df.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96012bbe09075239c432cf48a204ca2a35f85c16"},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1eccf4f87c29c6216d3d6d4f971bffea1641e44"},"cell_type":"markdown","source":"### Class imbalance"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"d7d54c0edc5228654daa5d1bbb3f7e17b509249c"},"cell_type":"code","source":"plt.figure(figsize = (10, 8))\nsns.countplot(df['target'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dc7a4b01b8118224a6346527007a786fa4d24e4"},"cell_type":"code","source":"print(\"Class repartition : \", (Counter(df['target'])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e42542a1bb153dd51f845a76d82b7fe29e78ab8"},"cell_type":"markdown","source":"There is way more 0s than 1s in our dataset. As mentionned, we could use data augmentation to balance classes. Therefore the prediction task will be easier."},{"metadata":{"_uuid":"b7c5a37c7151305c44764243c4e097b135b1afc2"},"cell_type":"markdown","source":"## Step 3: Tokenizing\n\nI am using Keras' Tokenizer to apply some text processing and to limit the size of the vocabulary"},{"metadata":{"trusted":true,"_uuid":"07b3163ffb71e5f94cd67075028dd75dac24cc3b"},"cell_type":"code","source":"len_voc = 100000","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87fc62fa686f54ac3531eddfadedb639d15c0ffd"},"cell_type":"markdown","source":"#### Tokenizing"},{"metadata":{"trusted":true,"_uuid":"363b9ece53b8e1800d03b188a50ddf84d0cb12a5"},"cell_type":"code","source":"def make_tokenizer(texts, len_voc):\n    from keras.preprocessing.text import Tokenizer\n    t = Tokenizer(num_words=len_voc)\n    t.fit_on_texts(texts)\n    return t","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"017eb151a1d1e927a491879bf068fabd0a7edc4d"},"cell_type":"code","source":"tokenizer = make_tokenizer(df['question_text'], len_voc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35dbfe75cd164f9623acce773ed217533a848414"},"cell_type":"code","source":"X = tokenizer.texts_to_sequences(df['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4bd88c36636ee394154376b59070afb1fb4d47ef"},"cell_type":"markdown","source":"I also apply padding, mostly to store X as an array."},{"metadata":{"trusted":true,"_uuid":"beb2ac9d36c9560261d52addfa3ea2bc49f19086"},"cell_type":"code","source":"from keras.preprocessing.sequence import pad_sequences\nX = pad_sequences(X, 70)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fca19348c54665f79bfcbecc981206280b8715da"},"cell_type":"code","source":"y = df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fe26fcf10d19757e572411da2a88591789e9ae0"},"cell_type":"markdown","source":"For visualization, I'm gonna need to see which index corresponds to which word"},{"metadata":{"trusted":true,"_uuid":"521babdf9ba5b1985609894f0230faa9fd156bb8"},"cell_type":"code","source":"index_word = {0: ''}\nfor word in tokenizer.word_index.keys():\n    index_word[tokenizer.word_index[word]] = word","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fce47b803d2f5b9261cc5c4720ff502a1a3f9f8a"},"cell_type":"markdown","source":"#### Embedding Matrix"},{"metadata":{"trusted":true,"_uuid":"be89625f116b26209f996d1b7ad1eb5d2ed8269f"},"cell_type":"code","source":"embed_mat = make_embedding_matrix(glove, tokenizer, len_voc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a0f98cd5cd6dcd656673e7fee48f3bc514cdf9f"},"cell_type":"markdown","source":"## Step 3 : Making a Synonym Dictionary\n\nWord vectors are made in a way that similar words have similar representation. Therefore we can use the $k$-nearest neighbours to get $k$ synonyms.\n\nAs the process takes a bit of time, I chose to compute 5 synonyms for the 20000 most frequent words."},{"metadata":{"trusted":true,"_uuid":"3effb5c72b4cecfe756d167d16f095dc548dff7e"},"cell_type":"code","source":"from sklearn.neighbors import NearestNeighbors\n\nsynonyms_number = 5\nword_number = 20000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64a13c6fd58e00a17e0c9f6a517c5b1e7ba43422"},"cell_type":"code","source":"nn = NearestNeighbors(n_neighbors=synonyms_number+1).fit(embed_mat) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f93d843f96f9e649afce4c7d65619f8427220f3"},"cell_type":"markdown","source":"#### We create Synonyms for the most frequent words"},{"metadata":{"trusted":true,"_uuid":"f60fae18a8960ae63d58618a2a836002ea89d80d"},"cell_type":"code","source":"neighbours_mat = nn.kneighbors(embed_mat[1:word_number])[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86016b8afe8a95ef8a69c875166f94bf1224e46c"},"cell_type":"code","source":"synonyms = {x[0]: x[1:] for x in neighbours_mat}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf85a5b3f6c78a0c7c630f6b60eb876987d1c3fb"},"cell_type":"markdown","source":"### Checking our synonyms"},{"metadata":{"trusted":true,"_uuid":"9ce5990f0c783a1462be7519e2064765e203f7be"},"cell_type":"code","source":"for x in np.random.randint(1, word_number, 10):\n    print(f\"{index_word[x]} : {[index_word[synonyms[x][i]] for i in range(synonyms_number-1)]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"265b273934f57e794346599a83100117969c4479"},"cell_type":"code","source":"index = np.random.randint(1, word_number, 9)\nplt.figure(figsize=(20,10))\n\nfor k in range(len(index)):\n    plt.subplot(3, 3, k+1)\n    \n    x = index[k]\n    text = ' '.join([index_word[x]] + [index_word[synonyms[x][i]] for i in range(synonyms_number-1)]) \n    wordcloud = WordCloud(stopwords=[]).generate((text))\n    plt.imshow(wordcloud, interpolation='bilinear')\n    plt.axis(\"off\")\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b76b514bd9ff9c46e2e9df92ba4dc8e9e31dac94"},"cell_type":"markdown","source":"#### Looks pretty good ! "},{"metadata":{"_uuid":"46e8882712c42236e854b6017933ca423381b712"},"cell_type":"markdown","source":"## Step 4 - Data Augmentation / Oversampling \n\n#### We work on 1 labelled texts. We apply the following algorithm to modify a sentence :\n\nFor each word in the sentence :\n* Keep it with probability $p$  (or if we don't have synonyms for it)\n* Randomly swap it with one of its synonyms with probability $1-p$"},{"metadata":{"trusted":true,"_uuid":"ba8fd07e8fb6f9d5a773eab51f41f2e4d9e40c58"},"cell_type":"code","source":"X_pos = X[y==1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"279f7e40ed61a210028e75bc3ec1efb8b427115e"},"cell_type":"code","source":"def modify_sentence(sentence, synonyms, p=0.5):\n    for i in range(len(sentence)):\n        if np.random.random() > p:\n            try:\n                syns = synonyms[sentence[i]]\n                sentence[i] = np.random.choice(syns)\n            except KeyError:\n                pass\n    return sentence","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"687d60db0ee84c897f3ed889e3f7f37ac40cf0e7"},"cell_type":"markdown","source":"### Let us preview our function"},{"metadata":{"trusted":true,"_uuid":"ad41e6e98a2359235d02c43337b51dca3faeedeb"},"cell_type":"code","source":"indexes = np.random.randint(0, X_pos.shape[0], 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c73c2fc1667d96e0a83e8cc1ecfaee4375f4d35e"},"cell_type":"code","source":"for x in X_pos[indexes]:\n    sample =  np.trim_zeros(x)\n    sentence = ' '.join([index_word[x] for x in sample])\n    print(sentence)\n\n    modified = modify_sentence(sample, synonyms)\n    sentence_m = ' '.join([index_word[x] for x in modified])\n    print(sentence_m)\n    \n    print(' ')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2bf85a0b0d56c8f1fe0a357862a6646a2edf9c44"},"cell_type":"markdown","source":"Looks pretty good, we now generate some texts"},{"metadata":{"trusted":true,"_uuid":"3d7c41a4457c8040209372d8249d6a7604b77b84"},"cell_type":"code","source":"n_texts = 30000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cbcfd4be68aeaddaa6f329f7cc372dc3a8e26f67"},"cell_type":"code","source":"indexes = np.random.randint(0, X_pos.shape[0], n_texts)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f78366a4183e704b35d7416e32aa5712afeb2a3"},"cell_type":"code","source":"X_gen = np.array([modify_sentence(x, synonyms) for x in X_pos[indexes]])\ny_gen = np.ones(n_texts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50698a4b9b2cf21d6370d8e88d90d265fb3af1fc"},"cell_type":"markdown","source":"\n \n #### The next work to do is to find a good value for $p$, and a correct number of samples to generate, and then feed it into a network.\n \n  ## Thanks for reading, hope it can be helpful to anyone !\n\n "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}