{"cells":[{"metadata":{"_uuid":"01f361ddc47e0b386595316fe3d7f4dabbd260db"},"cell_type":"markdown","source":" # Dealing with Class Imbalance with SMOTE\n\n### In this kernel, I will use a simple Deep Learning model and compare its performance on normal data and data augmented with SMOTE\n\n> Check https://arxiv.org/pdf/1106.1813.pdf\n\nI use SMOTE to add **sentence level** noise to our data.\n\n#### The model is the following one :\n* GloVe Embedding\n* Bidirectional GRU\n* MaxPool\n* Dense \n* Probably some Dropouts\n\n\n#### Feel free to give any feedback, it is always appreciated."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport keras\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom time import time\nfrom collections import Counter","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2837c162d484eca77c55913e8e1106643f33db34"},"cell_type":"markdown","source":"## How does SMOTE work ?\n\n> \" The minority class is over-sampled by taking each minority class sample and introducing synthetic examples along the line segments joining any/all of the k minority class nearest neighbors \"\n\n> \" Synthetic samples are generated in the following way: Take the difference between the feature vector (sample) under consideration and its nearest neighbor. Multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This causes the selection of a random point along the line segment between two specific features. This approach effectively forces the decision region of the minority class to become more general. \"\n\nI am using the class from imblearn,  see https://imbalanced-learn.org/en/stable/generated/imblearn.over_sampling.SMOTE.html"},{"metadata":{"trusted":true,"_uuid":"9342385261f03ece86a8aea59cb24c2e59387d30"},"cell_type":"code","source":"from imblearn.over_sampling import SMOTE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cad772858c76e97b7e590088a99ab9be47ea7d7c"},"cell_type":"code","source":"from sklearn.datasets import make_classification\nX, y = make_classification(n_samples=100, n_features=2, n_informative=2, n_redundant=0, n_classes=2, weights=[0.2, 0.8], class_sep=0.95, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bba6b36c5d42c0caef6771bda9f7afa2370050da"},"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nplt.title('Repartition before SMOTE')\nplt.scatter(X[y==1][:, 0], X[y==1][:, 1], label='class 1')\nplt.scatter(X[y==0][:, 0], X[y==0][:, 1], label='class 0')\nplt.legend()\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4e7f65cd8ec7394585f5a6d87a5e7418b87b5a6"},"cell_type":"code","source":"smt = SMOTE()\nX_smote, y_smote = smt.fit_resample(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3543fc2e288975b0592a14875c987b2ffc4d953"},"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nplt.title('Repartition after SMOTE')\nplt.scatter(X_smote[y_smote==1][:, 0], X_smote[y_smote==1][:, 1], label='class 1')\nplt.scatter(X_smote[y_smote==0][:, 0], X_smote[y_smote==0][:, 1], label='class 0')\nplt.legend()\nplt.grid(False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63530632439404a85540565eb31c6390bddb33e9"},"cell_type":"markdown","source":"## Loading data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.csv\")\nprint(\"Number of texts: \", df.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96012bbe09075239c432cf48a204ca2a35f85c16"},"cell_type":"code","source":"df = df.sample(30000)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1eccf4f87c29c6216d3d6d4f971bffea1641e44"},"cell_type":"markdown","source":"## Class imbalance"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"d7d54c0edc5228654daa5d1bbb3f7e17b509249c"},"cell_type":"code","source":"plt.figure(figsize = (10, 8))\nsns.countplot(df['target'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dc7a4b01b8118224a6346527007a786fa4d24e4"},"cell_type":"code","source":"print(Counter(df['target']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e42542a1bb153dd51f845a76d82b7fe29e78ab8"},"cell_type":"markdown","source":"There is way more 0s than 1s in our dataset, data is very unbalanced and one should consider using oversampling or undersampling.\n\nI don't recommand undersampling in Kaggle competitions, because you want to have as much data as possible for your training. "},{"metadata":{"_uuid":"b7c5a37c7151305c44764243c4e097b135b1afc2"},"cell_type":"markdown","source":"## Making Data for the network\nWe apply the following steps :\n* Splitting\n* Tokenizing\n* Padding"},{"metadata":{"trusted":true,"_uuid":"07b3163ffb71e5f94cd67075028dd75dac24cc3b"},"cell_type":"code","source":"max_len = 50\nlen_voc = 40000","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b23b1f6cebdb65ad5ea2627a2958f5810c447f1"},"cell_type":"markdown","source":"### Train/Test split\nIt is important to split before oversampling ! "},{"metadata":{"trusted":true,"_uuid":"324de03daf7579588961f5bc10aa91e2d65a420f"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ndf_train, df_test = train_test_split(df, test_size=0.5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87fc62fa686f54ac3531eddfadedb639d15c0ffd"},"cell_type":"markdown","source":"### Tokenizing"},{"metadata":{"trusted":true,"_uuid":"363b9ece53b8e1800d03b188a50ddf84d0cb12a5"},"cell_type":"code","source":"def make_tokenizer(texts, len_voc):\n    from keras.preprocessing.text import Tokenizer\n    t = Tokenizer(num_words=len_voc)\n    t.fit_on_texts(texts)\n    return t","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"017eb151a1d1e927a491879bf068fabd0a7edc4d"},"cell_type":"code","source":"tokenizer = make_tokenizer(df['question_text'], len_voc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35dbfe75cd164f9623acce773ed217533a848414"},"cell_type":"code","source":"X_train = tokenizer.texts_to_sequences(df_train['question_text'])\nX_test = tokenizer.texts_to_sequences(df_test['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbc447350c0ff1e2a8c736a4a1e9214ee0b53663"},"cell_type":"markdown","source":"### Padding"},{"metadata":{"trusted":true,"_uuid":"cd157c268771f2ee711d7fd6b10314322b882ab6"},"cell_type":"code","source":"from keras.preprocessing.sequence import pad_sequences\n\nX_train = pad_sequences(X_train, maxlen=max_len, padding='post', truncating='post')\nX_test = pad_sequences(X_test, maxlen=max_len, padding='post', truncating='post')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e82910797b3369bb821a1641e3bbeee9b015c2f"},"cell_type":"markdown","source":"### Targets"},{"metadata":{"trusted":true,"_uuid":"7b14b15365d6e3ec8b86a8e4705754cbb3e78c5f"},"cell_type":"code","source":"y_train = df_train['target'].values\ny_test = df_test['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fce47b803d2f5b9261cc5c4720ff502a1a3f9f8a"},"cell_type":"markdown","source":"### Embeddings"},{"metadata":{"trusted":true,"_uuid":"be89625f116b26209f996d1b7ad1eb5d2ed8269f"},"cell_type":"code","source":"def get_coefs(word,*arr): \n    return word, np.asarray(arr, dtype='float32')\n\ndef load_embedding(file):\n    if file == '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec':\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file) if len(o)>100)\n    else:\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file, encoding='latin'))\n    return embeddings_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a0f98cd5cd6dcd656673e7fee48f3bc514cdf9f"},"cell_type":"code","source":"def make_embedding_matrix(embedding, tokenizer, len_voc):\n    all_embs = np.stack(embedding.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n    word_index = tokenizer.word_index\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (len_voc, embed_size))\n    \n    for word, i in word_index.items():\n        if i >= len_voc:\n            continue\n        embedding_vector = embedding.get(word)\n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n    \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3effb5c72b4cecfe756d167d16f095dc548dff7e"},"cell_type":"code","source":"glove = load_embedding('../input/embeddings/glove.840B.300d/glove.840B.300d.txt')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64a13c6fd58e00a17e0c9f6a517c5b1e7ba43422"},"cell_type":"code","source":"embed_mat = make_embedding_matrix(glove, tokenizer, len_voc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e044b352fb3310e16e8e666efbddf3ab93585a91"},"cell_type":"code","source":"X_train_emb = embed_mat[X_train]\nX_test_emb = embed_mat[X_test]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"46e8882712c42236e854b6017933ca423381b712"},"cell_type":"markdown","source":"## Oversampling"},{"metadata":{"trusted":true,"_uuid":"88ea22938e3f0ee22490172e2135b7e95ee3f1b1"},"cell_type":"code","source":"train_size, max_len, embed_size = X_train_emb.shape\nX_train_emb_r = X_train_emb.reshape(train_size, max_len*embed_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb0de2db258f119045b8ee1595a186bcbe29f15d"},"cell_type":"code","source":"smt = SMOTE(sampling_strategy=0.2)\nX_smote, y_smote = smt.fit_sample(X_train_emb_r, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"587161813116583afa38da043b43665ef0be27d9"},"cell_type":"code","source":"X_smote = X_smote.reshape((X_smote.shape[0], max_len, embed_size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e896da77df8b9df281a588134c7b341adaa90d6c"},"cell_type":"code","source":"plt.figure(figsize = (10, 8))\nplt.subplot(1, 2, 1)\nsns.countplot(y_train)\nplt.title('Reparition before SMOTE')\nplt.subplot(1, 2, 2)\nsns.countplot(y_smote)\nplt.title('Reparition after SMOTE')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57533e8b1621dd25eff1196e9a183e2ef94a13bd"},"cell_type":"markdown","source":"## Now let us train a model"},{"metadata":{"_uuid":"2a5f324273d8e4726a6f0f9206170845d5ead890"},"cell_type":"markdown","source":"### Making model"},{"metadata":{"trusted":true,"_uuid":"14a2bfd9832b4f7f7fb89a8bd987f93e759f3cdb"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Dense, Bidirectional, CuDNNGRU, GlobalMaxPool1D, Input, Dropout\nfrom keras.optimizers import Adam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4584596a85f8380a0c25eea1e6364f672b114e25"},"cell_type":"code","source":"def make_model(max_len, len_voc=50000, embed_size=300):\n    inp = Input(shape=(max_len, 300))\n    x = Bidirectional(CuDNNGRU(64, return_sequences=True))(inp)\n    x = GlobalMaxPool1D()(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n    model = Model(inputs=inp, outputs=x)\n    model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.001), metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54d59f934f48046cca805f2d7684dba540e7eaa1"},"cell_type":"code","source":"model = make_model(max_len)\nmodel_smote = make_model(max_len)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"032df52e96dda468647fc10a138c3005127a01b6"},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2fcaba4e1d4c7e02f21bad4d3eb7440d17090e5"},"cell_type":"markdown","source":"### Callbacks"},{"metadata":{"trusted":true,"_uuid":"6d516d861a3a0c347401d688ca394b1333233c90"},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.1, patience=2, verbose=1, min_lr=0.000001)\ncheckpoints = ModelCheckpoint('weights.hdf5', monitor=\"val_acc\", mode=\"max\", verbose=True, save_best_only=True)\n\nreduce_lr_smote = ReduceLROnPlateau(monitor='val_acc', factor=0.1, patience=2, verbose=1, min_lr=0.000001)\ncheckpoints_smote = ModelCheckpoint('smote_weights.hdf5', monitor=\"val_acc\", mode=\"max\", verbose=True, save_best_only=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80580a6745b44b8b63ab725672c8ffafbab57e75"},"cell_type":"markdown","source":"### Fitting"},{"metadata":{"trusted":true,"_uuid":"8e1e47353ba3171ba1d798ee7c7c3e36577729c8","scrolled":false},"cell_type":"code","source":"model.fit(X_train_emb, y_train, batch_size=128, epochs=3, validation_data=[X_test_emb, y_test], callbacks=[checkpoints, reduce_lr])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"028cf6ef1b94a8ec4c5120d42d1fa6ae861205c8","scrolled":false},"cell_type":"code","source":"model_smote.fit(X_smote, y_smote, batch_size=128, epochs=3, validation_data=[X_test_emb, y_test], callbacks=[checkpoints_smote, reduce_lr_smote])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42d8b19a31626ec522acd15575754f073d662d80"},"cell_type":"code","source":"model.load_weights('weights.hdf5')\nmodel_smote.load_weights('smote_weights.hdf5')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2aebfe6e5889091b2d97f446373f5a1e2d4f015e"},"cell_type":"markdown","source":"### Predictions"},{"metadata":{"trusted":true,"_uuid":"35bd2078e02117d7fca467cc7068be4e3ae41822"},"cell_type":"code","source":"pred_test = model.predict([X_test_emb], verbose=1)\npred_test_smote = model_smote.predict([X_test_emb], batch_size=256, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"603d53b9394c43f2afefc6f46e0abee52d8755fd"},"cell_type":"markdown","source":"### Tweaking threshold"},{"metadata":{"trusted":true,"_uuid":"217d8ac6f655b35c7c0c82bffcc76ba50fee51dd"},"cell_type":"code","source":"def tweak_threshold(pred, truth):\n    from sklearn.metrics import f1_score\n    scores = []\n    for thresh in np.arange(0.1, 0.501, 0.01):\n        thresh = np.round(thresh, 2)\n        score = f1_score(truth, (pred>thresh).astype(int))\n        scores.append(score)\n    return round(np.max(scores), 4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77bdaceb60a5840c1561d8b9560ad7b29316f9b3"},"cell_type":"code","source":"print(f\"Scored {tweak_threshold(pred_test, y_test)} without SMOTE (test data)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3347d752a8b671019b448d092234bd786fa06bfb"},"cell_type":"code","source":"print(f\"Scored {tweak_threshold(pred_test_smote, y_test)} with SMOTE (test data)\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50698a4b9b2cf21d6370d8e88d90d265fb3af1fc"},"cell_type":"markdown","source":" ## Conclusion\n\nIt appears that SMOTE does not help improve the results. However, it makes the network learning faster.\n\n**Moreover, there is one big problem, this method is not compatible larger datasets.**\n\nYou have to apply SMOTE on embedded sentences, which takes way too much memory. \n\nA solution is to use a generator for our training, which realizes oversampling on batches. I've tried it, but my generator was very slow.\n\nSo I'm going to stick with these results for now, and try another data augmentation technique.\n\nIf you have any improvement idea feel free to let me know.\n\n#### Thanks for reading ! \n "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}