{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### This is the second kernel for me to share my thoughts with others.   This completation is a basic binary classification problem. To achieve this goal, there are so many machine learning algorithms and deep learning algorithms that can make this happend. Here I will use some deep learning algorithms like DNN, CNN, RNN, LSTM, GRU algorithms combined with Doc2Vec algorithms.\n\n#### Following all  advanced deep models are all based on my github project:https://github.com/lugq1990/neural-nets. \n#### I have not make it very well, I need your help  to make it better to be used. Please check it out. Thanks!\n### Let's start!"},{"metadata":{"trusted":true,"_uuid":"130ba7f4715a8046ddfb7a610e402569459e0d7c"},"cell_type":"code","source":"# import some libaries\nimport numpy as np\nimport pandas as pd\nimport os\nimport time\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom matplotlib import style\nimport seaborn as sns\nimport tensorflow as tf\nfrom sklearn import metrics\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Dense, Dropout, BatchNormalization, Activation, LSTM, GRU, BatchNormalization\nfrom tensorflow.keras.layers import Conv1D, GlobalAveragePooling1D, MaxPool1D, Flatten, Bidirectional\nfrom sklearn.model_selection import train_test_split\nfrom nltk.tokenize import word_tokenize\nfrom gensim.models import Doc2Vec\nfrom gensim.models.doc2vec import TaggedDocument","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e0aa2db34588ca54f6d2c3a7672c587873e50e8c"},"cell_type":"code","source":"# What data we have \nprint(os.listdir(\"../input/\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"531ccc4bcfa7fd7ee9de45154908686480836e0d"},"cell_type":"code","source":"# load train and test datasets\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\nprint(\"Train datasets shape:\", train.shape)\nprint(\"Test datasets shape:\", test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"379e19d79a102d4a68fa0e2739f1e1c99e670a22"},"cell_type":"code","source":"# Show some train datasets \nprint('Train data samples:')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72e145209e645aa4ea4d4829912fa36749b5af41"},"cell_type":"code","source":"print('Test data samples')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a396be0986fb56f2d56af4e11be8d6b8e66947e6"},"cell_type":"markdown","source":"#### This is text document datasets, so we have to convert the question_text column best to describe the information that can be used for seperating insincere or not. So for now, as far as I have learned that can be used for this problem is: CountVectorizer, TfidfVectorizer, Word2vec, Doc2vec .etc. \n#### But before we use the preprocessing algorithms, we have to get some basic info about our datasets."},{"metadata":{"trusted":true,"_uuid":"b848c9e601cf9f133ea29cb9c2f6e8e6f15eb61f"},"cell_type":"code","source":"# What ratio for insincere data\n# is_not_in_ratio = train.target.value_counts()[0]/len(train)\n# is_in_ratio = train.target.value_counts()[1]/len(train)\n\n# How many different parts numbers\nsns.countplot(train.target)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b9c08f90190d2be0ddcb5d558536a86930f710d"},"cell_type":"markdown","source":"#### Ok, this is a really unbalanced problem. So little insincere, this is a normal thing, after all the world is a normal world!\n#### But here I want to say one more words, For unbalanced datasets, we have  3 ways to solve it. One: use some data augument algorithms, such as SMOTE .etc. Two, we can give different weights for different classes. Three, we can tune some machine learning algorithm's parameters like class_weight of LogiticRegression. But for this problem, it is text datasets, first way maybe can use GAN to generate more datasets. But For time limited, I may not use this."},{"metadata":{"trusted":true,"_uuid":"876a96d85c7918d06af26e27baa12aa27d702332"},"cell_type":"code","source":"# This function is used to get some basic information(how many words and characters) about this text\ndef cfind(df):\n    df_new = df.copy()\n    data = df.question_text\n    df_new['Sentence_length'] = pd.Series([len(r) for r in data])\n    df_new['Word_num'] = pd.Series([len(r.split(' ')) for r in data])\n    return df_new\ntrain_new = cfind(train)\ntest_new = cfind(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"773953b444a5143d57cd09be645bd64fcefb4451"},"cell_type":"code","source":"# Plot the basic information\nfig, ax = plt.subplots(1, 2, figsize=(14, 10))\nsns.distplot(train_new.Sentence_length, ax=ax[0])\nax[0].set_title('Sentence Length distribution')\nsns.distplot(train_new.Word_num, ax=ax[1])\nax[1].set_title('Word number distribution')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c492b9ca3298988123d1551e8b03ecbf6ffb3d24"},"cell_type":"markdown","source":"#### Both of them are long tail distribution.  There are some question are more than 120 words. Haha, So many words can explain what they want."},{"metadata":{"trusted":true,"_uuid":"5892486a5f745c39ffdca65f5576bb75401b0ae5"},"cell_type":"code","source":"# Here I will split the data to train and validation data\ntrain_data, validation_data = train_test_split(train_new, test_size=.1, random_state=1234)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d3c596f9beaa3bdaf111afe704cfa35732f339c7"},"cell_type":"code","source":"# Here I will use Tokenizer to extract the keyword vector as baseline\n# I will use train data to fit the Tokenizer, then use this Tokenizer to extract the validation data\n# max_length = 100\n# max_features = 50000\n# token = Tokenizer(num_words=max_features)\n# token.fit_on_texts(list(np.asarray(train_data.question_text)))\n# xtrain = token.texts_to_sequences(np.asarray(train_data.question_text))\n# xvalidate = token.texts_to_sequences(np.asarray(validation_data.question_text))\n# xtest = token.texts_to_sequences(np.asarray(test_new.question_text))\n\n# # Because Tokenizer will split the sentence, for some sentence are smaller,\n# # so we have to pad the missing position\n# xtrain = pad_sequences(xtrain, maxlen=max_length)\n# xvalidate = pad_sequences(xvalidate, maxlen=max_length)\n# xtest = pad_sequences(xtest, maxlen=max_length)\n\n# ytrain = train_data.target\n# yvaliate = validation_data.target","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76febcd4e23a17c7c57b5ed567c7c5796e25dba8"},"cell_type":"code","source":"# Here I write a helper function to evaluate model\ndef evaluate(y, pred, p_diff_thre=False):\n    if pred.shape[1] == 2:\n        pred = np.argmax(pred, axis=1)\n    if y.shape[1] == 2:\n        y = np.argmax(y, axis=1)\n    score = metrics.f1_score(y, pred)\n    print('F1-score=%.4f'%(score))\n    if p_diff_thre:\n        for thr in np.arange(.1, .501, .01):\n            thr = np.round(thr, 2)\n            print('Threshold: %.3f, F1-Score: %.4f'%(thr, metrics.f1_score(y, (pred>thr).astype(int))))\n    return score\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9ada7eb6f108247e051f3438bd9831424d5304b"},"cell_type":"code","source":"### Because of training this LR model is so slow, not train it for now.\n# # Here is a baseline model: Logistic Regression\n# from sklearn.linear_model import LogisticRegression\n# lr_base = LogisticRegression()\n# lr_base.fit(xtrain, ytrain)\n# pred_lr_base = lr_base.predict(xvalidate)\n# evaluate(yvaliate, pred_lr_base)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ce41e0099f88a37ded0196a856b7ae65c56f96e"},"cell_type":"markdown","source":"#### So by using just LR, F1-score is so low: .0068. How should this be? So I will not use the Tokenizer result to do machine learning algorithms, because we need much better preprecossing to describe the datasets. Here is 2 ways to do: first is to train a Word2vec model or Doc2Vec model to get the vector result; Second is using the already trained vector to describe.\n\n1. First I will use Doc2vec model to train based on the data.\n2. I will train a Word2vec model, also combined with TFIDF algorithm\n3. I will use the already provided vector directory\n\n#### First Doc2vec show up!"},{"metadata":{"trusted":true,"_uuid":"5bccbb514c271c77b91905135cfed433b3f97356"},"cell_type":"code","source":"# Before I use Doc2Vec algorithm, I will use nltk to do word tokenizer.\n# First must Tagged each document with index\ntag_d = np.array(train_new.question_text)\ntagged_data = [TaggedDocument(words=word_tokenize(d.lower()), \n                              tags=[str(i)]) for i, d in enumerate(tag_d)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"066d737fbb71603cc5b3219009fecd8e7353acc2"},"cell_type":"code","source":"### Noted: to train Doc2vec model, there are some important parameters, like: alpha(learning rate), \n### vector_size(how many dimension of result vector), dm(whether to use distributed bags of words)\n### So if decided to use Doc2Vec model, have to tune these parameters to get a better representation.\n# After I have get tagged data, then I will start to train Doc2vec model\ndef doc2vec_training(tagged_data):\n    epochs = 10   # How many epochs to be trained\n    vector_size = 256  # How many dimensions\n    alpha = .025   # Initial learning rate\n    min_alpha = .00025  # Learning rate changes step\n    dm = 1   # Use Distributed bags of words\n    \n    # Start build model\n    model = Doc2Vec(vector_size=vector_size, alpha=alpha, min_alpha=min_alpha, min_count=1, dm=dm, workers=-1)\n    model.build_vocab(tagged_data)\n    \n    # start to train model\n    for e in range(epochs):\n        model.train(tagged_data, total_examples=model.corpus_count, epochs=model.iter)\n        # decrease learning rate\n        model.alpha -= min_alpha\n        model.min_alpha = model.alpha\n        print('Now is epochs: %02d'%(e))\n    print('Finished model training process')\n    \n    return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"473da02e086dc14971ea9e3c85415120976ea8ef"},"cell_type":"code","source":"# Because I want to get the return result data vector, \n# but I will also want to use this model to infer test data, so this is return model\ndoc2vec_model = doc2vec_training(tagged_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16e834d1f5a89d6d2eb7fd470e264fcab1d3afc3"},"cell_type":"code","source":"# After I have trained this model, I will use this model to infer train datasets\n# For now, I will not infer the test datasets, because I don't know whether this feature is best to test.\n# doc2vec_train = doc2vec_model.infer_vector(tag_d)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db18e736a9683d9066372426e379a45de9812315"},"cell_type":"code","source":"# Get final result from already trained doc2vec model.\ndoc2vec_train = doc2vec_model.docvecs.vectors_docs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8774b110f16e84c93a88a8f52599061313ab15ca"},"cell_type":"code","source":"# After I get the doc2vec result, then I will split this result to train and valiatation result.\nfrom tensorflow import keras\nlabel = np.asarray(train_new.target).reshape(-1, 1)\nlabel = keras.utils.to_categorical(label)\nxtrain_doc, xtest_doc, ytrain_doc, ytest_doc = train_test_split(doc2vec_train, label, test_size=.2, random_state=1234)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9bfac6b969113fb0541abccfadc1b6833b8507d1"},"cell_type":"code","source":"# Here I will build a DNN model as deep learning baseline\n\n# Here I write a DNN class for many other cases, \n# you can choose how many layers, how many units, whether to use dropout,\n# whether to use batchnormalization, also with optimizer! \nclass dnnNet(object):\n    def __init__(self, n_classes=2, n_dims=None, n_layers=3, n_units=64, use_dropout=True, drop_ratio=.5, use_batchnorm=True,\n                 metrics='accuracy', optimizer='rmsprop'):\n        self.n_classes = n_classes\n        self.n_dims = n_dims\n        self.n_layers = n_layers\n        self.n_units = n_units\n        self.use_dropout = use_dropout\n        self.drop_ratio = drop_ratio\n        self.use_batchnorm = use_batchnorm\n        self.metrics = metrics\n        self.optimizer = optimizer\n        self.model = self._init_model()\n\n    def _init_model(self):\n        if self.n_dims is None:\n            raise AttributeError('Data Dimension must be provided!')\n        inputs = Input(shape=(self.n_dims, ))\n\n        # this is dense block function.\n        def _dense_block(layers):\n            res = Dense(self.n_units)(layers)\n            if self.use_batchnorm:\n                res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            if self.use_dropout:\n                res = Dropout(self.drop_ratio)(res)\n            return res\n\n        for i in range(self.n_layers):\n            if i == 0:\n                res = _dense_block(inputs)\n            else: res = _dense_block(res)\n\n        if self.n_classes == 2:\n            out = Dense(self.n_classes, activation='sigmoid')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameters n_class must be provide up or equal 2!')\n\n        return model\n\n    # For fit function, auto randomly split the data to be train and validation datasets.\n    def fit(self, data, label, epochs=100, batch_size=256):\n        xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n        self.his = self.model.fit(xtrain, ytrain, epochs=epochs, batch_size=batch_size, verbose=1,\n                                  validation_data=(xvalidate, yvalidate))\n        print('Model evaluation on validation datasets accuracy:{:.4f}'.format(self.model.evaluate(xvalidate, yvalidate)[1]))\n        return self\n\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.6f}'.format(acc))\n        return acc\n    \n    def predict(self, data, batch_size=None):\n        return self.model.predict(data, batch_size=batch_size)\n    \n    def plot_acc_curve(self):\n        style.use('ggplot')\n\n        fig1, ax1 = plt.subplots(1, 1, figsize=(8, 6))\n        ax1.plot(self.his.history['acc'], label='Train Accuracy')\n        ax1.plot(self.his.history['val_acc'], label='Validation Accuracy')\n        ax1.set_title('Train and Validation Accuracy Curve')\n        ax1.set_xlabel('Epochs')\n        ax1.set_ylabel('Accuracy score')\n        plt.legend()\n\n        fig2, ax2 = plt.subplots(1, 1, figsize=(8, 6))\n        ax2.plot(self.his.history['loss'], label='Train Loss')\n        ax2.plot(self.his.history['val_loss'], label='Validation Loss')\n        ax2.set_title('Train and Validation Loss Curve')\n        ax2.set_xlabel('Epochs')\n        ax2.set_ylabel('Loss score')\n        plt.legend()\n        plt.show()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6336a0e5496137290063a8f7cce8a8797e9bde0e"},"cell_type":"code","source":"model_dnn = dnnNet(n_classes=2, n_dims=xtrain_doc.shape[1], n_layers=3, n_units=128)\nmodel_dnn.fit(xtrain_doc, ytrain_doc, epochs=10, batch_size=2048)\nacc = model_dnn.evaluate(xtest_doc, ytest_doc, batch_size=10240)\nmodel_dnn.plot_acc_curve()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba30ad579331889269b41c27a7244ef41eaca46f"},"cell_type":"code","source":"# Before we move forward, del some unused datasets\nimport gc\ndel train_new,train, doc2vec_model, doc2vec_train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"507d668261f83c65cc8bd419fd6efc36f89ef666"},"cell_type":"code","source":"# Because for CNN, LSTM, Residual network and DenseNet, data must up to 2 dimensions,I convert data is 3-D(batch, 16, 16)\nxtrain_deep = xtrain_doc.reshape(-1, 16, 16)\nxtest_deep = xtest_doc.reshape(-1, 16, 16)\nytrain_deep = ytrain_doc.copy()\nytest_deep = ytest_doc.copy()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce605cdfc19deda7248310e71b1837098b4b8dc5"},"cell_type":"markdown","source":"#### Here is a residual class, you can also choose how many residual block to use,  how many units to use, whether to use dense layer, how many dense layer to be used, how many units of dense layer, and optimizer and so on. You can also plot the train and validation accuracy and loss curve by using this model function in just on line!"},{"metadata":{"trusted":true,"_uuid":"87e7d948a75ee7fab0241997bac042e23520ab7a"},"cell_type":"code","source":"class residualNet(object):\n    def __init__(self, input_dim1=None, input_dim2=None, n_classes=2, n_layers=4, flatten=True, use_dense=True,\n                 n_dense_layers=1, conv_units=64, stride=1, padding='SAME', dense_units=128, drop_ratio=.5,\n                 optimizer='rmsprop', metrics='accuracy'):\n        self.input_dim1 = input_dim1\n        self.input_dim2 = input_dim2\n        self.n_classes = n_classes\n        self.n_layers = n_layers\n        self.flatten = flatten\n        self.use_dense = use_dense\n        self.n_dense_layers = n_dense_layers\n        self.conv_units = conv_units\n        self.stride = stride\n        self.padding = padding\n        self.dense_units = dense_units\n        self.drop_ratio = drop_ratio\n        self.optimizer = optimizer\n        self.metrics = metrics\n        self.model = self._init_model()\n\n    def _init_model(self):\n        inputs = Input(shape=(self.input_dim1, self.input_dim2))\n\n        # dense net residual block\n        def _res_block(layers):\n            res = Conv1D(self.conv_units, self.stride, padding=self.padding)(layers)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            res = Conv1D(self.input_dim2, self.stride, padding=self.padding)(res)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            return keras.layers.add([layers, res])\n\n        # construct residual block chain.\n        for i in range(self.n_layers):\n            if i == 0:\n                res = _res_block(inputs)\n            else:\n                res = _res_block(res)\n\n        # using flatten or global average pooling to process Convolution result\n        if self.flatten:\n            res = Flatten()(res)\n        else:\n            res = GlobalAveragePooling1D()(res)\n\n        # whether or not use dense net, also with how many layers to use\n        if self.use_dense:\n            for j in range(self.n_dense_layers):\n                res = Dense(self.dense_units)(res)\n                res = BatchNormalization()(res)\n                res = Activation('relu')(res)\n                res = Dropout(self.drop_ratio)(res)\n\n        if self.n_classes == 2:\n            out = Dense(self.n_classes, activation='sigmoid')(res)\n            model = Model(inputs, out)\n            print('Model structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameters n_classes must up to 2!')\n\n        return model\n\n    # Fit on given training data and label. Here I will auto random split the data to train and validation data,\n    # for test datasets, I will just use it if model already trained then I will evaluate the model.\n    def fit(self, data, label, epochs=100, batch_size=256):\n        # label is not encoding as one-hot, use keras util to convert it to one-hot\n        if len(label.shape) == 1:\n            label = keras.utils.to_categorical(label, num_classes=len(np.unique(label)))\n\n        xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n        self.his = self.model.fit(xtrain, ytrain, verbose=1, epochs=epochs,\n                                  validation_data=(xvalidate, yvalidate), batch_size=batch_size)\n        print('After training, model accuracy on validation datasets is {:.2f}%'.format(\n            self.model.evaluate(xvalidate, yvalidate)[1]*100))\n        return self\n\n    # this is evaluation function to evaluate already trained model.\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        if len(label.shape) == 1:\n            label = keras.utils.to_categorical(label, num_classes=len(np.unique(label)))\n\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.2f}%'.format(acc*100))\n        return acc\n    \n    def predict(self, data, batch_size=None):\n        return self.model.predict(data, batch_size=batch_size)\n    \n    # plot after training accuracy and loss curve.\n    def plot_acc_curve(self):\n        style.use('ggplot')\n\n        fig1, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['acc'], label='Train Accuracy')\n        ax.plot(self.his.history['val_acc'], label='Validation Accuracy')\n        ax.set_title('Train and Validation Accruacy Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Accuracy score')\n        plt.legend()\n\n        fig2, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['loss'], label='Traing Loss')\n        ax.plot(self.his.history['val_loss'], label='Validation Loss')\n        ax.set_title('Train and Validation Loss Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Loss score')\n\n        plt.legend()\n        plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5d5ee7fa73d2b447c953963eca9a20f810a6629"},"cell_type":"code","source":"### You must have noticed that Residual network will be more deep but with less parameters. This is amazing of Residual net.\n# Start to build a residual network\nmodel_res = residualNet(n_classes=2, input_dim1=16, input_dim2=16, n_layers=8, conv_units=128, dense_units=256)\nmodel_res.fit(xtrain_deep, ytrain_deep, epochs=2, batch_size=2048)\nmodel_res.evaluate(xtest_deep, ytest_deep, batch_size=10240)\nmodel_res.plot_acc_curve()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c52334207f374f53271fcb003cb61dd5abb3d1f"},"cell_type":"markdown","source":"#### This is LSTM model, you can also use GRU, Bidirectional LSTM or GRU. You can choose which to use with parameters, you can also choose how many layers to be used, how many units, whether to use BatchNormalization, or dropout and so many others to be choosen to build more advanced model."},{"metadata":{"trusted":true,"_uuid":"9c7b72ca5ba58aa50b3411e31c5baa346c4586d0"},"cell_type":"code","source":"class lstmNet(object):\n    def __init__(self, n_classes=2, input_dim1=None, input_dim2=None, n_layers=3, use_dropout=True, drop_ratio=.5,\n                 use_bidirec=False, use_gru=False, rnn_units=64, use_dense=True, dense_units=64, use_batch=True,\n                 metrics='accuracy', optimizer='rmsprop'):\n        self.n_classes = n_classes\n        self.input_dim1 = input_dim1\n        self.input_dim2 = input_dim2\n        self.n_layers = n_layers\n        self.use_dropout = use_dropout\n        self.drop_ratio = drop_ratio\n        self.use_bidierc = use_bidirec\n        self.use_gru = use_gru\n        self.rnn_units = rnn_units\n        self.use_dense = use_dense\n        self.use_batch = use_batch\n        self.dense_units = dense_units\n        self.metrics = metrics\n        self.optimizer = optimizer\n        self.model = self._init_model()\n\n    def _init_model(self):\n        inputs = Input(shape=(self.input_dim1, self.input_dim2))\n\n        def _lstm_block(layers, name_index=None):\n            if self.use_bidierc:\n                res = Bidirectional(LSTM(self.rnn_units, return_sequences=True,\n                                         recurrent_dropout=self.drop_ratio), name='bidi_lstm_'+str(name_index))(layers)\n            elif self.use_gru:\n                res = GRU(self.rnn_units, return_sequences=True,\n                          recurrent_dropout=self.drop_ratio, name='gru_'+str(name_index))(layers)\n            else:\n                res = LSTM(self.rnn_units, return_sequences=True,\n                           recurrent_dropout=self.drop_ratio, name='lstm_'+str(name_index))(layers)\n\n            if self.use_dropout:\n                res = Dropout(self.drop_ratio)(res)\n\n            return res\n\n        # No matter for LSTM, GRU, bidirection LSTM, final layer can not use 'return_sequences' output.\n        for i in range(self.n_layers - 1):\n            if i == 0:\n                res = _lstm_block(inputs, name_index=i)\n            else:\n                res = _lstm_block(res, name_index=i)\n\n        # final LSTM layer\n        if self.use_bidierc:\n            res = Bidirectional(LSTM(self.rnn_units), name='bire_final')(res)\n        elif self.use_gru:\n            res = GRU(self.rnn_units, name='gru_final')(res)\n        else:\n            res = LSTM(self.rnn_units, name='lstm_final')(res)\n\n        # whether or not to use Dense layer\n        if self.use_dense:\n            res = Dense(self.dense_units, name='dense_1')(res)\n            if self.use_batch:\n                res = BatchNormalization(name='batch_1')(res)\n            res = Activation('relu')(res)\n            if self.use_dropout:\n                res = Dropout(self.drop_ratio)(res)\n\n        if self.n_classes == 2:\n            out = Dense(self.n_classes, activation='sigmoid', name='out')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax', name='out')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameter n_class must be provide up or equals to 2!')\n\n        return model\n\n    def fit(self, data, label, epochs=100, batch_size=256):\n        #label = check_label_shape(label)\n        xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n        self.his = self.model.fit(xtrain, ytrain, epochs=epochs, batch_size=batch_size, verbose=1,\n                                  validation_data=(xvalidate, yvalidate))\n        print('Model evaluation on validation datasets accuracy:{:.2f}'.format(\n            self.model.evaluate(xvalidate, yvalidate)[1]*100))\n        return self\n\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        #label = check_label_shape(label)\n\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.2f}'.format(acc*100))\n        return acc\n\n    def plot_acc_curve(self, plot_acc=True, plot_loss=True, figsize=(8, 6)):\n        style.use('ggplot')\n\n        if plot_acc:\n            fig1, ax1 = plt.subplots(1, 1, figsize=figsize)\n            ax1.plot(self.his.history['acc'], label='Train accuracy')\n            ax1.plot(self.his.history['val_acc'], label='Validation accuracy')\n            ax1.set_title('Train and validation accuracy curve')\n            ax1.set_xlabel('Epochs')\n            ax1.set_ylabel('Accuracy score')\n            plt.legend()\n\n        if plot_loss:\n            fig2, ax2 = plt.subplots(1, 1, figsize=(8, 6))\n            ax2.plot(self.his.history['loss'], label='Train Loss')\n            ax2.plot(self.his.history['val_loss'], label='Validation Loss')\n            ax2.set_title('Train and validation loss curve')\n            ax2.set_xlabel('Epochs')\n            ax2.set_ylabel('Loss score')\n            plt.legend()\n\n        plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b74bce11f4880aa5888ace7207ebc1ee370f0570"},"cell_type":"code","source":"model_lstm = lstmNet(n_classes=2, input_dim1=16, input_dim2=16, n_layers=3)\nmodel_lstm.fit(xtrain_deep, ytrain_deep, epochs=2, batch_size=4086)\nmodel_lstm.evaluate(xtest_deep, ytest_deep, batch_size=4092)\nmodel_lstm.plot_acc_curve()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd875f0248e58952964082c51d61fcfaaab0ccf3"},"cell_type":"markdown","source":"#### Here is a more advanced model sturcture: DenseNet.  You can also choose to use basic residual or dense residual, also with Dropout and BatchNormalization to be choosen. You can check this class to find which can be usd."},{"metadata":{"trusted":true,"_uuid":"6f16be7bf620347a53da3ff93f24b614c3435ddc"},"cell_type":"code","source":"class denseNet(object):\n    def __init__(self, input_dim1=None, input_dim2=None, n_classes=2, basic_residual=False, n_layers=4, flatten=True, use_dense=True,\n                 n_dense_layers=1, conv_units=64, stride=1, padding='SAME', dense_units=128, drop_ratio=.5,\n                 optimizer='rmsprop', metrics='accuracy'):\n        self.input_dim1 = input_dim1\n        self.input_dim2 = input_dim2\n        self.n_classes = n_classes\n        self.basic_residual = basic_residual\n        self.n_layers = n_layers\n        self.flatten = flatten\n        self.use_dense = use_dense\n        self.n_dense_layers = n_dense_layers\n        self.conv_units = conv_units\n        self.stride = stride\n        self.padding = padding\n        self.dense_units = dense_units\n        self.drop_ratio = drop_ratio\n        self.optimizer = optimizer\n        self.metrics = metrics\n        self.model = self._init_model()\n\n    # this will build DenseNet or ResidualNet structure, this model is already compiled.\n    def _init_model(self):\n        inputs = Input(shape=(self.input_dim1, self.input_dim2))\n\n        # dense net residual block\n        def _res_block(layers, added_layers=inputs):\n            res = Conv1D(self.conv_units, self.stride, padding=self.padding)(layers)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            res = Conv1D(self.input_dim2, self.stride, padding=self.padding)(res)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            if self.basic_residual:\n                return keras.layers.add([res, layers])\n            else:\n                return keras.layers.add([res, added_layers])\n\n        # construct residual block chain.\n        for i in range(self.n_layers):\n            if i == 0:\n                res = _res_block(inputs)\n            else:\n                res = _res_block(res)\n\n        # using flatten or global average pooling to process Convolution result\n        if self.flatten:\n            res = Flatten()(res)\n        else:\n            res = GlobalAveragePooling1D()(res)\n\n        # whether or not use dense net, also with how many layers to use\n        if self.use_dense:\n            for j in range(self.n_dense_layers):\n                res = Dense(self.dense_units)(res)\n                res = BatchNormalization()(res)\n                res = Activation('relu')(res)\n                res = Dropout(self.drop_ratio)(res)\n\n        if self.n_classes == 2:\n            out = Dense(self.n_classes, activation='sigmoid')(res)\n            model = Model(inputs, out)\n            print('Model structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameters n_classes must up to 2!')\n\n        return model\n\n    # Fit on given training data and label. Here I will auto random split the data to train and validation data,\n    # for test datasets, I will just use it if model already trained then I will evaluate the model.\n    def fit(self, data, label, epochs=100, batch_size=256):\n        # self.model = self._init_model()\n        xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n        self.his = self.model.fit(xtrain, ytrain, verbose=1, epochs=epochs,\n                             validation_data=(xvalidate, yvalidate), batch_size=batch_size)\n        print('After training, model accuracy on validation datasets is {:.4f}'.format(self.model.evaluate(xvalidate, yvalidate)[1]))\n        return self\n\n    # evaluate model on test datasets.\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.6f}'.format(acc))\n        return acc\n\n    # plot after training accuracy and loss curve.\n    def plot_acc_curve(self):\n        style.use('ggplot')\n\n        fig1, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['acc'], label='Train Accuracy')\n        ax.plot(self.his.history['val_acc'], label='Validation Accuracy')\n        ax.set_title('Train and Validation Accruacy Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Accuracy score')\n        plt.legend()\n\n        fig2, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['loss'], label='Traing Loss')\n        ax.plot(self.his.history['val_loss'], label='Validation Loss')\n        ax.set_title('Train and Validation Loss Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Loss score')\n\n        plt.legend()\n        plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"920469880906e5c0cba152ebddceb743f2e246cf"},"cell_type":"code","source":"model_dense = denseNet(n_classes=2, input_dim1=16, input_dim2=16, n_layers=3, n_dense_layers=2, optimizer='sgd')\nmodel_dense.fit(xtrain_deep, ytrain_deep, epochs=2, batch_size=2048)\nmodel_dense.evaluate(xtest_deep, ytest_deep, batch_size=4092)\nmodel_dense.plot_acc_curve()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e13d9d5b524b3706cc798fa4c82e782df28298c"},"cell_type":"markdown","source":"### So there are many advanced deep neural networks that can be used for this problem. Here I just use Dec2Vec algorithm to get the inference features,  you can also use this models to train based on the vector that this competition given. There are also some great kernel that use provived features. Here they are:\n1. With attention: https://www.kaggle.com/shujian/different-embeddings-with-attention-fork-fork\n2.All features used: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\n#### If you have time, you can check them out. Use the already trained vectors combined with different deep learning model structure to fit. \n#### Again, Here is my github probject:https://github.com/lugq1990/neural-nets\n#### If you like to make this probject better to be used, don't hesitate to check it out! \n#### Thanks for your support!"},{"metadata":{"trusted":true,"_uuid":"0318cae3be9c5c6bacfe569929ca501af847ad2e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}