{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### Hi, every one. This kernel is based on this competition provided already trained embedding. I will also use my github project to use some advanced deep learning algorithms to train this datasets.\n### Here is my github project:   https://github.com/lugq1990/neural-nets.\n\n### I have also read one kernel that based on Doc2Vec to extract information to fit on deep learning models. You can also check it out to find how to use Doc2vec for deep learning:  https://www.kaggle.com/manrunning/dnn-resisual-densenet-lstm-gru-with-doc2vec\n### Go from here！\n"},{"metadata":{"trusted":true,"_uuid":"850a269fd8eb5e17cf46bfcf813eff027ba8e035"},"cell_type":"code","source":"# import some libaries\nimport numpy as np\nimport pandas as pd\nimport os\nimport time\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom matplotlib import style\nimport seaborn as sns\nimport tensorflow as tf\nfrom sklearn import metrics\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow import keras\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Dense, Dropout, BatchNormalization, Activation, LSTM, GRU, BatchNormalization\nfrom tensorflow.keras.layers import Conv1D, GlobalAveragePooling1D, MaxPool1D, Flatten, Bidirectional\nfrom tensorflow.keras.layers import Embedding, Reshape, CuDNNGRU, CuDNNLSTM\nfrom sklearn.model_selection import train_test_split\nfrom nltk.tokenize import word_tokenize\nfrom scikitplot.metrics import plot_confusion_matrix\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"458ade3fbe935c53102d6c126ced2ae26c791333"},"cell_type":"code","source":"# load train and test datasets\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\nprint(\"Train datasets shape:\", train.shape)\nprint(\"Test datasets shape:\", test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7bb773a29c3b2a082ba51e2fe5cb27f2eae1a60e"},"cell_type":"code","source":"# Show some train datasets \nprint('Train data samples:')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f551dff629b1dc842eb9e565cc3648501849f200"},"cell_type":"markdown","source":"#### This is text document datasets, so we have to convert the question_text column best to describe the information that can be used for seperating insincere or not. So for now, as far as I have learned that can be used for this problem is: CountVectorizer, TfidfVectorizer, Word2vec, Doc2vec .etc. \n#### But before we use the preprocessing algorithms, we have to get some basic info about our datasets."},{"metadata":{"trusted":true,"_uuid":"8ada74dd03eb8dc57709ea260b32b6f21657d8af"},"cell_type":"code","source":"# What ratio for insincere data\n# is_not_in_ratio = train.target.value_counts()[0]/len(train)\n# is_in_ratio = train.target.value_counts()[1]/len(train)\n\n# How many different parts numbers\nsns.countplot(train.target)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b51c9179f9d172828b771268c7020037c975ed2b"},"cell_type":"markdown","source":"#### Ok, this is a really unbalanced problem. So little insincere, this is a normal thing, after all the world is a normal world!\n#### But here I want to say one more words, For unbalanced datasets, we have  3 ways to solve it. One: use some data augument algorithms, such as SMOTE .etc. Two, we can give different weights for different classes. Three, we can tune some machine learning algorithm's parameters like class_weight of LogiticRegression. But for this problem, it is text datasets, first way maybe can use GAN to generate more datasets. But For time limited, I may not use this."},{"metadata":{"trusted":true,"_uuid":"8c68f1f40c3f60c3472d5932d1a92c76e9288c49"},"cell_type":"code","source":"# This function is used to get some basic information(how many words and characters) about this text\ndef cfind(df):\n    df_new = df.copy()\n    data = df.question_text\n    df_new['Sentence_length'] = pd.Series([len(r) for r in data])\n    df_new['Word_num'] = pd.Series([len(r.split(' ')) for r in data])\n    return df_new\ntrain_new = cfind(train)\ntest_new = cfind(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"28bd0e5ac9d5c1f6a0ded8c14c42a2270b4feb4b"},"cell_type":"code","source":"# Plot the basic information\nfig, ax = plt.subplots(1, 2, figsize=(14, 10))\nsns.distplot(train_new.Sentence_length, ax=ax[0])\nax[0].set_title('Sentence Length distribution')\nsns.distplot(train_new.Word_num, ax=ax[1])\nax[1].set_title('Word number distribution')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dbea2d4581e3e22d92c4fc36928c7812e577c91"},"cell_type":"markdown","source":"#### Both of them are long tail distribution.  There are some question are more than 120 words. Haha, So many words can explain what they want."},{"metadata":{"trusted":true,"_uuid":"2c919ba6794e7fa245c7bee5b36ff0f8fce36857"},"cell_type":"code","source":"# Here I will split the data to train and validation data\ntrain_data, validation_data = train_test_split(train_new, test_size=.1, random_state=1234)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c3a6345f1103f2d1d85b7bf8c975e26066f33c2"},"cell_type":"code","source":"# Here I will use Tokenizer to extract the keyword vector as baseline\n# I will use train data to fit the Tokenizer, then use this Tokenizer to extract the validation data\nmax_length = 100\nmax_features = 50000\ntoken = Tokenizer(num_words=max_features)\ntoken.fit_on_texts(list(np.asarray(train_data.question_text)))\nxtrain = token.texts_to_sequences(np.asarray(train_data.question_text))\nxvalidate = token.texts_to_sequences(np.asarray(validation_data.question_text))\nxtest = token.texts_to_sequences(np.asarray(test_new.question_text))\n\n# Because Tokenizer will split the sentence, for some sentence are smaller,\n# so we have to pad the missing position\nxtrain = pad_sequences(xtrain, maxlen=max_length)\nxvalidate = pad_sequences(xvalidate, maxlen=max_length)\nxtest = pad_sequences(xtest, maxlen=max_length)\n\nytrain = train_data.target\nyvaliate = validation_data.target","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c03a3b18447120fd1413ecaff752099f4fc2b1e"},"cell_type":"code","source":"# Here I write a helper function to evaluate model\ndef evaluate(y, pred):\n    f1_list = list()\n    thre_list = np.arange(0.1, 0.501, 0.01)\n    for thresh in thre_list:\n        thresh = np.round(thresh, 2)\n        f1 = metrics.f1_score(y, (pred>thresh).astype(int))\n        f1_list.append(f1)\n        print(\"F1 score at threshold {0} is {1}\".format(thresh, f1))\n    #return f1_list\n    plot_confusion_matrix(y, np.array(pd.Series(pred.reshape(-1,)).map(lambda x:1 if x>thre_list[np.argmax(f1_list)] else 0)))\n    print('Best Threshold: ',thre_list[np.argmax(f1_list)])\n    return thre_list[np.argmax(f1_list)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"779cf079fbc0a69009db4890ca5840c55387ea40"},"cell_type":"code","source":"# Here I will build a DNN model as deep learning baseline\n\n# Here I write a DNN class for many other cases, \n# you can choose how many layers, how many units, whether to use dropout,\n# whether to use batchnormalization, also with optimizer! \nclass dnnNet(object):\n    def __init__(self, n_classes=2, n_dims=None, n_layers=3, n_units=64, use_dropout=True, drop_ratio=.5, use_batchnorm=True,\n                 metrics='accuracy', optimizer='rmsprop', use_em=False, em_weights=None, em_input_dim=None, fit_split=False):\n        self.n_classes = n_classes\n        self.n_dims = n_dims\n        self.n_layers = n_layers\n        self.n_units = n_units\n        self.use_dropout = use_dropout\n        self.drop_ratio = drop_ratio\n        self.use_batchnorm = use_batchnorm\n        self.metrics = metrics\n        self.optimizer = optimizer\n        self.use_em = use_em\n        self.em_weights = em_weights\n        self.em_input_dim = em_input_dim\n        self.fit_split = fit_split\n        self.model = self._init_model()\n\n    def _init_model(self):\n        if self.n_dims is None:\n            raise AttributeError('Data Dimension must be provided!')\n        inputs = Input(shape=(self.n_dims, ))\n\n        # this is dense block function.\n        def _dense_block(layers):\n            res = Dense(self.n_units)(layers)\n            if self.use_batchnorm:\n                res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            if self.use_dropout:\n                res = Dropout(self.drop_ratio)(res)\n            return res\n\n        for i in range(self.n_layers):\n            if i == 0:\n                res = _dense_block(inputs)\n            else: res = _dense_block(res)\n\n        if self.n_classes == 2:\n            out = Dense(1, activation='sigmoid')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameters n_class must be provide up or equal 2!')\n\n        return model\n\n    # For fit function, auto randomly split the data to be train and validation datasets.\n    def fit(self, data, label, epochs=100, batch_size=256, vali_data=None, vali_label=None):\n        if self.fit_split:\n            xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n            self.his = self.model.fit(xtrain, ytrain, epochs=epochs, batch_size=batch_size, verbose=1,\n                                      validation_data=(xvalidate, yvalidate))\n            print('Model evaluation on validation datasets accuracy:{:.4f}'.format(self.model.evaluate(xvalidate, yvalidate)[1]))\n        else: \n            self.his = self.model.fit(data, label, epochs=epochs, batch_size=batch_size, verbose=1,\n                                      validation_data=(vali_data, vali_label))\n        return self\n\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.6f}'.format(acc))\n        return acc\n    \n    def predict(self, data, batch_size=None):\n        return self.model.predict(data, batch_size=batch_size)\n    \n    def plot_acc_curve(self):\n        style.use('ggplot')\n\n        fig1, ax1 = plt.subplots(1, 1, figsize=(8, 6))\n        ax1.plot(self.his.history['acc'], label='Train Accuracy')\n        ax1.plot(self.his.history['val_acc'], label='Validation Accuracy')\n        ax1.set_title('Train and Validation Accuracy Curve')\n        ax1.set_xlabel('Epochs')\n        ax1.set_ylabel('Accuracy score')\n        plt.legend()\n\n        fig2, ax2 = plt.subplots(1, 1, figsize=(8, 6))\n        ax2.plot(self.his.history['loss'], label='Train Loss')\n        ax2.plot(self.his.history['val_loss'], label='Validation Loss')\n        ax2.set_title('Train and Validation Loss Curve')\n        ax2.set_xlabel('Epochs')\n        ax2.set_ylabel('Loss score')\n        plt.legend()\n        plt.show()    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1bd4d0c66506725201e6922a3acd5a75d363042"},"cell_type":"markdown","source":"#### Here is a residual class, you can also choose how many residual block to use,  how many units to use, whether to use dense layer, how many dense layer to be used, how many units of dense layer, and optimizer and so on. You can also plot the train and validation accuracy and loss curve by using this model function in just on line!"},{"metadata":{"trusted":true,"_uuid":"e07261c025db19b93425484a04766445af7c5c50"},"cell_type":"code","source":"class residualNet(object):\n    def __init__(self, input_dim1=None, input_dim2=None, n_classes=2, n_layers=4, flatten=True, use_dense=True,\n                 n_dense_layers=1, conv_units=64, stride=1, padding='SAME', dense_units=128, drop_ratio=.5,\n                 optimizer='rmsprop', metrics='accuracy', use_em=False, em_weights=None, fit_split=False):\n        self.input_dim1 = input_dim1\n        self.input_dim2 = input_dim2\n        self.n_classes = n_classes\n        self.n_layers = n_layers\n        self.flatten = flatten\n        self.use_dense = use_dense\n        self.n_dense_layers = n_dense_layers\n        self.conv_units = conv_units\n        self.stride = stride\n        self.padding = padding\n        self.dense_units = dense_units\n        self.drop_ratio = drop_ratio\n        self.optimizer = optimizer\n        self.metrics = metrics\n        self.use_em = use_em\n        self.em_weights = em_weights\n        self.fit_split = fit_split\n        self.model = self._init_model()\n\n    def _init_model(self):\n        \n        if self.use_em:\n            inputs = Input(shape=(self.em_weights.shape[1], ))\n        else: inputs = Input(shape=(self.input_dim1, self.input_dim2))\n\n        # dense net residual block\n        def _res_block(layers):\n            res = Conv1D(self.conv_units, self.stride, padding=self.padding)(layers)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            if self.use_em:\n                res = Conv1D(self.em_weights.shape[1], self.stride, padding=self.padding)(res)\n            else: res = Conv1D(self.input_dim2, self.stride, padding=self.padding)(res)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            return keras.layers.add([layers, res])\n\n        # construct residual block chain.\n        for i in range(self.n_layers):\n            if i == 0:\n                if self.use_em:\n                    res = Embedding(max_features, em_size)(inputs)\n                    res = _res_block(res)\n                else: res = _res_block(inputs)\n            else:\n                res = _res_block(res)\n\n        # using flatten or global average pooling to process Convolution result\n        if self.flatten:\n            res = Flatten()(res)\n        else:\n            res = GlobalAveragePooling1D()(res)\n\n        # whether or not use dense net, also with how many layers to use\n        if self.use_dense:\n            for j in range(self.n_dense_layers):\n                res = Dense(self.dense_units)(res)\n                res = BatchNormalization()(res)\n                res = Activation('relu')(res)\n                res = Dropout(self.drop_ratio)(res)\n\n        if self.n_classes == 2:\n            out = Dense(1, activation='sigmoid')(res)\n            model = Model(inputs, out)\n            print('Model structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameters n_classes must up to 2!')\n\n        return model\n\n    # Fit on given training data and label. Here I will auto random split the data to train and validation data,\n    # for test datasets, I will just use it if model already trained then I will evaluate the model.\n    def fit(self, data, label, epochs=100, batch_size=256):\n        # label is not encoding as one-hot, use keras util to convert it to one-hot\n        if len(label.shape) == 1:\n            label = keras.utils.to_categorical(label, num_classes=len(np.unique(label)))\n\n        if self.fit_split:\n            xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n            self.his = self.model.fit(xtrain, ytrain, verbose=1, epochs=epochs,\n                                      validation_data=(xvalidate, yvalidate), batch_size=batch_size)\n            print('After training, model accuracy on validation datasets is {:.2f}%'.format(\n                self.model.evaluate(xvalidate, yvalidate)[1]*100))\n        else: self.his = self.model.fit(data, label, batch_size=batch_size, epochs=epochs, verbose=1)\n        return self\n\n    # this is evaluation function to evaluate already trained model.\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        if len(label.shape) == 1:\n            label = keras.utils.to_categorical(label, num_classes=len(np.unique(label)))\n\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.2f}%'.format(acc*100))\n        return acc\n    \n    def predict(self, data, batch_size=None):\n        return self.model.predict(data, batch_size=batch_size)\n    \n    # plot after training accuracy and loss curve.\n    def plot_acc_curve(self):\n        style.use('ggplot')\n\n        fig1, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['acc'], label='Train Accuracy')\n        if self.fit_split:\n            ax.plot(self.his.history['val_acc'], label='Validation Accuracy')\n        ax.set_title('Train and Validation Accruacy Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Accuracy score')\n        plt.legend()\n\n        fig2, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['loss'], label='Traing Loss')\n        if self.fit_split:\n            ax.plot(self.his.history['val_loss'], label='Validation Loss')\n        ax.set_title('Train and Validation Loss Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Loss score')\n\n        plt.legend()\n        plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91fcdc8270d010d595e0e114508b124d3d6507ef"},"cell_type":"markdown","source":"#### This is LSTM model, you can also use GRU, Bidirectional LSTM or GRU. You can choose which to use with parameters, you can also choose how many layers to be used, how many units, whether to use BatchNormalization, or dropout and so many others to be choosen to build more advanced model."},{"metadata":{"trusted":true,"_uuid":"b112277ca960a14fc82a5c160f6ca23530ab0466"},"cell_type":"code","source":"class lstmNet(object):\n    def __init__(self, n_classes=2, input_dim1=None, input_dim2=None, n_layers=3, use_dropout=True, drop_ratio=.5,\n                 use_bidirec=False, use_gru=False, rnn_units=64, use_dense=True, dense_units=64, use_batch=True,\n                 metrics='accuracy', optimizer='rmsprop', use_em=False, em_weights=None, em_input_dim=None, fit_split=False):\n        self.n_classes = n_classes\n        self.input_dim1 = input_dim1\n        self.input_dim2 = input_dim2\n        self.n_layers = n_layers\n        self.use_dropout = use_dropout\n        self.drop_ratio = drop_ratio\n        self.use_bidierc = use_bidirec\n        self.use_gru = use_gru\n        self.rnn_units = rnn_units\n        self.use_dense = use_dense\n        self.use_batch = use_batch\n        self.dense_units = dense_units\n        self.metrics = metrics\n        self.optimizer = optimizer\n        self.use_em = use_em\n        self.em_weights = em_weights\n        self.em_input_dim = em_input_dim\n        self.fit_split = fit_split\n        self.is_gpu = tf.test.is_gpu_available()\n        self.model = self._init_model()\n\n    def _init_model(self):\n        \n        if self.use_em:\n            inputs =  Input(shape=(self.em_input_dim, ))\n        else: inputs = Input(shape=(self.input_dim1, self.input_dim2))\n\n        def _lstm_block(layers, name_index=None):\n            if self.use_bidierc:\n                if self.is_gpu:\n                    res = Bidirectional(CuDNNLSTM(self.rnn_units, return_sequences=True),\n                                        name='bidi_lstm_'+str(name_index))(layers)\n                else: res = Bidirectional(LSTM(self.rnn_units, return_sequences=True,\n                                         recurrent_dropout=self.drop_ratio), name='bidi_lstm_'+str(name_index))(layers)\n            elif self.use_gru:\n                if self.is_gpu:\n                    res = CuDNNGRU(self.rnn_units, return_sequences=True)(layers)\n                else: res = GRU(self.rnn_units, return_sequences=True,\n                          recurrent_dropout=self.drop_ratio, name='gru_'+str(name_index))(layers)\n            else:\n                if self.is_gpu:\n                    res = CuDNNLSTM(self.rnn_units, return_sequences=True)(layers)\n                else: res = LSTM(self.rnn_units, return_sequences=True,\n                           recurrent_dropout=self.drop_ratio, name='lstm_'+str(name_index))(layers)\n\n            if self.use_dropout:\n                res = Dropout(self.drop_ratio)(res)\n\n            return res\n\n        # No matter for LSTM, GRU, bidirection LSTM, final layer can not use 'return_sequences' output.\n        for i in range(self.n_layers - 1):\n            if i == 0:\n                if self.use_em:\n                    res = Embedding(max_features, em_size)(inputs)\n                    res = _lstm_block(res, name_index=i)\n                else: res = _lstm_block(inputs, name_index=i)\n            else:\n                res = _lstm_block(res, name_index=i)\n\n        # final LSTM layer\n        if self.use_bidierc:\n            if self.is_gpu:\n                res = Bidirectional(CuDNNLSTM(self.rnn_units))(res)\n            else: res = Bidirectional(LSTM(self.rnn_units), name='bire_final')(res)\n        elif self.use_gru:\n            if self.is_gpu:\n                res = CuDNNGRU(self.rnn_units)(res)\n            else:res = GRU(self.rnn_units, name='gru_final')(res)\n        else:\n            if self.is_gpu:\n                res = CuDNNLSTM(self.rnn_units)(res)\n            else:\n                res = LSTM(self.rnn_units, name='lstm_final')(res)\n\n        # whether or not to use Dense layer\n        if self.use_dense:\n            res = Dense(self.dense_units, name='dense_1')(res)\n            if self.use_batch:\n                res = BatchNormalization(name='batch_1')(res)\n            res = Activation('relu')(res)\n            if self.use_dropout:\n                res = Dropout(self.drop_ratio)(res)\n\n        if self.n_classes == 2:\n            out = Dense(1, activation='sigmoid', name='out')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax', name='out')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameter n_class must be provide up or equals to 2!')\n\n        return model\n\n    def fit(self, data, label, epochs=100, batch_size=256, vali_data=None, vali_label=None):\n        #label = check_label_shape(label)\n        if self.fit_split:\n            xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n            self.his = self.model.fit(xtrain, ytrain, epochs=epochs, batch_size=batch_size, verbose=1,\n                                      validation_data=(xvalidate, yvalidate))\n            print('Model evaluation on validation datasets accuracy:{:.2f}'.format(\n                self.model.evaluate(xvalidate, yvalidate)[1]*100))\n        else: self.his = self.model.fit(data, label,\n                                        epochs=epochs, batch_size=batch_size, verbose=1, \n                                        validation_data=(vali_data,vali_label))\n        return self\n\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        #label = check_label_shape(label)\n\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.2f}'.format(acc*100))\n        return acc\n    \n    def predict(self, data, batch_size=None):\n        return self.model.predict(data, batch_size=batch_size)    \n\n    def plot_acc_curve(self, plot_acc=True, plot_loss=True, figsize=(8, 6)):\n        style.use('ggplot')\n\n        if plot_acc:\n            fig1, ax1 = plt.subplots(1, 1, figsize=figsize)\n            ax1.plot(self.his.history['acc'], label='Train accuracy')\n            if self.fit_split:\n                ax1.plot(self.his.history['val_acc'], label='Validation accuracy')\n            ax1.set_title('Train and validation accuracy curve')\n            ax1.set_xlabel('Epochs')\n            ax1.set_ylabel('Accuracy score')\n            plt.legend()\n\n        if plot_loss:\n            fig2, ax2 = plt.subplots(1, 1, figsize=(8, 6))\n            ax2.plot(self.his.history['loss'], label='Train Loss')\n            if self.fit_split:\n                ax2.plot(self.his.history['val_loss'], label='Validation Loss')\n            ax2.set_title('Train and validation loss curve')\n            ax2.set_xlabel('Epochs')\n            ax2.set_ylabel('Loss score')\n            plt.legend()\n\n        plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4bae8ff8aaa10fa73d648e0220cab4a9515852ed"},"cell_type":"markdown","source":"#### Here is a more advanced model sturcture: DenseNet.  You can also choose to use basic residual or dense residual, also with Dropout and BatchNormalization to be choosen. You can check this class to find which can be usd."},{"metadata":{"trusted":true,"_uuid":"8f67d30490b48623088ec9d884763acffcd4caf9"},"cell_type":"code","source":"class denseNet(object):\n    def __init__(self, input_dim1=None, input_dim2=None, n_classes=2, basic_residual=False, n_layers=4, flatten=True, use_dense=True,\n                 n_dense_layers=1, conv_units=64, stride=1, padding='SAME', dense_units=128, drop_ratio=.5,\n                 optimizer='rmsprop', metrics='accuracy', use_em=False, em_weights=None, em_input_dim=None, fit_split=False):\n        self.input_dim1 = input_dim1\n        self.input_dim2 = input_dim2\n        self.n_classes = n_classes\n        self.basic_residual = basic_residual\n        self.n_layers = n_layers\n        self.flatten = flatten\n        self.use_dense = use_dense\n        self.n_dense_layers = n_dense_layers\n        self.conv_units = conv_units\n        self.stride = stride\n        self.padding = padding\n        self.dense_units = dense_units\n        self.drop_ratio = drop_ratio\n        self.optimizer = optimizer\n        self.metrics = metrics\n        self.use_em = use_em\n        self.em_weights = em_weights\n        self.em_input_dim = em_input_dim\n        self.fit_split = fit_split\n        self.model = self._init_model()\n\n    # this will build DenseNet or ResidualNet structure, this model is already compiled.\n    def _init_model(self):\n        \n        if self.use_em:\n            inputs = Input(shape=(self.em_input_dim, ))\n        else:\n            inputs = Input(shape=(self.input_dim1, self.input_dim2))\n\n        # dense net residual block\n        def _res_block(layers, added_layers=inputs):\n            res = Conv1D(self.conv_units, self.stride, padding=self.padding)(layers)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n            \n            if self.use_em:\n                res = Conv1D(self.em_input_dim, self.stride, padding=self.padding)(res)\n            else: res = Conv1D(self.input_dim2, self.stride, padding=self.padding)(res)\n            res = BatchNormalization()(res)\n            res = Activation('relu')(res)\n            res = Dropout(self.drop_ratio)(res)\n\n            if self.basic_residual:\n                return keras.layers.add([res, layers])\n            else:\n                return keras.layers.add([res, added_layers])\n\n        # construct residual block chain.\n        for i in range(self.n_layers):\n            if i == 0:\n                if self.use_em:\n                    res = Embedding(max_features, em_size, weights=[self.em_weights])(inputs)\n                    res = _res_block(res)\n                else: res = _res_block(inputs)\n            else:\n                res = _res_block(res)\n\n        # using flatten or global average pooling to process Convolution result\n        if self.flatten:\n            res = Flatten()(res)\n        else:\n            res = GlobalAveragePooling1D()(res)\n\n        # whether or not use dense net, also with how many layers to use\n        if self.use_dense:\n            for j in range(self.n_dense_layers):\n                res = Dense(self.dense_units)(res)\n                res = BatchNormalization()(res)\n                res = Activation('relu')(res)\n                res = Dropout(self.drop_ratio)(res)\n\n        if self.n_classes == 2:\n            out = Dense(1, activation='sigmoid')(res)\n            model = Model(inputs, out)\n            print('Model structure:')\n            model.summary()\n            model.compile(loss='binary_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        elif self.n_classes > 2:\n            out = Dense(self.n_classes, activation='softmax')(res)\n            model = Model(inputs, out)\n            print('Model Structure:')\n            model.summary()\n            model.compile(loss='categorical_crossentropy', metrics=[self.metrics], optimizer=self.optimizer)\n        else:\n            raise AttributeError('parameters n_classes must up to 2!')\n\n        return model\n\n    # Fit on given training data and label. Here I will auto random split the data to train and validation data,\n    # for test datasets, I will just use it if model already trained then I will evaluate the model.\n    def fit(self, data, label, epochs=100, batch_size=256, vali_data=None, vali_label=None):\n        # self.model = self._init_model()\n        if self.fit_split:   # Whether or not to split the training data\n            xtrain, xvalidate, ytrain, yvalidate = train_test_split(data, label, test_size=.2, random_state=1234)\n            self.his = self.model.fit(xtrain, ytrain, verbose=1, epochs=epochs,\n                                 validation_data=(xvalidate, yvalidate), batch_size=batch_size)\n            print('After training, model accuracy on validation datasets is {:.4f}'.format(self.model.evaluate(xvalidate, yvalidate)[1]))\n        else:    \n            self.his = self.model.fit(data, label, verbose=1, \n                                      epochs=epochs, batch_size=batch_size, validation_data=(vali_data, vali_label))\n        return self\n\n    # evaluate model on test datasets.\n    def evaluate(self, data, label, batch_size=None, silent=False):\n        acc = self.model.evaluate(data, label, batch_size=batch_size)[1]\n        if not silent:\n            print('Model accuracy on Testsets : {:.6f}'.format(acc))\n        return acc\n    \n    def predict(self, data, batch_size=None):\n        return self.model.predict(data, batch_size=batch_size)\n    \n    # plot after training accuracy and loss curve.\n    def plot_acc_curve(self):\n        style.use('ggplot')\n\n        fig1, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['acc'], label='Train Accuracy')\n        if self.fit_split:\n            ax.plot(self.his.history['val_acc'], label='Validation Accuracy')\n        ax.set_title('Train and Validation Accruacy Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Accuracy score')\n        plt.legend()\n\n        fig2, ax = plt.subplots(1, 1, figsize=(8, 6))\n        ax.plot(self.his.history['loss'], label='Traing Loss')\n        if self.fit_split:\n            ax.plot(self.his.history['val_loss'], label='Validation Loss')\n        ax.set_title('Train and Validation Loss Curve')\n        ax.set_xlabel('Epochs')\n        ax.set_ylabel('Loss score')\n\n        plt.legend()\n        plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2d1660213e3609696e978437545ce2792ec01f0"},"cell_type":"code","source":"del train_new, train, train_data, validation_data\nimport gc\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab862e92c2c103cb9b5598b87bc0441d078aa278"},"cell_type":"markdown","source":"#### After I have build all advanced model structure, then I should get the pretrained embeddings.\n#### Above code just uses Doc2vec algorithm to get the features vector. Here I want to use the embedding result provived by competition.  There are 4 embeddings:\n* GoogleNews-vectors-negative300 - https://code.google.com/archive/p/word2vec/\n* glove.840B.300d - https://nlp.stanford.edu/projects/glove/\n* paragram_300_sl999 - https://cogcomp.org/page/resource_view/106\n* wiki-news-300d-1M - https://fasttext.cc/docs/en/english-vectors.html\n#### Here is Glove."},{"metadata":{"trusted":true,"_uuid":"1febca8e33e3bad01beee9dae3f1f3ef60df3718"},"cell_type":"code","source":"em_file = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word, *arr):\n    return word, np.asarray(arr, dtype='float32')\nembedding_index = dict(get_coefs(*d.split(' ')) for d in open(em_file))\n\nall_embs = np.stack(embedding_index.values())\nem_mean, em_std = all_embs.mean(), all_embs.std()\nem_size = all_embs.shape[1]\n\nword_index = token.word_index\nnb_words = min(max_features, len(word_index))\nem_matrix = np.random.normal(em_mean, em_std, (nb_words, em_size))\n# loop for every word\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    em_v = embedding_index.get(word)\n    if em_v is not None:\n        em_matrix[i] = em_v\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"773cdaafe2a30fa1ed8906d2b67a6bfb6c3af2b5"},"cell_type":"code","source":"# # Before fitting model, convert label to be 2D\n# ytrain_deep = keras.utils.to_categorical(ytrain)\n# yvaliate_deep = keras.utils.to_categorical(yvaliate)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbcabb15955dff12fc118987dce93a585edc7986"},"cell_type":"markdown","source":"#### Here is DNN model structure, you can tune the model parameters as you like to find whether can get better result."},{"metadata":{"trusted":true,"_uuid":"6269940156365769613aeb3931f84b402e5f6095","scrolled":true},"cell_type":"code","source":"# # Build DNN model\n# model_dnn = dnnNet(n_classes=2, n_dims=max_length, use_em=True, n_layers=3, n_units=512,\n#                    em_weights=em_matrix, em_input_dim=em_matrix.shape[1])\n# # Fit DNN model\n# model_dnn.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\n# # Evaluate DNN model based on validataion data\n# model_dnn.evaluate(xvalidate, yvaliate)\n\n# # Plot learning curve and validate curve\n# model_dnn.plot_acc_curve()\n\n# # Use trained model to make prediction based on validation data\n# pred_dnn = model_dnn.predict(xvalidate, batch_size=4092)\n\n# # Evaluate model result based on different threshold\n# evaluate(yvaliate, pred_dnn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4e5da97170d2946a47f42d345778c511458643a"},"cell_type":"markdown","source":"####  Here is DenseNet.\n#### Noted: You have to turn GPU on before you run this code(otherwise you may need over 1 hour to train one epoch!)."},{"metadata":{"trusted":true,"_uuid":"a2ffd08caa9ba5163b656c2e549cb10811894cc1","scrolled":true},"cell_type":"code","source":"# model_dense = denseNet(n_classes=2, use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\n\n# model_dense.fit(xtrain, ytrain , epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\n# model_dense.evaluate(xvalidate, yvaliate, batch_size=10240)\n\n# pred_dense = model_dense.predict(xvalidate)\n\n# evaluate(yvaliate, pred_dense)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1556adbefe9273dd28b78e65537363ef3f12fc99"},"cell_type":"markdown","source":"#### Here I find that for DNN and DenseNet model is not good at fitting on this datasets. So if you want to use them, you can tune them with different parameters."},{"metadata":{"_uuid":"fa9ae116e39d7474fd67d6a77b3dfc8b186a0cd6"},"cell_type":"markdown","source":"#### So according to the confusion matrix result, we can get that denseNet for now didn't make a good result(we have a high false negative rate!). Maybe we need to tune the class parameters to make a more powerful model to fit on this dataset!\n#### Noted: YOU have to turn GPU on, or you may need more than one hour to train it. I will also auto convert the model to fit on GPU."},{"metadata":{"trusted":true,"_uuid":"766fb55906cd71e6158aa0cf9a570dd715ac11e3"},"cell_type":"markdown","source":"#### Here is LSTM model. You can tune the model parameters as you like."},{"metadata":{"trusted":true,"_uuid":"3c6cbea15f22d55fcf3d58f53b4ed2c97531df9c"},"cell_type":"code","source":"model_lstm = lstmNet(n_classes=2, drop_ratio=.1,\n                     use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\nmodel_lstm.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\npred_lstm_glove = model_lstm.predict(xvalidate, batch_size=10240)\nevaluate(yvaliate, pred_lstm_glove)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"798f4be997ce60f66a39a68357127884a524349b"},"cell_type":"markdown","source":"#### Here I build a bidirectional LSTM to fit."},{"metadata":{"trusted":true,"_uuid":"c8422861a6273b15c7c89fb43d5ee5712019954e"},"cell_type":"code","source":"model_bidi_lstm = lstmNet(n_classes=2, use_bidirec=True, dense_units=128, drop_ratio=.1,\n                          use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\n\nmodel_bidi_lstm.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\npred_bidi_lstm_glove = model_lstm.predict(xvalidate, batch_size=10240)\nevaluate(yvaliate, pred_bidi_lstm_glove)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b64f75c38bd4570a49d38bea8112d0d0f2d326f"},"cell_type":"markdown","source":"#### Get LSTM and Bidi-LSTM prediction on testsets."},{"metadata":{"trusted":true,"_uuid":"31b6e374cf5b41c2e26d882cb663c69be924b0a8"},"cell_type":"code","source":"lstm_glove = model_lstm.predict(xtest)\nbidi_lstm_glove = model_bidi_lstm.predict(xtest)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cb1de9c1756fee9e829126c5e79d15f336b2b19"},"cell_type":"code","source":"del model_lstm, model_bidi_lstm\ndel embedding_index, all_embs, word_index, em_matrix\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"026e643be8554d360ff4de3b40b066c688a1d752"},"cell_type":"markdown","source":"#### Here is wiki-news"},{"metadata":{"trusted":true,"_uuid":"e18bf7f0f95927e2e70c95d0a93622379b15ac69"},"cell_type":"code","source":"em_file = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\ndef get_coefs(word, *arr):\n    return word, np.asarray(arr, dtype='float32')\nembedding_index = dict(get_coefs(*o.split(\" \")) for o in open(em_file) if len(o)>100)\n\nall_embs = np.stack(embedding_index.values())\nem_mean, em_std = all_embs.mean(), all_embs.std()\nem_size = all_embs.shape[1]\n\nword_index = token.word_index\nnb_words = min(max_features, len(word_index))\nem_matrix = np.random.normal(em_mean, em_std, (nb_words, em_size))\n# loop for every word\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    em_v = embedding_index.get(word)\n    if em_v is not None:\n        em_matrix[i] = em_v\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"66b34de57956bc388bdf00797d243b278836b519"},"cell_type":"code","source":"model_lstm = lstmNet(n_classes=2, drop_ratio=.1,\n                     use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\nmodel_lstm.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\npred_lstm_wiki = model_lstm.predict(xvalidate, batch_size=10240)\nevaluate(yvaliate, pred_lstm_wiki)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"51a0aa426c292256fbfce3a6d69fda8a6486d041"},"cell_type":"code","source":"model_bidi_lstm = lstmNet(n_classes=2, use_bidirec=True, dense_units=128, drop_ratio=.1,\n                          use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\n\nmodel_bidi_lstm.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\npred_bidi_lstm_wiki = model_lstm.predict(xvalidate, batch_size=10240)\nevaluate(yvaliate, pred_bidi_lstm_wiki)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"38bb3bb940aac5d2445130e6950ad38f178918bb"},"cell_type":"code","source":"lstm_wiki = model_lstm.predict(xtest)\nbidi_lstm_wiki = model_bidi_lstm.predict(xtest)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fd6c34a08808446c953c96c775934237e73fbca"},"cell_type":"code","source":"del model_lstm, model_bidi_lstm\ndel embedding_index, all_embs, word_index, em_matrix\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51ebff7eefe98d3b68a6c05a2d3a952f8f4050eb"},"cell_type":"markdown","source":"#### Here is Paragram embeddings"},{"metadata":{"trusted":true,"_uuid":"ba20073de01ff9d5b17960838ccbe39c62b31388"},"cell_type":"code","source":"em_file = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\ndef get_coefs(word, *arr):\n    return word, np.asarray(arr, dtype='float32')\nembedding_index = dict(get_coefs(*o.split(\" \")) for o in open(em_file, encoding=\"utf8\", errors='ignore') if len(o)>100)\n\nall_embs = np.stack(embedding_index.values())\nem_mean, em_std = all_embs.mean(), all_embs.std()\nem_size = all_embs.shape[1]\n\nword_index = token.word_index\nnb_words = min(max_features, len(word_index))\nem_matrix = np.random.normal(em_mean, em_std, (nb_words, em_size))\n# loop for every word\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    em_v = embedding_index.get(word)\n    if em_v is not None:\n        em_matrix[i] = em_v\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0d20774c96f5ff6d9ac4e1c60bce1144cfb9ece2"},"cell_type":"code","source":"model_lstm = lstmNet(n_classes=2, drop_ratio=.1,\n                     use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\nmodel_lstm.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\npred_lstm_para = model_lstm.predict(xvalidate, batch_size=10240)\nevaluate(yvaliate, pred_lstm_para)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f5f1407f08f90735ce47d456bb51d85ba3dec2b"},"cell_type":"code","source":"model_bidi_lstm = lstmNet(n_classes=2, use_bidirec=True, dense_units=128, drop_ratio=.1,\n                          use_em=True, em_weights=em_matrix, em_input_dim=max_length, optimizer='adam')\n\nmodel_bidi_lstm.fit(xtrain, ytrain, epochs=2, batch_size=512, vali_data=xvalidate, vali_label=yvaliate)\n\npred_bidi_lstm_para = model_lstm.predict(xvalidate, batch_size=10240)\nevaluate(yvaliate, pred_bidi_lstm_para)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"edf98e628048c5557d38de63cef171d0d6fea733"},"cell_type":"code","source":"lstm_para = model_lstm.predict(xtest)\nbidi_lstm_para = model_bidi_lstm.predict(xtest)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c03159e5995e23e22166f80e5860219d357d045"},"cell_type":"code","source":"del model_lstm, model_bidi_lstm\ndel embedding_index, all_embs, word_index, em_matrix\ngc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fab3836668132f08af3b082cc941c9826c172dcd"},"cell_type":"markdown","source":"#### Because of time limited, here I will make the submition file combining LSTM and Bidirectional LSTM to make the final result. \n#### If I have time, I will also use other embeddings to fit on this advanced model.\n#### Noted: you have to get the better threshold by using 'evaluate' function to get best f1-score. Then you use this threshold. But for now, I will use 0.32"},{"metadata":{"_uuid":"efa23b5a8c5cf53f103b953eb5adbfb6a7979c55"},"cell_type":"markdown","source":"Here I use a Linear Regression model to fit different model output to get different model weights(Before I just random give weights for different model). \nThanks to this kernel(you can also check it):https://www.kaggle.com/suicaokhoailang/blending-with-linear-regression-0-688-lb"},{"metadata":{"trusted":true,"_uuid":"4bf8daaf7a3938da384d97f4fef8129322b498e1"},"cell_type":"code","source":"model_num = 6\nfrom sklearn.linear_model import LinearRegression\nlr = LinearRegression()\npred_result = np.empty([len(pred_lstm_glove), model_num])\npred_result[:, 0] = pred_lstm_glove.reshape(-1, )\npred_result[:, 1] = pred_bidi_lstm_glove.reshape(-1, )\npred_result[:, 2] = pred_lstm_wiki.reshape(-1, )\npred_result[:, 3] = pred_bidi_lstm_wiki.reshape(-1, )\npred_result[:, 4] = pred_lstm_para.reshape(-1, )\npred_result[:, 5] = pred_bidi_lstm_para.reshape(-1, )\n\nlr.fit(pred_result, yvaliate)\n\nweights = lr.coef_\n\n# Here is just used for get combining result, and best threshold.\nsub_pred_weighted = np.sum([pred_result[:, i]*weights[i] for i in range(model_num)], axis=0)\nbest_thre = evaluate(yvaliate ,sub_pred_weighted)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b91f498ca62d91cd146ede72c4ef2d767f8ddce0"},"cell_type":"code","source":"# GET different model prediction result on test datasets\nsub_data = np.empty([len(xtest), model_num])\nsub_data[:, 0] = lstm_glove.reshape(-1, )\nsub_data[:, 1] = bidi_lstm_glove.reshape(-1, )\nsub_data[:, 2] = lstm_wiki.reshape(-1, )\nsub_data[:, 3] = bidi_lstm_glove.reshape(-1, )\nsub_data[:, 4] = lstm_para.reshape(-1, )\nsub_data[:, 5] = bidi_lstm_para.reshape(-1, )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46a14454e4800490b198ad00f610ba0122897063"},"cell_type":"code","source":"\n#sub_pred = 0.1 * pred_lstm_glove + 0.2*pred_bidi_lstm_glove + 0.1*pred_lstm_wiki + 0.2*pred_bidi_lstm_wiki+0.1*pred_lstm_para+0.3*pred_lstm_para\n# According to Linear Regression model result with different weights multiply with prediction.\nsub_pred = np.sum([sub_data[:, i]*weights[i] for i in range(model_num)], axis=0)\nsub_pred = (sub_pred > best_thre).astype(int)\n\nsub_df = pd.DataFrame({'qid':test.qid.values})\nsub_df['prediction'] = sub_pred\nsub_df.to_csv('submission.csv', index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4752e0b480e6f52217980523b279987f9c82546"},"cell_type":"markdown","source":"#### So there are many advanced deep neural networks that can be used for this problem. Here I just use some advanced deep learning models, you can also choose other model structure. There are also some great kernel that use provived features. Here they are:\n\nWith attention: https://www.kaggle.com/shujian/different-embeddings-with-attention-fork-fork \n2.All features used: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\n#### If you have time, you can check them out. Use the already trained vectors combined with different deep learning model structure to fit.\n#### Here is my github probject: https://github.com/lugq1990/neural-nets\n#### Thanks for your support!"},{"metadata":{"trusted":true,"_uuid":"f7d7f5e084fb53457ef06fe18b2ef28c57c1f18e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}