{"cells":[{"metadata":{"_uuid":"f5eb78d2e2c25a36d172b552d1b62af9edc6de42"},"cell_type":"markdown","source":"## Introduction\n\nAs this is my first competition that I'm \"seriously\" competing in (actually more trying to really learn all the ins and outs of different Neural Network architectures and how to apply them to NLP), and I've learned a lot from several kernels posted on this competition, I thought that I should share some learned experience as well.\n\nThe code below is intended to do a hyperparameter search over some defined model using hyperopt. It splits the training set up into a set to train the model on and a holdout set to test the model on. I'm optimising on the F1 score of the test set, so as to gauge how it would perform on the leaderboard in this competition. From the couple of runs that I submitted the configuration found as optimal by the model, the (optimized) F1 score returned by the model and the leaderboard matched very closely, so I feel like this could be a good way of testing the performance of your model. However, I didn't make an exhaustive study of this so I might have just gotten lucky on my couple of tries, so no guarantee that this will also work for you :).\n\n**Important Note** : This notebook is not intended to run on a kaggle kernel, since the time it will take to run the entire grid search (depending on the number of `max_evals` you pass to `hyperopt.fmin` of course) will take more time than the kernels on Kaggle will allow. What I did is open a google cloud account (you get 300 USD free, so that's pretty sweet), and I used the following instance from the marketplace to run this notebook on: http://jetware.io/appliances/aise/tensorflow110_keras22_python36_cuda92_notebook-180916. You can also create an instance yourself of course, but this marketplace solution makes it very easy to use GPUs on the Google Cloud Platform (trust me, I tried to create my own instance and was happy to find this instance after a couple of hours of frustration, but maybe that's just me).\n\n### Credits \n\nA lot of the code below is borrowed from other kernels, for which I have to thank the amazing authors, these are:\n>  https://www.kaggle.com/gmhost/gru-capsule (including the comments from @theoviel)\n\n>  https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/74214\n\n>  https://www.kaggle.com/shujian/single-rnn-with-4-folds-clr\n\n>  https://www.kaggle.com/theoviel/improve-your-score-with-text-preprocessing-v2\n\n>  https://www.kaggle.com/suicaokhoailang/beating-the-baseline-with-one-weird-trick-0-691\n\n>  https://www.kaggle.com/inspector/keras-hyperopt-example-sketch\n\n>  https://www.kaggle.com/spirosrap/bilstm-attention-kfold-clr-extra-features-capsule\n\nApologies if I missed your contribution (there's a lot of amazing kernels out there), please notify me in the comments and I will update the credits :).\n\n\nAll of the below is probably pretty basic for many people here, but I thought I'd just share it for other to learn from. And perhaps I made a couple of mistakes and I'll learn from someone seeing this kernel :). Anyhow, hope it helps"},{"metadata":{"ExecuteTime":{"end_time":"2019-01-10T11:12:20.042112Z","start_time":"2019-01-10T11:12:16.511636Z"},"trusted":true,"_uuid":"00ac7275757de046ec5b6251fca59f03ed16225b"},"cell_type":"code","source":"import sklearn\nimport tensorflow as tf\nimport numpy as np\nimport keras\nimport pandas as pd\n\nimport seaborn as sns\nsns.set_style('whitegrid')\n\nfrom sklearn.model_selection import StratifiedKFold, train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, CuDNNLSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D, LSTM, GRU\nfrom keras.layers import Bidirectional, GlobalMaxPool1D, GlobalMaxPooling1D, GlobalAveragePooling1D\nfrom keras.layers import Input, Embedding, Dense, Conv2D, MaxPool2D, concatenate\nfrom keras.layers import Reshape, Flatten, Concatenate, Dropout, SpatialDropout1D, BatchNormalization\nfrom keras.optimizers import *\nfrom keras.initializers import *\nfrom keras.activations import *\nfrom keras.callbacks import *\nfrom keras.models import Model\nfrom keras import backend as K\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers\n\nimport datetime\nimport timeit\n\nfrom keras.preprocessing import sequence\nfrom keras.models import Sequential\nfrom keras.layers.core import Dense, Dropout, Activation\nfrom keras.layers.embeddings import Embedding\nfrom keras.layers.recurrent import LSTM\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\n\nimport re\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ea4f8b6b0a36e6b7ade5690cbd1aa2f9ba88945"},"cell_type":"code","source":"## some config values \nmax_features = 95000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 70 # max number of words in a question to use\nembed_size = 300\nrun_name = 'LSTM_GTU_Attention_class_balance'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cfa104a7bfe5ef3b49630973c812068f0f39bbf","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\n\ndef clean_text(train_df):\n    print(\"Cleaning text data...\")\n\n    mispell_dict = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",  \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\", \"colour\": \"color\", \"centre\": \"center\", \"favourite\": \"favorite\", \"travelling\": \"traveling\", \"counselling\": \"counseling\", \"theatre\": \"theater\", \"cancelled\": \"canceled\", \"labour\": \"labor\", \"organisation\": \"organization\", \"wwii\": \"world war 2\", \"citicise\": \"criticize\", \"youtu \": \"youtube \", \"Qoura\": \"Quora\", \"sallary\": \"salary\", \"Whta\": \"What\", \"narcisist\": \"narcissist\", \"howdo\": \"how do\", \"whatare\": \"what are\", \"howcan\": \"how can\", \"howmuch\": \"how much\", \"howmany\": \"how many\", \"whydo\": \"why do\", \"doI\": \"do I\", \"theBest\": \"the best\", \"howdoes\": \"how does\", \"mastrubation\": \"masturbation\", \"mastrubate\": \"masturbate\", \"mastrubating\": 'masturbating', \"pennis\": \"penis\", \"Etherium\": \"Ethereum\", \"narcissit\": \"narcissist\", \"bigdata\": \"big data\", \"2k17\" : \"2017\", \"2k18\": \"2018\", \"qouta\": \"quota\", \"exboyfriend\" : \"ex boyfriend\", \"airhostess\" : \"air hostess\", \"whst\": \"what\", \"watsapp\": \"whatsapp\", \"demonitisation\": \"demonetization\", \"demonitization\": \"demonetization\", \"demonetisation\": \"demonetization\"}\n\n    def _get_mispell(mispell_dict):\n        mispell_re = re.compile(\"(%s)\" % \"|\".join(mispell_dict.keys()))\n        return mispell_dict, mispell_re\n\n    mispellings, mispellings_re = _get_mispell(mispell_dict)\n    def replace_typical_misspell(text):\n        def replace(match):\n            return mispellings[match.group(0)]\n        return mispellings_re.sub(replace, text)\n\n    # Lower the text\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.lower()\n\n    # Clean numbers\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"[0-9]{5,}\", r\"#####\")\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"[0-9]{4}\", r\"####\")\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"[0-9]{3}\", r\"###\")\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"[0-9]{2}\", r\"##\")\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"[0-9]*\\.[0-9]*\", r\"##\")\n\n    # Clean spellings\n    train_df[\"question_text\"] = train_df[\"question_text\"].apply(lambda x: replace_typical_misspell(x))\n    \n    # Clean the text\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"([^\\w\\s\\'\\\"])\", r\" \\1 \")\n    train_df[\"question_text\"] = train_df[\"question_text\"].str.replace(r\"\\s{2,}\", r\" \")\n    \n    return train_df\n\ntrain_df = clean_text(train_df)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-01-10T11:12:25.705814Z","start_time":"2019-01-10T11:12:23.132388Z"},"trusted":true,"_uuid":"4378a162e560ad6f8ccd7dfb27aca8719c8772c1"},"cell_type":"code","source":"def data(train_df):\n    \n    #train_df = pd.read_csv(\"input/train_clean.csv\")\n\n    X = train_df[\"question_text\"].values\n    y = train_df[\"target\"].values\n     \n    ## Tokenize the sentences\n    tokenizer = Tokenizer(num_words=max_features)\n    tokenizer.fit_on_texts(list(X))\n    X = tokenizer.texts_to_sequences(X)\n\n    ## Pad the sentences \n    X = pad_sequences(X, maxlen=maxlen)\n    \n    # Make a small train test split to somehow evaluate the model accuracy. \n    # As this data is noisy and this won't be the same as the test set in the \n    # competition, I'm not really sure if this is a great idea, but we'll run with it\n    \n    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.15, random_state = 42, \\\n                                                        shuffle = True, stratify = y)\n    \n    ###\n    # Embedding loading\n    ###\n    \n    #####\n    ### GLOVE\n    #####\n    \n    word_index = tokenizer.word_index\n    \n    print(\"Loading GloVe embedding...\")\n    \n    EMBEDDING_FILE = \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype=\"float32\")\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = -0.005838499,0.48782197\n    embed_size = all_embs.shape[1]\n\n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index))\n    embedding_matrix_glove = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix_glove[i] = embedding_vector\n    \n    del embeddings_index, all_embs, emb_mean, emb_std, nb_words, embedding_vector\n    \n    print(\"GloVe embedding loaded...\")\n    \n    ######\n    ### PARAGRAM Embedding\n    ######\n    \n    print(\"Loading Paragram embedding...\")\n    \n    EMBEDDING_FILE = \"../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt\"\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype=\"float32\")\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors=\"ignore\") if len(o)>100)\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = -0.0053247833,0.49346462\n    embed_size = all_embs.shape[1]\n    #print(emb_mean,emb_std,\"para\")\n\n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index))\n    embedding_matrix_para = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix_para[i] = embedding_vector\n    \n    del embeddings_index, all_embs, emb_mean, emb_std, nb_words, embedding_vector \n    print(\"Paragram embedding loaded...\")\n    \n    print(\"Concatenating embedding matrices...\")\n    \n    embedding_matrix = np.mean([embedding_matrix_para, embedding_matrix_glove], axis = 0)\n    \n    del embedding_matrix_para, embedding_matrix_glove\n    \n    print(\"Data loading done...\")\n    \n    return X_train, X_test, y_train, y_test, embedding_matrix #, max_features, maxlen, embed_size    \n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8db263246cf3eb38647be580c77ef0c02b02f8d"},"cell_type":"markdown","source":"## Extra model definitions"},{"metadata":{"trusted":true,"_uuid":"e1d4fa173e07f39fc0998e1e61533147481b0962"},"cell_type":"code","source":"####\n# Extra classes for the training\n####\n\nclass Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get(\"glorot_uniform\")\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim\n\nclass CyclicLR(Callback):\n    \"\"\"This callback implements a cyclical learning rate policy (CLR).\n    The method cycles the learning rate between two boundaries with\n    some constant frequency, as detailed in this paper (https://arxiv.org/abs/1506.01186).\n    The amplitude of the cycle can be scaled on a per-iteration or \n    per-cycle basis.\n    This class has three built-in policies, as put forth in the paper.\n    \"triangular\":\n        A basic triangular cycle w/ no amplitude scaling.\n    \"triangular2\":\n        A basic triangular cycle that scales initial amplitude by half each cycle.\n    \"exp_range\":\n        A cycle that scales initial amplitude by gamma**(cycle iterations) at each \n        cycle iteration.\n    For more detail, please see paper.\n\n    # Example\n        ```python\n            clr = CyclicLR(base_lr=0.001, max_lr=0.006,\n                                step_size=2000., mode='triangular')\n            model.fit(X_train, Y_train, callbacks=[clr])\n        ```\n\n    Class also supports custom scaling functions:\n        ```python\n            clr_fn = lambda x: 0.5*(1+np.sin(x*np.pi/2.))\n            clr = CyclicLR(base_lr=0.001, max_lr=0.006,\n                                step_size=2000., scale_fn=clr_fn,\n                                scale_mode='cycle')\n            model.fit(X_train, Y_train, callbacks=[clr])\n        ```    \n    # Arguments\n        base_lr: initial learning rate which is the\n            lower boundary in the cycle.\n        max_lr: upper boundary in the cycle. Functionally,\n            it defines the cycle amplitude (max_lr - base_lr).\n            The lr at any cycle is the sum of base_lr\n            and some scaling of the amplitude; therefore \n            max_lr may not actually be reached depending on\n            scaling function.\n        step_size: number of training iterations per\n            half cycle. Authors suggest setting step_size\n            2-8 x training iterations in epoch.\n        mode: one of {triangular, triangular2, exp_range}.\n            Default 'triangular'.\n            Values correspond to policies detailed above.\n            If scale_fn is not None, this argument is ignored.\n        gamma: constant in 'exp_range' scaling function:\n            gamma**(cycle iterations)\n        scale_fn: Custom scaling policy defined by a single\n            argument lambda function, where \n            0 <= scale_fn(x) <= 1 for all x >= 0.\n            mode paramater is ignored \n        scale_mode: {'cycle', 'iterations'}.\n            Defines whether scale_fn is evaluated on \n            cycle number or cycle iterations (training\n            iterations since start of cycle). Default is 'cycle'.\n    \"\"\"\n\n    def __init__(self, base_lr=0.001, max_lr=0.006, step_size=2000., mode=\"triangular\",\n                 gamma=1., scale_fn=None, scale_mode=\"cycle\"):\n        super(CyclicLR, self).__init__()\n\n        self.base_lr = base_lr\n        self.max_lr = max_lr\n        self.step_size = step_size\n        self.mode = mode\n        self.gamma = gamma\n        if scale_fn == None:\n            if self.mode == \"triangular\":\n                self.scale_fn = lambda x: 1.\n                self.scale_mode = \"cycle\"\n            elif self.mode == \"triangular2\":\n                self.scale_fn = lambda x: 1/(2.**(x-1))\n                self.scale_mode = \"cycle\"\n            elif self.mode == \"exp_range\":\n                self.scale_fn = lambda x: gamma**(x)\n                self.scale_mode = \"iterations\"\n        else:\n            self.scale_fn = scale_fn\n            self.scale_mode = scale_mode\n        self.clr_iterations = 0.\n        self.trn_iterations = 0.\n        self.history = {}\n\n        self._reset()\n\n    def _reset(self, new_base_lr=None, new_max_lr=None,\n               new_step_size=None):\n        \"\"\"Resets cycle iterations.\n        Optional boundary/step size adjustment.\n        \"\"\"\n        if new_base_lr != None:\n            self.base_lr = new_base_lr\n        if new_max_lr != None:\n            self.max_lr = new_max_lr\n        if new_step_size != None:\n            self.step_size = new_step_size\n        self.clr_iterations = 0.\n\n    def clr(self):\n        cycle = np.floor(1+self.clr_iterations/(2*self.step_size))\n        x = np.abs(self.clr_iterations/self.step_size - 2*cycle + 1)\n        if self.scale_mode == \"cycle\":\n            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(cycle)\n        else:\n            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(self.clr_iterations)\n\n    def on_train_begin(self, logs={}):\n        logs = logs or {}\n\n        if self.clr_iterations == 0:\n            K.set_value(self.model.optimizer.lr, self.base_lr)\n        else:\n            K.set_value(self.model.optimizer.lr, self.clr())        \n\n    def on_batch_end(self, epoch, logs=None):\n\n        logs = logs or {}\n        self.trn_iterations += 1\n        self.clr_iterations += 1\n\n        self.history.setdefault(\"lr\", []).append(K.get_value(self.model.optimizer.lr))\n        self.history.setdefault(\"iterations\", []).append(self.trn_iterations)\n\n        for k, v in logs.items():\n            self.history.setdefault(k, []).append(v)\n\n        K.set_value(self.model.optimizer.lr, self.clr())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de562b4da3c18cf2849824edec94bad0c84affae"},"cell_type":"code","source":"# Data loading...\n\nX_train, X_test, y_train, y_test, embedding_matrix = data(train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2bb2c4f679c9893f79a12a40650eb92f86e4a86"},"cell_type":"markdown","source":"## Hyperopt Model"},{"metadata":{"trusted":true,"_uuid":"633add0a3dbace29b660ee3e4ba04882e0fff8fe"},"cell_type":"code","source":"# Model based on hyperopt since I'm going crazy\nimport hyperopt\nfrom hyperopt import hp, fmin, tpe, hp, STATUS_OK, Trials\n# for better class weights see : https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras\nfrom sklearn.utils import class_weight\nimport json\n\n# hp.choice('dense_layers', np.arange(20, 80, 5, dtype=int))\n\nspace = {'k_folds' : 3,\n         'dropout_1d_rate' : hp.uniform('dropout_1d_rate', 0, 1),\n         'use_LSTM_layer' : hp.choice('use_LSTM_layer', [{'use_layer' : 'no'}, \\\n                                                         {'use_layer' : 'yes', \\\n                                                          'LSTM_layers' : hp.quniform('LSTM_layers', 20, 80, 13),\n                                                          'dropout_rate_lstm' : hp.uniform('dropout_rate_lstm', 0, 1)}]),\n         'use_GRU_layer' : hp.choice('use_GRU_layer', [{'use_layer' : 'no'}, \\\n                                                       {'use_layer' : 'yes', \\\n                                                        'GRU_layers' : hp.quniform('GRU_layers', 20, 80, 13),\n                                                        'dropout_rate_gru' : hp.uniform('dropout_rate_gru', 0, 1)}]),\n         'dense_layers' : hp.quniform('dense_layers', 20, 80, 13),\n         'dropout_rate_dense' : hp.uniform('dropout_rate_dense', 0, 1),\n         'batch_size' : hp.choice('batch_size', [512, 1024, 2048]),\n         'epochs' : hp.choice('epochs', [2,3,4])\n         #,'random_seed' : # leave random seed tuning for last...\n        }\n\n# https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras\nclass_weights = class_weight.compute_class_weight('balanced',\n                                                  np.unique(y_train),\n                                                  y_train)\n\ndef f1_smart(y_true, y_pred):\n    args = np.argsort(y_pred)\n    tp = y_true.sum()\n    fs = (tp - np.cumsum(y_true[args[:-1]])) / np.arange(y_true.shape[0] + tp - 1, tp, -1)\n    res_idx = np.argmax(fs)\n    return 2 * fs[res_idx], (y_pred[args[res_idx]] + y_pred[args[res_idx + 1]]) / 2\n\n###\n# REASONING : So as I've seen in the CV vs LB score on this competition, it seems that having more \n# epochs and more folds definitely increases the CV accuracy and loss, but gives a very variable\n# score on the LB. So the idea is to see which configuration quite quickly gives a good loss/accuracy, \n# and then try that out on the LB\n\n# What is a good metric to measure the loss by? Accuracy of the here defined test function is probably\n# not too bad, because it's truly a holdout (the test set that has been split out), \n# but maybe it should be a bit bigger than 10% (made it 15%)\n\n# Version 2 (which includes switching on and off of layers, and is posted to kaggle)\n# changes the above concern by not optimizing on the accuracy but on the F1 from the hold \n# out test set, which is far closer to what happens on Kaggle.\n\ndef objective(params):\n\n    max_features = 95000 \n    maxlen = 70 \n    embed_size = 300\n    \n    print('Currently searching over : {}'.format(params))\n    \n    ###\n    # kfolds\n    ###\n    \n    kfold = StratifiedKFold(n_splits=params['k_folds'], random_state=10, shuffle=True)\n    \n    ###\n    # define model, model based on a comment in the discussion section, which I couldn't\n    # find anymore, so thanks someone!\n    ###\n    \n    K.clear_session()       \n    inp = Input(shape=(maxlen,))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(rate = params['dropout_1d_rate'])(x)\n    if params['use_LSTM_layer']['use_layer'] == 'yes':\n        x = Bidirectional(CuDNNLSTM(units = int(round(params['use_LSTM_layer']['LSTM_layers'])), return_sequences=True, \n                                    kernel_initializer=glorot_normal(seed=12300), recurrent_initializer=orthogonal(gain=1.0, seed=10000)))(x)\n        #x = Dropout(rate = params['use_LSTM_layer']['dropout_rate_lstm'])(x)\n    if params['use_GRU_layer']['use_layer'] == 'yes':\n        x = Bidirectional(CuDNNGRU(units = int(round(params['use_GRU_layer']['GRU_layers'])), return_sequences=True, \n                                   kernel_initializer=glorot_normal(seed=12300), recurrent_initializer=orthogonal(gain=1.0, seed=10000)))(x)\n        #x = Dropout(rate = params['use_GRU_layer']['dropout_rate_gru'])(x)\n\n    x = Attention(maxlen)(x)\n    x = Dense(units = int(round(params['dense_layers'])), activation=\"linear\", kernel_initializer=glorot_normal(seed=12300))(x)\n    x = Dropout(rate = params['dropout_rate_dense'])(x)\n    x = BatchNormalization()(x)\n    #x = Activation(\"relu\")(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n    model = Model(inputs=inp, outputs=x)\n    # Use all GPUs\n    model.compile(loss=\"binary_crossentropy\", optimizer=Adam(), \\\n                  metrics = [\"accuracy\"])\n\n    filepath=\"weights_best.h5\"\n    # Checkpoint not really necessary since it only improves the run time at this moment\n    #checkpoint = ModelCheckpoint(filepath, monitor=\"val_loss\", verbose=2, save_best_only=True, mode=\"min\")\n    reduce_lr = ReduceLROnPlateau(monitor=\"val_loss\", factor=0.6, patience=1, min_lr=0.0001, verbose=2)\n    earlystopping = EarlyStopping(monitor=\"val_loss\", min_delta=0.001, patience=2, verbose=2, mode=\"auto\")\n    callbacks = [earlystopping, reduce_lr]\n    \n    ###\n    # Apply the folds over the model\n    ###\n\n    for i, (train_index, valid_index) in enumerate(kfold.split(X_train, y_train)):\n        Xk_train, Xk_val, Yk_train, Yk_val = X_train[train_index], X_train[valid_index], y_train[train_index], y_train[valid_index]\n\n        print(\"Currently in fold {}/{}\".format(i+1, params['k_folds']))\n        model.fit(Xk_train, Yk_train, batch_size=params['batch_size'], epochs=params['epochs'], \\\n                           validation_data=(Xk_val, Yk_val), callbacks=callbacks, class_weight=class_weights)\n        #model.load_weights(filepath) \n\n    score, acc = model.evaluate(X_test, y_test, verbose=0)\n    \n    # Also add f1 and find optimal threshold,\n    # this way we can compute f1 and optimize for that...\n    \n    pred_val_y = model.predict([X_test], batch_size=params['batch_size'], verbose=0)\n    f1, threshold = f1_smart(np.squeeze(y_test), np.squeeze(pred_val_y))\n    \n    print('Accuracy : {:5f}, Optimal F1 : {:5f}, at threshold : {:5f}'.format(acc, f1, threshold))\n    \n    ### Save to file what we've just done...\n    # I know this is redundant since we're already saving trials, but I had a notebook\n    # crash om me sometimes and this saves the intermediate results, so definitely helps\n    \n    ## https://stackoverflow.com/questions/33054527/python-3-5-typeerror-a-bytes-like-object-is-required-not-str-when-writing-t\n    with open(\"run_{}.txt\".format(run_name),\"a\") as f:\n        print(params, file=f)\n        print('\\n Accuracy : {}, F1 : {}, optimal threshold : {}'.format(acc, f1, threshold), file = f)  \n        f.close() \n    \n    return {\"loss\": -f1, \"status\": STATUS_OK, \"model\": model} # \"loss\" : -acc\n\ntrials = Trials()\n\nbest = fmin(objective, space, algo=tpe.suggest, trials=trials, max_evals=1) #200\n\nprint(hyperopt.space_eval(space, best))\nprint(trials.best_trial)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86110bf4936e81ebcb7134ac3c2dc7fcb53e056b"},"cell_type":"code","source":"import hyperopt\nprint(hyperopt.space_eval(space, best))\nprint(trials.best_trial)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69eece8fd1b7778e7b7361e5bc033451c9078547"},"cell_type":"code","source":"# remove the keras model since this one does not play nicely with the pickle\n# cleaned_trials_trials = []\n# for trial in trials.trials:\n#     del trial['result']['model']\n#     cleaned_trials_trials.append(trial)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8a7673cc88a60f4feda6f7a764155df107944e76"},"cell_type":"code","source":"# from https://github.com/hyperopt/hyperopt/issues/267\n# & https://github.com/hyperopt/hyperopt/wiki/FMin\n# for some nice postprocesing and plotting of the results see:\n# https://medium.com/district-data-labs/parameter-tuning-with-hyperopt-faa86acdfdce\n\n# Just saving the results here which should be interprable from the results\n# can't pickle the whole trials file because it contains Keras models and \n# those can't be pickled apparently :((((\n\n# import pickle\n\n# trials_trials = trials.trials\n# trials_losses = trials.losses()\n# trials_statuses = trials.statuses()\n\n# pickle.dump([cleaned_trials_trials, trials_losses, trials_statuses, space], open(\"hyperopt_result_{:%Y-%m-%d %H:%M:%S}.p\".format(datetime.datetime.now()), \"wb\"))\n# # trials = pickle.load(open(\"myfile.p\", \"rb\")) # for later reloading if necessary","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e251fe22748eb145b9d12da58db643e46868f5b"},"cell_type":"code","source":"# file = open(\"results_{:%Y-%m-%d %H:%M:%S}.txt\".format(datetime.datetime.now()),\"w\") \n# print(hyperopt.space_eval(space, best), file = file) \n# print(trials.best_trial, file = file)\n\n# file.close() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad68bca61af352d2302e10d1aa8c4830dea1db4b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"notify_time":"5"},"nbformat":4,"nbformat_minor":1}