{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from __future__ import division\nimport os\nimport random\nimport numpy as np\nimport tensorflow as tf\nfrom keras import backend as K\n\nimport pandas as pd\n\nfrom keras.layers import Input, Dropout, Dense, concatenate,  Embedding, Flatten, Activation, CuDNNLSTM,  Lambda\nfrom keras.layers import Conv1D, Bidirectional, SpatialDropout1D, BatchNormalization, multiply\nfrom keras.layers import GlobalMaxPooling1D, GlobalAveragePooling1D\nfrom keras import optimizers, callbacks, regularizers\nfrom keras.models import Model\n\n\nfrom collections import defaultdict\nfrom sklearn.feature_extraction.text import CountVectorizer\n\nfrom sklearn.metrics import log_loss\n\nimport re\n\nimport gc\nimport time\nimport nltk\n\nfrom nltk.stem import WordNetLemmatizer\nwordnet_lemmatizer = WordNetLemmatizer()\nfrom nltk.corpus import stopwords\nstop_words = set(stopwords.words('english'))\n\nfrom keras.preprocessing.text import Tokenizer\nfrom sklearn.model_selection import train_test_split\n\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"PATH = '../input/'\nEMBEDDINGS_PATH = '../input/embeddings/'\nWEIGHTS_PATH = './w0.h5'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Data preparation"},{"metadata":{"trusted":true},"cell_type":"code","source":"full_data = pd.read_csv(PATH+'train.csv',  encoding='utf-8', engine='python')\nfull_data['question_text'].fillna(u'unknownstring', inplace=True)\n\nprint (full_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"full_data.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sz = full_data['question_text'].apply(lambda x :len(x.split(' ')) )\n_ = plt.hist(sz, bins=64)\nplt.show()\n\nMAX_TEXT_LENGTH=40","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The plot shows the distribution of the number of words per question, for NN we need to use the same length for all inputs.\n\nBased on the plot, we notice that there are a few questions with number of words higher than 40. So let's fix the max length to 40. \n\nWe can also use 50 (maybe safer), but later on, I m expecting the mean number of words to be reduced as more advanced preprocessings will be used"},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess( x ):\n    x = re.sub( u\"\\s+\", u\" \", x ).strip()\n    x = x.split(' ')[:MAX_TEXT_LENGTH]\n    return ' '.join(x)\n\n\nX_train, X_test, y_train, y_test = train_test_split(  full_data.question_text.values, full_data.target.values, \n                                                    shuffle =True, test_size=0.5, random_state=42)\n\nX_train = np.array( [preprocess(x) for x in X_train] )\nX_test  = np.array( [preprocess(x) for x in X_test] )\n\nprint ( X_train.shape, X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We start by a simple preprocessing, replacing any multiple spaces (\\t, \\n) by a single space.\n\nDue to hardware constraints, time limit and mostly because we are in the prototyping phase, I will go for 50% train/test split. Later, we can switch to a 80% train/test split or even better to a real cross validation approach."},{"metadata":{},"cell_type":"markdown","source":"## 2. First NN using a bag of word approach"},{"metadata":{"trusted":true},"cell_type":"code","source":"def my_tokenizer(x):\n    return x.split(' ')\n\ncount_vectorizer = CountVectorizer( tokenizer=my_tokenizer, strip_accents = None,\n                                   lowercase=False, dtype=np.float32,  min_df=2,\n                                   ngram_range=(1, 1), stop_words=None, max_features=None)\n\nX_train_bow = count_vectorizer.fit_transform(X_train)\nX_test_bow  = count_vectorizer.transform(X_test)\n\nprint (X_train_bow.shape, X_test_bow.shape)\nprint ( len(count_vectorizer.vocabulary_ ) )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we only keep words with a frequency higher than 1\n\nthe outputs are sparse matrices"},{"metadata":{"trusted":true},"cell_type":"code","source":"def bowNN(data, seed=42):                                             \n    np.random.seed(seed)\n\n    bow_inpt = Input( shape=[data.shape[1]], dtype = 'float32',   sparse = True, name='bow_inpt',)  \n    \n    x = Dense(100 )(bow_inpt)   \n    x = Activation('relu')(x)\n    \n    x= Dense(1)(x)\n    x = Activation('sigmoid')(x)\n    \n    model = Model([bow_inpt],x)\n\n    return model\n\nbowNN(X_test_bow).summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nes = callbacks.EarlyStopping( patience=5 )\n\nmodel_bow = bowNN(X_test_bow,seed=0)\nmodel_bow.compile(loss=\"binary_crossentropy\", optimizer=optimizers.Adam())\n\nhistory = model_bow.fit(  X_train_bow, y_train, validation_data=(X_test_bow, y_test), callbacks=[es],\n             batch_size=2048, epochs=1000 , verbose=2)\n\nplt.plot(history.history['val_loss'])\nplt.plot(history.history['loss'])\nplt.legend( ['test', 'train'] )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = model_bow.predict( X_test_bow,batch_size=2048 )\nprint ( 'test score : ',log_loss(y_test, preds, eps = 1e-7) )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The performance we have is not the one corresponding to the optimal epoch with a loss of 0.1211. As we can see in the plots, we are overfitting.\n\nThe optimal weights from epoch 2 are lost, so we need to use a callback to save the optimal weights during training."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nes = callbacks.EarlyStopping( patience=5 )\n#below is the new callback to save the optimal weights\nmc = callbacks.ModelCheckpoint( filepath=WEIGHTS_PATH, monitor='val_loss', mode='min', save_best_only=True )\n\nmodel_bow = bowNN(X_test_bow,seed=0)\nmodel_bow.compile(loss=\"binary_crossentropy\", optimizer=optimizers.Adam())\n\nhistory = model_bow.fit(  X_train_bow, y_train, validation_data=(X_test_bow, y_test), callbacks=[es, mc],\n             batch_size=2048, epochs=1000 , verbose=2)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_bow.load_weights( WEIGHTS_PATH )\npreds = model_bow.predict( X_test_bow,batch_size=2048 )\nprint ( 'test score using the callback: ',log_loss(y_test, preds, eps = 1e-7) )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.Optimizers"},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizers_list = [('sgd',optimizers.SGD( lr=.1) ),\n                   ('sgd_momentum',optimizers.SGD(lr=.1, momentum=.9) ),\n                   ('adagrad',optimizers.Adagrad()),\n                   ('adadelta',optimizers.Adadelta()),\n                   ('adam', optimizers.Adam()) \n                  ]\n\nplt.figure(figsize=(20,5))\n\nfor optimizer in optimizers_list:\n    es = callbacks.EarlyStopping( patience=5 )\n\n    model_bow = bowNN(X_test_bow,seed=0)\n    model_bow.compile(loss=\"binary_crossentropy\", optimizer=optimizer[1])\n\n    history = model_bow.fit(  X_train_bow, y_train, validation_data=(X_test_bow, y_test), callbacks=[es],\n                 batch_size=2048, epochs=50, verbose=2 )\n    \n    \n    plt.plot(history.history['val_loss'])\n    \nplt.legend([x[0] for x in optimizers_list], loc='upper right')\nplt.title('model accuracy')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. second NN using embeddings"},{"metadata":{"trusted":true},"cell_type":"code","source":"word_frequency_dc=defaultdict(np.uint32)\n#function that computes the frequency of each word\ndef word_count(text):\n    text = text.split(' ')\n    for w in text:\n        word_frequency_dc[w]+=1\n\nfor x in X_train:#!!!!!\n    word_count(x) \n    \n# we are using only the train dataset (X_train) to create the vocabulary\n#we can't learn the embeddings of the words that are in Test but not in Train  anyway\n#because they are not seen during the training phase\n#For those unknown embeddings, we will, later, assign a single and shared embedding\n\n#This second dict will store the label encodings for each word in the vocabulary we created\n#Those encodings are needed as inputs for the NN\nencoding_dc = dict() #Dont use defautdict here!!!!!\n\n#We start the encoding at the value 1, we keep the value 0 for unknown words and padding\nlabelencoder=1\nfor key in word_frequency_dc:\n    encoding_dc[key]=labelencoder\n    labelencoder+=1\n    \nprint ('vocabulary size : ', len(encoding_dc))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess_keras(text):  \n    def get_encoding(w):\n        if w in encoding_dc:       #We will assign a 0 to any word that is not in Train\n            if encoding_dc[w]>1:   #For rare words (frequency==1), we return the 0 value\n                return encoding_dc[w]\n        return 0\n    \n    x = [ get_encoding(w) for w in text.split(' ') ]\n    x = x + (MAX_TEXT_LENGTH-len(x))*[0] #we apply a padding so all questions have the same length, the padding is 0.\n    return x\n\nX_train_emb = np.array( [ preprocess_keras(x) for x in X_train ] )\nX_test_emb  = np.array( [ preprocess_keras(x) for x in X_test ]  )\nprint ( X_train_emb.shape, X_test_emb.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_SIZE = 100\ndef embeddingNN(data, seed=42):                                             \n    np.random.seed(seed)\n\n    emb_inpt = Input( shape=[data.shape[1]], name='emb_inpt')  \n    x = Embedding(len( encoding_dc )+1, EMBEDDING_SIZE) (emb_inpt)\n    \n    x = CuDNNLSTM(64, return_sequences=False) (x)   \n\n    x= Dense(1)(x)\n    x = Activation('sigmoid')(x)\n    \n    model = Model([emb_inpt],x)\n\n    return model\nembeddingNN(X_test_emb).summary()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"es = callbacks.EarlyStopping( patience=3 )\n\nmodel_emb = embeddingNN(X_test_emb,seed=0)\noptimizer = optimizers.Adam(lr=1e-3)\nmodel_emb.compile(loss=\"binary_crossentropy\", optimizer=optimizer)\n\nmodel_emb.fit(  X_train_emb, y_train, validation_data=(X_test_emb, y_test), callbacks=[es],\n             batch_size=2048, epochs=1000 , verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Optimizing the preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"encoding_dc = dict()\nlabelencoder=1\nfor key in word_frequency_dc:\n    if word_frequency_dc[key]>1:# In the previous step, this condition was in the preprocess_keras function\n        encoding_dc[key]=labelencoder\n        labelencoder+=1\n    \nprint ('vocabulary size : ', len(encoding_dc))\n\ndef preprocess_keras(text):\n    def get_encoding(w):\n        if w in encoding_dc:\n            return encoding_dc[w]\n        return 0\n    \n    x = [ get_encoding(w) for w in text.split(' ') ]\n    x = x + (MAX_TEXT_LENGTH-len(x))*[0]\n    return x\nX_train_emb = np.array( [ preprocess_keras(x) for x in X_train ] )\nX_test_emb  = np.array( [ preprocess_keras(x) for x in X_test ]  )\nprint ( X_train_emb.shape, X_test_emb.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"we have now a vocabulary of size 121070 instead of 326921.\n\nIn the first preprocessing, we kept the rare words in the vocabulary, this means that their embeddings will be stored in memory and their weights updated in the NN even though they all have the same 0 label encoding.\n\nWith the second approach, we will only store the embeddings of frequent words.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"es = callbacks.EarlyStopping( patience=3 )\n\nmodel_emb = embeddingNN(X_test_emb,seed=0)\noptimizer = optimizers.Adam(lr=1e-3)\nmodel_emb.compile(loss=\"binary_crossentropy\", optimizer=optimizer)\n\nmodel_emb.fit(  X_train_emb, y_train, validation_data=(X_test_emb, y_test), callbacks=[es],\n             batch_size=2048, epochs=1000 , verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The NN trains now much faster because there are less weights updates to do"},{"metadata":{},"cell_type":"markdown","source":"## 6. Size of the embeddings"},{"metadata":{"trusted":true},"cell_type":"code","source":"for EMBEDDING_SIZE in [50, 100, 300]:\n\n    es = callbacks.EarlyStopping( patience=2 )\n\n    model_emb = embeddingNN(X_test_emb,seed=0)\n    optimizer = optimizers.Adam(lr=1e-3)\n    model_emb.compile(loss=\"binary_crossentropy\", optimizer=optimizer)\n\n    model_emb.fit(  X_train_emb, y_train, validation_data=(X_test_emb, y_test), callbacks=[es],\n                 batch_size=2048, epochs=1000, verbose=2 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Randomness"},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_SIZE = 100\nfor seed in range(3):\n    es = callbacks.EarlyStopping( patience=2)\n\n    model_emb = embeddingNN(X_test_emb,seed=seed)\n    optimizer = optimizers.Adam(lr=1e-3)\n    model_emb.compile(loss=\"binary_crossentropy\", optimizer=optimizer)\n\n    model_emb.fit(  X_train_emb, y_train, validation_data=(X_test_emb, y_test), callbacks=[es],\n                 batch_size=2048, epochs=1000 , verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Depending on the seed that we use. Performance using embeddings of size 100 can be as good as those of size 300 or worse than those of size 50. How can we decide which size is optimal?\n\nHow can we decide what NN architecture and parameters are the best if randomness is involved?\n\nPractical solution : For a given NN, make multiple runs with different seeds and average the results (BAGGING)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_results = dict()\ntrain_results = dict()\nfor EMBEDDING_SIZE in [50, 100, 300]:\n    predictions_test   = pd.DataFrame()\n    predictions_train  = pd.DataFrame()\n    for seed in range(3):\n        print ( 'Running Model with seed : ', seed, 'EMBEDDING_SIZE : ', EMBEDDING_SIZE )\n        es = callbacks.EarlyStopping( patience=2, monitor='val_loss', mode='min' )\n        mc = callbacks.ModelCheckpoint( filepath=WEIGHTS_PATH, monitor='val_loss', mode='min', save_best_only=True )\n\n        model = embeddingNN(X_test_emb,seed=seed)# seed value change at each iteration\n        optimizer = optimizers.Adam(lr=1e-3)\n        model.compile(loss=\"binary_crossentropy\", optimizer=optimizer)\n\n        model.fit(  X_train_emb, y_train, validation_data=(X_test_emb, y_test), callbacks=[es,mc],\n                     batch_size=2048, epochs=1000, verbose=2 )\n\n        model.load_weights(WEIGHTS_PATH)\n\n        p = model.predict(X_test_emb, batch_size=4096)\n        predictions_test[str(seed)] = p.flatten()\n        \n        p = model.predict(X_train_emb, batch_size=4096)\n        predictions_train[str(seed)] = p.flatten()\n\n\n    print('*'*50)\n\n    test_results[EMBEDDING_SIZE]  =   log_loss(y_test, predictions_test.mean(axis=1), eps = 1e-7) \n    train_results[EMBEDDING_SIZE]  =   log_loss(y_train, predictions_train.mean(axis=1), eps = 1e-7) \n    \nprint ( test_results )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#We can see that a high dimensional (300) embedding gives the best results\n{50: 0.11738275673258461, 100: 0.11665867249362337, 300: 0.11523046791034251}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 8. Tuning the batch size and the learning rate\nThe following code will run the same model with different combinations of batch size and learning rate"},{"metadata":{"trusted":true},"cell_type":"code","source":"run_this_code = False\nif run_this_code:\n    train_results = defaultdict(dict)\n    test_results  = defaultdict(dict)\n\n    for batch in [8192, 4096, 2048, 1024, 512]:\n        for lr in [1e-4, 5e-4, 2e-4, 1e-3, 2e-3, 5e-3, 1e-2, 2e-2, 1e-1]:\n            print ('                         Running Batch size : ', batch, 'Learning Rate : ', lr)\n            predictions_train = pd.DataFrame()\n            predictions_test  = pd.DataFrame()\n            for seed in range(3):\n                print ( 'Running Model with seed : ', seed )\n                es = callbacks.EarlyStopping( patience=3, monitor='val_loss', mode='min' )\n                mc = callbacks.ModelCheckpoint( filepath=WEIGHTS_PATH, monitor='val_loss', mode='min', save_best_only=True )\n\n                model = embeddingNN(X_test_emb,seed=seed)\n                optimizer = optimizers.Adam(lr=lr)\n                model.compile(loss=\"binary_crossentropy\", optimizer=optimizer)\n\n                history = model.fit(  X_train_emb, y_train, validation_data=(X_test_emb, y_test), callbacks=[es,mc],\n                             batch_size=batch, epochs=1000, verbose=2 )\n\n                model.load_weights(WEIGHTS_PATH)\n\n                p = model.predict(X_test_emb, batch_size=1024)\n                predictions_test[str(seed)] = p.flatten()\n\n                p = model.predict(X_train_emb, batch_size=1024)\n                predictions_train[str(seed)] = p.flatten()\n\n                del model, history\n                gc.collect()\n                print ( 'BAGGING SCORE Test: ' , log_loss(y_test,  predictions_test.mean(axis=1), eps = 1e-7) )\n                print ( 'BAGGING SCORE Train: ', log_loss(y_train, predictions_train.mean(axis=1), eps = 1e-7) )\n            print('*'*50)\n            print('')\n\n            test_results[batch][lr]  = log_loss(y_test, predictions_test.mean(axis=1), eps = 1e-7) \n            train_results[batch][lr] = log_loss(y_train, predictions_train.mean(axis=1), eps = 1e-7) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_results = defaultdict(dict,\n            {512: {0.0001: 0.11761565550762154,\n              0.0002: 0.11698579357144163,\n              0.0005: 0.11528072624713684,\n              0.001: 0.11436596154065809,\n              0.002: 0.1135644061933917,\n              0.005: 0.11297268478063005,\n              0.01: 0.11313634372237788,\n              0.02: 0.11971199273251448,\n              0.1: 0.21723223044694112},\n             1024: {0.0001: 0.11825559436878119,\n              0.0002: 0.11722233273331623,\n              0.0005: 0.11577194832604475,\n              0.001: 0.11503769556471356,\n              0.002: 0.11409356554357286,\n              0.005: 0.11365796929952472,\n              0.01: 0.11303597386041803,\n              0.02: 0.11768156400204072,\n              0.1: 0.18597669868338992},\n             2048: {0.0001: 0.11913019800933457,\n              0.0002: 0.11773831646842034,\n              0.0005: 0.11689393794967122,\n              0.001: 0.11675256992165887,\n              0.002: 0.11527158769441004,\n              0.005: 0.11456169586485342,\n              0.01: 0.11301434994527887,\n              0.02: 0.1139797450196391,\n              0.1: 0.16730059463991115},\n             4096: {0.0001: 0.12076714592512715,\n              0.0002: 0.1184375450744624,\n              0.0005: 0.1182501223359833,\n              0.001: 0.1181256172965128,\n              0.002: 0.11741912951601145,\n              0.005: 0.11483459988136224,\n              0.01: 0.11376491009917047,\n              0.02: 0.11450277734120927,\n              0.1: 0.17871693992139087},\n             8192: {0.0001: 0.1212227536500414,\n              0.0002: 0.11957891040232248,\n              0.0005: 0.11839599140357852,\n              0.001: 0.11763529282024128,\n              0.002: 0.11738116102242745,\n              0.005: 0.11674775211161043,\n              0.01: 0.11420364314091092,\n              0.02: 0.11468364987433355,\n              0.1: 0.151295587411758}})\npd.DataFrame(test_results)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- A learning rate 0.01 seems to be the best option when combined with a batchsize of 512, 1024, 2048\n- Some published papers suggest that for sgd based optimizers, using small batch sizes leads to a better generalization (https://openreview.net/pdf?id=BJij4yg0Z , https://openreview.net/forum?id=H1oyRlYgg)\n- We should maximize the ratio LR / BS in order to avoid sharp minima in the los function (https://www.research.ed.ac.uk/portal/files/75846467/width_of_minima_reached_by_stochastic_gradient_descent_is_influenced_by_learning_rate_to_batch_size_ratio.pdf)\n\n\nSo we should probably use LR = 0.01 and BS=1024. But I will use a BS of 2048 instead during the prototyping phase (NN will run faster so we can test more things). Later, we can switch back to LR=1024"},{"metadata":{},"cell_type":"markdown","source":"- When prototyping, choose the largest BS without sacrificing too much the performance\n- I have found in practice that the optimal batch size is independent of NN architecture, so always tune the batch size first, then take the highest learning rate possible."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}