{"cells":[{"metadata":{},"cell_type":"markdown","source":"Forked from Basic Feature Benchmark, courtesy inversion (https://www.kaggle.com/inversion/basic-feature-benchmark)"},{"metadata":{},"cell_type":"markdown","source":"# Executive Summary\nThis Kernel demonstrates a \"zero feature extraction\" method to solve the problem. Keeps trainable parameter count very low.\n\nThis Kernel demonstrates a \"Leave Two Out\" k-fold cross-validation scheme to insure accurate results."},{"metadata":{},"cell_type":"markdown","source":"## Revision History\nv01 - first public version\nv02 - Added \"relu\" activation for pseudo-residual (detail coefficients) layer for better backprop learning performance\nv02 - corrected k-fold wording, added \"Van Halen\" activation function eplanation"},{"metadata":{},"cell_type":"markdown","source":"# A \"zero feature extraction\" method to solve the problem.\n\nI am against complicated feature extraction algorithms prior to machine learning. It seems to defeat the purpose of having artificial intelligence. Also, if you are able to extract features well enough, you can just use a nearest neighbor (kNN) algorithm so a good sorting algorithm is more useful than an iterative error minimization algorithm.\n\nThere is no need to extract features. Most feature extraction uses Fourier Transforms, or the FFT version of the same to extract frequency components from a signal. When the signals are not continuous, then a windowing algorithm and/or a wavelet convolution algorithm is used. The problem with the latter is the selection of the optimal wavelet filter.\n\nIn this Kernel, I show you how to not only use Convolutional Neural Networks (CNN) to create a Discrete Wavelet Transform (DWT) and Perceptron’s (Dense) layers to extract statistical and trend information from the DWT; but I also demonstrate that the algorithm optimizes to the best wavelet filters at each layer of the transformation.\n\nAs you may recall, DWT convolves the signal with a low pass filter g[0], called a wavelet. The results of this convolution are called the approximation coefficients. Simultaneously, the original signal is convolved with a quadrature mirror of the original filter h[0], which extracts all the remaining high frequency components. The results of this convolution are called the detail coefficients. \n\nSince the original signal minus the approximation coefficients should equal the detail coefficients, I sometimes refer to the latter as “the residual.”\n\nDuring Keras based on Tensorflow CNN training, done in batches, I have not found a convenient way to get copies of the trained wavelet filters in the middle of training. For this reason, I propose the use of a pseudo-residual in place of the original residual. The pseudo-residual is calculated as follows:\n\nStep 1, calculate the approximation coefficients (these are precisely the outputs of a “Conv” Keras layer)\n\nStep 2, since we do not have the original wavelet data (called the “kernels” in Keras documentation) we have to make some inferences. We infer that if the output of the convolution is very large, then it must have been a good match with the wavelet. Similarly, If it is a large negative number, it must be a good match, but flipped 180 degrees in phase (upside-down). Only if the values are relatively small, is there a poor match with the wavelet shape. So in step 2, we use an activation function on the “Conv” output (approximation coefficients) that first takes the hyperbolic tangent (‘tanh’ in Keras), and then the absolute value. I called this the \"Van Halen\" activation function, named after the logo of a band from when I was growing up. See below for details.\n\nStep 3, we multiply the activated approximation coefficients by the original signal. \n\nStep 4, finally we subtract step 3 from the original signal, leaving mostly what was poorly approximated, the residual, and hence the detail coefficients.\n\nSee the code below for details.\n\nNOTE: some variations from traditional DWT’s used in literature…\n\n1 – Normally after each transformation, the signals are down sampled by two (2). Here we allow down sampling by any integer value (AveragePooling in Keras terminology)\n\n2 – Normally, we calculate statistical details at each layer of transformation. Here we take an additional step, and down sample to a constant shape (MaxPooling) to remove phase information (so if a particular signal element is shifted by a couple of pixels or samples, the result is the same)\n\n3 – Normally statistical details are calculated using traditional statistical analysis (mean, mode, median, standard deviation, it’s square the variance, etc). Here we use a single layer perceptron (Dense in Keras terminology) to sort out any such patterns.\n\n"},{"metadata":{},"cell_type":"markdown","source":"# A \"Leave Two Out\" k-fold cross-validation scheme to insure accurate results.\n\nWhen using a k-fold cross validation scheme, we take a portion of the training set and save it for validation. The number of times this is repeated is “k”. When using a traditional “Leave One Out” (LOO) k-fold cross-validation, we are taking the most rigorous approach. We leave one of the experiments (one of the people in a an experiment involving humans, or one earthquake experiment here). This is the most rigorous because we are not seeding the training set with “parts of that experiment” – since in the real world, we won’t get that luxury.\nThe modification I made in this Kernel is to pair up experiments (so “Leave Two Out”) yielding a k value of 8. So it is an 8-fold cross validation with LTO experiments.\nEach of these creates a solution neural network, and together they form an ensemble of 8 solutions, whose results are averaged as an ensemble.\nNote: Each solution is deleted before calculating the next, to save memory in Kaggle.\n"},{"metadata":{},"cell_type":"markdown","source":"# \"Van Halen\" Activation Layer for pseudo-residual calculation.\n\nI apologize for using terminology that might not be familiar to younger folks, so let me explain the term \"Van Halen\" activation layer:\n\nWe want the activation function to equal one (+1) whenever the convolution yields a relatively large positive or negative value. This implies a good correlation, or match, between the wavelet (kernel) and the original signal that was filtered. If the convolution yields a relatively small positive or negative value, we want the activation function to trend towards 0.\n\nI gave this the name of the \"Van Halen\" activation function, named after the logo of a band from when I was younger. The function y = VanHalen(x) tends towards +1 as x tends towards + and - infinity, and is 0 when x = 0.\n\n![Van Halen logo](https://upload.wikimedia.org/wikipedia/en/3/35/Van_Halen_-_Van_Halen_II.jpg )\n\nThe Van Halen activation function is simply the 'tanh' activation function in Keras shown in the blue dotted line below, followed by taking its absolute value.\n\n![Hyperbolic Functions](https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/Sinh_cosh_tanh.svg/330px-Sinh_cosh_tanh.svg.png)\n\nI could not find a graph of y = abs(tanh(x)) anywhere online, so I guess it must not be used much, but I hope the logo of the band helps you visualize it in your mind.\n\nimages courtesy wikipedia"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom tqdm import tqdm\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.svm import NuSVR\nfrom sklearn.metrics import mean_absolute_error\n\n# For future use - to use CNN to extract features instead pf simple min/max/etc..\nfrom keras.layers import * \nfrom keras.models import Model, Sequential, load_model\nfrom keras import backend as K \nfrom keras import optimizers \nfrom keras.callbacks import * \nfrom keras.backend import clear_session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rows = 150_000\nsegment = 0\nfor chunk in tqdm(pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float16}, chunksize=rows * 100)) :\n    if (segment == 0) :\n        X_train = np.int16(chunk.acoustic_data.values)\n        y_raw = np.float16(chunk.time_to_failure.values)\n    else: \n        X_train = np.concatenate((X_train, np.int16(chunk.acoustic_data.values)))\n        y_raw = np.concatenate((y_raw, np.float16(chunk.time_to_failure.values)))\n    segment += 1\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# find start and endpoints of the sixteen (16) Earthquake Trials\n# Including the \"cleanup\" of segments that traverse experiment boundaries\nstart = np.zeros(17, dtype=np.int32)\nend = np.zeros(17, dtype=np.int32)\nindex = 0\nfor i in tqdm(range (0,y_raw.shape[0] - rows, rows)) :\n    if i == 0:\n        start[index] = i;\n        end[index] = i;\n    if y_raw[i+rows] > y_raw[i] :\n        end[index] = i\n        index += 1\n        start[index] = i\n        end[index] = i + rows\n        # \"Clean Up\" the segment that traverses the experment boundary\n        # This will be the beginning segment of a new experiment.\n        boundary = 0\n        for j in range(rows - 1) :\n            if y_raw[i + j + 1] > y_raw[i + j] :\n                boundary = j\n        # \"Clean up\" for now means zero out seismic data.\n        X_train[i:i+boundary] = 0\n# Now let's see how big each experiment is, in terms of the maximum number of 150000 sample rows \nrunning_count = 0\nfor i in range (16) :\n    count = np.int32((end[i] - start[i])/rows)\n    print (\"Experiment\", i, \"has\", count, \"segments of 150000 samples each\")\n    running_count += count\n    start[i] = np.int32((start[i])/rows)\n    end[i] = np.int32((end[i])/rows)\n\nsegments = np.int32(end[15])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = np.reshape(X_train[0:(segments * rows)], (segments, rows,1))\nprint(X_train.shape)\nprint(\"Sanity check, shape, segments, and running count\", segments, X_train.shape[0], running_count)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Example of cleaned up segment starting a new experiment\")\nplt.plot(X_train[start[8]])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = np.zeros(segments)\nfor segment in tqdm(range(segments)):\n    seg = y_raw[segment*rows:segment*rows+rows]\n    y_train[segment] = seg[-1]\nmedian = np.median(y_train)\noverall_max = y_train.max()\nprint (\"time to failure median\", median, \"and maximum\", overall_max)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The original arget data\")\nplt.plot(y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clean up unused RAM \ndel y_raw\ndel chunk","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rows = 150_000\nsegment = 0\nfor seg_id in tqdm(submission.index):\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    if (segment == 0) :\n        X_test = np.int16(seg.acoustic_data.values)\n    else: \n        X_test = np.concatenate((X_test, np.int16(seg.acoustic_data.values)))\n    segment += 1\ntest_segments = segment","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot the head of the seismic acoustic data\nplt.plot(X_test[0:150000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test = np.reshape(X_test, (test_segments, rows,1))\nX_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sanity check - plot the head of the seismic acoustic data\nplt.plot(X_test[0,:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Back to CNN creation"},{"metadata":{},"cell_type":"markdown","source":"Discrete Wavelet Transform (courtesy Wikipedia.org)\n![Discrete Wavelet Transform (courtesy Wikipedia.org)](https://upload.wikimedia.org/wikipedia/commons/2/22/Wavelets_-_Filter_Bank.png)\n\nWhere g(n) is the low pass filter, yielding the approximation coefficients, and\nh(n) is the corresponding high pass filter (quadrature mirror of g(n)) yielding the detail coefficients"},{"metadata":{"trusted":true},"cell_type":"code","source":"# DWT (diagram above) using pseudo-residual\n        # PSEUDO-RESIDUAL\n        # the above variable \"conv\" represents an approximation of the original (scaled) signal created by convolving\n        # one of the \"scale\" number of filters of size \"wavelet_size\" with the original signal and coming up with \n        # convolution values \"conv\" at each point (strides = 1) along that signal.\n        # we coud use these convolved values, along with the wavelet (filter) to recreate an approximation of the\n        # original signal - if only KERAS ON TESORFLOW would give us a function to see those filter values.\n        # I cannot see such a function. \n        #\n        # Nevertheless, we do know that the greater the value of \"conv\" at a particular point, the closer the match of the\n        # original signal was to the wavelet filter shape, centered at that point. The smaller the value, the worse\n        # the match. \n        #\n        # In traditional wavelet decomposition, at each iteration, the residual is calculated.\n        # The residual is the difference between the original signal and the aforementioned approximation. We can't\n        # calculate the approximation (as discussed above) but we do know it should be closest to the original signal\n        # when \"conv\" is greatest, and farthest away from the original signal when \"conv\" is lowest.\n        #\n        # I therefore propose to create a \"pseudo-residual\" which approximates some aspects of an actual residual.\n        # by -1 to +1 limiting the values of \"conv\" ('tanh' activation), taking the absolute value, \n        # and then multiplying that by the original signal, we can create this \n        # pseudoresidual - a function who most closely equals the original signal\n        # in places where the wavelet fileter FAILS to closely match, and most closely equals zero in places where the wavelet\n        # filter SUCCEEDS in closely matching the original signal.\n        \nimport math\nfrom math import floor, log\n\n# Make a CNN Model to learn features\ndef make_model(rows = 150000, linear = True) :\n    # rows is the size of each row of sequential data\n    # linear is whether or not the target is a linear value or not (e.g. binary w/output activation is sigmoid)\n    \n    scale = 32              # the size of the CNN or similarly, the number of wavelet shapes to try and learn\n    dropout = 0.1           # the percentage of training examples to randomly skip each epoch \n    pool_base = 32          # The number of values to average when pooling (DWT uses 2, but here we can modify this)\n    wavelet_size = 32       # The size of the individual convolution or similarly, the number of data points in the wavelet filter\n\n    input_layer = Input((rows,1))\n\n    \n    # This is how far repeated pooling (scaling down) can go before we're down to < wavelet_size sample\n    # padding = \"valid\" for no padding instead of padding = \"same\" for padding with 0's\n    pad = \"same\"\n    depth_low = int(math.floor(math.log(wavelet_size)/math.log(pool_base))) \n    depth = int(math.floor(math.log(rows)/math.log(pool_base))) - depth_low\n    \n    print (\"Depth :\", depth,\"\\n\")\n\n    ########\n    #  A   #\n    ########\n    # MAY NEED TO LOOK INTO ACTIVATION FUNCTION FURTHER. \n    # SHOULD WE SWITCH TO A VAN HALEN SHAPED ACTIVATION FOR 180 PHASE INCLUSION?\n    # v36 added this as the absolute value of the tanh activation function\n    seq_conv = []\n    seq_conv_pool = []\n\n    seq_residual = []\n    seq_residual_pool = []\n    \n    # create the first level\n    seq_conv_pool.append(BatchNormalization()(input_layer))\n    seq_conv_pool[0] = Dropout(dropout)(seq_conv_pool[0])\n\n    for i in range(depth) :\n\n        # find the output of filter g(i) and therefore the detail coefficients\n        seq_conv.append(Conv1D(scale, wavelet_size, strides = 1, padding=pad)(seq_conv_pool[i]))\n\n        # create the pseudo-residual which is an approximation of the output of filter h(i), and therefore the detail coefficients\n        # apply activation function \n        seq_residual.append(Activation('tanh')(seq_conv[i]))\n        seq_residual[i] = Lambda(lambda x: abs(x))(seq_residual[i])\n        # remove the approximation from the original signal, leaving the detail coefficients (residual)\n        seq_residual[i] = Multiply()([seq_residual[i], seq_conv_pool[i]])\n        seq_residual[i] = Subtract()([seq_conv_pool[i], seq_residual[i]])\n\n        # apply the standard \"relu\" activation function for use in NN training (doesn't seem to converge well unless we do this)\n        seq_conv[i] = Activation('relu')(seq_conv[i])\n        \n        # Worked for g(i), so trying on h(i) too...apply the standard \"relu\" activation function\n        seq_residual[i] = Activation('relu')(seq_residual[i])\n        \n        # apply downscaling to g(i) to be used to calculate the next g(i)\n        seq_conv_pool.append((AveragePooling1D(pool_size = pool_base)(seq_conv[i])))\n\n        # finish up by pooling all layers down to the same size\n        # this removes remaining phase information and chunks the entire time sequence into approx \"wavelet_size\" number of chunks based on depth calculations\n        seq_residual_pool.append(MaxPooling1D(pool_size = pool_base ** (depth - i))(seq_residual[i]))\n        seq_conv_pool[i] = MaxPooling1D(pool_size = pool_base ** (depth - i))(seq_conv[i])\n\n\n    # one last one without the pooling afterwards\n    seq_conv.append(Conv1D(scale, wavelet_size, strides = 1, padding=pad)(seq_conv_pool[depth]))\n    seq_conv[depth] = ReLU(max_value = 1)(seq_conv[depth])\n    \n\n    ########\n    #  B   #\n    ########\n    # Give each of the above convolutions of convolutions their own dense layer\n    # first by flattening the convolutional layer and then by adding a simple relu activated set of perceptrons\n    fscp = []\n    dscp = []\n    for i in range(depth) :\n        # for each layer of depth i, concatenate the g(i) output and the h(i) output and flatten\n        fscp.append(concatenate([seq_conv_pool[i], seq_residual_pool[i]]))\n        fscp[i] = Flatten()(fscp[i])\n        # then create a single perceptron (dense layer) to represent that layer\n        dscp.append(Dense(1, activation='relu')(fscp[i]))\n    # One last one without the pooling afterwards\n    fscp.append(Flatten()(seq_conv[depth]))\n    dscp.append(Dense(1, activation='relu')(fscp[depth]))\n\n    # concatenate all of these dense layers into one.\n    con_cat_d2 = dscp[0]\n    for i in range(depth) :\n        con_cat_d2 = concatenate([con_cat_d2,dscp[i+1]])\n        \n\n    \n    ########\n    #  C   #\n    ########\n    # Penultimate layer before output simply provides an ability to resolve simple but nolinear separation \n#    dense7 = Dense(scale, activation='relu')(con_cat_d2)   \n    dense7 = con_cat_d2\n    \n    # output layer\n    if (linear == True) : \n        dense8 = Dense(1, activation='linear')(dense7)\n#        dense8 = Dense(1, activation='relu')(dense7)\n    else :\n        dense8 = Dense(1, activation='sigmoid')(dense7)\n\n    output_layer = dense8\n    \n    model2 = Model(input_layer, output_layer)\n\n    if (linear == True) : \n        model2.compile(loss='mse', optimizer='adam')\n    else :\n        # by all rights, should probably use binary_crossentropy below, but seems to get stuck if I do (mse works). Will investigate further at some other time.\n        model2.compile(loss='binary_crossentropy', optimizer='adam')\n    return model2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(X_train.shape[0], X_test.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# we will be slowing down the learning using \"Dropouts\" (see above) so the patience needed to exit local minima can be large\npatience = 10 # Overrunning CPU time of 9 hours... so lowereed patience from 15\n# Probably will never reach this many epochs, but want to use a number larger than what we expect\nepochs = 300\n\ny_pred_train = np.zeros(X_train.shape[0])\ny_pred_test = np.zeros(X_test.shape[0])\n\n# Use LTO from the 16 earthquake experiments, so eight pairs of experiments to leave out\nfor fold in range (0,16,2) :\n    \n    if fold >= 14 :\n        X_t = X_train[start[0]:end[13]]\n        X_v = X_train[start[14]:end[15]]\n        y_t = y_train[start[0]:end[13]]\n        y_v = y_train[start[14]:end[15]]\n    elif fold == 0 :\n        X_t = X_train[start[2]:end[15]]\n        X_v = X_train[start[0]:end[1]]\n        y_t = y_train[start[2]:end[15]]\n        y_v = y_train[start[0]:end[1]]\n    else :\n        X_t = np.concatenate((X_train[start[0]:end[fold - 1]], X_train[start[fold + 2]:end[15]]))\n        X_v = X_train[start[fold]:end[fold + 1]]\n        y_t = np.concatenate((y_train[start[0]:end[fold - 1]], y_train[start[fold + 2]:end[15]]))\n        y_v = y_train[start[fold]:end[fold + 1]]\n    \n    print('Fold', fold)\n    K.clear_session()\n    \n    model = make_model()\n\n    checkpointer = ModelCheckpoint('LANL_value', verbose=1, save_best_only=True)\n\n    earlystopper = EarlyStopping(patience = patience, verbose=1) \n\n    results = model.fit(X_t, y_t, \n                    epochs = epochs, batch_size = 32,\n                    callbacks=[earlystopper, checkpointer], \n                    validation_data = [X_v, y_v])\n\n    model = load_model('LANL_value')\n    \n    y_pred_train += model.predict(X_train)[:,0] / 8\n    y_pred_test += model.predict(X_test)[:,0] / 8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# y_pred_train *= 8\n# fold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SCORE FOR SCALED TRAINING SET\nplt.figure(figsize=(6, 6))\nplt.scatter(y_train, y_pred_train)\nplt.xlim(0, 20)\nplt.ylim(0, 20)\nplt.xlabel('actual', fontsize=12)\nplt.ylabel('predicted', fontsize=12)\nplt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score = mean_absolute_error(y_train, y_pred_train)\nprint(f'Score: {score:0.3f}')\nbest_mult = 1\nbest_offset = 0\nbest_score = score\nfor imult in tqdm(range (1, 500, 1)) :\n    mult = imult / 100\n    for ioffset in range (-50, 50, 1) :\n        offset = ioffset / 10\n        score = mean_absolute_error(y_train, ((y_pred_train + offset) * mult))\n        if score < best_score :\n            best_score = score\n            best_offset = offset\n            best_mult = mult","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Best multiplier and offset are\", best_mult, best_offset)\nprint(\"Best score\", best_score)\n\n# Use the same multiplier and offset as discovered above.\ny_pred_train += best_offset\ny_pred_train *= best_mult\n\n# get rid of invalid answers (< 0)\nfor i in range(segments) :\n    if y_pred_train[i] < 0 :\n        y_pred_train[i] = 0\n        \n# SCORE FOR SCALED TRAINING SET\nplt.figure(figsize=(6, 6))\nplt.scatter(y_train, y_pred_train)\nplt.xlim(0, 20)\nplt.ylim(0, 20)\nplt.xlabel('actual', fontsize=12)\nplt.ylabel('predicted', fontsize=12)\nplt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"saving Un-Scaled TIME_TO_FAILURE estimates\")\n# y_pred = y_pred_test\n# submission['time_to_failure'] = y_pred\n# submission.to_csv('unscaled_submission.csv')\n# \n# print(\"done!\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use the same multiplier and offset as discovered above.\ny_pred_test += best_offset\ny_pred_test *= best_mult\n# get rid of invalid answers (< 0)\nfor i in range(test_segments) :\n    if y_pred_test[i] < 0 :\n        y_pred_test[i] = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"saving TIME_TO_FAILURE estimates\")\ny_pred = y_pred_test\nsubmission['time_to_failure'] = y_pred\nsubmission.to_csv('submission.csv')\n\nprint(\"done!\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(y_pred_test)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}