{"cells":[{"metadata":{"_uuid":"6ad951d6352a465a126da5de93e1c5016386c4b4"},"cell_type":"markdown","source":"# Introduction\nI constructed this Kernel from [MichaelMayer](https://www.kaggle.com/mayer79)'s great [RNN Starter Kernel](https://www.kaggle.com/mayer79/rnn-starter/code) code. Thanks for that! I've been using a similar Kernel with a Notebook but it's got plenty of other stuff in there which makes it more than just bare bones. Since this is not really my work, I figured I would share this Kernel with the community. It's meant to be MichaelMayer's starter code converted to a Notebook and nothing more. I had to make some changes, which I've indicated in the code.\n\n# Basic idea of the Kernel\nAt test time, we will see a time series x of length 150 000 to predict the next earthquake. The idea of this kernel is to split x into contiguous chunks of size 1000 and feed summary statistics calculated from these chunks into a current neural net with 150 000 / 1000 = 150 time steps. \n\n# Validation\nValidation is a very complex and crucial element of this competition. Here, the validation generator is fed with data until the second earthquake, while the training generator solely picks chunks after the second earthquake. In order to reach a better LB score, you will need to change this so that the training generator has access to the full data.\n\n# Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\nfrom tqdm import tqdm\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a79066c13602706f8f5650c26df4543343751a24"},"cell_type":"markdown","source":"# Setup"},{"metadata":{"trusted":true,"_uuid":"d7603a75fef6a3699a80c83f1a022db846ec4b62"},"cell_type":"code","source":"# Fix seeds\nfrom numpy.random import seed\nseed(639)\nfrom tensorflow import set_random_seed\nset_random_seed(5944)\n\nfloat_data = pd.read_csv(\"../input/train.csv\", dtype={\"acoustic_data\": np.float32, \"time_to_failure\": np.float32}).values","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Helper functions\n## Helper function for data generator\nExtracts mean, standard deviation, and quantiles per time step.\nCan easily be extended. Expects a two dimensional array."},{"metadata":{"trusted":true,"_uuid":"f613168286f30f0b726a5a87b19496b886e5799d"},"cell_type":"code","source":"def extract_features(z):\n     return np.c_[z.mean(axis=1), \n                  np.transpose(np.percentile(np.abs(z), q=[0, 50, 75, 100], axis=1)),\n                  z.std(axis=1)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f5e008338780fbabe1bbca8db240321eb2d43ac4"},"cell_type":"markdown","source":"## Create features\nFor a given ending position \"last_index\", we split the last 150 000 values of *x* into 150 pieces of length 1000 each. So *n_steps * step_length* should equal 150 000.\nFrom each piece, a set features are extracted. This results in a feature matrix of dimension *(150 time steps x features)*.  "},{"metadata":{"trusted":true,"_uuid":"44b656fdd10ab9f20d07cb9796fb368077aa91ba"},"cell_type":"code","source":"def create_X(x, last_index=None, n_steps=150, step_length=1000):\n    if last_index == None:\n        last_index=len(x)\n       \n    assert last_index - n_steps * step_length >= 0\n\n    # Reshaping and approximate standardization with mean 5 and std 3.\n    # ORIGINAL: I changed this becuase I got an No OpKernel was registered to support Op 'CudnnRNN' error\n    #temp = (x[(last_index - n_steps * step_length):last_index].reshape(n_steps, -1) - 5 ) / 3\n    # MY CHANGE: This doesn't fix things, I get the same errors\n    temp = (x[(last_index - n_steps * step_length):last_index].reshape(n_steps, -1).astype(np.float32) - 5 ) / 3\n    \n    # Extracts features of sequences of full length 1000, of the last 100 values and finally also \n    # of the last 10 observations. \n    return np.c_[extract_features(temp),\n                 extract_features(temp[:, -step_length // 10:]),\n                 extract_features(temp[:, -step_length // 100:])]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c822f8139814beef638d67db525d4947ac50b50"},"cell_type":"markdown","source":"## Generate features"},{"metadata":{"trusted":true,"_uuid":"ff24351374ff15e299019aa74cb8a58434cf5d04"},"cell_type":"code","source":"# Query \"create_X\" to figure out the number of features\nn_features = create_X(float_data[0:150000]).shape[1]\nprint(\"Our RNN is based on %i features\"% n_features)\n    \n# The generator endlessly selects \"batch_size\" ending positions of sub-time series. For each ending position,\n# the \"time_to_failure\" serves as target, while the features are created by the function \"create_X\".\ndef generator(data, min_index=0, max_index=None, batch_size=16, n_steps=150, step_length=1000):\n    if max_index is None:\n        max_index = len(data) - 1\n     \n    while True:\n        # Pick indices of ending positions\n        rows = np.random.randint(min_index + n_steps * step_length, max_index, size=batch_size)\n         \n        # Initialize feature matrices and targets\n        samples = np.zeros((batch_size, n_steps, n_features))\n        targets = np.zeros(batch_size, )\n        \n        for j, row in enumerate(rows):\n            samples[j] = create_X(data[:, 0], last_index=row, n_steps=n_steps, step_length=step_length)\n            targets[j] = data[row - 1, 1]\n        yield samples, targets\n        \nbatch_size = 32\n\n# Position of second (of 16) earthquake. Used to have a clean split\n# between train and validation\nsecond_earthquake = 50085877\nfloat_data[second_earthquake, 1]\n\n# Initialize generators\n# train_gen = generator(float_data, batch_size=batch_size) # Use this for better score\ntrain_gen = generator(float_data, batch_size=batch_size, min_index=second_earthquake + 1)\nvalid_gen = generator(float_data, batch_size=batch_size, max_index=second_earthquake)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04a3f10117cf8a343a14b543a89da29e17e37b57"},"cell_type":"markdown","source":"# Define the model"},{"metadata":{"trusted":true,"_uuid":"9cb0f322b75ff0b99cfa387c4cdf4e0ba1b6bd57"},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, CuDNNGRU\nfrom keras.optimizers import adam\nfrom keras.callbacks import ModelCheckpoint\n\ncb = [ModelCheckpoint(\"model.hdf5\", save_best_only=True, period=3)]\n\nmodel = Sequential()\nmodel.add(CuDNNGRU(48, input_shape=(None, n_features)))\nmodel.add(Dense(10, activation='relu'))\nmodel.add(Dense(1))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ad308c89964016bdd68f4e47c54c6cdb4dcdd99"},"cell_type":"markdown","source":"# Compile and fit model"},{"metadata":{"trusted":true,"_uuid":"205c34084b16a4f12d02670732948fa2885ea1e8"},"cell_type":"code","source":"model.compile(optimizer=adam(lr=0.0005), loss=\"mae\")\n\nhistory = model.fit_generator(train_gen,\n                              steps_per_epoch=1000,\n                              epochs=30,\n                              verbose=0,\n                              callbacks=cb,\n                              validation_data=valid_gen,\n                              validation_steps=200)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f018417dfa11a7949f0d6ee60360c8086d652ee8"},"cell_type":"markdown","source":"# Visualize accuracies"},{"metadata":{"trusted":true,"_uuid":"5955f00b93ddff2c34ac06c34e956d16f01de620"},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndef perf_plot(history, what = 'loss'):\n    x = history.history[what]\n    val_x = history.history['val_' + what]\n    epochs = np.asarray(history.epoch) + 1\n    \n    plt.plot(epochs, x, 'bo', label = \"Training \" + what)\n    plt.plot(epochs, val_x, 'b', label = \"Validation \" + what)\n    plt.title(\"Training and validation \" + what)\n    plt.xlabel(\"Epochs\")\n    plt.legend()\n    plt.show()\n    return None\n\nperf_plot(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"690bac2b88759da70329c0a2ff5d37a2976db7e6"},"cell_type":"markdown","source":"# Load submission file"},{"metadata":{"trusted":true,"_uuid":"80137e94051a31311cf02ef720cf4ce5ecc404fc"},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id', dtype={\"time_to_failure\": np.float32})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d2ebcc27d44ef5fabf4edf6657cf3fb1f1c3973"},"cell_type":"markdown","source":"## Prepare submission data\nLoad each test data, create the feature matrix, get numeric prediction\n"},{"metadata":{"trusted":true,"_uuid":"a18c660aa93a7aefa19a304b1cbc72a52c337bab"},"cell_type":"code","source":"for i, seg_id in enumerate(tqdm(submission.index)):\n  #  print(i)\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    x = seg['acoustic_data'].values\n    submission.time_to_failure[i] = model.predict(np.expand_dims(create_X(x), 0))\n\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ef3dfcb47c3dc06e936e859a7f9e519a52e8085"},"cell_type":"markdown","source":"## Save submission file"},{"metadata":{"trusted":true,"_uuid":"3806f69a539d7d9eeadeb5691ab6f243d53db5f6"},"cell_type":"code","source":"submission.to_csv('submission.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}