{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nnp.random.seed(42) # Set seed for random\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport time\nimport datetime\npd.options.display.precision = 15\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport os\nprint(os.listdir(\"../input\"))\n\nfrom tqdm import tqdm_notebook\n\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout\nfrom tensorflow.keras.layers import Conv1D, MaxPooling1D\nfrom tensorflow.keras.layers import GlobalAveragePooling1D\nfrom tensorflow.keras.layers import BatchNormalization, Activation\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\nfrom tensorflow.keras.initializers import he_normal\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nfrom scipy.signal import spectrogram\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"**First, Import the data with pd.read_csv**"},{"metadata":{"trusted":true,"_uuid":"aa5bcaf9ebb733ebcb3769c58333c37927f6fc40"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float32})\nprint(\"Train Data Loaded\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"edb8c52369a5ba0e03f6f0d888457f87e54426e4"},"cell_type":"markdown","source":"**Split into features and targets**"},{"metadata":{"trusted":true,"_uuid":"3f658b7f6b7694c35e20f3ce2acb787bb473d6aa"},"cell_type":"code","source":"# Split into features and targets\nX_train = train['acoustic_data'].values\ny_train = train['time_to_failure'].values\n# Clear the old stuff\ndel train","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f025e2a2302ea6980db50cfdeddb0a69ac92de8"},"cell_type":"markdown","source":"**Look at a histogram of the y_train values**"},{"metadata":{"trusted":true,"_uuid":"3238895fa07ff56bed7ec3f429b83b3c407277fa"},"cell_type":"code","source":"plt.hist(y_train,bins=50,facecolor='blue',density=True)\nplt.title('Histogram of all time_to_failure values')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d370961728d2defc530b94026e5367499e241c38"},"cell_type":"markdown","source":"**Now to choose starting indices for our training time domain samples**\n\nThere's some interesting discussion seen in <https://www.kaggle.com/allunia/shaking-earth> about what is the best way to split the training data up according to the points of failure in time_to_failure. I've experimented a bit with including and excluding the actual points of failure from the final training data, and I noticed better performance by not excluding the actual points of failure (meaning there could be a point of failure not related to the target time_to_failure within a training segment).\n\nI also did something a little cheeky and peeked at a histogram of the predictions from the kernel <https://www.kaggle.com/artgor/earthquakes-fe-more-features-and-samples>. The author has an MAE of 1.497 from this kernel so it's reasonable to assume that the distribution of his final predictions are fairly close to the actual distribution, and thus I opted to intentionally bias my training data with a similar distribution (a slightly bimodal distribution around 4.0 seconds and 8.0 seconds). This may or may not be dubious when considering the 87% of test data not included in the current test set, but we can worry about that later.\n\nI split the train data up by three overlapping segments, attempting to produce a nearly bimodal distribution of the training data for time_to_failures around ~4 seconds and ~8 seconds. Ultimately it looks like more of a positively skewed gaussian distribution, but I think it's close enough to the original goal. I'm also deleting variables as soon as they aren't needed anymore to free up memory in the kernel. At the end of the code segment I plot a histogram of the y_train values to examine the distribution."},{"metadata":{"trusted":true,"_uuid":"0681f42ab6dfb061c4e434a0fbe8a8bd500f9710"},"cell_type":"code","source":"#rows = 150_000\n#X_train = X_train[:int(np.floor(X_train.shape[0] / rows))*rows]\n#y_train = y_train[:int(np.floor(y_train.shape[0] / rows))*rows]\n#X_train= X_train.reshape((-1, rows, 1))\n#y_train = y_train[rows-1::rows]\n#train_starts = np.arange(rows,rows*4194,rows)\n\n# Look at histogram of y_train values\n#n, bins, patches = plt.hist(y_train,bins=50,facecolor='blue',density=True)\n#plt.title('Histogram of y_train values')\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true,"scrolled":true,"_uuid":"924fe4ea82e59b4cd12299ba8b12f04a1fd2f721"},"cell_type":"code","source":"# First quartile, y_train < 4.0\nQ1_indices = np.where(y_train<6.0)[0]\nQ1_idx = Q1_indices > 150000\nQ1_indices = Q1_indices[Q1_idx]\nQ1_starts = np.random.choice(a=Q1_indices,size=4000)\ndel Q1_indices, Q1_idx\n# Second Quartile, y_train > 4 && y_train < 8\nQ2_indicesX = np.where(y_train>=4.0)[0]\nQ2_indices1 = y_train >= 4.0\nQ2_times = y_train[Q2_indices1]\ndel Q2_indices1     # free up memory asap\nQ2_indices2 = Q2_times < 10.0\nQ2_indices = Q2_indicesX[Q2_indices2]    # Limit between 3 and 9\ndel Q2_indices2, Q2_indicesX     # free up memory asap\nidx = Q2_indices > 150000\nQ2_indices = Q2_indices[idx]\nQ2_starts = np.random.choice(a=Q2_indices,size=3000)\ndel Q2_indices, Q2_times, idx\n# Third quartile, y_train >= 7\nQ3_indices = np.where(y_train>=4.0)[0]\nQ3_idx = Q3_indices > 150000\nQ3_indices = Q3_indices[Q3_idx]\nQ3_starts = np.random.choice(a=Q3_indices,size=4000)\ndel Q3_idx, Q3_indices\n\n# Concatenate the starts arrays\ntrain_starts = np.r_[Q1_starts, Q2_starts, Q3_starts]\ndel Q1_starts, Q2_starts, Q3_starts\n\n# Get y_tr\ny_train = y_train[train_starts]\n\n# Look at histogram of y_tr values\nplt.hist(y_train,bins=50,facecolor='blue',density=True)\nplt.title('Histogram of y_tr values')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d688c57a9f91408ae0fe2ef0bf77a6f2b805c21e"},"cell_type":"markdown","source":"**Generate the spectrograms**\n\nIn this segment, we iterate through our train_starts indices, extract the time domain signal (always length 150k samples), spectrogram the signal with scipy.signal.spectrogram, and store it in a three dimensional array of spectrograms.\n\nA primary concern to address when using spectrograms is the time/frequency resolution trade-off. Time and frequency are of course inversely related, so to gain frequency resolution we must lose resolution in time, and vice-versa. This time/frequency resolution is determined primarily by our NPERSEG (length of window), while NOVERLAP (number of samples to overlap each segment) will produce a smoothing effect in the time domain. What might the best values be for these 'hyperparameters'?\n\nWell I did a reasonably thorough, although not entirely exhaustive, analysis of a number values for these parameters. Details are below...\n\n**NPERSEG (frame size):** I experimented with values closer to traditional frame sizes used in spectrograms initially, i.e. 1024, 512, and 256 samples. I never acheived very good performance from the CNN with these, and began experimenting with very small frame sizes. Hence the reader can observe a frame size (NPERSEG) of 4 samples used below. Note that such a frame size would be pretty ridiculous in a normal spectrogram analysis.\n\n**NOVERLAP (overlap in samples):** When using NPERSEG 256 or greater, having NOVERLAP in range from 25% to 90% had no distinguishable effect on performance. However, when using very small frame sizes (NPERSEG = 4, 8) I found that using NOVERLAP equivalent to 25-75% overlap was significantly detrimental to performance. Also note that introducing overlap makes your spectrograms larger, thus potentially reducing the amount of spectrograms one can generate before running out of memory.\n\nFor the sampling rate (fs), I just made it one. It's kind of arbitrary and not explicitly given, although I supposed it might be interesting to calculate it from the time_to_failure...\n\nI also noticed that there wasn't really much information present in the third frequency of the spectrogram output (Gxx), so I just sliced it out before saving Gxx to Gxx_train"},{"metadata":{"trusted":true,"_uuid":"32c5818da18785f4e7dd991d230858993497ef1a"},"cell_type":"code","source":"# Gxx params\nNPERSEG = 4\nNOVERLAP = 0\nGxx_train = np.zeros([len(y_train),37500,2])\n# Choose a random length between 100k and 300k\nsig_length = 150000\nfor ii, seg in tqdm_notebook(enumerate(train_starts)):\n    # Grab the chunk of signal\n    x_time = X_train[train_starts[ii]-sig_length:train_starts[ii]]\n    # Spectrogram the signal\n    _, __, Gxx = spectrogram(x_time,fs=1.0,window='hann', noverlap=NOVERLAP, \n                             return_onesided=True, nperseg=NPERSEG, \n                             scaling='density', mode='magnitude')\n    # Remove all frequencies above indice 2\n    Gxx = Gxx[0:2,:]\n    # Reshape Gxx as a 3D array\n    Gxx = Gxx.reshape([1,Gxx.shape[1],Gxx.shape[0]])\n    # Allocate into main 3D array\n    Gxx_train[ii,:,:] = Gxx[0,:,:]\n        \n        \n# Clear memory from train data\ndel X_train, x_time, sig_length","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"efd7f05195d07a6d0b0d80195f3ba48cf9f8c917"},"cell_type":"markdown","source":"**Scaling?**\nInitially I experimented with standard scaling, min-max scaling (-1 to 1), as well as simply normalizing to the maximum value from the dataset. While I find it a bit strange, not scaling the data had the best performance in the end. Also the kernel <https://www.kaggle.com/fanconic/earthquake-cnn> used a Conv1D model on just the unscaled time series data and had pretty good performance too (spoiler alert - better than any of my spectrogram based models)."},{"metadata":{"_uuid":"ee60f413ff77fbc904e6ab90ae35e5ebe3943ea4"},"cell_type":"markdown","source":"**Split into training and validation sets**"},{"metadata":{"trusted":true,"_uuid":"24bb8d99c17e4bb31e787b685424b9cce9af85a7"},"cell_type":"code","source":"# Split train and validation sets\nindices = np.arange(0,Gxx_train.shape[0],1)\ntrain_indices, val_indices = train_test_split(indices,test_size=0.2,random_state=42)\ndel indices\n# Split up with indices\nGxx_val = Gxx_train[val_indices,:,:]\nGxx_train = Gxx_train[train_indices,:,:]\ny_val = y_train[val_indices]\ny_train = y_train[train_indices]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06f41653d1f89ad0eb6b65490f0315e0f484f9b0"},"cell_type":"markdown","source":"**Build the 1D Convolutional model and fit it to the training data**\n\nI took a bit of inspiration from <https://www.kaggle.com/fanconic/earthquake-cnn> to get my convolution model up and running. Notably switching from MaxPooling1D after the final convolutional layer to GlobalAveragePooling1D significantly improved performance of my model.\n\nModel architecture is as follows:\n[Conv1D] > [Conv1D] > [MaxPool1D] > [Conv1D] > [Conv1D] > [GlobalAveragePooling1D] > [FC-16] > Output [FC-1]\n\nInitially I had used Adam optimization as seen in the above mentioned kernel, however I found that the model would achieve its best performance around epoch two or three, and then diverge strongly. It also wasn't very consistent about where it ended up in terms of the validation MAE. I switched to simple stochastic gradient descent (SGD) with a bit of momentum and a lower learning rate to adress the issue, although the issue still seemed to be present. I then introduced learning rate scheduling through the decay hyperparameter in SGD, which really seemed to help address the divergence issue."},{"metadata":{"trusted":true,"_uuid":"233b1bfb5c76ba649c4664897a7818ac0d42dc4c","scrolled":true},"cell_type":"code","source":"# Lets build a CNN model\nprint('Building CNN Model 1') # Diagnostics purposes\ninput_shape = (Gxx.shape[1],Gxx.shape[2])\nbatch_size = 64\ndecay = 0.0001 / 100\nkernel_initializer = tf.keras.initializers.RandomUniform(seed=1)\n# Conv2D model\nwith tf.device('/gpu:0'):\n    mdl1 = tf.keras.models.Sequential()\n    mdl1.add(Conv1D(filters=16,kernel_size=10, padding=\"same\", input_shape=input_shape,\n                    kernel_initializer=kernel_initializer, activation='relu'))\n    mdl1.add(Conv1D(filters=16,kernel_size=10, padding=\"same\", kernel_initializer=kernel_initializer, activation='relu'))\n    mdl1.add(MaxPooling1D(pool_size=100))\n    mdl1.add(Conv1D(filters=32,kernel_size=10,padding=\"same\", kernel_initializer=kernel_initializer, activation='relu'))\n    mdl1.add(Conv1D(filters=32,kernel_size=10,padding=\"same\", kernel_initializer=kernel_initializer, activation='relu'))\n    mdl1.add(GlobalAveragePooling1D())\n    mdl1.add(Dense(16, kernel_initializer=kernel_initializer, activation='relu'))\n    mdl1.add(Dense(1,activation='linear'))\n    # Early Stopping and stuff\n    earlyStopping = EarlyStopping(monitor='val_loss',\n                              patience=10,\n                              verbose=1,\n                              mode='min',\n                              )\n    mcp_save = ModelCheckpoint('.mdl1_wts.hdf5',\n                           save_best_only=True,\n                           monitor='val_loss',\n                           mode='min')\n    \n    # Compile the model\n    mdl1.compile(loss=tf.keras.losses.mean_absolute_error,\n                optimizer=tf.keras.optimizers.SGD(0.01,momentum=0.7,decay=decay),\n                metrics=['mae'])\n\n\n    t1 = time.time()\n    mdl1.fit(Gxx_train,y_train,\n            batch_size=batch_size,\n            epochs=100,\n            verbose=1,\n            validation_data= (Gxx_val, y_val),\n            callbacks=[earlyStopping, mcp_save]\n            )\n    t_total = time.time() - t1\n    print(\"Time for train: \",str(t_total/60**2),\" hours\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"608de8780fd28c154b01dd8dc1922a4ab38d43dd"},"cell_type":"markdown","source":"**Ensemble of CNN models (blending)**\n\nHaving noticed the performance of my previous model top out around 1.9 MAE, I figured I might try blending a couple of different CNN models and see if that would improve the score.\n\nI actually use the same model in both cases below as I did above, however I seed the random uniform initialization differently for each of the three CNN models. This is a trick I actually saw used in a tutorial for Google's AdaNet, which used an ensemble of two identical CNNs with different random initializations to improve the overall classification accuracy of their model. On the surface this doesn't seem that useful, as generally one would expect the models to both converge to the global optimum -. but I think facing the current issue with the strong divergence immediatley after reaching the optimal solution, this could produce a similar performance boost.\n"},{"metadata":{"trusted":true,"_uuid":"d3943cc112a6f2af98b16654aba8a5d0ca09a86e"},"cell_type":"code","source":"# CNN model with different initialization\nprint('Building CNN Model 2') # Diagnostics purposes\ninput_shape = (Gxx.shape[1],Gxx.shape[2])\nbatch_size = 64\nkernel_initializer = tf.keras.initializers.RandomNormal(seed=11)\n# Conv2D model\nwith tf.device('/gpu:0'):\n    mdl2 = tf.keras.models.Sequential()\n    mdl2.add(Conv1D(filters=16,kernel_size=10, padding=\"same\", input_shape=input_shape,\n                    kernel_initializer=kernel_initializer, activation='relu'))\n    mdl2.add(Conv1D(filters=16,kernel_size=10, padding=\"same\", kernel_initializer=kernel_initializer, activation='relu'))\n    mdl2.add(MaxPooling1D(pool_size=100))\n    mdl2.add(Conv1D(filters=32,kernel_size=10,padding=\"same\",kernel_initializer=kernel_initializer, activation='relu'))\n    mdl2.add(Conv1D(filters=32,kernel_size=10,padding=\"same\",kernel_initializer=kernel_initializer, activation='relu'))\n    mdl2.add(GlobalAveragePooling1D())\n    mdl2.add(Dense(16,activation='relu',kernel_initializer=kernel_initializer))\n    mdl2.add(Dense(1,activation='linear'))\n    # Early Stopping and stuff\n    earlyStopping = EarlyStopping(monitor='val_loss',\n                              patience=10,\n                              verbose=1,\n                              mode='min',\n                              )\n    mcp_save = ModelCheckpoint('.mdl2_wts.hdf5',\n                           save_best_only=True,\n                           monitor='val_loss',\n                           mode='min')\n    \n    # Compile the model\n    mdl2.compile(loss=tf.keras.losses.mean_absolute_error,\n                optimizer=tf.keras.optimizers.SGD(0.01,momentum=0.7,decay=decay),\n                metrics=['mae'])\n\n\n    t1 = time.time()\n    mdl2.fit(Gxx_train,y_train,\n            batch_size=batch_size,\n            epochs=100,\n            verbose=1,\n            validation_data= (Gxx_val, y_val),\n            callbacks=[earlyStopping, mcp_save]\n            )\n    t_total = time.time() - t1\n    print(\"Time for train: \",str(t_total/60**2),\" hours\")\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"146c79818dadf61fe846b0231021d5952536ff13"},"cell_type":"code","source":"# Simple CNN model\nprint('Building CNN Model 3') # Diagnostics purposes\ninput_shape = (Gxx.shape[1],Gxx.shape[2])\nbatch_size = 64\nkernel_initializer = tf.keras.initializers.RandomNormal(seed=12)\n# Conv2D model\nwith tf.device('/gpu:0'):\n    mdl3 = tf.keras.models.Sequential()\n    mdl3.add(Conv1D(filters=16,kernel_size=10, padding=\"same\", input_shape=input_shape,\n                    kernel_initializer=kernel_initializer, activation='relu'))\n    mdl3.add(Conv1D(filters=16,kernel_size=10, padding=\"same\", \n                    kernel_initializer=kernel_initializer, activation='relu'))\n    mdl3.add(MaxPooling1D(pool_size=100, strides=1))\n    mdl3.add(Conv1D(filters=32,kernel_size=10,padding=\"same\",\n                    kernel_initializer=kernel_initializer, activation='relu'))\n    mdl3.add(Conv1D(filters=32,kernel_size=10,padding=\"same\",\n                    kernel_initializer=kernel_initializer, activation='relu'))\n    mdl3.add(GlobalAveragePooling1D())\n    mdl3.add(Dense(16,activation='relu',kernel_initializer=kernel_initializer))\n    mdl3.add(Dense(1,activation='linear'))\n    # Early Stopping and stuff\n    earlyStopping = EarlyStopping(monitor='val_loss',\n                              patience=10,\n                              verbose=1,\n                              mode='min',\n                              )\n    mcp_save = ModelCheckpoint('.mdl3_wts.hdf5',\n                           save_best_only=True,\n                           monitor='val_loss',\n                           mode='min')\n    \n    # Compile the model\n    mdl3.compile(loss=tf.keras.losses.mean_absolute_error,\n                optimizer=tf.keras.optimizers.SGD(0.01,momentum=0.7,decay=decay),\n                metrics=['mae'])\n\n\n    t1 = time.time()\n    mdl3.fit(Gxx_train,y_train,\n            batch_size=batch_size,\n            epochs=100,\n            verbose=1,\n            validation_data= (Gxx_val, y_val),\n            callbacks=[earlyStopping, mcp_save]\n            )\n    t_total = time.time() - t1\n    print(\"Time for train: \",str(t_total/60**2),\" hours\")\n\n# Clear Gxx_train, y_tr\ndel Gxx_train, y_train","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de34765d2881d1bec9a1caae315d9360d10674ca"},"cell_type":"markdown","source":"**Get the test data and generate spectrograms of it**\n\nNow we read in the sample_submission.csv file to get the list of test files, and extract spectrograms of each time series."},{"metadata":{"trusted":true,"_uuid":"c7e2ed32dd7a9f0e36967bb52f3213fef592773b"},"cell_type":"code","source":"# Format test data\nsubmission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\n# Convert X_test to Gxx\nGxx_test = np.zeros([len(submission),37500,2])\nfor ii, seg in tqdm_notebook(enumerate(submission.index)):\n    # Grab the chunk of signal\n    x_time = pd.read_csv('../input/test/' + seg + '.csv').values.astype(np.int16)\n    # Spectrogram the signal\n    _, __, Gxx = spectrogram(x_time.reshape(-1),fs=1.0,window='hann', nperseg=NPERSEG, \n                             noverlap=NOVERLAP, return_onesided=True, \n                             scaling='density', mode='magnitude')\n    # Remove information for freqs above indice 32\n    Gxx = Gxx[0:2,:]\n    # Reshape Gxx as a 3D array\n    Gxx = Gxx.reshape([1,Gxx.shape[1],Gxx.shape[0]])\n    # Allocate into main 3D array\n    Gxx_test[ii,:,:] = Gxx[0,:,:]      ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df0ba50a84352db70c5f968f97056a011bfe637a"},"cell_type":"markdown","source":"**Restore the best model weights, make predictions on the test set**\n\nWe'll also plot a histogram of the final predictions, as well as the validation set, to see how they ended up. "},{"metadata":{"trusted":true,"_uuid":"7fedc6d308304978553049cb8f986b4132e90ce2"},"cell_type":"code","source":"# Get predictions for each test instance\nt1 = time.time()\nwith tf.device('/gpu:0'):\n    mdl1.load_weights(\".mdl1_wts.hdf5\")   # Restore best weights\n    mdl2.load_weights(\".mdl2_wts.hdf5\")\n    mdl3.load_weights(\".mdl3_wts.hdf5\")\n    # Evaluate on validation set\n    loss1, mae_best1 = mdl1.evaluate(Gxx_val,y_val)\n    loss2, mae_best2 = mdl2.evaluate(Gxx_val,y_val)\n    loss3, mae_best3 = mdl3.evaluate(Gxx_val,y_val)\n    # Get predictions from models to look at distributions later\n    mdl1_val_preds = mdl1.predict(Gxx_val,batch_size=32)\n    mdl2_val_preds = mdl2.predict(Gxx_val,batch_size=32)\n    mdl3_val_preds = mdl3.predict(Gxx_val,batch_size=32)\n    # Print what the best validation MAE was for each model\n    print('Mdl1 Best Validation MAE: ',str(mae_best1))\n    print('Mdl2 Best Validation MAE: ',str(mae_best2))\n    print('Mdl3 Best Validation MAE: ',str(mae_best3))\n    # Get predictions on the test set for each model\n    mdl1_test_preds = mdl1.predict(Gxx_test,batch_size=32)\n    mdl2_test_preds = mdl2.predict(Gxx_test,batch_size=32)\n    mdl3_test_preds = mdl3.predict(Gxx_test,batch_size=32)\nt_total = time.time() - t1\nprint(\"Time for test predictions: \",str(t_total/60),\" minutes\")\n\n# Prepare the submission\nval_preds = (mdl1_val_preds + mdl2_val_preds + mdl3_val_preds) / 3\ntest_preds = (mdl1_test_preds + mdl2_test_preds + mdl3_test_preds) / 3 # Blending CNN models\nsubmission['time_to_failure'] = test_preds\n\n# Convert the submission to .csv\nsubmission.to_csv('submission.csv')\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5edbba2f2fa4b791be93b71c00f5222c4b9ecfae"},"cell_type":"code","source":"# Look at validation distributions\nBINS = np.linspace(0,16,100)\nplt.figure(figsize=(6.0,3.0),dpi=150)\nplt.hist(mdl1_val_preds, bins=BINS, density=True, label='Mdl1',alpha=0.2)\nplt.hist(mdl2_val_preds, bins=BINS, density=True, label='Mdl2',alpha=0.2)\nplt.hist(mdl3_val_preds, bins=BINS, density=True, label='Mdl3',alpha=0.2)\nplt.hist(val_preds, bins=BINS, density=True, label='Blend',alpha=0.3)\nplt.title('Validation Predictions Distribution')\nplt.legend(loc='upper right')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"63aad5e7d69d91fc70c8074c395ce2eb55f2ac4b"},"cell_type":"code","source":"# Look at the submission distributions\nBINS2 = np.linspace(0,16,100)\nplt.figure(figsize=(6.0,3.0),dpi=150)\nplt.hist(mdl1_test_preds, bins=BINS2, density=True, label='Mdl1', alpha=0.4)\nplt.hist(mdl2_test_preds, bins=BINS2, density=True, label='Mdl2', alpha=0.4)\nplt.hist(mdl3_test_preds, bins=BINS2, density=True, label='Mdl3', alpha=0.4)\nplt.hist(test_preds,bins=BINS2,density=True,label='Blend', alpha=0.5)\nplt.title('Test Predictions Distributions')\nplt.legend(loc='upper right')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fca38a9730cf6f150c99bf6ddc396a50e9dd203e"},"cell_type":"markdown","source":"**Final Analysis?**\n\nI don't really think spectrograms are going to be the way to go. <https://www.kaggle.com/fanconic/earthquake-cnn> did better than my spectrogram based method, using just the time series data. My hypothesis would be that gaining knowledge of the frequencies at the expense of time resolution probably isn't ideal for predicting time_to_failure. It was fun to try and optimize this method though."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}