{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1><center><font size=\"6\">CNNlstm less features </font></center></h1>\n\n<br>\n\n# <a id='0'>Content</a>\n\n- <a href='#1'>Introduction</a>  \n- <a href='#2'>Prepare the data analysis</a>  \n- <a href='#3'>Calculate aggregated features</a>\n- <a href='#4'>New features exploration</a>  \n- <a href='#5'>Conclusions</a>  \n- <a href='#6'>References</a>  \n"},{"metadata":{},"cell_type":"markdown","source":"# <a id='1'>Introduction</a>  \n\n## Simulated earthquake experiment\nThe data are from an experiment conducted on rock in a double direct shear geometry subjected to bi-axial loading, a classic laboratory earthquake model.\n\nTwo fault gouge layers are sheared simultaneously while subjected to a constant normal load and a prescribed shear velocity. The laboratory faults fail in repetitive cycles of stick and slip that is meant to mimic the cycle of loading and failure on tectonic faults. While the experiment is considerably simpler than a fault in Earth, it shares many physical characteristics.\n\nLos Alamos' initial work showed that the prediction of laboratory earthquakes from continuous seismic data is possible in the case of quasi-periodic laboratory seismic cycles.\n\n## Competition\nIn this competition, the team has provided a much more challenging dataset with considerably more aperiodic earthquake failures.\nObjective of the competition is to predict the failures for each test set."},{"metadata":{},"cell_type":"markdown","source":"# <a id='2'>Prepare the data analysis</a>  \n\n## Load packages"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport os\nimport time\nimport logging\nimport datetime\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport xgboost as xgb\nimport lightgbm as lgb\nfrom scipy import stats\nfrom tqdm import tqdm_notebook\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import StandardScaler\nfrom tqdm import tqdm\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"PATH=\"../input/\"\nos.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"There are {} files in test folder\".format(len(os.listdir(os.path.join(PATH, 'test' )))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\ntrain_df = pd.read_csv(os.path.join(PATH,'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_failure': np.float32})\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Train: rows:{} cols:{}\".format(train_df.shape[0], train_df.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='3'>Calculate aggregated features</a>  "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"rows = 150000\nsegments = int(np.floor(train_df.shape[0] / rows))\nprint(\"Number of segments: \", segments)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_X = pd.DataFrame(index=range(segments), dtype=np.float64)\ntrain_y = pd.DataFrame(index=range(segments), dtype=np.float64, columns=['time_to_failure'])\ntrain_X .shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def create_features(seg_id, seg, X):\n    xc = pd.Series(seg['acoustic_data'].values)   \n    zc = np.fft.fft(xc)\n    realFFT = np.real(zc)\n    imagFFT = np.imag(zc)\n    X.loc[seg_id, 'mean'] = xc.mean()\n    X.loc[seg_id, 'std'] = xc.std()\n    X.loc[seg_id, 'max'] = xc.max()\n    X.loc[seg_id, 'min'] = xc.min()\n    X.loc[seg_id, 'sum'] = xc.sum()\n    X.loc[seg_id, 'mad'] = xc.mad()\n  \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# iterate over all segments\nfor seg_id in tqdm_notebook(range(segments)):\n    seg = train_df.iloc[seg_id*rows:seg_id*rows+rows]\n    create_features(seg_id, seg, train_X)\n    train_y.loc[seg_id, 'time_to_failure'] = seg['time_to_failure'].values[-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" train_X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\ntest_X = pd.DataFrame(columns=train_X.columns, dtype=np.float64, index=submission.index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for seg_id in tqdm_notebook(test_X.index):\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    create_features(seg_id, seg, test_X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Train X: {} y: {} Test X: {}\".format(train_X.shape, train_y.shape, test_X.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_X.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test_X.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='4'>New features exploration</a> \n\n\n## Aggregated features\n\nLet's visualize the new features distributions. The graphs below shows the distplot (histograms and density plots) for all the new features, for train (<font color=\"green\">green</font>) and test (<font color=\"blue\">blue</font>) data."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_distplot(feature):\n    plt.figure(figsize=(16,6))\n    plt.title(\"Distribution of {} values in the train and test set\".format(feature))\n    sns.distplot(train_X[feature],color=\"green\", kde=True,bins=120, label='train')\n    sns.distplot(test_X[feature],color=\"blue\", kde=True,bins=120, label='test')\n    plt.legend()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_distplot_features(features, nlines=3, colors=['green', 'blue'], df1=train_X, df2=test_X):\n    i = 0\n    plt.figure()\n    fig, ax = plt.subplots(nlines,2,figsize=(16,4*nlines))\n    for feature in features:\n        i += 1\n        plt.subplot(nlines,2,i)\n        sns.distplot(df1[feature],color=colors[0], kde=True,bins=40, label='train')\n        sns.distplot(df2[feature],color=colors[1], kde=True,bins=40, label='test')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"features = ['mean', 'std', 'max', 'min', 'sum', 'mad']\n\nplot_distplot_features(features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Scaled features\n\nLet's scale now the aggregated features and show again the resulting graphs.   \nWe are fiting the scaler with both train and test data.\nWe use <font color=\"red\">red</font> from train and <font color=\"magenta\">magenta</font> for test data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"scaler = StandardScaler()\nscaler.fit(pd.concat([train_X, test_X]))\nscaled_train_X = pd.DataFrame(scaler.transform(train_X), columns=train_X.columns)\nscaled_test_X = pd.DataFrame(scaler.transform(test_X), columns=test_X.columns)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"features = ['mean', 'std', 'max', 'min', 'sum', 'mad']\nplot_distplot_features(features, nlines=4, colors=['red', 'magenta'], df1=scaled_train_X, df2=scaled_test_X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Aggregated features and time to failure\n\nLet's also show aggregated features and time to failure on the same graph. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_acc_agg_ttf_data(feature, title=\"Averaged accoustic data and ttf\"):\n    fig, ax1 = plt.subplots(figsize=(16, 8))\n    plt.title('Averaged accoustic data ({}) and time to failure'.format(feature))\n    plt.plot(train_X[feature], color='r')\n    ax1.set_xlabel('training samples')\n    ax1.set_ylabel('acoustic data ({})'.format(feature), color='r')\n    plt.legend(['acoustic data ({})'.format(feature)], loc=(0.01, 0.95))\n    ax2 = ax1.twinx()\n    plt.plot(train_y, color='b')\n    ax2.set_ylabel('time to failure', color='b')\n    plt.legend(['time to failure'], loc=(0.01, 0.9))\n    plt.grid(True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_acc_agg_ttf_data('mean')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_acc_agg_ttf_data('std')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_acc_agg_ttf_data('max')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plot_acc_agg_ttf_data('min')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_acc_agg_ttf_data('sum')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_acc_agg_ttf_data('mad')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='5'>Rearranging data </a> "},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_point=351\nendpoint=train_X.shape[0]-1\ntttt=validation_point-endpoint\n\nX_train=scaled_train_X.values[validation_point:endpoint,].reshape(tttt,3,2)\nprint('X_train.shape',X_train.shape)\ny_train=train_y.values[validation_point:endpoint,]\nprint('y_train.shape',y_train.shape)\n\nX_validation=scaled_train_X.values[0:validation_point-1,].reshape(validation_point-1,3,2)\nprint('X_validation.shape',X_validation.shape)\ny_validation=train_y.values[0:validation_point-1,]\nprint('y_validation.shape',y_validation.shape)\n\nX_train.shape[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='5'>Defining Model </a> "},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, CuDNNGRU, SimpleRNN, LSTM ,  Dropout, Activation, Flatten, Input, Conv1D, MaxPooling1D\nfrom keras.optimizers import adam\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.optimizers import RMSprop\nimport datetime\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#used to help some of the timing functions\nnow = datetime.datetime.now\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i = (X_train.shape[1],X_train.shape[2])\nmodel = Sequential ()\nmodel.add(Conv1D(2, 2, activation='relu', input_shape= i))\nmodel.add(MaxPooling1D(2))\nmodel.add(LSTM(50,  return_sequences=True))\nmodel.add(LSTM(10))\nmodel.add(Dense(240))\nmodel.add(Dense(120))\nmodel.add(Dense(60))\nmodel.add(Dense(30))\nmodel.add(Dense(1))\n\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='5'>Compile and fit model </a>                                            "},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras\nfrom keras.optimizers import RMSprop\nopt = keras.optimizers.adam(lr=.005)\n\nmodel.compile(loss=\"mae\",\n              optimizer=opt, metrics=['mean_absolute_error'])\n             # metrics=['accuracy'])\n\n\nbatch_size = 128 # mini-batch with 32 examples\nepochs = 50\nt = now()\n\nhistory = model.fit(\n    X_train, y_train,\n    batch_size=batch_size,\n    epochs=epochs,\n    verbose=1,\n    validation_data=(X_validation ,y_validation ))\nprint('Training time: %s' % (now() - t))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='5'>Load submission file </a>                                                                                         \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id', dtype={\"time_to_failure\": np.float32})\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='5'>Prepare submission data </a>                              "},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test=scaled_test_X.values.reshape(test_X.shape[0],3,2)\nprint(X_test.shape)\n\nfor i, seg_id in enumerate(tqdm(submission.index)):\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    x = seg['acoustic_data'].values\n    submission.time_to_failure[i]= model.predict(np.expand_dims(X_test[i], 0))\n    \n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_newfeatures=submission\nsubmission_newfeatures.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='5'>Save submission file </a>                              "},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_newfeatures.to_csv('submission_newfeatures.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a id='6'>References</a>  \n\n[1] LANL Earthquake Prediction, https://www.kaggle.com/c/LANL-Earthquake-Prediction  \n[2] Shaking Earth, https://www.kaggle.com/allunia/shaking-earth  \n[3] Earthquake FE - more features and samles, https://www.kaggle.com/artgor/earthquakes-fe-more-features-and-samples  \n[4] Laboratory observations of slow earthquakes and the spectrum of tectonic fault slip modes, https://www.nature.com/articles/ncomms11104   \n[5] Machine Learning Predicts Laboratory Earthquakes, https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2017GL074677  \n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}