{"cells":[{"metadata":{},"cell_type":"markdown","source":"# LANL Earthquake Prediction\n# Can you predict upcoming laboratory earthquakes?\n# Maicon Moda"},{"metadata":{},"cell_type":"markdown","source":"#### Data Description\n\nThe goal of this competition is to use seismic signals to predict the timing of laboratory earthquakes. The data comes from a well-known experimental set-up used to study earthquake physics. The acoustic_data input signal is used to predict the time remaining before the next laboratory earthquake (time_to_failure).\n\nThe training data is a single, continuous segment of experimental data. The test data consists of a folder containing many small segments. The data within each test file is continuous, but the test files do not represent a continuous segment of the experiment; thus, the predictions cannot be assumed to follow the same regular pattern seen in the training file.\n\nFor each seg_id in the test folder, you should predict a single time_to_failure corresponding to the time between the last row of the segment and the next laboratory earthquake.\n\n#### File descriptions\n1. train.csv - A single, continuous training segment of experimental data.\n2. test - A folder containing many small segments of test data.\n3. sample_sumbission.csv - A sample submission file in the correct format.\n\n#### Data fields\n1. acoustic_data - the seismic signal [int16]\n2. time_to_failure - the time (in seconds) until the next laboratory earthquake [float64]\n3. seg_id - the test segment ids for which predictions should be made (one prediction per segment)"},{"metadata":{"trusted":false},"cell_type":"code","source":"import os\npath = os.getcwd()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(os.listdir('../input'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Pandas and numpy for data manipulation\nimport pandas as pd\nimport numpy as np\n\n# No warnings about setting value on copy of slice\npd.options.mode.chained_assignment = None\n\n# pandas doesn't show us all the decimals\npd.options.display.precision = 15\n\n# Matplotlib visualization\nimport matplotlib.pyplot as plt\nplt.style.use('fivethirtyeight')\n%matplotlib inline\n\n# Scipy\nfrom scipy import stats\nfrom scipy.signal import hann\nfrom scipy.signal import hilbert\nfrom scipy.signal import convolve\n\n# Scaling values\nfrom sklearn.preprocessing import StandardScaler\n\n# Splitting data into training and testing\nfrom sklearn.model_selection import train_test_split\n\n# Metric\nfrom sklearn.metrics import mean_absolute_error\n\n# Machine Learning\nfrom xgboost.sklearn import XGBRegressor\n\n# Hyperparameter tuning\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\n\n# Others\nfrom sklearn.linear_model import LinearRegression\nfrom tqdm import tqdm_notebook\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Read in data into a dataframe \ntrain = pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float32})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Exploration"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Show dataframe columns\nprint(train.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Display top of dataframe\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Display bottom of dataframe\ntrain.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Display the shape of dataframe\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# See the column data types and non-missing values\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\ntrain_ad_sample_df = train['acoustic_data'].values[::100]\ntrain_ttf_sample_df = train['time_to_failure'].values[::100]\n\ndef plot_acc_ttf_data(train_ad_sample_df, train_ttf_sample_df, title=\"Acoustic data and time to failure: 1% sampled data\"):\n    fig, ax1 = plt.subplots(figsize=(12, 8))\n    plt.title(title)\n    plt.plot(train_ad_sample_df, color='r')\n    ax1.set_ylabel('acoustic data', color='r')\n    plt.legend(['acoustic data'], loc=(0.01, 0.95))\n    ax2 = ax1.twinx()\n    plt.plot(train_ttf_sample_df, color='b')\n    ax2.set_ylabel('time to failure', color='b')\n    plt.legend(['time to failure'], loc=(0.01, 0.9))\n    plt.grid(True)\n\nplot_acc_ttf_data(train_ad_sample_df, train_ttf_sample_df)\ndel train_ad_sample_df\ndel train_ttf_sample_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\ntrain_ad_sample_df = train['acoustic_data'].values[:6291455]\ntrain_ttf_sample_df = train['time_to_failure'].values[:6291455]\nplot_acc_ttf_data(train_ad_sample_df, train_ttf_sample_df, title=\"Acoustic data and time to failure: 1% of data\")\ndel train_ad_sample_df\ndel train_ttf_sample_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Engineering"},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\nrows = 150000\nsegments = int(np.floor(train.shape[0] / rows))\nprint(\"Number of segments: \", segments)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\ndef add_trend_feature(arr, abs_values=False):\n    idx = np.array(range(len(arr)))\n    if abs_values:\n        arr = np.abs(arr)\n    lr = LinearRegression()\n    lr.fit(idx.reshape(-1, 1), arr)\n    return lr.coef_[0]\n\ndef classic_sta_lta(x, length_sta, length_lta):\n    sta = np.cumsum(x ** 2)\n    # Convert to float\n    sta = np.require(sta, dtype=np.float)\n    # Copy for LTA\n    lta = sta.copy()\n    # Compute the STA and the LTA\n    sta[length_sta:] = sta[length_sta:] - sta[:-length_sta]\n    sta /= length_sta\n    lta[length_lta:] = lta[length_lta:] - lta[:-length_lta]\n    lta /= length_lta\n    # Pad zeros\n    sta[:length_lta - 1] = 0\n    # Avoid division by zero by setting zero values to tiny float\n    dtiny = np.finfo(0.0).tiny\n    idx = lta < dtiny\n    lta[idx] = dtiny\n    return sta / lta","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\n# Create a training file with simple derived features\nX_train = pd.DataFrame(index=range(segments), dtype=np.float64)\ny_train = pd.DataFrame(index=range(segments), dtype=np.float64, columns=['time_to_failure'])\ntotal_mean = train['acoustic_data'].mean()\ntotal_std = train['acoustic_data'].std()\ntotal_max = train['acoustic_data'].max()\ntotal_min = train['acoustic_data'].min()\ntotal_sum = train['acoustic_data'].sum()\ntotal_abs_sum = np.abs(train['acoustic_data']).sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\ndef create_features(seg_id, seg, X):\n    xc = pd.Series(seg['acoustic_data'].values)\n    zc = np.fft.fft(xc)\n    \n    X.loc[seg_id, 'mean'] = xc.mean()\n    X.loc[seg_id, 'std'] = xc.std()\n    X.loc[seg_id, 'max'] = xc.max()\n    X.loc[seg_id, 'min'] = xc.min()\n    \n    #FFT transform values\n    realFFT = np.real(zc)\n    imagFFT = np.imag(zc)\n    X.loc[seg_id, 'Rmean'] = realFFT.mean()\n    X.loc[seg_id, 'Rstd'] = realFFT.std()\n    X.loc[seg_id, 'Rmax'] = realFFT.max()\n    X.loc[seg_id, 'Rmin'] = realFFT.min()\n    X.loc[seg_id, 'Imean'] = imagFFT.mean()\n    X.loc[seg_id, 'Istd'] = imagFFT.std()\n    X.loc[seg_id, 'Imax'] = imagFFT.max()\n    X.loc[seg_id, 'Imin'] = imagFFT.min()\n    X.loc[seg_id, 'Rmean_last_5000'] = realFFT[-5000:].mean()\n    X.loc[seg_id, 'Rstd__last_5000'] = realFFT[-5000:].std()\n    X.loc[seg_id, 'Rmax_last_5000'] = realFFT[-5000:].max()\n    X.loc[seg_id, 'Rmin_last_5000'] = realFFT[-5000:].min()\n    X.loc[seg_id, 'Rmean_last_15000'] = realFFT[-15000:].mean()\n    X.loc[seg_id, 'Rstd_last_15000'] = realFFT[-15000:].std()\n    X.loc[seg_id, 'Rmax_last_15000'] = realFFT[-15000:].max()\n    X.loc[seg_id, 'Rmin_last_15000'] = realFFT[-15000:].min()\n    \n    X.loc[seg_id, 'mean_change_abs'] = np.mean(np.diff(xc))\n    X.loc[seg_id, 'mean_change_rate'] = np.mean(np.nonzero((np.diff(xc) / xc[:-1]))[0])\n    X.loc[seg_id, 'abs_max'] = np.abs(xc).max()\n    X.loc[seg_id, 'abs_min'] = np.abs(xc).min()\n    \n    X.loc[seg_id, 'std_first_50000'] = xc[:50000].std()\n    X.loc[seg_id, 'std_last_50000'] = xc[-50000:].std()\n    X.loc[seg_id, 'std_first_10000'] = xc[:10000].std()\n    X.loc[seg_id, 'std_last_10000'] = xc[-10000:].std()\n    \n    X.loc[seg_id, 'avg_first_50000'] = xc[:50000].mean()\n    X.loc[seg_id, 'avg_last_50000'] = xc[-50000:].mean()\n    X.loc[seg_id, 'avg_first_10000'] = xc[:10000].mean()\n    X.loc[seg_id, 'avg_last_10000'] = xc[-10000:].mean()\n    \n    X.loc[seg_id, 'min_first_50000'] = xc[:50000].min()\n    X.loc[seg_id, 'min_last_50000'] = xc[-50000:].min()\n    X.loc[seg_id, 'min_first_10000'] = xc[:10000].min()\n    X.loc[seg_id, 'min_last_10000'] = xc[-10000:].min()\n    \n    X.loc[seg_id, 'max_first_50000'] = xc[:50000].max()\n    X.loc[seg_id, 'max_last_50000'] = xc[-50000:].max()\n    X.loc[seg_id, 'max_first_10000'] = xc[:10000].max()\n    X.loc[seg_id, 'max_last_10000'] = xc[-10000:].max()\n    \n    X.loc[seg_id, 'max_to_min'] = xc.max() / np.abs(xc.min())\n    X.loc[seg_id, 'max_to_min_diff'] = xc.max() - np.abs(xc.min())\n    X.loc[seg_id, 'count_big'] = len(xc[np.abs(xc) > 500])\n    X.loc[seg_id, 'sum'] = xc.sum()\n    \n    X.loc[seg_id, 'mean_change_rate_first_50000'] = np.mean(np.nonzero((np.diff(xc[:50000]) / xc[:50000][:-1]))[0])\n    X.loc[seg_id, 'mean_change_rate_last_50000'] = np.mean(np.nonzero((np.diff(xc[-50000:]) / xc[-50000:][:-1]))[0])\n    X.loc[seg_id, 'mean_change_rate_first_10000'] = np.mean(np.nonzero((np.diff(xc[:10000]) / xc[:10000][:-1]))[0])\n    X.loc[seg_id, 'mean_change_rate_last_10000'] = np.mean(np.nonzero((np.diff(xc[-10000:]) / xc[-10000:][:-1]))[0])\n    \n    X.loc[seg_id, 'q95'] = np.quantile(xc, 0.95)\n    X.loc[seg_id, 'q99'] = np.quantile(xc, 0.99)\n    X.loc[seg_id, 'q05'] = np.quantile(xc, 0.05)\n    X.loc[seg_id, 'q01'] = np.quantile(xc, 0.01)\n    \n    X.loc[seg_id, 'abs_q95'] = np.quantile(np.abs(xc), 0.95)\n    X.loc[seg_id, 'abs_q99'] = np.quantile(np.abs(xc), 0.99)\n    X.loc[seg_id, 'abs_q05'] = np.quantile(np.abs(xc), 0.05)\n    X.loc[seg_id, 'abs_q01'] = np.quantile(np.abs(xc), 0.01)\n    \n    X.loc[seg_id, 'trend'] = add_trend_feature(xc)\n    X.loc[seg_id, 'abs_trend'] = add_trend_feature(xc, abs_values=True)\n    X.loc[seg_id, 'abs_mean'] = np.abs(xc).mean()\n    X.loc[seg_id, 'abs_std'] = np.abs(xc).std()\n    \n    X.loc[seg_id, 'mad'] = xc.mad()\n    X.loc[seg_id, 'kurt'] = xc.kurtosis()\n    X.loc[seg_id, 'skew'] = xc.skew()\n    X.loc[seg_id, 'med'] = xc.median()\n    \n    X.loc[seg_id, 'Hilbert_mean'] = np.abs(hilbert(xc)).mean()\n    X.loc[seg_id, 'Hann_window_mean'] = (convolve(xc, hann(150), mode='same') / sum(hann(150))).mean()\n    X.loc[seg_id, 'classic_sta_lta1_mean'] = classic_sta_lta(xc, 500, 10000).mean()\n    X.loc[seg_id, 'classic_sta_lta2_mean'] = classic_sta_lta(xc, 5000, 100000).mean()\n    X.loc[seg_id, 'classic_sta_lta3_mean'] = classic_sta_lta(xc, 3333, 6666).mean()\n    X.loc[seg_id, 'classic_sta_lta4_mean'] = classic_sta_lta(xc, 10000, 25000).mean()\n    X.loc[seg_id, 'Moving_average_700_mean'] = xc.rolling(window=700).mean().mean(skipna=True)\n    X.loc[seg_id, 'Moving_average_1500_mean'] = xc.rolling(window=1500).mean().mean(skipna=True)\n    X.loc[seg_id, 'Moving_average_3000_mean'] = xc.rolling(window=3000).mean().mean(skipna=True)\n    X.loc[seg_id, 'Moving_average_6000_mean'] = xc.rolling(window=6000).mean().mean(skipna=True)\n    ewma = pd.Series.ewm\n    X.loc[seg_id, 'exp_Moving_average_300_mean'] = (ewma(xc, span=300).mean()).mean(skipna=True)\n    X.loc[seg_id, 'exp_Moving_average_3000_mean'] = ewma(xc, span=3000).mean().mean(skipna=True)\n    X.loc[seg_id, 'exp_Moving_average_30000_mean'] = ewma(xc, span=6000).mean().mean(skipna=True)\n    no_of_std = 2\n    X.loc[seg_id, 'MA_700MA_std_mean'] = xc.rolling(window=700).std().mean()\n    X.loc[seg_id,'MA_700MA_BB_high_mean'] = (X.loc[seg_id, 'Moving_average_700_mean'] + no_of_std * X.loc[seg_id, 'MA_700MA_std_mean']).mean()\n    X.loc[seg_id,'MA_700MA_BB_low_mean'] = (X.loc[seg_id, 'Moving_average_700_mean'] - no_of_std * X.loc[seg_id, 'MA_700MA_std_mean']).mean()\n    X.loc[seg_id, 'MA_400MA_std_mean'] = xc.rolling(window=400).std().mean()\n    X.loc[seg_id,'MA_400MA_BB_high_mean'] = (X.loc[seg_id, 'Moving_average_700_mean'] + no_of_std * X.loc[seg_id, 'MA_400MA_std_mean']).mean()\n    X.loc[seg_id,'MA_400MA_BB_low_mean'] = (X.loc[seg_id, 'Moving_average_700_mean'] - no_of_std * X.loc[seg_id, 'MA_400MA_std_mean']).mean()\n    X.loc[seg_id, 'MA_1000MA_std_mean'] = xc.rolling(window=1000).std().mean()\n    \n    X.loc[seg_id, 'iqr'] = np.subtract(*np.percentile(xc, [75, 25]))\n    X.loc[seg_id, 'q999'] = np.quantile(xc,0.999)\n    X.loc[seg_id, 'q001'] = np.quantile(xc,0.001)\n    X.loc[seg_id, 'ave10'] = stats.trim_mean(xc, 0.1)\n    \n    for windows in [10, 100, 1000]:\n        x_roll_std = xc.rolling(windows).std().dropna().values\n        x_roll_mean = xc.rolling(windows).mean().dropna().values\n        \n        X.loc[seg_id, 'ave_roll_std_' + str(windows)] = x_roll_std.mean()\n        X.loc[seg_id, 'std_roll_std_' + str(windows)] = x_roll_std.std()\n        X.loc[seg_id, 'max_roll_std_' + str(windows)] = x_roll_std.max()\n        X.loc[seg_id, 'min_roll_std_' + str(windows)] = x_roll_std.min()\n        X.loc[seg_id, 'q01_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.01)\n        X.loc[seg_id, 'q05_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.05)\n        X.loc[seg_id, 'q95_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.95)\n        X.loc[seg_id, 'q99_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.99)\n        X.loc[seg_id, 'av_change_abs_roll_std_' + str(windows)] = np.mean(np.diff(x_roll_std))\n        X.loc[seg_id, 'av_change_rate_roll_std_' + str(windows)] = np.mean(np.nonzero((np.diff(x_roll_std) / x_roll_std[:-1]))[0])\n        X.loc[seg_id, 'abs_max_roll_std_' + str(windows)] = np.abs(x_roll_std).max()\n        \n        X.loc[seg_id, 'ave_roll_mean_' + str(windows)] = x_roll_mean.mean()\n        X.loc[seg_id, 'std_roll_mean_' + str(windows)] = x_roll_mean.std()\n        X.loc[seg_id, 'max_roll_mean_' + str(windows)] = x_roll_mean.max()\n        X.loc[seg_id, 'min_roll_mean_' + str(windows)] = x_roll_mean.min()\n        X.loc[seg_id, 'q01_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.01)\n        X.loc[seg_id, 'q05_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.05)\n        X.loc[seg_id, 'q95_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.95)\n        X.loc[seg_id, 'q99_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.99)\n        X.loc[seg_id, 'av_change_abs_roll_mean_' + str(windows)] = np.mean(np.diff(x_roll_mean))\n        X.loc[seg_id, 'av_change_rate_roll_mean_' + str(windows)] = np.mean(np.nonzero((np.diff(x_roll_mean) / x_roll_mean[:-1]))[0])\n        X.loc[seg_id, 'abs_max_roll_mean_' + str(windows)] = np.abs(x_roll_mean).max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction\n# iterate over all segments\nfor seg_id in tqdm_notebook(range(segments)):\n    seg = train.iloc[seg_id*rows:seg_id*rows+rows]\n    create_features(seg_id, seg, X_train)\n    y_train.loc[seg_id, 'time_to_failure'] = seg['time_to_failure'].values[-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"y_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Process Test Data"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Read in data into a dataframe \nsubmission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X_test = pd.DataFrame(columns=X_train.columns, dtype=np.float64, index=submission.index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for seg_id in tqdm_notebook(X_test.index):\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    create_features(seg_id, seg, X_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save New Datasets"},{"metadata":{"trusted":false},"cell_type":"code","source":"X_train.to_csv('training_features.csv', index = False)\nX_test.to_csv('testing_features.csv', index = False)\ny_train.to_csv('training_labels.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Read in data into dataframes \ntrain_features = pd.read_csv('training_features.csv')\ntrain_labels = pd.read_csv('training_labels.csv')\ntest_features = pd.read_csv('testing_features.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split Into Training and Testing Sets"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Split into training and testing set\nX_train, X_test, y_train, y_test = train_test_split(train_features, train_labels, test_size = 0.2, random_state = 42)\n\nprint(X_train.shape)\nprint(X_test.shape)\nprint(y_train.shape)\nprint(y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Scaling Features"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Create the scaler object with a range of 0-1\nscaler = StandardScaler()\n\n# Fit on the training data\nscaler.fit(X_train)\n\n# Transform both the training and testing data\nX_train = scaler.transform(X_train)\nX_test = scaler.transform(X_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluating and Comparing Machine Learning Models"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Function to calculate mean absolute error\ndef mae(y_true, y_pred):\n    return mean_absolute_error(y_true.values.flatten(), y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Takes in a model, trains the model, and evaluates the model on the test set\ndef fit_and_evaluate(model):\n    \n    # Train the model\n    model.fit(X_train, y_train.values.flatten())\n    \n    # Make predictions and evaluate\n    model_pred = model.predict(X_test)\n    model_mae = mae(y_test, model_pred)\n        \n    # Return the performance metric\n    return model_mae","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"xgboost = XGBRegressor()\nxgboost_mae = fit_and_evaluate(xgboost)\n\nprint('XGBoost Regression Performance: MAE = %0.3f' % xgboost_mae)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Model Optimization "},{"metadata":{"trusted":false},"cell_type":"code","source":"# Define the grid of hyperparameters to search\none_to_left = stats.beta(10, 1)  \nfrom_zero_positive = stats.expon(0, 50)\n\nhyperparameter_grid = {'n_estimators': stats.randint(3, 40),\n                       'max_depth': stats.randint(3, 40),\n                       'learning_rate': stats.uniform(0.05, 0.4),\n                       'colsample_bytree': one_to_left,\n                       'subsample': one_to_left,\n                       'gamma': stats.uniform(0, 10),\n                       'reg_alpha': from_zero_positive,\n                       'min_child_weight': from_zero_positive}","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Create the model to use for hyperparameter tuning\nmodel = XGBRegressor(nthreads=-1, random_state = 42)\n\n# Set up the random search with 10-fold cross validation\nrandom_cv = RandomizedSearchCV(estimator=model,\n                               param_distributions=hyperparameter_grid,\n                               cv=10, n_iter=100, \n                               scoring = 'neg_mean_absolute_error',\n                               n_jobs = -1, verbose = 1, \n                               return_train_score = True,\n                               random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Fit on the training data\nrandom_cv.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Get all of the cv results and sort by the test performance\nrandom_results = pd.DataFrame(random_cv.cv_results_).sort_values('mean_test_score', ascending = False)\n\nrandom_results.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"random_cv.best_estimator_","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Default model\ndefault_model = XGBRegressor(nthreads=-1, random_state = 42)\n\n# Select the best model\nfinal_model = random_cv.best_estimator_\n\nfinal_model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"default_model.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"final_model.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"default_pred = default_model.predict(X_test)\nfinal_pred = final_model.predict(X_test)\n\nprint('Default model performance on the test set: MAE = %0.3f.' % mae(y_test, default_pred))\nprint('Final model performance on the test set:   MAE = %0.3f.' % mae(y_test, final_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"final_pred","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Read in data into a dataframe \nsubmission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Collecting submission dataset\nX_submission = test_features","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X_submission.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"X_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Create the scaler object with a range of 0-1\nscaler = StandardScaler()\n\n# Fit on the training data\nscaler.fit(X_submission)\n\n# Transform the test data\nX_submission = scaler.transform(X_submission)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Make predictions on the submission set\nmodel_pred = final_model.predict(X_submission)\nmodel_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Final dataset with predictions\nsubmission['time_to_failure'] = model_pred\n\nsubmission.to_csv('submission.csv', index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"submission.head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## End"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}