{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"![](http://retrofittingca.com/wp-content/uploads/2017/02/PROOFED-RetrofittingCA-WhyEarthquakesAreSoHardtoPredictJanuary312017-PIC.jpg)\n"},{"metadata":{"_uuid":"65471153dfe762b8fc524fd7fca5f8aa0913e0dc"},"cell_type":"markdown","source":"# <div style=\"text-align: center\"> Reader can find new and interesting feature engineering, model modifications and Bayesian Hyperparameter Optimization\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":" <a id=\"top\"></a> <br>\n## Notebook  Content\n3. [Feature Engineering (continued)](#1)\n1. [New Features](#2)\n1. [Model ](#3)\n1. [Bayesian Hyperparamter Optimization ](#4)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"070d0a8fad4e8feddebc992a4bb04968b47008e7"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom tqdm import tqdm_notebook\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.svm import NuSVR, SVR\nfrom sklearn.metrics import mean_absolute_error\npd.options.display.precision = 15\nfrom sklearn.svm import NuSVR, SVR\nimport lightgbm as lgb\nimport xgboost as xgb\nimport time\nimport datetime\n\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold, KFold, RepeatedKFold\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error\nfrom sklearn.linear_model import Ridge, RidgeCV\nimport gc\nfrom catboost import CatBoostRegressor\nimport seaborn as sns\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ff208e2e3dfb4961fa7a00543aa53c5dbad0261"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float64})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b47c07d51efa05bfbe8d517bad43ebc0d2b067ad"},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n#  3-Feature Engineering  \n\n\"#\"-commented out code is optional, possibly features lie also there"},{"metadata":{"_uuid":"6915850c3a7111f9219916cda14959bd96354826"},"cell_type":"markdown","source":"**a) Sharp-rise indicator**\nAs already discussed we want an indicator that tells explicitly about the sharp rise in value. "},{"metadata":{"trusted":true,"_uuid":"770eeedcb268cac5e9b46abffa5e2ffa523ac713"},"cell_type":"code","source":"#min_1 = train.acoustic_data.mean() - 3 * train.acoustic_data.std()\n#max_1 = train.acoustic_data.mean() + 3 * train.acoustic_data.std() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"220e0b4e6b364387d64238960d62e8a32ff92958"},"cell_type":"code","source":"#train[\"sharp_rise1\"] = np.where((train.acoustic_data >= min_1) & (train.acoustic_data <= max_1), 0, 100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e130a416d24174b6a8d29b1cb1fe1f59b31f27e"},"cell_type":"markdown","source":"Another assumption. Lets say that earthquake is does not contain one sharp jump in signal values, but a few medium sized ones. (standard deviation 2) So we should give our algorithm the opportunity to take that also into account, but with a smaller value"},{"metadata":{"trusted":true,"_uuid":"41b0b5c6b3739459bc40f5ce0db069eb0c266641"},"cell_type":"code","source":"#del min_1,max_1\n#gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1bdc7dba218d79d29ee70b3147c601bb5fb9f1ef"},"cell_type":"code","source":"#min_2 = train.acoustic_data.mean() - 2 * train.acoustic_data.std()\n#max_2 = train.acoustic_data.mean() + 2 * train.acoustic_data.std() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2811f4d5e8b4e8eda976544aff779ee63ab6b48a"},"cell_type":"code","source":"#train[\"sharp_rise2\"] = np.where((train.acoustic_data >= min_2) & (train.acoustic_data <= max_2), 0, 50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa2190e4232aa2e949ff5fa7902af7d831e3c7d0"},"cell_type":"code","source":"#del min_2,max_2\n#gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea759b58b79b82258c5c2aa577d9167721479851"},"cell_type":"markdown","source":"Now before I come to aggregations I would like to touch upon second (red graph) from the previous Part 1. I said that we can see from the analysis that there are some distinct earthquake times and we should analyse them. Than I noticed [allunia](https://www.kaggle.com/allunia/shaking-earth) did GREAT work already on that. In essence we want to inspect the **differences**"},{"metadata":{"trusted":true,"_uuid":"397af18dc6c98cd2a29e02ea25b938fb64fc084d"},"cell_type":"code","source":"#differences = np.diff(train.time_to_failure)\n#train = train.drop(train.index[len(train)-1])\n#train[\"differences\"] = differences\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4904519442d0779c60fb5c35f8d79c8832ee297"},"cell_type":"code","source":"#train.differences.unique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0a3ab5d8d12df3d0dc588721206cc9cc9cffb74"},"cell_type":"markdown","source":"Hmm if we look closely, 3 digits are almost the same **BUT** not exactly as in the allunia kernel. The three values are basically \n\n* -1.00000000e-09 +- (-1.00000000e-09)\n* -9.95496600e-04 +- (-9.95496600e-04)\n* -1.09549550e-03 +- (-1.09549550e-03)"},{"metadata":{"trusted":true,"_uuid":"7bda4923a08971669ce4b7cd0e2bc45a058ed2b2"},"cell_type":"code","source":"#train[\"differences\"] = np.around(train[\"differences\"],10)\n#train.differences.unique()\n#train = train.convert_objects(convert_numeric=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ee033b37c05d3a5c921953c41cda703c33c9e1c"},"cell_type":"markdown","source":"Now we could write a function (or a one-liner) that checks whether the value in differences column is in one of these 3 intervals than set it to the fixed values. But it wont make much difference if we let it be. What we do want to create is additional variables/features from this one, if we do the rounding up right we can avoid it:"},{"metadata":{"trusted":true,"_uuid":"6ab509d20a8bcf126a94d3c8d735a9eeb016dce0"},"cell_type":"code","source":"#train[\"change\"]=train.differences * 1e9 + 1\n\n#train.change.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1bcd68aa09f04bb1ea74ff3d7b72d5564eb8539c"},"cell_type":"code","source":"#train[\"change\"] = np.around(train[\"change\"],3)\n#train[\"change\"]=np.floor(train[\"change\"])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbfc1744faf782497220bfaa6693ec62aa8e97e5"},"cell_type":"code","source":"#train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d2011e8834ac7cbdc757d8bf6d344af432ea75a"},"cell_type":"markdown","source":"**IMPLEMENTATION** So how do we implement these values. Problem is that we engineered these features based on **target** variable, so when we try to do create the same features on the test set that wont be possible (target values on the test set is what we are trying to predict) But luckily, acoustic_data are all integers. So they will take finite number of values. After we create these 4 Columns for the train data, we can map these values depending on the values of acoustic_data in the test set, since they  are all integers. Ofcourse some of the values wont be matched but we can just impute these values (ffil or bfil seams reasonable) **Potential problem** We have to check 150 000 * 2600 rows of data to map these values, I am afraid it will be to costly on the memory when working with pandas. For someone who implemented dusk as I advised in first tutorial it should work fine. [Memory solutions](https://www.kaggle.com/zikazika/memory-problems)"},{"metadata":{"trusted":true,"_uuid":"53c94ac3812f81edfa062446ef1702ed27979070"},"cell_type":"code","source":"#columns = ['sharp_rise1', 'sharp_rise2',\"differences\",\"change\"]\n#train.drop(columns, inplace=True, axis=1)\n#del differences\n#gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"769b16f202b3d7bf61fcebb623641e42151d2812"},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n# **Features:**\n#  Hilbert transform\n# Hann Window\n# classic_sta_lta\n\nClarification [consult](http://docs.obspy.org/tutorial/code_snippets/trigger_tutorial.html)\n# +\n# Various variations of moving averages\nRegarding MA values and their derivation. WE can gauge the lookback horizont and combinations of MA variables with feature importance. So I did have some baseline, than I saw what variables are the most potent and I played around a bit until I found some indicators. \nSame logic can be applied to quantiles. After all x is just an series of integer values, and we know from eda that right before earthquake (but not exactly next milisecond) accustic values will be huge ---> in the top of the quantiles. 99 quantile is too much as we can see from graphs but around 95 is the sweetspot---->modify the values of q\n\n\n**IMPLICATION** Implication of the same thinking that went into modifying quantiles. If we know what we know about values of signal right before earthquake than modifying the values \"std_last_10000\" and other similiar variables should make an impact. Why? Simple hypothesis: For example in the last 5000 values of accoustic_signal we will find huge standard deviation. That can be powerful predicator\n\n\n# GOAL: I think an avic reader can find a systemic way to gauge these values, and not only trial&error"},{"metadata":{"trusted":true,"_uuid":"6f11c39bf45fbd030a22e44d9df482dbdad37745"},"cell_type":"code","source":"from scipy.signal import hilbert\nfrom scipy.signal import hann\nfrom scipy.signal import convolve\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f079efe84195831ab6bf83fefe79f869115ce9ca"},"cell_type":"code","source":"def classic_sta_lta(x, length_sta, length_lta):\n    \n    sta = np.cumsum(x ** 2)\n\n    # Convert to float\n    sta = np.require(sta, dtype=np.float)\n\n    # Copy for LTA\n    lta = sta.copy()\n\n    # Compute the STA and the LTA\n    sta[length_sta:] = sta[length_sta:] - sta[:-length_sta]\n    sta /= length_sta\n    lta[length_lta:] = lta[length_lta:] - lta[:-length_lta]\n    lta /= length_lta\n\n    # Pad zeros\n    sta[:length_lta - 1] = 0\n\n    # Avoid division by zero by setting zero values to tiny float\n    dtiny = np.finfo(0.0).tiny\n    idx = lta < dtiny\n    lta[idx] = dtiny\n\n    return sta / lta","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c06f3c2343801147f36586df44443dfb39646a43"},"cell_type":"markdown","source":"AS in [artgor](https://www.kaggle.com/artgor/earthquakes-fe-more-features-and-samples), be sure to change pd.Series(x), to be ablo to create new features, since rolling works on pandas series object"},{"metadata":{"trusted":true,"_uuid":"68a8ac57071bad7897d26635350b45a41873bbf1"},"cell_type":"code","source":"# Create a training file with simple derived features\nrows = 150_000\nsegments = int(np.floor(train.shape[0] / rows))\n\nX_tr = pd.DataFrame(index=range(segments), dtype=np.float64,\n                       columns=['mean', 'std', 'max', 'min',\n                               'mean_change_abs', 'mean_change_rate', 'abs_max', 'abs_min',\n                               'std_first_50000', 'std_last_50000', 'std_first_10000', 'std_last_10000',\n                               'avg_first_50000', 'avg_last_50000', 'avg_first_10000', 'avg_last_10000',\n                               'min_first_50000', 'min_last_50000', 'min_first_10000', 'min_last_10000',\n                               'max_first_50000', 'max_last_50000', 'max_first_10000', 'max_last_10000',\n                               'max_to_min', 'max_to_min_diff', 'count_big', 'sum',\n                               'mean_change_rate_first_50000', 'mean_change_rate_last_50000', 'mean_change_rate_first_10000', 'mean_change_rate_last_10000','q70','q75','q60','q65','q85',\"q90\",'q80','q95','q99','Hilbert_mean','Hann_window_mean','classic_sta_lta1_mean','classic_sta_lta2_mean','classic_sta_lta3_mean','classic_sta_lta4_mean','Moving_average_700_mean','Moving_average_1500_mean','Moving_average_3000_mean','Moving_average_6000_mean','exp_Moving_average_300_mean','exp_Moving_average_3000_mean','exp_Moving_average_30000_mean','MA_700MA_std_mean','MA_700MA_BB_high_mean','MA_700MA_BB_low_mean','MA_400MA_std_mean','MA_400MA_BB_high_mean','MA_400MA_BB_low_mean','MA_1000MA_std_mean'])\ny_tr = pd.DataFrame(index=range(segments), dtype=np.float64,\n                       columns=['time_to_failure'])\n\ntotal_mean = train['acoustic_data'].mean()\ntotal_std = train['acoustic_data'].std()\ntotal_max = train['acoustic_data'].max()\ntotal_min = train['acoustic_data'].min()\ntotal_sum = train['acoustic_data'].sum()\ntotal_abs_max = np.abs(train['acoustic_data']).sum()\n\nfor segment in tqdm_notebook(range(segments)):\n    seg = train.iloc[segment*rows:segment*rows+rows]\n    x = pd.Series(seg['acoustic_data'].values)\n    y = seg['time_to_failure'].values[-1]\n    \n    y_tr.loc[segment, 'time_to_failure'] = y\n    X_tr.loc[segment, 'mean'] = x.mean()\n    X_tr.loc[segment, 'std'] = x.std()\n    X_tr.loc[segment, 'max'] = x.max()\n    X_tr.loc[segment, 'min'] = x.min()\n    \n    \n    X_tr.loc[segment, 'mean_change_abs'] = np.mean(np.diff(x))\n    X_tr.loc[segment, 'mean_change_rate'] = np.mean(np.nonzero((np.diff(x) / x[:-1]))[0])\n    X_tr.loc[segment, 'abs_max'] = np.abs(x).max()\n    X_tr.loc[segment, 'abs_min'] = np.abs(x).min()\n    \n    X_tr.loc[segment, 'std_first_50000'] = x[:50000].std()\n    X_tr.loc[segment, 'std_last_50000'] = x[-50000:].std()\n    X_tr.loc[segment, 'std_first_10000'] = x[:10000].std()\n    X_tr.loc[segment, 'std_last_10000'] = x[-10000:].std()\n    \n    X_tr.loc[segment, 'avg_first_50000'] = x[:50000].mean()\n    X_tr.loc[segment, 'avg_last_50000'] = x[-50000:].mean()\n    X_tr.loc[segment, 'avg_first_10000'] = x[:10000].mean()\n    X_tr.loc[segment, 'avg_last_10000'] = x[-10000:].mean()\n    \n    X_tr.loc[segment, 'min_first_50000'] = x[:50000].min()\n    X_tr.loc[segment, 'min_last_50000'] = x[-50000:].min()\n    X_tr.loc[segment, 'min_first_10000'] = x[:10000].min()\n    X_tr.loc[segment, 'min_last_10000'] = x[-10000:].min()\n    \n    X_tr.loc[segment, 'max_first_50000'] = x[:50000].max()\n    X_tr.loc[segment, 'max_last_50000'] = x[-50000:].max()\n    X_tr.loc[segment, 'max_first_10000'] = x[:10000].max()\n    X_tr.loc[segment, 'max_last_10000'] = x[-10000:].max()\n    \n    X_tr.loc[segment, 'max_to_min'] = x.max() / np.abs(x.min())\n    X_tr.loc[segment, 'max_to_min_diff'] = x.max() - np.abs(x.min())\n    X_tr.loc[segment, 'count_big'] = len(x[np.abs(x) > 500])\n    X_tr.loc[segment, 'sum'] = x.sum()\n    \n    X_tr.loc[segment, 'mean_change_rate_first_50000'] = np.mean(np.nonzero((np.diff(x[:50000]) / x[:50000][:-1]))[0])\n    X_tr.loc[segment, 'mean_change_rate_last_50000'] = np.mean(np.nonzero((np.diff(x[-50000:]) / x[-50000:][:-1]))[0])\n    X_tr.loc[segment, 'mean_change_rate_first_10000'] = np.mean(np.nonzero((np.diff(x[:10000]) / x[:10000][:-1]))[0])\n    X_tr.loc[segment, 'mean_change_rate_last_10000'] = np.mean(np.nonzero((np.diff(x[-10000:]) / x[-10000:][:-1]))[0])\n    \n     #new:'q70','q75','q60','q65'\n    X_tr.loc[segment, 'q70'] = np.quantile(x, 0.70)    \n    X_tr.loc[segment, 'q75'] = np.quantile(x, 0.75)   \n    X_tr.loc[segment, 'q60'] = np.quantile(x, 0.60)    \n    X_tr.loc[segment, 'q65'] = np.quantile(x, 0.65)    \n    X_tr.loc[segment, 'q85'] = np.quantile(x, 0.85)\n    X_tr.loc[segment, 'q90'] = np.quantile(x, 0.90)\n    X_tr.loc[segment, 'q80'] = np.quantile(x, 0.80)\n    X_tr.loc[segment, 'q95'] = np.quantile(x, 0.95)\n    X_tr.loc[segment, 'q99'] = np.quantile(x, 0.99)\n\n\n    X_tr.loc[segment, 'Hilbert_mean'] = np.abs(hilbert(x)).mean()\n    X_tr.loc[segment, 'Hann_window_mean'] = (convolve(x, hann(150), mode='same') / sum(hann(150))).mean()\n    X_tr.loc[segment, 'classic_sta_lta1_mean'] = classic_sta_lta(x, 500, 10000).mean()\n    X_tr.loc[segment, 'classic_sta_lta2_mean'] = classic_sta_lta(x, 5000, 100000).mean()\n    X_tr.loc[segment, 'classic_sta_lta3_mean'] = classic_sta_lta(x, 3333, 6666).mean()\n    X_tr.loc[segment, 'classic_sta_lta4_mean'] = classic_sta_lta(x, 10000, 25000).mean()\n    X_tr.loc[segment, 'Moving_average_700_mean'] = x.rolling(window=700).mean().mean(skipna=True)\n    X_tr.loc[segment, 'Moving_average_1500_mean'] = x.rolling(window=1500).mean().mean(skipna=True)\n    X_tr.loc[segment, 'Moving_average_3000_mean'] = x.rolling(window=3000).mean().mean(skipna=True)\n    X_tr.loc[segment, 'Moving_average_6000_mean'] = x.rolling(window=6000).mean().mean(skipna=True)\n    ewma = pd.Series.ewm\n    X_tr.loc[segment, 'exp_Moving_average_300_mean'] = (ewma(x, span=300).mean()).mean(skipna=True)\n    X_tr.loc[segment, 'exp_Moving_average_3000_mean'] = ewma(x, span=3000).mean().mean(skipna=True)\n    X_tr.loc[segment, 'exp_Moving_average_30000_mean'] = ewma(x, span=6000).mean().mean(skipna=True)\n    no_of_std = 2\n    X_tr.loc[segment, 'MA_700MA_std_mean'] = x.rolling(window=700).std().mean()\n    X_tr.loc[segment,'MA_700MA_BB_high_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] + no_of_std * X_tr.loc[segment, 'MA_700MA_std_mean']).mean()\n    X_tr.loc[segment,'MA_700MA_BB_low_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] - no_of_std * X_tr.loc[segment, 'MA_700MA_std_mean']).mean()\n    X_tr.loc[segment, 'MA_400MA_std_mean'] = x.rolling(window=400).std().mean()\n    X_tr.loc[segment,'MA_400MA_BB_high_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] + no_of_std * X_tr.loc[segment, 'MA_400MA_std_mean']).mean()\n    X_tr.loc[segment,'MA_400MA_BB_low_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] - no_of_std * X_tr.loc[segment, 'MA_400MA_std_mean']).mean()\n    X_tr.loc[segment, 'MA_1000MA_std_mean'] = x.rolling(window=1000).std().mean()\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e7b78771c4ffdb99370e0ebca28df6c509ee51c"},"cell_type":"markdown","source":"We took 150 000 samples as in test set, now lets take any 150 000 and append to the sequential 150 000 samples..."},{"metadata":{"trusted":true,"_uuid":"fc6ce5ac049bc75e88436b81b2527ab537d1ebaa"},"cell_type":"code","source":"fsegments = 10000\n\nX_tr1 = pd.DataFrame(index=range(segments), dtype=np.float64,\n                       columns=['mean', 'std', 'max', 'min',\n                               'mean_change_abs', 'mean_change_rate', 'abs_max', 'abs_min',\n                               'std_first_50000', 'std_last_50000', 'std_first_10000', 'std_last_10000',\n                               'avg_first_50000', 'avg_last_50000', 'avg_first_10000', 'avg_last_10000',\n                               'min_first_50000', 'min_last_50000', 'min_first_10000', 'min_last_10000',\n                               'max_first_50000', 'max_last_50000', 'max_first_10000', 'max_last_10000',\n                               'max_to_min', 'max_to_min_diff', 'count_big', 'sum',\n                               'mean_change_rate_first_50000', 'mean_change_rate_last_50000', 'mean_change_rate_first_10000', 'mean_change_rate_last_10000','q70','q75','q60','q65','q85',\"q90\",'q80',\n                               'q95','q99','Hilbert_mean','Hann_window_mean','classic_sta_lta1_mean','classic_sta_lta2_mean','classic_sta_lta3_mean','classic_sta_lta4_mean','Moving_average_700_mean','Moving_average_1500_mean','Moving_average_3000_mean','Moving_average_6000_mean','exp_Moving_average_300_mean','exp_Moving_average_3000_mean','exp_Moving_average_30000_mean','MA_700MA_std_mean','MA_700MA_BB_high_mean','MA_700MA_BB_low_mean','MA_400MA_std_mean','MA_400MA_BB_high_mean','MA_400MA_BB_low_mean','MA_1000MA_std_mean'])\ny_tr1 = pd.DataFrame(index=range(segments), dtype=np.float64,\n                       columns=['time_to_failure'])\n\ntotal_mean = train['acoustic_data'].mean()\ntotal_std = train['acoustic_data'].std()\ntotal_max = train['acoustic_data'].max()\ntotal_min = train['acoustic_data'].min()\ntotal_sum = train['acoustic_data'].sum()\ntotal_abs_max = np.abs(train['acoustic_data']).sum()\n\nfor segment in tqdm_notebook(range(segments)):\n    ind = np.random.randint(0, train.shape[0]-150001)\n    seg = train.iloc[ind:ind+rows]\n    x = pd.Series(seg['acoustic_data'].values)\n    y = seg['time_to_failure'].values[-1]\n\n   \n    y_tr1.loc[segment, 'time_to_failure'] = y\n\n    X_tr1.loc[segment, 'mean'] = x.mean()\n    X_tr1.loc[segment, 'std'] = x.std()\n    X_tr1.loc[segment, 'max'] = x.max()\n    X_tr1.loc[segment, 'min'] = x.min()\n    \n    \n    X_tr1.loc[segment, 'mean_change_abs'] = np.mean(np.diff(x))\n    X_tr1.loc[segment, 'mean_change_rate'] = np.mean(np.nonzero((np.diff(x) / x[:-1]))[0])\n    X_tr1.loc[segment, 'abs_max'] = np.abs(x).max()\n    X_tr1.loc[segment, 'abs_min'] = np.abs(x).min()\n    \n    X_tr1.loc[segment, 'std_first_50000'] = x[:50000].std()\n    X_tr1.loc[segment, 'std_last_50000'] = x[-50000:].std()\n    X_tr1.loc[segment, 'std_first_10000'] = x[:10000].std()\n    X_tr1.loc[segment, 'std_last_10000'] = x[-10000:].std()\n    \n    X_tr1.loc[segment, 'avg_first_50000'] = x[:50000].mean()\n    X_tr1.loc[segment, 'avg_last_50000'] = x[-50000:].mean()\n    X_tr1.loc[segment, 'avg_first_10000'] = x[:10000].mean()\n    X_tr1.loc[segment, 'avg_last_10000'] = x[-10000:].mean()\n    \n    X_tr1.loc[segment, 'min_first_50000'] = x[:50000].min()\n    X_tr1.loc[segment, 'min_last_50000'] = x[-50000:].min()\n    X_tr1.loc[segment, 'min_first_10000'] = x[:10000].min()\n    X_tr1.loc[segment, 'min_last_10000'] = x[-10000:].min()\n    \n    X_tr1.loc[segment, 'max_first_50000'] = x[:50000].max()\n    X_tr1.loc[segment, 'max_last_50000'] = x[-50000:].max()\n    X_tr1.loc[segment, 'max_first_10000'] = x[:10000].max()\n    X_tr1.loc[segment, 'max_last_10000'] = x[-10000:].max()\n    \n    X_tr1.loc[segment, 'max_to_min'] = x.max() / np.abs(x.min())\n    X_tr1.loc[segment, 'max_to_min_diff'] = x.max() - np.abs(x.min())\n    X_tr1.loc[segment, 'count_big'] = len(x[np.abs(x) > 500])\n    X_tr1.loc[segment, 'sum'] = x.sum()\n    \n    X_tr1.loc[segment, 'mean_change_rate_first_50000'] = np.mean(np.nonzero((np.diff(x[:50000]) / x[:50000][:-1]))[0])\n    X_tr1.loc[segment, 'mean_change_rate_last_50000'] = np.mean(np.nonzero((np.diff(x[-50000:]) / x[-50000:][:-1]))[0])\n    X_tr1.loc[segment, 'mean_change_rate_first_10000'] = np.mean(np.nonzero((np.diff(x[:10000]) / x[:10000][:-1]))[0])\n    X_tr1.loc[segment, 'mean_change_rate_last_10000'] = np.mean(np.nonzero((np.diff(x[-10000:]) / x[-10000:][:-1]))[0])\n   \n    \n    #new:\n    \n    X_tr1.loc[segment, 'q70'] = np.quantile(x, 0.70)    \n    X_tr1.loc[segment, 'q75'] = np.quantile(x, 0.75)   \n    X_tr1.loc[segment, 'q60'] = np.quantile(x, 0.60)    \n    X_tr1.loc[segment, 'q65'] = np.quantile(x, 0.65) \n    X_tr1.loc[segment, 'q85'] = np.quantile(x, 0.85)\n    X_tr1.loc[segment, 'q90'] = np.quantile(x, 0.90)\n    X_tr1.loc[segment, 'q80'] = np.quantile(x, 0.80)\n    X_tr1.loc[segment, 'q95'] = np.quantile(x, 0.95)\n    X_tr1.loc[segment, 'q99'] = np.quantile(x, 0.99)\n\n\n    #new:\n    \n\n    X_tr1.loc[segment, 'Hilbert_mean'] = np.abs(hilbert(x)).mean()\n    X_tr1.loc[segment, 'Hann_window_mean'] = (convolve(x, hann(150), mode='same') / sum(hann(150))).mean()\n    X_tr1.loc[segment, 'classic_sta_lta1_mean'] = classic_sta_lta(x, 500, 10000).mean()\n    X_tr1.loc[segment, 'classic_sta_lta2_mean'] = classic_sta_lta(x, 5000, 100000).mean()\n    X_tr1.loc[segment, 'classic_sta_lta3_mean'] = classic_sta_lta(x, 3333, 6666).mean()\n    X_tr1.loc[segment, 'classic_sta_lta4_mean'] = classic_sta_lta(x, 10000, 25000).mean()\n    X_tr1.loc[segment, 'Moving_average_700_mean'] = x.rolling(window=700).mean().mean(skipna=True)\n    X_tr1.loc[segment, 'Moving_average_1500_mean'] = x.rolling(window=1500).mean().mean(skipna=True)\n    X_tr1.loc[segment, 'Moving_average_3000_mean'] = x.rolling(window=3000).mean().mean(skipna=True)\n    X_tr1.loc[segment, 'Moving_average_6000_mean'] = x.rolling(window=6000).mean().mean(skipna=True)\n    ewma = pd.Series.ewm\n    X_tr1.loc[segment, 'exp_Moving_average_300_mean'] = (ewma(x, span=300).mean()).mean(skipna=True)\n    X_tr1.loc[segment, 'exp_Moving_average_3000_mean'] = ewma(x, span=3000).mean().mean(skipna=True)\n    X_tr1.loc[segment, 'exp_Moving_average_30000_mean'] = ewma(x, span=6000).mean().mean(skipna=True)\n    no_of_std = 2 \n    X_tr1.loc[segment, 'MA_700MA_std_mean'] = x.rolling(window=700).std().mean()\n    X_tr1.loc[segment,'MA_700MA_BB_high_mean'] = (X_tr1.loc[segment, 'Moving_average_700_mean'] + no_of_std * X_tr1.loc[segment, 'MA_700MA_std_mean']).mean()\n    X_tr1.loc[segment,'MA_700MA_BB_low_mean'] = (X_tr1.loc[segment, 'Moving_average_700_mean'] - no_of_std * X_tr1.loc[segment, 'MA_700MA_std_mean']).mean()\n    X_tr1.loc[segment, 'MA_400MA_std_mean'] = x.rolling(window=400).std().mean()\n    X_tr1.loc[segment,'MA_400MA_BB_high_mean'] = (X_tr1.loc[segment, 'Moving_average_700_mean'] + no_of_std * X_tr1.loc[segment, 'MA_400MA_std_mean']).mean()\n    X_tr1.loc[segment,'MA_400MA_BB_low_mean'] = (X_tr1.loc[segment, 'Moving_average_700_mean'] - no_of_std * X_tr1.loc[segment, 'MA_400MA_std_mean']).mean()\n    X_tr1.loc[segment, 'MA_1000MA_std_mean'] = x.rolling(window=1000).std().mean()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"780eaa1f66eeca597acc3868ed88683a74efbe2d"},"cell_type":"code","source":"X_tr.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8543c819a94fd24d6b026c68c319fef78f455c6c","scrolled":true},"cell_type":"code","source":"X_tr = X_tr.append(X_tr1)\ny_tr = y_tr.append(y_tr1)\nprint(f'{X_tr.shape[0]} samples in new train data now.')\ndel train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e15fecede3c9fb95067d6d6e92a76015fc829026"},"cell_type":"code","source":"scaler = StandardScaler()\nscaler.fit(X_tr)\nX_train_scaled = pd.DataFrame(scaler.transform(X_tr), columns=X_tr.columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac09381ed8a411e691e883e55e5282600ee90ef3"},"cell_type":"markdown","source":"Test data set:"},{"metadata":{"trusted":true,"_uuid":"0cd281e33490b850cae447442a2d628ff95fff42","scrolled":false},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\nX_test = pd.DataFrame(columns=X_tr.columns, dtype=np.float64, index=submission.index)\nplt.figure(figsize=(22, 16))\n\nfor i, seg_id in enumerate(tqdm_notebook(X_test.index)):\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    \n    x = pd.Series(seg['acoustic_data'].values)\n    X_test.loc[seg_id, 'mean'] = x.mean()\n    X_test.loc[seg_id, 'std'] = x.std()\n    X_test.loc[seg_id, 'max'] = x.max()\n    X_test.loc[seg_id, 'min'] = x.min()\n        \n    X_test.loc[seg_id, 'mean_change_abs'] = np.mean(np.diff(x))\n    X_test.loc[seg_id, 'mean_change_rate'] = np.mean(np.nonzero((np.diff(x) / x[:-1]))[0])\n    X_test.loc[seg_id, 'abs_max'] = np.abs(x).max()\n    X_test.loc[seg_id, 'abs_min'] = np.abs(x).min()\n    \n    X_test.loc[seg_id, 'std_first_50000'] = x[:50000].std()\n    X_test.loc[seg_id, 'std_last_50000'] = x[-50000:].std()\n    X_test.loc[seg_id, 'std_first_10000'] = x[:10000].std()\n    X_test.loc[seg_id, 'std_last_10000'] = x[-10000:].std()\n    \n    X_test.loc[seg_id, 'avg_first_50000'] = x[:50000].mean()\n    X_test.loc[seg_id, 'avg_last_50000'] = x[-50000:].mean()\n    X_test.loc[seg_id, 'avg_first_10000'] = x[:10000].mean()\n    X_test.loc[seg_id, 'avg_last_10000'] = x[-10000:].mean()\n    \n    X_test.loc[seg_id, 'min_first_50000'] = x[:50000].min()\n    X_test.loc[seg_id, 'min_last_50000'] = x[-50000:].min()\n    X_test.loc[seg_id, 'min_first_10000'] = x[:10000].min()\n    X_test.loc[seg_id, 'min_last_10000'] = x[-10000:].min()\n    \n    X_test.loc[seg_id, 'max_first_50000'] = x[:50000].max()\n    X_test.loc[seg_id, 'max_last_50000'] = x[-50000:].max()\n    X_test.loc[seg_id, 'max_first_10000'] = x[:10000].max()\n    X_test.loc[seg_id, 'max_last_10000'] = x[-10000:].max()\n    \n    X_test.loc[seg_id, 'max_to_min'] = x.max() / np.abs(x.min())\n    X_test.loc[seg_id, 'max_to_min_diff'] = x.max() - np.abs(x.min())\n    X_test.loc[seg_id, 'count_big'] = len(x[np.abs(x) > 500])\n    X_test.loc[seg_id, 'sum'] = x.sum()\n    \n    X_test.loc[seg_id, 'mean_change_rate_first_50000'] = np.mean(np.nonzero((np.diff(x[:50000]) / x[:50000][:-1]))[0])\n    X_test.loc[seg_id, 'mean_change_rate_last_50000'] = np.mean(np.nonzero((np.diff(x[-50000:]) / x[-50000:][:-1]))[0])\n    X_test.loc[seg_id, 'mean_change_rate_first_10000'] = np.mean(np.nonzero((np.diff(x[:10000]) / x[:10000][:-1]))[0])\n    X_test.loc[seg_id, 'mean_change_rate_last_10000'] = np.mean(np.nonzero((np.diff(x[-10000:]) / x[-10000:][:-1]))[0])\n   \n\n\n    #new\n    \n    \n    X_test.loc[seg_id, 'q70'] = np.quantile(x, 0.70)    \n    X_test.loc[seg_id, 'q75'] = np.quantile(x, 0.75)   \n    X_test.loc[seg_id, 'q60'] = np.quantile(x, 0.60)    \n    X_test.loc[seg_id, 'q65'] = np.quantile(x, 0.65) \n    X_test.loc[seg_id, 'q85'] = np.quantile(x, 0.85)\n    X_test.loc[seg_id, 'q90'] = np.quantile(x, 0.90)\n    X_test.loc[seg_id, 'q80'] = np.quantile(x, 0.80)\n    X_test.loc[seg_id, 'q95'] = np.quantile(x,0.95)\n    X_test.loc[seg_id, 'q99'] = np.quantile(x,0.99)\n    \n\n    \n    X_test.loc[seg_id, 'Hilbert_mean'] = np.abs(hilbert(x)).mean()\n    X_test.loc[seg_id, 'Hann_window_mean'] = (convolve(x, hann(150), mode='same') / sum(hann(150))).mean()\n    X_test.loc[seg_id, 'classic_sta_lta1_mean'] = classic_sta_lta(x, 500, 10000).mean()\n    X_test.loc[seg_id, 'classic_sta_lta2_mean'] = classic_sta_lta(x, 5000, 100000).mean()\n    X_test.loc[seg_id, 'classic_sta_lta3_mean'] = classic_sta_lta(x, 3333, 6666).mean()\n    X_test.loc[seg_id, 'classic_sta_lta4_mean'] = classic_sta_lta(x, 10000, 25000).mean()\n    X_test.loc[seg_id, 'Moving_average_700_mean'] = x.rolling(window=700).mean().mean(skipna=True)\n    X_test.loc[seg_id, 'Moving_average_1500_mean'] = x.rolling(window=1500).mean().mean(skipna=True)\n    X_test.loc[seg_id, 'Moving_average_3000_mean'] = x.rolling(window=3000).mean().mean(skipna=True)\n    X_test.loc[seg_id, 'Moving_average_6000_mean'] = x.rolling(window=6000).mean().mean(skipna=True)\n    ewma = pd.Series.ewm\n    X_test.loc[seg_id, 'exp_Moving_average_300_mean'] = (ewma(x, span=300).mean()).mean(skipna=True)\n    X_test.loc[seg_id, 'exp_Moving_average_3000_mean'] = ewma(x, span=3000).mean().mean(skipna=True)\n    X_test.loc[seg_id, 'exp_Moving_average_30000_mean'] = ewma(x, span=6000).mean().mean(skipna=True)\n    no_of_std = 2\n    X_test.loc[seg_id, 'MA_700MA_std_mean'] = x.rolling(window=700).std().mean()\n    X_test.loc[seg_id,'MA_700MA_BB_high_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] + no_of_std * X_test.loc[seg_id, 'MA_700MA_std_mean']).mean()\n    X_test.loc[seg_id,'MA_700MA_BB_low_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] - no_of_std * X_test.loc[seg_id, 'MA_700MA_std_mean']).mean()\n    X_test.loc[seg_id, 'MA_400MA_std_mean'] = x.rolling(window=400).std().mean()\n    X_test.loc[seg_id,'MA_400MA_BB_high_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] + no_of_std * X_test.loc[seg_id, 'MA_400MA_std_mean']).mean()\n    X_test.loc[seg_id,'MA_400MA_BB_low_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] - no_of_std * X_test.loc[seg_id, 'MA_400MA_std_mean']).mean()\n    X_test.loc[seg_id, 'MA_1000MA_std_mean'] = x.rolling(window=1000).std().mean()\n   \n    X_test_scaled = pd.DataFrame(scaler.transform(X_test), columns=X_tr1.columns)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8d7131972c91333ddb06464b93e87e809d0809c"},"cell_type":"markdown","source":"# NOTE\nI know what are you thinking, how much sense does it make to take a mean of a moving average and other derivations of it but according to the feature importance of lbgm it does. In a contrast if we were to take all 150 000 rows and calculate what we have calculated for each of them. Than (approximately) 9 minutes times 150 000 is around\n# **ONE LIFETIME**\n worth of time of calculations. Now I do understand that it is actually not that since we are taking the mean for every one ofthe variables etc.. but STILL it is a long time!"},{"metadata":{"trusted":true,"_uuid":"6425d48ce0e5618b99b8690017479de2f5c3f187"},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n# **Model**\n**Model** xbgoost, lgbm and NuSVR with some parameter tweeking"},{"metadata":{"trusted":true,"_uuid":"a05b60941b28cbc0b22a2938d382957aeb5e8d06"},"cell_type":"code","source":"n_fold = 5\nfolds = KFold(n_splits=n_fold, shuffle=True, random_state=11)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a72da0253de26132214abde1231fa14970b22682"},"cell_type":"code","source":"def train_model(X=X_train_scaled, X_test=X_test_scaled, y=y_tr, params=None, folds=folds, model_type='lgb', plot_feature_importance=False, model=None):\n\n    oof = np.zeros(len(X))\n    prediction = np.zeros(len(X_test))\n    scores = []\n    feature_importance = pd.DataFrame()\n    for fold_n, (train_index, valid_index) in enumerate(folds.split(X)):\n        print('Fold', fold_n, 'started at', time.ctime())\n        X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]\n        y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n        \n        if model_type == 'lgb':\n            model = lgb.LGBMRegressor(**params, n_estimators = 50000, n_jobs = -1)\n            model.fit(X_train, y_train, \n                    eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric='mae',\n                    verbose=10000, early_stopping_rounds=200)\n            \n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test, num_iteration=model.best_iteration_)\n            \n        if model_type == 'xgb':\n            train_data = xgb.DMatrix(data=X_train, label=y_train, feature_names=X_tr.columns)\n            valid_data = xgb.DMatrix(data=X_valid, label=y_valid, feature_names=X_tr.columns)\n\n            watchlist = [(train_data, 'train'), (valid_data, 'valid_data')]\n            model = xgb.train(dtrain=train_data, num_boost_round=20000, evals=watchlist, early_stopping_rounds=200, verbose_eval=500, params=params)\n            y_pred_valid = model.predict(xgb.DMatrix(X_valid, feature_names=X_tr.columns), ntree_limit=model.best_ntree_limit)\n            y_pred = model.predict(xgb.DMatrix(X_test, feature_names=X_tr.columns), ntree_limit=model.best_ntree_limit)\n            \n        if model_type == 'rcv':\n            model = RidgeCV(alphas=(0.01, 0.1, 1.0, 10.0, 100.0, 1000.0), scoring='neg_mean_absolute_error', cv=5)\n            model.fit(X_train, y_train)\n            print(model.alpha_)\n\n            y_pred_valid = model.predict(X_valid).reshape(-1,)\n            score = mean_absolute_error(y_valid, y_pred_valid)\n            print(f'Fold {fold_n}. MAE: {score:.4f}.')\n            print('')\n            \n            y_pred = model.predict(X_test).reshape(-1,)\n        \n        if model_type == 'sklearn':\n            model = model\n            model.fit(X_train, y_train)\n            \n            y_pred_valid = model.predict(X_valid).reshape(-1,)\n            score = mean_absolute_error(y_valid, y_pred_valid)\n            print(f'Fold {fold_n}. MAE: {score:.4f}.')\n            print('')\n            \n            y_pred = model.predict(X_test).reshape(-1,)\n        \n        if model_type == 'cat':\n            model = CatBoostRegressor(iterations=20000,  eval_metric='MAE', **params)\n            model.fit(X_train, y_train, eval_set=(X_valid, y_valid), cat_features=[], use_best_model=True, verbose=False)\n\n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test)\n        \n        oof[valid_index] = y_pred_valid.reshape(-1,)\n        scores.append(mean_absolute_error(y_valid, y_pred_valid))\n\n        prediction += y_pred    \n        \n        if model_type == 'lgb':\n            # feature importance\n            fold_importance = pd.DataFrame()\n            fold_importance[\"feature\"] = X.columns\n            fold_importance[\"importance\"] = model.feature_importances_\n            fold_importance[\"fold\"] = fold_n + 1\n            feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n\n    prediction /= n_fold\n    \n    print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))\n    \n    if model_type == 'lgb':\n        feature_importance[\"importance\"] /= n_fold\n        if plot_feature_importance:\n            cols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n                by=\"importance\", ascending=False)[:50].index\n\n            best_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\n            plt.figure(figsize=(16, 12));\n            sns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\n            plt.title('LGB Features (avg over folds)');\n        \n            return oof, prediction, feature_importance\n        return oof, prediction\n    \n    else:\n        return oof, prediction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c3f801fc49c69b4591cce71f6cc96db3ea2242f"},"cell_type":"code","source":"from bayes_opt import BayesianOptimization","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"354b4aa0c90257deee21be9b07d5dd962b95572e"},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n#  Bayesian Hyperparameter Optimisation:\nRead up on it [tutorial](https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f). \n\n\nTo search deeper, (it does have a limit tough!) change **init_points, n_iter values**\n\n\nAlso reader can change the interval of values if one has some better conviction of where the true values lie.\n\nFinally one may also want to fixate some parameter values, and search for select-subgroup. Time does grow exponentially with the number of parameters to be searched. For the ommited values default will be taken!"},{"metadata":{"trusted":true,"_uuid":"0c69b45b0ce61f0385227aa9512db7453c8a15fb"},"cell_type":"code","source":"X = X_train_scaled\ny = y_tr\ntrain_data = lgb.Dataset(data=X, label=y, free_raw_data=False)\ndef lgb_eval(num_leaves, feature_fraction, max_depth , min_split_gain, min_child_weight,bagging_freq,reg_alpha,reg_lambda):\n        params = {\n            \"objective\" : \"regression\", \"bagging_fraction\" : 0.8,\n            \"min_child_samples\": 20, \"boosting\": \"gbdt\",\n            \"learning_rate\" : 0.01, \"subsample\" : 0.8, \"colsample_bytree\" : 0.8, \"verbosity\": -1, \"metric\" : 'mae'\n        }\n        params[\"bagging_freq\"] = int(round(bagging_freq))\n        params[\"reg_alpha\"] = reg_alpha\n        params[\"reg_lambda\"] = reg_lambda\n        params['feature_fraction'] = max(min(feature_fraction, 1), 0)\n        params['max_depth'] = int(round(max_depth))\n        params['num_leaves'] = int(round(num_leaves))\n        params['min_split_gain'] = min_split_gain\n        params['min_child_weight'] = min_child_weight\n        cv_result = lgb.cv(params, train_data, nfold=5, seed=123, verbose_eval =200,stratified=False)\n        return (-1.0 * np.array(cv_result['l1-mean'])).max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca182669dd9a56530ec98b1b66d6554f3d89ffdc"},"cell_type":"code","source":"lgbBO = BayesianOptimization(lgb_eval, {'num_leaves': (24, 80),\n                                        'feature_fraction': (0.1, 1),\n                                        'max_depth': (2, 30),\n                                        'min_split_gain': (0.001, 1),\n                                        'min_child_weight': (1, 30),\n                                        \"reg_alpha\": (0,3),\n                                        \"reg_lambda\":(0,3),\n                                        \"bagging_freq\": (1,10)}\n                            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"bbf5c3ec73bac5b5e5505b914f0881b08440e280"},"cell_type":"code","source":"lgbBO.maximize(init_points=5, n_iter=15,acq='ei')\n# Use the expected improvement acquisition function to handle negative numbers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b204aeede4adfb1acaee6d42fb52650f00ef1650"},"cell_type":"code","source":"lgb_params = {'num_leaves': 80,\n              'min_child_weight': 28,\n              'min_split_gain': 0.745,\n          'min_data_in_leaf': 79,\n          'objective': 'huber',\n          'max_depth': 25,\n          'learning_rate': 0.01,\n          \"boosting\": \"gbdt\",\n          \"bagging_freq\": 4,\n          \"bagging_fraction\": 0.8126672064208567,\n          \"bagging_seed\": 11,\n          \"metric\": 'mae',\n          \"verbosity\": -1,\n          'reg_alpha': 0.1058,\n          'reg_lambda': 0.2209,\n          'feature_fraction': 0.9201\n         }\noof_lgb, prediction_lgb, feature_importance = train_model(params=lgb_params, model_type='lgb', plot_feature_importance=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88be23548282c1084ba5739e9a02f77a05d8182f"},"cell_type":"markdown","source":"**XGBoost**"},{"metadata":{"trusted":true,"_uuid":"3ad095f13ded40ba5e2254da562fadff23b371f4"},"cell_type":"code","source":"dtrain = xgb.DMatrix(X, label=y)\ndef xgb_evaluate(max_depth, gamma, colsample_bytree,learning_rate,reg_alpha,reg_lambda,min_child_weight):\n    params = {'eval_metric': 'mae',\n              'max_depth': int(round(max_depth)),\n              'subsample': 0.8,\n              'eta': 0.1,\n              'gamma': gamma,\n              'colsample_bytree': colsample_bytree,\n              \"silent\":1,\n              \"learning_rate\":learning_rate,\n              \"reg_alpha\":reg_alpha,\n              \"reg_lambda\":reg_lambda,\n              \"min_child_weight\":min_child_weight\n              \n             }\n\n    cv_result = xgb.cv(params, dtrain, num_boost_round=1000, nfold=3)    \n\n    return (-1.0 * np.array(cv_result['test-mae-mean'])).max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"731f6721762b806323fa6825a986ebfba3508f41","scrolled":true},"cell_type":"code","source":"xgb_bo = BayesianOptimization(xgb_evaluate, {'max_depth': (3, 30), \n                                             'gamma': (0, 1),\n                                             'colsample_bytree': (0.3, 1),\n                                             \"learning_rate\": (0.0, 1.0),\n                                             \"reg_alpha\": (1.0, 10.0),\n                                             \"reg_lambda\":(1.0, 10.0),\n                                             \"min_child_weight\":(0, 10)\n                                            })\n# Use the expected improvement acquisition function to handle negative numbers\nxgb_bo.maximize(init_points=5, n_iter=15, acq='ei')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"303d32d16863f348bf0d9e34525ee890599f22c0"},"cell_type":"code","source":"xgb_params = {'eta': 0.05,\n              'gamma': 0.5913,\n              'colsample_bytree': 0.9692,\n              \"learning_rate\": 0.04425,\n              \"reg_alpha\":  1.226,\n              \"reg_lambda\": 4.834,\n              \"min_child_weight\": 5,\n              'max_depth': 27,\n              'subsample': 0.9,\n              'objective': 'reg:linear',\n              'eval_metric': 'mae',\n              'silent': True,\n              'nthread': 4}\noof_xgb, prediction_xgb = train_model(params=xgb_params, model_type='xgb')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ab5dc148530603269b8766a4fca884e2759d04e7"},"cell_type":"markdown","source":"Submission"},{"metadata":{"trusted":true,"_uuid":"b7bd96cbf6ab13122c894d658c02c3f660de78ae"},"cell_type":"code","source":"submission['time_to_failure'] = (prediction_lgb + prediction_xgb) / 2\nprint(submission.head())\nsubmission.to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6fb83a9d664b4f69250ed44ad104e854f9f24af"},"cell_type":"markdown","source":"Based on [artgor](https://www.kaggle.com/artgor/earthquakes-fe-more-features-and-samples)\n\n\n**Support appreciated :)**"},{"metadata":{"trusted":true,"_uuid":"313b9137d0c1fb8c76df520e949b597b85c1a4af"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6441551fa3c6ddde6d0f5cb90d28db8fd707c88"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85d5b6ff90cc7422deb77e1216158df59a109984"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6567033b6cdca8c125b3577cdbd1ceefa18c7da3"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd68d00e849d3a55c2207428e35134f1fe7f86e6"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a362e00567426059e9cb4a0d23cdd35d2cbf1bdd"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}