{"cells":[{"metadata":{"_uuid":"a7de248e7fed02ea935b19a6ab3a4ccd410a123f"},"cell_type":"markdown","source":"# <span style=\"color:red;\">Machine learning-detected signal predicts time to earthquake</span>"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d604d766f395f08e394b6637fde3a62b892a37ad"},"cell_type":"code","source":"%%html\n<h1><b>LIVE EARTHQUACK MAP</b></h1>\n<iframe width=\"800\" height=\"600\" src=\"https://ds.iris.edu/seismon/\" allowfullscreen style=\"align:center;\"></iframe>","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd8e82d3a54c20a6f94a963ccfd6bca3e2e9596d"},"cell_type":"markdown","source":"<span style=\"color:blue;\"><strong>ABOUT COMPETITION</strong></span>\n-------------------------------------\n<div class=\"competition-overview__content\"><div><div class=\"markdown-converter__text--rendered\"><img src=\"https://storage.googleapis.com/kaggle-media/competitions/LANL/nik-shuliahin-585307-unsplash.jpg\" alt=\"map\" width=\"300\" style=\"float:right;\" class=\"hoverZoomLink\">\n> <p>Forecasting earthquakes is one of the most important problems in Earth science because of their devastating consequences. Current scientific studies related to earthquake forecasting focus on three key points: <b>when</b> the event will occur, <b>where</b> it will occur, and <b>how large</b> it will be.</p>  \n\n<span style=\"color:blue;\"><strong>WHAT THEY WANT:</strong></span>\n> <p>In this competition, you will address <b>when</b> the earthquake will take place. Specifically, you’ll predict the time remaining before laboratory earthquakes occur from real-time seismic data. </p>\n\n<span style=\"color:blue;\"><strong>CHALLANGE:</strong></span>\n> <p>If this challenge is solved and the **physics are ultimately shown to scale from the laboratory to the field**, researchers will have the potential to **improve earthquake hazard assessments** that could **save lives and billions of dollars in infrastructure.**This challenge is hosted by  <a href=\"https://www.lanl.gov/\" rel=\"nofollow\">Los Alamos National Laboratory</a> which enhances national security by ensuring the safety of the U.S. nuclear stockpile, developing technologies to reduce threats from weapons of mass destruction, and solving problems related to energy, environment, infrastructure, health, and global security concerns.</p>\n\n### <span style=\"color:red;\"><strong>SUBMISSION FORMAT</strong></span>\n> * Submissions are evaluated using the [**mean absolute error**](https://en.wikipedia.org/wiki/Mean_absolute_error) between the predicted time remaining before the next lab earthquake and the act remaining time.\n\n### <span style=\"color:red;\"><strong>Submission File</strong></span>\n\nFor each `seg_id` in the test set folder, you must predict `time_to_failure`, which is the remaining time before the next lab earthquake. The file should contain a header and have the following format:\n\n    seg_id,time_to_failure\n    seg_00030f,0\n    seg_0012b5,0\n    seg_00184e,0\n    ...\n\n\n<span style=\"color:red;\">**GOAL OF COMPETITION**</span>\n-----------------\n\n* ***The goal of this competition is to use seismic signals to predict the timing of laboratory earthquakes.*** The *data comes from a well-known experimental set-up used to study earthquake physics.* The` acoustic_data` input signal is used to **predict the time remaining before the next laboratory earthquake (time_to_failure).**\n* The ***training data** is a **single, continuous segment of experimental data.** The ***test data*** consists of a folder containing many **small segments.** The data within each **test file is continuous, but the test files do not represent a continuous segment of the experiment**; thus, the **predictions cannot be assumed to follow the same regular pattern seen in the training file.**\n* For each `seg_id` in the test folder, you should predict a single `time_to_failure` corresponding to the time between the **last row of the segment and the next laboratory earthquake.**\n\n<span style=\"color:Red;\">**DATA DESCRIPTION**</span>\n----\n### <span style=\"color:blue;\">**File descriptions**: </span>\n* **train.csv** - A single, continuous training segment of experimental data.\n* **test** - A folder containing many small segments of test data.\n* **sample_sumbission.csv** - A sample submission file in the correct format.\n\n### <span style=\"color:blue;\">**Data fields**:</span>\n* **acoustic_data** - the seismic signal [int16]\n* **time_to_failure** - the time (in seconds) until the next laboratory earthquake [float64]\n* **seg_id** - the test segment ids for which predictions should be made (one prediction per segment)"},{"metadata":{"_uuid":"97ad6ded5feb8cb0d25adffb4ec908e677a0c6b8"},"cell_type":"markdown","source":"<strong><span style=\"color:Red;\">*Article By Los Alamos National Laboratory*</span></strong>\n<strong><span style=\"color:black;\">‘Fingerprint’ of fault displacement also forecasts magnitude of rupture</span></strong>\n-----------------------------------------------------------------------\n\nLOS ALAMOS, N.M., Dec. 17, 2018—Machine-learning research published in two related papers today in _Nature Geosciences_ reports the detection of seismic signals accurately predicting the Cascadia fault’s slow slippage, a type of failure observed to precede large earthquakes in other subduction zones.\n\n* **Los Alamos National Laboratory researchers applied machine learning to analyze Cascadia data and discovered the megathrust broadcasts a constant tremor, a fingerprint of the fault’s displacement.** More importantly, they **found a direct parallel between the loudness of the fault’s acoustic signal and its physical changes**. Cascadia’s groans, previously discounted as meaningless noise, foretold its fragility.\n* **“Cascadia’s behavior was buried in the data**. Until **machine learning revealed precise patterns**, we all discarded the **continuous signal as noise, but it was full of rich information.** We discovered a **highly predictable sound pattern that indicates slippage and fault failure**,” said Los Alamos scientist Paul Johnson. “We also found a precise link between the fragility of the fault and the signal’s strength, which can help us more accurately predict a megaquake.”  \n* The **new papers** were authored by ***Johnson, Bertrand Rouet-Leduc and Claudia Hulbert*** from the ***Laboratory’s Earth and Environmental Sciences Division, Christopher Ren from the Laboratory’s Intelligence and Space Research Division and collaborators at Pennsylvania State University.***\n* **Machine learning crunches massive seismic data sets to find distinct patterns by learning from self-adjusting algorithms to create decision trees that select and retest a series of questions and answers.** Last year, the team simulated an earthquake in a laboratory, using steel blocks interacting with rocks and pistons, and recorded sounds that they analyzed by machine learning. They discovered that the numerous seismic signals, previously discounted as meaningless noise, pinpointed when the simulated fault would slip, a major advance towards earthquake prediction. Faster, more powerful quakes had louder signals.\n* The team decided to apply their new paradigm to the real world: Cascadia. Recent research reveals that Cascadia has been active, but noted activity has been seemingly random. This team analyzed 12 years of real data from seismic stations in the region and found similar signals and results: Cascadia’s constant tremors quantify the displacement of the slowly slipping portion of the subduction zone. In the laboratory, the authors identified a similar signal that accurately predicted a broad range of fault failure. Careful monitoring in Cascadia may provide new information on the locked zone to provide an early warning system.\n\nThe papers:\n-------------------\n\n*   [Similarity of fast and slow earthquakes illuminated by machine learning](https://www.nature.com/articles/s41561-018-0272-8 \"Machine learning\"), Nature Geoscience, Dec. 17, 2018\n*   [Continuous chatter of the Cascadia subduction zone revealed by machine learning,](https://www.nature.com/articles/s41561-018-0274-6) Nature Geoscience, Dec. 17, 2018\n\n**REFERENCES:**  \nhttps://www.lanl.gov/discover/news-release-archive/2018/December/1217-machine-learning.php"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"31a1da835dccc79f74431e31fd215bbe8dca54b1"},"cell_type":"code","source":"%%html\n\n<h1> <strong><span style=\"color:blue;\">Can We Predict Earthquakes?</span></strong></h1>\n<iframe width=\"800\" height=\"400\" src=\"https://www.youtube.com/embed/uUEzGcRJIZE\" style=\"align:center;\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc65ad71d7c6f1e21fdb204519b4bdd7c8f02148"},"cell_type":"markdown","source":"## Outline\n---\n* [**1.Read Data**](#1.Read-Data)\n* [**2.Simple Exploration**](#2.Simple-Exploration)\n* [**3.Feature Engineering**](#3.Feature-Engineering)\n* [**4.Sanity Check**](#4.Sanity-Check)\n* [**5.Data Transformation**](#5.Data-Transformation)\n* [**6.Model Training**](#6.Model-Training)\n\t* [**1.nuSVR**](#1.nuSVR)\n\t* [**2.SVR**](#2.SVR)\n\t* [**3.BayesianRidge**](#3.BayesianRidge)\n\t* [**4.LightGBM Regression**](#4.LightGBM-Regression)\n\t* [**5.CatBoost Regression**](#5.CatBoost-Regression)\n* [**7.Prediction**](#7.Prediction)\n* [**8.Stacking using lightgbm**](#8.Stacking-using-lightgbm)\n* [**9.Total Analysis of all model**](#9.Total-Analysis-of-all-model)\n* [**10.Best Model**](#10.Best-Model)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport gc\nimport os\nprint(os.listdir(\"../input\"))\nfrom mlxtend.regressor import stacking_regression\n# pandas doesn't show us all the decimals\npd.options.display.precision = 15\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nplt.style.use(\"fivethirtyeight\")\nfrom tqdm import tqdm\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.svm import NuSVR,LinearSVR, SVR\nfrom sklearn.metrics import mean_absolute_error,r2_score, make_scorer\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn import ensemble\nfrom sklearn.kernel_ridge import KernelRidge\nfrom sklearn.linear_model import LinearRegression, BayesianRidge\nfrom catboost import CatBoostRegressor\nfrom lightgbm import LGBMRegressor\nfrom sklearn.metrics import accuracy_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a38af4daf48c3073f32bff5bca27852cc26ee76b"},"cell_type":"markdown","source":"## <span style=\"color:blue;\"><strong>1.Read Data</strong></span>"},{"metadata":{"trusted":true,"_uuid":"207e7262626cadcacc7c93c52259324de03ef05d"},"cell_type":"code","source":"%%time\ntrain = pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float64})\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f5cce599c25e20de7a20bfb176e51e9cc3ac661c"},"cell_type":"markdown","source":"## <span style=\"color:blue;\"><strong>2.Simple Exploration</strong></span>"},{"metadata":{"trusted":true,"_uuid":"be37c0794b387b4cdbc635721d36ca3849017303"},"cell_type":"code","source":"plt.figure(figsize=(20, 5))\nplt.plot(train['acoustic_data'].values[::100], color='blue', label='Acoustic Data')\nplt.legend()\nplt.ylabel(\"Acoustic Data value\")\nplt.xlabel(\"Total Value Count\")\nplt.title('Acoustic Data')\nplt.show()\nplt.figure(figsize=(20, 5))\nplt.plot(train['time_to_failure'].values[::100], color='red', label='Time_to_failure')\nplt.legend()\nplt.ylabel(\"Time Data value\")\nplt.xlabel(\"Time Value Count\")\nplt.title('Time to failure')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e2e142d357951c90972995548025c798f0236f0"},"cell_type":"code","source":"def add_trend_feature(arr, abs_values=False):\n    idx = np.array(range(len(arr)))\n    if abs_values:\n        arr = np.abs(arr)\n    lr = LinearRegression()\n    lr.fit(idx.reshape(-1, 1), arr)\n    return lr.coef_[0]\n\ndef rmse(predictions, targets):\n    return np.sqrt(((predictions - targets) ** 2).mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c443fa4802f764310d45a3533b99f771bdb6cba"},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"479d67bb885b3bfabfc7e33b05392f17c6eb961d"},"cell_type":"markdown","source":"## <span style=\"color:blue;\"><strong>3.Feature Engineering</strong></span>"},{"metadata":{"trusted":true,"_uuid":"9c09b65c17af1181582d333179466b9c9b284a9b"},"cell_type":"code","source":"# Create a training file with simple derived features\n# Feature Engineering : https://www.kaggle.com/gpreda/lanl-earthquake-eda-and-prediction?scriptVersionId=9550007\n\nrows = 150_000\nsegments = int(np.floor(train.shape[0] / rows))\n\nX_train = pd.DataFrame(index=range(segments), dtype=np.float64,\n                       columns=['ave', 'std', 'max', 'min','q95','q99', 'q05','q01',\n                                'abs_max', 'abs_mean', 'abs_std', 'trend', 'abs_trend'])\ny_train = pd.DataFrame(index=range(segments), dtype=np.float64,\n                       columns=['time_to_failure'])\n\nfor segment in tqdm(range(segments)):\n    seg = train.iloc[segment*rows:segment*rows+rows]\n    x = seg['acoustic_data'].values\n    y = seg['time_to_failure'].values[-1]\n    \n    y_train.loc[segment, 'time_to_failure'] = y\n    \n    X_train.loc[segment, 'ave'] = x.mean()\n    X_train.loc[segment, 'std'] = x.std()\n    X_train.loc[segment, 'max'] = x.max()\n    X_train.loc[segment, 'min'] = x.min()\n    X_train.loc[segment, 'q95'] = np.quantile(x,0.95)\n    X_train.loc[segment, 'q99'] = np.quantile(x,0.99)\n    X_train.loc[segment, 'q05'] = np.quantile(x,0.05)\n    X_train.loc[segment, 'q01'] = np.quantile(x,0.01)\n    \n    X_train.loc[segment, 'abs_max'] = np.abs(x).max()\n    X_train.loc[segment, 'abs_mean'] = np.abs(x).mean()\n    X_train.loc[segment, 'abs_std'] = np.abs(x).std()\n    X_train.loc[segment, 'trend'] = add_trend_feature(x)\n    X_train.loc[segment, 'abs_trend'] = add_trend_feature(x, abs_values=True)\n    \n    \ndisplay(X_train.head())\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42fbb964faa2e62f96a0411b7403940752631937"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>4.Sanity Check</strong></span>"},{"metadata":{"trusted":true,"_uuid":"dde8362d5d4678adcf245cbdedf851bfa043c56c"},"cell_type":"code","source":"%%time\naxs = pd.scatter_matrix(X_train[::100], figsize=(20,12), diagonal='kde')\ndisplay(X_train[::100].corr())\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9da9d785a0799c37c3d9a0755d6654d9aa224222"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>5.Data Transformation</strong></span>"},{"metadata":{"trusted":true,"_uuid":"180cf35c19457394d3e1e901134d1da27574a46e"},"cell_type":"code","source":"%%time\nscaler = StandardScaler()\nscaler.fit(X_train)\nX_train_scaled = scaler.transform(X_train)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bdbfa4a7e2d48d7719b0cc31019c4c6238113eae"},"cell_type":"code","source":"%%time\naxs = pd.scatter_matrix(X_train[::100], figsize=(20,12), diagonal='kde')\ndisplay(X_train[::100].corr())\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2ba7ad4101163fe429578552c6ea3dac0335f93"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>6.Model Training</strong></span>"},{"metadata":{"_uuid":"3b6f4e2d8fe019cb0c3f7fb3766e5be2f70e296d"},"cell_type":"markdown","source":"## 1.nuSVR"},{"metadata":{"trusted":true,"_uuid":"27b5b7a2aa326d7c8aa6d3cc318d95424b7256ec","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"def nusvr_code(NuSVR,X_train_scaled,y_train):\n    svm1 = NuSVR(nu=0.95, gamma=0.62,C=2.45)\n    svm1.fit(X_train_scaled, y_train.values.flatten())\n    y_pred = svm1.predict(X_train_scaled)\n    plt.figure(figsize=(20, 6))\n    plt.scatter(y_train.values.flatten(), y_pred)\n    plt.xlim(0, 20)\n    plt.ylim(0, 20)\n    plt.xlabel('actual', fontsize=12)\n    plt.ylabel('predicted', fontsize=12)\n    plt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\n    plt.title('NuSVR')\n    plt.show()\n    score = rmse(y_train.values.flatten(), y_pred)\n    print(f'RMSE Score: {score:0.3f}')\n    score = r2_score(y_train.values.flatten(), y_pred)\n    print(f'Score: {score:0.3f}')\n    return (y_pred,score,svm1)\n    \ny_pred_nusvr, score, svm1 = nusvr_code(NuSVR,X_train_scaled,y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0ee0693718682e6bd5993c9df8a7210879daabd"},"cell_type":"markdown","source":"## 2.SVR"},{"metadata":{"trusted":true,"_uuid":"48027c7598f4cf943140ce8cde9ec97808950a90"},"cell_type":"code","source":"def svr_code(SVR,X_train_scaled,y_train):\n    svm3 = SVR(C=1000, verbose = 1)\n    svm3.fit(X_train_scaled, y_train.values.flatten())\n    y_pred = svm3.predict(X_train_scaled)\n    plt.figure(figsize=(20, 6))\n    plt.scatter(y_train.values.flatten(), y_pred)\n    plt.xlim(0, 20)\n    plt.ylim(0, 20)\n    plt.xlabel('actual', fontsize=12)\n    plt.ylabel('predicted', fontsize=12)\n    plt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\n    plt.title('SVR')\n    plt.show()\n    score = rmse(y_train.values.flatten(), y_pred)\n    print(f'RMSE Score: {score:0.3f}')\n    score = r2_score(y_train.values.flatten(), y_pred)\n    print(f'Score: {score:0.3f}')\n    return (y_pred,score, svm3)\n    \ny_pred_SVR, score, svm3 = svr_code(SVR,X_train_scaled,y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e622b50af739ad66e28864109d6574020fa4c0b"},"cell_type":"markdown","source":"## 3.Kernel Ridge"},{"metadata":{"trusted":true,"_uuid":"4e4f4405ed25f45fa2881f0893cd8d5bfed0fbb6"},"cell_type":"code","source":"def br_code(BayesianRidge,X_train_scaled,y_train):\n    svm2 = KernelRidge(kernel='rbf',alpha = 0.05, gamma = 0.06)\n    svm2.fit(X_train_scaled, y_train.values.flatten())\n    y_pred = svm2.predict(X_train_scaled)\n    plt.figure(figsize=(20, 6))\n    plt.scatter(y_train.values.flatten(), y_pred)\n    plt.xlim(0, 20)\n    plt.ylim(0, 20)\n    plt.xlabel('actual', fontsize=12)\n    plt.ylabel('predicted', fontsize=12)\n    plt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\n    plt.title('Kernel Ridge Regression')\n    plt.show()\n    score = rmse(y_train.values.flatten(), y_pred)\n    print(f'RMSE Score: {score:0.3f}')\n    score = r2_score(y_train.values.flatten(), y_pred)\n    print(f'Score: {score:0.3f}')\n    return (y_pred,score, svm2)\n    \ny_pred_Bayesian, score, svm2 = br_code(BayesianRidge,X_train_scaled,y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a894ac439a7d2fabfe4cc11a9ab751f2aa6b5fb"},"cell_type":"markdown","source":"## 4.LightGBM Regression"},{"metadata":{"trusted":true,"_uuid":"a2574254e13bf3b4feae3a77e3b3aceb846db91c"},"cell_type":"code","source":"svm5 = LGBMRegressor(num_leaves=31, max_depth=-1, learning_rate=0.01, n_estimators=1000)\nsvm5.fit(X_train_scaled, y_train.values.flatten())\ny_pred_lgb = svm5.predict(X_train_scaled)\nplt.figure(figsize=(20, 6))\nplt.scatter(y_train.values.flatten(), y_pred_lgb)\nplt.xlim(0, 20)\nplt.ylim(0, 20)\nplt.xlabel('actual', fontsize=12)\nplt.ylabel('predicted', fontsize=12)\nplt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\nplt.title('Light GBM')\nscore = rmse(y_train.values.flatten(), y_pred_lgb)\nprint(f'RMSE Score: {score:0.3f}')\nscore = r2_score(y_train.values.flatten(), y_pred_lgb)\nprint(f'Score: {score:0.3f}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b794010fcddf5975f20dba6b74f5ebe913c3dd9"},"cell_type":"markdown","source":"## 5.CatBoost Regression"},{"metadata":{"trusted":true,"_uuid":"d17696441a9c9dfa2aa624c70f0b490f56f285dd","scrolled":true},"cell_type":"code","source":"def cat_code(CatBoostRegressor,X_train_scaled,y_train):\n    svm4 = CatBoostRegressor(depth=8)\n    svm4.fit(X_train_scaled, y_train.values.flatten())\n    y_pred = svm4.predict(X_train_scaled)\n    plt.figure(figsize=(20, 6))\n    plt.scatter(y_train.values.flatten(), y_pred)\n    plt.xlim(0, 20)\n    plt.ylim(0, 20)\n    plt.xlabel('actual', fontsize=12)\n    plt.ylabel('predicted', fontsize=12)\n    plt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\n    plt.title('CATBOOST')\n    plt.show()\n    score = rmse(y_train.values.flatten(), y_pred)\n    print(f'RMSE Score: {score:0.3f}')\n    score = r2_score(y_train.values.flatten(), y_pred)\n    print(f'Score: {score:0.3f}')\n    return (y_pred,score, svm4)\n    \ny_pred_cat, score, svm4 = cat_code(CatBoostRegressor,X_train_scaled,y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b147d13ca112228abd117056f8ef3c9200c6900f"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>7.Prediction</strong></span>"},{"metadata":{"trusted":true,"_uuid":"03adea012f55b38c40478073e63aa0da6570a8a3"},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\nX_test = pd.DataFrame(columns=X_train.columns, dtype=np.float64, index=submission.index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d1d287427b80f7d13e774c7c1ebb814b40613145"},"cell_type":"code","source":"for seg_id in tqdm(X_test.index):\n    seg = pd.read_csv('../input/test/' + seg_id + '.csv')\n    \n    x = seg['acoustic_data'].values\n    \n    X_test.loc[seg_id, 'ave'] = x.mean()\n    X_test.loc[seg_id, 'std'] = x.std()\n    X_test.loc[seg_id, 'max'] = x.max()\n    X_test.loc[seg_id, 'min'] = x.min()\n    X_test.loc[seg_id, 'q95'] = np.quantile(x,0.95)\n    X_test.loc[seg_id, 'q99'] = np.quantile(x,0.99)\n    X_test.loc[seg_id, 'q05'] = np.quantile(x,0.05)\n    X_test.loc[seg_id, 'q01'] = np.quantile(x,0.01)\n    \n    X_test.loc[seg_id, 'abs_max'] = np.abs(x).max()\n    X_test.loc[seg_id, 'abs_mean'] = np.abs(x).mean()\n    X_test.loc[seg_id, 'abs_std'] = np.abs(x).std()\n    X_test.loc[seg_id, 'trend'] = add_trend_feature(x)\n    X_test.loc[seg_id, 'abs_trend'] = add_trend_feature(x, abs_values=True)    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae60ffc04becd6b6a832cf193df9eb79a2c984b9"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>8.Stacking using lightgbm</strong></span>"},{"metadata":{"trusted":true,"_uuid":"4d87da3ab9e484f72faab2e23da40cc246262814"},"cell_type":"code","source":"f = [y_pred_nusvr,y_pred_SVR,y_pred_cat,y_pred_lgb,y_pred_Bayesian]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d29130dbc919afb24817f295b93f715150a0929"},"cell_type":"code","source":"f = np.transpose(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"821cb62cf72fdb1bee4a43b05ba7103f21a22d2b"},"cell_type":"code","source":"f.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e6a69ec95122531b879ade95bd96b5ede3da30c8"},"cell_type":"code","source":"svm6 = LGBMRegressor()\nsvm6.fit(f, y_train.values.flatten())\ny_pred_stack = svm6.predict(f)\nplt.figure(figsize=(20, 6))\nplt.scatter(y_train.values.flatten(), y_pred_lgb)\nplt.xlim(0, 20)\nplt.ylim(0, 20)\nplt.xlabel('actual', fontsize=12)\nplt.ylabel('predicted', fontsize=12)\nplt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\nplt.title('Light GBM Stacking')\nscore = rmse(y_train.values.flatten(), y_pred_stack)\nprint(f'RMSE Score: {score:0.3f}')\nscore = r2_score(y_train.values.flatten(), y_pred_lgb)\nprint(f'Score: {score:0.3f}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8749bfd6e74311d1a877b1a6ac37fbce6c455b0d"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>9.Total Analysis of all model</strong></span>\n\n| Model | RMSE | F1SCORE |\n|--|--|--|\n|**nuSVR**|**2.647**| **0.48** |\n|**SVR**|**2.635**|**0.485**|\n|**Kernel Ridge**|**2.653**|**0.478**|\n|**Lightgbm**|**2.029**|**0.695**|\n|**Catboost**|**2.491**|**0.54**|\n|**Stacking**|**1.200**|**0.695**|"},{"metadata":{"_kg_hide-output":false,"_kg_hide-input":false,"trusted":true,"_uuid":"06c6ac241a07f9e4c4cd8ce34170021e249eed13"},"cell_type":"code","source":"d = {'Model': ['nuSVR', 'SVR','Kernel Ridge','Lightgbm','Catboost','Stacking'], 'RMSE': [2.647,2.635,2.653,2.029,2.491,1.20],'F1_Score': [0.48,0.485,0.478,0.695,0.54,0.695]}\nanalysis_df = pd.DataFrame(d)\n# display(analysis_df)\n\nanalysis_df.index = analysis_df.Model\ndel analysis_df['Model']\ndisplay(analysis_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"7714d9e2bbdf903a5efeb0acd43603fbfa798832"},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nplt.plot(analysis_df.index,analysis_df.RMSE,'mD-',animated=True)\nplt.scatter(analysis_df.index, analysis_df.RMSE,s=y*50, cmap=\"Blues\", alpha=0.4, edgecolors=\"grey\", linewidth=2)\nplt.xlabel('Model')\nplt.ylabel('RMSE')\nplt.title(\"RMSE by Model\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67f126ac16ba8e69ecb957f197087b044c10d25e"},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nplt.plot(analysis_df.index,analysis_df.F1_Score,'rD-',animated=True)\nplt.scatter(analysis_df.index, analysis_df.F1_Score,s=y*50, cmap=\"Blues\", alpha=0.4, edgecolors=\"grey\", linewidth=2)\nplt.xlabel('Model')\nplt.ylabel('F1_Score')\nplt.title(\"F1_Score by Model\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"737f548688c1fc9adcd1333cbc2c7c61a3b494a4"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>8.Blending</strong></span>"},{"metadata":{"trusted":true,"_uuid":"882c6bacfdf703a02453091ddcb5eb31679153fd"},"cell_type":"code","source":"svm5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3719ae094356b6da701f473798efad03c75b2334"},"cell_type":"code","source":"from mlxtend.regressor import StackingRegressor\nsclf = StackingRegressor(regressors=[svm1,svm2,svm3,svm4,svm5,svm6], \n                          meta_regressor=SVR())\n\nsclf.fit(X_train_scaled, y_train.values.flatten())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22f836588678d03b7e1677e04c3a94ce945300e5"},"cell_type":"code","source":"y_pred_final = sclf.predict(X_train_scaled)\nplt.figure(figsize=(20, 6))\nplt.scatter(y_train.values.flatten(), y_pred_final)\nplt.xlim(0, 20)\nplt.ylim(0, 20)\nplt.xlabel('actual', fontsize=12)\nplt.ylabel('predicted', fontsize=12)\nplt.plot([(0, 0), (20, 20)], [(0, 0), (20, 20)])\nplt.title('Light GBM')\nscore = rmse(y_train.values.flatten(), y_pred_final)\nprint(f'RMSE Score: {score:0.3f}')\nscore = r2_score(y_train.values.flatten(), y_pred_final)\nprint(f'Score: {score:0.3f}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33e172004b1c5398f615a2fe81a45842fc60edbb"},"cell_type":"code","source":"X_test_scaled = scaler.transform(X_test)\n\nsubmission['time_to_failure'] = sclf.predict(X_test_scaled)\n# submission['time_to_failure1'] = svm1.predict(X_test_scaled)\n# submission['time_to_failure2'] = svm2.predict(X_test_scaled)\n# submission['time_to_failure3'] = svm3.predict(X_test_scaled)\n# submission['time_to_failure4'] = svm4.predict(X_test_scaled)\n# submission['time_to_failure5'] = svm5.predict(X_test_scaled)\n# submission['time_to_failure'] = (submission['time_to_failure1']+submission['time_to_failure2']+submission['time_to_failure3']+submission['time_to_failure4']+submission['time_to_failure5'])/5\n\n# del submission['time_to_failure1'],submission['time_to_failure2'],submission['time_to_failure3'],submission['time_to_failure4'],submission['time_to_failure5']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"216564a2b1b0ffa055ec031985d6e9bcc1809de0"},"cell_type":"code","source":"submission.to_csv(\"Advance_stack.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de982a7bcea447ce2c751d89978451eb514f7921"},"cell_type":"markdown","source":"# <span style=\"color:blue;\"><strong>10.Best Model</strong></span>"},{"metadata":{"trusted":true,"_uuid":"39b7a7fee534599f169ec6ab7d151879a30527bc"},"cell_type":"code","source":"submission1 = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\nsubmission1['time_to_failure'] = svm5.predict(X_test_scaled)\nsubmission1.to_csv(\"submission_lgbbestmodel.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fb160a6c54d6b0ddc35f196a398dcaf950ff81f"},"cell_type":"code","source":"submission1 = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\nsubmission1['time_to_failure'] = svm3.predict(X_test_scaled)\nsubmission1.to_csv(\"submission_svrbestmodel.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e004a34e0fa3d4ceb22087fe7b86d4402ecd47ba"},"cell_type":"code","source":"submission1 = pd.read_csv('../input/sample_submission.csv', index_col='seg_id')\nsubmission1['time_to_failure'] = svm1.predict(X_test_scaled)\nsubmission1.to_csv(\"submission_nusvrbestmodel.csv\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}