{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Often we get the time series data but due to lack of other features, we cannot convert the time series problem into a regression (or classification) problem. Recently, I came across [\"tsfresh\" package](https://tsfresh.readthedocs.io/en/latest/text/introduction.html) which helps in extracting features from time series data such as: mean, max, min, median, 0.4 quantile, 0.7 quantile, linear trend attribute intercept etc. Once we extract these features, the problem converts to a machine learning problem instead of a time series one, and we can apply ML models instead of applying time series models.\n\nIt is possible to calculate these statistical figures for given data by writing codes manually as well, but tsfresh automates the process. Also, tsfresh can extract more meaningful (features which have more impact on time series data) parameters as compared to which we can think of by writing manual code.  ","metadata":{}},{"cell_type":"code","source":"import os\nos.listdir(\"../input/LANL-Earthquake-Prediction\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-02T23:07:33.526456Z","iopub.execute_input":"2023-10-02T23:07:33.526836Z","iopub.status.idle":"2023-10-02T23:07:33.537346Z","shell.execute_reply.started":"2023-10-02T23:07:33.526796Z","shell.execute_reply":"2023-10-02T23:07:33.535849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\ntrain_data = pd.read_csv(os.path.join(\"../input/LANL-Earthquake-Prediction\",'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_failure': np.float64})","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:07:36.428101Z","iopub.execute_input":"2023-10-02T23:07:36.428470Z","iopub.status.idle":"2023-10-02T23:10:27.851673Z","shell.execute_reply.started":"2023-10-02T23:07:36.428432Z","shell.execute_reply":"2023-10-02T23:10:27.850240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am using the earthquake data here. It is a time series data having two fields.\n* **acoustic_data** = the time when signal was generated at the epicenter \n* **time_to_failure** = the time taken in seconds when quake is felt on the surface of earth","metadata":{}},{"cell_type":"code","source":"train_data.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:10:44.143192Z","iopub.execute_input":"2023-10-02T23:10:44.143883Z","iopub.status.idle":"2023-10-02T23:10:44.180498Z","shell.execute_reply.started":"2023-10-02T23:10:44.143832Z","shell.execute_reply":"2023-10-02T23:10:44.179324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Apparently, all 10 rows are showing same \"time_to_failure\" above. Is it really so? Let's try to see it with increased precision.","metadata":{}},{"cell_type":"code","source":"pd.options.display.precision = 12\ntrain_data.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:10:58.644000Z","iopub.execute_input":"2023-10-02T23:10:58.644781Z","iopub.status.idle":"2023-10-02T23:10:58.657939Z","shell.execute_reply.started":"2023-10-02T23:10:58.644719Z","shell.execute_reply":"2023-10-02T23:10:58.656632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can spot the difference!","metadata":{}},{"cell_type":"code","source":"train_data.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:11:03.877224Z","iopub.execute_input":"2023-10-02T23:11:03.877647Z","iopub.status.idle":"2023-10-02T23:11:03.886023Z","shell.execute_reply.started":"2023-10-02T23:11:03.877597Z","shell.execute_reply":"2023-10-02T23:11:03.884946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:11:10.687149Z","iopub.execute_input":"2023-10-02T23:11:10.687607Z","iopub.status.idle":"2023-10-02T23:11:10.708990Z","shell.execute_reply.started":"2023-10-02T23:11:10.687575Z","shell.execute_reply":"2023-10-02T23:11:10.706965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The train data set is huge here. Let us work with a fraction of data set. Our main purpose is to extract features here. We are not going to solve the entire ML modelling in this notebook.","metadata":{}},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:19:17.381100Z","iopub.execute_input":"2023-10-02T23:19:17.381469Z","iopub.status.idle":"2023-10-02T23:19:17.579777Z","shell.execute_reply.started":"2023-10-02T23:19:17.381428Z","shell.execute_reply":"2023-10-02T23:19:17.578684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 1st sample = 0.0025% of total data\ntrain_acoustic_data_sample_1 = train_data['acoustic_data'].values[::40000]\ntrain_time_to_failure_sample_1 = train_data['time_to_failure'].values[::40000]","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:19:26.867094Z","iopub.execute_input":"2023-10-02T23:19:26.867387Z","iopub.status.idle":"2023-10-02T23:19:26.872150Z","shell.execute_reply.started":"2023-10-02T23:19:26.867362Z","shell.execute_reply":"2023-10-02T23:19:26.871174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\ndef plot_data(train_ad_sample_df, train_ttf_sample_df):\n    fig, ax = plt.subplots(2,1, figsize=(13, 10))\n    ax[0].set_title(\"Acoustic Data: {:.4f} % sampled data\".format(float(train_ad_sample_df.shape[0]/train_data.shape[0])*100))\n    ax[0].plot(train_ad_sample_df, color='red')\n    ax[0].set_ylabel('acoustic data', color='red')\n    ax[0].set_xlabel('index', color='red')\n    ax[1].set_title(\"Time to Failure: {:.4f} % sampled data\".format(float(train_ad_sample_df.shape[0]/train_data.shape[0])*100))\n    ax[1].plot(train_ttf_sample_df, color='green')\n    ax[1].set_ylabel('time to failure', color='green')\n    ax[1].set_xlabel('index', color='green')","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:19:30.291582Z","iopub.execute_input":"2023-10-02T23:19:30.291990Z","iopub.status.idle":"2023-10-02T23:19:30.299522Z","shell.execute_reply.started":"2023-10-02T23:19:30.291947Z","shell.execute_reply":"2023-10-02T23:19:30.298922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_data(train_acoustic_data_sample_1, train_time_to_failure_sample_1)\ndel train_acoustic_data_sample_1\ndel train_time_to_failure_sample_1","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:19:33.868582Z","iopub.execute_input":"2023-10-02T23:19:33.869216Z","iopub.status.idle":"2023-10-02T23:19:34.211950Z","shell.execute_reply.started":"2023-10-02T23:19:33.869175Z","shell.execute_reply":"2023-10-02T23:19:34.210744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation for 1st Sample Data Set :** We have taken 0.0025% sample data here, where each data point is situated at 40000 gap. The 'acoustic data' is not varying much whereas 'time to failure' is varying aggressively when plotted against the index.","metadata":{}},{"cell_type":"code","source":"# 2nd sample = 0.25% of total data\ntrain_acoustic_data_sample_2 = train_data['acoustic_data'].values[::400]\ntrain_time_to_failure_sample_2 = train_data['time_to_failure'].values[::400]","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:23.656120Z","iopub.execute_input":"2023-10-02T23:43:23.656764Z","iopub.status.idle":"2023-10-02T23:43:23.662810Z","shell.execute_reply.started":"2023-10-02T23:43:23.656720Z","shell.execute_reply":"2023-10-02T23:43:23.661608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_data(train_acoustic_data_sample_2, train_time_to_failure_sample_2)\ndel train_acoustic_data_sample_2\ndel train_time_to_failure_sample_2","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:25.927180Z","iopub.execute_input":"2023-10-02T23:43:25.927743Z","iopub.status.idle":"2023-10-02T23:43:26.528855Z","shell.execute_reply.started":"2023-10-02T23:43:25.927693Z","shell.execute_reply":"2023-10-02T23:43:26.528119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation for 2nd Sample Data Set :** We have taken 0.25% sample data here, where each data point is situated at 400 gap. The 'acoustic data' is now varying comparatively high w.r.t. index. 'Time to failure' is varying on a similar aggressive level as before when plotted against the index.","metadata":{}},{"cell_type":"code","source":"# 3rd sample = 5% of total data\ntrain_acoustic_data_sample_3 = train_data['acoustic_data'].values[::20]\ntrain_time_to_failure_sample_3 = train_data['time_to_failure'].values[::20]","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:30.336439Z","iopub.execute_input":"2023-10-02T23:43:30.336790Z","iopub.status.idle":"2023-10-02T23:43:30.342177Z","shell.execute_reply.started":"2023-10-02T23:43:30.336758Z","shell.execute_reply":"2023-10-02T23:43:30.341021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_data(train_acoustic_data_sample_3, train_time_to_failure_sample_3)\ndel train_acoustic_data_sample_3\ndel train_time_to_failure_sample_3","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:32.918661Z","iopub.execute_input":"2023-10-02T23:43:32.919014Z","iopub.status.idle":"2023-10-02T23:43:38.323657Z","shell.execute_reply.started":"2023-10-02T23:43:32.918981Z","shell.execute_reply":"2023-10-02T23:43:38.322585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation for 3rd Sample Data Set :** We have taken 5% sample data here, where each data point is situated at 20 gap. Now, both 'acoustic data' and 'time to failure' are varying aggressively w.r.t. index.\n\n**Overall Observation:**\nAs we are increasing our sample size, the variation range of both 'acoustic data' and 'time to failure' are increasing.","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:42.455355Z","iopub.execute_input":"2023-10-02T23:43:42.455700Z","iopub.status.idle":"2023-10-02T23:43:42.559868Z","shell.execute_reply.started":"2023-10-02T23:43:42.455667Z","shell.execute_reply":"2023-10-02T23:43:42.558727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tsfresh\nfrom tsfresh import select_features\nfrom tsfresh.utilities.dataframe_functions import impute\nfrom tsfresh import extract_features\nfrom tsfresh.feature_extraction import EfficientFCParameters\nfrom tsfresh.feature_extraction.settings import from_columns","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:44.911840Z","iopub.execute_input":"2023-10-02T23:43:44.912186Z","iopub.status.idle":"2023-10-02T23:43:46.748528Z","shell.execute_reply.started":"2023-10-02T23:43:44.912156Z","shell.execute_reply":"2023-10-02T23:43:46.747795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sample training data\ntrain_data = train_data[:6000000]\ntrain_data = train_data.reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:53.289938Z","iopub.execute_input":"2023-10-02T23:43:53.290640Z","iopub.status.idle":"2023-10-02T23:43:53.335007Z","shell.execute_reply.started":"2023-10-02T23:43:53.290590Z","shell.execute_reply":"2023-10-02T23:43:53.333555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will segregate 6000000 records into 600 segments each having 10000 rows. Each segment is then allotted an id. So our data will be having ids from 1 till 600. The purpose of creating \"id\" and \"index\" columns in the data set is: \"tsfresh\" requires the data to follow a [particular format](https://tsfresh.readthedocs.io/en/latest/text/data_formats.html).","metadata":{}},{"cell_type":"code","source":"rows = 10000\nidlist = []\nfor n in range(1,601):  #600 segments\n    idlist = idlist + [n for i in range(rows)]","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:43:55.598335Z","iopub.execute_input":"2023-10-02T23:43:55.598692Z","iopub.status.idle":"2023-10-02T23:44:07.204929Z","shell.execute_reply.started":"2023-10-02T23:43:55.598660Z","shell.execute_reply":"2023-10-02T23:44:07.203765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:14.520129Z","iopub.execute_input":"2023-10-02T23:44:14.520692Z","iopub.status.idle":"2023-10-02T23:44:14.654964Z","shell.execute_reply.started":"2023-10-02T23:44:14.520639Z","shell.execute_reply":"2023-10-02T23:44:14.653533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data['id'] = idlist\ntrain_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:16.503687Z","iopub.execute_input":"2023-10-02T23:44:16.504113Z","iopub.status.idle":"2023-10-02T23:44:17.606554Z","shell.execute_reply.started":"2023-10-02T23:44:16.504070Z","shell.execute_reply":"2023-10-02T23:44:17.605491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.tail()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:19.391933Z","iopub.execute_input":"2023-10-02T23:44:19.392311Z","iopub.status.idle":"2023-10-02T23:44:19.403479Z","shell.execute_reply.started":"2023-10-02T23:44:19.392277Z","shell.execute_reply":"2023-10-02T23:44:19.402391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:21.663630Z","iopub.execute_input":"2023-10-02T23:44:21.663979Z","iopub.status.idle":"2023-10-02T23:44:21.783532Z","shell.execute_reply.started":"2023-10-02T23:44:21.663949Z","shell.execute_reply":"2023-10-02T23:44:21.782604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, we are now keeping only 'acoustic data' as our independent variable 'x' and separating out the 'time to failure' as dependent variable 'y'.","metadata":{}},{"cell_type":"code","source":"y = train_data['time_to_failure']\nx = train_data.drop(columns = 'time_to_failure')","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:24.086096Z","iopub.execute_input":"2023-10-02T23:44:24.086456Z","iopub.status.idle":"2023-10-02T23:44:24.167688Z","shell.execute_reply.started":"2023-10-02T23:44:24.086421Z","shell.execute_reply":"2023-10-02T23:44:24.166134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:26.727511Z","iopub.execute_input":"2023-10-02T23:44:26.727842Z","iopub.status.idle":"2023-10-02T23:44:26.736692Z","shell.execute_reply.started":"2023-10-02T23:44:26.727812Z","shell.execute_reply":"2023-10-02T23:44:26.735516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_data","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:29.528627Z","iopub.execute_input":"2023-10-02T23:44:29.529001Z","iopub.status.idle":"2023-10-02T23:44:29.533637Z","shell.execute_reply.started":"2023-10-02T23:44:29.528965Z","shell.execute_reply":"2023-10-02T23:44:29.532558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From 'y', we will now separate out 600 data points (equally distant at 10000 gap) for using them as a \"target\" later for feature extraction purpose.","metadata":{}},{"cell_type":"code","source":"target = y[9999::10000]","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:32.792105Z","iopub.execute_input":"2023-10-02T23:44:32.792481Z","iopub.status.idle":"2023-10-02T23:44:32.798314Z","shell.execute_reply.started":"2023-10-02T23:44:32.792441Z","shell.execute_reply":"2023-10-02T23:44:32.796804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:34.512169Z","iopub.execute_input":"2023-10-02T23:44:34.512517Z","iopub.status.idle":"2023-10-02T23:44:34.520010Z","shell.execute_reply.started":"2023-10-02T23:44:34.512483Z","shell.execute_reply":"2023-10-02T23:44:34.519220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target.index = range(1,601)","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:37.008608Z","iopub.execute_input":"2023-10-02T23:44:37.008947Z","iopub.status.idle":"2023-10-02T23:44:37.013722Z","shell.execute_reply.started":"2023-10-02T23:44:37.008920Z","shell.execute_reply":"2023-10-02T23:44:37.012610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:39.373927Z","iopub.execute_input":"2023-10-02T23:44:39.374272Z","iopub.status.idle":"2023-10-02T23:44:39.380481Z","shell.execute_reply.started":"2023-10-02T23:44:39.374243Z","shell.execute_reply":"2023-10-02T23:44:39.379970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:45.447287Z","iopub.execute_input":"2023-10-02T23:44:45.447919Z","iopub.status.idle":"2023-10-02T23:44:45.567694Z","shell.execute_reply.started":"2023-10-02T23:44:45.447871Z","shell.execute_reply":"2023-10-02T23:44:45.566544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x = x.rename(columns = {'index':'time'})","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:50.417020Z","iopub.execute_input":"2023-10-02T23:44:50.417657Z","iopub.status.idle":"2023-10-02T23:44:50.439478Z","shell.execute_reply.started":"2023-10-02T23:44:50.417618Z","shell.execute_reply":"2023-10-02T23:44:50.438841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will extract full features using 'x' as independent variable data set.","metadata":{}},{"cell_type":"code","source":"extracted_features = extract_features(x, column_id=\"id\", column_sort=\"time\", default_fc_parameters=EfficientFCParameters())","metadata":{"execution":{"iopub.status.busy":"2023-10-02T23:44:53.961287Z","iopub.execute_input":"2023-10-02T23:44:53.961767Z","iopub.status.idle":"2023-10-02T23:54:18.121614Z","shell.execute_reply.started":"2023-10-02T23:44:53.961738Z","shell.execute_reply":"2023-10-02T23:54:18.120411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us have a look at what is the entire list of features we have extracted.","metadata":{}},{"cell_type":"code","source":"extracted_features.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:10.974027Z","iopub.execute_input":"2023-10-03T00:00:10.974494Z","iopub.status.idle":"2023-10-03T00:00:11.003645Z","shell.execute_reply.started":"2023-10-03T00:00:10.974453Z","shell.execute_reply":"2023-10-03T00:00:11.002443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, we have extracted 773 features i.e. too many to deal with. We need to have only the features having the highest impact on \"target\" (using **Regression Model** here to judge the impact). For that purpose, we will use a threshold called **\"FDR level\"** which is the theoretical expected percentage of irrelevant features among all created features. ","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:21.009991Z","iopub.execute_input":"2023-10-03T00:00:21.010547Z","iopub.status.idle":"2023-10-03T00:00:21.159905Z","shell.execute_reply.started":"2023-10-03T00:00:21.010513Z","shell.execute_reply":"2023-10-03T00:00:21.158475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before performing smaller feature set generation, we need to impute the big feature set first.","metadata":{}},{"cell_type":"code","source":"impute(extracted_features)","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:23.298462Z","iopub.execute_input":"2023-10-03T00:00:23.298771Z","iopub.status.idle":"2023-10-03T00:00:23.616810Z","shell.execute_reply.started":"2023-10-03T00:00:23.298746Z","shell.execute_reply":"2023-10-03T00:00:23.616116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#The fdr level is the threshold of feature importance. Ii is set as very low to get smaller number of features.\nsmall_feat_set = select_features(extracted_features, target, fdr_level = 0.005, ml_task = 'regression')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:27.178081Z","iopub.execute_input":"2023-10-03T00:00:27.178505Z","iopub.status.idle":"2023-10-03T00:00:27.634681Z","shell.execute_reply.started":"2023-10-03T00:00:27.178467Z","shell.execute_reply":"2023-10-03T00:00:27.633764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:30.114942Z","iopub.execute_input":"2023-10-03T00:00:30.115554Z","iopub.status.idle":"2023-10-03T00:00:30.123002Z","shell.execute_reply.started":"2023-10-03T00:00:30.115511Z","shell.execute_reply":"2023-10-03T00:00:30.122126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = target.values.reshape(600,1)","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:31.882485Z","iopub.execute_input":"2023-10-03T00:00:31.882942Z","iopub.status.idle":"2023-10-03T00:00:31.887846Z","shell.execute_reply.started":"2023-10-03T00:00:31.882892Z","shell.execute_reply":"2023-10-03T00:00:31.887031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_feat_set.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:34.474149Z","iopub.execute_input":"2023-10-03T00:00:34.474821Z","iopub.status.idle":"2023-10-03T00:00:34.481149Z","shell.execute_reply.started":"2023-10-03T00:00:34.474775Z","shell.execute_reply":"2023-10-03T00:00:34.480271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_feat_set.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:36.737599Z","iopub.execute_input":"2023-10-03T00:00:36.738116Z","iopub.status.idle":"2023-10-03T00:00:36.749953Z","shell.execute_reply.started":"2023-10-03T00:00:36.738074Z","shell.execute_reply":"2023-10-03T00:00:36.748698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, now we can see that from our 773 features, really important features are only 27 in number!","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:39.818471Z","iopub.execute_input":"2023-10-03T00:00:39.819163Z","iopub.status.idle":"2023-10-03T00:00:39.961835Z","shell.execute_reply.started":"2023-10-03T00:00:39.819095Z","shell.execute_reply":"2023-10-03T00:00:39.960494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us check if there is any pair of features having high multicollinearity.","metadata":{}},{"cell_type":"code","source":"# Correlation Heatmap\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncorr = small_feat_set.corr()\nmask = np.triu(np.ones_like(corr, dtype=np.bool))\nf, ax = plt.subplots(figsize=(20, 20))\ncmap = sns.diverging_palette(1, 200, as_cmap=True)\nsns.heatmap(corr, mask=mask, cmap=cmap, vmax=None, center=0,square=True, annot=False, linewidths=.5, cbar_kws={\"shrink\": 0.8})","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:42.857335Z","iopub.execute_input":"2023-10-03T00:00:42.857702Z","iopub.status.idle":"2023-10-03T00:00:43.617762Z","shell.execute_reply.started":"2023-10-03T00:00:42.857667Z","shell.execute_reply":"2023-10-03T00:00:43.615835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to drop the highly correlated features to avoid perfect multicollinearity. For that, we need to spot the highly correlated feature pairs.","metadata":{}},{"cell_type":"code","source":"%%javascript\nIPython.OutputArea.prototype._should_scroll = function() {\n    return False;\n}","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:49.682502Z","iopub.execute_input":"2023-10-03T00:00:49.682887Z","iopub.status.idle":"2023-10-03T00:00:49.691225Z","shell.execute_reply.started":"2023-10-03T00:00:49.682851Z","shell.execute_reply":"2023-10-03T00:00:49.689903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Spot the categorical feature pairs with high correlation\nthreshold = 0.9999\nhigh_corrs = (corr[abs(corr) > threshold][corr != 1.0]).unstack().dropna().to_dict()\nunique_high_corrs = pd.DataFrame(list(set([(tuple(sorted(key)), high_corrs[key]) for key in high_corrs])), columns=['feature_pair', 'correlation_coefficient'])\nunique_high_corrs = unique_high_corrs.loc[abs(unique_high_corrs['correlation_coefficient']).argsort()[::-1]]\npd.options.display.max_colwidth = 200\nunique_high_corrs","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:52.793985Z","iopub.execute_input":"2023-10-03T00:00:52.794377Z","iopub.status.idle":"2023-10-03T00:00:52.821402Z","shell.execute_reply.started":"2023-10-03T00:00:52.794343Z","shell.execute_reply":"2023-10-03T00:00:52.820243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:00:56.729776Z","iopub.execute_input":"2023-10-03T00:00:56.730244Z","iopub.status.idle":"2023-10-03T00:00:56.857571Z","shell.execute_reply.started":"2023-10-03T00:00:56.730198Z","shell.execute_reply":"2023-10-03T00:00:56.855575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_feat_set = small_feat_set.drop(['acoustic_data__count_above_mean', 'acoustic_data__mean', \n                                      'acoustic_data__linear_trend__attr_\"intercept\"', \n                                      'acoustic_data__agg_linear_trend__attr_\"intercept\"__chunk_len_10__f_agg_\"mean\"',\n'acoustic_data__agg_linear_trend__attr_\"intercept\"__chunk_len_5__f_agg_\"mean\"'],axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:01:13.602467Z","iopub.execute_input":"2023-10-03T00:01:13.602833Z","iopub.status.idle":"2023-10-03T00:01:13.609929Z","shell.execute_reply.started":"2023-10-03T00:01:13.602797Z","shell.execute_reply":"2023-10-03T00:01:13.608833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_feat_set.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:01:15.482655Z","iopub.execute_input":"2023-10-03T00:01:15.483070Z","iopub.status.idle":"2023-10-03T00:01:15.488604Z","shell.execute_reply.started":"2023-10-03T00:01:15.483008Z","shell.execute_reply":"2023-10-03T00:01:15.487693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:01:17.786725Z","iopub.execute_input":"2023-10-03T00:01:17.787379Z","iopub.status.idle":"2023-10-03T00:01:17.936275Z","shell.execute_reply.started":"2023-10-03T00:01:17.787334Z","shell.execute_reply":"2023-10-03T00:01:17.934503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we have only 23 important features!","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:01:19.723513Z","iopub.execute_input":"2023-10-03T00:01:19.724279Z","iopub.status.idle":"2023-10-03T00:01:19.729541Z","shell.execute_reply.started":"2023-10-03T00:01:19.724230Z","shell.execute_reply":"2023-10-03T00:01:19.728328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Alright. We will now determine the feature importance of our extracted features using various ML models like Random Forest Regressor and Extra Trees Regressor respectively.","metadata":{}},{"cell_type":"code","source":"import eli5\nfrom eli5.sklearn import PermutationImportance\nfrom sklearn.ensemble import RandomForestRegressor as rf\n\nperm = PermutationImportance(rf(n_estimators=100, random_state=12345).fit(small_feat_set,target),random_state=56789).fit(small_feat_set,target)\neli5.show_weights(perm, feature_names = small_feat_set.columns.tolist())","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:01:23.706407Z","iopub.execute_input":"2023-10-03T00:01:23.706901Z","iopub.status.idle":"2023-10-03T00:01:34.119779Z","shell.execute_reply.started":"2023-10-03T00:01:23.706865Z","shell.execute_reply":"2023-10-03T00:01:34.118726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesRegressor as et\n\nperm = PermutationImportance(et(max_features='auto').fit(small_feat_set,target),random_state=12345).fit(small_feat_set,target)\neli5.show_weights(perm, feature_names = small_feat_set.columns.tolist())","metadata":{"execution":{"iopub.status.busy":"2023-10-03T00:01:39.602442Z","iopub.execute_input":"2023-10-03T00:01:39.603010Z","iopub.status.idle":"2023-10-03T00:01:41.868118Z","shell.execute_reply.started":"2023-10-03T00:01:39.602972Z","shell.execute_reply":"2023-10-03T00:01:41.867253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From both the models, we have found that \"acoustic_data__c3__lag_1\", \"acoustic_data__c3__lag_2\", \"acoustic_data__c3__lag_3\" have high influence on the target variable. ","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}