{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"G2Net: End-to-End Pipeline
\n
\n\n# Introduction\nGravitational Waves (GW) is an astronomical event that takes place due to the collision among Black Holes or merging of Neutron stars. These signals are unimaginably tiny ripples in the fabric of space-time, and when captured by the detectors, signals gets buried in detector noise.\n\n## Objective:\nIn this Kaggle Challenge, we will be exploring how these mixed signal should be analysed and from there how GW can be detected from the signals. We will approach this problem as __binary classification__.\n\n## Data:\nHere, the training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (__LIGO Hanford__, __LIGO Livingston__, and __Virgo__). Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal. The task is to identify when a signal is present in the data (`target=1`). The simulated part is the gravitational wave signal, while the detector noise is real.\n\n__Files__\n* `train/` - the training set files, one npy file per observation; labels are provided in a files shown below\n* `test/` - the test set files; you must predict the probability that the observation contains a gravitational wave\n* `training_labels.csv` - target values of whether the associated signal contains a gravitational wave\n* `sample_submission.csv` - a sample submission file in the correct format","metadata":{}},{"cell_type":"code","source":"!pip install -q dtaidistance astropy","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport json\nimport random\nimport collections\nfrom pprint import pprint\n\nimport numpy as np\nimport pandas as pd\nfrom scipy import spatial\nfrom scipy import signal\n\nfrom IPython.display import display, HTML, Markdown, clear_output\nfrom tqdm.notebook import tqdm\n\nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot\nfrom plotly.subplots import make_subplots\nimport plotly.figure_factory as ff\nimport plotly.graph_objects as go\nfrom dtaidistance import dtw\nfrom dtaidistance import dtw_visualisation as dtwvis\n\nfrom astropy.timeseries import LombScargle\nimport librosa\n\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.preprocessing import RobustScaler, MinMaxScaler\nfrom sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV, cross_val_score\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.preprocessing import label_binarize\nfrom sklearn.svm import SVC\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.metrics import precision_recall_fscore_support, confusion_matrix, accuracy_score, classification_report, roc_auc_score, roc_curve, auc, plot_roc_curve\nfrom sklearn.linear_model import LogisticRegression\nimport xgboost as xgb\n\ninit_notebook_mode(connected=True)\n\nrandom.seed(369)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Exploration\n\n## Data Loading and Distribution\nFirst we load the `training_labels.csv` to see the distribution of the signals, check for any missing values and the distribution of the target variable.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/g2net-gravitational-wave-detection/training_labels.csv\")\ndisplay(train_df.head(10))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# finding the number of missing values across the columns\ndisplay(Markdown('Display the missing values:'))\ndisplay(train_df.isnull().sum())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# finding the frequency of the target variables\nfreq_df = train_df.groupby('target').count().reset_index()\nfreq_df['target'] = freq_df['target'].astype(str)\nfreq_fig = px.bar(freq_df, x='target', y='id', \n color='target', title='Target Class Distribution')\nfreq_fig.update_layout(height=400, template='plotly_white')\nfreq_fig.add_layout_image(\n dict(\n source='https://img.icons8.com/color/452/black-hole.png',\n x=0.7, y=0.9, sizex=0.25, sizey=0.25,\n )\n)\niplot(freq_fig)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Although the target distribution is textbook, we need to ensure we are not going to overfit the model when training.\n\nAs mentioned in the data description, we have 3 detector sites and each of them have signal values of duration 2 seconds sampled at 2048 Hz frequency. Now, we have to explore this data in deoth.\n\n## Histogram\n\nWe can start off by filtering the data based on the target varaible and taking a sample of data points from each sites to calculate sample histogram distribution of the data. Then we will superimpose the result of data with signal and with no signal of the correspondig sites and see how does the data differ.","metadata":{}},{"cell_type":"code","source":"def hist_plot(data, group_labels, det_colors, title):\n hist_fig = ff.create_distplot(data, group_labels, show_hist=False, colors=det_colors)\n hist_fig.update_layout(height=400, title_text=title, template=\"plotly_white\")\n iplot(hist_fig)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sig_det_df = train_df[train_df[\"target\"] == 1]\ndet_df = train_df[train_df[\"target\"] == 0]\n\ndet1_sd, det2_sd, det3_sd = [], [], []\ndisplay(Markdown(\"For Target 1: Signal + Detector\"))\nfor each_id in tqdm(random.sample(sig_det_df.index.tolist(), 100)):\n img_id = train_df.loc[each_id, \"id\"]\n target_var = train_df.loc[each_id, \"target\"]\n file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(\n img_id[0], img_id[1], img_id[2], img_id\n )\n train_arr = np.load(file_path)\n train_arr_df = pd.DataFrame.from_records(train_arr)\n det1_sd.append(train_arr_df.iloc[0, :])\n det2_sd.append(train_arr_df.iloc[1, :])\n det3_sd.append(train_arr_df.iloc[2, :])\n\n\ndet1_d, det2_d, det3_d = [], [], []\ndisplay(Markdown(\"For Target 0: Detector\"))\nfor each_id in tqdm(random.sample(det_df.index.tolist(), 100)):\n img_id = train_df.loc[each_id, \"id\"]\n target_var = train_df.loc[each_id, \"target\"]\n file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(\n img_id[0], img_id[1], img_id[2], img_id\n )\n train_arr = np.load(file_path)\n train_arr_df = pd.DataFrame.from_records(train_arr)\n det1_d.append(train_arr_df.iloc[0, :])\n det2_d.append(train_arr_df.iloc[1, :])\n det3_d.append(train_arr_df.iloc[2, :])\n\n\ndet1_sd_df, det2_sd_df, det3_sd_df = pd.DataFrame(det1_sd), pd.DataFrame(det2_sd), pd.DataFrame(det3_sd)\ndet1_d_df, det2_d_df, det3_d_df = pd.DataFrame(det1_d), pd.DataFrame(det2_d), pd.DataFrame(det3_d)\n\ndet1_sd_df_mean, det2_sd_df_mean, det3_sd_df_mean = det1_sd_df.mean(axis=0), det2_sd_df.mean(axis=0), det3_sd_df.mean(axis=0)\ndet1_d_df_mean, det2_d_df_mean, det3_d_df_mean = det1_d_df.mean(axis=0), det2_d_df.mean(axis=0), det3_d_df.mean(axis=0)\n\nall_det_hist_data = [det1_sd_df_mean, det1_d_df_mean, det2_sd_df_mean, det2_d_df_mean, det3_sd_df_mean, det3_d_df_mean]\nall_det_colors = ['red', 'orange', 'purple', 'blue', 'darkgreen', 'lightgreen']\n\ngroup_labels = ['Site #1 Target 1', 'Site #1 Target 0', \n 'Site #2 Target 1', 'Site #2 Target 0', \n 'Site #3 Target 1', 'Site #3 Target 0']\n \nhist_plot(all_det_hist_data, group_labels, all_det_colors, 'Sample Average Histogram of All Site Detection')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Visualization: Time-domain\n\nWe first split up the data against the 2 classes and observe how does the data on individual level looks like.","metadata":{}},{"cell_type":"code","source":"# let us randomly select any 2 training files with target 1 and 0 to see how does the data looks like.\n\ndef visualize_series(each_id):\n display(Markdown(f\"Selected ID: {each_id}\"))\n img_id = train_df.loc[each_id, \"id\"]\n target_var = train_df.loc[each_id, \"target\"]\n file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(\n img_id[0], img_id[1], img_id[2], img_id\n )\n train_arr = np.load(file_path)\n train_arr_df = pd.DataFrame.from_records(train_arr)\n display(train_arr_df)\n \n if target_var == 1:\n color = [\"red\", \"purple\", \"darkgreen\"]\n else:\n color = [\"orange\", \"blue\", \"lightgreen\"]\n\n train_fig = go.Figure()\n\n for idx in list(range(train_arr_df.shape[0])):\n train_fig.add_trace(\n go.Scatter(\n x=list(range(train_arr_df.shape[1])),\n y=train_arr_df.iloc[idx, :],\n mode=\"lines\",\n name=f\"Detector_{idx+1}\",\n marker_color=color[idx],\n yaxis=[\"y\", \"y2\", \"y3\"][idx]\n )\n )\n\n for idx in list(range(train_arr_df.shape[0])):\n train_fig.add_trace(\n go.Scatter(\n x=list(range(train_arr_df.shape[1])),\n y=train_arr_df.iloc[idx, :],\n mode=\"lines\",\n name=f\"Detector_{idx+1}\",\n showlegend=False,\n marker_color=color[idx],\n yaxis=\"y4\"\n )\n )\n\n train_fig.update_layout(\n hovermode=\"x\",\n xaxis=dict(\n autorange=True,\n range=[0, train_arr_df.shape[1]],\n rangeslider=dict(autorange=True, range=[0, train_arr_df.shape[1]]),\n ),\n yaxis=dict(\n anchor=\"x\",\n autorange=True,\n domain=[0, 0.2],\n linecolor=\"red\",\n side=\"left\",\n type=\"linear\",\n zeroline=False,\n ),\n yaxis2=dict(\n anchor=\"x\",\n autorange=True,\n domain=[0.25, 0.45],\n linecolor=\"blue\",\n side=\"left\",\n type=\"linear\",\n zeroline=False,\n ),\n yaxis3=dict(\n anchor=\"x\",\n autorange=True,\n domain=[0.5, 0.7],\n linecolor=\"green\",\n side=\"left\",\n type=\"linear\",\n zeroline=False,\n ),\n yaxis4=dict(\n anchor=\"x\",\n autorange=True,\n domain=[0.75, 0.95],\n side=\"left\",\n type=\"linear\",\n zeroline=False,\n ),\n title_text=f\"GW Observation ID: {img_id} and target: {target_var} across 3 centers\",\n template=\"plotly_white\"\n )\n iplot(train_fig)\n\n\nsig_det_df = train_df[train_df[\"target\"] == 1]\ndet_df = train_df[train_df[\"target\"] == 0]\n\ndisplay(Markdown(\"For Target 1: Signal + Detector\"))\nfor each_id in random.sample(sig_det_df.index.tolist(), 2):\n visualize_series(each_id)\n\ndisplay(Markdown(\"For Target 0: Detector\"))\nfor each_id in random.sample(det_df.index.tolist(), 2):\n visualize_series(each_id)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We don't see much difference! This was expected, isn't it? These waves are so faint that it is difficult to observe using these time series plots. Let's try to understand it from another perspective.\n\nWe are randomly selecting `N` series from each observation center that __has the signal__. For simplicity, let us assume that `S1` is the data obtained from center 1. Now we select `M` random data that has __no signal__. We are going to compute the _time-domain similarity_ score of `S1` against all the `M` datapoints that we selected using __cross-correlation__ and try to see that which has the closest similarity. In that way, we can get a rough estimation of how visually should we see the data with signal and without signal.\n\nSince it is randomly selected, we can expect some error in the estimation because the approach in which the data has been generated was not mentioned - hence difficult to validate. Also, there is no naming convention to properly compare.","metadata":{}},{"cell_type":"code","source":"sig_det_df = train_df[train_df[\"target\"] == 1]\ndet_df = train_df[train_df[\"target\"] == 0]\n\n\ndet1_rec_sd, det1_rec_d_tmp, det1_rec_d = [], [], []\ndet2_rec_sd, det2_rec_d_tmp, det2_rec_d = [], [], []\ndet3_rec_sd, det3_rec_d_tmp, det3_rec_d = [], [], []\n\nfor sd_id in tqdm(random.sample(sig_det_df.index.tolist(), 10)):\n sd_img_id = sig_det_df.loc[sd_id, 'id']\n file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(sd_img_id[0], sd_img_id[1], sd_img_id[2], sd_img_id)\n train_arr = np.load(file_path)\n train_arr_df = pd.DataFrame.from_records(train_arr)\n \n det1_rec_sd.append(train_arr_df.iloc[0, :])\n det2_rec_sd.append(train_arr_df.iloc[1, :])\n det3_rec_sd.append(train_arr_df.iloc[2, :])\n\nfor d_id in tqdm(random.sample(det_df.index.tolist(), 100)):\n d_img_id = det_df.loc[d_id, 'id']\n file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(d_img_id[0], d_img_id[1], d_img_id[2], d_img_id)\n train_arr = np.load(file_path)\n train_arr_df = pd.DataFrame.from_records(train_arr)\n \n det1_rec_d_tmp.append(train_arr_df.iloc[0, :])\n det2_rec_d_tmp.append(train_arr_df.iloc[1, :])\n det3_rec_d_tmp.append(train_arr_df.iloc[2, :])\n \n# performing the time-domain, frequency-domain and power similarity\n\ndef compute_similarity(ref_rec,input_rec):\n ## Time domain similarity\n ref_time = np.correlate(ref_rec,ref_rec)\n inp_time = np.correlate(ref_rec,input_rec)\n diff_time = abs(ref_time-inp_time)\n\n ## Freq domain similarity\n ref_freq = np.correlate(np.fft.fft(ref_rec),np.fft.fft(ref_rec)) \n inp_freq = np.correlate(np.fft.fft(ref_rec),np.fft.fft(input_rec))\n diff_freq = abs(ref_freq-inp_freq)\n\n ## Power similarity\n ref_power = np.sum(ref_rec**2)\n inp_power = np.sum(input_rec**2)\n diff_power = abs(ref_power-inp_power)\n\n return (diff_time+diff_freq+diff_power)/3\n\n\nfor rec_sd1, rec_sd2, rec_sd3 in zip(tqdm(det1_rec_sd), det2_rec_sd, det3_rec_sd):\n diff_rec1, diff_rec2, diff_rec3 = [], [], []\n for rec_d1 in det1_rec_d_tmp:\n sim = compute_similarity(rec_d1, rec_sd1)\n diff_rec1.append(sim)\n \n for rec_d2 in det2_rec_d_tmp:\n sim = compute_similarity(rec_d2, rec_sd2)\n diff_rec2.append(sim)\n \n for rec_d3 in det3_rec_d_tmp:\n sim = compute_similarity(rec_d3, rec_sd3)\n diff_rec3.append(sim)\n \n det1_rec_d.append(det1_rec_d_tmp[diff_rec1.index(min(diff_rec1))])\n det2_rec_d.append(det2_rec_d_tmp[diff_rec2.index(min(diff_rec2))])\n det3_rec_d.append(det3_rec_d_tmp[diff_rec3.index(min(diff_rec3))])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we calculated the similar patterns of `signal+detector` and `detector` only across all the 3 sites, let's visualize it in using __heatmap__.","metadata":{}},{"cell_type":"code","source":"def show_heatmap(title, sd_rec, d_rec):\n hm_det_fig = make_subplots(rows=1, cols=2, subplot_titles=(\"Signal+Detector\", \"Detector\"))\n hm_det_fig.add_trace(\n go.Heatmap(z=sd_rec, legendgroup='group1'), \n row=1, col=1)\n hm_det_fig.add_trace(\n go.Heatmap(z=d_rec, legendgroup='group1', showscale=False), \n row=1, col=2)\n\n hm_det_fig.update_layout(height=300, showlegend=False, title_text=title)\n iplot(hm_det_fig)\n \nshow_heatmap(\"Detector 1 Record Heatmap\", det1_rec_sd, det1_rec_d)\nshow_heatmap(\"Detector 2 Record Heatmap\", det2_rec_sd, det2_rec_d)\nshow_heatmap(\"Detector 3 Record Heatmap\", det3_rec_sd, det3_rec_d)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Of course, we would expect different result everytime we run but we can still estimate how different would the observation look like if we do encounter a signal against a one that doesn't have.\n\nNow that we have overall comparison of the signals across sites, we will select the signals that are deemed simialr in a site and check how similar are they using __Dynamic Time Warping (DTW)__. ","metadata":{}},{"cell_type":"code","source":"# visualise similar signal from site 1\nd, paths = dtw.warping_paths(det1_rec_sd[0], det1_rec_d[0], window=128)\nbest_path = dtw.best_path(paths)\ndisplay(Markdown('Site #1'))\ndisplay(dtwvis.plot_warpingpaths(det1_rec_sd[0], det1_rec_d[0], paths, best_path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__UPDATE__: From these diagrams, we are witnessing a better understanding of how much the `signal+detector` data and `detector` data are similar. _I had to deleted some of these diagrams to avoid slowing down the notebook_.\n\n## Data Visualization: Frequency-domain\n\nSo far we have covered time-domain visualization. Now let us inspect frequency-domain based analysis.\n\n__Periodograms__\n\n_Power Spectral Density (PSD)_ is the measure of signal's power content versus frequency. __Lomb-Scargle periodogram__ is a commonly used statistical tool designed to detect and test the significance of weak periodic signals with uneven temporal sampling, and it is pretty commonly used in the field of astronomy.","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"# performing the Lomb-Scargle PSD\ndef ls_psd(y):\n freq, pwr = LombScargle(list(range(4096)), \n y).autopower(normalization='psd', \n minimum_frequency=0.98, maximum_frequency=1.02)\n return freq, pwr\n\nrecords = [det1_rec_sd, det1_rec_d, det2_rec_sd, det2_rec_d, det3_rec_sd, det3_rec_d]\n\nls_psd_res = []\nfor rec in records:\n frequency, power = ls_psd(rec[1])\n ls_psd_res.append([frequency, power])\n\npsd_fig = make_subplots(rows=3, cols=2, \n subplot_titles=['Site 1: Signal+Detector', 'Site 1: Detector',\n 'Site 2: Signal+Detector', 'Site 2: Detector',\n 'Site 3: Signal+Detector', 'Site 3: Detector'])\n\npsd_fig.add_trace(go.Scatter(x=ls_psd_res[0][0], y=ls_psd_res[0][1], mode='lines'), row=1, col=1)\npsd_fig.add_trace(go.Scatter(x=ls_psd_res[1][0], y=ls_psd_res[1][1], mode='lines'), row=1, col=2)\npsd_fig.add_trace(go.Scatter(x=ls_psd_res[2][0], y=ls_psd_res[2][1], mode='lines'), row=2, col=1)\npsd_fig.add_trace(go.Scatter(x=ls_psd_res[3][0], y=ls_psd_res[3][1], mode='lines'), row=2, col=2)\npsd_fig.add_trace(go.Scatter(x=ls_psd_res[4][0], y=ls_psd_res[4][1], mode='lines'), row=3, col=1)\npsd_fig.add_trace(go.Scatter(x=ls_psd_res[5][0], y=ls_psd_res[5][1], mode='lines'), row=3, col=2)\npsd_fig.update_layout(showlegend=False, title_text='Lomb-Scargle PSD')\niplot(psd_fig)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, in all the sites having the signal, the signal power is higher in the records having `signal+detector` compared to the `detector` by a magnitude of 2-3.\n\n## CQT\n\nThe constant-Q transform, simply known as CQT transforms a data series to the frequency domain. It is related to the Fourier transform.","metadata":{}},{"cell_type":"code","source":"def compute_cqt(data):\n return np.abs(librosa.cqt(np.array(data)/np.max(np.array(data)), sr=2048, n_bins=50, pad_mode='median'))\n\nshow_heatmap(\"Detector 1 CQT Heatmap\", compute_cqt(det1_rec_sd[1]), compute_cqt(det1_rec_d[1]))\nshow_heatmap(\"Detector 2 CQT Heatmap\", compute_cqt(det2_rec_sd[1]), compute_cqt(det2_rec_d[1]))\nshow_heatmap(\"Detector 3 CQT Heatmap\", compute_cqt(det3_rec_sd[1]), compute_cqt(det3_rec_d[1]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preparation and Processing\n\nFirst, we are taking a sample of the original dataset and perform the data preparation to perform the classifier model.\n\nHere, we have 3 detector sites and each of them have records of signal getting detected (target = 1) and with no signal (target = 0). So instead of the splitting the consolidated record of `training_labels.csv` into __train__ and __validation__ data points, we will split it up based on detector site.\n\nWe will be performing the __Mel-frequency cepstral coefficients (MFCC)__ to obtain the necessary frequency components that will be beneficial for this entire analysis.","metadata":{}},{"cell_type":"code","source":"sig_det_df = train_df[train_df[\"target\"] == 1]\ndet_df = train_df[train_df[\"target\"] == 0]\n\ndet1_total_sd, det2_total_sd, det3_total_sd = {}, {}, {}\ndet1_total_d, det2_total_d, det3_total_d = {}, {}, {}\n\nfor sd_id in tqdm(random.sample(sig_det_df.index.tolist(), 2000)):\n sd_img_id = sig_det_df.loc[sd_id, 'id']\n sd_file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(sd_img_id[0], sd_img_id[1], sd_img_id[2], sd_img_id)\n sd_arr = np.load(sd_file_path, mmap_mode='r')\n sd_arr_df = pd.DataFrame.from_records(sd_arr)\n \n sd1, sd2, sd3 = np.array(sd_arr_df.iloc[0, :]), np.array(sd_arr_df.iloc[1, :]), np.array(sd_arr_df.iloc[2, :])\n \n det1_total_sd[sd_img_id] = librosa.feature.mfcc(sd1/max(sd1), sr=2048).flatten()\n det2_total_sd[sd_img_id] = librosa.feature.mfcc(sd2/max(sd2), sr=2048).flatten()\n det3_total_sd[sd_img_id] = librosa.feature.mfcc(sd3/max(sd3), sr=2048).flatten()\n\nfor d_id in tqdm(random.sample(det_df.index.tolist(), 2000)):\n d_img_id = det_df.loc[d_id, 'id']\n d_file_path = \"../input/g2net-gravitational-wave-detection/train/{}/{}/{}/{}.npy\".format(d_img_id[0], d_img_id[1], d_img_id[2], d_img_id)\n d_arr = np.load(d_file_path, mmap_mode='r')\n d_arr_df = pd.DataFrame.from_records(d_arr)\n \n d1, d2, d3 = np.array(d_arr_df.iloc[0, :]), np.array(d_arr_df.iloc[1, :]), np.array(d_arr_df.iloc[2, :])\n \n det1_total_d[d_img_id] = librosa.feature.mfcc(d1/max(d1), sr=2048).flatten()\n det2_total_d[d_img_id] = librosa.feature.mfcc(d2/max(d2), sr=2048).flatten()\n det3_total_d[d_img_id] = librosa.feature.mfcc(d3/max(d3), sr=2048).flatten()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So now we obtained records of site 1,2,3 having the signals and no signals separately - basically `det1_total_sd` contains data of site 1 with `s`ignal+`d`etector and `det1_total_d` contains data of site 1 with only `d`etector. Likewise for site 2 and site 3.\n\nNext, we will concat the records of site 1 in a single data point - basically , `det1_total_sd + det1_total_d`. Likewise for site 2 and site 3.","metadata":{}},{"cell_type":"code","source":"# site 1\ndet1_total_sd = pd.DataFrame.from_dict(det1_total_sd, orient='index')\ndet1_total_sd['target'] = [1]*det1_total_sd.shape[0]\n\ndet1_total_d = pd.DataFrame.from_dict(det1_total_d, orient='index')\ndet1_total_d['target'] = [0]*det1_total_d.shape[0]\n\ndet1_total = pd.concat([det1_total_sd, det1_total_d])\n\n# site 2\ndet2_total_sd = pd.DataFrame.from_dict(det2_total_sd, orient='index')\ndet2_total_sd['target'] = [1]*det2_total_sd.shape[0]\n\ndet2_total_d = pd.DataFrame.from_dict(det2_total_d, orient='index')\ndet2_total_d['target'] = [0]*det2_total_d.shape[0]\n\ndet2_total = pd.concat([det2_total_sd, det2_total_d])\n\n# site 3\ndet3_total_sd = pd.DataFrame.from_dict(det3_total_sd, orient='index')\ndet3_total_sd['target'] = [1]*det3_total_sd.shape[0]\n\ndet3_total_d = pd.DataFrame.from_dict(det3_total_d, orient='index')\ndet3_total_d['target'] = [0]*det3_total_d.shape[0]\n\ndet3_total = pd.concat([det3_total_sd, det3_total_d])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we got the records of the respective sites, we will perform data split as per the convention - we get train and validation data points of all the sites individually. So we will concat them and get the final train and validation dataset.","metadata":{}},{"cell_type":"code","source":"df_train_1, df_valid_1, y_train_1, y_valid_1 = train_test_split(\n det1_total.iloc[:, :-1], \n det1_total.iloc[:, -1],\n test_size=0.2, \n random_state=36, \n stratify=det1_total.iloc[:, -1]\n)\n\ndf_train_2, df_valid_2, y_train_2, y_valid_2 = train_test_split(\n det2_total.iloc[:, :-1], \n det2_total.iloc[:, -1],\n test_size=0.2, \n random_state=36, \n stratify=det2_total.iloc[:, -1]\n)\n\ndf_train_3, df_valid_3, y_train_3, y_valid_3 = train_test_split(\n det3_total.iloc[:, :-1], \n det3_total.iloc[:, -1],\n test_size=0.2, \n random_state=36, \n stratify=det3_total.iloc[:, -1]\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Decided to take mean of the columns of data points for each site and obtain the distribution. It was observed that the data distribution is different for site 3. So I performed scaling and saw that the data across all sites are relatively aligned with each other.","metadata":{}},{"cell_type":"code","source":"# perform scaling\n\nsclr = MinMaxScaler()\nx_train_1, x_train_2, x_train_3 = sclr.fit_transform(np.array(df_train_1)), \\\nsclr.fit_transform(np.array(df_train_2)), sclr.fit_transform(np.array(df_train_3))\n\nx_valid_1, x_valid_2, x_valid_3 = sclr.fit_transform(np.array(df_valid_1)), \\\nsclr.fit_transform(np.array(df_valid_2)), sclr.fit_transform(np.array(df_valid_3))\n\n# displaying the data distribution before and after scaling\nhist_data = [df_train_1.mean(axis=0), df_train_2.mean(axis=0), df_train_3.mean(axis=0),\n df_valid_1.mean(axis=0), df_valid_2.mean(axis=0), df_valid_3.mean(axis=0)]\n\nscaled_hist_data = [x_train_1.mean(axis=0), x_train_2.mean(axis=0), x_train_3.mean(axis=0),\n x_valid_1.mean(axis=0), x_valid_2.mean(axis=0), x_valid_3.mean(axis=0)]\ndet_colors = ['red', 'purple', 'darkgreen', 'orange', 'blue', 'lightgreen']\ngroup_labels = ['Train 1', 'Train 2', 'Train 3', 'Valid 1', 'Valid 2', 'Valid 3']\n\nhist_plot(hist_data, group_labels, det_colors, 'Sample Average Histogram of All Site Detection (Original Data)')\nhist_plot(scaled_hist_data, group_labels, det_colors, 'Sample Average Histogram of All Site Detection (Scaled Data)')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we can observe that scaled data is relatively well distributed compared to the original one. Now we will concat the datapoints from all these sites.","metadata":{}},{"cell_type":"code","source":"df_train_final = pd.concat([pd.DataFrame.from_records(x_train_1), \n pd.DataFrame.from_records(x_train_2), \n pd.DataFrame.from_records(x_train_3)])\ny_train_final = pd.concat([y_train_1, y_train_2, y_train_3])\ndf_valid_final = pd.concat([pd.DataFrame.from_records(x_valid_1), \n pd.DataFrame.from_records(x_valid_2), \n pd.DataFrame.from_records(x_valid_3)])\ny_valid_final = pd.concat([y_valid_1, y_valid_2, y_valid_3])\n\ndisplay(Markdown(f'__Train Dimension:__ {df_train_final.shape}'))\ndisplay(Markdown(f'__Valid Dimension:__ {df_valid_final.shape}'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Importance and Selection\n\nFeature importance refers to a class of techniques for assigning scores to input features to a predictive model that indicates the relative importance of each feature when making a prediction.\n\nThe scores are useful and can be used in a range of situations in a predictive modeling problem, such as:\n* Better understanding the data.\n* Better understanding a model.\n* Reducing the number of input features.\n\nThere are various ways by which this operation can be performed:\n* Coefficients as Feature Importance\n* Decision Tree Feature Importance\n* Permutation Feature Importance\n\nWe will proceed with __Decision Tree Feature Importance__.","metadata":{}},{"cell_type":"code","source":"def calculate_feature_importance(model, model_name, x_train, y_train):\n model_fe = model.fit(x_train, y_train)\n\n fe = model_fe.feature_importances_\n fe_sorted = [fe[i] for i in fe.argsort()]\n feature_names = [f'feature_{i}' for i in fe.argsort()]\n \n mean_fe_score = np.mean(fe)\n selected_fe = [i for i, scr in enumerate(fe) if scr>mean_fe_score]\n\n fe_fig = go.Figure()\n fe_fig.add_trace(go.Bar(x=feature_names, y=fe_sorted))\n fe_fig.update_layout(height=400, title_text=f'{model_name} Feature Importance')\n iplot(fe_fig)\n \n return selected_fe","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rf_selected_features = calculate_feature_importance(RandomForestClassifier(random_state=369, n_jobs=-1, class_weight='balanced'),\n 'Random Forest', df_train_final, y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xgb_selected_features = calculate_feature_importance(xgb.XGBClassifier(objective=\"binary:logistic\", booster='gbtree', eval_metric='error', \n random_state=369, use_label_encoder=False, n_jobs=-1),\n 'XGBoost', df_train_final, y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt_selected_features = calculate_feature_importance(DecisionTreeClassifier(random_state=369, class_weight='balanced', max_features='auto'),\n 'Decision Tree', df_train_final, y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, usually we would be considering the common features obtained from these methods. So we take intersection of the features and obtain the final features.","metadata":{}},{"cell_type":"code","source":"display(Markdown('__Number of relevant features__'))\ndisplay(Markdown(f'Random Forest: {len(rf_selected_features)} || XGBoost {len(xgb_selected_features)} || Decision Tree {len(dt_selected_features)}'))\n\n# instersection\nfinal_selected_features = list(set(rf_selected_features) & set(xgb_selected_features) & set(dt_selected_features))\ndisplay(Markdown('__Number of common relevant features__'))\ndisplay(len(final_selected_features))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling\nWe will be performing a range of binary classifier operation using Grid-Search Cross Validation technique by passing a set of parameters corresponding to the classifier we are passing, then perform ensembling technique and assess how much good result are we obtaining.","metadata":{}},{"cell_type":"code","source":"# defining the utils\n\n#function for cross validation\ndef cross_valid_result(classifier, params, x_train, y_train):\n skf = StratifiedKFold(shuffle=True, random_state=369)\n grid_search = GridSearchCV(classifier, param_grid=params,\n cv=skf.split(x_train, y_train),\n scoring='accuracy', n_jobs=-1, verbose=2)\n model_opt = grid_search.fit(x_train, y_train)\n\n cv_res_df = pd.DataFrame(data=model_opt.cv_results_)\n cv_res_df.drop(['mean_fit_time', 'std_fit_time', 'mean_score_time', \n 'std_score_time', 'std_test_score', 'params'], axis=1, inplace=True)\n cv_res_df_cols = list(cv_res_df.columns)\n cv_res_df_cols.insert(0, cv_res_df_cols.pop())\n cv_res_df = cv_res_df.reindex(columns=cv_res_df_cols).sort_values(by='rank_test_score')\n \n opt_params = model_opt.best_params_\n classifier = classifier.set_params(**opt_params)\n \n final_model = classifier.fit(x_train, y_train)\n y_train_pred = final_model.predict(x_train)\n clear_output()\n \n display(Markdown('__Cross Validation Result__'))\n display(cv_res_df.head().round(4))\n \n cv_scores = cross_val_score(classifier, x_train, y_train, scoring='accuracy')\n mean_cv_score = np.mean(cv_scores)\n display(Markdown('__Mean Cross Validation Score__'))\n display(round(mean_cv_score, 4))\n \n display(Markdown('__Best Parameters__'))\n for k,v in opt_params.items():\n display(Markdown(f'__{k}__: {v}'))\n \n return final_model, y_train_pred\n\n\n# function for roc plot\ndef roc_plot(x_train, y_train, x_valid, y_valid, model, model_name):\n y_train_pred_probs = model.predict_proba(x_train)[:, 1]\n y_train_auc = roc_auc_score(y_train, y_train_pred_probs)\n yt_fpr, yt_tpr, _ = roc_curve(y_train, y_train_pred_probs, pos_label = 1)\n\n y_valid_pred_probs = model.predict_proba(x_valid)[:, 1]\n y_valid_auc = roc_auc_score(y_valid, y_valid_pred_probs)\n yv_fpr, yv_tpr, _ = roc_curve(y_valid, y_valid_pred_probs, pos_label = 1)\n\n roc_fig = make_subplots(rows=1, cols=2, subplot_titles=['Train Data', 'Validation Data'])\n\n roc_fig.add_trace(go.Scatter(x=yt_fpr, y=yt_tpr, mode='lines', \n name=f'Train AUC = {round(y_train_auc, 4)}', line_shape='linear'), row=1, col=1)\n roc_fig.add_trace(go.Scatter(x=[0,1], y=[0,1], showlegend=False,\n line=dict(dash='dash'), line_shape='linear'), row=1, col=1)\n\n roc_fig.add_trace(go.Scatter(x=yv_fpr, y=yv_tpr, mode='lines', \n name=f'Validation AUC = {round(y_valid_auc, 4)}', line_shape='linear'), row=1, col=2)\n roc_fig.add_trace(go.Scatter(x=[0,1], y=[0,1], showlegend=False,\n line=dict(dash='dash'), line_shape='linear'), row=1, col=2)\n\n roc_fig.update_layout(title=f'ROC of {model_name}', title_x=0.5)\n\n roc_fig.update_xaxes(title_text=\"False Positive Rate\", row=1, col=1)\n roc_fig.update_xaxes(title_text=\"False Positive Rate\", row=1, col=2)\n\n roc_fig.update_yaxes(title_text=\"True Positive Rate\", row=1, col=1)\n roc_fig.update_yaxes(title_text=\"True Positive Rate\", row=1, col=2)\n iplot(roc_fig)\n \n\n# function to calculate model metrics\ndef model_metric(model, y_train, y_pred_train, x_valid, y_valid, model_name):\n classes = [0,1]\n acc_train = accuracy_score(y_train, y_pred_train)*100\n class_report_train = classification_report(y_train, y_pred_train, \n target_names=classes, output_dict=True)\n y_pred_valid = model.predict(x_valid)\n acc_valid = accuracy_score(y_valid, y_pred_valid)*100 \n class_report_valid = classification_report(y_valid, y_pred_valid, \n target_names=classes, output_dict = True)\n \n display(Markdown('__Overall Statistics__'))\n display(Markdown('Accuracy on TRAIN DATA: __{}%__'. format (round(acc_train,4)))) \n display(Markdown('Accuracy on VALIDATION DATA: __{}%__'.format(round(acc_valid,4))))\n \n cr_df_train = pd.DataFrame(class_report_train).T \n cr_df_valid = pd.DataFrame(class_report_valid).T\n \n cm_train = confusion_matrix(y_train,y_pred_train)\n cm_valid = confusion_matrix(y_valid,y_pred_valid)\n \n train_avg_metric = cr_df_train[len(classes)+1:].T \n valid_avg_metric = cr_df_valid[len(classes)+1:].T\n train_avg_metric['micro avg'] = precision_recall_fscore_support(y_train, y_pred_train, average='micro')\n valid_avg_metric['micro avg'] = precision_recall_fscore_support(y_valid, y_pred_valid, average='micro')\n train_avg_metric = train_avg_metric.iloc[:-1,:]\n valid_avg_metric = valid_avg_metric.iloc[:-1,:]\n \n display(Markdown(f'_Overall Average Metrics of **{model_name}**: **TRAIN DATA**_')) \n display(train_avg_metric)\n display(Markdown(f'_Overall Average Metrics of **{model_name}**: **VALIDATION DATA**_')) \n display(valid_avg_metric)\n \n cm_fig_t = ff.create_annotated_heatmap(cm_train, x=classes, y=classes, colorscale='darkmint', showscale=True)\n cm_fig_t.update_layout(title='Confusion Matrix TRAIN DATA', xaxis_title=\"Predicted\", yaxis_title=\"Reference\",\n height=400, width=400, title_x=0.5)\n iplot(cm_fig_t)\n \n cm_fig_v = ff.create_annotated_heatmap(cm_valid, x=classes, y=classes, colorscale='darkmint', showscale=True)\n cm_fig_v.update_layout(title='Confusion Matrix VALIDATION DATA', xaxis_title=\"Predicted\", yaxis_title=\"Reference\",\n height=400, width=400, title_x=0.5)\n iplot(cm_fig_v)\n \n return acc_train, acc_valid\n\n\nmodels_acc_train, models_acc_valid = [], []","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## RandomForest","metadata":{}},{"cell_type":"code","source":"rf_params = {'n_estimators': [100, 150], \n 'max_depth': [10, 20],\n 'max_leaf_nodes': [20, 40]}\n\nrf_classifier = RandomForestClassifier(random_state=369, n_jobs=-1, class_weight='balanced')\nrf_model, rf_y_train_pred = cross_valid_result(rf_classifier, rf_params, \n df_train_final.iloc[:, final_selected_features], \n y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rf_t_acc, rf_v_acc = model_metric(rf_model, y_train_final, rf_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'Random Forest')\n\nmodels_acc_train.append(rf_t_acc)\nmodels_acc_valid.append(rf_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n rf_model, 'Random Forest')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Logistic Regression","metadata":{}},{"cell_type":"code","source":"lr_params = {'max_iter': [100, 200], \n 'tol': [1e-3, 1e-4],\n 'C': [1, 2]}\n\nlr_classifier = LogisticRegression(solver='liblinear', random_state=369)\nlr_model, lr_y_train_pred = cross_valid_result(lr_classifier, lr_params, \n df_train_final.iloc[:, final_selected_features], \n y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_t_acc, lr_v_acc = model_metric(lr_model, y_train_final, lr_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'Logistic Regression')\n\nmodels_acc_train.append(lr_t_acc)\nmodels_acc_valid.append(lr_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n lr_model, 'Logistic Regression')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Decision Tree","metadata":{}},{"cell_type":"code","source":"dt_params = {'max_depth': [50, 100],\n 'min_samples_leaf': [1, 2]}\ndt_classifier = DecisionTreeClassifier(random_state=369, class_weight='balanced', max_features='auto')\n\ndt_model, dt_y_train_pred = cross_valid_result(dt_classifier, dt_params, \n df_train_final.iloc[:, final_selected_features], \n y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt_t_acc, dt_v_acc = model_metric(dt_model, y_train_final, dt_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'Decision Tree')\n\nmodels_acc_train.append(dt_t_acc)\nmodels_acc_valid.append(dt_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n dt_model, 'Decision Tree')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## AdaBoost","metadata":{}},{"cell_type":"code","source":"ab_params = {'n_estimators': [100, 150]}\nab_classifier = AdaBoostClassifier(random_state=369)\n\nab_model, ab_y_train_pred = cross_valid_result(ab_classifier, ab_params, \n df_train_final.iloc[:, final_selected_features], \n y_train_final)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ab_t_acc, ab_v_acc = model_metric(ab_model, y_train_final, ab_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'AdaBoost')\n\nmodels_acc_train.append(ab_t_acc)\nmodels_acc_valid.append(ab_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n ab_model, 'AdaBoost')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Gaussian Naive Bayes","metadata":{}},{"cell_type":"code","source":"nb_classifier = GaussianNB()\nnb_model = nb_classifier.fit(df_train_final.iloc[:, final_selected_features], y_train_final)\nnb_y_train_pred = nb_model.predict(df_train_final.iloc[:, final_selected_features])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nb_t_acc, nb_v_acc = model_metric(nb_model, y_train_final, nb_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'Naive Bayes')\n\nmodels_acc_train.append(nb_t_acc)\nmodels_acc_valid.append(nb_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n nb_model, 'Naive Bayes')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## XGBoost","metadata":{}},{"cell_type":"code","source":"xgb_classifier = xgb.XGBClassifier(objective=\"binary:logistic\", booster='gbtree',\n random_state=369, use_label_encoder=False, n_jobs=-1)\n\nxgb_model = xgb_classifier.fit(df_train_final.iloc[:, final_selected_features], y_train_final)\nxgb_y_train_pred = xgb_model.predict(df_train_final.iloc[:, final_selected_features])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xgb_t_acc, xgb_v_acc = model_metric(xgb_model, y_train_final, xgb_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'XG Boost')\n\nmodels_acc_train.append(xgb_t_acc)\nmodels_acc_valid.append(xgb_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n xgb_model, 'XG Boost')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble Method","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import VotingClassifier\n\neclf = VotingClassifier(estimators=[('rf', rf_model), \n ('lr', lr_model), \n ('nb', nb_model),\n ('dt', dt_model),\n ('ab', ab_model),\n ('xgb', xgb_model)\n ],\n voting='soft', n_jobs=-1)\n\nens_model = eclf.fit(df_train_final.iloc[:, final_selected_features], y_train_final)\nens_y_train_pred = ens_model.predict(df_train_final.iloc[:, final_selected_features])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ens_t_acc, ens_v_acc = model_metric(ens_model, y_train_final, ens_y_train_pred, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, 'Ensembled')\n\nmodels_acc_train.append(ens_t_acc)\nmodels_acc_valid.append(ens_v_acc)\n\nroc_plot(df_train_final.iloc[:, final_selected_features], y_train_final, \n df_valid_final.iloc[:, final_selected_features], y_valid_final, \n ens_model, 'Ensembled')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Comparison","metadata":{}},{"cell_type":"code","source":"model_name_list = ['RF', 'LR', 'DT', 'AB', 'NB', 'XGB', 'Ensemble']\n\ncomp_fig = go.Figure(data=[\n go.Bar(name='Train', x=model_name_list, y=models_acc_train),\n go.Bar(name='Validation', x=model_name_list, y=models_acc_valid)\n])\n\n# Change the bar mode\ncomp_fig.update_layout(barmode='group')\ncomp_fig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we can expect the best model to vary.\n\nHowever, this is not the best version. I will continue to explore other methods and get better result - both in terms of dataset creation and model developement.\n\n# To Be Continued...","metadata":{}}]}