{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":41880,"databundleVersionId":5677426,"sourceType":"competition"}],"dockerImageVersionId":30648,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":9440.392896,"end_time":"2024-02-08T16:47:27.927581","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-02-08T14:10:07.534685","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# importing all the libraries\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport warnings\nimport os\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split\n\n#to remove warnings\nwarnings.filterwarnings(action = \"ignore\", category = DeprecationWarning ) \nwarnings.filterwarnings(action = \"ignore\", category = FutureWarning ) ","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":5.19059,"end_time":"2024-02-08T14:10:16.115669","exception":false,"start_time":"2024-02-08T14:10:10.925079","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:25:43.190652Z","iopub.execute_input":"2024-02-19T07:25:43.190925Z","iopub.status.idle":"2024-02-19T07:25:45.162250Z","shell.execute_reply.started":"2024-02-19T07:25:43.190895Z","shell.execute_reply":"2024-02-19T07:25:45.161356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Data Exploration","metadata":{"papermill":{"duration":0.008747,"end_time":"2024-02-08T14:10:16.133602","exception":false,"start_time":"2024-02-08T14:10:16.124855","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Combining all tdcsfog '.csv' train files\ntdcsfog_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/train/tdcsfog'\ntdcsfog_list = []\n\nfor file_name in os.listdir(tdcsfog_path):\n    if file_name.endswith('.csv') and file_name != '003f117e14.csv': # removing the file if same as in test set\n        file_path = os.path.join(tdcsfog_path, file_name)\n        df = pd.read_csv(file_path)\n\n        tdcsfog_list.append(df)\n\ntdcsfog = pd.concat(tdcsfog_list)\nmask = (tdcsfog['StartHesitation'] + tdcsfog['Turn'] + tdcsfog['Walking']) != 0 # condition to remove nonfog events\n\n# Append the rows that meet the condition to a new DataFrame\ntdcsfog = tdcsfog[mask].copy()\ntdcsfog\n\n#before 7062672 rows, after 2190622","metadata":{"papermill":{"duration":20.020772,"end_time":"2024-02-08T14:10:36.179585","exception":false,"start_time":"2024-02-08T14:10:16.158813","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:25:45.164274Z","iopub.execute_input":"2024-02-19T07:25:45.164784Z","iopub.status.idle":"2024-02-19T07:26:03.204989Z","shell.execute_reply.started":"2024-02-19T07:25:45.164750Z","shell.execute_reply":"2024-02-19T07:26:03.204059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Combining all defog '.csv' train files\ndefog_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/train/defog'\ndefog_list = []\n\nfor file_name in os.listdir(defog_path):\n    if file_name.endswith('.csv') and file_name != '02ab235146.csv': # removing the file if same as in test set\n        file_path = os.path.join(defog_path, file_name)\n        df = pd.read_csv(file_path)\n        defog_list.append(df)\n\ndefog = pd.concat(defog_list)\nmask = (defog['StartHesitation'] + defog['Turn'] + defog['Walking']) != 0 # condition to remove nonfog events\n\n# Append the rows that meet the condition to a new DataFrame\ndefog = defog[mask].copy()\n\ndefog\n#before 13525702 rows, after 685847","metadata":{"papermill":{"duration":27.106014,"end_time":"2024-02-08T14:11:03.295059","exception":false,"start_time":"2024-02-08T14:10:36.189045","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:03.210200Z","iopub.execute_input":"2024-02-19T07:26:03.210491Z","iopub.status.idle":"2024-02-19T07:26:28.608506Z","shell.execute_reply.started":"2024-02-19T07:26:03.210468Z","shell.execute_reply":"2024-02-19T07:26:28.607559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog = defog[(defog['Valid'] == True) & (defog['Task'] == True)] #leaving only true states\nprint(defog)","metadata":{"papermill":{"duration":0.045616,"end_time":"2024-02-08T14:11:03.354392","exception":false,"start_time":"2024-02-08T14:11:03.308776","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:28.609579Z","iopub.execute_input":"2024-02-19T07:26:28.609839Z","iopub.status.idle":"2024-02-19T07:26:28.637178Z","shell.execute_reply.started":"2024-02-19T07:26:28.609817Z","shell.execute_reply":"2024-02-19T07:26:28.636292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#dropping rows\nrowstodrop = ['Valid', 'Task'] \ndefog = defog.drop(rowstodrop, axis=1)\ndefog","metadata":{"papermill":{"duration":0.049785,"end_time":"2024-02-08T14:11:03.414739","exception":false,"start_time":"2024-02-08T14:11:03.364954","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:28.638578Z","iopub.execute_input":"2024-02-19T07:26:28.638931Z","iopub.status.idle":"2024-02-19T07:26:28.668802Z","shell.execute_reply.started":"2024-02-19T07:26:28.638899Z","shell.execute_reply":"2024-02-19T07:26:28.667810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Data Exploration","metadata":{"papermill":{"duration":0.013133,"end_time":"2024-02-08T14:11:03.441718","exception":false,"start_time":"2024-02-08T14:11:03.428585","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#do the bar plot for tdcsfog\n\n# columns to analyze\nselected_columns = ['StartHesitation', 'Turn', 'Walking']\n\n# Count occurrences of 1 in each row for the specified columns\ncounts = tdcsfog[selected_columns].sum()\n\npercentages = (counts / counts.sum()) * 100\n\n# Plotting\ny_rows = selected_columns\nbars = plt.bar(y_rows, counts)\nplt.xlabel('Activity')\nplt.ylabel('Count of 1s')\nplt.title('Count of 1s for Each Activity')\n\n# Annotate each bar with its percentage value\nfor bar, percentage in zip(bars, percentages):\n    plt.text(bar.get_x() + bar.get_width() / 2 - 0.15, bar.get_height() + 0.5,\n             f'{percentage:.2f}%', ha='center', va='bottom', color='black')\n\nplt.show()\n\npercentages_all = (counts / len(tdcsfog)) * 100\nprint(percentages_all) #within all","metadata":{"papermill":{"duration":0.363686,"end_time":"2024-02-08T14:11:03.816009","exception":false,"start_time":"2024-02-08T14:11:03.452323","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:28.670034Z","iopub.execute_input":"2024-02-19T07:26:28.670305Z","iopub.status.idle":"2024-02-19T07:26:28.904990Z","shell.execute_reply.started":"2024-02-19T07:26:28.670282Z","shell.execute_reply":"2024-02-19T07:26:28.904072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#do the bar plot for defog\n\n# Count occurrences of 1 in each row for the specified columns\ncounts = defog[selected_columns].sum()\n\npercentages = (counts / counts.sum()) * 100 #within all 1s\n\n# Plotting\ny_rows = selected_columns\nbars = plt.bar(y_rows, counts)\nplt.xlabel('Activity')\nplt.ylabel('Count of 1s')\nplt.title('Count of 1s for Each Activity')\n\n# Annotate each bar with its percentage value\nfor bar, percentage in zip(bars, percentages):\n    plt.text(bar.get_x() + bar.get_width() / 2 - 0.15, bar.get_height() + 0.5,\n             f'{percentage:.2f}%', ha='center', va='bottom', color='black')\n\nplt.show()\n\npercentages_all = (counts / len(defog)) * 100\nprint(percentages_all) #within all","metadata":{"papermill":{"duration":0.277739,"end_time":"2024-02-08T14:11:04.105018","exception":false,"start_time":"2024-02-08T14:11:03.827279","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:28.906334Z","iopub.execute_input":"2024-02-19T07:26:28.906778Z","iopub.status.idle":"2024-02-19T07:26:29.087916Z","shell.execute_reply.started":"2024-02-19T07:26:28.906745Z","shell.execute_reply":"2024-02-19T07:26:29.086781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Split Data in Train / Test Set","metadata":{"papermill":{"duration":0.010608,"end_time":"2024-02-08T14:11:04.127527","exception":false,"start_time":"2024-02-08T14:11:04.116919","status":"completed"},"tags":[]}},{"cell_type":"code","source":"x_rows = ['AccV', 'AccML','AccAP']\ny_rows = ['StartHesitation', 'Turn', 'Walking']","metadata":{"papermill":{"duration":0.019377,"end_time":"2024-02-08T14:11:04.157845","exception":false,"start_time":"2024-02-08T14:11:04.138468","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:29.089126Z","iopub.execute_input":"2024-02-19T07:26:29.089423Z","iopub.status.idle":"2024-02-19T07:26:29.093686Z","shell.execute_reply.started":"2024-02-19T07:26:29.089399Z","shell.execute_reply":"2024-02-19T07:26:29.092708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from imblearn.under_sampling import RandomUnderSampler","metadata":{"papermill":{"duration":0.154187,"end_time":"2024-02-08T14:11:04.322692","exception":false,"start_time":"2024-02-08T14:11:04.168505","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:29.094843Z","iopub.execute_input":"2024-02-19T07:26:29.095137Z","iopub.status.idle":"2024-02-19T07:26:29.402686Z","shell.execute_reply.started":"2024-02-19T07:26:29.095113Z","shell.execute_reply":"2024-02-19T07:26:29.401585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# splitting tdcsfog\nX_tdcsfog = tdcsfog[x_rows]\ny_tdcsfog = tdcsfog[y_rows]\nX_tdcsfog_train, X_tdcsfog_test, y_tdcsfog_train, y_tdcsfog_test = train_test_split(X_tdcsfog, y_tdcsfog, test_size=0.2, random_state=42, stratify=y_tdcsfog)\n\nprint('done')","metadata":{"papermill":{"duration":21.375741,"end_time":"2024-02-08T14:11:25.709539","exception":false,"start_time":"2024-02-08T14:11:04.333798","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:29.407860Z","iopub.execute_input":"2024-02-19T07:26:29.408422Z","iopub.status.idle":"2024-02-19T07:26:45.167671Z","shell.execute_reply.started":"2024-02-19T07:26:29.408396Z","shell.execute_reply":"2024-02-19T07:26:45.166702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#splitting defog\nX_defog = defog[x_rows]\ny_defog = defog[y_rows]\nX_defog_train, X_defog_test, y_defog_train, y_defog_test = train_test_split(X_defog, y_defog, test_size=0.2, random_state=42, stratify=y_defog)\n\nprint('done')","metadata":{"papermill":{"duration":6.530837,"end_time":"2024-02-08T14:11:32.251360","exception":false,"start_time":"2024-02-08T14:11:25.720523","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:45.168989Z","iopub.execute_input":"2024-02-19T07:26:45.169356Z","iopub.status.idle":"2024-02-19T07:26:50.181607Z","shell.execute_reply.started":"2024-02-19T07:26:45.169323Z","shell.execute_reply":"2024-02-19T07:26:50.180599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# reshaping and undersampling data to deal with imbalanced dataset\nX_train1_reshaped = X_tdcsfog_train.values\nX_test1_reshaped = X_tdcsfog_test.values\n\ny_train1_np = y_tdcsfog_train.values\ny_test1_np = y_tdcsfog_test.values\n\nundersampler = RandomUnderSampler(sampling_strategy='all', random_state=42)\nX_train1_reshaped, y_train1_np = undersampler.fit_resample(X_train1_reshaped, y_train1_np)\n\n# Reshape NumPy arrays\nX_train1_reshaped = X_train1_reshaped.reshape((X_train1_reshaped.shape[0], 1, X_train1_reshaped.shape[1]))\nX_test1_reshaped = X_test1_reshaped.reshape((X_tdcsfog_test.shape[0], 1, X_tdcsfog_test.shape[1]))\ny_train1_np_reshaped = y_train1_np.reshape((y_train1_np.shape[0], 1, y_train1_np.shape[1]))\ny_test1_np_reshaped = y_test1_np.reshape((y_test1_np.shape[0], 1, y_test1_np.shape[1]))\nprint('done')","metadata":{"papermill":{"duration":0.774924,"end_time":"2024-02-08T14:11:33.037299","exception":false,"start_time":"2024-02-08T14:11:32.262375","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:50.183057Z","iopub.execute_input":"2024-02-19T07:26:50.183512Z","iopub.status.idle":"2024-02-19T07:26:51.380453Z","shell.execute_reply.started":"2024-02-19T07:26:50.183478Z","shell.execute_reply":"2024-02-19T07:26:51.379455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# reshaping and undersampling data to deal with imbalanced dataset\nX_train2_reshaped = X_defog_train.values\nX_test2_reshaped = X_defog_test.values\n\ny_train2_np = y_defog_train.values\ny_test2_np = y_defog_test.values\n\nundersampler = RandomUnderSampler(sampling_strategy='all', random_state=42)\nX_train2_reshaped, y_train2_np = undersampler.fit_resample(X_train2_reshaped, y_train2_np)\n\n# Reshape NumPy arrays\nX_train2_reshaped = X_train2_reshaped.reshape((X_train2_reshaped.shape[0], 1, X_train2_reshaped.shape[1]))\nX_test2_reshaped = X_test2_reshaped.reshape((X_defog_test.shape[0], 1, X_defog_test.shape[1]))\ny_train2_np_reshaped = y_train2_np.reshape((y_train2_np.shape[0], 1, y_train2_np.shape[1]))\ny_test2_np_reshaped = y_test2_np.reshape((y_test2_np.shape[0], 1, y_test2_np.shape[1]))\nprint('done')","metadata":{"papermill":{"duration":0.205103,"end_time":"2024-02-08T14:11:33.253613","exception":false,"start_time":"2024-02-08T14:11:33.048510","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:51.381681Z","iopub.execute_input":"2024-02-19T07:26:51.381992Z","iopub.status.idle":"2024-02-19T07:26:51.589688Z","shell.execute_reply.started":"2024-02-19T07:26:51.381966Z","shell.execute_reply":"2024-02-19T07:26:51.588575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def precision(y_true, y_pred):\n    true_positives = tf.keras.backend.sum(tf.keras.backend.round(tf.keras.backend.clip(y_true * y_pred, 0, 1)))\n    predicted_positives = tf.keras.backend.sum(tf.keras.backend.round(tf.keras.backend.clip(y_pred, 0, 1)))\n    precision = true_positives / (predicted_positives + tf.keras.backend.epsilon())\n    return precision","metadata":{"execution":{"iopub.status.busy":"2024-02-19T07:26:51.590978Z","iopub.execute_input":"2024-02-19T07:26:51.591301Z","iopub.status.idle":"2024-02-19T07:26:51.598049Z","shell.execute_reply.started":"2024-02-19T07:26:51.591273Z","shell.execute_reply":"2024-02-19T07:26:51.597002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers, models\n\n# Create the input layer\ninputs = tf.keras.Input(shape=(1, 3))\n\n# Model 1\nx1 = layers.LSTM(64, return_sequences=True)(inputs)\nx1 = layers.Dropout(0.2)(x1)\nx1 = layers.LSTM(64, return_sequences=True)(x1)\nx1 = layers.Dropout(0.2)(x1)\nx1 = layers.LSTM(64, return_sequences=True)(x1)\nx1 = layers.Dropout(0.2)(x1)\nx1 = layers.LSTM(64, return_sequences=True)(x1)\nx1 = layers.Dropout(0.2)(x1)\noutputs1 = layers.Dense(3, activation='softmax')(x1)\nmodel1 = tf.keras.Model(inputs=inputs, outputs=outputs1)\nmodel1.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[precision])\nmodel1.summary()\n\n# Model 2\nx2 = layers.LSTM(64, return_sequences=True)(inputs)\nx2 = layers.Dropout(0.2)(x2)\nx2 = layers.LSTM(64, return_sequences=True)(x2)\nx2 = layers.Dropout(0.2)(x2)\nx2 = layers.LSTM(64, return_sequences=True)(x2)\nx2 = layers.Dropout(0.2)(x2)\nx2 = layers.LSTM(64, return_sequences=True)(x2)\nx2 = layers.Dropout(0.2)(x2)\noutputs2 = layers.Dense(3, activation='softmax')(x2)\nmodel2 = tf.keras.Model(inputs=inputs, outputs=outputs2)\nmodel2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[precision])\nmodel2.summary()\n\nmodel1.fit(X_train1_reshaped, y_train1_np_reshaped, epochs=50, validation_data=(X_test1_reshaped, y_test1_np_reshaped))\nmodel2.fit(X_train2_reshaped, y_train2_np_reshaped, epochs=50, validation_data=(X_test2_reshaped, y_test2_np_reshaped))\nprint('done')","metadata":{"papermill":{"duration":9001.724195,"end_time":"2024-02-08T16:41:34.988950","exception":false,"start_time":"2024-02-08T14:11:33.264755","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:26:51.599177Z","iopub.execute_input":"2024-02-19T07:26:51.599578Z","iopub.status.idle":"2024-02-19T07:27:30.152999Z","shell.execute_reply.started":"2024-02-19T07:26:51.599485Z","shell.execute_reply":"2024-02-19T07:27:30.151581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# opening and preparing the test data set\n\ntdcsfog_test_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/tdcsfog'\ntdcsfog_test_list = []\n\nfor file_name in os.listdir(tdcsfog_test_path):\n    if file_name.endswith('.csv'):\n        file_path = os.path.join(tdcsfog_test_path, file_name)\n        file = pd.read_csv(file_path)\n        file['Id'] = file_name[:-4] + '_' + file['Time'].apply(str)\n        file.Time = file.Time / (len(file) - 1)\n        tdcsfog_test_list.append(file)\n\ntdcsfog_test = pd.concat(tdcsfog_test_list, axis=0)\n\nX_tdcsfog_test = tdcsfog_test[x_rows]\n\nX_test_reshaped_tdcsfog = X_tdcsfog_test.values.reshape((X_tdcsfog_test.shape[0], 1, X_tdcsfog_test.shape[1]))\n\n# predicting with model\nprobabilities_1 = model1.predict(X_test_reshaped_tdcsfog)\n\nX_tdcsfog_test['StartHesitation'] = probabilities_1[:, 0, 0]\nX_tdcsfog_test['Turn'] = probabilities_1[:, 0, 1]\nX_tdcsfog_test['Walking'] = probabilities_1[:, 0, 2]","metadata":{"papermill":{"duration":11.133439,"end_time":"2024-02-08T16:42:37.154957","exception":false,"start_time":"2024-02-08T16:42:26.021518","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.153955Z","iopub.status.idle":"2024-02-19T07:27:30.154327Z","shell.execute_reply.started":"2024-02-19T07:27:30.154141Z","shell.execute_reply":"2024-02-19T07:27:30.154155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# opening and preparing the test data set\n\ndefog_test_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog'\ndefog_test_list = []\n\nfor file_name in os.listdir(defog_test_path):\n    if file_name.endswith('.csv'):\n        file_path = os.path.join(defog_test_path, file_name)\n        file = pd.read_csv(file_path)\n        file['Id'] = file_name[:-4] + '_' + file['Time'].apply(str)\n        file.Time = file.Time / (len(file) - 1)\n        defog_test_list.append(file)\n\ndefog_test = pd.concat(defog_test_list, axis=0)\n\nX_defog_test = defog_test[x_rows]\n\nX_test_reshaped_defog = X_defog_test.values.reshape((X_defog_test.shape[0], 1, X_defog_test.shape[1]))\n\n# predicting with model\nprobabilities_2 = model2.predict(X_test_reshaped_defog)\n\nX_defog_test['StartHesitation'] = probabilities_2[:, 0, 0]\nX_defog_test['Turn'] = probabilities_2[:, 0, 1]\nX_defog_test['Walking'] = probabilities_2[:, 0, 2]","metadata":{"papermill":{"duration":44.915547,"end_time":"2024-02-08T16:43:31.118533","exception":false,"start_time":"2024-02-08T16:42:46.202986","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.155738Z","iopub.status.idle":"2024-02-19T07:27:30.156075Z","shell.execute_reply.started":"2024-02-19T07:27:30.155911Z","shell.execute_reply":"2024-02-19T07:27:30.155925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_tdcsfog_test","metadata":{"papermill":{"duration":9.092264,"end_time":"2024-02-08T16:44:08.115562","exception":false,"start_time":"2024-02-08T16:43:59.023298","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.157729Z","iopub.status.idle":"2024-02-19T07:27:30.158039Z","shell.execute_reply.started":"2024-02-19T07:27:30.157886Z","shell.execute_reply":"2024-02-19T07:27:30.157899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_defog_test","metadata":{"papermill":{"duration":9.554611,"end_time":"2024-02-08T16:44:26.938522","exception":false,"start_time":"2024-02-08T16:44:17.383911","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.159617Z","iopub.status.idle":"2024-02-19T07:27:30.159930Z","shell.execute_reply.started":"2024-02-19T07:27:30.159776Z","shell.execute_reply":"2024-02-19T07:27:30.159788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# putting id column\ndefog_id = defog_test.iloc[:]['Id']\nX_defog_test['Id'] = defog_test['Id']\nprint('done')","metadata":{"papermill":{"duration":9.29327,"end_time":"2024-02-08T16:44:45.556382","exception":false,"start_time":"2024-02-08T16:44:36.263112","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.161242Z","iopub.status.idle":"2024-02-19T07:27:30.161587Z","shell.execute_reply.started":"2024-02-19T07:27:30.161404Z","shell.execute_reply":"2024-02-19T07:27:30.161417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# putting id column\ntdcsfog_id = tdcsfog_test['Id']\nX_tdcsfog_test.insert(0, 'Id', tdcsfog_id)\nprint('done')","metadata":{"papermill":{"duration":9.074989,"end_time":"2024-02-08T16:45:03.887197","exception":false,"start_time":"2024-02-08T16:44:54.812208","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.162769Z","iopub.status.idle":"2024-02-19T07:27:30.163098Z","shell.execute_reply.started":"2024-02-19T07:27:30.162938Z","shell.execute_reply":"2024-02-19T07:27:30.162952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Concatenate the data frames\nconcatenated_dataframe = pd.concat([X_tdcsfog_test, X_defog_test], axis=0)\n\n# Display the resulting concatenated DataFrame\nprint(concatenated_dataframe)","metadata":{"papermill":{"duration":9.459186,"end_time":"2024-02-08T16:45:22.498186","exception":false,"start_time":"2024-02-08T16:45:13.039000","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.164334Z","iopub.status.idle":"2024-02-19T07:27:30.164676Z","shell.execute_reply.started":"2024-02-19T07:27:30.164487Z","shell.execute_reply":"2024-02-19T07:27:30.164499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"concatenated_dataframe","metadata":{"papermill":{"duration":9.267567,"end_time":"2024-02-08T16:45:40.990744","exception":false,"start_time":"2024-02-08T16:45:31.723177","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.165679Z","iopub.status.idle":"2024-02-19T07:27:30.166006Z","shell.execute_reply.started":"2024-02-19T07:27:30.165846Z","shell.execute_reply":"2024-02-19T07:27:30.165860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#removing x_rows and preparing submission file\nx_rows = ['AccV', 'AccML', 'AccAP']\nconcatenated_dataframe = concatenated_dataframe.drop(x_rows, axis=1)\nprint('done')","metadata":{"papermill":{"duration":9.107541,"end_time":"2024-02-08T16:45:59.328833","exception":false,"start_time":"2024-02-08T16:45:50.221292","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.168208Z","iopub.status.idle":"2024-02-19T07:27:30.168677Z","shell.execute_reply.started":"2024-02-19T07:27:30.168423Z","shell.execute_reply":"2024-02-19T07:27:30.168442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"concatenated_dataframe","metadata":{"papermill":{"duration":9.33997,"end_time":"2024-02-08T16:46:17.924589","exception":false,"start_time":"2024-02-08T16:46:08.584619","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.169811Z","iopub.status.idle":"2024-02-19T07:27:30.170259Z","shell.execute_reply.started":"2024-02-19T07:27:30.170029Z","shell.execute_reply":"2024-02-19T07:27:30.170048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"concatenated_dataframe.describe()","metadata":{"papermill":{"duration":9.334094,"end_time":"2024-02-08T16:46:36.680455","exception":false,"start_time":"2024-02-08T16:46:27.346361","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.171924Z","iopub.status.idle":"2024-02-19T07:27:30.172263Z","shell.execute_reply.started":"2024-02-19T07:27:30.172098Z","shell.execute_reply":"2024-02-19T07:27:30.172113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = concatenated_dataframe\nsubmission","metadata":{"papermill":{"duration":9.240373,"end_time":"2024-02-08T16:46:54.985643","exception":false,"start_time":"2024-02-08T16:46:45.745270","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.173156Z","iopub.status.idle":"2024-02-19T07:27:30.173482Z","shell.execute_reply.started":"2024-02-19T07:27:30.173322Z","shell.execute_reply":"2024-02-19T07:27:30.173335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index = False)","metadata":{"papermill":{"duration":10.845956,"end_time":"2024-02-08T16:47:15.160938","exception":false,"start_time":"2024-02-08T16:47:04.314982","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-19T07:27:30.174892Z","iopub.status.idle":"2024-02-19T07:27:30.175229Z","shell.execute_reply.started":"2024-02-19T07:27:30.175063Z","shell.execute_reply":"2024-02-19T07:27:30.175077Z"},"trusted":true},"execution_count":null,"outputs":[]}]}