{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nfrom copy import deepcopy\nimport glob\nimport matplotlib.pyplot as plt\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\n\nfrom sklearn.metrics import average_precision_score, accuracy_score\nfrom sklearn.model_selection import train_test_split, RandomizedSearchCV\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-30T08:52:00.877610Z","iopub.execute_input":"2023-05-30T08:52:00.878024Z","iopub.status.idle":"2023-05-30T08:52:02.547861Z","shell.execute_reply.started":"2023-05-30T08:52:00.877994Z","shell.execute_reply":"2023-05-30T08:52:02.546673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Let's see what we can do...","metadata":{}},{"cell_type":"markdown","source":"**Short Agenda**  \n\nFirst of all we are going to put all data from DeFoG and tdcs_FoG train folders in one DataFrame with meta info.  \nThe next step is to check the feature correlation. Here we try to find some interesting relationships.  \nPrepare predictive model.","metadata":{}},{"cell_type":"markdown","source":"## \"Cooking\" single DataFrame of defog/tdcsfog datasets with meta","metadata":{}},{"cell_type":"code","source":"## - these double symbols mean I skip this code to try to get inside 9 hour limit","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.549889Z","iopub.execute_input":"2023-05-30T08:52:02.550356Z","iopub.status.idle":"2023-05-30T08:52:02.557212Z","shell.execute_reply.started":"2023-05-30T08:52:02.550312Z","shell.execute_reply":"2023-05-30T08:52:02.555843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create variable with data directory for easy operating in future  \n\ndata_directory = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/'","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.558515Z","iopub.execute_input":"2023-05-30T08:52:02.558928Z","iopub.status.idle":"2023-05-30T08:52:02.571775Z","shell.execute_reply.started":"2023-05-30T08:52:02.558897Z","shell.execute_reply":"2023-05-30T08:52:02.570494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read and save meta info in DataFrames\n\ndefog_meta = pd.read_csv(data_directory + 'defog_metadata.csv')\ntdcsfog_meta = pd.read_csv(data_directory + 'tdcsfog_metadata.csv')","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.574326Z","iopub.execute_input":"2023-05-30T08:52:02.574774Z","iopub.status.idle":"2023-05-30T08:52:02.609598Z","shell.execute_reply.started":"2023-05-30T08:52:02.574740Z","shell.execute_reply":"2023-05-30T08:52:02.608641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creat function to display main info about datasets\n\ndef main_info (dataset):\n    info = dataset.info()\n    describe = dataset.describe()\n    return print(info,\"\\n\"*2, describe,\"\\n\"*2)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.612363Z","iopub.execute_input":"2023-05-30T08:52:02.613095Z","iopub.status.idle":"2023-05-30T08:52:02.618469Z","shell.execute_reply.started":"2023-05-30T08:52:02.613058Z","shell.execute_reply":"2023-05-30T08:52:02.617632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Display 5 rows of each DataFrame and their main info.","metadata":{}},{"cell_type":"code","source":"for dataset in [defog_meta, tdcsfog_meta]:\n    print(f'main information:')\n    main_info(dataset)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.620145Z","iopub.execute_input":"2023-05-30T08:52:02.620828Z","iopub.status.idle":"2023-05-30T08:52:02.687384Z","shell.execute_reply.started":"2023-05-30T08:52:02.620798Z","shell.execute_reply":"2023-05-30T08:52:02.686018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_meta.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.689417Z","iopub.execute_input":"2023-05-30T08:52:02.689767Z","iopub.status.idle":"2023-05-30T08:52:02.711160Z","shell.execute_reply.started":"2023-05-30T08:52:02.689737Z","shell.execute_reply":"2023-05-30T08:52:02.709922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_meta.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.713095Z","iopub.execute_input":"2023-05-30T08:52:02.713538Z","iopub.status.idle":"2023-05-30T08:52:02.726596Z","shell.execute_reply.started":"2023-05-30T08:52:02.713496Z","shell.execute_reply":"2023-05-30T08:52:02.725163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# defog/tdcsfog full info func\n\ndef prepare_fog_table (df_type):\n    full_data = pd.DataFrame()\n    subdatas = glob.glob(data_directory + f'train/{df_type}/*')\n    \n    for subdata in subdatas:\n        sub_data = pd.read_csv(subdata)\n        sub_data['Id'] = subdata.split(sep='/')[-1].split(sep='.')[0]\n        \n        if df_type == 'defog':\n            sub_data['Visit']      = (defog_meta\n                                      .loc[defog_meta['Id'] == subdata.split(sep='/')[-1]\\\n                                           .split(sep='.')[0], 'Visit']\n                                      .to_list()[0])\n            ##sub_data['Medication'] = (defog_meta\n            ##                          .loc[defog_meta['Id'] == subdata.split(sep='/')[-1]\\\n            ##                               .split(sep='.')[0], 'Medication']\n            ##                          .to_list()[0])\n            ##sub_data['Subject']    = (defog_meta\n            ##                          .loc[defog_meta['Id'] == subdata.split(sep='/')[-1]\\\n            ##                               .split(sep='.')[0], 'Subject']\n            ##                          .to_list()[0])\n            ##sub_data['Test']       = 0\n            ##sub_data['Type']       = df_type\n            full_data              = pd.concat([full_data, sub_data]).reset_index(drop=True)\n            \n        else:\n            sub_data['Visit']      = (tdcsfog_meta\n                                      .loc[tdcsfog_meta['Id'] == subdata.split(sep='/')[-1]\\\n                                           .split(sep='.')[0], 'Visit']\n                                      .to_list()[0])\n            ##sub_data['Medication'] = (tdcsfog_meta\n            ##                          .loc[tdcsfog_meta['Id'] == subdata.split(sep='/')[-1]\\\n            ##                               .split(sep='.')[0], 'Medication']\n            ##                          .to_list()[0])\n            ##sub_data['Subject']    = (tdcsfog_meta\n            ##                          .loc[tdcsfog_meta['Id'] == subdata.split(sep='/')[-1]\\\n            ##                               .split(sep='.')[0], 'Subject']\n            ##                          .to_list()[0])\n            ##sub_data['Test']       = (tdcsfog_meta\n            ##                          .loc[tdcsfog_meta['Id'] == subdata.split(sep='/')[-1]\\\n            ##                               .split(sep='.')[0], 'Test']\n            ##                          .to_list()[0])\n            ##sub_data['Type']       = df_type\n            full_data              = pd.concat([full_data, sub_data]).reset_index(drop=True)\n            \n    return full_data","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.728301Z","iopub.execute_input":"2023-05-30T08:52:02.728775Z","iopub.status.idle":"2023-05-30T08:52:02.745339Z","shell.execute_reply.started":"2023-05-30T08:52:02.728734Z","shell.execute_reply":"2023-05-30T08:52:02.744174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creat defog full DataFrame and display result\n\ndefog_full_table = prepare_fog_table('defog')\n##defog_full_table","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:52:02.750177Z","iopub.execute_input":"2023-05-30T08:52:02.750573Z","iopub.status.idle":"2023-05-30T08:53:53.750194Z","shell.execute_reply.started":"2023-05-30T08:52:02.750528Z","shell.execute_reply":"2023-05-30T08:53:53.748976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creat tdcsfog full DataFrame and display result\n\ntdcsfog_full_table = prepare_fog_table('tdcsfog')\n##tdcsfog_full_table","metadata":{"execution":{"iopub.status.busy":"2023-05-30T08:53:53.751860Z","iopub.execute_input":"2023-05-30T08:53:53.752245Z","iopub.status.idle":"2023-05-30T09:00:00.572825Z","shell.execute_reply.started":"2023-05-30T08:53:53.752214Z","shell.execute_reply":"2023-05-30T09:00:00.571122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop rows with Valid/Task == False in defog data\n\ndefog_full_table = (defog_full_table\n                    .loc[(defog_full_table.Valid == True)&(defog_full_table.Task == True)]\n                    .drop(['Valid','Task'], axis=1)\n                    .reset_index(drop=True)\n                   )","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:00.574697Z","iopub.execute_input":"2023-05-30T09:00:00.575476Z","iopub.status.idle":"2023-05-30T09:00:01.209470Z","shell.execute_reply.started":"2023-05-30T09:00:00.575434Z","shell.execute_reply":"2023-05-30T09:00:01.208160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creat a single DataFrame for both datasets\n\nmega_data = (pd.concat([defog_full_table,\n                       tdcsfog_full_table])\n             .reset_index(drop=True)\n            )","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:01.211004Z","iopub.execute_input":"2023-05-30T09:00:01.211372Z","iopub.status.idle":"2023-05-30T09:00:02.624517Z","shell.execute_reply.started":"2023-05-30T09:00:01.211341Z","shell.execute_reply":"2023-05-30T09:00:02.623336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the result\n\n##mega_data.head(), \\\n##mega_data.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:02.625795Z","iopub.execute_input":"2023-05-30T09:00:02.626130Z","iopub.status.idle":"2023-05-30T09:00:02.630985Z","shell.execute_reply.started":"2023-05-30T09:00:02.626101Z","shell.execute_reply":"2023-05-30T09:00:02.629832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# digitize the data of the medication column\n\nmega_data = mega_data.replace({'on':1, 'off':0})","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:02.632923Z","iopub.execute_input":"2023-05-30T09:00:02.633654Z","iopub.status.idle":"2023-05-30T09:00:08.258355Z","shell.execute_reply.started":"2023-05-30T09:00:02.633614Z","shell.execute_reply":"2023-05-30T09:00:08.257232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Chek missing values\n\n##mega_data.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:08.259690Z","iopub.execute_input":"2023-05-30T09:00:08.260028Z","iopub.status.idle":"2023-05-30T09:00:08.264746Z","shell.execute_reply.started":"2023-05-30T09:00:08.260000Z","shell.execute_reply":"2023-05-30T09:00:08.263518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Chek duplicates\n\n##mega_data.duplicated().sum()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:08.266278Z","iopub.execute_input":"2023-05-30T09:00:08.266624Z","iopub.status.idle":"2023-05-30T09:00:08.284564Z","shell.execute_reply.started":"2023-05-30T09:00:08.266594Z","shell.execute_reply":"2023-05-30T09:00:08.283481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is better to reduce the memory usage. Thx to ARJANGROEN.","metadata":{}},{"cell_type":"code","source":"def reduce_memory_usage(df):\n    \n    start_mem = df.memory_usage().sum() / 1024 ** 2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype.name\n        if ((col_type != 'datetime64[ns]') & (col_type != 'category')):\n            if (col_type != 'object'):\n                c_min = df[col].min()\n                c_max = df[col].max()\n\n                if str(col_type)[:3] == 'int':\n                    if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                        df[col] = df[col].astype(np.int8)\n                    elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                        df[col] = df[col].astype(np.int16)\n                    elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                        df[col] = df[col].astype(np.int32)\n                    elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                        df[col] = df[col].astype(np.int64)\n\n                else:\n                    if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                        df[col] = df[col].astype(np.float16)\n                    elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                        df[col] = df[col].astype(np.float32)\n                    else:\n                        pass\n            else:\n                df[col] = df[col].astype('category')\n    mem_usg = df.memory_usage().sum() / 1024 ** 2 \n    print(\"Memory usage became: \",mem_usg,\" MB\")\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:08.286221Z","iopub.execute_input":"2023-05-30T09:00:08.286669Z","iopub.status.idle":"2023-05-30T09:00:08.301596Z","shell.execute_reply.started":"2023-05-30T09:00:08.286639Z","shell.execute_reply":"2023-05-30T09:00:08.300209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mega_data = reduce_memory_usage(mega_data)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:08.302902Z","iopub.execute_input":"2023-05-30T09:00:08.303253Z","iopub.status.idle":"2023-05-30T09:00:10.561541Z","shell.execute_reply.started":"2023-05-30T09:00:08.303225Z","shell.execute_reply":"2023-05-30T09:00:10.560358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check the feature correlation","metadata":{}},{"cell_type":"code","source":"# visualize pairwise correlation of columns\n\nnumeric_columns = ['Time', \n                   'AccV', \n                   'AccML', \n                   'AccAP',\n                   'StartHesitation', \n                   'Turn', \n                   'Walking', \n                   'Visit', \n                   ##'Test', \n                   ##'Medication'\n                   ]\nplt.figure(figsize=(8,8))\nsns.heatmap(mega_data[numeric_columns].corr(), annot=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:10.562971Z","iopub.execute_input":"2023-05-30T09:00:10.563283Z","iopub.status.idle":"2023-05-30T09:00:13.902681Z","shell.execute_reply.started":"2023-05-30T09:00:10.563255Z","shell.execute_reply":"2023-05-30T09:00:13.901429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see unique values of 'visit'\n\nmega_data.Visit.unique()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:13.904517Z","iopub.execute_input":"2023-05-30T09:00:13.904934Z","iopub.status.idle":"2023-05-30T09:00:13.974821Z","shell.execute_reply.started":"2023-05-30T09:00:13.904896Z","shell.execute_reply":"2023-05-30T09:00:13.973624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dislplay relitionship between number of visits and FoG issues.","metadata":{}},{"cell_type":"code","source":"((mega_data.groupby('Visit')[['StartHesitation','Turn','Walking']]\n .mean()\n .round(4)*100)\n .plot(kind='bar',\n      title='FoG vs Visits',\n      xlabel='number of visits',\n      ylabel='case_1/total_cases ratio',\n      grid=True,\n      figsize=(8,6)))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:13.976402Z","iopub.execute_input":"2023-05-30T09:00:13.977040Z","iopub.status.idle":"2023-05-30T09:00:14.857011Z","shell.execute_reply.started":"2023-05-30T09:00:13.977001Z","shell.execute_reply":"2023-05-30T09:00:14.855706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Expected to see another picture... \n\nIn our case, a greater number of visits corresponds to a greater frequency for all types of FoG.","metadata":{}},{"cell_type":"markdown","source":"## Create predictive models","metadata":{}},{"cell_type":"markdown","source":"According results from previus section we are going to:  \n- fit the model to predict Turn-FoG on data related to visits 4 and 5;  \n- fit the model to predict StartHesitation-FoG on data related to 13 visits;  \n- fit the model to predict Walking-FoG on data related to 20 visits.  \n\nFeatures for Turn-FoG and StartHesitation-FoG models are:\n- ['AccV','AccML','AccAP']  \n\nFeatures for Walking-FoG model and StartHesitation-FoG are:  \n- ['Time','AccV','AccML','AccAP']","metadata":{}},{"cell_type":"markdown","source":"### Turn_FoG","metadata":{}},{"cell_type":"code","source":"# create train-test dataframe for Turn-FoG\n\nturn_fog_data = (mega_data.loc[(mega_data.Visit == 4)|(mega_data.Visit == 5)]\n                 .reset_index(drop=True))","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:14.858666Z","iopub.execute_input":"2023-05-30T09:00:14.859791Z","iopub.status.idle":"2023-05-30T09:00:14.981898Z","shell.execute_reply.started":"2023-05-30T09:00:14.859747Z","shell.execute_reply":"2023-05-30T09:00:14.980616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# feature's and target's sets\n\nfeature_turn = turn_fog_data[['AccV','AccML','AccAP']]\n\ntarget_turn = turn_fog_data.Turn\n\nturn_fog_data.shape, feature_turn.shape, target_turn.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:14.983219Z","iopub.execute_input":"2023-05-30T09:00:14.983596Z","iopub.status.idle":"2023-05-30T09:00:14.997044Z","shell.execute_reply.started":"2023-05-30T09:00:14.983540Z","shell.execute_reply":"2023-05-30T09:00:14.995913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare data for train/valid\n\nX_train_turn, X_valid_turn, y_train_turn, y_valid_turn = (\n    train_test_split(feature_turn, target_turn, random_state=13))\n\nX_train_turn.shape, X_valid_turn.shape,\\\ny_train_turn.shape, y_valid_turn.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:14.998315Z","iopub.execute_input":"2023-05-30T09:00:14.998669Z","iopub.status.idle":"2023-05-30T09:00:15.210013Z","shell.execute_reply.started":"2023-05-30T09:00:14.998641Z","shell.execute_reply":"2023-05-30T09:00:15.208834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I understand that using Random Forest in this case is not good idea, but unfortunately this is the best I know yet ;).","metadata":{}},{"cell_type":"code","source":"##%%time\n# RandomForest\n\n# Parameters\n##param_grid = {'clf__max_depth': range(2,6,1)}\n\n# Create a decision tree classifier\n##model = Pipeline([('scaler', StandardScaler()), \n##                 ('clf', RandomForestClassifier(random_state=13,\n##                                                min_samples_leaf = 5,\n##                                                min_samples_split = 7,\n##                                                class_weight='balanced'))])\n# Create the random search object\n##random_search = RandomizedSearchCV(model, \n##                                   param_grid, \n##                                   cv=5, \n##                                   scoring='average_precision',\n##                                   random_state=13)\n\n# Fit the random search to the data\n##random_search.fit(X_train_turn, y_train_turn)\n                \n# Print the best parameters found\n##print(random_search.best_params_, random_search.best_score_)\n\n##model_rf_turn = deepcopy(random_search.best_estimator_)\n#model_rf_turn.set_params(clf__n_estimators=200)\n\n##predicted_valid_turn = model_rf_turn.predict(X_valid_turn)\n\n##aps = average_precision_score(\n##    y_valid_turn, \n##    predicted_valid_turn)\n\n##acc = accuracy_score(\n##    y_valid_turn, \n##    predicted_valid_turn)\n\n##print('Average precision score on validation data:', aps)\n##print('Accuracy score on validation data:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:15.211517Z","iopub.execute_input":"2023-05-30T09:00:15.211876Z","iopub.status.idle":"2023-05-30T09:00:15.220830Z","shell.execute_reply.started":"2023-05-30T09:00:15.211848Z","shell.execute_reply":"2023-05-30T09:00:15.219643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# RandomForest without Randomized, goal: to get inside 9 hours\n\n# Create a decision tree classifier\nmodel_rf_turn = Pipeline([('scaler', StandardScaler()), \n                 ('clf', RandomForestClassifier(random_state=13,\n                                                min_samples_leaf = 5,\n                                                min_samples_split = 7,\n                                                max_depth = 5,\n                                                class_weight='balanced'))])\n\n# Fit the model_rf_turn to the data\nmodel_rf_turn.fit(X_train_turn, y_train_turn)\n\npredicted_valid_turn = model_rf_turn.predict(X_valid_turn)\n\naps = average_precision_score(\n    y_valid_turn, \n    predicted_valid_turn)\n\nacc = accuracy_score(\n    y_valid_turn, \n    predicted_valid_turn)\n\nprint('Average precision score on validation data:', aps)\nprint('Accuracy score on validation data:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:00:15.222260Z","iopub.execute_input":"2023-05-30T09:00:15.224372Z","iopub.status.idle":"2023-05-30T09:02:28.127891Z","shell.execute_reply.started":"2023-05-30T09:00:15.224325Z","shell.execute_reply":"2023-05-30T09:02:28.126702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### StartHesitation-FoG","metadata":{}},{"cell_type":"code","source":"# create train-test dataframe for Turn-FoG\n\nStartHesitation_fog_data = (mega_data.loc[mega_data.Visit == 13]\n                 .reset_index(drop=True))","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:02:28.134765Z","iopub.execute_input":"2023-05-30T09:02:28.135116Z","iopub.status.idle":"2023-05-30T09:02:28.185226Z","shell.execute_reply.started":"2023-05-30T09:02:28.135088Z","shell.execute_reply":"2023-05-30T09:02:28.183966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# feature's and target's sets\n\nfeature_StartHesitation = StartHesitation_fog_data[['AccV','AccML','AccAP']]\n\ntarget_StartHesitation = StartHesitation_fog_data.StartHesitation\n\nStartHesitation_fog_data.shape, feature_StartHesitation.shape, target_StartHesitation.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:02:28.186805Z","iopub.execute_input":"2023-05-30T09:02:28.187160Z","iopub.status.idle":"2023-05-30T09:02:28.198973Z","shell.execute_reply.started":"2023-05-30T09:02:28.187129Z","shell.execute_reply":"2023-05-30T09:02:28.197738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare data for train/valid\n\nX_train_StartHesitation, X_valid_StartHesitation, \\\ny_train_StartHesitation, y_valid_StartHesitation = (\n    train_test_split(feature_StartHesitation, target_StartHesitation, random_state=13))\n\nX_train_StartHesitation.shape, X_valid_StartHesitation.shape,\\\ny_train_StartHesitation.shape, y_valid_StartHesitation.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:02:28.200427Z","iopub.execute_input":"2023-05-30T09:02:28.200769Z","iopub.status.idle":"2023-05-30T09:02:28.318325Z","shell.execute_reply.started":"2023-05-30T09:02:28.200742Z","shell.execute_reply":"2023-05-30T09:02:28.316642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# RandomForest\n\n# Parameters\n##param_grid = {'clf__max_depth': range(4,6,1)}\n\n# Create a decision tree classifier\n##model = Pipeline([('scaler', StandardScaler()), \n##                 ('clf', RandomForestClassifier(random_state=13,\n##                                                min_samples_leaf = 9,\n##                                                min_samples_split = 2,\n##                                                class_weight='balanced'))])\n# Create the random search object\n##random_search = RandomizedSearchCV(model, \n##                                   param_grid, \n##                                   cv=5, \n##                                   scoring='average_precision',\n##                                   random_state=13)\n\n# Fit the random search to the data\n##random_search.fit(X_train_StartHesitation, y_train_StartHesitation)\n                \n# Print the best parameters found\n##print(random_search.best_params_, random_search.best_score_)\n\n##model_rf_StartHesitation = deepcopy(random_search.best_estimator_)\n#model_rf_StartHesitation.set_params(clf__n_estimators=200)\n\n##predicted_valid_StartHesitation = (\n##    model_rf_StartHesitation.predict(X_valid_StartHesitation))\n\n##aps = average_precision_score(\n##    y_valid_StartHesitation, \n##    predicted_valid_StartHesitation)\n\n##acc = accuracy_score(\n##    y_valid_StartHesitation, \n##    predicted_valid_StartHesitation)\n\n##print('Average precision score on validation data:', aps)\n##print('Accuracy score on validation data:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:02:28.320392Z","iopub.execute_input":"2023-05-30T09:02:28.320846Z","iopub.status.idle":"2023-05-30T09:02:28.330349Z","shell.execute_reply.started":"2023-05-30T09:02:28.320800Z","shell.execute_reply":"2023-05-30T09:02:28.328854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# RandomForest\n\n# Create a decision tree classifier\nmodel_rf_StartHesitation = Pipeline([('scaler', StandardScaler()), \n                 ('clf', RandomForestClassifier(random_state=13,\n                                                min_samples_leaf = 9,\n                                                min_samples_split = 2,\n                                                max_depth = 5,\n                                                class_weight='balanced'))])\n\n# Fit the model_rf_StartHesitation to the data\nmodel_rf_StartHesitation.fit(X_train_StartHesitation, y_train_StartHesitation)\n                \npredicted_valid_StartHesitation = (\n    model_rf_StartHesitation.predict(X_valid_StartHesitation))\n\naps = average_precision_score(\n    y_valid_StartHesitation, \n    predicted_valid_StartHesitation)\n\nacc = accuracy_score(\n    y_valid_StartHesitation, \n    predicted_valid_StartHesitation)\n\nprint('Average precision score on validation data:', aps)\nprint('Accuracy score on validation data:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:02:28.331677Z","iopub.execute_input":"2023-05-30T09:02:28.332026Z","iopub.status.idle":"2023-05-30T09:03:32.243248Z","shell.execute_reply.started":"2023-05-30T09:02:28.331999Z","shell.execute_reply":"2023-05-30T09:03:32.242043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Walking-FoG","metadata":{}},{"cell_type":"code","source":"# create train-test dataframe for Turn-FoG\n\nWalking_fog_data = (mega_data.loc[mega_data.Visit == 20]\n                 .reset_index(drop=True))","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:03:32.244892Z","iopub.execute_input":"2023-05-30T09:03:32.245224Z","iopub.status.idle":"2023-05-30T09:03:32.297316Z","shell.execute_reply.started":"2023-05-30T09:03:32.245197Z","shell.execute_reply":"2023-05-30T09:03:32.296066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# feature's and target's sets\n\nfeature_Walking = Walking_fog_data[['Time','AccV','AccML','AccAP']]\n\ntarget_Walking = Walking_fog_data.Walking\n\nWalking_fog_data.shape, feature_Walking.shape, target_Walking.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:03:32.298777Z","iopub.execute_input":"2023-05-30T09:03:32.299530Z","iopub.status.idle":"2023-05-30T09:03:32.315614Z","shell.execute_reply.started":"2023-05-30T09:03:32.299487Z","shell.execute_reply":"2023-05-30T09:03:32.314410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare data for train/valid\n\nX_train_Walking, X_valid_Walking, \\\ny_train_Walking, y_valid_Walking = (\n    train_test_split(feature_Walking, target_Walking, random_state=13))\n\nX_train_Walking.shape, X_valid_Walking.shape,\\\ny_train_Walking.shape, y_valid_Walking.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:03:32.317482Z","iopub.execute_input":"2023-05-30T09:03:32.317917Z","iopub.status.idle":"2023-05-30T09:03:32.421586Z","shell.execute_reply.started":"2023-05-30T09:03:32.317881Z","shell.execute_reply":"2023-05-30T09:03:32.420602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# RandomForest\n\n# Parameters\n##param_grid = {'clf__max_depth': range(4,6,1)}\n\n# Create a decision tree classifier\n##model = Pipeline([('scaler', StandardScaler()), \n##                 ('clf', RandomForestClassifier(random_state=13,\n##                                                min_samples_leaf = 9,\n##                                                min_samples_split = 2,\n##                                                class_weight='balanced'))])\n# Create the random search object\n##random_search = RandomizedSearchCV(model, \n##                                   param_grid, \n##                                   cv=5, \n##                                   scoring='average_precision',\n##                                   random_state=13)\n\n# Fit the random search to the data\n##random_search.fit(X_train_Walking, y_train_Walking)\n                \n# Print the best parameters found\n##print(random_search.best_params_, random_search.best_score_)\n\n##model_rf_Walking = deepcopy(random_search.best_estimator_)\n#model_rf_Walking.set_params(clf__n_estimators=200)\n\n##predicted_valid_Walking = (\n##    model_rf_Walking.predict(X_valid_Walking))\n\n##aps = average_precision_score(\n##    y_valid_Walking, \n##    predicted_valid_Walking)\n\n##acc = accuracy_score(\n##    y_valid_Walking, \n##    predicted_valid_Walking)\n\n##print('Average precision score on validation data:', aps)\n##print('Accuracy score on validation data:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:03:32.423071Z","iopub.execute_input":"2023-05-30T09:03:32.423402Z","iopub.status.idle":"2023-05-30T09:03:32.428867Z","shell.execute_reply.started":"2023-05-30T09:03:32.423374Z","shell.execute_reply":"2023-05-30T09:03:32.427911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# RandomForest\n\n# Create a decision tree classifier\nmodel_rf_Walking = Pipeline([('scaler', StandardScaler()), \n                 ('clf', RandomForestClassifier(random_state=13,\n                                                min_samples_leaf = 9,\n                                                min_samples_split = 2,\n                                                max_depth = 5,\n                                                class_weight='balanced'))])\n\n# Fit the model_rf_Walking to the data\nmodel_rf_Walking.fit(X_train_Walking, y_train_Walking)\n\npredicted_valid_Walking = (\n    model_rf_Walking.predict(X_valid_Walking))\n\naps = average_precision_score(\n    y_valid_Walking, \n    predicted_valid_Walking)\n\nacc = accuracy_score(\n    y_valid_Walking, \n    predicted_valid_Walking)\n\nprint('Average precision score on validation data:', aps)\nprint('Accuracy score on validation data:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:03:32.429955Z","iopub.execute_input":"2023-05-30T09:03:32.430478Z","iopub.status.idle":"2023-05-30T09:04:18.007744Z","shell.execute_reply.started":"2023-05-30T09:03:32.430450Z","shell.execute_reply":"2023-05-30T09:04:18.006604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 'Cooking' submit file","metadata":{}},{"cell_type":"markdown","source":"Main idea is to predict FoG issue separatly for the Turn/StartHesitation/Walking cases. Then concat predicted series to submit dataframe and 'clean' it by special function to avoid several positive ('1') FoG moments in the same time (row). \n\nWe set positive prioryte like that: \n- 1st - most frequent case (Turn);\n- 2nd - model with highest prediction ability (Walking);\n- 3rd - the last one (StartHesitation).","metadata":{}},{"cell_type":"code","source":"# function for read and save test data\n\ndef prepare_test_data (df_type):\n    full_data = pd.DataFrame()\n    subdatas = glob.glob(data_directory + f'test/{df_type}/*')\n    \n    for subdata in subdatas:\n        sub_data = pd.read_csv(subdata)\n        sub_data['Id'] = subdata.split(sep='/')[-1].split(sep='.')[0] + '_' + sub_data['Time'].astype(str)\n        full_data = pd.concat([full_data, sub_data]).reset_index(drop=True)\n            \n    return full_data","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:18.009283Z","iopub.execute_input":"2023-05-30T09:04:18.009752Z","iopub.status.idle":"2023-05-30T09:04:18.019061Z","shell.execute_reply.started":"2023-05-30T09:04:18.009710Z","shell.execute_reply":"2023-05-30T09:04:18.017704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_tdcsfog = prepare_test_data('tdcsfog')\ntest_tdcsfog.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:18.020818Z","iopub.execute_input":"2023-05-30T09:04:18.021288Z","iopub.status.idle":"2023-05-30T09:04:18.068543Z","shell.execute_reply.started":"2023-05-30T09:04:18.021257Z","shell.execute_reply":"2023-05-30T09:04:18.067359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_defog = prepare_test_data('defog')\ntest_defog.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:18.070072Z","iopub.execute_input":"2023-05-30T09:04:18.070492Z","iopub.status.idle":"2023-05-30T09:04:18.762806Z","shell.execute_reply.started":"2023-05-30T09:04:18.070454Z","shell.execute_reply":"2023-05-30T09:04:18.761618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**defog**","metadata":{}},{"cell_type":"code","source":"# data for StartHesitation/Turn models\ntest_defog_pred = test_defog.drop('Id', axis=1)\ntest_defog_short = test_defog.drop(['Time', 'Id'], axis=1)\n\n# defog_StartHesitation\ndefog_StartHesitation_test = model_rf_StartHesitation.predict(test_defog_short)\n\n# defog_Turn\ndefog_Turn_test = model_rf_turn.predict(test_defog_short)\n\n# defog_Walking\ndefog_Walking_test = model_rf_Walking.predict(test_defog_pred)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:18.764353Z","iopub.execute_input":"2023-05-30T09:04:18.765164Z","iopub.status.idle":"2023-05-30T09:04:23.071065Z","shell.execute_reply.started":"2023-05-30T09:04:18.765123Z","shell.execute_reply":"2023-05-30T09:04:23.069915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# defog table\n\ndefog_StartHesitation_test = pd.Series(defog_StartHesitation_test, name='StartHesitation')\ndefog_Turn_test = pd.Series(defog_Turn_test, name='Turn')\ndefog_Walking_test = pd.Series(defog_Walking_test, name='Walking')\n\ntest_defog_predicted = pd.concat([defog_StartHesitation_test, \n                                  defog_Turn_test, \n                                  defog_Walking_test], axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.072349Z","iopub.execute_input":"2023-05-30T09:04:23.072715Z","iopub.status.idle":"2023-05-30T09:04:23.081155Z","shell.execute_reply.started":"2023-05-30T09:04:23.072686Z","shell.execute_reply":"2023-05-30T09:04:23.079863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# display what we got\n##print(test_defog_predicted.StartHesitation.unique())\n##print(test_defog_predicted.Turn.unique())\n##print(test_defog_predicted.Walking.unique())","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.082660Z","iopub.execute_input":"2023-05-30T09:04:23.083449Z","iopub.status.idle":"2023-05-30T09:04:23.099926Z","shell.execute_reply.started":"2023-05-30T09:04:23.083418Z","shell.execute_reply":"2023-05-30T09:04:23.099122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**tdcsfog**","metadata":{"execution":{"iopub.status.busy":"2023-05-18T21:11:48.854696Z","iopub.execute_input":"2023-05-18T21:11:48.855271Z","iopub.status.idle":"2023-05-18T21:11:48.866777Z","shell.execute_reply.started":"2023-05-18T21:11:48.855223Z","shell.execute_reply":"2023-05-18T21:11:48.865421Z"}}},{"cell_type":"code","source":"# data for StartHesitation/Turn models\ntest_tdcsfog_pred = test_tdcsfog.drop('Id', axis=1)\ntest_tdcsfog_short = test_tdcsfog.drop(['Time', 'Id'], axis=1)\n\n# defog_StartHesitation\ntdcsfog_StartHesitation_test = model_rf_StartHesitation.predict(test_tdcsfog_short)\n\n# defog_Turn\ntdcsfog_Turn_test = model_rf_turn.predict(test_tdcsfog_short)\n\n# defog_Walking\ntdcsfog_Walking_test = model_rf_Walking.predict(test_tdcsfog_pred)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.101397Z","iopub.execute_input":"2023-05-30T09:04:23.101895Z","iopub.status.idle":"2023-05-30T09:04:23.258937Z","shell.execute_reply.started":"2023-05-30T09:04:23.101861Z","shell.execute_reply":"2023-05-30T09:04:23.257624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# tdcsfog table\n\ntdcsfog_StartHesitation_test = pd.Series(tdcsfog_StartHesitation_test, name='StartHesitation')\ntdcsfog_Turn_test = pd.Series(tdcsfog_Turn_test, name='Turn')\ntdcsfog_Walking_test = pd.Series(tdcsfog_Walking_test, name='Walking')\n\ntest_tdcsfog_predicted = pd.concat([tdcsfog_StartHesitation_test, \n                                  tdcsfog_Turn_test, \n                                  tdcsfog_Walking_test], axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.260290Z","iopub.execute_input":"2023-05-30T09:04:23.260646Z","iopub.status.idle":"2023-05-30T09:04:23.269145Z","shell.execute_reply.started":"2023-05-30T09:04:23.260612Z","shell.execute_reply":"2023-05-30T09:04:23.267747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# display what we got\n\n##print(test_tdcsfog_predicted.StartHesitation.unique())\n##print(test_tdcsfog_predicted.Turn.unique())\n##print(test_tdcsfog_predicted.Walking.unique())","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.270455Z","iopub.execute_input":"2023-05-30T09:04:23.270947Z","iopub.status.idle":"2023-05-30T09:04:23.281053Z","shell.execute_reply.started":"2023-05-30T09:04:23.270913Z","shell.execute_reply":"2023-05-30T09:04:23.279966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's check cross sections between FOG\n\n##test_tdcsfog_predicted.loc[\n##    (test_tdcsfog_predicted.StartHesitation == 1)&(test_tdcsfog_predicted.Turn == 1)]\\\n##.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.282300Z","iopub.execute_input":"2023-05-30T09:04:23.282783Z","iopub.status.idle":"2023-05-30T09:04:23.294092Z","shell.execute_reply.started":"2023-05-30T09:04:23.282753Z","shell.execute_reply":"2023-05-30T09:04:23.292882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, let's use our clean function. Unfortunately, clean function is out of time limit(((","metadata":{}},{"cell_type":"code","source":"# clean function\n\n##def submit_chek (data_frame):\n##    submit_checked = pd.DataFrame() \n##    for row in range(len(data_frame)):\n##        sample = data_frame.iloc[row:row+1]\n##        if sample.iloc[0,2] == 1:\n##            temp_sample = pd.DataFrame(\n##                {'StartHesitation':[0], \n##                 'Turn':[0], \n##                 'Walking':[1]})\n##            submit_checked = pd.concat(\n##                [submit_checked, temp_sample]).reset_index(drop=True)\n            \n##        elif sample.iloc[0,1] == 1:\n##            temp_sample = pd.DataFrame(\n##                {'StartHesitation':[0], \n##                 'Turn':[1], \n##                 'Walking':[0]})\n##            submit_checked = pd.concat(\n##                [submit_checked, temp_sample]).reset_index(drop=True)\n                \n##        elif sample.iloc[0,0] == 1:\n##            temp_sample = pd.DataFrame(\n##                {'StartHesitation':[1], \n##                 'Turn':[0], \n##                 'Walking':[0]})\n##            submit_checked = pd.concat(\n##                [submit_checked, temp_sample])\n            \n##        else:\n##            temp_sample = pd.DataFrame({'StartHesitation':[0], \n##                                        'Turn':[0], \n##                                        'Walking':[0]})\n##            submit_checked = pd.concat(\n##                [submit_checked, temp_sample]).reset_index(drop=True)\n            \n##    return submit_checked","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.295273Z","iopub.execute_input":"2023-05-30T09:04:23.295627Z","iopub.status.idle":"2023-05-30T09:04:23.307314Z","shell.execute_reply.started":"2023-05-30T09:04:23.295546Z","shell.execute_reply":"2023-05-30T09:04:23.306117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pandas way\n\ndef submit_chek (df):\n    df.loc[(df.StartHesitation == 1)&(df.Turn == 1)&(df.Walking == 1),\n           ['StartHesitation','Turn','Walking']] = [0,1,0]\n    df.loc[(df.StartHesitation == 1)&(df.Turn == 1)&(df.Walking == 0),\n           ['StartHesitation','Turn','Walking']] = [0,1,0]\n    df.loc[(df.StartHesitation == 1)&(df.Turn == 0)&(df.Walking == 1),\n           ['StartHesitation','Turn','Walking']] = [0,0,1]\n    df.loc[(df.StartHesitation == 0)&(df.Turn == 1)&(df.Walking == 1),\n           ['StartHesitation','Turn','Walking']] = [0,1,0]\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.309326Z","iopub.execute_input":"2023-05-30T09:04:23.309802Z","iopub.status.idle":"2023-05-30T09:04:23.322853Z","shell.execute_reply.started":"2023-05-30T09:04:23.309764Z","shell.execute_reply":"2023-05-30T09:04:23.321756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# \"clean\" defog/tdcs data\n\ntest_tdcsfog_predicted_clean = submit_chek(test_tdcsfog_predicted)\ntest_defog_predicted_clean = submit_chek(test_defog_predicted)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.324205Z","iopub.execute_input":"2023-05-30T09:04:23.324533Z","iopub.status.idle":"2023-05-30T09:04:23.355413Z","shell.execute_reply.started":"2023-05-30T09:04:23.324505Z","shell.execute_reply":"2023-05-30T09:04:23.354310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Creat submission file**","metadata":{}},{"cell_type":"code","source":"total_data = (pd.concat(\n    [test_tdcsfog_predicted_clean, \n     test_defog_predicted_clean])\n              .reset_index(drop=True))\n\ntotal_data.head(),\\\ntotal_data.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.356726Z","iopub.execute_input":"2023-05-30T09:04:23.357058Z","iopub.status.idle":"2023-05-30T09:04:23.366609Z","shell.execute_reply.started":"2023-05-30T09:04:23.357031Z","shell.execute_reply":"2023-05-30T09:04:23.365789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##total_data = (pd.concat(\n##    [test_tdcsfog_predicted, \n##     test_defog_predicted])\n##              .reset_index(drop=True))\n\n##total_data.head(),\\\n##total_data.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.367447Z","iopub.execute_input":"2023-05-30T09:04:23.367787Z","iopub.status.idle":"2023-05-30T09:04:23.378376Z","shell.execute_reply.started":"2023-05-30T09:04:23.367748Z","shell.execute_reply":"2023-05-30T09:04:23.377130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create 'Id_Time' column for submition file\n\ndef id_for_submission(tdcsfog, defog):\n    df_tdcsfog = tdcsfog\n    df_defog = defog\n    \n    df = pd.concat([df_tdcsfog,\n                    df_defog]).reset_index(drop=True)\n    \n    submission_id = df.drop(['Time',\n                         'AccV', \n                         'AccML', \n                         'AccAP'], axis=1)\n    \n    return submission_id","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.379682Z","iopub.execute_input":"2023-05-30T09:04:23.379983Z","iopub.status.idle":"2023-05-30T09:04:23.392377Z","shell.execute_reply.started":"2023-05-30T09:04:23.379959Z","shell.execute_reply":"2023-05-30T09:04:23.391089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save predictions in format used for competition scoring\n\nsubmission = id_for_submission(test_tdcsfog, test_defog)\nsubmission = pd.concat([submission, total_data], axis=1)\n\nsubmission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:23.393870Z","iopub.execute_input":"2023-05-30T09:04:23.394234Z","iopub.status.idle":"2023-05-30T09:04:24.366542Z","shell.execute_reply.started":"2023-05-30T09:04:23.394206Z","shell.execute_reply":"2023-05-30T09:04:24.365324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ta-Da!","metadata":{}},{"cell_type":"code","source":"submission","metadata":{"execution":{"iopub.status.busy":"2023-05-30T09:04:24.368002Z","iopub.execute_input":"2023-05-30T09:04:24.368423Z","iopub.status.idle":"2023-05-30T09:04:24.383387Z","shell.execute_reply.started":"2023-05-30T09:04:24.368384Z","shell.execute_reply":"2023-05-30T09:04:24.382099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's try to check how \"cleaning\" function could help.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nOf course, the use of this model will not allow achieving high prediction accuracy, but it will allow to work out the main approaches to model preparation. This approach has not shown the worst result and can be the basis for assessing the introduction of any changes to the model and training features. Further it would be nice to work with Threshold. But, unfortunately, it is difficult to evaluate the changes made on the test sample due to the high duration of the execution of the notebook, more than 9 hours. It is also very unfortunate that it was not possible to evaluate the effectiveness of the cleanup function on the test sample for the same reason.","metadata":{}}]}