{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"The purpose of this notebook is to perform EDA on the dataset of kaggle competition <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction\" style=\"text-decoration:none\">Parkinson's Freezing of Gait Prediction</a>, so as to figure out the relationship among those sub-dataset like `subjects.csv`, `events.csv`, `tasks.csv`, `defog_metadata.csv`, `tdcsfog_metadata.csv`, `daily_metadata.csv` and the others in the `test/` sub-folder, `train/` sub-folder and `unlabeled/` sub-folder. \n\nThis work is time-consuming and requires patience, but it is inevitable and worthwhile for the later processes like feature engineering and modeling.\n\n<br>\n\n# Import Libraries","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport glob\nimport pathlib\nfrom tqdm.auto import tqdm\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2023-06-09T03:18:02.605440Z","iopub.execute_input":"2023-06-09T03:18:02.605820Z","iopub.status.idle":"2023-06-09T03:18:03.326627Z","shell.execute_reply.started":"2023-06-09T03:18:02.605787Z","shell.execute_reply":"2023-06-09T03:18:03.325308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n# Loading Dataset","metadata":{}},{"cell_type":"code","source":"data_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/'\n\ntrain_defog = glob.glob(data_root + 'train/defog/**')\ntrain_tdcsfog = glob.glob(data_root + 'train/tdcsfog/**')\ntrain_notype = glob.glob(data_root + 'train/notype/**')\n\ntest_defog = glob.glob(data_root + 'test/defog/**')\ntest_tdcsfog = glob.glob(data_root + 'test/tdcsfog/**')\n\nsubjects = pd.read_csv(data_root + 'subjects.csv')\ntasks = pd.read_csv(data_root + 'tasks.csv')\nevents = pd.read_csv(data_root + 'events.csv')\n\ndaily_metadata=pd.read_csv(data_root + 'daily_metadata.csv')\ntdcsfog_metadata=pd.read_csv(data_root + 'tdcsfog_metadata.csv')\ndefog_metadata=pd.read_csv(data_root + 'defog_metadata.csv')\n\nsub = pd.read_csv(data_root + 'sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.328093Z","iopub.execute_input":"2023-06-09T03:18:03.328448Z","iopub.status.idle":"2023-06-09T03:18:03.646579Z","shell.execute_reply.started":"2023-06-09T03:18:03.328419Z","shell.execute_reply":"2023-06-09T03:18:03.645399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n<br>\n\n# Data Descriptions","metadata":{}},{"cell_type":"markdown","source":"## training and testing subfolders","metadata":{}},{"cell_type":"code","source":"print(\"train_defog: \", len(train_defog))\nprint(train_defog[0])                           # print the first example\nprint()\n\nprint(\"train_tdcsfog: \", len(train_tdcsfog))\nprint(train_tdcsfog[0])                         # print the first example\nprint()\n\nprint(\"train_notype: \", len(train_notype))\nprint(train_notype[0])                         # print the first example","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.648136Z","iopub.execute_input":"2023-06-09T03:18:03.648596Z","iopub.status.idle":"2023-06-09T03:18:03.657428Z","shell.execute_reply.started":"2023-06-09T03:18:03.648563Z","shell.execute_reply":"2023-06-09T03:18:03.655940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"test_defog: \", len(test_defog))\nprint(test_defog[0])                           # print the first example\nprint()\n\nprint(\"test_tdcsfog: \", len(test_tdcsfog))\nprint(test_tdcsfog[0])                         # print the first example\nprint()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.661191Z","iopub.execute_input":"2023-06-09T03:18:03.661570Z","iopub.status.idle":"2023-06-09T03:18:03.674741Z","shell.execute_reply.started":"2023-06-09T03:18:03.661539Z","shell.execute_reply":"2023-06-09T03:18:03.673324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n## events\n\n**events.csv** : Metadata for each FoG event in all data series. The event times agree with the labels in the data series.\n\n* `Id` The data series the event occured in.\n* `Init` Time (s) the event began.\n* `Completion` Time (s) the event ended.\n* `Type` Whether **StartHesitation**, **Turn**, or **Walking**.\n* `Kinetic` Whether the event was kinetic (**1**) and involved movement, or akinetic (**0**) and static.","metadata":{}},{"cell_type":"code","source":"events.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.676051Z","iopub.execute_input":"2023-06-09T03:18:03.676532Z","iopub.status.idle":"2023-06-09T03:18:03.702712Z","shell.execute_reply.started":"2023-06-09T03:18:03.676488Z","shell.execute_reply":"2023-06-09T03:18:03.701748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.704255Z","iopub.execute_input":"2023-06-09T03:18:03.704583Z","iopub.status.idle":"2023-06-09T03:18:03.711329Z","shell.execute_reply.started":"2023-06-09T03:18:03.704557Z","shell.execute_reply":"2023-06-09T03:18:03.710165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.712960Z","iopub.execute_input":"2023-06-09T03:18:03.713722Z","iopub.status.idle":"2023-06-09T03:18:03.737024Z","shell.execute_reply.started":"2023-06-09T03:18:03.713666Z","shell.execute_reply":"2023-06-09T03:18:03.735767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.describe(include=[\"O\"])","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.738687Z","iopub.execute_input":"2023-06-09T03:18:03.739076Z","iopub.status.idle":"2023-06-09T03:18:03.758409Z","shell.execute_reply.started":"2023-06-09T03:18:03.739045Z","shell.execute_reply":"2023-06-09T03:18:03.757119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.Type.value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.759991Z","iopub.execute_input":"2023-06-09T03:18:03.760355Z","iopub.status.idle":"2023-06-09T03:18:03.773206Z","shell.execute_reply.started":"2023-06-09T03:18:03.760325Z","shell.execute_reply":"2023-06-09T03:18:03.772116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.Type.unique().tolist()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.774586Z","iopub.execute_input":"2023-06-09T03:18:03.775393Z","iopub.status.idle":"2023-06-09T03:18:03.789492Z","shell.execute_reply.started":"2023-06-09T03:18:03.775359Z","shell.execute_reply":"2023-06-09T03:18:03.788021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.Id.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.792044Z","iopub.execute_input":"2023-06-09T03:18:03.793084Z","iopub.status.idle":"2023-06-09T03:18:03.807360Z","shell.execute_reply.started":"2023-06-09T03:18:03.793030Z","shell.execute_reply":"2023-06-09T03:18:03.806024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.Id.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.809359Z","iopub.execute_input":"2023-06-09T03:18:03.810287Z","iopub.status.idle":"2023-06-09T03:18:03.820583Z","shell.execute_reply.started":"2023-06-09T03:18:03.810239Z","shell.execute_reply":"2023-06-09T03:18:03.819675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.Kinetic.unique()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.827087Z","iopub.execute_input":"2023-06-09T03:18:03.828026Z","iopub.status.idle":"2023-06-09T03:18:03.837191Z","shell.execute_reply.started":"2023-06-09T03:18:03.827963Z","shell.execute_reply":"2023-06-09T03:18:03.835685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n## subjects\n\n**subjects.csv** : Metadata for each `Subject` in the study, including their `Age` and `Sex` as well as:\n* `Visit` Only available for subjects in the ***daily*** and ***defog*** datasets.\n* `YearsSinceDx` Years since Parkinson's diagnosis.\n* `UPDRSIIIOn/UPDRSIIIOff` Unified Parkinson's Disease Rating Scale score during on/off medication respectively.\n* `NFOGQ` Self-report FoG questionnaire score. See: https://pubmed.ncbi.nlm.nih.gov/19660949/\n","metadata":{}},{"cell_type":"code","source":"subjects.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.841206Z","iopub.execute_input":"2023-06-09T03:18:03.841619Z","iopub.status.idle":"2023-06-09T03:18:03.863870Z","shell.execute_reply.started":"2023-06-09T03:18:03.841586Z","shell.execute_reply":"2023-06-09T03:18:03.862860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.865627Z","iopub.execute_input":"2023-06-09T03:18:03.866003Z","iopub.status.idle":"2023-06-09T03:18:03.873561Z","shell.execute_reply.started":"2023-06-09T03:18:03.865972Z","shell.execute_reply":"2023-06-09T03:18:03.872181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.875033Z","iopub.execute_input":"2023-06-09T03:18:03.875431Z","iopub.status.idle":"2023-06-09T03:18:03.895408Z","shell.execute_reply.started":"2023-06-09T03:18:03.875402Z","shell.execute_reply":"2023-06-09T03:18:03.893740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.describe()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.899009Z","iopub.execute_input":"2023-06-09T03:18:03.899372Z","iopub.status.idle":"2023-06-09T03:18:03.944990Z","shell.execute_reply.started":"2023-06-09T03:18:03.899343Z","shell.execute_reply":"2023-06-09T03:18:03.943757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.describe(include=[\"O\"])","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.949047Z","iopub.execute_input":"2023-06-09T03:18:03.949454Z","iopub.status.idle":"2023-06-09T03:18:03.967596Z","shell.execute_reply.started":"2023-06-09T03:18:03.949421Z","shell.execute_reply":"2023-06-09T03:18:03.965969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.Subject.value_counts().sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.969992Z","iopub.execute_input":"2023-06-09T03:18:03.970540Z","iopub.status.idle":"2023-06-09T03:18:03.984252Z","shell.execute_reply.started":"2023-06-09T03:18:03.970494Z","shell.execute_reply":"2023-06-09T03:18:03.983043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.Subject.value_counts(dropna=False).shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:03.985900Z","iopub.execute_input":"2023-06-09T03:18:03.986316Z","iopub.status.idle":"2023-06-09T03:18:04.000638Z","shell.execute_reply.started":"2023-06-09T03:18:03.986284Z","shell.execute_reply":"2023-06-09T03:18:03.999318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_ = subjects.Subject.value_counts()\n\nsubjects[subjects.Subject.isin(df_[df_>=2].index.values)]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.001974Z","iopub.execute_input":"2023-06-09T03:18:04.002338Z","iopub.status.idle":"2023-06-09T03:18:04.037777Z","shell.execute_reply.started":"2023-06-09T03:18:04.002308Z","shell.execute_reply":"2023-06-09T03:18:04.036213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4>It shows that there are 136 different subjects, some of them have more than one records in the `subjects.csv` and this is because they visit at least twice.</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n## tasks\n\n**tasks.csv** : Task metadata for series in the ***defog*** dataset. (Not relevant for the series in the ***tdcsfog*** or ***daily*** datasets.)\n\n* `Id` The data series where the task was measured.\n* `Begin` Time (s) the task began.\n* `End` Time (s) the task ended.\n* `Task` One of seven tasks types in the DeFOG protocol, described on <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/overview/additional-data-documentation\" style=\"text-decoration:none\">this page</a>.","metadata":{}},{"cell_type":"code","source":"tasks.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.039051Z","iopub.execute_input":"2023-06-09T03:18:04.039928Z","iopub.status.idle":"2023-06-09T03:18:04.053894Z","shell.execute_reply.started":"2023-06-09T03:18:04.039892Z","shell.execute_reply":"2023-06-09T03:18:04.052653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.055158Z","iopub.execute_input":"2023-06-09T03:18:04.055488Z","iopub.status.idle":"2023-06-09T03:18:04.068879Z","shell.execute_reply.started":"2023-06-09T03:18:04.055462Z","shell.execute_reply":"2023-06-09T03:18:04.067572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.070645Z","iopub.execute_input":"2023-06-09T03:18:04.071779Z","iopub.status.idle":"2023-06-09T03:18:04.093764Z","shell.execute_reply.started":"2023-06-09T03:18:04.071730Z","shell.execute_reply":"2023-06-09T03:18:04.092321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.describe(include=[\"O\"])","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.095390Z","iopub.execute_input":"2023-06-09T03:18:04.095903Z","iopub.status.idle":"2023-06-09T03:18:04.122708Z","shell.execute_reply.started":"2023-06-09T03:18:04.095855Z","shell.execute_reply":"2023-06-09T03:18:04.121371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.Task.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.124527Z","iopub.execute_input":"2023-06-09T03:18:04.125548Z","iopub.status.idle":"2023-06-09T03:18:04.135576Z","shell.execute_reply.started":"2023-06-09T03:18:04.125500Z","shell.execute_reply":"2023-06-09T03:18:04.134220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.Task.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.137138Z","iopub.execute_input":"2023-06-09T03:18:04.137544Z","iopub.status.idle":"2023-06-09T03:18:04.150491Z","shell.execute_reply.started":"2023-06-09T03:18:04.137509Z","shell.execute_reply":"2023-06-09T03:18:04.149448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4> It is said above that there were seven tasks types in the DeFOG protocol. Here we can find that seven types expand to 31 kinds totally.</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n## daily_metadata\n\nEach series in the ***daily*** dataset is identified by the `Subject` id. This file also contains the time of day the recording began.","metadata":{}},{"cell_type":"code","source":"daily_metadata.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.151520Z","iopub.execute_input":"2023-06-09T03:18:04.151889Z","iopub.status.idle":"2023-06-09T03:18:04.169824Z","shell.execute_reply.started":"2023-06-09T03:18:04.151857Z","shell.execute_reply":"2023-06-09T03:18:04.168818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_metadata.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.171341Z","iopub.execute_input":"2023-06-09T03:18:04.171681Z","iopub.status.idle":"2023-06-09T03:18:04.184545Z","shell.execute_reply.started":"2023-06-09T03:18:04.171645Z","shell.execute_reply":"2023-06-09T03:18:04.183586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_metadata.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.186477Z","iopub.execute_input":"2023-06-09T03:18:04.187412Z","iopub.status.idle":"2023-06-09T03:18:04.208373Z","shell.execute_reply.started":"2023-06-09T03:18:04.187357Z","shell.execute_reply":"2023-06-09T03:18:04.207357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_metadata.describe(include=[\"O\"])","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.210246Z","iopub.execute_input":"2023-06-09T03:18:04.210629Z","iopub.status.idle":"2023-06-09T03:18:04.230836Z","shell.execute_reply.started":"2023-06-09T03:18:04.210599Z","shell.execute_reply":"2023-06-09T03:18:04.229574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.Subject.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.232319Z","iopub.execute_input":"2023-06-09T03:18:04.233336Z","iopub.status.idle":"2023-06-09T03:18:04.240285Z","shell.execute_reply.started":"2023-06-09T03:18:04.233299Z","shell.execute_reply":"2023-06-09T03:18:04.239285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n## defog_metadata\n\n**defog_metadata.csv** : Identifies each series in the ***defog*** dataset by a unique `Subject`, `Visit`, `Medication` condition.","metadata":{}},{"cell_type":"code","source":"defog_metadata.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.242129Z","iopub.execute_input":"2023-06-09T03:18:04.243161Z","iopub.status.idle":"2023-06-09T03:18:04.259080Z","shell.execute_reply.started":"2023-06-09T03:18:04.243119Z","shell.execute_reply":"2023-06-09T03:18:04.257769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_metadata.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.261583Z","iopub.execute_input":"2023-06-09T03:18:04.262430Z","iopub.status.idle":"2023-06-09T03:18:04.270306Z","shell.execute_reply.started":"2023-06-09T03:18:04.262390Z","shell.execute_reply":"2023-06-09T03:18:04.269096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_metadata.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.272156Z","iopub.execute_input":"2023-06-09T03:18:04.272912Z","iopub.status.idle":"2023-06-09T03:18:04.289134Z","shell.execute_reply.started":"2023-06-09T03:18:04.272870Z","shell.execute_reply":"2023-06-09T03:18:04.287802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_metadata.describe(include=[\"O\"])","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.290578Z","iopub.execute_input":"2023-06-09T03:18:04.290960Z","iopub.status.idle":"2023-06-09T03:18:04.309280Z","shell.execute_reply.started":"2023-06-09T03:18:04.290930Z","shell.execute_reply":"2023-06-09T03:18:04.307998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_metadata.Id.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.310836Z","iopub.execute_input":"2023-06-09T03:18:04.311799Z","iopub.status.idle":"2023-06-09T03:18:04.324124Z","shell.execute_reply.started":"2023-06-09T03:18:04.311765Z","shell.execute_reply":"2023-06-09T03:18:04.322850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_metadata.Subject.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.326040Z","iopub.execute_input":"2023-06-09T03:18:04.326745Z","iopub.status.idle":"2023-06-09T03:18:04.336565Z","shell.execute_reply.started":"2023-06-09T03:18:04.326682Z","shell.execute_reply":"2023-06-09T03:18:04.335528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n## tdcsfog_metadata\n\n**tdcsfog_metadata.csv** : Identifies each series in the ***tdcsfog*** dataset by a unique `Subject, Visit, Test, Medication` condition.\n\n* `Visit` Lab visits consist of a baseline assessment, two post-treatment assessments for different treatment stages, and one follow-up assessment.\n* `Test` Which of three test types was performed, with 3 the most challenging.\n* `Medication` Subjects may have been either off or on anti-parkinsonian medication during the recording.","metadata":{}},{"cell_type":"code","source":"tdcsfog_metadata.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.338174Z","iopub.execute_input":"2023-06-09T03:18:04.338536Z","iopub.status.idle":"2023-06-09T03:18:04.357630Z","shell.execute_reply.started":"2023-06-09T03:18:04.338507Z","shell.execute_reply":"2023-06-09T03:18:04.356725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.369885Z","iopub.execute_input":"2023-06-09T03:18:04.370865Z","iopub.status.idle":"2023-06-09T03:18:04.378224Z","shell.execute_reply.started":"2023-06-09T03:18:04.370818Z","shell.execute_reply":"2023-06-09T03:18:04.377163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.info()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.379574Z","iopub.execute_input":"2023-06-09T03:18:04.380599Z","iopub.status.idle":"2023-06-09T03:18:04.403034Z","shell.execute_reply.started":"2023-06-09T03:18:04.380555Z","shell.execute_reply":"2023-06-09T03:18:04.401412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.describe()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.404477Z","iopub.execute_input":"2023-06-09T03:18:04.405515Z","iopub.status.idle":"2023-06-09T03:18:04.433337Z","shell.execute_reply.started":"2023-06-09T03:18:04.405470Z","shell.execute_reply":"2023-06-09T03:18:04.431852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.describe(include=[\"O\"])","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.434904Z","iopub.execute_input":"2023-06-09T03:18:04.436420Z","iopub.status.idle":"2023-06-09T03:18:04.457627Z","shell.execute_reply.started":"2023-06-09T03:18:04.436368Z","shell.execute_reply":"2023-06-09T03:18:04.456427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.Subject.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.459128Z","iopub.execute_input":"2023-06-09T03:18:04.459577Z","iopub.status.idle":"2023-06-09T03:18:04.468030Z","shell.execute_reply.started":"2023-06-09T03:18:04.459537Z","shell.execute_reply":"2023-06-09T03:18:04.466770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.Visit.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.469477Z","iopub.execute_input":"2023-06-09T03:18:04.470273Z","iopub.status.idle":"2023-06-09T03:18:04.484769Z","shell.execute_reply.started":"2023-06-09T03:18:04.470240Z","shell.execute_reply":"2023-06-09T03:18:04.483765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.Test.unique().shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.486264Z","iopub.execute_input":"2023-06-09T03:18:04.486609Z","iopub.status.idle":"2023-06-09T03:18:04.498935Z","shell.execute_reply.started":"2023-06-09T03:18:04.486581Z","shell.execute_reply":"2023-06-09T03:18:04.497609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n## `train/` folders\n\n\n\nContaining the data series in the training set within three subfolders: **tdcsfog/**, **defog/**, and **notype/**. Series in the **notype** folder are from the ***defog*** dataset but lack event-type annotations. The fields present in these series vary by folder.\n* `Time` An integer timestep. Series from the ***tdcsfog*** dataset are recorded at <font color=red>128Hz (128 timesteps per second)</font>, while series from the ***defog*** and ***daily*** series are recorded at <font color=red>100Hz (100 timesteps per second)</font>.\n\n* `AccV`, `AccML`, and `AccAP` Acceleration from a lower-back sensor on three axes: V - vertical, ML - mediolateral, AP - anteroposterior. Data is in units of <font color=red> m/s^2 </font> for **tdcsfog/** and <font color=red> g </font> for **defog/** and **notype/**.\n\n* `StartHesitation`, `Turn`, `Walking` Indicator variables for the occurrence of each of the event types.\n\n* `Event` Indicator variable for the occurrence of any FOG-type event. Present only in the **notype** series, which lack type-level annotations.\n\n* `Valid` There were cases during the video annotation that were hard for the annotator to decide if there was an Akinetic (i.e., essentially no movement) FoG or the subject stopped voluntarily. Only event annotations where the series is marked **true** should be considered as unambiguous.\n\n* `Task` Series were only annotated where this value is **true**. Portions marked **false** should be considered unannotated.\n\nNote that the `Valid` and `Task` fields are only present in the ***defog*** dataset. They are not relevant for the tdcsfog data.\n\n(<a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/data\" style=\"text-decoration:none\">Source</a>)\n\n<br>\n\n\n### train/defog/\n\n<font color=red>Note that</font> data series in the ***DeFOG*** dataset are annotated with `Valid` and `Task` labels (in addition to the event labels). <font color=red>Only the portions of the series where both are **true** should you consider to be annotated</font>. Though not included in the test set series, the metric is aware of these labels and will ignore predictions on the unannotated portions of these series (where either label is **false**).  (<a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/overview/evaluation\" style=\"text-decoration:none\">Source</a>)","metadata":{}},{"cell_type":"code","source":"# train/defog/\nprint(len(train_defog))\n\nid_train_defog = list(map(lambda x:x.split(\"/\")[-1].split(\".\")[0], train_defog))\nprint(\"Before set(): \", len(id_train_defog))\n\nid_train_defog = sorted(list(set(id_train_defog)))\nprint(\"After set() : \", len(set(id_train_defog)))\nid_train_defog[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.500332Z","iopub.execute_input":"2023-06-09T03:18:04.500744Z","iopub.status.idle":"2023-06-09T03:18:04.520769Z","shell.execute_reply.started":"2023-06-09T03:18:04.500678Z","shell.execute_reply":"2023-06-09T03:18:04.519264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_defog[0])\n\ntrain_defog_0 = pd.read_csv(train_defog[0])\ntrain_defog_0","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.522576Z","iopub.execute_input":"2023-06-09T03:18:04.522989Z","iopub.status.idle":"2023-06-09T03:18:04.710261Z","shell.execute_reply.started":"2023-06-09T03:18:04.522956Z","shell.execute_reply":"2023-06-09T03:18:04.708903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_defog_0[(train_defog_0.Valid==True) & (train_defog_0.Task==True)]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.711516Z","iopub.execute_input":"2023-06-09T03:18:04.711893Z","iopub.status.idle":"2023-06-09T03:18:04.737855Z","shell.execute_reply.started":"2023-06-09T03:18:04.711862Z","shell.execute_reply":"2023-06-09T03:18:04.736513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_defog_0[(train_defog_0.Valid==True) & (train_defog_0.Task==True) & (train_defog_0.Turn==1)]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.739421Z","iopub.execute_input":"2023-06-09T03:18:04.739901Z","iopub.status.idle":"2023-06-09T03:18:04.763740Z","shell.execute_reply.started":"2023-06-09T03:18:04.739868Z","shell.execute_reply":"2023-06-09T03:18:04.762852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_defog[90])\n\ntrain_defog_90 = pd.read_csv(train_defog[90])\ntrain_defog_90","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.764971Z","iopub.execute_input":"2023-06-09T03:18:04.765295Z","iopub.status.idle":"2023-06-09T03:18:04.946593Z","shell.execute_reply.started":"2023-06-09T03:18:04.765269Z","shell.execute_reply":"2023-06-09T03:18:04.945486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_defog_90[(train_defog_90.Valid==True) & (train_defog_90.Task==True)]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.947817Z","iopub.execute_input":"2023-06-09T03:18:04.948145Z","iopub.status.idle":"2023-06-09T03:18:04.974825Z","shell.execute_reply.started":"2023-06-09T03:18:04.948118Z","shell.execute_reply":"2023-06-09T03:18:04.973959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n### train/tdcsfog/","metadata":{}},{"cell_type":"code","source":"# train/tdcsfog/\nprint(len(train_tdcsfog))\n\nid_train_tdcsfog = list(map(lambda x:x.split(\"/\")[-1].split(\".\")[0], train_tdcsfog))\nprint(\"Before set(): \", len(id_train_tdcsfog))\n\nid_train_tdcsfog = sorted(list(set(id_train_tdcsfog)))\nprint(\"After set(): \", len(set(id_train_tdcsfog)))\nid_train_tdcsfog[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.977462Z","iopub.execute_input":"2023-06-09T03:18:04.977901Z","iopub.status.idle":"2023-06-09T03:18:04.990236Z","shell.execute_reply.started":"2023-06-09T03:18:04.977866Z","shell.execute_reply":"2023-06-09T03:18:04.988952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_tdcsfog[0])\n\ntrain_tdcsfog_0 = pd.read_csv(train_tdcsfog[0])\ntrain_tdcsfog_0","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:04.991614Z","iopub.execute_input":"2023-06-09T03:18:04.992069Z","iopub.status.idle":"2023-06-09T03:18:05.027739Z","shell.execute_reply.started":"2023-06-09T03:18:04.992036Z","shell.execute_reply":"2023-06-09T03:18:05.026391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_tdcsfog_0[train_tdcsfog_0.Turn != 0]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.029445Z","iopub.execute_input":"2023-06-09T03:18:05.029959Z","iopub.status.idle":"2023-06-09T03:18:05.051327Z","shell.execute_reply.started":"2023-06-09T03:18:05.029913Z","shell.execute_reply":"2023-06-09T03:18:05.049975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_tdcsfog[832])\n\ntrain_tdcsfog_832 = pd.read_csv(train_tdcsfog[832])\ntrain_tdcsfog_832","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.052829Z","iopub.execute_input":"2023-06-09T03:18:05.053210Z","iopub.status.idle":"2023-06-09T03:18:05.082214Z","shell.execute_reply.started":"2023-06-09T03:18:05.053177Z","shell.execute_reply":"2023-06-09T03:18:05.080993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_tdcsfog_832[train_tdcsfog_832.Turn != 0]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.083714Z","iopub.execute_input":"2023-06-09T03:18:05.084107Z","iopub.status.idle":"2023-06-09T03:18:05.105312Z","shell.execute_reply.started":"2023-06-09T03:18:05.084075Z","shell.execute_reply":"2023-06-09T03:18:05.103785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n### train/notype/","metadata":{}},{"cell_type":"code","source":"# train/notype/\nprint(len(train_notype))\n\nid_train_notype = list(map(lambda x:x.split(\"/\")[-1].split(\".\")[0], train_notype))\nprint(\"Before set(): \", len(id_train_notype))\n\nid_train_notype = sorted(list(set(id_train_notype)))\nprint(\"After set(): \", len(set(id_train_notype)))\nid_train_notype[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.106567Z","iopub.execute_input":"2023-06-09T03:18:05.107292Z","iopub.status.idle":"2023-06-09T03:18:05.119961Z","shell.execute_reply.started":"2023-06-09T03:18:05.107258Z","shell.execute_reply":"2023-06-09T03:18:05.118403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(id_train_defog[:5])\n\nset(id_train_notype) & set(id_train_defog)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.121951Z","iopub.execute_input":"2023-06-09T03:18:05.122353Z","iopub.status.idle":"2023-06-09T03:18:05.136924Z","shell.execute_reply.started":"2023-06-09T03:18:05.122319Z","shell.execute_reply":"2023-06-09T03:18:05.135766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4>The **ids** hidden in `train/defog/` and `train/notype/` are mutually exclusive.</font>","metadata":{}},{"cell_type":"code","source":"print(train_notype[0])\n\ntrain_notype_0 = pd.read_csv(train_notype[0])\ntrain_notype_0","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.138213Z","iopub.execute_input":"2023-06-09T03:18:05.138553Z","iopub.status.idle":"2023-06-09T03:18:05.409764Z","shell.execute_reply.started":"2023-06-09T03:18:05.138525Z","shell.execute_reply":"2023-06-09T03:18:05.408554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_notype_0[train_notype_0.Event != 0]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.411409Z","iopub.execute_input":"2023-06-09T03:18:05.411779Z","iopub.status.idle":"2023-06-09T03:18:05.432038Z","shell.execute_reply.started":"2023-06-09T03:18:05.411740Z","shell.execute_reply":"2023-06-09T03:18:05.430856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n## `unlabeled/` folders\n\nContaining the unannotated data series from the ***daily*** dataset, <font color=red>one series per subject</font>. Forty-five of the subjects also have series in the ***defog*** dataset, some in the training split and some in the test split. Accelerometer data has units of ` g `.","metadata":{}},{"cell_type":"code","source":"# unlabeled/\nunlabeled = glob.glob(data_root + \"unlabeled/**\")\nprint(len(unlabeled))\nunlabeled[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.433654Z","iopub.execute_input":"2023-06-09T03:18:05.434725Z","iopub.status.idle":"2023-06-09T03:18:05.444002Z","shell.execute_reply.started":"2023-06-09T03:18:05.434661Z","shell.execute_reply":"2023-06-09T03:18:05.442676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unlabeled_0 = pd.read_parquet(unlabeled[0])\nunlabeled_0.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:05.445801Z","iopub.execute_input":"2023-06-09T03:18:05.447317Z","iopub.status.idle":"2023-06-09T03:18:09.918742Z","shell.execute_reply.started":"2023-06-09T03:18:05.447282Z","shell.execute_reply":"2023-06-09T03:18:09.917417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"id_unlabeled = list(map(lambda x:x.split(\"/\")[-1].split(\".\")[0], unlabeled))\nprint(\"Before set(): \", len(id_unlabeled))\n\nid_unlabeled = sorted(list(set(id_unlabeled)))\nprint(\"After set(): \", len(set(id_unlabeled)))\nid_unlabeled[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:09.920778Z","iopub.execute_input":"2023-06-09T03:18:09.922063Z","iopub.status.idle":"2023-06-09T03:18:09.932497Z","shell.execute_reply.started":"2023-06-09T03:18:09.922020Z","shell.execute_reply":"2023-06-09T03:18:09.931052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(set(id_unlabeled) & set(daily_metadata.Id.unique()))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:09.934990Z","iopub.execute_input":"2023-06-09T03:18:09.935390Z","iopub.status.idle":"2023-06-09T03:18:09.946338Z","shell.execute_reply.started":"2023-06-09T03:18:09.935356Z","shell.execute_reply":"2023-06-09T03:18:09.945496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sorted(list(set(daily_metadata.Id.unique().tolist()))) == id_unlabeled","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:09.947506Z","iopub.execute_input":"2023-06-09T03:18:09.948524Z","iopub.status.idle":"2023-06-09T03:18:09.963244Z","shell.execute_reply.started":"2023-06-09T03:18:09.948491Z","shell.execute_reply":"2023-06-09T03:18:09.961878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4>Unique **ids** in the `daily_metadata.csv` are the same as those hidden in the `unlabeled/` subfolders.</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n## `test` folders\n\nOnly the `Time`, `AccV`, `AccML`, and `AccAP` fields are provided for the test series. See the <a href=\"https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/overview/evaluation\" style=\"text-decoration:none\">Evaluation</a> for details on how the hidden `Valid` and `Task` annotations affect scoring.\n\n\n### test/defog/","metadata":{}},{"cell_type":"code","source":"# test/defog/\nprint(len(test_defog))\n\nid_test_defog = list(map(lambda x:x.split(\"/\")[-1].split(\".\")[0], test_defog))\nprint(\"Before set(): \", len(id_test_defog))\n\nid_test_defog = sorted(list(set(id_test_defog)))\nprint(\"After set(): \", len(set(id_test_defog)))\nid_test_defog[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:09.965106Z","iopub.execute_input":"2023-06-09T03:18:09.966417Z","iopub.status.idle":"2023-06-09T03:18:09.979243Z","shell.execute_reply.started":"2023-06-09T03:18:09.966358Z","shell.execute_reply":"2023-06-09T03:18:09.977822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(test_defog[0])\n\ntest_defog_0 = pd.read_csv(test_defog[0])\ntest_defog_0","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:09.981140Z","iopub.execute_input":"2023-06-09T03:18:09.981634Z","iopub.status.idle":"2023-06-09T03:18:10.313343Z","shell.execute_reply.started":"2023-06-09T03:18:09.981591Z","shell.execute_reply":"2023-06-09T03:18:10.311986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n\n### test/tdcsfog/","metadata":{}},{"cell_type":"code","source":"# test/tdcsfog/\nprint(len(test_tdcsfog))\n\nid_test_tdcsfog = list(map(lambda x:x.split(\"/\")[-1].split(\".\")[0], test_tdcsfog))\nprint(\"Before set(): \", len(id_test_tdcsfog))\n\nid_test_tdcsfog = sorted(list(set(id_test_tdcsfog)))\nprint(\"After set(): \", len(set(id_test_tdcsfog)))\nid_test_tdcsfog[:5]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.315101Z","iopub.execute_input":"2023-06-09T03:18:10.315587Z","iopub.status.idle":"2023-06-09T03:18:10.328172Z","shell.execute_reply.started":"2023-06-09T03:18:10.315544Z","shell.execute_reply":"2023-06-09T03:18:10.326564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(test_tdcsfog[0])\n\ntest_tdcsfog_0 = pd.read_csv(test_tdcsfog[0])\ntest_tdcsfog_0","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.329818Z","iopub.execute_input":"2023-06-09T03:18:10.330233Z","iopub.status.idle":"2023-06-09T03:18:10.363952Z","shell.execute_reply.started":"2023-06-09T03:18:10.330200Z","shell.execute_reply":"2023-06-09T03:18:10.362580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n<br>\n<br>\n\n# EDA\n\n\n## Check Subject\n\n\n### `defog_metadata.Subject` & `tdcsfog_metadata.Subject`","metadata":{}},{"cell_type":"code","source":"print(\"defog subjects counts: \", defog_metadata.Subject.unique().shape)\nprint(\"tdcsfog subjects counts: \", tdcsfog_metadata.Subject.unique().shape)\n\nprint(\"Union len: \", len(set(tdcsfog_metadata.Subject.unique()) | set(defog_metadata.Subject.unique())))\nprint(\"Intersection len: \", len(set(tdcsfog_metadata.Subject.unique()) & set(defog_metadata.Subject.unique())))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.365508Z","iopub.execute_input":"2023-06-09T03:18:10.365881Z","iopub.status.idle":"2023-06-09T03:18:10.376524Z","shell.execute_reply.started":"2023-06-09T03:18:10.365850Z","shell.execute_reply":"2023-06-09T03:18:10.375055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Subjects counts: \", subjects.Subject.unique().shape)\n\nprint(\"Issubset: \", set(tdcsfog_metadata.Subject.unique()).issubset(set(subjects.Subject.unique())))\n\nprint()\nprint(\"Issubset: \", set(defog_metadata.Subject.unique()).issubset(set(subjects.Subject.unique())))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.378849Z","iopub.execute_input":"2023-06-09T03:18:10.380382Z","iopub.status.idle":"2023-06-09T03:18:10.392754Z","shell.execute_reply.started":"2023-06-09T03:18:10.380330Z","shell.execute_reply":"2023-06-09T03:18:10.391233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4>**Subject**s in the `defog_metadata.csv` (**45**) and those in the `tdcsfog_metadata.csv` (**62**) are no intersection at all. The Union of them takes up a proportion of **107/136** of the total number of the **Subject**s in the `subjects.csv` (**136**).</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n### `daily_metadata.Subject`","metadata":{}},{"cell_type":"code","source":"print(\"daily subjects counts: \", daily_metadata.Subject.unique().shape)\n\nprint(\"Issubset: \", set(defog_metadata.Subject.unique()).issubset(set(daily_metadata.Subject.unique())))\nprint(len(set(defog_metadata.Subject.unique()) & set(daily_metadata.Subject.unique())))\nprint()\n\n\nprint(\"Issubset: \", set(tdcsfog_metadata.Subject.unique()).issubset(set(daily_metadata.Subject.unique())))\nprint(len(set(tdcsfog_metadata.Subject.unique()) & set(daily_metadata.Subject.unique())))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.394631Z","iopub.execute_input":"2023-06-09T03:18:10.395082Z","iopub.status.idle":"2023-06-09T03:18:10.409574Z","shell.execute_reply.started":"2023-06-09T03:18:10.395046Z","shell.execute_reply":"2023-06-09T03:18:10.408083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Issubset: \", set(daily_metadata.Subject.unique()).issubset(set(subjects.Subject.unique())))\n\nset(defog_metadata.Subject.unique())   | \\\nset(tdcsfog_metadata.Subject.unique()) | \\\nset(daily_metadata.Subject.unique())   == set(subjects.Subject.unique())","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.411388Z","iopub.execute_input":"2023-06-09T03:18:10.414205Z","iopub.status.idle":"2023-06-09T03:18:10.440612Z","shell.execute_reply.started":"2023-06-09T03:18:10.414161Z","shell.execute_reply":"2023-06-09T03:18:10.438975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4> **Subject**s in the `daily_metadata.csv` is a subset of `subjects.csv`'s. And the union of **Subject**s of `daily_metadata.csv`, `defog_metadata.csv`, `tdcsfog_metadata.csv` forms the **Subject**s of `subject.csv`.</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n## Check Id\n\n### `tasks.Id` & `defog_metadata.Id` & `train/defog/` & `train/notype/`","metadata":{}},{"cell_type":"code","source":"print(\"len(defog_metadata.Id.unique()): \", len(defog_metadata.Id.unique()))\nprint(\"len(id_train_defog)            : \", len(id_train_defog))\nprint(\"len(tasks.Id.unique())         : \", len(tasks.Id.unique()))\n\nprint(\"Issubset: \", set(id_train_defog).issubset(set(defog_metadata.Id.unique())))\n\nprint(\"Equal   : \", set(tasks.Id.unique()) == set(defog_metadata.Id.unique()))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.442403Z","iopub.execute_input":"2023-06-09T03:18:10.442789Z","iopub.status.idle":"2023-06-09T03:18:10.461682Z","shell.execute_reply.started":"2023-06-09T03:18:10.442757Z","shell.execute_reply":"2023-06-09T03:18:10.460260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(set(id_train_notype) & set(id_train_defog))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.463833Z","iopub.execute_input":"2023-06-09T03:18:10.464982Z","iopub.status.idle":"2023-06-09T03:18:10.474264Z","shell.execute_reply.started":"2023-06-09T03:18:10.464923Z","shell.execute_reply":"2023-06-09T03:18:10.473226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(set(id_train_notype) & set(defog_metadata.Id.unique()))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.475847Z","iopub.execute_input":"2023-06-09T03:18:10.476258Z","iopub.status.idle":"2023-06-09T03:18:10.491319Z","shell.execute_reply.started":"2023-06-09T03:18:10.476213Z","shell.execute_reply":"2023-06-09T03:18:10.490379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(set(id_train_notype) | set(id_train_defog)) == set(defog_metadata.Id.unique())","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.493042Z","iopub.execute_input":"2023-06-09T03:18:10.493848Z","iopub.status.idle":"2023-06-09T03:18:10.516935Z","shell.execute_reply.started":"2023-06-09T03:18:10.493803Z","shell.execute_reply":"2023-06-09T03:18:10.515326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4>\n    \nConclusion:\n* tasks.Id.unique() = defog_metadata.Id.unique()\n* set(id_train_notype) | set(id_train_defog)) == set(defog_metadata.Id.unique()\n</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n### `tdcsfog_metadata.Id` & `train/tdcdfog/`","metadata":{}},{"cell_type":"code","source":"print(\"len(tdcsfog_metadata.Id.unique()): \", len(tdcsfog_metadata.Id.unique()))\nprint(\"len(id_train_tdcsfog)            :\", len(id_train_tdcsfog))\n\nset(tdcsfog_metadata.Id.unique()) == set(id_train_tdcsfog)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.519642Z","iopub.execute_input":"2023-06-09T03:18:10.520680Z","iopub.status.idle":"2023-06-09T03:18:10.539615Z","shell.execute_reply.started":"2023-06-09T03:18:10.520627Z","shell.execute_reply":"2023-06-09T03:18:10.538538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4>set(id_train_tdcsfog) == set(tdcsfog_metadata.Id.unique()).</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n### `daily_metadata.Id`  & `unlabeled/`","metadata":{}},{"cell_type":"code","source":"sorted(list(set(daily_metadata.Id.unique()))) == id_unlabeled","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.541428Z","iopub.execute_input":"2023-06-09T03:18:10.545004Z","iopub.status.idle":"2023-06-09T03:18:10.554200Z","shell.execute_reply.started":"2023-06-09T03:18:10.544964Z","shell.execute_reply":"2023-06-09T03:18:10.552666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Unique **ids** in the `daily_metadata.csv` are the same as those hidden in the `unlabeled/` subfolders.","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n### `events.Id` & `defog_metadata.Id` & `tdcsfog_metadata.Id` & `daily_metadata.Id`","metadata":{}},{"cell_type":"code","source":"print(\"len(events.Id.unique())        : \", len(events.Id.unique()))\nprint(\"len(defog_metadata.Id.unique()): \", len(defog_metadata.Id.unique()))\n\nprint(\"Issubset                       : \", set(defog_metadata.Id.unique()).issubset(set(events.Id.unique())))\n\nprint(\"Intersection len               : \", len(set(defog_metadata.Id.unique()) & set(events.Id.unique())))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.557009Z","iopub.execute_input":"2023-06-09T03:18:10.558571Z","iopub.status.idle":"2023-06-09T03:18:10.572152Z","shell.execute_reply.started":"2023-06-09T03:18:10.558525Z","shell.execute_reply":"2023-06-09T03:18:10.570764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"len(events.Id.unique())          : \", len(events.Id.unique()))\nprint(\"len(tdcsfog_metadata.Id.unique()): \", len(tdcsfog_metadata.Id.unique()))\n\nprint(\"Issubset        : \", set(tdcsfog_metadata.Id.unique()).issubset(set(events.Id.unique())))\n\nprint(\"Intersection len: \", len(set(tdcsfog_metadata.Id.unique()) & set(events.Id.unique())))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.573834Z","iopub.execute_input":"2023-06-09T03:18:10.575529Z","iopub.status.idle":"2023-06-09T03:18:10.589343Z","shell.execute_reply.started":"2023-06-09T03:18:10.575481Z","shell.execute_reply":"2023-06-09T03:18:10.587963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(set(defog_metadata.Id.unique()) & set(tdcsfog_metadata.Id.unique()))\n\nprint(len((set(defog_metadata.Id.unique()) | set(tdcsfog_metadata.Id.unique()))  &  set(events.Id.unique())))\n\n(set(tdcsfog_metadata.Id.unique()) & set(events.Id.unique())  | \\\n set(defog_metadata.Id.unique())   & set(events.Id.unique())) == set(events.Id.unique())","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.591079Z","iopub.execute_input":"2023-06-09T03:18:10.591476Z","iopub.status.idle":"2023-06-09T03:18:10.610912Z","shell.execute_reply.started":"2023-06-09T03:18:10.591444Z","shell.execute_reply":"2023-06-09T03:18:10.609266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>","metadata":{}},{"cell_type":"code","source":"print(\"len(events.Id.unique())        : \", len(events.Id.unique()))\nprint(\"len(daily_metadata.Id.unique()): \", len(daily_metadata.Id.unique()))\n\nprint(\"Issubset        : \", set(daily_metadata.Id.unique()).issubset(set(events.Id.unique())))\n\nprint(\"Intersection len: \", len(set(daily_metadata.Id.unique()) & set(events.Id.unique())))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.612734Z","iopub.execute_input":"2023-06-09T03:18:10.613605Z","iopub.status.idle":"2023-06-09T03:18:10.624919Z","shell.execute_reply.started":"2023-06-09T03:18:10.613570Z","shell.execute_reply":"2023-06-09T03:18:10.623529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<font color=red size=4> **ids** in `defog_metadata.csv` and in `tdcsfog_metadata.csv` are mutually exclusive and they make up **ids** in `events.csv`. **ids** in `daily_metadata.csv` have no any relation with those in `events.csv`.</font>","metadata":{}},{"cell_type":"markdown","source":"<br>\n\n### `defog_metadata.Id` & `tdcsfog_metadata.Id` & `daily_metadata.Id`","metadata":{}},{"cell_type":"code","source":"len(set(defog_metadata.Id.unique()) & set(daily_metadata.Id.unique()))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.626681Z","iopub.execute_input":"2023-06-09T03:18:10.627618Z","iopub.status.idle":"2023-06-09T03:18:10.638825Z","shell.execute_reply.started":"2023-06-09T03:18:10.627575Z","shell.execute_reply":"2023-06-09T03:18:10.637545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(set(tdcsfog_metadata.Id.unique()) & set(daily_metadata.Id.unique()))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.641010Z","iopub.execute_input":"2023-06-09T03:18:10.641846Z","iopub.status.idle":"2023-06-09T03:18:10.650079Z","shell.execute_reply.started":"2023-06-09T03:18:10.641810Z","shell.execute_reply":"2023-06-09T03:18:10.649236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check Nan","metadata":{}},{"cell_type":"code","source":"defog_metadata.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.651678Z","iopub.execute_input":"2023-06-09T03:18:10.652325Z","iopub.status.idle":"2023-06-09T03:18:10.664397Z","shell.execute_reply.started":"2023-06-09T03:18:10.652292Z","shell.execute_reply":"2023-06-09T03:18:10.663238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_metadata.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.665684Z","iopub.execute_input":"2023-06-09T03:18:10.666055Z","iopub.status.idle":"2023-06-09T03:18:10.681469Z","shell.execute_reply.started":"2023-06-09T03:18:10.666026Z","shell.execute_reply":"2023-06-09T03:18:10.680134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.682843Z","iopub.execute_input":"2023-06-09T03:18:10.683295Z","iopub.status.idle":"2023-06-09T03:18:10.702065Z","shell.execute_reply.started":"2023-06-09T03:18:10.683261Z","shell.execute_reply":"2023-06-09T03:18:10.700303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.704943Z","iopub.execute_input":"2023-06-09T03:18:10.705931Z","iopub.status.idle":"2023-06-09T03:18:10.718392Z","shell.execute_reply.started":"2023-06-09T03:18:10.705885Z","shell.execute_reply":"2023-06-09T03:18:10.717030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.719934Z","iopub.execute_input":"2023-06-09T03:18:10.720317Z","iopub.status.idle":"2023-06-09T03:18:10.732549Z","shell.execute_reply.started":"2023-06-09T03:18:10.720285Z","shell.execute_reply":"2023-06-09T03:18:10.730969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_metadata.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T03:18:10.734160Z","iopub.execute_input":"2023-06-09T03:18:10.734735Z","iopub.status.idle":"2023-06-09T03:18:10.744033Z","shell.execute_reply.started":"2023-06-09T03:18:10.734675Z","shell.execute_reply":"2023-06-09T03:18:10.743053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<br>\n<br>\n<br>\n\n# Some Conclusions\n\n<font size=4>\n\n* `subjects.csv`: It shows that there are **136** different **Subject**s, some of them have more than one records in the `subjects.csv` and this is because they visit at least twice.\n    \n* **Subject**s in the `defog_metadata.csv` (**45**) and those in the `tdcsfog_metadata.csv` (62) are no intersection at all. The Union of them takes up a proportion of **107/136** of the total number of the **Subject**s in the `subjects.csv` (**136**).\n    \n* **Subject**s in the `daily_metadata.csv` is a subset of `subjects.csv`'s. And the union of **Subject**s of `daily_metadata.csv`, `defog_metadata.csv`, `tdcsfog_metadata.csv` forms the **Subject**s of `subject.csv`.\n    \n    \n<br>\n    \n* `tasks.csv`: It is said that there were seven **tasks types** in the DeFOG protocol. Here we can find that seven types expand to 31 kinds totally.\n\n<br>\n    \n* The **ids** hidden in `train/defog/` and `train/notype/` are mutually exclusive.\n* `set(id_train_notype) | set(id_train_defog))` equals to `set(defog_metadata.Id.unique()`.\n* `tasks.Id.unique()` equals to `defog_metadata.Id.unique()`.\n\n    \n* `set(id_train_tdcsfog)` equals to `set(tdcsfog_metadata.Id.unique())`.\n\n    \n* **ids** in `defog_metadata.csv` and in `tdcsfog_metadata.csv` are mutually exclusive and they make up **ids** in `events.csv`. \n* **ids** in `daily_metadata.csv` have no any relation with those in `events.csv`.\n\n    \n* Unique **ids** in the `daily_metadata.csv` are the same as those hidden in the `unlabeled/` subfolders.\n\n</font>","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}