{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Basic Data Exploration\n[Aviv Levi](https://www.linkedin.com/in/aviv-levi815/) - March 2023\n\n----------\n<b>'Parkinson's Freezing of Gait Prediction'</b>\n\nThis is my first EDA for this competition.\n\nI decided to explore the metadata csv's before the actual 3D data.\nBasically, for every csv, I checked if the data is one->one ratio or one->many, \nlater exploring each column - and maybe exploring some obvious derived features.\n\n# TL;DR\n1. <b>daily_metadata</b>. 1->1 ratio. most subjects visit once. || *no nan values*\n\n2. <b>defog metadata</b>. 1-> many ratio, subjects visit once or twice roughly 50/50, and are on medications 50% of the time || *no nan values*\n\n3. <b>events</b>. Most events' duration is under 7 seconds, but some are much longer Most events were of type Turn. || **There are over 1000 events with unknown Type&Kinetic**\nMost events involved movement \nthere is one id with way more data than the rest - **60dfb26b2c** \n\n4. <b>subjects</b>. || for a subject with nan values in any column - it does not have a record without a nan value,\nThere are more male subjects.\nThe subject's age is ~Normally distributed\nMean ~ 67, std ~ 8 **There seems to be a predictive line from the score without medication and with medication.** || || nans: Visit(62) , UPDRSIII_On(1), UPDRSIII_Off(41) \n\n5. <b>tasks</b>.There are 20 tasks on average per Id, longest Task is Rest1, also note Turning-DT is pretty long.\nUsually, subjects have 2-4 visits.\nLooks like there is a large group of subjects that always take medications and another large group that takes medications 50% of the time || with no nans\n","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport plotly.express as px","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_m = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/daily_metadata.csv\")\ndefog_m = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/defog_metadata.csv\")\ntdcsfog_m = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/tdcsfog_metadata.csv\")\nevents = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/events.csv\")\nss = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/sample_submission.csv\")\nsubjects = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/subjects.csv\")\ntasks = pd.read_csv(\"/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/tasks.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:12.896892Z","iopub.execute_input":"2023-03-20T15:10:12.897270Z","iopub.status.idle":"2023-03-20T15:10:13.273735Z","shell.execute_reply.started":"2023-03-20T15:10:12.897234Z","shell.execute_reply":"2023-03-20T15:10:13.272489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# looking into daily_metadata","metadata":{}},{"cell_type":"code","source":"daily_m.head(1)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.275040Z","iopub.execute_input":"2023-03-20T15:10:13.275360Z","iopub.status.idle":"2023-03-20T15:10:13.304875Z","shell.execute_reply.started":"2023-03-20T15:10:13.275331Z","shell.execute_reply":"2023-03-20T15:10:13.303536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_m.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.307276Z","iopub.execute_input":"2023-03-20T15:10:13.307898Z","iopub.status.idle":"2023-03-20T15:10:13.318183Z","shell.execute_reply.started":"2023-03-20T15:10:13.307850Z","shell.execute_reply":"2023-03-20T15:10:13.316792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_m.shape[0], daily_m['Id'].nunique(), daily_m[\"Subject\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.320062Z","iopub.execute_input":"2023-03-20T15:10:13.320550Z","iopub.status.idle":"2023-03-20T15:10:13.337924Z","shell.execute_reply.started":"2023-03-20T15:10:13.320502Z","shell.execute_reply":"2023-03-20T15:10:13.336572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_m.Visit.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.339103Z","iopub.execute_input":"2023-03-20T15:10:13.339497Z","iopub.status.idle":"2023-03-20T15:10:13.357032Z","shell.execute_reply.started":"2023-03-20T15:10:13.339461Z","shell.execute_reply":"2023-03-20T15:10:13.355828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_m[\"Beginning of recording [00:00-23:59]\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.358868Z","iopub.execute_input":"2023-03-20T15:10:13.359331Z","iopub.status.idle":"2023-03-20T15:10:13.372578Z","shell.execute_reply.started":"2023-03-20T15:10:13.359283Z","shell.execute_reply":"2023-03-20T15:10:13.371501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Looking into defog metadata","metadata":{}},{"cell_type":"code","source":"defog_m.head(1)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.373723Z","iopub.execute_input":"2023-03-20T15:10:13.374090Z","iopub.status.idle":"2023-03-20T15:10:13.391315Z","shell.execute_reply.started":"2023-03-20T15:10:13.374028Z","shell.execute_reply":"2023-03-20T15:10:13.390379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_m.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.392311Z","iopub.execute_input":"2023-03-20T15:10:13.392669Z","iopub.status.idle":"2023-03-20T15:10:13.407053Z","shell.execute_reply.started":"2023-03-20T15:10:13.392636Z","shell.execute_reply":"2023-03-20T15:10:13.405566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_m.shape[0], defog_m[\"Id\"].nunique(), defog_m[\"Subject\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:40:21.679878Z","iopub.execute_input":"2023-03-20T15:40:21.680861Z","iopub.status.idle":"2023-03-20T15:40:21.690471Z","shell.execute_reply.started":"2023-03-20T15:40:21.680799Z","shell.execute_reply":"2023-03-20T15:40:21.689266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_m[\"Visit\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.424819Z","iopub.execute_input":"2023-03-20T15:10:13.425143Z","iopub.status.idle":"2023-03-20T15:10:13.439589Z","shell.execute_reply.started":"2023-03-20T15:10:13.425113Z","shell.execute_reply":"2023-03-20T15:10:13.438291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_m[\"Medication\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.441281Z","iopub.execute_input":"2023-03-20T15:10:13.442441Z","iopub.status.idle":"2023-03-20T15:10:13.453193Z","shell.execute_reply.started":"2023-03-20T15:10:13.442374Z","shell.execute_reply":"2023-03-20T15:10:13.451661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_m[\"Subject\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.455093Z","iopub.execute_input":"2023-03-20T15:10:13.456240Z","iopub.status.idle":"2023-03-20T15:10:13.469358Z","shell.execute_reply.started":"2023-03-20T15:10:13.456188Z","shell.execute_reply":"2023-03-20T15:10:13.468136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# # Looking into events\n## TL ; DR\n* ### Most event's duration is under 7 seconds, but some are much longer\n* ### Most events were of type Turn.\n* ### Most events involved movement","metadata":{}},{"cell_type":"code","source":"events.head(1)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.471196Z","iopub.execute_input":"2023-03-20T15:10:13.472339Z","iopub.status.idle":"2023-03-20T15:10:13.488578Z","shell.execute_reply.started":"2023-03-20T15:10:13.472291Z","shell.execute_reply":"2023-03-20T15:10:13.487335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.490064Z","iopub.execute_input":"2023-03-20T15:10:13.490428Z","iopub.status.idle":"2023-03-20T15:10:13.500509Z","shell.execute_reply.started":"2023-03-20T15:10:13.490376Z","shell.execute_reply":"2023-03-20T15:10:13.499291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### There are over 1000 events with unknown Type&Kinetic","metadata":{}},{"cell_type":"code","source":"events.shape[0], events[\"Id\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.502156Z","iopub.execute_input":"2023-03-20T15:10:13.502630Z","iopub.status.idle":"2023-03-20T15:10:13.516762Z","shell.execute_reply.started":"2023-03-20T15:10:13.502580Z","shell.execute_reply":"2023-03-20T15:10:13.515553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### mini observation:  events has 1-> many (Id -> rows)","metadata":{}},{"cell_type":"code","source":"value_counts = events.Id.value_counts().values\npx.bar(value_counts, labels = {'value': 'count'})","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:13.517963Z","iopub.execute_input":"2023-03-20T15:10:13.518419Z","iopub.status.idle":"2023-03-20T15:10:15.165986Z","shell.execute_reply.started":"2023-03-20T15:10:13.518356Z","shell.execute_reply":"2023-03-20T15:10:15.164658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### there is one id with way more data than the rest - 60dfb26b2c    ","metadata":{}},{"cell_type":"code","source":"events.Init.sort_values().reset_index(drop=True).plot(kind=\"hist\", bins=50, figsize=(10,3))","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:15.168203Z","iopub.execute_input":"2023-03-20T15:10:15.168594Z","iopub.status.idle":"2023-03-20T15:10:15.508551Z","shell.execute_reply.started":"2023-03-20T15:10:15.168558Z","shell.execute_reply":"2023-03-20T15:10:15.507261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### low Init value is very common","metadata":{}},{"cell_type":"code","source":"events.Completion.sort_values().reset_index(drop=True).plot(kind=\"hist\", bins=50, figsize=(10,3))","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:15.510119Z","iopub.execute_input":"2023-03-20T15:10:15.510699Z","iopub.status.idle":"2023-03-20T15:10:15.770544Z","shell.execute_reply.started":"2023-03-20T15:10:15.510651Z","shell.execute_reply":"2023-03-20T15:10:15.769655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_hist(df, col,  color = None):\n    f = px.histogram(df, x=col, color = color)\n    f.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:15.772219Z","iopub.execute_input":"2023-03-20T15:10:15.772613Z","iopub.status.idle":"2023-03-20T15:10:15.778296Z","shell.execute_reply.started":"2023-03-20T15:10:15.772577Z","shell.execute_reply":"2023-03-20T15:10:15.776912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events[\"duration\"] = events[\"Completion\"] - events[\"Init\"]\nplot_hist(events, \"duration\")\n","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:15.779888Z","iopub.execute_input":"2023-03-20T15:10:15.780859Z","iopub.status.idle":"2023-03-20T15:10:16.022285Z","shell.execute_reply.started":"2023-03-20T15:10:15.780817Z","shell.execute_reply":"2023-03-20T15:10:16.021352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* ### Most events duration is under 7 seconds, but some are much longer\n","metadata":{}},{"cell_type":"code","source":"events.Type.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.023632Z","iopub.execute_input":"2023-03-20T15:10:16.024533Z","iopub.status.idle":"2023-03-20T15:10:16.034798Z","shell.execute_reply.started":"2023-03-20T15:10:16.024489Z","shell.execute_reply":"2023-03-20T15:10:16.033508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* ### Most events were of type Turn.\n","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"events.Kinetic.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.036744Z","iopub.execute_input":"2023-03-20T15:10:16.037358Z","iopub.status.idle":"2023-03-20T15:10:16.052754Z","shell.execute_reply.started":"2023-03-20T15:10:16.037292Z","shell.execute_reply":"2023-03-20T15:10:16.051804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* ### Mosts events involved movement","metadata":{}},{"cell_type":"markdown","source":"# Looking into Subjects","metadata":{}},{"cell_type":"code","source":"subjects.head(1)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.054104Z","iopub.execute_input":"2023-03-20T15:10:16.055368Z","iopub.status.idle":"2023-03-20T15:10:16.073109Z","shell.execute_reply.started":"2023-03-20T15:10:16.055327Z","shell.execute_reply":"2023-03-20T15:10:16.071872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.shape[0], subjects['Subject'].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.074569Z","iopub.execute_input":"2023-03-20T15:10:16.075648Z","iopub.status.idle":"2023-03-20T15:10:16.091141Z","shell.execute_reply.started":"2023-03-20T15:10:16.075607Z","shell.execute_reply":"2023-03-20T15:10:16.089854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects[\"Subject\"].value_counts().max()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.093125Z","iopub.execute_input":"2023-03-20T15:10:16.093659Z","iopub.status.idle":"2023-03-20T15:10:16.106569Z","shell.execute_reply.started":"2023-03-20T15:10:16.093578Z","shell.execute_reply":"2023-03-20T15:10:16.105346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.Visit.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.108139Z","iopub.execute_input":"2023-03-20T15:10:16.108879Z","iopub.status.idle":"2023-03-20T15:10:16.128223Z","shell.execute_reply.started":"2023-03-20T15:10:16.108840Z","shell.execute_reply":"2023-03-20T15:10:16.127312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Some Subjects appeare twice","metadata":{}},{"cell_type":"code","source":"subjects.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.135377Z","iopub.execute_input":"2023-03-20T15:10:16.135852Z","iopub.status.idle":"2023-03-20T15:10:16.146939Z","shell.execute_reply.started":"2023-03-20T15:10:16.135786Z","shell.execute_reply":"2023-03-20T15:10:16.145514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visit_unknown_id = subjects.query(\"Visit.isnull()\", engine=\"python\").Subject\nsubjects.query(\"Subject in @visit_unknown_id\").Visit.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.149797Z","iopub.execute_input":"2023-03-20T15:10:16.150710Z","iopub.status.idle":"2023-03-20T15:10:16.171305Z","shell.execute_reply.started":"2023-03-20T15:10:16.150655Z","shell.execute_reply":"2023-03-20T15:10:16.170379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"UPDRSIII_Off_unknown_id = subjects.query(\"UPDRSIII_Off.isnull()\", engine=\"python\").Subject\nsubjects.query(\"Subject in @UPDRSIII_Off_unknown_id\").UPDRSIII_Off.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.172935Z","iopub.execute_input":"2023-03-20T15:10:16.173588Z","iopub.status.idle":"2023-03-20T15:10:16.187734Z","shell.execute_reply.started":"2023-03-20T15:10:16.173549Z","shell.execute_reply":"2023-03-20T15:10:16.186795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"UPDRSIII_On_unknown_id = subjects.query(\"UPDRSIII_On.isnull()\", engine=\"python\").Subject\nsubjects.query(\"Subject in @UPDRSIII_On_unknown_id\").UPDRSIII_On.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.189081Z","iopub.execute_input":"2023-03-20T15:10:16.189686Z","iopub.status.idle":"2023-03-20T15:10:16.205376Z","shell.execute_reply.started":"2023-03-20T15:10:16.189649Z","shell.execute_reply":"2023-03-20T15:10:16.204070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### for a subject with nan values in any column - it does not have a record without a nan value.","metadata":{}},{"cell_type":"code","source":"subjects.Sex.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.207090Z","iopub.execute_input":"2023-03-20T15:10:16.207648Z","iopub.status.idle":"2023-03-20T15:10:16.220603Z","shell.execute_reply.started":"2023-03-20T15:10:16.207599Z","shell.execute_reply":"2023-03-20T15:10:16.219318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### There are more male subjects.","metadata":{}},{"cell_type":"code","source":"plot_hist(subjects,'Age', color=\"Sex\")","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.222260Z","iopub.execute_input":"2023-03-20T15:10:16.223192Z","iopub.status.idle":"2023-03-20T15:10:16.293086Z","shell.execute_reply.started":"2023-03-20T15:10:16.223143Z","shell.execute_reply":"2023-03-20T15:10:16.291872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects.Age.describe()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.294832Z","iopub.execute_input":"2023-03-20T15:10:16.295389Z","iopub.status.idle":"2023-03-20T15:10:16.309916Z","shell.execute_reply.started":"2023-03-20T15:10:16.295341Z","shell.execute_reply":"2023-03-20T15:10:16.308366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Subject's age is ~Normally distributed\n#### * Mean ~ 67, std ~ 8\n\n### There are more male subjects","metadata":{}},{"cell_type":"code","source":"subjects[\"age_Dx\"] = subjects[\"Age\"] - subjects[\"YearsSinceDx\"]\nplot_hist(subjects,'age_Dx', color=\"Sex\")","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.311548Z","iopub.execute_input":"2023-03-20T15:10:16.311927Z","iopub.status.idle":"2023-03-20T15:10:16.376178Z","shell.execute_reply.started":"2023-03-20T15:10:16.311892Z","shell.execute_reply":"2023-03-20T15:10:16.375215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"medication_effect_on_off = subjects[[\"UPDRSIII_On\", \"UPDRSIII_Off\"]].dropna()\nmedication_effect = medication_effect_on_off['UPDRSIII_Off'] - medication_effect_on_off[\"UPDRSIII_On\"]\nmedication_effect_on_off.plot(kind=\"scatter\", x=\"UPDRSIII_On\", y=\"UPDRSIII_Off\")","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.379951Z","iopub.execute_input":"2023-03-20T15:10:16.380294Z","iopub.status.idle":"2023-03-20T15:10:16.633587Z","shell.execute_reply.started":"2023-03-20T15:10:16.380260Z","shell.execute_reply":"2023-03-20T15:10:16.632273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  There seems to be a predictive line from the score without medication and with medication.\n","metadata":{}},{"cell_type":"code","source":"plot_hist(subjects, col=\"NFOGQ\")","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:16.635604Z","iopub.execute_input":"2023-03-20T15:10:16.636000Z","iopub.status.idle":"2023-03-20T15:10:16.694644Z","shell.execute_reply.started":"2023-03-20T15:10:16.635966Z","shell.execute_reply":"2023-03-20T15:10:16.693261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I'm not sure if this column is useful - it was used to show the Reliability of the new freezing of gait.","metadata":{}},{"cell_type":"markdown","source":"# Looking into tasks","metadata":{}},{"cell_type":"code","source":"tasks.head(1)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:10:34.755006Z","iopub.execute_input":"2023-03-20T15:10:34.755483Z","iopub.status.idle":"2023-03-20T15:10:34.768993Z","shell.execute_reply.started":"2023-03-20T15:10:34.755433Z","shell.execute_reply":"2023-03-20T15:10:34.767707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:51:20.427280Z","iopub.execute_input":"2023-03-20T15:51:20.427844Z","iopub.status.idle":"2023-03-20T15:51:20.442058Z","shell.execute_reply.started":"2023-03-20T15:51:20.427791Z","shell.execute_reply":"2023-03-20T15:51:20.440509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tasks.groupby(\"Id\")[\"Id\"].agg(\"count\").describe()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:11:37.738569Z","iopub.execute_input":"2023-03-20T15:11:37.738954Z","iopub.status.idle":"2023-03-20T15:11:37.754115Z","shell.execute_reply.started":"2023-03-20T15:11:37.738918Z","shell.execute_reply":"2023-03-20T15:11:37.752680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### There are 20 tasks on average per Id","metadata":{}},{"cell_type":"code","source":"tasks[\"Duration\"] = tasks[\"End\"] - tasks[\"Begin\"]\ntasks.groupby(\"Task\")[\"Duration\"].agg(\"mean\").sort_values()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:15:10.854880Z","iopub.execute_input":"2023-03-20T15:15:10.856100Z","iopub.status.idle":"2023-03-20T15:15:10.872318Z","shell.execute_reply.started":"2023-03-20T15:15:10.856032Z","shell.execute_reply":"2023-03-20T15:15:10.870044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Longest Task is Rest1, also note Turning-DT is pretty long","metadata":{}},{"cell_type":"code","source":"plot_hist(tasks, col=\"Duration\")","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:13:25.319859Z","iopub.execute_input":"2023-03-20T15:13:25.320290Z","iopub.status.idle":"2023-03-20T15:13:25.379736Z","shell.execute_reply.started":"2023-03-20T15:13:25.320251Z","shell.execute_reply":"2023-03-20T15:13:25.378484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_m.head(1)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:16:55.897572Z","iopub.execute_input":"2023-03-20T15:16:55.898976Z","iopub.status.idle":"2023-03-20T15:16:55.912651Z","shell.execute_reply.started":"2023-03-20T15:16:55.898920Z","shell.execute_reply":"2023-03-20T15:16:55.911076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_m.shape[0], tdcsfog_m[\"Id\"].nunique(), tdcsfog_m[\"Subject\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:17:37.307572Z","iopub.execute_input":"2023-03-20T15:17:37.309114Z","iopub.status.idle":"2023-03-20T15:17:37.325533Z","shell.execute_reply.started":"2023-03-20T15:17:37.309028Z","shell.execute_reply":"2023-03-20T15:17:37.323589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Subject -> Vist || 1 -> many","metadata":{}},{"cell_type":"code","source":"tdcsfog_m.Visit.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:18:28.955494Z","iopub.execute_input":"2023-03-20T15:18:28.956021Z","iopub.status.idle":"2023-03-20T15:18:28.966807Z","shell.execute_reply.started":"2023-03-20T15:18:28.955969Z","shell.execute_reply":"2023-03-20T15:18:28.965806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Usually subjects have 2-4 visits","metadata":{}},{"cell_type":"code","source":"tdcsfog_m[\"Medication_numeric\"] = np.where(tdcsfog_m[\"Medication\"] == \"on\", 1, 0)","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:20:58.117019Z","iopub.execute_input":"2023-03-20T15:20:58.117472Z","iopub.status.idle":"2023-03-20T15:20:58.124636Z","shell.execute_reply.started":"2023-03-20T15:20:58.117427Z","shell.execute_reply":"2023-03-20T15:20:58.123518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"medication_fraction = tdcsfog_m.groupby('Subject')[\"Medication_numeric\"].agg(\"mean\").reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:22:28.741224Z","iopub.execute_input":"2023-03-20T15:22:28.741636Z","iopub.status.idle":"2023-03-20T15:22:28.751256Z","shell.execute_reply.started":"2023-03-20T15:22:28.741601Z","shell.execute_reply":"2023-03-20T15:22:28.749790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(medication_fraction, 'Medication_numeric')","metadata":{"execution":{"iopub.status.busy":"2023-03-20T15:22:48.694206Z","iopub.execute_input":"2023-03-20T15:22:48.695342Z","iopub.status.idle":"2023-03-20T15:22:48.752573Z","shell.execute_reply.started":"2023-03-20T15:22:48.695293Z","shell.execute_reply":"2023-03-20T15:22:48.751439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Looks like there is a large group of subjects that always take medications and another large group that takes medications 50% of the time","metadata":{}}]}