{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nThe goal of this competition is to **detect freezing of gait (FOG)**, a debilitating symptom that afflicts many people **with Parkinson’s disease**. It is requred to **develop a machine learning model trained on data collected from a wearable 3D lower back sensor** to better understand **when and why FOG episodes occur**.","metadata":{"papermill":{"duration":0.026599,"end_time":"2023-05-14T06:28:35.319714","exception":false,"start_time":"2023-05-14T06:28:35.293115","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Import Libraries","metadata":{"papermill":{"duration":0.024773,"end_time":"2023-05-14T06:28:35.368114","exception":false,"start_time":"2023-05-14T06:28:35.343341","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom scipy import stats as st\nimport scipy as sp","metadata":{"papermill":{"duration":1.340262,"end_time":"2023-05-14T06:28:36.731675","exception":false,"start_time":"2023-05-14T06:28:35.391413","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:00.986917Z","iopub.execute_input":"2023-05-28T17:14:00.988085Z","iopub.status.idle":"2023-05-28T17:14:02.521509Z","shell.execute_reply.started":"2023-05-28T17:14:00.988039Z","shell.execute_reply":"2023-05-28T17:14:02.520280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The **tDCS FOG (tdcsfog)** dataset, comprising data series collected **in the lab**, as subjects completed a FOG-provoking protocol.\n- The **DeFOG (defog)** dataset, comprising data series collected **in the subject's home**, as subjects completed a FOG-provoking protocol","metadata":{}},{"cell_type":"markdown","source":"# Read CSV Files\n\nThe train dataset is made up of the defog, notype, and tdcsfog folders. **The tdcsfog folder includes** more than 800 csv files and accounts for **the majority of the information contained in the whole dataset for this competition**. Moreover, csv files in the edcsfog folder include annotation: StartHesitation, Turn, and Walking.","metadata":{"papermill":{"duration":0.023554,"end_time":"2023-05-14T06:28:36.7803","exception":false,"start_time":"2023-05-14T06:28:36.756746","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# one tdcsfog file\ntdcsfog_003f117e14 = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/train/tdcsfog/003f117e14.csv')\ntdcsfog_003f117e14.head(5)","metadata":{"papermill":{"duration":0.093859,"end_time":"2023-05-14T06:28:36.897803","exception":false,"start_time":"2023-05-14T06:28:36.803944","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.525774Z","iopub.execute_input":"2023-05-28T17:14:02.526552Z","iopub.status.idle":"2023-05-28T17:14:02.597826Z","shell.execute_reply.started":"2023-05-28T17:14:02.526508Z","shell.execute_reply":"2023-05-28T17:14:02.596999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_003f117e14.tail(5)","metadata":{"papermill":{"duration":0.043739,"end_time":"2023-05-14T06:28:36.965448","exception":false,"start_time":"2023-05-14T06:28:36.921709","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.599347Z","iopub.execute_input":"2023-05-28T17:14:02.599996Z","iopub.status.idle":"2023-05-28T17:14:02.611904Z","shell.execute_reply.started":"2023-05-28T17:14:02.599941Z","shell.execute_reply":"2023-05-28T17:14:02.611044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This looks like a table of motion sensor data, possibly from a wearable device, with each row representing a particular moment in time (as indicated by the \"Time\" column) and containing measurements of acceleration in three dimensions (**\"AccV\", \"AccML\", \"AccAP\"**). **The values in these columns indicate the acceleration of the device along the vertical, medial-lateral, and anterior-posterior axes, respectively.**\n\n**The \"StartHesitation\", \"Turn\", and \"Walking\" columns appear to be binary variables indicating whether or not the corresponding activity is taking place at a given moment in time.** However, in this table, all values in these columns are 0, so it is not possible to predict the status of these variables based on the data in this table alone.","metadata":{"papermill":{"duration":0.02441,"end_time":"2023-05-14T06:28:37.014642","exception":false,"start_time":"2023-05-14T06:28:36.990232","status":"completed"},"tags":[]}},{"cell_type":"code","source":"tdcsfog_003f117e14.describe()","metadata":{"papermill":{"duration":0.081188,"end_time":"2023-05-14T06:28:37.120626","exception":false,"start_time":"2023-05-14T06:28:37.039438","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.614383Z","iopub.execute_input":"2023-05-28T17:14:02.614971Z","iopub.status.idle":"2023-05-28T17:14:02.666929Z","shell.execute_reply.started":"2023-05-28T17:14:02.614926Z","shell.execute_reply":"2023-05-28T17:14:02.665708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The \"mean\", \"std\", \"min\", \"25%\", \"50%\", \"75%\", and \"max\" values for each variable provide information about the distribution and range of the data. For example, the mean value for \"AccV\" is -9.15, indicating that the device was tilted slightly downwards on average. The standard deviation of 1.38 for \"AccV\" suggests that the device orientation varied quite a bit over time. The maximum and minimum values for each variable provide an idea of the range of motion that was captured by the sensor over the course of the data collection period.\n\nHere, **only Turn has the max value of 1**. Thus, **only a turn event occurred** in this experiment.","metadata":{"papermill":{"duration":0.024087,"end_time":"2023-05-14T06:28:37.170636","exception":false,"start_time":"2023-05-14T06:28:37.146549","status":"completed"},"tags":[]}},{"cell_type":"code","source":"tdcsfog_003f117e14[tdcsfog_003f117e14['Turn'] == 1]","metadata":{"papermill":{"duration":0.049704,"end_time":"2023-05-14T06:28:37.245875","exception":false,"start_time":"2023-05-14T06:28:37.196171","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.668470Z","iopub.execute_input":"2023-05-28T17:14:02.670674Z","iopub.status.idle":"2023-05-28T17:14:02.693768Z","shell.execute_reply.started":"2023-05-28T17:14:02.670637Z","shell.execute_reply":"2023-05-28T17:14:02.692705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The sensor data was collected during this period of time (from 1103 to 1890) when the subject was making turns.**","metadata":{"papermill":{"duration":0.024466,"end_time":"2023-05-14T06:28:37.295047","exception":false,"start_time":"2023-05-14T06:28:37.270581","status":"completed"},"tags":[]}},{"cell_type":"code","source":"tdcsfog_003f117e14.info()","metadata":{"papermill":{"duration":0.048689,"end_time":"2023-05-14T06:28:37.369083","exception":false,"start_time":"2023-05-14T06:28:37.320394","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.695530Z","iopub.execute_input":"2023-05-28T17:14:02.696266Z","iopub.status.idle":"2023-05-28T17:14:02.720637Z","shell.execute_reply.started":"2023-05-28T17:14:02.696222Z","shell.execute_reply":"2023-05-28T17:14:02.719369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we **normalize \"Time\"**.","metadata":{"papermill":{"duration":0.024676,"end_time":"2023-05-14T06:28:37.419157","exception":false,"start_time":"2023-05-14T06:28:37.394481","status":"completed"},"tags":[]}},{"cell_type":"code","source":"tdcsfog_003f117e14.Time = tdcsfog_003f117e14.Time / (len(tdcsfog_003f117e14) -1)\ntdcsfog_003f117e14","metadata":{"papermill":{"duration":0.048979,"end_time":"2023-05-14T06:28:37.493508","exception":false,"start_time":"2023-05-14T06:28:37.444529","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.722635Z","iopub.execute_input":"2023-05-28T17:14:02.723556Z","iopub.status.idle":"2023-05-28T17:14:02.743061Z","shell.execute_reply.started":"2023-05-28T17:14:02.723501Z","shell.execute_reply":"2023-05-28T17:14:02.741702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# one defog file\ndefog_02ea782681 = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/train/defog/02ea782681.csv')\ndefog_02ea782681","metadata":{"papermill":{"duration":0.320253,"end_time":"2023-05-14T06:28:38.562489","exception":false,"start_time":"2023-05-14T06:28:38.242236","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:02.745022Z","iopub.execute_input":"2023-05-28T17:14:02.745723Z","iopub.status.idle":"2023-05-28T17:14:03.132614Z","shell.execute_reply.started":"2023-05-28T17:14:02.745683Z","shell.execute_reply":"2023-05-28T17:14:03.131746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we **normalize \"Time\"**.","metadata":{"papermill":{"duration":0.027918,"end_time":"2023-05-14T06:28:38.61869","exception":false,"start_time":"2023-05-14T06:28:38.590772","status":"completed"},"tags":[]}},{"cell_type":"code","source":"defog_02ea782681.Time = defog_02ea782681.Time / (len(defog_02ea782681) - 1)\ndefog_02ea782681","metadata":{"papermill":{"duration":0.055216,"end_time":"2023-05-14T06:28:38.700193","exception":false,"start_time":"2023-05-14T06:28:38.644977","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:03.134263Z","iopub.execute_input":"2023-05-28T17:14:03.134967Z","iopub.status.idle":"2023-05-28T17:14:03.160633Z","shell.execute_reply.started":"2023-05-28T17:14:03.134910Z","shell.execute_reply":"2023-05-28T17:14:03.159381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This table appears to contain sensor data and annotations related to a task performed by a subject. The columns **\"Time\", \"AccV\", \"AccML\", and \"AccAP\" likely contain motion sensor data similar to the previous examples**.\n\nThe columns \"StartHesitation\", \"Turn\", and \"Walking\" appear to correspond to whether the subject is performing certain activities at each time point, similar to the previous example. However, in this case, there are **additional columns \"Valid\" and \"Task\" that provide information about the validity of the annotations and the task performed**.\n\nThe **\"Valid\"** column appears to indicate whether the annotations for a given time point are considered to be **reliable or not**. The **\"Task\"** column appears to indicate whether the task was performed during a given time point. Portions of the data marked **\"False\"** in this column should be considered **unannotated and not used in analysis**.\n\nIt's possible that this dataset was collected in the context of studying movement disorders or other neurological conditions, where accurate annotation of sensor data is important for clinical diagnosis and treatment planning. **This dataset could potentially be used to develop machine learning models to predict task performance or identify patterns of sensor data associated with certain movements or behaviors.** However, given the additional complexity introduced by the \"Valid\" and \"Task\" columns, **careful attention** would need to be paid to cleaning and preprocessing the data before using it for analysis or modeling.\n\n**Next, let's see its test file.**","metadata":{"papermill":{"duration":0.026487,"end_time":"2023-05-14T06:28:38.754424","exception":false,"start_time":"2023-05-14T06:28:38.727937","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Take All the CSV Files in the Train tdcsfog Folder","metadata":{"papermill":{"duration":0.02634,"end_time":"2023-05-14T06:28:38.924186","exception":false,"start_time":"2023-05-14T06:28:38.897846","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Set the directory path to the folder containing the CSV files.\ntdcsfog_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/train/tdcsfog'\n\n# Initialize an empty list to store the dataframes.\ntdcsfog_list = []\n\n# Loop through each file in the directory and read it into a dataframe.\nfor file_name in os.listdir(tdcsfog_path):\n    if file_name.endswith('.csv'):\n        file_path = os.path.join(tdcsfog_path, file_name)\n        file = pd.read_csv(file_path)\n        file.Time = file.Time / (len(file) - 1)\n        tdcsfog_list.append(file)\n\n# Concatenate the dataframes vertically using pd.concat().\ntdcsfog = pd.concat(tdcsfog_list, axis = 0)\n\n# Show the concatenated dataframe.\ntdcsfog = tdcsfog.reset_index(drop=True)\ntdcsfog","metadata":{"papermill":{"duration":18.008288,"end_time":"2023-05-14T06:28:56.959744","exception":false,"start_time":"2023-05-14T06:28:38.951456","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:03.166353Z","iopub.execute_input":"2023-05-28T17:14:03.166703Z","iopub.status.idle":"2023-05-28T17:14:22.289625Z","shell.execute_reply.started":"2023-05-28T17:14:03.166674Z","shell.execute_reply":"2023-05-28T17:14:22.288533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Take All the CSV Files in the Train defog Folder","metadata":{"papermill":{"duration":0.033964,"end_time":"2023-05-14T06:33:55.834755","exception":false,"start_time":"2023-05-14T06:33:55.800791","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Set the directory path to the folder containing the CSV files.\ndefog_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/train/defog'\n\n# Initialize an empty list to store the dataframes.\ndefog_list = []\n\n# Loop through each file in the directory and read it into a dataframe.\nfor file_name in os.listdir(defog_path):\n    if file_name.endswith('.csv'):\n        file_path = os.path.join(defog_path, file_name)\n        file = pd.read_csv(file_path)\n        file.Time = file.Time / (len(file) - 1)\n        defog_list.append(file)\n\n# Concatenate the dataframes vertically using pd.concat().\ndefog = pd.concat(defog_list, axis = 0)\n\n# Show the concatenated dataframe.\ndefog = defog.reset_index(drop=True)\ndefog","metadata":{"papermill":{"duration":24.575548,"end_time":"2023-05-14T06:34:20.444682","exception":false,"start_time":"2023-05-14T06:33:55.869134","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:22.291004Z","iopub.execute_input":"2023-05-28T17:14:22.291328Z","iopub.status.idle":"2023-05-28T17:14:50.768156Z","shell.execute_reply.started":"2023-05-28T17:14:22.291301Z","shell.execute_reply":"2023-05-28T17:14:50.766878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog = defog[(defog['Task'] == 1) & (defog['Valid'] == 1)]","metadata":{"papermill":{"duration":0.512136,"end_time":"2023-05-14T06:34:22.448614","exception":false,"start_time":"2023-05-14T06:34:21.936478","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:50.769425Z","iopub.execute_input":"2023-05-28T17:14:50.769751Z","iopub.status.idle":"2023-05-28T17:14:51.022404Z","shell.execute_reply.started":"2023-05-28T17:14:50.769722Z","shell.execute_reply":"2023-05-28T17:14:51.021338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog = defog.iloc[:, :7]","metadata":{"papermill":{"duration":0.084822,"end_time":"2023-05-14T06:34:22.568697","exception":false,"start_time":"2023-05-14T06:34:22.483875","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:51.023721Z","iopub.execute_input":"2023-05-28T17:14:51.024049Z","iopub.status.idle":"2023-05-28T17:14:51.109887Z","shell.execute_reply.started":"2023-05-28T17:14:51.024022Z","shell.execute_reply":"2023-05-28T17:14:51.108737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog.describe()","metadata":{"papermill":{"duration":1.969354,"end_time":"2023-05-14T06:34:24.573134","exception":false,"start_time":"2023-05-14T06:34:22.60378","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-28T17:14:51.111990Z","iopub.execute_input":"2023-05-28T17:14:51.112375Z","iopub.status.idle":"2023-05-28T17:14:51.969132Z","shell.execute_reply.started":"2023-05-28T17:14:51.112339Z","shell.execute_reply":"2023-05-28T17:14:51.967919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog.describe()","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:14:51.970551Z","iopub.execute_input":"2023-05-28T17:14:51.971357Z","iopub.status.idle":"2023-05-28T17:14:53.578259Z","shell.execute_reply.started":"2023-05-28T17:14:51.971322Z","shell.execute_reply":"2023-05-28T17:14:53.577161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Look at AccV, AccML and AccP distributions","metadata":{}},{"cell_type":"markdown","source":"# AccV\n\nWe will try to look at two distributions (**tdcsfog_AccV and defog_AccV**) and find are they probably the same or not","metadata":{}},{"cell_type":"code","source":"\nConst = 0.10197\ntdcsfog_AccV = tdcsfog['AccV'] * Const\ndefog_AccV = defog['AccV']\nprint(tdcsfog_AccV.mean())\nprint(defog_AccV.mean())","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:14:53.579658Z","iopub.execute_input":"2023-05-28T17:14:53.580061Z","iopub.status.idle":"2023-05-28T17:14:53.627698Z","shell.execute_reply.started":"2023-05-28T17:14:53.580029Z","shell.execute_reply":"2023-05-28T17:14:53.626706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data is in units of **m/s^2** for tdcsfog/ and **g** for defog/ and notype/\n\n1 m/s^2 to g-unit = 0.10197 g-unit","metadata":{}},{"cell_type":"code","source":"sns.histplot(data=defog_AccV,  bins=len(defog_AccV), stat=\"density\",\n      element=\"step\", fill=False, cumulative=True, common_norm=False,legend=True,label='defog_AccV');\n\nsns.histplot(data=tdcsfog_AccV, bins=len(tdcsfog_AccV), stat=\"density\",\n      element=\"step\", fill=False, cumulative=True, common_norm=False,legend=True,label='tdcsfog_AccV');\n\nplt.xlabel('AccV')\nplt.ylabel('Density AccV'+ '\\n')\nplt.ylim([0.0, 1.15])\nplt.legend(bbox_to_anchor=(1.2, 1), loc='lower right')\nplt.title('Cumulative distribution function' + '\\n')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:14:53.628975Z","iopub.execute_input":"2023-05-28T17:14:53.629330Z","iopub.status.idle":"2023-05-28T17:15:24.707137Z","shell.execute_reply.started":"2023-05-28T17:14:53.629301Z","shell.execute_reply":"2023-05-28T17:15:24.705992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(\n    data = [defog_AccV, tdcsfog_AccV],\n    palette='rainbow', \n    orient='h',\n    showfliers=False,\n);\nplt.title('AccV' + '\\n'),\nplt.xlabel('value AccV'),\nplt.yticks([0, 1],labels=['defog','tdcsfog']),\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:24.708555Z","iopub.execute_input":"2023-05-28T17:15:24.708907Z","iopub.status.idle":"2023-05-28T17:15:25.314785Z","shell.execute_reply.started":"2023-05-28T17:15:24.708877Z","shell.execute_reply":"2023-05-28T17:15:25.313706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# AccML\n\nWe will try to look at two distributions (**tdcsfog_AccML and defog_AccML**) and find are they probably the same or not","metadata":{}},{"cell_type":"code","source":"tdcsfog_AccML = tdcsfog['AccML'] * Const\ndefog_AccML = defog['AccML']\nprint('mean tdcsfog AccML =', tdcsfog_AccML.mean())\nprint('mean defog AccML =', defog_AccML.mean())","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:25.316483Z","iopub.execute_input":"2023-05-28T17:15:25.316915Z","iopub.status.idle":"2023-05-28T17:15:25.356931Z","shell.execute_reply.started":"2023-05-28T17:15:25.316874Z","shell.execute_reply":"2023-05-28T17:15:25.355853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(\n    data = [defog_AccML, tdcsfog_AccML],\n    palette='rainbow', \n    orient='h',\n    showfliers=False,\n);\nplt.title('AccML' + '\\n'),\nplt.xlabel('value AccML'),\nplt.yticks([0, 1],labels=['defog','tdcsfog']),\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:25.358617Z","iopub.execute_input":"2023-05-28T17:15:25.359492Z","iopub.status.idle":"2023-05-28T17:15:26.089255Z","shell.execute_reply.started":"2023-05-28T17:15:25.359457Z","shell.execute_reply":"2023-05-28T17:15:26.087875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.histplot(data=defog_AccML,  bins=len(defog_AccML), stat=\"density\",\n      element=\"step\", fill=False, cumulative=True, common_norm=False,legend=True,label='defog_AccML');\n\nsns.histplot(data=tdcsfog_AccML, bins=len(tdcsfog_AccML), stat=\"density\",\n      element=\"step\", fill=False, cumulative=True, common_norm=False,legend=True,label='tdcsfog_AccML');\n\nplt.xlabel('AccML')\nplt.ylabel('Density AccML'+ '\\n')\nplt.ylim([0.0, 1.15])\nplt.legend(bbox_to_anchor=(1.2, 1), loc='lower right')\nplt.title('Cumulative distribution function' + '\\n')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:26.091974Z","iopub.execute_input":"2023-05-28T17:15:26.092902Z","iopub.status.idle":"2023-05-28T17:15:56.120777Z","shell.execute_reply.started":"2023-05-28T17:15:26.092856Z","shell.execute_reply":"2023-05-28T17:15:56.119643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# AccAP\n\nWe will try to look at two distributions (**tdcsfog_AccAP and defog_AccAP**) are they probably the same or not","metadata":{}},{"cell_type":"code","source":"tdcsfog_AccAP = tdcsfog['AccAP'] * Const\ndefog_AccAP = defog['AccAP']\nprint('mean tdcsfog AccAP =', tdcsfog_AccAP.mean())\nprint('mean defog AccAP =', defog_AccAP.mean())","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:56.122803Z","iopub.execute_input":"2023-05-28T17:15:56.123304Z","iopub.status.idle":"2023-05-28T17:15:56.162031Z","shell.execute_reply.started":"2023-05-28T17:15:56.123261Z","shell.execute_reply":"2023-05-28T17:15:56.161045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(\n    data = [defog_AccAP, tdcsfog_AccAP],\n    palette='rainbow', \n    orient='h',\n    showfliers=False,\n);\nplt.title('AccP' + '\\n'),\nplt.xlabel('value AccP'),\nplt.yticks([0, 1],labels=['defog','tdcsfog']),\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:56.163378Z","iopub.execute_input":"2023-05-28T17:15:56.164361Z","iopub.status.idle":"2023-05-28T17:15:56.714714Z","shell.execute_reply.started":"2023-05-28T17:15:56.164324Z","shell.execute_reply":"2023-05-28T17:15:56.713722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.histplot(data=defog_AccAP,  bins=len(defog_AccAP), stat=\"density\",\n      element=\"step\", fill=False, cumulative=True, common_norm=False,legend=True,label='defog_AccAP');\n\nsns.histplot(data=tdcsfog_AccAP, bins=len(tdcsfog_AccAP), stat=\"density\",\n      element=\"step\", fill=False, cumulative=True, common_norm=False,legend=True,label='tdcsfog_AccAP');\n\nplt.xlabel('AccAP')\nplt.ylabel('Density AccAP'+ '\\n')\nplt.ylim([0.0, 1.15])\nplt.legend(bbox_to_anchor=(1.2, 1), loc='lower right')\nplt.title('Cumulative distribution function' + '\\n')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:15:56.716660Z","iopub.execute_input":"2023-05-28T17:15:56.717395Z","iopub.status.idle":"2023-05-28T17:16:26.918298Z","shell.execute_reply.started":"2023-05-28T17:15:56.717361Z","shell.execute_reply":"2023-05-28T17:16:26.917202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Statistical function","metadata":{}},{"cell_type":"code","source":"def t_test_verbose(a, sample2=None, mean=None, fn=None):\n    abar = a.mean()\n    avar = a.var(ddof=1)  \n    na = a.size\n    adof = na - 1\n    conf_int = st.t.interval(0.95, len(a)-1, \n                                loc=np.mean(a), scale=st.sem(a))\n\n    if type(a) == type(sample2):\n        bbar = sample2.mean()\n        bvar = sample2.var(ddof=1)\n        nb = sample2.size\n        bdof = nb - 1\n\n        dof = (avar/na + bvar/nb)**2 /   \\\n              (avar**2/(na**2*adof) + bvar**2/(nb**2*bdof))\n        return {'p-value'           : \n                   fn(a, sample2, equal_var=False).pvalue,  \n                'degrees of freedom       ' : dof,  #t_test(a, b),   \n                'confidential interval 95%' : conf_int,         \n                'n1             ' : a.count(),\n                'n2             ' : sample2.count(),\n                'average tdcsfog' : a.mean(),\n                'average defog'   : sample2.mean(),\n                'variance tdcsfog ' : a.var(),\n                'variance defog ' : sample2.var(),\n                't-statistic' : fn( a, sample2, equal_var=False ).statistic} \n   ","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:26.920108Z","iopub.execute_input":"2023-05-28T17:16:26.920456Z","iopub.status.idle":"2023-05-28T17:16:26.931535Z","shell.execute_reply.started":"2023-05-28T17:16:26.920425Z","shell.execute_reply":"2023-05-28T17:16:26.930202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Statistical significance AccV\n\nH0: Distribution of defog AccP **is NOT the same** as tdcsfog \n\nH1: Distribution of defog AccP **is the same** as tdcsfog","metadata":{}},{"cell_type":"code","source":"results_AccV = st.ttest_ind(tdcsfog_AccV, defog_AccV)\nalpha_V = 0.05\nprint('p-value:', results_AccV.pvalue)\nif results_AccV.pvalue < alpha_V:\n    print('Reject H0: probably the same distributions')\nelse:\n    print('Cannot reject H0: probably different distribution') ","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:26.933036Z","iopub.execute_input":"2023-05-28T17:16:26.933431Z","iopub.status.idle":"2023-05-28T17:16:27.009031Z","shell.execute_reply.started":"2023-05-28T17:16:26.933386Z","shell.execute_reply":"2023-05-28T17:16:27.008117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(t_test_verbose(tdcsfog_AccV, sample2=defog_AccV, fn=st.ttest_ind))","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:27.010544Z","iopub.execute_input":"2023-05-28T17:16:27.011209Z","iopub.status.idle":"2023-05-28T17:16:27.428868Z","shell.execute_reply.started":"2023-05-28T17:16:27.011176Z","shell.execute_reply":"2023-05-28T17:16:27.427725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Statistical significance AccML\n\nH0: Distribution of defog AccML **is NOT the same** as tdcsfog \n\nH1: Distribution of defog AccML **is the same** as tdcsfog","metadata":{}},{"cell_type":"code","source":"results_AccML = st.ttest_ind(tdcsfog_AccML, defog_AccML)\nalpha_ML = 0.05\nprint('p-value:', results_AccML.pvalue)\nif results_AccML.pvalue < alpha_ML:\n    print('Reject H0: probably the same distributions')\nelse:\n    print('Cannot reject H0: probably different distribution') ","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:27.434326Z","iopub.execute_input":"2023-05-28T17:16:27.434698Z","iopub.status.idle":"2023-05-28T17:16:27.500471Z","shell.execute_reply.started":"2023-05-28T17:16:27.434668Z","shell.execute_reply":"2023-05-28T17:16:27.499253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(t_test_verbose(tdcsfog_AccML, sample2=defog_AccML, fn=st.ttest_ind))","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:27.502026Z","iopub.execute_input":"2023-05-28T17:16:27.502361Z","iopub.status.idle":"2023-05-28T17:16:27.929390Z","shell.execute_reply.started":"2023-05-28T17:16:27.502333Z","shell.execute_reply":"2023-05-28T17:16:27.928181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Statistical significance AccAP\n\nH0: Distribution of defog AccAP **is NOT the same** as tdcsfog \n\nH1: Distribution of defog AccAP **is the same** as tdcsfog","metadata":{}},{"cell_type":"code","source":"results_AP = st.ttest_ind(tdcsfog_AccAP, defog_AccAP)\nalpha_AP = 0.05\nprint('p-value:', results_AP.pvalue)\nif results_AP.pvalue < alpha_AP:\n    print('Reject H0: probably the same distributions')\nelse:\n    print('Cannot reject H0: probably different distribution')  ","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:27.930711Z","iopub.execute_input":"2023-05-28T17:16:27.931075Z","iopub.status.idle":"2023-05-28T17:16:27.996913Z","shell.execute_reply.started":"2023-05-28T17:16:27.931045Z","shell.execute_reply":"2023-05-28T17:16:27.995708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(t_test_verbose(tdcsfog_AccAP, sample2=defog_AccAP, fn=st.ttest_ind))","metadata":{"execution":{"iopub.status.busy":"2023-05-28T17:16:27.999285Z","iopub.execute_input":"2023-05-28T17:16:27.999751Z","iopub.status.idle":"2023-05-28T17:16:28.427550Z","shell.execute_reply.started":"2023-05-28T17:16:27.999709Z","shell.execute_reply":"2023-05-28T17:16:28.426529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion","metadata":{}},{"cell_type":"markdown","source":"As we can see distributions of AccP, AccV and AccML probably are the same","metadata":{}}]}