{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**TensorFlow Data Validation(TFDV)** is a library in the TensorFlow Extended(TFX) platform which can be used for visualizing descriptive statistics or for finding anomalies between training , evaluation, and serving data. This also can be used with TensorFlow core platform. There are multiple aspects of TFDV suitable for serving environment and here we are limiting our focus only on the visualization.\n\nUnderstanding data and learning patterns is crucial in the process of building models. This is more relevant and challenging when we have voluminous and fragmented data as we have in this challenge.\n\nWe can benefit in many ways: \n\n    1. Peek at the data to get reasonable understanding before model building \n    2. Possible outliers detection \n    3. Missing data analysis \n    \nWe will visualize different files to get some understanding on the method and results.","metadata":{}},{"cell_type":"code","source":"!pip install -U pip\n!pip install --upgrade -q 'tensorflow_data_validation[visualization]<2'\nimport pkg_resources\nimport importlib\nimportlib.reload(pkg_resources)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-28T07:45:01.359876Z","iopub.execute_input":"2023-04-28T07:45:01.360395Z","iopub.status.idle":"2023-04-28T07:45:30.248664Z","shell.execute_reply.started":"2023-04-28T07:45:01.360355Z","shell.execute_reply":"2023-04-28T07:45:30.247388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow_data_validation as tfdv\nprint('TF version:', tf.__version__)\nprint('TFDV version:', tfdv.version.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:45:30.253814Z","iopub.execute_input":"2023-04-28T07:45:30.255273Z","iopub.status.idle":"2023-04-28T07:45:30.263838Z","shell.execute_reply.started":"2023-04-28T07:45:30.255219Z","shell.execute_reply":"2023-04-28T07:45:30.261874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# daily_data","metadata":{}},{"cell_type":"code","source":"daily_metadata = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/daily_metadata.csv')\ndaily_metadata","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:45:30.265468Z","iopub.execute_input":"2023-04-28T07:45:30.268516Z","iopub.status.idle":"2023-04-28T07:45:30.295512Z","shell.execute_reply.started":"2023-04-28T07:45:30.268466Z","shell.execute_reply":"2023-04-28T07:45:30.294139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_meta_stats = tfdv.generate_statistics_from_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/daily_metadata.csv')\ntfdv.visualize_statistics(daily_meta_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:45:30.299855Z","iopub.execute_input":"2023-04-28T07:45:30.300367Z","iopub.status.idle":"2023-04-28T07:45:32.915936Z","shell.execute_reply.started":"2023-04-28T07:45:30.300302Z","shell.execute_reply":"2023-04-28T07:45:32.914393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_series_data = pd.read_parquet('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/unlabeled/00c4c9313d.parquet')\ndaily_series_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:45:32.917858Z","iopub.execute_input":"2023-04-28T07:45:32.918298Z","iopub.status.idle":"2023-04-28T07:45:37.112531Z","shell.execute_reply.started":"2023-04-28T07:45:32.918257Z","shell.execute_reply":"2023-04-28T07:45:37.111131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"daily_stats = tfdv.generate_statistics_from_dataframe(daily_series_data)\ntfdv.visualize_statistics(daily_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:45:37.114292Z","iopub.execute_input":"2023-04-28T07:45:37.115411Z","iopub.status.idle":"2023-04-28T07:46:19.589775Z","shell.execute_reply.started":"2023-04-28T07:45:37.115345Z","shell.execute_reply":"2023-04-28T07:46:19.588016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**defog_metadata** ","metadata":{}},{"cell_type":"code","source":"defog_metadata = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/defog_metadata.csv')\ndefog_metadata.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:19.592123Z","iopub.execute_input":"2023-04-28T07:46:19.592639Z","iopub.status.idle":"2023-04-28T07:46:19.613519Z","shell.execute_reply.started":"2023-04-28T07:46:19.592585Z","shell.execute_reply":"2023-04-28T07:46:19.611912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_meta_stats = tfdv.generate_statistics_from_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/defog_metadata.csv')\ntfdv.visualize_statistics(defog_meta_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:19.617684Z","iopub.execute_input":"2023-04-28T07:46:19.618099Z","iopub.status.idle":"2023-04-28T07:46:22.415248Z","shell.execute_reply.started":"2023-04-28T07:46:19.618064Z","shell.execute_reply":"2023-04-28T07:46:22.413609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The above is typical interactive output of 'visualize_statistics'. Contains:**\n\n1. Type of features - Numeric or Categorical\n2. Type of data - int, string, float\n3. General stats such as min, max, stddev,unique,top values.\n4. Count and missing data.137 are total number of subjects. There are no missing values.\n5. Interactive charts of distribution for each feature. Can be changed to log scale or 'expand' to view zoomed charts.\n6. Categorical features can be seen as raw data. For example click on 'SHOW RAW DATA' in Medication feature to know that there were 69 subjects on medication and 68 were not.","metadata":{}},{"cell_type":"markdown","source":"**tdcsfog_metadata**","metadata":{}},{"cell_type":"code","source":"tdcsfog_metadata = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/tdcsfog_metadata.csv')\ntdcsfog_metadata.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:22.417468Z","iopub.execute_input":"2023-04-28T07:46:22.419132Z","iopub.status.idle":"2023-04-28T07:46:22.442483Z","shell.execute_reply.started":"2023-04-28T07:46:22.419062Z","shell.execute_reply":"2023-04-28T07:46:22.440275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_meta_stats = tfdv.generate_statistics_from_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/tdcsfog_metadata.csv')\ntfdv.visualize_statistics(tdcsfog_meta_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:22.450077Z","iopub.execute_input":"2023-04-28T07:46:22.451848Z","iopub.status.idle":"2023-04-28T07:46:25.213679Z","shell.execute_reply.started":"2023-04-28T07:46:22.451761Z","shell.execute_reply":"2023-04-28T07:46:25.211842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Comparing with defog metadata:\n\n1. Maximum visit is 20 compared to 2 in defog. With median of 4, this may be an outlier. Need to be checked.\n2. tdcsFog has additional numeric feature - Test, which has three almost equal counts.\n3. Out of 62 subjects some have as many as 24 test Ids (train series data) and some only 2 or 3(click on 'SHOW RAW DATA' in Subject feature).","metadata":{}},{"cell_type":"markdown","source":"**Subjects**","metadata":{}},{"cell_type":"code","source":"subjects = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/subjects.csv')\nsubjects.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:25.215571Z","iopub.execute_input":"2023-04-28T07:46:25.216116Z","iopub.status.idle":"2023-04-28T07:46:25.242507Z","shell.execute_reply.started":"2023-04-28T07:46:25.216079Z","shell.execute_reply":"2023-04-28T07:46:25.240535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subjects_stats = tfdv.generate_statistics_from_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/subjects.csv')\ntfdv.visualize_statistics(subjects_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:25.244515Z","iopub.execute_input":"2023-04-28T07:46:25.245748Z","iopub.status.idle":"2023-04-28T07:46:27.899006Z","shell.execute_reply.started":"2023-04-28T07:46:25.245697Z","shell.execute_reply":"2023-04-28T07:46:27.897951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This data has missing values.\n\nSelect 'Sort by' to 'Amount missing/zero' to view missing values and zeros as percentages.\n\nNFOGQ- Self-report FoG questionnaire score has about 12% zero values. Need to check whether these are to be treated as missing values.","metadata":{}},{"cell_type":"markdown","source":"**defog_series_data - Target Analysis:**","metadata":{"execution":{"iopub.status.busy":"2023-03-20T16:04:38.686622Z","iopub.execute_input":"2023-03-20T16:04:38.687143Z","iopub.status.idle":"2023-03-20T16:04:38.695533Z","shell.execute_reply.started":"2023-03-20T16:04:38.687082Z","shell.execute_reply":"2023-03-20T16:04:38.693740Z"}}},{"cell_type":"markdown","source":"**Let us anlyze the combined csv files in train folder for defog and tdcsfog data setries.**\n\nlocation:[defog-tdcsfog-data](https://www.kaggle.com/datasets/viji1609/freezing-gait-prediction-defogtdcsfog-data)\n\n","metadata":{}},{"cell_type":"code","source":"defog_series_data = pd.read_parquet('/kaggle/input/freezing-gait-prediction-defogtdcsfog-data/defog_series_data.parquet')\ndefog_series_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:27.900477Z","iopub.execute_input":"2023-04-28T07:46:27.901451Z","iopub.status.idle":"2023-04-28T07:46:33.964849Z","shell.execute_reply.started":"2023-04-28T07:46:27.901411Z","shell.execute_reply":"2023-04-28T07:46:33.963080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_series_data = pd.read_parquet('/kaggle/input/freezing-gait-prediction-defogtdcsfog-data/tdcsfog_series_data.parquet')\ntdcsfog_series_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:33.966791Z","iopub.execute_input":"2023-04-28T07:46:33.967559Z","iopub.status.idle":"2023-04-28T07:46:35.916260Z","shell.execute_reply.started":"2023-04-28T07:46:33.967516Z","shell.execute_reply":"2023-04-28T07:46:35.915083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Target data analysis**","metadata":{}},{"cell_type":"markdown","source":"Let us check the target values of both defog & tdcsfog data","metadata":{}},{"cell_type":"code","source":"subm = pd.read_csv('/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/sample_submission.csv')\nsubm.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:35.917950Z","iopub.execute_input":"2023-04-28T07:46:35.918977Z","iopub.status.idle":"2023-04-28T07:46:36.255885Z","shell.execute_reply.started":"2023-04-28T07:46:35.918937Z","shell.execute_reply":"2023-04-28T07:46:36.253976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targets = ['StartHesitation', 'Turn', 'Walking']","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:36.257942Z","iopub.execute_input":"2023-04-28T07:46:36.258324Z","iopub.status.idle":"2023-04-28T07:46:36.264778Z","shell.execute_reply.started":"2023-04-28T07:46:36.258290Z","shell.execute_reply":"2023-04-28T07:46:36.263216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_targets_stats = tfdv.generate_statistics_from_dataframe(defog_series_data[targets].reset_index(drop=True))\ntfdv.visualize_statistics(defog_targets_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:36.266672Z","iopub.execute_input":"2023-04-28T07:46:36.267148Z","iopub.status.idle":"2023-04-28T07:46:41.473448Z","shell.execute_reply.started":"2023-04-28T07:46:36.267102Z","shell.execute_reply":"2023-04-28T07:46:41.471612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_targets_stats = tfdv.generate_statistics_from_dataframe(tdcsfog_series_data[targets].reset_index(drop=True))\ntfdv.visualize_statistics(tdcsfog_targets_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:41.475148Z","iopub.execute_input":"2023-04-28T07:46:41.475568Z","iopub.status.idle":"2023-04-28T07:46:44.006007Z","shell.execute_reply.started":"2023-04-28T07:46:41.475531Z","shell.execute_reply":"2023-04-28T07:46:44.004373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can compare both the targets side by side to have better understanding.","metadata":{}},{"cell_type":"code","source":"tfdv.visualize_statistics(lhs_statistics=defog_targets_stats, rhs_statistics=tdcsfog_targets_stats,\n                          lhs_name='defog_targets_stats', rhs_name='tdcsfog_targets_stats')\n","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:44.007880Z","iopub.execute_input":"2023-04-28T07:46:44.008348Z","iopub.status.idle":"2023-04-28T07:46:44.019819Z","shell.execute_reply.started":"2023-04-28T07:46:44.008294Z","shell.execute_reply":"2023-04-28T07:46:44.017281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following are some obvious differences in defog and tdcsfog targets:\n\n1. All values are zeros for 'start hesitation' in defog and 95.68% in tdcsfog.\n2. In general defog has less positive values compared to tdcsfog.","metadata":{}},{"cell_type":"code","source":"defog_features = ['Time', 'AccV', 'AccML', 'AccAP','Valid', 'Task', 'Id', 'Subject', 'series_type']","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:44.022193Z","iopub.execute_input":"2023-04-28T07:46:44.023682Z","iopub.status.idle":"2023-04-28T07:46:44.038790Z","shell.execute_reply.started":"2023-04-28T07:46:44.023600Z","shell.execute_reply":"2023-04-28T07:46:44.036857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"defog_features_stats = tfdv.generate_statistics_from_dataframe(defog_series_data[defog_features].reset_index(drop=True))\ntfdv.visualize_statistics(defog_features_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:46:44.041004Z","iopub.execute_input":"2023-04-28T07:46:44.041616Z","iopub.status.idle":"2023-04-28T07:47:08.269811Z","shell.execute_reply.started":"2023-04-28T07:46:44.041571Z","shell.execute_reply":"2023-04-28T07:47:08.268147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_features = ['Time', 'AccV', 'AccML', 'AccAP','Id', 'Subject', 'series_type']","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:47:08.272155Z","iopub.execute_input":"2023-04-28T07:47:08.273120Z","iopub.status.idle":"2023-04-28T07:47:08.280855Z","shell.execute_reply.started":"2023-04-28T07:47:08.273060Z","shell.execute_reply":"2023-04-28T07:47:08.279755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tdcsfog_features_stats = tfdv.generate_statistics_from_dataframe(tdcsfog_series_data[tdcsfog_features].reset_index(drop=True))\ntfdv.visualize_statistics(tdcsfog_features_stats)","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:47:08.282467Z","iopub.execute_input":"2023-04-28T07:47:08.283893Z","iopub.status.idle":"2023-04-28T07:47:19.864813Z","shell.execute_reply.started":"2023-04-28T07:47:08.283808Z","shell.execute_reply":"2023-04-28T07:47:19.863286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfdv.visualize_statistics(lhs_statistics=defog_features_stats, rhs_statistics=tdcsfog_features_stats,\n                          lhs_name='defog_features_stats', rhs_name='tdcsfog_features_stats')\n","metadata":{"execution":{"iopub.status.busy":"2023-04-28T07:47:19.866797Z","iopub.execute_input":"2023-04-28T07:47:19.867201Z","iopub.status.idle":"2023-04-28T07:47:19.884289Z","shell.execute_reply.started":"2023-04-28T07:47:19.867164Z","shell.execute_reply":"2023-04-28T07:47:19.882139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Comparing features:**\n\n\n   1. Valid/Task are available only for defog.\n   2. defog Subject (RAW DATA) have 38 subjects 91 train data series. tdcsfog tests have 62 subjects with 833 train data series.\n   2. There seems to be variation in the range of Time and acceleration values. This may be due to unit of measurement. As per organizers:\n    \n    \"\n    Series from the tdcsfog dataset are recorded at 128Hz (128 timesteps per second), while series from the defog and daily series are recorded at 100Hz (100 timesteps per second).\n    AccV, AccML, and AccAP Acceleration from a lower-back sensor on three axes: V - vertical, ML - mediolateral, AP - anteroposterior. Data is in units of m/s^2 for tdcsfog/ and g for defog/.\n    \"","metadata":{}},{"cell_type":"markdown","source":"**To summarize,**\n\n1. Few examples above show the simplicity and usefulness of visualization using TFDV.\n2. Though we have seen some explanations for each file, all can be used for any file depending on the need.\n3. Also visualization at any stage, for example after data transformations if needed.\n\n**Feel free to explore by using the interactive features and also add files not included above to get more out of the data!**\n\n**Thank you**","metadata":{}}]}