Importing Libraries
","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport scipy.stats as stats\nimport seaborn as sns\n\nimport os\nfrom concurrent.futures import ThreadPoolExecutor\nfrom tqdm import tqdm\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T16:27:52.405225Z","iopub.execute_input":"2024-11-18T16:27:52.405700Z","iopub.status.idle":"2024-11-18T16:27:55.270896Z","shell.execute_reply.started":"2024-11-18T16:27:52.405657Z","shell.execute_reply":"2024-11-18T16:27:55.269812Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Loading Data
","metadata":{}},{"cell_type":"markdown","source":"Take first id 00115b9f as an example. ","metadata":{}},{"cell_type":"code","source":"df = pd.read_parquet(\"/kaggle/input/child-mind-institute-problematic-internet-use/series_train.parquet/id=00115b9f/part-0.parquet\")\ndf.drop('step', axis=1, inplace=True)\ndf","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:12:14.862806Z","iopub.execute_input":"2024-11-18T20:12:14.863297Z","iopub.status.idle":"2024-11-18T20:12:14.987765Z","shell.execute_reply.started":"2024-11-18T20:12:14.863253Z","shell.execute_reply":"2024-11-18T20:12:14.986529Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Data Exploration
","metadata":{}},{"cell_type":"markdown","source":"Data descriptions:\n\nSeries are used as training data, partitioned by id. Each series is a continuous recording of accelerometer data for a single subject spanning many days.\n\nid - The patient identifier corresponding to the id field in train/test.csv.\n\nstep - An integer timestep for each observation within a series.\n\nX, Y, Z - Measure of acceleration, in g, experienced by the wrist-worn watch along each standard axis.\n\nenmo - As calculated and described by the wristpy package, ENMO is the Euclidean Norm Minus One of all accelerometer signals (along each of the x-, y-, and z-axis, measured in g-force) with negative values rounded to zero. Zero values are indicative of periods of no motion. While no standard measure of acceleration exists in this space, this is one of the several commonly computed features.\n\nanglez - As calculated and described by the wristpy package, Angle-Z is a metric derived from individual accelerometer components and refers to the angle of the arm relative to the horizontal plane.\n\nnon-wear_flag - A flag (0: watch is being worn, 1: the watch is not worn) to help determine periods when the watch has been removed, based on the GGIR definition, which uses the standard deviation and range of the accelerometer data.\n\nlight - Measure of ambient light in lux. See here for details.\n\nbattery_voltage - A measure of the battery voltage in mV.\n\ntime_of_day - Time of day representing the start of a 5s window that the data has been sampled over, with format %H:%M:%S.%9f.\n\nweekday - The day of the week, coded as an integer with 1 being Monday and 7 being Sunday.\n\nquarter - The quarter of the year, an integer from 1 to 4.\n\nrelative_date_PCIAT - The number of days (integer) since the PCIAT test was administered (negative days indicate that the actigraphy data has been collected before the test was administered).","metadata":{}},{"cell_type":"code","source":"df.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:12:25.598002Z","iopub.execute_input":"2024-11-18T20:12:25.598455Z","iopub.status.idle":"2024-11-18T20:12:25.771400Z","shell.execute_reply.started":"2024-11-18T20:12:25.598419Z","shell.execute_reply":"2024-11-18T20:12:25.770203Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:12:41.535739Z","iopub.execute_input":"2024-11-18T20:12:41.536327Z","iopub.status.idle":"2024-11-18T20:12:41.556165Z","shell.execute_reply.started":"2024-11-18T20:12:41.536274Z","shell.execute_reply":"2024-11-18T20:12:41.554912Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The 'quarter' coloumn has only one value: 3.","metadata":{}},{"cell_type":"code","source":"df['quarter'].unique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:12:44.918488Z","iopub.execute_input":"2024-11-18T20:12:44.918967Z","iopub.status.idle":"2024-11-18T20:12:44.929293Z","shell.execute_reply.started":"2024-11-18T20:12:44.918926Z","shell.execute_reply":"2024-11-18T20:12:44.927842Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The 'non-wear_flag' coloumn has only one value: 0.","metadata":{}},{"cell_type":"code","source":"df['non-wear_flag'].unique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:12:49.397890Z","iopub.execute_input":"2024-11-18T20:12:49.398343Z","iopub.status.idle":"2024-11-18T20:12:49.407635Z","shell.execute_reply.started":"2024-11-18T20:12:49.398303Z","shell.execute_reply":"2024-11-18T20:12:49.406389Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Q1. How are the features distributed?
","metadata":{}},{"cell_type":"markdown","source":"'X', 'Y', 'Z' (Measure of acceleration, in g, experienced by the wrist-worn watch along each standard axis) show similar patterns. Most active regions lie within (-1, 1). Both 'enmo' and 'light' have very large outliers. See the differences between '%75's (0,04; 15) and 'max's(4; 2633). 'non-wear_flag'(0) means watch is being worn. Measurements are distributed along all 7 weekdays in quarter 3 only. Positive 'relative_date_PCIAT' days indicate that the actigraphy data have been collected after the test is administered.","metadata":{}},{"cell_type":"code","source":"sns.set_theme(rc={'figure.figsize':(15,12)})\nplt.figure(figsize=(15,12))\n\nfig, axes = plt.subplots(3, 4)\nfor i in range(3):\n for j in range(4):\n sns.kdeplot(ax=axes[i,j], data=df, x=df.columns[i*4+j])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:12:52.858203Z","iopub.execute_input":"2024-11-18T20:12:52.858671Z","iopub.status.idle":"2024-11-18T20:13:10.980505Z","shell.execute_reply.started":"2024-11-18T20:12:52.858628Z","shell.execute_reply":"2024-11-18T20:13:10.979142Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Two columns 'enmo' and 'light' show non-Gaussian distributions. After log transformation, they look more like Gaussian distributions. Skewnesses are also reduced. \"1\" is added to deal with the 0 values in the columns.","metadata":{}},{"cell_type":"code","source":"df['enmo'] = np.log(1 + df['enmo'])\ndf['light'] = np.log(1 + df['light'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:13:24.076582Z","iopub.execute_input":"2024-11-18T20:13:24.077050Z","iopub.status.idle":"2024-11-18T20:13:24.089080Z","shell.execute_reply.started":"2024-11-18T20:13:24.077008Z","shell.execute_reply":"2024-11-18T20:13:24.087669Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.set_theme(rc={'figure.figsize':(10,5)})\nplt.figure(figsize=(10,5))\n\nfig, axes = plt.subplots(1, 2)\nsns.kdeplot(ax=axes[0], data=df, x='enmo')\nsns.kdeplot(ax=axes[1], data=df, x='light')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:13:25.928269Z","iopub.execute_input":"2024-11-18T20:13:25.928778Z","iopub.status.idle":"2024-11-18T20:13:29.572183Z","shell.execute_reply.started":"2024-11-18T20:13:25.928731Z","shell.execute_reply":"2024-11-18T20:13:29.571057Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"I add another feature 'cubic_root' to describe the geometric mean accelaration along 3 standard axes. Clearly it shows a nearly symmetric distribution along 0.0 axis.","metadata":{}},{"cell_type":"code","source":"df['geo_mean'] = np.cbrt(df['X'] * df['Y'] * df['Z'])\n\nfig, ax = plt.subplots(1, 1)\nsns.kdeplot(ax=ax, data=df, x='geo_mean')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:13:35.028452Z","iopub.execute_input":"2024-11-18T20:13:35.028939Z","iopub.status.idle":"2024-11-18T20:13:36.968687Z","shell.execute_reply.started":"2024-11-18T20:13:35.028897Z","shell.execute_reply":"2024-11-18T20:13:36.967503Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Below are the QQ-plots and histogram distributions, with skewnesses printed on the figures.","metadata":{}},{"cell_type":"code","source":"def qq_and_skewness_plots(data, num_col):\n \n f, ax = plt.subplots(1, 2, figsize=(12, 5))\n\n # QQ plot\n stats.probplot(data[num_col], dist=\"norm\", plot=ax[0], fit=True)\n ax[0].set_title(f'QQ Plot for {num_col}')\n\n # Skewness plot\n sns.histplot(data[num_col], kde=True, ax=ax[1])\n ax[1].set_title(f'Distribution of {num_col}')\n\n # Calculate skewness value\n skewness_value = stats.skew(data[num_col])\n\n # Display skewness value on the plot\n ax[1].text(0.5, 0.5, f'Skewness: {skewness_value:.2f}', transform=ax[1].transAxes, \n horizontalalignment='center', verticalalignment='center', fontsize=16, color='red')\n\n plt.tight_layout() # Adjust spacing between subplots\n\n plt.show()\n\nfor col in df.columns:\n qq_and_skewness_plots(df, col)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:13:43.498744Z","iopub.execute_input":"2024-11-18T20:13:43.499167Z","iopub.status.idle":"2024-11-18T20:14:29.392149Z","shell.execute_reply.started":"2024-11-18T20:13:43.499130Z","shell.execute_reply":"2024-11-18T20:14:29.390816Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Q2.How are the numerical variables varied in response to time_of_day and weekday?
","metadata":{}},{"cell_type":"markdown","source":"Take 'X' for example. On Monday the object is most active at later time. From Tuesday to Sunday, active across all day long. Super active after lunch on Friday to Sunday. Similar patterns are observed for both 'Y' and 'Z'.","metadata":{}},{"cell_type":"code","source":"sns.set_theme(rc={'figure.figsize':(12,8)})\nplt.figure(figsize=(12,8))\nsns.scatterplot(data=df, x='time_of_day', y='X', hue='weekday')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:14:34.966482Z","iopub.execute_input":"2024-11-18T20:14:34.966988Z","iopub.status.idle":"2024-11-18T20:15:00.007084Z","shell.execute_reply.started":"2024-11-18T20:14:34.966944Z","shell.execute_reply":"2024-11-18T20:15:00.005872Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Highest 'enmo's are recorded at later time on Thursday and Friday.","metadata":{}},{"cell_type":"code","source":"sns.scatterplot(data=df, x='time_of_day', y='enmo', hue='weekday')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:15:07.109734Z","iopub.execute_input":"2024-11-18T20:15:07.110207Z","iopub.status.idle":"2024-11-18T20:15:29.780087Z","shell.execute_reply.started":"2024-11-18T20:15:07.110164Z","shell.execute_reply":"2024-11-18T20:15:29.778637Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Q3 How many counts do numerical features have by weekdays?
","metadata":{}},{"cell_type":"markdown","source":"Take 'X' for example. Its distribution is not even across seven days. Friday is the most. Sunday comes second. Tuesday and Wednesday are the fewest.","metadata":{}},{"cell_type":"code","source":"df.groupby('weekday')['X'].count().reset_index()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:15:48.125069Z","iopub.execute_input":"2024-11-18T20:15:48.125516Z","iopub.status.idle":"2024-11-18T20:15:48.146160Z","shell.execute_reply.started":"2024-11-18T20:15:48.125478Z","shell.execute_reply":"2024-11-18T20:15:48.144500Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This figure shows how all of the training samples are taken by weekdays. Friday is the most frequent one. Sunday comes second. Tuesday and Wednesday are the least.","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=df, x='weekday')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:15:51.319819Z","iopub.execute_input":"2024-11-18T20:15:51.320853Z","iopub.status.idle":"2024-11-18T20:15:51.670139Z","shell.execute_reply.started":"2024-11-18T20:15:51.320811Z","shell.execute_reply":"2024-11-18T20:15:51.669087Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Separate 'relative_date_PCIAT' into several ranges. Most of the PCIAT tests were administered about 40 days ago. Tests administered about 60 days ago are the fewest.","metadata":{}},{"cell_type":"code","source":"df['relative_date_PCIAT_range'] = np.where(df['relative_date_PCIAT'].between(40, 50, inclusive='left'), '40s', \n np.where(df['relative_date_PCIAT'].between(50, 60, inclusive='left'), '50s',\n np.where(df['relative_date_PCIAT'].between(60, 70, inclusive='left'), '60s',\n np.where(df['relative_date_PCIAT'].between(70, 80, inclusive='left'), '70s', '80+'))))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:15:55.648114Z","iopub.execute_input":"2024-11-18T20:15:55.648592Z","iopub.status.idle":"2024-11-18T20:15:55.716835Z","shell.execute_reply.started":"2024-11-18T20:15:55.648521Z","shell.execute_reply":"2024-11-18T20:15:55.715677Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.groupby('relative_date_PCIAT_range')['X'].count().reset_index()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:15:58.329612Z","iopub.execute_input":"2024-11-18T20:15:58.330671Z","iopub.status.idle":"2024-11-18T20:15:58.370660Z","shell.execute_reply.started":"2024-11-18T20:15:58.330613Z","shell.execute_reply":"2024-11-18T20:15:58.368847Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.countplot(data=df, x='relative_date_PCIAT_range')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:16:06.096698Z","iopub.execute_input":"2024-11-18T20:16:06.097226Z","iopub.status.idle":"2024-11-18T20:16:06.540058Z","shell.execute_reply.started":"2024-11-18T20:16:06.097185Z","shell.execute_reply":"2024-11-18T20:16:06.538902Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Q4 Other Interesting Plots.
","metadata":{}},{"cell_type":"markdown","source":"Pick three dimensions 'X', 'Y' and 'Z' and draw a 3D scatter plot. Most of the measurement data lie densely within an oval shaped area.","metadata":{}},{"cell_type":"code","source":"sns.set_style (\"darkgrid\")\nplot_mean = 3\n\nplot1 = df['X']\nplot2 = df['Y']\nplot3 = df['Z']\n\nplt.figure (figsize = (15, 10))\nseaborn_plot = plt.axes (projection='3d')\nprint (type (seaborn_plot))\nseaborn_plot.scatter3D (plot1, plot2, plot3)\nseaborn_plot.set_xlabel ('x')\nseaborn_plot.set_ylabel ('y')\nseaborn_plot.set_zlabel ('z')\nplt.show ()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:16:18.492292Z","iopub.execute_input":"2024-11-18T20:16:18.492726Z","iopub.status.idle":"2024-11-18T20:16:26.737845Z","shell.execute_reply.started":"2024-11-18T20:16:18.492691Z","shell.execute_reply":"2024-11-18T20:16:26.736633Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Pick two dimensions 'X' and 'Y' and draw a 2D contour plot.","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"white\")\nsns.kdeplot(x=df['X'], y=df['Y'], cmap=\"Reds\", fill=True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-18T20:16:26.739723Z","iopub.execute_input":"2024-11-18T20:16:26.740073Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Conclusions.
","metadata":{}},{"cell_type":"markdown","source":"Four questions are answered through exploratory data analysis. This object is just the first example. Other objects may have similar or different patterns.","metadata":{}}]}