![\"little](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F5425440ae67e89b87b0a0019d4f6c007%2F1%20daylight%201.png?generation=1727208015377175&alt=media\")
\n
![\"interpolation\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F6dbb1775a9ec891d8dec9f8b5d6390a4%2Finterpolation.png?generation=1727208001135576&alt=media\")
\n
![\"accident\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F8faac983ba1c8881c9990bb7f0574c8c%2F3%20accident.png?generation=1727207988946467&alt=media\")
\nMore samples and the source code can be found in the EDA which makes sense.
\nEDIT: The competition host has given more background information about the actigraphy data here.
Actigraphy files tell us a lot about people's lives. Look at the following:
\nActigraphy files tell us a lot about people's lives. Look at the following:
\n@ambrosm I agree, we have a lot of such issues in the data. A glance at the time series plots elicits a lot of such misnomers. Is the data actual/ synthetic?
", "rawMarkdown": "ambrosm I agree, we have a lot of such issues in the data. A glance at the time series plots elicits a lot of such misnomers. Is the data actual/ synthetic?", "votes": null }, { "id": "2997916", "postDate": "09/25/2024 03:10:55", "content": "Great observation! I agree, the data seems to be very noisy… I am in the process of doing this analysis as well, and for now I can add that some points in the description of the files are incorrect or misleading:
\n\n\nsome participants were given an accelerometer to wear for up to 30 days continually
\n
But they actually wore the device for 1 to 81 days (calculated as the number of unique days in relative_date_PCIAT).
\n\n\ntime_of_day - Time of day representing the start of a 5s window that the data has been sampled over, with format %H:%M:%S.%9f.
\n
Time differences between steps can be much larger than 5 seconds, e.g. up to 43 hours for id=0417c91e. To illustrate, this is data from one of the days of this recording (x axis is relative to PCIAT date + time):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12193662%2Fa77693caf5b902c0efedbaad92a6fa9e%2FScreenshot%202024-09-25%20055257.png?generation=1727232809996184&alt=media\")
The bottom plot (in red) is for the time difference between steps in seconds, with a maximum gap of about 1 hour (rigth end). During these irregular periods, measurements are still being recorded, but it is unclear whether these data points are valid (they intentionally cut off some time periods, or there were technical issues/glitches of the devices).
\nNot sure how to clean this actigraphy data… maybe remove suspicious periods?
", "rawMarkdown": "Great observation! I agree, the data seems to be very noisy... I am in the process of doing this analysis as well, and for now I can add that some points in the description of the files are incorrect or misleading: \n\n>some participants were given an accelerometer to wear for up to 30 days continually\n\nBut they actually wore the device for 1 to 81 days (calculated as the number of unique days in relative_date_PCIAT).\n\n>time_of_day - Time of day representing the start of a 5s window that the data has been sampled over, with format %H:%M:%S.%9f.\n\nTime differences between steps can be much larger than 5 seconds, e.g. up to 43 hours for id=0417c91e. To illustrate, this is data from one of the days of this recording (x axis is relative to PCIAT date + time):\n\n\nThe bottom plot (in red) is for the time difference between steps in seconds, with a maximum gap of about 1 hour (rigth end). During these irregular periods, measurements are still being recorded, but it is unclear whether these data points are valid (they intentionally cut off some time periods, or there were technical issues/glitches of the devices).\n\nNot sure how to clean this actigraphy data... maybe remove suspicious periods?", "votes": null }, { "id": "2999319", "postDate": "09/26/2024 15:06:29", "content": "Other explanation of point 3: The nine year old boy got pissed at the wristband and dropped it from a skyscraper.
", "rawMarkdown": "Other explanation of point 3: The nine year old boy got pissed at the wristband and dropped it from a skyscraper.", "votes": null }, { "id": "2999422", "postDate": "09/26/2024 16:24:41", "content": "or maybe he's been working in a coal mine since birth… who knows :)
", "rawMarkdown": "or maybe he's been working in a coal mine since birth... who knows :)", "votes": null }, { "id": "2999679", "postDate": "09/26/2024 19:17:46", "content": "It is absolutely true that the data does not seem to realistic. However, wouldn't averaging the data over long periods of time fix the effect of the rubbish data. It is absolutely true that the interpolation issue would override this.
\nThresholiding could help with the other kinds of anomalies right?
", "rawMarkdown": "It is absolutely true that the data does not seem to realistic. However, wouldn't averaging the data over long periods of time fix the effect of the rubbish data. It is absolutely true that the interpolation issue would override this.\n\nThresholiding could help with the other kinds of anomalies right?", "votes": null }, { "id": "2999719", "postDate": "09/26/2024 20:14:26", "content": "Correct, implementation of lag features and rolling statistics can help reduce noise.
", "rawMarkdown": "Correct, implementation of lag features and rolling statistics can help reduce noise.", "votes": null }, { "id": "3047135", "postDate": "11/16/2024 09:35:12", "content": "Really great insights!!
", "rawMarkdown": "Really great insights!!", "votes": null } ], "comments": [ { "id": 2997720, "author_name": "ravi20076", "author_url": "", "post_date": "09/24/2024 20:39:14", "content": "@ambrosm I agree, we have a lot of such issues in the data. A glance at the time series plots elicits a lot of such misnomers. Is the data actual/ synthetic?
", "votes": null, "replies": [] }, { "id": 2997916, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "09/25/2024 03:10:55", "content": "Great observation! I agree, the data seems to be very noisy… I am in the process of doing this analysis as well, and for now I can add that some points in the description of the files are incorrect or misleading:
\n\n\nsome participants were given an accelerometer to wear for up to 30 days continually
\n
But they actually wore the device for 1 to 81 days (calculated as the number of unique days in relative_date_PCIAT).
\n\n\ntime_of_day - Time of day representing the start of a 5s window that the data has been sampled over, with format %H:%M:%S.%9f.
\n
Time differences between steps can be much larger than 5 seconds, e.g. up to 43 hours for id=0417c91e. To illustrate, this is data from one of the days of this recording (x axis is relative to PCIAT date + time):
\n
The bottom plot (in red) is for the time difference between steps in seconds, with a maximum gap of about 1 hour (rigth end). During these irregular periods, measurements are still being recorded, but it is unclear whether these data points are valid (they intentionally cut off some time periods, or there were technical issues/glitches of the devices).
\nNot sure how to clean this actigraphy data… maybe remove suspicious periods?
", "votes": null, "replies": [] }, { "id": 2999319, "author_name": "antdes", "author_url": "", "post_date": "09/26/2024 15:06:29", "content": "Other explanation of point 3: The nine year old boy got pissed at the wristband and dropped it from a skyscraper.
", "votes": null, "replies": [] }, { "id": 2999422, "author_name": "evanhislupus", "author_url": "", "post_date": "09/26/2024 16:24:41", "content": "or maybe he's been working in a coal mine since birth… who knows :)
", "votes": null, "replies": [] }, { "id": 2999679, "author_name": "luccarodriguez", "author_url": "", "post_date": "09/26/2024 19:17:46", "content": "It is absolutely true that the data does not seem to realistic. However, wouldn't averaging the data over long periods of time fix the effect of the rubbish data. It is absolutely true that the interpolation issue would override this.
\nThresholiding could help with the other kinds of anomalies right?
", "votes": null, "replies": [ { "id": 2999719, "author_name": "ethanp345", "author_url": "", "post_date": "09/26/2024 20:14:26", "content": "Correct, implementation of lag features and rolling statistics can help reduce noise.
", "votes": null, "replies": [] } ] }, { "id": 3047135, "author_name": "beforeikillyou", "author_url": "", "post_date": "11/16/2024 09:35:12", "content": "Really great insights!!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2997696": "Actigraphy files tell us a lot about people's lives. Look at the following:\n\n1. This 15-year old left-handed boy never moves much (enmo < 0.5), and he saw daylight only once in a whole month. Maybe he's in prison:\n\n\n\n2. This time series shows some strange ramps in the illuminance measurement. It looks like data were missing and somebody filled the blanks by linear interpolation. I'd prefer to get the raw data without undocumented preprocessing applied:\n\n\n\n3. This poor nine-year old boy (id 0668373f) had a quiet life for two and a half weeks, then he had an accident with an acceleration of 12 g. After the accident, the device immediately stopped recording data; let's hope that the boy survived!\n\n\n\n\nMore samples and the source code can be found in the [EDA which makes sense](https://www.kaggle.com/code/ambrosm/piu-eda-which-makes-sense).\n\nEDIT: The competition host has given more background information about the actigraphy data [here](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/537145).", "2997720": "ambrosm I agree, we have a lot of such issues in the data. A glance at the time series plots elicits a lot of such misnomers. Is the data actual/ synthetic?", "2997916": "Great observation! I agree, the data seems to be very noisy... I am in the process of doing this analysis as well, and for now I can add that some points in the description of the files are incorrect or misleading: \n\n>some participants were given an accelerometer to wear for up to 30 days continually\n\nBut they actually wore the device for 1 to 81 days (calculated as the number of unique days in relative_date_PCIAT).\n\n>time_of_day - Time of day representing the start of a 5s window that the data has been sampled over, with format %H:%M:%S.%9f.\n\nTime differences between steps can be much larger than 5 seconds, e.g. up to 43 hours for id=0417c91e. To illustrate, this is data from one of the days of this recording (x axis is relative to PCIAT date + time):\n\n\nThe bottom plot (in red) is for the time difference between steps in seconds, with a maximum gap of about 1 hour (rigth end). During these irregular periods, measurements are still being recorded, but it is unclear whether these data points are valid (they intentionally cut off some time periods, or there were technical issues/glitches of the devices).\n\nNot sure how to clean this actigraphy data... maybe remove suspicious periods?", "2999319": "Other explanation of point 3: The nine year old boy got pissed at the wristband and dropped it from a skyscraper.", "2999422": "or maybe he's been working in a coal mine since birth... who knows :)", "2999679": "It is absolutely true that the data does not seem to realistic. However, wouldn't averaging the data over long periods of time fix the effect of the rubbish data. It is absolutely true that the interpolation issue would override this.\n\nThresholiding could help with the other kinds of anomalies right?", "2999719": "Correct, implementation of lag features and rolling statistics can help reduce noise.", "3047135": "Really great insights!!" }, "source": "meta" }