{
  "id": 537145,
  "title": "A note on the actigraphy data",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/537145",
  "author_name": "AdammbSantorelli",
  "post_date": "2024-10-01T15:26:40.866000",
  "votes": 38,
  "comment_count": 21,
  "views": 0,
  "content": "<p>Hi all,</p>\n<p>Adam from CMI here, as I have seen quite a bit of discussion in regards to the actigraphy data I would like to share some comments that might alleviate some concerns that have been arising.</p>\n<p>Data was collected with these <a href=\"https://theactigraph.com/actigraph-wgt3x-bt\" target=\"_blank\">watches</a>. Note that there is a setting called <code>idle_sleep_mode</code>, read more about it <a href=\"https://actigraphcorp.my.site.com/support/s/article/Idle-Sleep-Mode-Explained\" target=\"_blank\">here</a>. Unfortunately, approximately half the data was collected with this setting enabled. As one could infer from the name, during periods of no motion (sleep, idle time, or non-wear) the watch will not record any data. This explains the observed time gaps (non-5s resolution in ENMO) that are seen in some participants. At this moment we are not doing any imputing/modifications to any of the recorded acceleration data as we cannot guarantee what is actually happening during the <code>idle_sleep_mode</code> times. An additional note on data collected with <code>idle_sleep_mode</code> enabled, it causes issues with our non-wear detection algorithm since there are, potentially, no samples in the given time window being considered. It is a current issue we are exploring on how to best solve.</p>\n<p>As for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate. This is the caused for the observed 'ramping' in some participants, this should only occur when <code>idle_sleep_mode</code> is enabled).</p>\n<p>Feel free to ask any actigraphy data related questions here and I'll answer. </p>\n<p>Thanks, hope this helps.</p>",
  "messages": [
    {
      "id": 3004151,
      "postDate": "2024-10-01T15:26:40.867Z",
      "content": "<p>Hi all,</p>\n<p>Adam from CMI here, as I have seen quite a bit of discussion in regards to the actigraphy data I would like to share some comments that might alleviate some concerns that have been arising.</p>\n<p>Data was collected with these <a href=\"https://theactigraph.com/actigraph-wgt3x-bt\" target=\"_blank\">watches</a>. Note that there is a setting called <code>idle_sleep_mode</code>, read more about it <a href=\"https://actigraphcorp.my.site.com/support/s/article/Idle-Sleep-Mode-Explained\" target=\"_blank\">here</a>. Unfortunately, approximately half the data was collected with this setting enabled. As one could infer from the name, during periods of no motion (sleep, idle time, or non-wear) the watch will not record any data. This explains the observed time gaps (non-5s resolution in ENMO) that are seen in some participants. At this moment we are not doing any imputing/modifications to any of the recorded acceleration data as we cannot guarantee what is actually happening during the <code>idle_sleep_mode</code> times. An additional note on data collected with <code>idle_sleep_mode</code> enabled, it causes issues with our non-wear detection algorithm since there are, potentially, no samples in the given time window being considered. It is a current issue we are exploring on how to best solve.</p>\n<p>As for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate. This is the caused for the observed 'ramping' in some participants, this should only occur when <code>idle_sleep_mode</code> is enabled).</p>\n<p>Feel free to ask any actigraphy data related questions here and I'll answer. </p>\n<p>Thanks, hope this helps.</p>",
      "rawMarkdown": "Hi all,\n\nAdam from CMI here, as I have seen quite a bit of discussion in regards to the actigraphy data I would like to share some comments that might alleviate some concerns that have been arising.\n\nData was collected with these [watches](https://theactigraph.com/actigraph-wgt3x-bt). Note that there is a setting called `idle_sleep_mode`, read more about it [here](https://actigraphcorp.my.site.com/support/s/article/Idle-Sleep-Mode-Explained). Unfortunately, approximately half the data was collected with this setting enabled. As one could infer from the name, during periods of no motion (sleep, idle time, or non-wear) the watch will not record any data. This explains the observed time gaps (non-5s resolution in ENMO) that are seen in some participants. At this moment we are not doing any imputing/modifications to any of the recorded acceleration data as we cannot guarantee what is actually happening during the `idle_sleep_mode` times. An additional note on data collected with `idle_sleep_mode` enabled, it causes issues with our non-wear detection algorithm since there are, potentially, no samples in the given time window being considered. It is a current issue we are exploring on how to best solve.\n\nAs for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate. This is the caused for the observed 'ramping' in some participants, this should only occur when `idle_sleep_mode` is enabled).\n\nFeel free to ask any actigraphy data related questions here and I'll answer. \n\nThanks, hope this helps.",
      "votes": 37
    },
    {
      "id": 3006274,
      "postDate": "2024-10-04T01:32:04.617Z",
      "content": "<p>Hi,</p>\n<p>Could you please confirm if the actigraphy data would be missing for majority of hidden test data, as we are seeing in train and test sample data? (only 996 out of 3960 in train, 2 out of 20 in sampled test have coresponding actigraphy data)</p>\n<p>Thanks,</p>",
      "rawMarkdown": "Hi,\n\nCould you please confirm if the actigraphy data would be missing for majority of hidden test data, as we are seeing in train and test sample data? (only 996 out of 3960 in train, 2 out of 20 in sampled test have coresponding actigraphy data)\n\nThanks,",
      "votes": 4,
      "replies": [
        {
          "id": 3014735,
          "postDate": "2024-10-11T14:41:12.543Z",
          "content": "<p>Sorry, I cannot disclose what percentage of the hidden test data does or does not have actigraphy data. One could assume that much like in reality, sometimes we are able to collect data from all modalities and other times we cannot, and of course cannot predict which exact sets of data we will have access to for all incoming new data.</p>",
          "rawMarkdown": "Sorry, I cannot disclose what percentage of the hidden test data does or does not have actigraphy data. One could assume that much like in reality, sometimes we are able to collect data from all modalities and other times we cannot, and of course cannot predict which exact sets of data we will have access to for all incoming new data."
        }
      ]
    },
    {
      "id": 3063657,
      "postDate": "2024-12-04T18:14:21.557Z",
      "content": "<p>Can you provide us the original data before linear interpolation?</p>",
      "rawMarkdown": "Can you provide us the original data before linear interpolation?",
      "votes": 1
    },
    {
      "id": 3012389,
      "postDate": "2024-10-09T02:08:58.397Z",
      "content": "<p>\"As for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate.\"</p>\n<p>Hi, Did the test data were being upsampled as well?<br>\nJust find it strange, when I remove battery voltage LB score reduced….</p>",
      "rawMarkdown": "\"As for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate.\"\n\nHi, Did the test data were being upsampled as well?\nJust find it strange, when I remove battery voltage LB score reduced....",
      "votes": 1
    },
    {
      "id": 3008146,
      "postDate": "2024-10-06T09:30:29.720Z",
      "content": "<p>Hi Adam,</p>\n<p>Thanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?</p>\n<p>Thanks for your help!</p>",
      "rawMarkdown": "Hi Adam,\n\nThanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?\n\nThanks for your help!",
      "votes": 1,
      "replies": [
        {
          "id": 3008418,
          "postDate": "2024-10-06T14:56:47.840Z",
          "content": "<p>Hi,<br>\nWe are currently investigating how best to handle <code>idle_sleep_mode</code> enabled data, but it is also an important aspect of this competition!<br>\nMy best answer would be that as per the Actigraph description, during those time gaps the sensor is not measuring any changes in acceleration. Of course the question is: is that due to non-wear or periods of stillness?<br>\nAdditionally, while the interpolated battery voltage is probably 'close' to the truth during those periods, I would caution using the interpolated light data during those time gaps as it would not be indicative of real data.</p>",
          "rawMarkdown": "Hi,\nWe are currently investigating how best to handle `idle_sleep_mode` enabled data, but it is also an important aspect of this competition!\nMy best answer would be that as per the Actigraph description, during those time gaps the sensor is not measuring any changes in acceleration. Of course the question is: is that due to non-wear or periods of stillness?\nAdditionally, while the interpolated battery voltage is probably 'close' to the truth during those periods, I would caution using the interpolated light data during those time gaps as it would not be indicative of real data."
        }
      ]
    },
    {
      "id": 3004524,
      "postDate": "2024-10-01T23:09:06.320Z",
      "content": "<p>Thanks very much for clarifying this and giving a clear explanation for the differences in the parquet data files!<br>\nfyi, I recently realized I can unify the data format by removing non-wears and also very low activity times in files that have non-zero non-wear flags -- effectively applying <code>idle_sleep_mode</code> 😆</p>",
      "rawMarkdown": "Thanks very much for clarifying this and giving a clear explanation for the differences in the parquet data files!\nfyi, I recently realized I can unify the data format by removing non-wears and also very low activity times in files that have non-zero non-wear flags -- effectively applying `idle_sleep_mode` 😆",
      "votes": 2
    },
    {
      "id": 3067634,
      "postDate": "2024-12-09T13:37:39.563Z",
      "content": "<p>great!!<br>\nI've gained a lot</p>",
      "rawMarkdown": "great!!\nI've gained a lot",
      "votes": -1
    },
    {
      "id": 3035328,
      "postDate": "2024-11-03T09:25:34.907Z",
      "content": "<p>Doubt in how to get started with this problem ?</p>\n<p>I am stuck with this problem statement and would like to understand the initial approach to tackle such a problem.</p>\n<ul>\n<li><p>There are many labels missing. How can I train my model based on the remaining data?</p></li>\n<li><p>The number of columns or features differs between the Test and Train datasets. How can I incorporate this? How can I predict the Test set without the complete set of features used in the Train set?</p></li>\n<li><p>There is a set of data in “parquet” format, available for only a subset of IDs. I thought of concatenating the parquet data with the CSV data, but this is not possible as some IDs do not have parquet data since the wrist accelerometer was worn by selected participants only. What should I do?</p></li>\n</ul>\n<p>I am new to these types of problems, so please help me learn from this situation.</p>",
      "rawMarkdown": "Doubt in how to get started with this problem ?\n\nI am stuck with this problem statement and would like to understand the initial approach to tackle such a problem.\n\n- There are many labels missing. How can I train my model based on the remaining data?\n\n- The number of columns or features differs between the Test and Train datasets. How can I incorporate this? How can I predict the Test set without the complete set of features used in the Train set?\n\n- There is a set of data in “parquet” format, available for only a subset of IDs. I thought of concatenating the parquet data with the CSV data, but this is not possible as some IDs do not have parquet data since the wrist accelerometer was worn by selected participants only. What should I do?\n\nI am new to these types of problems, so please help me learn from this situation.",
      "votes": -3
    },
    {
      "id": 3010051,
      "postDate": "2024-10-08T16:03:54.357Z",
      "content": "<p>Hi, are you intrested in become my team member in child mind problematic internet use. If you intrested please contact me.</p>\n<p>Thank you</p>",
      "rawMarkdown": "Hi, are you intrested in become my team member in child mind problematic internet use. If you intrested please contact me.\n\nThank you",
      "votes": -4
    },
    {
      "id": 3008945,
      "postDate": "2024-10-07T10:02:49.347Z",
      "content": "<p>Hi Adam,</p>\n<p>Thanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?</p>\n<p>Thanks for your help!</p>",
      "rawMarkdown": "Hi Adam,\n\nThanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?\n\nThanks for your help!",
      "votes": -1
    },
    {
      "id": 3007379,
      "postDate": "2024-10-05T09:20:36.800Z",
      "content": "<p>Hi there! I am wondering how do we interpret <code>weekday</code> column: does 1 mean Sunday or Monday?</p>",
      "rawMarkdown": "Hi there! I am wondering how do we interpret `weekday` column: does 1 mean Sunday or Monday?",
      "replies": [
        {
          "id": 3007380,
          "postDate": "2024-10-05T09:21:48.893Z",
          "content": "<p>Furthermore, it looks like the wristpy repository is trying to implement the functionalities of the R GGIR package. I have gone through the entire documentation of it and I would love to contribute to wristpy if needed!</p>",
          "rawMarkdown": "Furthermore, it looks like the wristpy repository is trying to implement the functionalities of the R GGIR package. I have gone through the entire documentation of it and I would love to contribute to wristpy if needed!",
          "replies": [
            {
              "id": 3008421,
              "postDate": "2024-10-06T14:58:01.390Z",
              "content": "<p>Thanks! We welcome any bug reports/issues and of course PRs for any functionality you might think of. Currently, we are very close to an 'official' public release, and have quite a few new features planned for immediate post-release updates!</p>",
              "rawMarkdown": "Thanks! We welcome any bug reports/issues and of course PRs for any functionality you might think of. Currently, we are very close to an 'official' public release, and have quite a few new features planned for immediate post-release updates!",
              "votes": 2
            }
          ]
        },
        {
          "id": 3008406,
          "postDate": "2024-10-06T14:39:13.997Z",
          "content": "<p>Hi, <br>\nAs was stated in the Data section: \"The day of the week, coded as an integer with 1 being Monday and 7 being Sunday.\"<br>\nIt was extracting using the <a href=\"https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.weekday.html\" target=\"_blank\">polars .weekday() expression</a> from a datetime pl Series.</p>",
          "rawMarkdown": "Hi, \nAs was stated in the Data section: \"The day of the week, coded as an integer with 1 being Monday and 7 being Sunday.\"\nIt was extracting using the [polars .weekday() expression](https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.weekday.html) from a datetime pl Series.",
          "votes": 2
        }
      ]
    },
    {
      "id": 3028233,
      "postDate": "2024-10-25T18:28:15.537Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 3028187,
      "postDate": "2024-10-25T17:36:30.770Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 3013923,
      "postDate": "2024-10-10T17:33:52.563Z",
      "content": "<p>Hi,<br>\nCould you please confirm the ratio of missing actigraphy data in hidden test data?</p>",
      "rawMarkdown": "Hi,\nCould you please confirm the ratio of missing actigraphy data in hidden test data?",
      "votes": -2,
      "isDeleted": true
    },
    {
      "id": 3069939,
      "postDate": "2024-12-12T04:29:20.693Z",
      "content": "<p>Thank you very much.</p>",
      "rawMarkdown": "Thank you very much."
    },
    {
      "id": 3065039,
      "postDate": "2024-12-06T09:17:33.040Z",
      "content": "<p>thank you for the clarification!</p>",
      "rawMarkdown": "thank you for the clarification!"
    },
    {
      "id": 3015015,
      "postDate": "2024-10-11T20:31:49.740Z",
      "content": "<p>Thank you, this is interesting.</p>",
      "rawMarkdown": "Thank you, this is interesting."
    }
  ],
  "comments": [
    {
      "id": 3006274,
      "author_name": "Yannan Chen",
      "author_url": "",
      "post_date": "2024-10-04T01:32:04.617000",
      "content": "<p>Hi,</p>\n<p>Could you please confirm if the actigraphy data would be missing for majority of hidden test data, as we are seeing in train and test sample data? (only 996 out of 3960 in train, 2 out of 20 in sampled test have coresponding actigraphy data)</p>\n<p>Thanks,</p>",
      "votes": 4,
      "replies": [
        {
          "id": 3014735,
          "author_name": "AdammbSantorelli",
          "author_url": "",
          "post_date": "2024-10-11T14:41:12.543000",
          "content": "<p>Sorry, I cannot disclose what percentage of the hidden test data does or does not have actigraphy data. One could assume that much like in reality, sometimes we are able to collect data from all modalities and other times we cannot, and of course cannot predict which exact sets of data we will have access to for all incoming new data.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3063657,
      "author_name": "madmax0404",
      "author_url": "",
      "post_date": "2024-12-04T18:14:21.557000",
      "content": "<p>Can you provide us the original data before linear interpolation?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3012389,
      "author_name": "Ruby Hartono",
      "author_url": "",
      "post_date": "2024-10-09T02:08:58.397000",
      "content": "<p>\"As for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate.\"</p>\n<p>Hi, Did the test data were being upsampled as well?<br>\nJust find it strange, when I remove battery voltage LB score reduced….</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3008146,
      "author_name": "CYXISDTC",
      "author_url": "",
      "post_date": "2024-10-06T09:30:29.720000",
      "content": "<p>Hi Adam,</p>\n<p>Thanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?</p>\n<p>Thanks for your help!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3008418,
          "author_name": "AdammbSantorelli",
          "author_url": "",
          "post_date": "2024-10-06T14:56:47.840000",
          "content": "<p>Hi,<br>\nWe are currently investigating how best to handle <code>idle_sleep_mode</code> enabled data, but it is also an important aspect of this competition!<br>\nMy best answer would be that as per the Actigraph description, during those time gaps the sensor is not measuring any changes in acceleration. Of course the question is: is that due to non-wear or periods of stillness?<br>\nAdditionally, while the interpolated battery voltage is probably 'close' to the truth during those periods, I would caution using the interpolated light data during those time gaps as it would not be indicative of real data.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3004524,
      "author_name": "Daniel Dewey",
      "author_url": "",
      "post_date": "2024-10-01T23:09:06.320000",
      "content": "<p>Thanks very much for clarifying this and giving a clear explanation for the differences in the parquet data files!<br>\nfyi, I recently realized I can unify the data format by removing non-wears and also very low activity times in files that have non-zero non-wear flags -- effectively applying <code>idle_sleep_mode</code> 😆</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 3067634,
      "author_name": "ann0ymous",
      "author_url": "",
      "post_date": "2024-12-09T13:37:39.563000",
      "content": "<p>great!!<br>\nI've gained a lot</p>",
      "votes": -1,
      "replies": []
    },
    {
      "id": 3035328,
      "author_name": "Freak2209",
      "author_url": "",
      "post_date": "2024-11-03T09:25:34.907000",
      "content": "<p>Doubt in how to get started with this problem ?</p>\n<p>I am stuck with this problem statement and would like to understand the initial approach to tackle such a problem.</p>\n<ul>\n<li><p>There are many labels missing. How can I train my model based on the remaining data?</p></li>\n<li><p>The number of columns or features differs between the Test and Train datasets. How can I incorporate this? How can I predict the Test set without the complete set of features used in the Train set?</p></li>\n<li><p>There is a set of data in “parquet” format, available for only a subset of IDs. I thought of concatenating the parquet data with the CSV data, but this is not possible as some IDs do not have parquet data since the wrist accelerometer was worn by selected participants only. What should I do?</p></li>\n</ul>\n<p>I am new to these types of problems, so please help me learn from this situation.</p>",
      "votes": -3,
      "replies": []
    },
    {
      "id": 3010051,
      "author_name": "Sadaf Mughees",
      "author_url": "",
      "post_date": "2024-10-08T16:03:54.357000",
      "content": "<p>Hi, are you intrested in become my team member in child mind problematic internet use. If you intrested please contact me.</p>\n<p>Thank you</p>",
      "votes": -4,
      "replies": []
    },
    {
      "id": 3008945,
      "author_name": "Swapnil Dwivedi",
      "author_url": "",
      "post_date": "2024-10-07T10:02:49.347000",
      "content": "<p>Hi Adam,</p>\n<p>Thanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?</p>\n<p>Thanks for your help!</p>",
      "votes": -1,
      "replies": []
    },
    {
      "id": 3007379,
      "author_name": "Hongyu Yang",
      "author_url": "",
      "post_date": "2024-10-05T09:20:36.800000",
      "content": "<p>Hi there! I am wondering how do we interpret <code>weekday</code> column: does 1 mean Sunday or Monday?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3007380,
          "author_name": "Hongyu Yang",
          "author_url": "",
          "post_date": "2024-10-05T09:21:48.893000",
          "content": "<p>Furthermore, it looks like the wristpy repository is trying to implement the functionalities of the R GGIR package. I have gone through the entire documentation of it and I would love to contribute to wristpy if needed!</p>",
          "votes": 0,
          "replies": [
            {
              "id": 3008421,
              "author_name": "AdammbSantorelli",
              "author_url": "",
              "post_date": "2024-10-06T14:58:01.390000",
              "content": "<p>Thanks! We welcome any bug reports/issues and of course PRs for any functionality you might think of. Currently, we are very close to an 'official' public release, and have quite a few new features planned for immediate post-release updates!</p>",
              "votes": 2,
              "replies": []
            }
          ]
        },
        {
          "id": 3008406,
          "author_name": "AdammbSantorelli",
          "author_url": "",
          "post_date": "2024-10-06T14:39:13.997000",
          "content": "<p>Hi, <br>\nAs was stated in the Data section: \"The day of the week, coded as an integer with 1 being Monday and 7 being Sunday.\"<br>\nIt was extracting using the <a href=\"https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.weekday.html\" target=\"_blank\">polars .weekday() expression</a> from a datetime pl Series.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 3028233,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-10-25T18:28:15.537000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3028187,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-10-25T17:36:30.770000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3013923,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-10-10T17:33:52.563000",
      "content": "<p>Hi,<br>\nCould you please confirm the ratio of missing actigraphy data in hidden test data?</p>",
      "votes": -2,
      "replies": []
    },
    {
      "id": 3069939,
      "author_name": "Mengsihan12138",
      "author_url": "",
      "post_date": "2024-12-12T04:29:20.693000",
      "content": "<p>Thank you very much.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3065039,
      "author_name": "jiali wu",
      "author_url": "",
      "post_date": "2024-12-06T09:17:33.040000",
      "content": "<p>thank you for the clarification!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3015015,
      "author_name": "Irina-Kondratenko",
      "author_url": "",
      "post_date": "2024-10-11T20:31:49.740000",
      "content": "<p>Thank you, this is interesting.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3004151": "Hi all,\n\nAdam from CMI here, as I have seen quite a bit of discussion in regards to the actigraphy data I would like to share some comments that might alleviate some concerns that have been arising.\n\nData was collected with these [watches](https://theactigraph.com/actigraph-wgt3x-bt). Note that there is a setting called `idle_sleep_mode`, read more about it [here](https://actigraphcorp.my.site.com/support/s/article/Idle-Sleep-Mode-Explained). Unfortunately, approximately half the data was collected with this setting enabled. As one could infer from the name, during periods of no motion (sleep, idle time, or non-wear) the watch will not record any data. This explains the observed time gaps (non-5s resolution in ENMO) that are seen in some participants. At this moment we are not doing any imputing/modifications to any of the recorded acceleration data as we cannot guarantee what is actually happening during the `idle_sleep_mode` times. An additional note on data collected with `idle_sleep_mode` enabled, it causes issues with our non-wear detection algorithm since there are, potentially, no samples in the given time window being considered. It is a current issue we are exploring on how to best solve.\n\nAs for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate. This is the caused for the observed 'ramping' in some participants, this should only occur when `idle_sleep_mode` is enabled).\n\nFeel free to ask any actigraphy data related questions here and I'll answer. \n\nThanks, hope this helps.",
    "3006274": "Hi,\n\nCould you please confirm if the actigraphy data would be missing for majority of hidden test data, as we are seeing in train and test sample data? (only 996 out of 3960 in train, 2 out of 20 in sampled test have coresponding actigraphy data)\n\nThanks,",
    "3063657": "Can you provide us the original data before linear interpolation?",
    "3012389": "\"As for the 'secondary' sensor information (light, battery voltage). These are measured at different sampling rates (from each other and from acceleration), and for the sake of exporting data they were upsampled (linear interpolation) to match the 5s sampling rate.\"\n\nHi, Did the test data were being upsampled as well?\nJust find it strange, when I remove battery voltage LB score reduced....",
    "3008146": "Hi Adam,\n\nThanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?\n\nThanks for your help!",
    "3004524": "Thanks very much for clarifying this and giving a clear explanation for the differences in the parquet data files!\nfyi, I recently realized I can unify the data format by removing non-wears and also very low activity times in files that have non-zero non-wear flags -- effectively applying `idle_sleep_mode` 😆",
    "3067634": "great!!\nI've gained a lot",
    "3035328": "Doubt in how to get started with this problem ?\n\nI am stuck with this problem statement and would like to understand the initial approach to tackle such a problem.\n\n- There are many labels missing. How can I train my model based on the remaining data?\n\n- The number of columns or features differs between the Test and Train datasets. How can I incorporate this? How can I predict the Test set without the complete set of features used in the Train set?\n\n- There is a set of data in “parquet” format, available for only a subset of IDs. I thought of concatenating the parquet data with the CSV data, but this is not possible as some IDs do not have parquet data since the wrist accelerometer was worn by selected participants only. What should I do?\n\nI am new to these types of problems, so please help me learn from this situation.",
    "3010051": "Hi, are you intrested in become my team member in child mind problematic internet use. If you intrested please contact me.\n\nThank you",
    "3008945": "Hi Adam,\n\nThanks for the clarification on the actigraphy data. Do you have any recommendations on how to handle the missing data for participants with idle_sleep_mode enabled, especially regarding data imputation and non-wear detection? Also, are there any specific considerations when analyzing the interpolated light and battery voltage data?\n\nThanks for your help!",
    "3007379": "Hi there! I am wondering how do we interpret `weekday` column: does 1 mean Sunday or Monday?",
    "3028233": "",
    "3028187": "",
    "3013923": "Hi,\nCould you please confirm the ratio of missing actigraphy data in hidden test data?",
    "3069939": "Thank you very much.",
    "3065039": "thank you for the clarification!",
    "3015015": "Thank you, this is interesting."
  }
}