{
  "id": 216780,
  "title": "Timestamp errors or data just not sorted?",
  "url": "/competitions/indoor-location-navigation/discussion/216780",
  "author_name": "",
  "post_date": "2021-02-04T02:55:26.540000600Z",
  "votes": 14,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I found several places where smaller timestamp follows bigger.<br>\nFor example in file<br>\n<code>/kaggle/input/indoor-location-navigation/test/046cfa46be49fc10834815c6.txt</code><br>\n0000000008631 follows 0000000008991</p>\n<p><a href=\"https://postimg.cc/svG5yb4z\" target=\"_blank\"><img src=\"https://i.postimg.cc/zX07wrLy/Screenshot-from-2021-02-04-05-36-05.png\" alt=\"Screenshot-from-2021-02-04-05-36-05.png\"></a></p>\n<p><a href=\"https://postimg.cc/w37mcnPN\" target=\"_blank\"><img src=\"https://i.postimg.cc/Kc5DF2D0/Screenshot-from-2021-02-04-05-37-21.png\" alt=\"Screenshot-from-2021-02-04-05-37-21.png\"></a></p>\n<p>(trace parsing script taken from <a href=\"https://www.kaggle.com/c/indoor-location-navigation/discussion/215381\" target=\"_blank\">this post</a>)</p>\n<p>Is this data just not sorted or has errors in timestamps? Maybe this is related to the data collection process?</p>",
  "messages": [
    {
      "id": "1185188",
      "postDate": "02/04/2021 02:55:26",
      "content": "<p>I found several places where smaller timestamp follows bigger.<br>\nFor example in file<br>\n<code>/kaggle/input/indoor-location-navigation/test/046cfa46be49fc10834815c6.txt</code><br>\n0000000008631 follows 0000000008991</p>\n<p><a href=\"https://postimg.cc/svG5yb4z\" target=\"_blank\"><img src=\"https://i.postimg.cc/zX07wrLy/Screenshot-from-2021-02-04-05-36-05.png\" alt=\"Screenshot-from-2021-02-04-05-36-05.png\"></a></p>\n<p><a href=\"https://postimg.cc/w37mcnPN\" target=\"_blank\"><img src=\"https://i.postimg.cc/Kc5DF2D0/Screenshot-from-2021-02-04-05-37-21.png\" alt=\"Screenshot-from-2021-02-04-05-37-21.png\"></a></p>\n<p>(trace parsing script taken from <a href=\"https://www.kaggle.com/c/indoor-location-navigation/discussion/215381\" target=\"_blank\">this post</a>)</p>\n<p>Is this data just not sorted or has errors in timestamps? Maybe this is related to the data collection process?</p>",
      "rawMarkdown": "I found several places where smaller timestamp follows bigger.\nFor example in file\n`/kaggle/input/indoor-location-navigation/test/046cfa46be49fc10834815c6.txt`\n0000000008631 follows 0000000008991\n\n[![Screenshot-from-2021-02-04-05-36-05.png](https://i.postimg.cc/zX07wrLy/Screenshot-from-2021-02-04-05-36-05.png)](https://postimg.cc/svG5yb4z)\n\n\n[![Screenshot-from-2021-02-04-05-37-21.png](https://i.postimg.cc/Kc5DF2D0/Screenshot-from-2021-02-04-05-37-21.png)](https://postimg.cc/w37mcnPN)\n\n(trace parsing script taken from [this post](https://www.kaggle.com/c/indoor-location-navigation/discussion/215381))\n\nIs this data just not sorted or has errors in timestamps? Maybe this is related to the data collection process?",
      "votes": null
    },
    {
      "id": "1190693",
      "postDate": "02/07/2021 22:55:38",
      "content": "<p>I'm having the same doubt </p>",
      "rawMarkdown": "I'm having the same doubt",
      "votes": null
    },
    {
      "id": "1208947",
      "postDate": "02/18/2021 15:24:52",
      "content": "<p>I am new to this competition and have just downloaded the dataset. I did not get a chance to inspect the files. But what comes to my mind is- Both the timestamps correspond to different types. Time 8991 is accelerometer while 8631 i beacon type. So my guess is that is should be okay while running any form of solution. It would be however, surprising to find, for example, time : 8000 for type beacon followed by time: 7500 for the same type beacon. can you post if you come across such a case?</p>",
      "rawMarkdown": "I am new to this competition and have just downloaded the dataset. I did not get a chance to inspect the files. But what comes to my mind is- Both the timestamps correspond to different types. Time 8991 is accelerometer while 8631 i beacon type. So my guess is that is should be okay while running any form of solution. It would be however, surprising to find, for example, time : 8000 for type beacon followed by time: 7500 for the same type beacon. can you post if you come across such a case?",
      "votes": null
    },
    {
      "id": "1211569",
      "postDate": "02/20/2021 10:34:41",
      "content": "<p>Two possibilities: assuming different log sources flush their buffers at different times, you should expect some bunching and being out of line. I suspect that's what's happening for TYPE_BEACON, at least: you see the TYPE_BEACON data flushed in different intervals than the sensor data, so they show up as \"blocks\". </p>\n<p>The second possibility is that it has to do with different time sources being used. The sensors carry their own time while the beacon and WiFi data uses system time <a href=\"https://github.com/location-competition/indoor-location-competition-20\" target=\"_blank\">https://github.com/location-competition/indoor-location-competition-20</a></p>\n<p>Most likely it's a combination of the two factors. TYPE_WIFI is also sometimes out of line with the sensor data, but only carries one timestamp per block.</p>\n<p>So tl;dr: it's likely safe to assume that data is ordered for each set of classes (sensors, beacon and wifi), but not across the classes, so you should do your own sorting if you need a coherent log across the different classes of information. It is also likely that the times don't match up perfectly across classes</p>",
      "rawMarkdown": "Two possibilities: assuming different log sources flush their buffers at different times, you should expect some bunching and being out of line. I suspect that's what's happening for TYPE_BEACON, at least: you see the TYPE_BEACON data flushed in different intervals than the sensor data, so they show up as \"blocks\". \n\nThe second possibility is that it has to do with different time sources being used. The sensors carry their own time while the beacon and WiFi data uses system time https://github.com/location-competition/indoor-location-competition-20\n\nMost likely it's a combination of the two factors. TYPE_WIFI is also sometimes out of line with the sensor data, but only carries one timestamp per block.\n\nSo tl;dr: it's likely safe to assume that data is ordered for each set of classes (sensors, beacon and wifi), but not across the classes, so you should do your own sorting if you need a coherent log across the different classes of information. It is also likely that the times don't match up perfectly across classes",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1190693,
      "author_name": "tpothjuan",
      "author_url": "",
      "post_date": "02/07/2021 22:55:38",
      "content": "<p>I'm having the same doubt </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1208947,
      "author_name": "shreyasinha2506",
      "author_url": "",
      "post_date": "02/18/2021 15:24:52",
      "content": "<p>I am new to this competition and have just downloaded the dataset. I did not get a chance to inspect the files. But what comes to my mind is- Both the timestamps correspond to different types. Time 8991 is accelerometer while 8631 i beacon type. So my guess is that is should be okay while running any form of solution. It would be however, surprising to find, for example, time : 8000 for type beacon followed by time: 7500 for the same type beacon. can you post if you come across such a case?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1211569,
      "author_name": "nepharan",
      "author_url": "",
      "post_date": "02/20/2021 10:34:41",
      "content": "<p>Two possibilities: assuming different log sources flush their buffers at different times, you should expect some bunching and being out of line. I suspect that's what's happening for TYPE_BEACON, at least: you see the TYPE_BEACON data flushed in different intervals than the sensor data, so they show up as \"blocks\". </p>\n<p>The second possibility is that it has to do with different time sources being used. The sensors carry their own time while the beacon and WiFi data uses system time <a href=\"https://github.com/location-competition/indoor-location-competition-20\" target=\"_blank\">https://github.com/location-competition/indoor-location-competition-20</a></p>\n<p>Most likely it's a combination of the two factors. TYPE_WIFI is also sometimes out of line with the sensor data, but only carries one timestamp per block.</p>\n<p>So tl;dr: it's likely safe to assume that data is ordered for each set of classes (sensors, beacon and wifi), but not across the classes, so you should do your own sorting if you need a coherent log across the different classes of information. It is also likely that the times don't match up perfectly across classes</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1185188": "I found several places where smaller timestamp follows bigger.\nFor example in file\n`/kaggle/input/indoor-location-navigation/test/046cfa46be49fc10834815c6.txt`\n0000000008631 follows 0000000008991\n\n[![Screenshot-from-2021-02-04-05-36-05.png](https://i.postimg.cc/zX07wrLy/Screenshot-from-2021-02-04-05-36-05.png)](https://postimg.cc/svG5yb4z)\n\n\n[![Screenshot-from-2021-02-04-05-37-21.png](https://i.postimg.cc/Kc5DF2D0/Screenshot-from-2021-02-04-05-37-21.png)](https://postimg.cc/w37mcnPN)\n\n(trace parsing script taken from [this post](https://www.kaggle.com/c/indoor-location-navigation/discussion/215381))\n\nIs this data just not sorted or has errors in timestamps? Maybe this is related to the data collection process?",
    "1190693": "I'm having the same doubt",
    "1208947": "I am new to this competition and have just downloaded the dataset. I did not get a chance to inspect the files. But what comes to my mind is- Both the timestamps correspond to different types. Time 8991 is accelerometer while 8631 i beacon type. So my guess is that is should be okay while running any form of solution. It would be however, surprising to find, for example, time : 8000 for type beacon followed by time: 7500 for the same type beacon. can you post if you come across such a case?",
    "1211569": "Two possibilities: assuming different log sources flush their buffers at different times, you should expect some bunching and being out of line. I suspect that's what's happening for TYPE_BEACON, at least: you see the TYPE_BEACON data flushed in different intervals than the sensor data, so they show up as \"blocks\". \n\nThe second possibility is that it has to do with different time sources being used. The sensors carry their own time while the beacon and WiFi data uses system time https://github.com/location-competition/indoor-location-competition-20\n\nMost likely it's a combination of the two factors. TYPE_WIFI is also sometimes out of line with the sensor data, but only carries one timestamp per block.\n\nSo tl;dr: it's likely safe to assume that data is ordered for each set of classes (sensors, beacon and wifi), but not across the classes, so you should do your own sorting if you need a coherent log across the different classes of information. It is also likely that the times don't match up perfectly across classes"
  },
  "source": "meta"
}