{
  "id": 224987,
  "title": "Data time-ordering",
  "url": "/competitions/indoor-location-navigation/discussion/224987",
  "author_name": "Maksim",
  "post_date": "2021-03-10T13:32:05.244000",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>So, I have checked. Data has wrong time-ordering. I found 1 791 126 cases when ordering was wrong. For example, from one of train files: <br>\n1578462618794<br>\n1578462618826<br>\n1578462618814</p>\n<p>Here is my code to check:</p>\n<pre><code>train_path = '../input/indoor-location-navigation/train/'\ntrain_folders = os.listdir(train_path)\n\nmiss_timed_rows = 0\ntrain_files = 0\nfor mall in train_folders:\n    floors = os.listdir(train_path+mall)\n    for floor in floors:\n        cases = os.listdir(train_path+mall+'/'+floor)\n        for case in cases:\n            with open(train_path+mall+'/'+floor+'/'+case) as file:\n                train_files += 1\n                temp = file.read().split('\\n')\n                first = -1\n                second = -1\n                for row in temp[:len(temp)-1]:\n                    if row[0] == '#':\n                        continue\n                    parts = row.split('\\t')\n                    second = int(parts[0])\n                    if second &lt; first:\n                        miss_timed_rows += 1\n                        first = second\n                    else:\n                        first = second\nprint(miss_timed_rows)\nprint(train_files)\n</code></pre>",
  "messages": [
    {
      "id": 1233498,
      "postDate": "2021-03-10T13:32:05.243Z",
      "content": "<p>So, I have checked. Data has wrong time-ordering. I found 1 791 126 cases when ordering was wrong. For example, from one of train files: <br>\n1578462618794<br>\n1578462618826<br>\n1578462618814</p>\n<p>Here is my code to check:</p>\n<pre><code>train_path = '../input/indoor-location-navigation/train/'\ntrain_folders = os.listdir(train_path)\n\nmiss_timed_rows = 0\ntrain_files = 0\nfor mall in train_folders:\n    floors = os.listdir(train_path+mall)\n    for floor in floors:\n        cases = os.listdir(train_path+mall+'/'+floor)\n        for case in cases:\n            with open(train_path+mall+'/'+floor+'/'+case) as file:\n                train_files += 1\n                temp = file.read().split('\\n')\n                first = -1\n                second = -1\n                for row in temp[:len(temp)-1]:\n                    if row[0] == '#':\n                        continue\n                    parts = row.split('\\t')\n                    second = int(parts[0])\n                    if second &lt; first:\n                        miss_timed_rows += 1\n                        first = second\n                    else:\n                        first = second\nprint(miss_timed_rows)\nprint(train_files)\n</code></pre>",
      "rawMarkdown": "So, I have checked. Data has wrong time-ordering. I found 1 791 126 cases when ordering was wrong. For example, from one of train files: \n1578462618794\n1578462618826\n1578462618814\n\nHere is my code to check:\n```\ntrain_path = '../input/indoor-location-navigation/train/'\ntrain_folders = os.listdir(train_path)\n\nmiss_timed_rows = 0\ntrain_files = 0\nfor mall in train_folders:\n    floors = os.listdir(train_path+mall)\n    for floor in floors:\n        cases = os.listdir(train_path+mall+'/'+floor)\n        for case in cases:\n            with open(train_path+mall+'/'+floor+'/'+case) as file:\n                train_files += 1\n                temp = file.read().split('\\n')\n                first = -1\n                second = -1\n                for row in temp[:len(temp)-1]:\n                    if row[0] == '#':\n                        continue\n                    parts = row.split('\\t')\n                    second = int(parts[0])\n                    if second < first:\n                        miss_timed_rows += 1\n                        first = second\n                    else:\n                        first = second\nprint(miss_timed_rows)\nprint(train_files)\n```",
      "votes": 1
    },
    {
      "id": 1244166,
      "postDate": "2021-03-18T18:20:55.023Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1244166,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-03-18T18:20:55.023000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1233498": "So, I have checked. Data has wrong time-ordering. I found 1 791 126 cases when ordering was wrong. For example, from one of train files: \n1578462618794\n1578462618826\n1578462618814\n\nHere is my code to check:\n```\ntrain_path = '../input/indoor-location-navigation/train/'\ntrain_folders = os.listdir(train_path)\n\nmiss_timed_rows = 0\ntrain_files = 0\nfor mall in train_folders:\n    floors = os.listdir(train_path+mall)\n    for floor in floors:\n        cases = os.listdir(train_path+mall+'/'+floor)\n        for case in cases:\n            with open(train_path+mall+'/'+floor+'/'+case) as file:\n                train_files += 1\n                temp = file.read().split('\\n')\n                first = -1\n                second = -1\n                for row in temp[:len(temp)-1]:\n                    if row[0] == '#':\n                        continue\n                    parts = row.split('\\t')\n                    second = int(parts[0])\n                    if second < first:\n                        miss_timed_rows += 1\n                        first = second\n                    else:\n                        first = second\nprint(miss_timed_rows)\nprint(train_files)\n```",
    "1244166": ""
  }
}