{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Purpose:\n\nCalculate how long each trace takes to complete, calculated as:  \n\n(startTime - endTime)/(1000\\*60)  \n\nCompare the distribution between the train and test sets.\n\n### Interesting findings:\n\n* Maximum trace time: **18.2 minutes**\n* Many train traces took less than a minute to complete, while test traces always take more than a minute.\n    * The distribution of test trace completion times looks suspiciously like the train distribution shifted by 1 minute.\n* Roughly 2.0% of train traces took more than **5 minutes** to complete, compared to 4.8% of test traces.\n    * Roughly 2.7% of test traces took more than **6 minutes** to complete.\n* Minor data quality issue: The file **../input/indoor-location-navigation/train/5cd56b83e2acfd2d33b5cab0/B2/5cf72539e9d9c9000852f45b.txt** does not have an 'endTime' line."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport glob\n\npath_train = '../input/indoor-location-navigation/train/*/*/*'\npath_test = '../input/indoor-location-navigation/test/*'\n\ndef get_time_diffs(path, train=True, n_files=-1):    \n    # Get file paths and initialize return list\n    all_files = glob.glob(path)\n    time_diffs = []\n    trace_ids = []\n    \n    # Loop through each file\n    for f in all_files:\n        \n        # Get the trace text\n        with open(f) as f_contents:\n            lines = f_contents.readlines()\n        \n        try:\n            if train:\n                # '#\\tstartTime:1560913369586\\n'\n                start_time = int(lines[0].replace('\\n', '').split(':')[-1])\n                end_time = int(lines[-1].replace('\\n', '').split(':')[-1])\n            else:\n                # '#\\tstartTime\\t0000000000000\\n'\n                start_time = int(lines[0].replace('\\n', '').split('\\t')[-1])\n                end_time = int(lines[-1].replace('\\n', '').split('\\t')[-1])\n        except:\n            print('Unable to parse for file: {}'.format(f))\n        \n        # Append results\n        time_diffs.append((end_time - start_time)/(1000*60))\n        \n    return time_diffs\n        \ntrain_diffs = get_time_diffs(path_train)\ntest_diffs = get_time_diffs(path_test, train=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bins = np.linspace(0, int(max(train_diffs + test_diffs)) + 1, 100)\nplt.hist(train_diffs, bins, alpha=0.5, label='train', density=True)\nplt.hist(test_diffs, bins, alpha=0.5, label='test', density=True)\nplt.legend(loc='upper right')\nplt.xlabel(\"Trace Completion Time (minutes)\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Minimum train trace time: {}'.format(min(train_diffs)))\nprint('Minimum test trace time: {}'.format(min(test_diffs)))\n\nprint('\\nMaximum train trace time: {}'.format(max(train_diffs)))\nprint('Maximum test trace time: {}'.format(max(test_diffs)))\n\ntmp = sum(d > 4 for d in train_diffs)/len(train_diffs)\nprint('\\nRatio train trace time > 5 minutes: {}'.format(tmp))\ntmp = sum(d > 5 for d in test_diffs)/len(test_diffs)\nprint('Ratio test trace time > 5 minutes: {}'.format(tmp))\ntmp = sum(d > 6 for d in test_diffs)/len(test_diffs)\nprint('Ratio test trace time > 6 minutes: {}'.format(tmp))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}