{
  "id": 5537,
  "title": "HF FFT Data only once every second?",
  "url": "/competitions/belkin-energy-disaggregation-competition/discussion/5537",
  "author_name": "",
  "post_date": "2013-08-25T16:02:32.403Z",
  "votes": null,
  "comment_count": 2,
  "views": 1157,
  "content": "<p>In reference [4], it says that the HF FFT data was computed 244 times per second, meaning I would expect each UNIX timestamp in&nbsp;TimeTicksHF to be spaced apart by about 1/244 = 0.004 sec. However, the actual reality is that each timestamp is spaced apart by&nbsp;1.0667 sec.</p>\n<p>Why such low temporal resolution? From [4], it was suggested that we should use sliding window averages of 25 vectors to cancel out the noise - but taking 25 consecutive vectors with this data would mean we only get 2 samples per minute.</p>\n<p>Am I missing something?</p>",
  "messages": [
    {
      "id": "29445",
      "postDate": "08/25/2013 16:02:32",
      "content": "<p>In reference [4], it says that the HF FFT data was computed 244 times per second, meaning I would expect each UNIX timestamp in&nbsp;TimeTicksHF to be spaced apart by about 1/244 = 0.004 sec. However, the actual reality is that each timestamp is spaced apart by&nbsp;1.0667 sec.</p>\n<p>Why such low temporal resolution? From [4], it was suggested that we should use sliding window averages of 25 vectors to cancel out the noise - but taking 25 consecutive vectors with this data would mean we only get 2 samples per minute.</p>\n<p>Am I missing something?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29989",
      "postDate": "09/02/2013 13:02:33",
      "content": "<p>bump</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "30028",
      "postDate": "09/03/2013 01:41:19",
      "content": "<p>Hi Nitai,</p>\n<p>First of all the sampling hardware is not the same as that used in reference [4]. This was changed to be both much better and be commercially viable.</p>\n<p>The Belkin hardware samples at 2Mhz and takes a 8192 point FFT. This still results in ~244.14 vectors per second. &nbsp;Since it is impractical to upload such high bandwidth data on an average homeowner's internet connection, we apply proprietary compression techniques to the data and then upload it. Unfortunately I cannot go into details of what we exactly do. The end result is that you get 1 FFT per second (or more specifically, like you said 1.0667).</p>\n<p>Though I cannot disclose what transpires in going from 244 vectors/sec to 1 vector/sec, I can very well tell you that we ran multitudes of machine learning statistics to convince us that the information that we are after is not all lost. In other words, the &quot;information gain&quot; remains pretty much intact.</p>\n<p>In [4] the averaging window of 25 was to reduce the 244 vector/sec down to ~9.7 vectors/sec. But as you see, we have already given you 1 vector/sec., so there is really no need to apply that 25 vector window averaging. Of course you can if you see fit, but personally I would not average it to anything below ~1 vector per 3-4 seconds.</p>\n<p>Sidhant</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 29989,
      "author_name": "nitaidean",
      "author_url": "",
      "post_date": "09/02/2013 13:02:33",
      "content": "<p>bump</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 30028,
      "author_name": "sidhantgupta",
      "author_url": "",
      "post_date": "09/03/2013 01:41:19",
      "content": "<p>Hi Nitai,</p>\n<p>First of all the sampling hardware is not the same as that used in reference [4]. This was changed to be both much better and be commercially viable.</p>\n<p>The Belkin hardware samples at 2Mhz and takes a 8192 point FFT. This still results in ~244.14 vectors per second. &nbsp;Since it is impractical to upload such high bandwidth data on an average homeowner's internet connection, we apply proprietary compression techniques to the data and then upload it. Unfortunately I cannot go into details of what we exactly do. The end result is that you get 1 FFT per second (or more specifically, like you said 1.0667).</p>\n<p>Though I cannot disclose what transpires in going from 244 vectors/sec to 1 vector/sec, I can very well tell you that we ran multitudes of machine learning statistics to convince us that the information that we are after is not all lost. In other words, the &quot;information gain&quot; remains pretty much intact.</p>\n<p>In [4] the averaging window of 25 was to reduce the 244 vector/sec down to ~9.7 vectors/sec. But as you see, we have already given you 1 vector/sec., so there is really no need to apply that 25 vector window averaging. Of course you can if you see fit, but personally I would not average it to anything below ~1 vector per 3-4 seconds.</p>\n<p>Sidhant</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29445": "",
    "29989": "",
    "30028": ""
  },
  "source": "meta"
}