{
  "id": 5935,
  "title": "False positives and mis-tagged appliances",
  "url": "/competitions/belkin-energy-disaggregation-competition/discussion/5935",
  "author_name": "",
  "post_date": "2013-10-01T11:52:07.490Z",
  "votes": null,
  "comment_count": 1,
  "views": 1104,
  "content": "<p>I am wondering how Kaggle/Belkin intend to deal with the related issues of false positives and appliances that are marked differently in the tagged data and in the test set.</p>\n<p>By &quot;false positives&quot; I am referring to appliances with clear On/Off signatures that are marked as &quot;on&quot; for periods in the back-end solution even when there is no signal (of any kind) in the data for the corresponding on and off times.</p>\n<p>By &quot;mis-tagged appliances&quot; I am referring to appliances with clear signatures that appear in the data as well as in the TaggingInfo fields of the training set but that seem to be marked in the back-end solution as something different from what the tag in the tagged data would seem to indicate.&nbsp;</p>\n<p>There seems to be some self consistency in the public fold of the test data that in some cases is different from what we see in the tagged data. &nbsp;When there are more consistent samples in the test data than there are in the tagged data, it makes sense to assume that one human error occurred in the tagging process than to believe that multiple human errors all occurred in exactly the same way in the processing of the test data.</p>\n<p>A few of the TaggingInfo fields are clearly off by more than 60 seconds and there are many tags that do not follow the guidelines posted earlier by Sidhant. Those guidelines stated that tags should always be on the exact minute boundary (multiples of 60 seconds) and should refer to whether an On/Off event happened anywhere in the 60 seconds following the tag.</p>\n<p>&nbsp;</p>\n<p>Our submissions (and presumably the back-end solution they are graded against) follow these guidelines by definition but the TaggingInfo fields have not been fixed to comply with these rules since the competition started.</p>\n<p>Would it be correct to assume that the public and private folds of the test set went through the same process and that the division into public and private folds only occurred after the processing was done?</p>\n<p><span style=\"line-height: 1.4\">If that is the case, consistency between the public and private folds of the data sets should be more reliable than the correspondence between the tagged and test sets.</span></p>\n<p><span style=\"line-height: 1.4\">If/when Kaggle/Belkin is going to fix things, I hope they address the issues with the tagged training sets and avoid introducing an artificial difference between the public and private folds.</span></p>\n<p><span style=\"line-height: 1.4\">In either case, Admins, &nbsp;please let us know how you are handling these issues.</span></p>\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "31856",
      "postDate": "10/01/2013 11:52:07",
      "content": "<p>I am wondering how Kaggle/Belkin intend to deal with the related issues of false positives and appliances that are marked differently in the tagged data and in the test set.</p>\n<p>By &quot;false positives&quot; I am referring to appliances with clear On/Off signatures that are marked as &quot;on&quot; for periods in the back-end solution even when there is no signal (of any kind) in the data for the corresponding on and off times.</p>\n<p>By &quot;mis-tagged appliances&quot; I am referring to appliances with clear signatures that appear in the data as well as in the TaggingInfo fields of the training set but that seem to be marked in the back-end solution as something different from what the tag in the tagged data would seem to indicate.&nbsp;</p>\n<p>There seems to be some self consistency in the public fold of the test data that in some cases is different from what we see in the tagged data. &nbsp;When there are more consistent samples in the test data than there are in the tagged data, it makes sense to assume that one human error occurred in the tagging process than to believe that multiple human errors all occurred in exactly the same way in the processing of the test data.</p>\n<p>A few of the TaggingInfo fields are clearly off by more than 60 seconds and there are many tags that do not follow the guidelines posted earlier by Sidhant. Those guidelines stated that tags should always be on the exact minute boundary (multiples of 60 seconds) and should refer to whether an On/Off event happened anywhere in the 60 seconds following the tag.</p>\n<p>&nbsp;</p>\n<p>Our submissions (and presumably the back-end solution they are graded against) follow these guidelines by definition but the TaggingInfo fields have not been fixed to comply with these rules since the competition started.</p>\n<p>Would it be correct to assume that the public and private folds of the test set went through the same process and that the division into public and private folds only occurred after the processing was done?</p>\n<p><span style=\"line-height: 1.4\">If that is the case, consistency between the public and private folds of the data sets should be more reliable than the correspondence between the tagged and test sets.</span></p>\n<p><span style=\"line-height: 1.4\">If/when Kaggle/Belkin is going to fix things, I hope they address the issues with the tagged training sets and avoid introducing an artificial difference between the public and private folds.</span></p>\n<p><span style=\"line-height: 1.4\">In either case, Admins, &nbsp;please let us know how you are handling these issues.</span></p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31942",
      "postDate": "10/02/2013 16:02:06",
      "content": "<p>Noam,</p>\n<p>&nbsp;</p>\n<p>Point noted and understood. Based on the public fold data models that have been submitted by you guys, we will also be looking into some of the related private data set instances. We will try and post something about how the data and changes were handled if possible. Our foremost priority will be to look into the models uploaded and update the solution if errors are found so that participants get time to absorb the outcome.&nbsp;</p>\n<p>&nbsp;</p>\n<p>Regards,</p>\n<p>Jinesh</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 31942,
      "author_name": "jinesh0",
      "author_url": "",
      "post_date": "10/02/2013 16:02:06",
      "content": "<p>Noam,</p>\n<p>&nbsp;</p>\n<p>Point noted and understood. Based on the public fold data models that have been submitted by you guys, we will also be looking into some of the related private data set instances. We will try and post something about how the data and changes were handled if possible. Our foremost priority will be to look into the models uploaded and update the solution if errors are found so that participants get time to absorb the outcome.&nbsp;</p>\n<p>&nbsp;</p>\n<p>Regards,</p>\n<p>Jinesh</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "31856": "",
    "31942": ""
  },
  "source": "meta"
}