In house 2, there is clearly a missing training example for "Laundry Room Lights", which is ID = 19. We were assured there would be 3 training examples for each device, but this one only has 2 in the AllTaggingInfo. However, it is clear from looking at the signal that there are 3, but the last one is unlabeled (also, the labels for this device are up to 50 seconds off from the actual signal timestamps).
\nI've attached an image below demonstrating the issue.
\n\n
Note: in the image, the positive green/red vertical lines represent where the AllTaggingInfo says the signal is, and the negative corresponding lines show where the electric signal probably is.
", "messages": [ { "id": "30048", "postDate": "09/03/2013 11:54:21", "content": "In house 2, there is clearly a missing training example for "Laundry Room Lights", which is ID = 19. We were assured there would be 3 training examples for each device, but this one only has 2 in the AllTaggingInfo. However, it is clear from looking at the signal that there are 3, but the last one is unlabeled (also, the labels for this device are up to 50 seconds off from the actual signal timestamps).
\nI've attached an image below demonstrating the issue.
\n\n
Note: in the image, the positive green/red vertical lines represent where the AllTaggingInfo says the signal is, and the negative corresponding lines show where the electric signal probably is.
", "rawMarkdown": "", "votes": null }, { "id": "30051", "postDate": "09/03/2013 12:34:14", "content": "Upon looking further into the training examples of House 2, I found that 12 out of the 37 devices in the house have only 2 training examples (instead of the minimum assured 3), and in most of them there appears to indeed be a 3rd signal that is simply not labelled in the AllTaggingInfo.mat file.
\nHas anyone else noticed this?
", "rawMarkdown": "", "votes": null }, { "id": "30114", "postDate": "09/04/2013 00:22:39", "content": "I have noticed it as well.
\nIn some cases there are four signals where only two are marked.
\nBased on the discussion about tags being rounded to the nearest minute, it did not bother me to find two pairs of events under a single tag. In theory, if a device is turned on and off twice within a minute, that should only be tagged once. I did not bother to check if they were indeed all within a minute because as you said - some of the tags were more than 50 seconds off anyway.
", "rawMarkdown": "", "votes": null }, { "id": "30127", "postDate": "09/04/2013 03:41:09", "content": "Is it only in H2? Is it against the rules to go into the tags and add in the missing tags myself for this house?
", "rawMarkdown": "", "votes": null }, { "id": "30128", "postDate": "09/04/2013 04:18:58", "content": "This is only my guess, but the rules say: "Entries may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records. "
\nTherefore, we cannot hand label anything in the validation set, but it doesn't say anything about hand labeling the training set. I think we can add the tags ourselves as long as it is only in the training set.
\n\n
That's my guess. You may still want to hear an official answer.
", "rawMarkdown": "", "votes": null }, { "id": "30145", "postDate": "09/04/2013 12:06:15", "content": "Yeah... plus, this isn't really even adding the "tag". I don't intend to go and manually enter the exact timestamp of the electric ON/OFF events that are missing - just indicate in the AllTaggingInfo that there is another training example in the vicinity where it appears.
\nI know this is a RL data collection corpus, and while we can't expect perfection from the users doing the labeling (for instance, having them sync up their tagging time with the actual electrical event is asking too much), I think it's fair to expect them to at least have a label present for each time they performed an ON/OFF pair. We shouldn't have to "guess" at whether there are more unlabeled training examples hiding in the training data.
\nPlus, it seems to me that the other 3 houses are much better about this. House 2 seems somewhat of an outlier when it comes to labeling practices.
", "rawMarkdown": "", "votes": null }, { "id": "30303", "postDate": "09/06/2013 12:38:00", "content": "It seems like there a number of devices in all houses that have less than 3 examples. I'm looking at house 1 now, for example, and both washer and dryer have only 1 example in the training set. Are there just unlabeled examples in the set, or did the training protocol not follow procedure in this case? It's difficult to learn a signal with only 1 example...
", "rawMarkdown": "", "votes": null }, { "id": "30310", "postDate": "09/06/2013 13:25:35", "content": "I agree.
\n", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30051, "author_name": "nitaidean", "author_url": "", "post_date": "09/03/2013 12:34:14", "content": "
Upon looking further into the training examples of House 2, I found that 12 out of the 37 devices in the house have only 2 training examples (instead of the minimum assured 3), and in most of them there appears to indeed be a 3rd signal that is simply not labelled in the AllTaggingInfo.mat file.
\nHas anyone else noticed this?
", "votes": null, "replies": [] }, { "id": 30114, "author_name": "noamtene", "author_url": "", "post_date": "09/04/2013 00:22:39", "content": "I have noticed it as well.
\nIn some cases there are four signals where only two are marked.
\nBased on the discussion about tags being rounded to the nearest minute, it did not bother me to find two pairs of events under a single tag. In theory, if a device is turned on and off twice within a minute, that should only be tagged once. I did not bother to check if they were indeed all within a minute because as you said - some of the tags were more than 50 seconds off anyway.
", "votes": null, "replies": [] }, { "id": 30127, "author_name": "nitaidean", "author_url": "", "post_date": "09/04/2013 03:41:09", "content": "Is it only in H2? Is it against the rules to go into the tags and add in the missing tags myself for this house?
", "votes": null, "replies": [] }, { "id": 30128, "author_name": "luistp001", "author_url": "", "post_date": "09/04/2013 04:18:58", "content": "This is only my guess, but the rules say: "Entries may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records. "
\nTherefore, we cannot hand label anything in the validation set, but it doesn't say anything about hand labeling the training set. I think we can add the tags ourselves as long as it is only in the training set.
\n\n
That's my guess. You may still want to hear an official answer.
", "votes": null, "replies": [] }, { "id": 30145, "author_name": "nitaidean", "author_url": "", "post_date": "09/04/2013 12:06:15", "content": "Yeah... plus, this isn't really even adding the "tag". I don't intend to go and manually enter the exact timestamp of the electric ON/OFF events that are missing - just indicate in the AllTaggingInfo that there is another training example in the vicinity where it appears.
\nI know this is a RL data collection corpus, and while we can't expect perfection from the users doing the labeling (for instance, having them sync up their tagging time with the actual electrical event is asking too much), I think it's fair to expect them to at least have a label present for each time they performed an ON/OFF pair. We shouldn't have to "guess" at whether there are more unlabeled training examples hiding in the training data.
\nPlus, it seems to me that the other 3 houses are much better about this. House 2 seems somewhat of an outlier when it comes to labeling practices.
", "votes": null, "replies": [] }, { "id": 30303, "author_name": "nitaidean", "author_url": "", "post_date": "09/06/2013 12:38:00", "content": "It seems like there a number of devices in all houses that have less than 3 examples. I'm looking at house 1 now, for example, and both washer and dryer have only 1 example in the training set. Are there just unlabeled examples in the set, or did the training protocol not follow procedure in this case? It's difficult to learn a signal with only 1 example...
", "votes": null, "replies": [] }, { "id": 30310, "author_name": "noamtene", "author_url": "", "post_date": "09/06/2013 13:25:35", "content": "I agree.
\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30048": "", "30051": "", "30114": "", "30127": "", "30128": "", "30145": "", "30303": "", "30310": "" }, "source": "meta" }