{ "id": 5227, "title": "test data mislabeled", "url": "/competitions/belkin-energy-disaggregation-competition/discussion/5227", "author_name": "", "post_date": "2013-07-29T17:44:45.963Z", "votes": 1, "comment_count": 15, "views": 4411, "content": "

hey,

\n

in the file "Testing_07_09_1341817201.mat" (H1) there is a clear dishwasher (appliance 11). However, if we predict the dishwasher being ON during

\n

the observed time interval, the score gets worse. 

\n

This was our only prediction which means that the test data is mislabeled.

\n

Can someone confirm this issue?

\n

 

\n

X

", "messages": [ { "id": "27775", "postDate": "07/29/2013 17:44:45", "content": "

hey,

\n

in the file "Testing_07_09_1341817201.mat" (H1) there is a clear dishwasher (appliance 11). However, if we predict the dishwasher being ON during

\n

the observed time interval, the score gets worse. 

\n

This was our only prediction which means that the test data is mislabeled.

\n

Can someone confirm this issue?

\n

 

\n

X

", "rawMarkdown": "", "votes": null }, { "id": "27801", "postDate": "07/30/2013 01:22:57", "content": "

Hi Jessica,

\n

Thanks for bringing this to our attention. I will look into it and get back to you. We will fix the backend solution files as necessary.

\n

Sidhant

", "rawMarkdown": "", "votes": null }, { "id": "28030", "postDate": "08/01/2013 23:47:45", "content": "

Can you please let us know when the fix happens so we know which scores are fixed and which ones are not?

", "rawMarkdown": "", "votes": null }, { "id": "28059", "postDate": "08/02/2013 14:47:28", "content": "

Belkin has provided a corrected test set and we've made the swap. All of your submissions are now re-scored (or will be within a few minutes) with the corrected test set.

", "rawMarkdown": "", "votes": null }, { "id": "28064", "postDate": "08/02/2013 15:15:52", "content": "

Are the csv files fixed also--or was the error affecting the .mat files only?  Thanks.

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "28065", "postDate": "08/02/2013 15:19:04", "content": "

This was a change to the test set solution (the answers). As far as I know, there were no corrections necessary to the datasets themselves.

", "rawMarkdown": "", "votes": null }, { "id": "28185", "postDate": "08/04/2013 21:24:41", "content": "

It seems that Jessica's dishwasher was not the only device missing from the backend solution files.

\n

Can you please look at my submissions and let me know if there are other mislabeled test data?

", "rawMarkdown": "", "votes": null }, { "id": "28207", "postDate": "08/05/2013 14:55:40", "content": "

[quote=Noam Tene;28185]

\n

Can you please look at my submissions and let me know if there are other mislabeled test data?

\n

[/quote]

\n

You'll have to give more specific information to get this to happen. Rechecking the entire dataset is a lot of work. If you do not wish to divulge the error(s) publicly, use the message feature to send the house, appliance, and time to one of the admins.

", "rawMarkdown": "", "votes": null }, { "id": "28217", "postDate": "08/05/2013 18:20:08", "content": "

Have the submissions been re-scored? Seems to me the scores haven't changed (at least I expected some change for my submissions), but of course it may be the case that the relabeling doesn't affect any of the submissions yet. Thanks for some hints on that issue.

", "rawMarkdown": "", "votes": null }, { "id": "28218", "postDate": "08/05/2013 18:21:31", "content": "

Yes, all submissions were re-scored.

", "rawMarkdown": "", "votes": null }, { "id": "28219", "postDate": "08/05/2013 19:10:49", "content": "

I have edited the description for my first two scored submissions "to give more specific information" about why I think that "Jessica's dishwasher was not the only device missing from the backend solution files".  My descriptions now describe the "houses, appliances, and times" to be checked.

\n

I am not sure how to use the message feature to send private messages to one of the admins.

\n

When I tried to email William, I got a message saying that: "You need to obtain more points in competitions before you'll be able to contact other Kaggle users."

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "28220", "postDate": "08/05/2013 19:20:04", "content": "

Michael,

\n

It seems that even if they fixed the dishwasher Jessica pointed out to them, there are other tagged device activations that are still missing from the backend solution.  If they only fixed that dishwasher and your submissions did not include it, there is no reason to expect your scores to change.

", "rawMarkdown": "", "votes": null }, { "id": "28232", "postDate": "08/05/2013 23:39:04", "content": "

Noam,

\n

 

\n

We have double-checked the data that you mentioned and labels associated with them are correct in the backend solution files. 

\n

 

\n

I would also recommend you to check this very well written explanation by Sidhant in one of the earlier posts regarding events and timestamps : https://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5083/zero-length-events-in-h1-tagged-training-10-24-1351062001-mat/27155#post27155

\n

 

\n

Thanks,

\n

Jinesh

", "rawMarkdown": "", "votes": null }, { "id": "28235", "postDate": "08/06/2013 00:34:21", "content": "

Jinesh, I already read Sidhant's well written explanation before I posted here.  It is interesting but irrelevant to my question.  If you look at the data, it would be very hard to miss the dishwasher and yet when I predict the dishwasher being on I get a worse score.  I am having exactly the same problem that Jessica had.  It's just on a different day and it is still happening even after the last fix to the backend solution.

\n

Sidhant said that: "we will never get high resolution start/stop of an appliance from a human performing labeling" which is very reasonable and easy to work around.  If at least one dishwasher operation was mislabeled, how likely is it that the human(s) performing the labeling neglected, forgot or mishandled the dishwasher label on more than just one occasion? 

", "rawMarkdown": "", "votes": null }, { "id": "28323", "postDate": "08/06/2013 22:06:10", "content": "

William, 

\n

Is there a "message feature to send the house, appliance, and time to one of the admins" that is available to me?

", "rawMarkdown": "", "votes": null }, { "id": "31176", "postDate": "09/19/2013 12:28:21", "content": "

[quote=Jinesh ;28232]

\n

We have double-checked the data that you mentioned and labels associated with them are correct in the backend solution files. 

\n

 [/quote]

\n

LOL, that was posted 44 days ago.

\n

The score for the submission containing this data changed yesterday from worse than "All Off" to better than "All Off".

\n

Now they finally are correct in the backend solution files.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 27801, "author_name": "sidhantgupta", "author_url": "", "post_date": "07/30/2013 01:22:57", "content": "

Hi Jessica,

\n

Thanks for bringing this to our attention. I will look into it and get back to you. We will fix the backend solution files as necessary.

\n

Sidhant

", "votes": null, "replies": [] }, { "id": 28030, "author_name": "noamtene", "author_url": "", "post_date": "08/01/2013 23:47:45", "content": "

Can you please let us know when the fix happens so we know which scores are fixed and which ones are not?

", "votes": null, "replies": [] }, { "id": 28059, "author_name": "wcukierski", "author_url": "", "post_date": "08/02/2013 14:47:28", "content": "

Belkin has provided a corrected test set and we've made the swap. All of your submissions are now re-scored (or will be within a few minutes) with the corrected test set.

", "votes": null, "replies": [] }, { "id": 28064, "author_name": "uilenspiegel", "author_url": "", "post_date": "08/02/2013 15:15:52", "content": "

Are the csv files fixed also--or was the error affecting the .mat files only?  Thanks.

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 28065, "author_name": "wcukierski", "author_url": "", "post_date": "08/02/2013 15:19:04", "content": "

This was a change to the test set solution (the answers). As far as I know, there were no corrections necessary to the datasets themselves.

", "votes": null, "replies": [] }, { "id": 28185, "author_name": "noamtene", "author_url": "", "post_date": "08/04/2013 21:24:41", "content": "

It seems that Jessica's dishwasher was not the only device missing from the backend solution files.

\n

Can you please look at my submissions and let me know if there are other mislabeled test data?

", "votes": null, "replies": [] }, { "id": 28207, "author_name": "wcukierski", "author_url": "", "post_date": "08/05/2013 14:55:40", "content": "

[quote=Noam Tene;28185]

\n

Can you please look at my submissions and let me know if there are other mislabeled test data?

\n

[/quote]

\n

You'll have to give more specific information to get this to happen. Rechecking the entire dataset is a lot of work. If you do not wish to divulge the error(s) publicly, use the message feature to send the house, appliance, and time to one of the admins.

", "votes": null, "replies": [] }, { "id": 28217, "author_name": "michael23", "author_url": "", "post_date": "08/05/2013 18:20:08", "content": "

Have the submissions been re-scored? Seems to me the scores haven't changed (at least I expected some change for my submissions), but of course it may be the case that the relabeling doesn't affect any of the submissions yet. Thanks for some hints on that issue.

", "votes": null, "replies": [] }, { "id": 28218, "author_name": "wcukierski", "author_url": "", "post_date": "08/05/2013 18:21:31", "content": "

Yes, all submissions were re-scored.

", "votes": null, "replies": [] }, { "id": 28219, "author_name": "noamtene", "author_url": "", "post_date": "08/05/2013 19:10:49", "content": "

I have edited the description for my first two scored submissions "to give more specific information" about why I think that "Jessica's dishwasher was not the only device missing from the backend solution files".  My descriptions now describe the "houses, appliances, and times" to be checked.

\n

I am not sure how to use the message feature to send private messages to one of the admins.

\n

When I tried to email William, I got a message saying that: "You need to obtain more points in competitions before you'll be able to contact other Kaggle users."

\n

 

", "votes": null, "replies": [] }, { "id": 28220, "author_name": "noamtene", "author_url": "", "post_date": "08/05/2013 19:20:04", "content": "

Michael,

\n

It seems that even if they fixed the dishwasher Jessica pointed out to them, there are other tagged device activations that are still missing from the backend solution.  If they only fixed that dishwasher and your submissions did not include it, there is no reason to expect your scores to change.

", "votes": null, "replies": [] }, { "id": 28232, "author_name": "jinesh0", "author_url": "", "post_date": "08/05/2013 23:39:04", "content": "

Noam,

\n

 

\n

We have double-checked the data that you mentioned and labels associated with them are correct in the backend solution files. 

\n

 

\n

I would also recommend you to check this very well written explanation by Sidhant in one of the earlier posts regarding events and timestamps : https://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5083/zero-length-events-in-h1-tagged-training-10-24-1351062001-mat/27155#post27155

\n

 

\n

Thanks,

\n

Jinesh

", "votes": null, "replies": [] }, { "id": 28235, "author_name": "noamtene", "author_url": "", "post_date": "08/06/2013 00:34:21", "content": "

Jinesh, I already read Sidhant's well written explanation before I posted here.  It is interesting but irrelevant to my question.  If you look at the data, it would be very hard to miss the dishwasher and yet when I predict the dishwasher being on I get a worse score.  I am having exactly the same problem that Jessica had.  It's just on a different day and it is still happening even after the last fix to the backend solution.

\n

Sidhant said that: "we will never get high resolution start/stop of an appliance from a human performing labeling" which is very reasonable and easy to work around.  If at least one dishwasher operation was mislabeled, how likely is it that the human(s) performing the labeling neglected, forgot or mishandled the dishwasher label on more than just one occasion? 

", "votes": null, "replies": [] }, { "id": 28323, "author_name": "noamtene", "author_url": "", "post_date": "08/06/2013 22:06:10", "content": "

William, 

\n

Is there a "message feature to send the house, appliance, and time to one of the admins" that is available to me?

", "votes": null, "replies": [] }, { "id": 31176, "author_name": "noamtene", "author_url": "", "post_date": "09/19/2013 12:28:21", "content": "

[quote=Jinesh ;28232]

\n

We have double-checked the data that you mentioned and labels associated with them are correct in the backend solution files. 

\n

 [/quote]

\n

LOL, that was posted 44 days ago.

\n

The score for the submission containing this data changed yesterday from worse than "All Off" to better than "All Off".

\n

Now they finally are correct in the backend solution files.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "27775": "", "27801": "", "28030": "", "28059": "", "28064": "", "28065": "", "28185": "", "28207": "", "28217": "", "28218": "", "28219": "", "28220": "", "28232": "", "28235": "", "28323": "", "31176": "" }, "source": "meta" }