There is a very clear dishwasher in 'H2/Testing_07_17_1342508401.mat' but the negative contribution to my score when I mark it seems to indicate that it is marked as "no dishwasher" in the public fold.
\nHas anyone else had trouble with this one?
\nDid all the dishwashers get fixed when Jessica reported the one in H1 as missing? Or are there still problems with the backend solution?
\n", "messages": [ { "id": "30107", "postDate": "09/03/2013 23:05:16", "content": "
There is a very clear dishwasher in 'H2/Testing_07_17_1342508401.mat' but the negative contribution to my score when I mark it seems to indicate that it is marked as "no dishwasher" in the public fold.
\nHas anyone else had trouble with this one?
\nDid all the dishwashers get fixed when Jessica reported the one in H1 as missing? Or are there still problems with the backend solution?
\n", "rawMarkdown": "", "votes": null }, { "id": "30108", "postDate": "09/03/2013 23:14:01", "content": "
I haven't tried with the dishwasher yet, but I had that problem with two other appliances in H2. My program is completely sure that they are present, but they don't improve the public leaderboard score.
\n", "rawMarkdown": "", "votes": null }, { "id": "30116", "postDate": "09/04/2013 00:45:23", "content": "
Thanks Luis.
\nI have had the same problem with more than four appliances on H1 and some on H4. Some of those are even longer than this dishwasher and they really hurt the overall score. The only way I know to be sure which one is wrong is to use a submission to test it (like Jessica did) and there are only so many submissions. Besides, the final scoring will be done on the private fold where we have no submissions to go by.
\nI am not always sure which one of two devices I marked is scored wrong. I do know that I marked each of the devices and got it scored as right at least once but I can not see any difference between the good and the bad "guesses".
\nThe annoying part is that by this time I was expecting to have more than just a guess. I feel pretty confident about my calls but as long as the scores are getting worse for what looks like good calls, I guess we are all still just guessing about what the private fold would look like.
\n", "rawMarkdown": "", "votes": null }, { "id": "30131", "postDate": "09/04/2013 05:01:02", "content": "
I completely agree with you Noam.
", "rawMarkdown": "", "votes": null }, { "id": "30139", "postDate": "09/04/2013 09:39:35", "content": "Noam,
\nI've made similar experiences. Not only for the dishwasher in H2, but also for a few other devices in various households.
", "rawMarkdown": "", "votes": null }, { "id": "30143", "postDate": "09/04/2013 11:40:49", "content": "I am trying to think of ways to improve the experience for all of us and develop better algorithms for Belkin without violating competition rules and without giving away too much information to each other (after all we are competing against each other).
\nWe can not tell if a score is worse than we expected because X minutes are scored differently in the public fold or because 2*X minutes are in the private fold and do not contribute to the public score. Still, when we observe the score getting worse for what we think should be a better submission we can bring it to the attention of the administrators. I have added (private) notes to many of my submissions that indicate to the Kaggle and Belkin administrators where I have high confidence in my "guesses" which are inconsistent with the submission score.
\nI doubt that Kaggle/Belkin have the resources to check each of our individual questions, after all the whole point of the competition is for us to do the work for them. Many of our individual guesses might be wrong so chasing each one would be a waste of their time. However, if most of the top 5 contestants on the leaderboard disagree with the back-end score for any specific device at a specific time, I think it would be worthwhile for them to check the backend solution for a possible error. Doing so would help them improve not only the accuracy of the public fold, but the validity of the private fold scores as well.
\nIf the top five contestants all add private notes like I did, the administrators could check for cases where many of us agree that the backend solution is wrong without requiring any direct communication between contestants. They could then make weekly updates to the backend solution as more errors like the H1 diswasher Jessica found early on are discovered.
\n\n
In order for this to work well, two things need to happen:
\nA. Several of the leading contestants need to join me in posting private notes on their submissions
\nB. Kaggle/Belkin need to let us know that adding those notes would not be a waste of our time.
\nI believe that the benefit Belkin derives from this competition would be much higher if we spend our time developing better algorithms instead of trying to second guess the errors in the back end solution so I really hope that they do something about it soon.
", "rawMarkdown": "", "votes": null }, { "id": "30144", "postDate": "09/04/2013 12:02:23", "content": "Noam,
\nI fully agree with your previous posting.
\nIf the admins will let us know that this is no waste of time, I will try to add private notes to my future submissions (in those cases where scores just don't seem to make sense).
", "rawMarkdown": "", "votes": null }, { "id": "30170", "postDate": "09/04/2013 16:22:33", "content": "I also agree with you Noam. I can also add notes to some of my past submissions.
\n\n
Now, let's wait for an answer from Kaggle and/or Belkin.
", "rawMarkdown": "", "votes": null }, { "id": "30324", "postDate": "09/06/2013 16:13:27", "content": "[quote=Noam Tene;30107]There is a very clear dishwasher in 'H2/Testing_07_17_1342508401.mat' but the negative contribution to my score when I mark it seems to indicate that it is marked as "no dishwasher" in the public fold.[/quote]
\nI have found a dishwasher in 'H2/Testing_07_17_1342508401.mat' and it looks to be confirmed by public score.
", "rawMarkdown": "", "votes": null }, { "id": "30454", "postDate": "09/07/2013 21:09:37", "content": "Perhaps the answer to this issue is related to this the discussion in this thread:
\nhttps://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5317/question-about-the-evaluation-function
\nThe test data is split in two halves, and one of them is used for the public leaderboard. This split is performed randomly, but not at the level of single timestamps, but rather in entire periods. So, each occurrence of a given appliance in the validation set (the one used for the public score) is either completely evaluated in the score or completely ignored. If we focus on a single appliance in a given submission, and it is accurately predicted but the period considered is not in the validation set, we wouldn't observe any improvement.
\nCan this be the reason for the disagreements reported in this thread?
", "rawMarkdown": "", "votes": null }, { "id": "30471", "postDate": "09/08/2013 06:54:33", "content": "Jose, if you read my earlier posts you'll see that I already account for the effect of the public/private fold. The private fold can explain why the score does not get better when you make a good call. But it does not explain why the Public score gets worse.
\nWhen you make more than one guess in a submission and get a lower public score: at least some of your guesses must be marked differently in the public fold of the back end solution.
\nIf you have high confidence in all your "guesses" and still get a worse score when you submit them, you know that at least some of them are wrong but not necessarily which ones. It is still possible to misidentify the source of the negative contribution.
\n", "rawMarkdown": "", "votes": null }, { "id": "30738", "postDate": "09/11/2013 16:28:28", "content": "
Now I have found (clear power signatures) extra two dishwashers in H2, at 1342633299 and at 1342724796, but they decrease the score.
", "rawMarkdown": "", "votes": null }, { "id": "30741", "postDate": "09/11/2013 17:02:28", "content": "Victor, is it possible that think only one of them caused the decrease in the score?
", "rawMarkdown": "", "votes": null }, { "id": "30746", "postDate": "09/11/2013 17:48:32", "content": "[quote=Noam Tene;30741]Victor, is it possible that think only one of them caused the decrease in the score?[/quote]
\nYes, it looks like the first one decreases, but the second one does not affect.
", "rawMarkdown": "", "votes": null }, { "id": "30754", "postDate": "09/11/2013 20:03:03", "content": "While we are at it. Does anyone else think there should be an H4 Dishwasher On Sep 12 between 1347485286 and 1347486420?
", "rawMarkdown": "", "votes": null }, { "id": "30860", "postDate": "09/13/2013 15:56:17", "content": "[quote=Noam Tene;30754]While we are at it. Does anyone else think there should be an H4 Dishwasher On Sep 12 between 1347485286 and 1347486420? [/quote]
\nHere it is (the second plot).
", "rawMarkdown": "", "votes": null }, { "id": "30996", "postDate": "09/16/2013 19:05:59", "content": "Both Belkin and Kaggle have chosen to answer on other threads while ignoring this one.
\nUntil, they post something to let us know that we are not "wasting our time" it seems that reporting errors in the back-end solution is worse than useless.
\nIf I know that Belkin is consistently wrong on in specific situations and you don't, that gives me an advantage. If Belkin were to fix their errors in the back end solution today (or a few days after a report for future reports) letting them know about their errors would make the challenge more interesting and the competition scientifically useful.
\nHowever, sending them information about their errors only to have on them fix the back-end solution on the day before the deadline will remove my advantage with very little benefit to me.
\nIt seems that Belkin/Kaggle have chosen to take the second route :(
\n\n
\n
\n
\n
", "rawMarkdown": "", "votes": null }, { "id": "31003", "postDate": "09/16/2013 19:41:55", "content": "
Noam, Belkin is working on the best way to address these. It takes time to sort out the valid changes and they are working on it. Make the best prediction method you can and let them worry about the validity of the solution set. With 30+ appliances from 4 houses over a period of several days, it's unlikely that luck will play the significant role (of course, it usually plays some role) in determining the winner, even if the ground truth is flawed.
\nAlso, keep in mind that we are not going to respond in the affirmative/negative to test set changes. The leaderboard is there to motivate competition and make it fun, but it is not an oracle against which you should play 20 questions (nor will changes to the public fold of the test set affect the final outcome). A response for or against these inquiries would essentially turn test data into training data.
", "rawMarkdown": "", "votes": null }, { "id": "31014", "postDate": "09/16/2013 21:59:05", "content": "Thanks for explaining the reasoning behind your earlier silence on this thread.
\n", "rawMarkdown": "", "votes": null }, { "id": "31066", "postDate": "09/18/2013 00:38:45", "content": "
[quote=Noam Tene;30754]
\nWhile we are at it. Does anyone else think there should be an H4 Dishwasher On Sep 12 between 1347485286 and 1347486420?
\n[/quote]
\nI haven't had the same experiences with all the claimed appliances I've read across the various threads but this one I agree with.
", "rawMarkdown": "", "votes": null }, { "id": "31113", "postDate": "09/18/2013 18:02:21", "content": "Please see the announcement here for the way we are handling these suggestions:
\nhttps://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5745/good-calls-result-in-worse-scores/31109#post31109
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30108, "author_name": "luistp001", "author_url": "", "post_date": "09/03/2013 23:14:01", "content": "I haven't tried with the dishwasher yet, but I had that problem with two other appliances in H2. My program is completely sure that they are present, but they don't improve the public leaderboard score.
\n", "votes": null, "replies": [] }, { "id": 30116, "author_name": "noamtene", "author_url": "", "post_date": "09/04/2013 00:45:23", "content": "
Thanks Luis.
\nI have had the same problem with more than four appliances on H1 and some on H4. Some of those are even longer than this dishwasher and they really hurt the overall score. The only way I know to be sure which one is wrong is to use a submission to test it (like Jessica did) and there are only so many submissions. Besides, the final scoring will be done on the private fold where we have no submissions to go by.
\nI am not always sure which one of two devices I marked is scored wrong. I do know that I marked each of the devices and got it scored as right at least once but I can not see any difference between the good and the bad "guesses".
\nThe annoying part is that by this time I was expecting to have more than just a guess. I feel pretty confident about my calls but as long as the scores are getting worse for what looks like good calls, I guess we are all still just guessing about what the private fold would look like.
\n", "votes": null, "replies": [] }, { "id": 30131, "author_name": "luistp001", "author_url": "", "post_date": "09/04/2013 05:01:02", "content": "
I completely agree with you Noam.
", "votes": null, "replies": [] }, { "id": 30139, "author_name": "michael23", "author_url": "", "post_date": "09/04/2013 09:39:35", "content": "Noam,
\nI've made similar experiences. Not only for the dishwasher in H2, but also for a few other devices in various households.
", "votes": null, "replies": [] }, { "id": 30143, "author_name": "noamtene", "author_url": "", "post_date": "09/04/2013 11:40:49", "content": "I am trying to think of ways to improve the experience for all of us and develop better algorithms for Belkin without violating competition rules and without giving away too much information to each other (after all we are competing against each other).
\nWe can not tell if a score is worse than we expected because X minutes are scored differently in the public fold or because 2*X minutes are in the private fold and do not contribute to the public score. Still, when we observe the score getting worse for what we think should be a better submission we can bring it to the attention of the administrators. I have added (private) notes to many of my submissions that indicate to the Kaggle and Belkin administrators where I have high confidence in my "guesses" which are inconsistent with the submission score.
\nI doubt that Kaggle/Belkin have the resources to check each of our individual questions, after all the whole point of the competition is for us to do the work for them. Many of our individual guesses might be wrong so chasing each one would be a waste of their time. However, if most of the top 5 contestants on the leaderboard disagree with the back-end score for any specific device at a specific time, I think it would be worthwhile for them to check the backend solution for a possible error. Doing so would help them improve not only the accuracy of the public fold, but the validity of the private fold scores as well.
\nIf the top five contestants all add private notes like I did, the administrators could check for cases where many of us agree that the backend solution is wrong without requiring any direct communication between contestants. They could then make weekly updates to the backend solution as more errors like the H1 diswasher Jessica found early on are discovered.
\n\n
In order for this to work well, two things need to happen:
\nA. Several of the leading contestants need to join me in posting private notes on their submissions
\nB. Kaggle/Belkin need to let us know that adding those notes would not be a waste of our time.
\nI believe that the benefit Belkin derives from this competition would be much higher if we spend our time developing better algorithms instead of trying to second guess the errors in the back end solution so I really hope that they do something about it soon.
", "votes": null, "replies": [] }, { "id": 30144, "author_name": "michael23", "author_url": "", "post_date": "09/04/2013 12:02:23", "content": "Noam,
\nI fully agree with your previous posting.
\nIf the admins will let us know that this is no waste of time, I will try to add private notes to my future submissions (in those cases where scores just don't seem to make sense).
", "votes": null, "replies": [] }, { "id": 30170, "author_name": "luistp001", "author_url": "", "post_date": "09/04/2013 16:22:33", "content": "I also agree with you Noam. I can also add notes to some of my past submissions.
\n\n
Now, let's wait for an answer from Kaggle and/or Belkin.
", "votes": null, "replies": [] }, { "id": 30324, "author_name": "nedelko", "author_url": "", "post_date": "09/06/2013 16:13:27", "content": "[quote=Noam Tene;30107]There is a very clear dishwasher in 'H2/Testing_07_17_1342508401.mat' but the negative contribution to my score when I mark it seems to indicate that it is marked as "no dishwasher" in the public fold.[/quote]
\nI have found a dishwasher in 'H2/Testing_07_17_1342508401.mat' and it looks to be confirmed by public score.
", "votes": null, "replies": [] }, { "id": 30454, "author_name": "joseleiva", "author_url": "", "post_date": "09/07/2013 21:09:37", "content": "Perhaps the answer to this issue is related to this the discussion in this thread:
\nhttps://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5317/question-about-the-evaluation-function
\nThe test data is split in two halves, and one of them is used for the public leaderboard. This split is performed randomly, but not at the level of single timestamps, but rather in entire periods. So, each occurrence of a given appliance in the validation set (the one used for the public score) is either completely evaluated in the score or completely ignored. If we focus on a single appliance in a given submission, and it is accurately predicted but the period considered is not in the validation set, we wouldn't observe any improvement.
\nCan this be the reason for the disagreements reported in this thread?
", "votes": null, "replies": [] }, { "id": 30471, "author_name": "noamtene", "author_url": "", "post_date": "09/08/2013 06:54:33", "content": "Jose, if you read my earlier posts you'll see that I already account for the effect of the public/private fold. The private fold can explain why the score does not get better when you make a good call. But it does not explain why the Public score gets worse.
\nWhen you make more than one guess in a submission and get a lower public score: at least some of your guesses must be marked differently in the public fold of the back end solution.
\nIf you have high confidence in all your "guesses" and still get a worse score when you submit them, you know that at least some of them are wrong but not necessarily which ones. It is still possible to misidentify the source of the negative contribution.
\n", "votes": null, "replies": [] }, { "id": 30738, "author_name": "nedelko", "author_url": "", "post_date": "09/11/2013 16:28:28", "content": "
Now I have found (clear power signatures) extra two dishwashers in H2, at 1342633299 and at 1342724796, but they decrease the score.
", "votes": null, "replies": [] }, { "id": 30741, "author_name": "noamtene", "author_url": "", "post_date": "09/11/2013 17:02:28", "content": "Victor, is it possible that think only one of them caused the decrease in the score?
", "votes": null, "replies": [] }, { "id": 30746, "author_name": "nedelko", "author_url": "", "post_date": "09/11/2013 17:48:32", "content": "[quote=Noam Tene;30741]Victor, is it possible that think only one of them caused the decrease in the score?[/quote]
\nYes, it looks like the first one decreases, but the second one does not affect.
", "votes": null, "replies": [] }, { "id": 30754, "author_name": "noamtene", "author_url": "", "post_date": "09/11/2013 20:03:03", "content": "While we are at it. Does anyone else think there should be an H4 Dishwasher On Sep 12 between 1347485286 and 1347486420?
", "votes": null, "replies": [] }, { "id": 30860, "author_name": "nedelko", "author_url": "", "post_date": "09/13/2013 15:56:17", "content": "[quote=Noam Tene;30754]While we are at it. Does anyone else think there should be an H4 Dishwasher On Sep 12 between 1347485286 and 1347486420? [/quote]
\nHere it is (the second plot).
", "votes": null, "replies": [] }, { "id": 30996, "author_name": "noamtene", "author_url": "", "post_date": "09/16/2013 19:05:59", "content": "Both Belkin and Kaggle have chosen to answer on other threads while ignoring this one.
\nUntil, they post something to let us know that we are not "wasting our time" it seems that reporting errors in the back-end solution is worse than useless.
\nIf I know that Belkin is consistently wrong on in specific situations and you don't, that gives me an advantage. If Belkin were to fix their errors in the back end solution today (or a few days after a report for future reports) letting them know about their errors would make the challenge more interesting and the competition scientifically useful.
\nHowever, sending them information about their errors only to have on them fix the back-end solution on the day before the deadline will remove my advantage with very little benefit to me.
\nIt seems that Belkin/Kaggle have chosen to take the second route :(
\n\n
\n
\n
\n
", "votes": null, "replies": [] }, { "id": 31003, "author_name": "wcukierski", "author_url": "", "post_date": "09/16/2013 19:41:55", "content": "
Noam, Belkin is working on the best way to address these. It takes time to sort out the valid changes and they are working on it. Make the best prediction method you can and let them worry about the validity of the solution set. With 30+ appliances from 4 houses over a period of several days, it's unlikely that luck will play the significant role (of course, it usually plays some role) in determining the winner, even if the ground truth is flawed.
\nAlso, keep in mind that we are not going to respond in the affirmative/negative to test set changes. The leaderboard is there to motivate competition and make it fun, but it is not an oracle against which you should play 20 questions (nor will changes to the public fold of the test set affect the final outcome). A response for or against these inquiries would essentially turn test data into training data.
", "votes": null, "replies": [] }, { "id": 31014, "author_name": "noamtene", "author_url": "", "post_date": "09/16/2013 21:59:05", "content": "Thanks for explaining the reasoning behind your earlier silence on this thread.
\n", "votes": null, "replies": [] }, { "id": 31066, "author_name": "rosnfeld", "author_url": "", "post_date": "09/18/2013 00:38:45", "content": "
[quote=Noam Tene;30754]
\nWhile we are at it. Does anyone else think there should be an H4 Dishwasher On Sep 12 between 1347485286 and 1347486420?
\n[/quote]
\nI haven't had the same experiences with all the claimed appliances I've read across the various threads but this one I agree with.
", "votes": null, "replies": [] }, { "id": 31113, "author_name": "wcukierski", "author_url": "", "post_date": "09/18/2013 18:02:21", "content": "Please see the announcement here for the way we are handling these suggestions:
\nhttps://www.kaggle.com/c/belkin-energy-disaggregation-competition/forums/t/5745/good-calls-result-in-worse-scores/31109#post31109
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30107": "", "30108": "", "30116": "", "30131": "", "30139": "", "30143": "", "30144": "", "30170": "", "30324": "", "30454": "", "30471": "", "30738": "", "30741": "", "30746": "", "30754": "", "30860": "", "30996": "", "31003": "", "31014": "", "31066": "", "31113": "" }, "source": "meta" }