{ "id": 38769, "title": "should we spam the discussion forum to fix the LB ?!!! ", "url": "/competitions/carvana-image-masking-challenge/discussion/38769", "author_name": "ahmed anis", "post_date": "2017-08-30T16:55:25.841000", "votes": -1, "comment_count": 16, "views": 0, "content": "

title.

", "messages": [ { "id": 217517, "postDate": "2017-08-30T20:59:43.243Z", "content": "

I just officially glided out of the top10%, let's hope the fix comes soon so I can get back full speed on this :) by the way, is it planned to increase the visible digits (99.7XX)?

", "rawMarkdown": "I just officially glided out of the top10%, let's hope the fix comes soon so I can get back full speed on this :) by the way, is it planned to increase the visible digits (99.7XX)? ", "votes": 3 }, { "id": 217464, "postDate": "2017-08-30T17:00:56.543Z", "content": "

Please do not do this. We will get the fix up as soon as possible.

", "rawMarkdown": "Please do not do this. We will get the fix up as soon as possible.", "votes": 2, "replies": [ { "id": 217467, "postDate": "2017-08-30T17:03:21.350Z", "content": "

Thx for your reply,\ncan we have any expected time when the fix will be applied? \nThanks

", "rawMarkdown": "Thx for your reply,\ncan we have any expected time when the fix will be applied? \nThanks" }, { "id": 217469, "postDate": "2017-08-30T17:06:57.287Z", "content": "

I can't give a firm timeline, but it should be a matter of days, not weeks. The code is written and all that remains is testing.

", "rawMarkdown": "I can't give a firm timeline, but it should be a matter of days, not weeks. The code is written and all that remains is testing." }, { "id": 217499, "postDate": "2017-08-30T18:53:09.717Z", "content": "

Thanks for the update. However, the timeframe (\"days not weeks\") still seems too long and vague. Right now most of us are\"flying blind\" when it comes to estimating how well our models are doing. In light of all of this, would it be possible to extend the competition deadline?

", "rawMarkdown": "Thanks for the update. However, the timeframe (\"days not weeks\") still seems too long and vague. Right now most of us are\"flying blind\" when it comes to estimating how well our models are doing. In light of all of this, would it be possible to extend the competition deadline?", "votes": 3 }, { "id": 217502, "postDate": "2017-08-30T19:01:49.697Z", "content": "

I am waiting for the LB bugfix as well. \nBut how about using local validation till then? :P Proper local validation framework will be useful at the end as well. I found the submission creation (and upload) too long anyway so I prefer to test ideas with local validation.

", "rawMarkdown": "I am waiting for the LB bugfix as well. \nBut how about using local validation till then? :P Proper local validation framework will be useful at the end as well. I found the submission creation (and upload) too long anyway so I prefer to test ideas with local validation.", "votes": 4 }, { "id": 217528, "postDate": "2017-08-30T21:43:29.247Z", "content": "

Kaggle team, I hope you'll take this as constructive criticism. While I can appreciate some seemingly simple problems may have integration challenges to implement, the fact this problem has gone on so long is really damaging your credibility. Kaggle is in the business of supporting competitions that deal with pushing the frontiers of data science to the edge. So how can it be that a problem that compromises the spirit of competition and is fundamentally an elementary data problem (rounding, truncating, sorting) take this long? The irony of this is really setting in and creating negative perceptions. You missed out on an opportunity to jump on this one and get it solved quickly when it was pointed out weeks ago. At a minimum Kaggle could have just backed out of the truncated score all together on this competition given the negligible risk of leaderboard probing on 100k images at 1918x1280 resolution. Given that didn't happen, the implementation should proceed with much higher urgency.\nIt's not even clear to me when I submit a solution if it is better or worse than previous given the conflicting info between my sorted submissions the the message I get on whether there was an improvement.
\nPlease, all we want is for this to be fixed so we can measure the value of our work. Right now the competitors, Carvana, and Kaggle are all losing.

", "rawMarkdown": "Kaggle team, I hope you'll take this as constructive criticism. While I can appreciate some seemingly simple problems may have integration challenges to implement, the fact this problem has gone on so long is really damaging your credibility. Kaggle is in the business of supporting competitions that deal with pushing the frontiers of data science to the edge. So how can it be that a problem that compromises the spirit of competition and is fundamentally an elementary data problem (rounding, truncating, sorting) take this long? The irony of this is really setting in and creating negative perceptions. You missed out on an opportunity to jump on this one and get it solved quickly when it was pointed out weeks ago. At a minimum Kaggle could have just backed out of the truncated score all together on this competition given the negligible risk of leaderboard probing on 100k images at 1918x1280 resolution. Given that didn't happen, the implementation should proceed with much higher urgency.\nIt's not even clear to me when I submit a solution if it is better or worse than previous given the conflicting info between my sorted submissions the the message I get on whether there was an improvement. \nPlease, all we want is for this to be fixed so we can measure the value of our work. Right now the competitors, Carvana, and Kaggle are all losing.", "votes": 13 }, { "id": 217530, "postDate": "2017-08-30T21:53:01.733Z", "rawMarkdown": "", "isDeleted": true }, { "id": 217536, "postDate": "2017-08-30T22:28:58.280Z", "content": "

@beluga: Of course I use the local validation as well. But due to the large size and incredibly long time to train each one of the models, my favorite form of local validation (CV) is very inefficient. It takes up to 5 days for a 5-fold CV, vs. five minutes for uploading stuff here. If there ever was a competition that could help us with giving us extra insight into how well we are doing with our models, this would be it.

\n\n

Another potentially major issue with inaccurately ranked LB is that we are hindered in deciding if we want to team up with someone else. What if I want to team up with someone who seems to be really high on LB, but it turns out that my best model is actually better than his/her?

", "rawMarkdown": "@beluga: Of course I use the local validation as well. But due to the large size and incredibly long time to train each one of the models, my favorite form of local validation (CV) is very inefficient. It takes up to 5 days for a 5-fold CV, vs. five minutes for uploading stuff here. If there ever was a competition that could help us with giving us extra insight into how well we are doing with our models, this would be it.\n\nAnother potentially major issue with inaccurately ranked LB is that we are hindered in deciding if we want to team up with someone else. What if I want to team up with someone who seems to be really high on LB, but it turns out that my best model is actually better than his/her?", "votes": 2 }, { "id": 217539, "postDate": "2017-08-30T23:36:04.170Z", "content": "

@Bojan\nI don't think there are that many downsides besides losing some ranking points. At the very least the scores are sorted by the bins.

", "rawMarkdown": "@Bojan\nI don't think there are that many downsides besides losing some ranking points. At the very least the scores are sorted by the bins." }, { "id": 217725, "postDate": "2017-08-31T18:54:05.960Z", "content": "

On the other post it was said it would be days too. And then that it would be yesterday or the day before. So it already took weeks. And it took weeks before that to get the problem acknowledged.

\n\n

I mean, you are owned by Google now... Are we really relying on only one dev and if he has a day off then nobody can do anything?

\n\n

At least can we have some temporary fix in the mean while? The score on the source code was one. Why removing it so fast?

", "rawMarkdown": "On the other post it was said it would be days too. And then that it would be yesterday or the day before. So it already took weeks. And it took weeks before that to get the problem acknowledged.\n\nI mean, you are owned by Google now... Are we really relying on only one dev and if he has a day off then nobody can do anything?\n\nAt least can we have some temporary fix in the mean while? The score on the source code was one. Why removing it so fast?" }, { "id": 217726, "postDate": "2017-08-31T18:55:59.940Z", "content": "

The LB has been just fixed. https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38818

\n\n

I could use one more digit as well but at least it should show the correct rank now.

", "rawMarkdown": "The LB has been just fixed. https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38818\n\n\nI could use one more digit as well but at least it should show the correct rank now.", "votes": 1 }, { "id": 217730, "postDate": "2017-08-31T19:09:37.743Z", "content": "

thanks Beluga! I missed this one

", "rawMarkdown": "thanks Beluga! I missed this one" }, { "id": 217731, "postDate": "2017-08-31T19:12:23.880Z", "content": "

@charles and others: we're sorry the response to this took so long to land. We indeed take your feedback constructively and will do our best to respond in a more timely matter to future issues.

\n\n

In case you're wondering why the long fix for this, we were delayed by two contributing factors (not making excuses, just giving you clarity):

\n\n
    \n
  1. We tend to be more careful when it comes to touching leaderboard code. Deploying something that accidentally leaks private scores is in nobody's best interests.
    2. \n
  2. I had an unexpected death in the family and have only just returned to work. The engineer who created this fix was uncomfortable launching without my review. (I would normally not share this level of personal detail here, but we admins are humans, and in this case it clearly delayed our response).
    3. \n
", "rawMarkdown": "@charles and others: we're sorry the response to this took so long to land. We indeed take your feedback constructively and will do our best to respond in a more timely matter to future issues.\n\nIn case you're wondering why the long fix for this, we were delayed by two contributing factors (not making excuses, just giving you clarity):\n\n 1. We tend to be more careful when it comes to touching leaderboard code. Deploying something that accidentally leaks private scores is in nobody's best interests.\n 2. I had an unexpected death in the family and have only just returned to work. The engineer who created this fix was uncomfortable launching without my review. (I would normally not share this level of personal detail here, but we admins are humans, and in this case it clearly delayed our response).", "votes": 12 }, { "id": 217751, "postDate": "2017-08-31T20:04:02.187Z", "content": "

@Williams,

\n\n

Sorry for your loss and all your efforts in taking care of this issue is highly appreciated.

\n\n

I believe @Charle's comment is coming from the frustrations most of us feel especially with the dateline closing in on us. Training, predicting and submitting for this competition takes a long time even for those who have the best HW setup.

", "rawMarkdown": "@Williams,\n\nSorry for your loss and all your efforts in taking care of this issue is highly appreciated. \n\nI believe @Charle's comment is coming from the frustrations most of us feel especially with the dateline closing in on us. Training, predicting and submitting for this competition takes a long time even for those who have the best HW setup.", "votes": 4 }, { "id": 217787, "postDate": "2017-08-31T22:56:39.487Z", "content": "

@William - Sorry for your loss. there are things in life far more important than leaderboard bugs. glad you're back.

", "rawMarkdown": "@William - Sorry for your loss. there are things in life far more important than leaderboard bugs. glad you're back.", "votes": 6 } ] }, { "id": 217463, "postDate": "2017-08-30T16:55:25.840Z", "content": "

title.

", "rawMarkdown": "title.", "votes": -1 } ], "comments": [ { "id": 217517, "author_name": "Davide Boschetto", "author_url": "", "post_date": "2017-08-30T20:59:43.243000", "content": "

I just officially glided out of the top10%, let's hope the fix comes soon so I can get back full speed on this :) by the way, is it planned to increase the visible digits (99.7XX)?

", "votes": 3, "replies": [] }, { "id": 217464, "author_name": "Will Cukierski", "author_url": "", "post_date": "2017-08-30T17:00:56.543000", "content": "

Please do not do this. We will get the fix up as soon as possible.

", "votes": 2, "replies": [ { "id": 217467, "author_name": "ahmed anis", "author_url": "", "post_date": "2017-08-30T17:03:21.350000", "content": "

Thx for your reply,\ncan we have any expected time when the fix will be applied? \nThanks

", "votes": 0, "replies": [] }, { "id": 217469, "author_name": "Will Cukierski", "author_url": "", "post_date": "2017-08-30T17:06:57.287000", "content": "

I can't give a firm timeline, but it should be a matter of days, not weeks. The code is written and all that remains is testing.

", "votes": 0, "replies": [] }, { "id": 217499, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2017-08-30T18:53:09.717000", "content": "

Thanks for the update. However, the timeframe (\"days not weeks\") still seems too long and vague. Right now most of us are\"flying blind\" when it comes to estimating how well our models are doing. In light of all of this, would it be possible to extend the competition deadline?

", "votes": 3, "replies": [] }, { "id": 217502, "author_name": "beluga", "author_url": "", "post_date": "2017-08-30T19:01:49.697000", "content": "

I am waiting for the LB bugfix as well. \nBut how about using local validation till then? :P Proper local validation framework will be useful at the end as well. I found the submission creation (and upload) too long anyway so I prefer to test ideas with local validation.

", "votes": 4, "replies": [] }, { "id": 217528, "author_name": "David Austin", "author_url": "", "post_date": "2017-08-30T21:43:29.247000", "content": "

Kaggle team, I hope you'll take this as constructive criticism. While I can appreciate some seemingly simple problems may have integration challenges to implement, the fact this problem has gone on so long is really damaging your credibility. Kaggle is in the business of supporting competitions that deal with pushing the frontiers of data science to the edge. So how can it be that a problem that compromises the spirit of competition and is fundamentally an elementary data problem (rounding, truncating, sorting) take this long? The irony of this is really setting in and creating negative perceptions. You missed out on an opportunity to jump on this one and get it solved quickly when it was pointed out weeks ago. At a minimum Kaggle could have just backed out of the truncated score all together on this competition given the negligible risk of leaderboard probing on 100k images at 1918x1280 resolution. Given that didn't happen, the implementation should proceed with much higher urgency.\nIt's not even clear to me when I submit a solution if it is better or worse than previous given the conflicting info between my sorted submissions the the message I get on whether there was an improvement.
\nPlease, all we want is for this to be fixed so we can measure the value of our work. Right now the competitors, Carvana, and Kaggle are all losing.

", "votes": 13, "replies": [] }, { "id": 217530, "author_name": "", "author_url": "", "post_date": "2017-08-30T21:53:01.733000", "content": "", "votes": 0, "replies": [] }, { "id": 217536, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2017-08-30T22:28:58.280000", "content": "

@beluga: Of course I use the local validation as well. But due to the large size and incredibly long time to train each one of the models, my favorite form of local validation (CV) is very inefficient. It takes up to 5 days for a 5-fold CV, vs. five minutes for uploading stuff here. If there ever was a competition that could help us with giving us extra insight into how well we are doing with our models, this would be it.

\n\n

Another potentially major issue with inaccurately ranked LB is that we are hindered in deciding if we want to team up with someone else. What if I want to team up with someone who seems to be really high on LB, but it turns out that my best model is actually better than his/her?

", "votes": 2, "replies": [] }, { "id": 217539, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2017-08-30T23:36:04.170000", "content": "

@Bojan\nI don't think there are that many downsides besides losing some ranking points. At the very least the scores are sorted by the bins.

", "votes": 0, "replies": [] }, { "id": 217725, "author_name": "Charles Jansen", "author_url": "", "post_date": "2017-08-31T18:54:05.960000", "content": "

On the other post it was said it would be days too. And then that it would be yesterday or the day before. So it already took weeks. And it took weeks before that to get the problem acknowledged.

\n\n

I mean, you are owned by Google now... Are we really relying on only one dev and if he has a day off then nobody can do anything?

\n\n

At least can we have some temporary fix in the mean while? The score on the source code was one. Why removing it so fast?

", "votes": 0, "replies": [] }, { "id": 217726, "author_name": "beluga", "author_url": "", "post_date": "2017-08-31T18:55:59.940000", "content": "

The LB has been just fixed. https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38818

\n\n

I could use one more digit as well but at least it should show the correct rank now.

", "votes": 1, "replies": [] }, { "id": 217730, "author_name": "Charles Jansen", "author_url": "", "post_date": "2017-08-31T19:09:37.743000", "content": "

thanks Beluga! I missed this one

", "votes": 0, "replies": [] }, { "id": 217731, "author_name": "Will Cukierski", "author_url": "", "post_date": "2017-08-31T19:12:23.880000", "content": "

@charles and others: we're sorry the response to this took so long to land. We indeed take your feedback constructively and will do our best to respond in a more timely matter to future issues.

\n\n

In case you're wondering why the long fix for this, we were delayed by two contributing factors (not making excuses, just giving you clarity):

\n\n
    \n
  1. We tend to be more careful when it comes to touching leaderboard code. Deploying something that accidentally leaks private scores is in nobody's best interests.
    2. \n
  2. I had an unexpected death in the family and have only just returned to work. The engineer who created this fix was uncomfortable launching without my review. (I would normally not share this level of personal detail here, but we admins are humans, and in this case it clearly delayed our response).
    3. \n
", "votes": 12, "replies": [] }, { "id": 217751, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2017-08-31T20:04:02.187000", "content": "

@Williams,

\n\n

Sorry for your loss and all your efforts in taking care of this issue is highly appreciated.

\n\n

I believe @Charle's comment is coming from the frustrations most of us feel especially with the dateline closing in on us. Training, predicting and submitting for this competition takes a long time even for those who have the best HW setup.

", "votes": 4, "replies": [] }, { "id": 217787, "author_name": "David Austin", "author_url": "", "post_date": "2017-08-31T22:56:39.487000", "content": "

@William - Sorry for your loss. there are things in life far more important than leaderboard bugs. glad you're back.

", "votes": 6, "replies": [] } ] } ], "raw_markdown_by_id": { "217517": "I just officially glided out of the top10%, let's hope the fix comes soon so I can get back full speed on this :) by the way, is it planned to increase the visible digits (99.7XX)? ", "217464": "Please do not do this. We will get the fix up as soon as possible.", "217463": "title." } }