First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps.
\n\nRoot Cause Analysis and Countermeasures
\n\nI'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both.
\n\nIt's actually quite embarrassing when I did the initial investigation after reading the report of leakage in the forum. Airbus had provided location information so the Train / Test split would not include overlapping images. This point was literally the first note in my data processing script, \"Split by location.\" Yet, it got missed in the details.
\n\nIt is fairly well known that a person's brain can overlook something extremely important as a result of being out-of-routine, in a stressful situation, sleep deprived, or any combination of the three. I happened to be in all three zones when the first mistake happened.
\n\nIt is wrong to blame a process breakdown on a stressful week. (It was actually quite bad, and included a death in the family.) Humans have an upper limit of capability, even when they're performing at their best, and that limit is never completely mistake-free. That's why I am a strong proponent of standard checklists. In fact, at Kaggle we do independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.
\n\nThe location information that would have prompted the independent reviewer to ask the obvious question about Train/Test split was not included as part of the competition data (by design), and thus the unintended side effect was that it did not get the scrutiny it should have. Had I provided a list of additional considerations to the reviewer, this would have been caught.
\n\nTo keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this.
\n\nAgain, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!
", "messages": [ { "id": 377371, "postDate": "2018-08-29T03:31:49.820Z", "content": "First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps.
\n\nRoot Cause Analysis and Countermeasures
\n\nI'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both.
\n\nIt's actually quite embarrassing when I did the initial investigation after reading the report of leakage in the forum. Airbus had provided location information so the Train / Test split would not include overlapping images. This point was literally the first note in my data processing script, \"Split by location.\" Yet, it got missed in the details.
\n\nIt is fairly well known that a person's brain can overlook something extremely important as a result of being out-of-routine, in a stressful situation, sleep deprived, or any combination of the three. I happened to be in all three zones when the first mistake happened.
\n\nIt is wrong to blame a process breakdown on a stressful week. (It was actually quite bad, and included a death in the family.) Humans have an upper limit of capability, even when they're performing at their best, and that limit is never completely mistake-free. That's why I am a strong proponent of standard checklists. In fact, at Kaggle we do independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.
\n\nThe location information that would have prompted the independent reviewer to ask the obvious question about Train/Test split was not included as part of the competition data (by design), and thus the unintended side effect was that it did not get the scrutiny it should have. Had I provided a list of additional considerations to the reviewer, this would have been caught.
\n\nTo keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this.
\n\nAgain, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!
", "rawMarkdown": "First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps.\n\n - A new set of Test images will be provided for the competition in a few weeks\n - In the meantime, the competition will still be \"open\" in the sense that the current data will be available and you can submit to the leaderboard\n - To ensure everyone is on an even playing field, the current Test masks will be released (timing: Wed, Aug 29, by EOD)\n - Once new test images are collected, we'll \"reset\" the data by moving current Test images into the Train folder, upload the new Test data, and reset the leaderboard\n - Once the new data has been uploaded, we will extend the competition deadline to account for the delay\n\n\n**Root Cause Analysis and Countermeasures**\n\nI'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both.\n\nIt's actually quite embarrassing when I did the initial investigation after reading the report of leakage in the forum. Airbus had provided location information so the Train / Test split would not include overlapping images. This point was literally the first note in my data processing script, \"Split by location.\" Yet, it got missed in the details.\n\nIt is fairly well known that a person's brain can overlook something extremely important as a result of being out-of-routine, in a stressful situation, sleep deprived, or any combination of the three. I happened to be in all three zones when the first mistake happened. \n\nIt is wrong to blame a process breakdown on a stressful week. (It was actually quite bad, and included a death in the family.) Humans have an upper limit of capability, even when they're performing at their best, and that limit is never completely mistake-free. That's why I am a [strong proponent of standard checklists](https://www.kaggle.com/general/19959). In fact, at Kaggle we _do_ independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.\n\nThe location information that would have prompted the independent reviewer to ask the obvious question about Train/Test split was not included as part of the competition data (by design), and thus the unintended side effect was that it did not get the scrutiny it should have. Had I provided a list of additional considerations to the reviewer, this would have been caught. \n\nTo keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this. \n\nAgain, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!", "votes": 150 }, { "id": 377530, "postDate": "2018-08-29T10:26:00.760Z", "content": "It takes courage to admit one's error. You have shown a great deal of courage here. Respect.
", "rawMarkdown": "It takes courage to admit one's error. You have shown a great deal of courage here. Respect.", "votes": 52 }, { "id": 377408, "postDate": "2018-08-29T05:44:09.570Z", "content": "Thank you for your fast and meaningful response to this situation. The overall tone of your message and willingness to put yourself in the fire and not shifting the blame left and right sounds reassuring that eventually we will get to the point when competitions will flow uninterrupted and as intended.
\n\nSorry for your family loss, please take some rest, sleep well and relax a bit, so all of us could enjoy more and more flawless competitions in the future.
", "rawMarkdown": "Thank you for your fast and meaningful response to this situation. The overall tone of your message and willingness to put yourself in the fire and not shifting the blame left and right sounds reassuring that eventually we will get to the point when competitions will flow uninterrupted and as intended. \n\nSorry for your family loss, please take some rest, sleep well and relax a bit, so all of us could enjoy more and more flawless competitions in the future. ", "votes": 33 }, { "id": 382220, "postDate": "2018-09-05T22:27:27.440Z", "content": "Thank you all for your patience, Airbus has offered new data for the competition. We are assessing the time required to gather and annotate the new data and we should have an update on timing in about a week.
", "rawMarkdown": "Thank you all for your patience, Airbus has offered new data for the competition. We are assessing the time required to gather and annotate the new data and we should have an update on timing in about a week. ", "votes": 20, "replies": [ { "id": 387443, "postDate": "2018-09-14T23:05:32.067Z", "content": "Thank you
", "rawMarkdown": "Thank you", "votes": 1 }, { "id": 388183, "postDate": "2018-09-16T13:58:45.920Z", "content": "Was this done?
", "rawMarkdown": "Was this done?", "votes": 1 }, { "id": 388186, "postDate": "2018-09-16T14:02:19.270Z", "content": "Any updates?
", "rawMarkdown": "Any updates?", "votes": 16 }, { "id": 392378, "postDate": "2018-09-23T16:51:34.660Z", "content": "Hello Kaggle team,
\n\nHope everything is going well. Any update on this? Could this prompt an extension in the timeline?
\n\nThanks and appreciate all your hard work!
", "rawMarkdown": "Hello Kaggle team,\n\nHope everything is going well. Any update on this? Could this prompt an extension in the timeline?\n\nThanks and appreciate all your hard work! " } ] }, { "id": 378143, "postDate": "2018-08-30T02:48:38.923Z", "content": "Thank you Inversion! Your honesty and boldness is much appreciated.
\n\nAlso, thanks for all the hard work that you and your team puts in to provide us this platform.
\n\nSo at my work, last week I realized that I had pulled data from wrong sql server and my analysis for whole week was rendered useless. It was quite embarrassing to face my client about that. But, I am quite fortunate I do not have hundreds of people, scrutinizing my mistake and giving me hundreds of suggestions on what I could have done to prevent that error.
\n\nThe point being all of us make mistakes, even very silly ones, however strong counter measures. Yes, it is important to scrutinize mistakes but it is equally important to show compassion when a mistake is accepted.
", "rawMarkdown": "Thank you Inversion! Your honesty and boldness is much appreciated.\n\nAlso, thanks for all the hard work that you and your team puts in to provide us this platform.\n\nSo at my work, last week I realized that I had pulled data from wrong sql server and my analysis for whole week was rendered useless. It was quite embarrassing to face my client about that. But, I am quite fortunate I do not have hundreds of people, scrutinizing my mistake and giving me hundreds of suggestions on what I could have done to prevent that error.\n\nThe point being all of us make mistakes, even very silly ones, however strong counter measures. Yes, it is important to scrutinize mistakes but it is equally important to show compassion when a mistake is accepted.\n\n\n\n ", "votes": 14 }, { "id": 377908, "postDate": "2018-08-29T23:44:30.810Z", "content": "I thought this was the Airbus challenge, not the Titanic one?
", "rawMarkdown": "I thought this was the Airbus challenge, not the Titanic one?", "votes": 7, "replies": [ { "id": 399789, "postDate": "2018-10-06T19:10:47.530Z", "rawMarkdown": "" } ] }, { "id": 377405, "postDate": "2018-08-29T05:41:19.307Z", "content": "Sounds like a good and fair approach to the problem at hand. And once the new data is released, for people a good change to see how much their current network was actually overfitting ;)
\n\nSo thanks for coming up with this solution so quickly and really sorry to hear about your personal loss.
", "rawMarkdown": "Sounds like a good and fair approach to the problem at hand. And once the new data is released, for people a good change to see how much their current network was actually overfitting ;)\n\nSo thanks for coming up with this solution so quickly and really sorry to hear about your personal loss.", "votes": 7 }, { "id": 380089, "postDate": "2018-09-01T16:54:52.963Z", "content": "I appreciate the honesty and the reasonable actions that have been taken so far. \nThanks inversion. :)
", "rawMarkdown": "I appreciate the honesty and the reasonable actions that have been taken so far. \nThanks inversion. :)", "votes": 7 }, { "id": 382162, "postDate": "2018-09-05T20:14:44.123Z", "content": "Any news on when the new test data will be available for download?
", "rawMarkdown": "Any news on when the new test data will be available for download?", "votes": 5 }, { "id": 377620, "postDate": "2018-08-29T13:10:03.797Z", "content": "Good members of staff are the ones that take good actions in a good timely manner. You've shown a good example.
", "rawMarkdown": "Good members of staff are the ones that take good actions in a good timely manner. You've shown a good example.", "votes": 6 }, { "id": 380042, "postDate": "2018-09-01T15:00:27.740Z", "content": "Do you have an estimated date for the new test set release?
", "rawMarkdown": "Do you have an estimated date for the new test set release?", "votes": 4 }, { "id": 379108, "postDate": "2018-08-30T21:55:20.947Z", "content": "I came to join this competition and found @Andres Torrubia's leak post.
\n\n@Inversion, thanks for taking responsibility and I am sorry for the loss in your family.
", "rawMarkdown": "I came to join this competition and found @Andres Torrubia's leak post.\n\n@Inversion, thanks for taking responsibility and I am sorry for the loss in your family.\n", "votes": 1 }, { "id": 377652, "postDate": "2018-08-29T14:46:15.133Z", "content": "Do the images also have timestamps? It might be possible to track ships, even if the images are not overlapping. You might consider checking the timestamps as well. Or just release the space-time stamps and add the additional twist of tracking as part of the competition. The other reason its important to know space-time info is to make cross-validation easier and more robust. Whether there are ships in an image or not, I don't want images from nearby locations taken at the same or similar timestamp on different folds.
", "rawMarkdown": "Do the images also have timestamps? It might be possible to track ships, even if the images are not overlapping. You might consider checking the timestamps as well. Or just release the space-time stamps and add the additional twist of tracking as part of the competition. The other reason its important to know space-time info is to make cross-validation easier and more robust. Whether there are ships in an image or not, I don't want images from nearby locations taken at the same or similar timestamp on different folds. ", "votes": 1, "replies": [ { "id": 377722, "postDate": "2018-08-29T16:47:30.003Z", "content": "No they do not have timestamps. My understanding is that the images are chips from one larger image, therefore from the same date, covering something like 60km wide and up to 600km long.
", "rawMarkdown": "No they do not have timestamps. My understanding is that the images are chips from one larger image, therefore from the same date, covering something like 60km wide and up to 600km long." }, { "id": 377731, "postDate": "2018-08-29T16:56:19.733Z", "content": "Well the images do not have spatial information either. But the data that Inversion has access to DOES have spatial information. I am referring to the data that HE has and uses to slice up the images between train and test sets.
", "rawMarkdown": "Well the images do not have spatial information either. But the data that Inversion has access to DOES have spatial information. I am referring to the data that HE has and uses to slice up the images between train and test sets.", "votes": 2 }, { "id": 377736, "postDate": "2018-08-29T17:02:03.673Z", "content": "You´re right.
", "rawMarkdown": "You´re right." } ] }, { "id": 377543, "postDate": "2018-08-29T10:42:34.543Z", "content": "Don't sweat it too much - we all make mistakes. It's the willingness to learn from them that matters.
", "rawMarkdown": "Don't sweat it too much - we all make mistakes. It's the willingness to learn from them that matters.", "votes": 1 }, { "id": 390287, "postDate": "2018-09-20T01:25:41.380Z", "content": "I am sorry for your loss, Inversion and for your family. Hang in there. This will still be a nice competition when reset. Kaggle is still a great place to learn and get feedback. Thank you for your work.
", "rawMarkdown": "I am sorry for your loss, Inversion and for your family. Hang in there. This will still be a nice competition when reset. Kaggle is still a great place to learn and get feedback. Thank you for your work.", "votes": 2 }, { "id": 379447, "postDate": "2018-08-31T11:18:12.673Z", "content": "After all.. we are humans ..!
\nThis one incident can't change the good work you have done so far.. Sorry about the loss in your family. Take care.
@inversion Could you please confirm if the new dataset is released ?
", "rawMarkdown": " @inversion Could you please confirm if the new dataset is released ?", "votes": 2, "replies": [ { "id": 378962, "postDate": "2018-08-30T17:47:42.097Z", "content": "The current test mask (not new test images) will be available in a couple of hours. I'll make a forum post when it's ready.
", "rawMarkdown": "The current test mask (not new test images) will be available in a couple of hours. I'll make a forum post when it's ready.", "votes": 3 }, { "id": 379126, "postDate": "2018-08-30T23:09:49.080Z", "content": "Sure. Np, thank you for taking time to fix this.
", "rawMarkdown": "Sure. Np, thank you for taking time to fix this.", "votes": 1 } ] }, { "id": 377831, "postDate": "2018-08-29T20:13:09.117Z", "content": "We all make mistakes, hope you dont stress over it too much inversion and keep up the good work.
", "rawMarkdown": "We all make mistakes, hope you dont stress over it too much inversion and keep up the good work.", "votes": 2 }, { "id": 383531, "postDate": "2018-09-08T23:19:51.490Z", "content": "I am new to the competition and forgive if sound like I didn't spend enough time investigating or fact checking. Please tell me I understand correctly: due to leakage we now have train.zip and test.zip both annotated as a full training set. On top of this we will be provided a fully new test set. Is it correct interpretation ? Thanks a lot !
\n\nSo now there is no point of making submission for LB probing ?
", "rawMarkdown": "I am new to the competition and forgive if sound like I didn't spend enough time investigating or fact checking. Please tell me I understand correctly: due to leakage we now have train.zip and test.zip both annotated as a full training set. On top of this we will be provided a fully new test set. Is it correct interpretation ? Thanks a lot !\n\nSo now there is no point of making submission for LB probing ?", "replies": [ { "id": 383654, "postDate": "2018-09-09T07:45:49.353Z", "content": "Correct, we are waiting till the new test set is out, the timeline and leaderboard will be reset afterwards
", "rawMarkdown": "Correct, we are waiting till the new test set is out, the timeline and leaderboard will be reset afterwards", "votes": 8 }, { "id": 383904, "postDate": "2018-09-09T22:40:29.033Z", "content": "Thank you AIFORCE
", "rawMarkdown": "Thank you AIFORCE" } ] }, { "id": 377461, "postDate": "2018-08-29T07:48:43.263Z", "content": "Thanks. A new test set might include not only new locations, but also known locations shot at different time.
", "rawMarkdown": "Thanks. A new test set might include not only new locations, but also known locations shot at different time." }, { "id": 377859, "postDate": "2018-08-29T21:10:11.280Z", "content": "Experience is simply the name we give our mistakes. At least this time you admins did a nice job responding promptly, honestly admitting your mistake and scheduling corrective actions. Damage limitation was the only thing you could do afterwards and you're doing it fair enough.
", "rawMarkdown": "Experience is simply the name we give our mistakes. At least this time you admins did a nice job responding promptly, honestly admitting your mistake and scheduling corrective actions. Damage limitation was the only thing you could do afterwards and you're doing it fair enough.", "votes": 1 }, { "id": 380507, "postDate": "2018-09-02T20:54:46.767Z", "content": "take care & respect. @inversion, could you please help, to review the compliance report sent from me? it is about Home-Credit final result been removed. final result was 33rd. I have sent many emails before the deadline to explain the case, but got no response yet. would be appreciate for your attention. thanks in advance.
", "rawMarkdown": "take care & respect. @inversion, could you please help, to review the compliance report sent from me? it is about Home-Credit final result been removed. final result was 33rd. I have sent many emails before the deadline to explain the case, but got no response yet. would be appreciate for your attention. thanks in advance.", "votes": -1 }, { "id": 377413, "postDate": "2018-08-29T06:00:32.013Z", "content": "\n\n\nThat's why I am a strong proponent of standard checklists. In fact, at Kaggle we do independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.\n I welcome and appreciate your feedback (even when it's brutally blunt)!
\n
Please consider some ideas in this thread
", "rawMarkdown": "> That's why I am a strong proponent of standard checklists. In fact, at Kaggle we do independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.\n> I welcome and appreciate your feedback (even when it's brutally blunt)!\n\nPlease consider some ideas in this [thread](https://www.kaggle.com/c/airbus-ship-detection/discussion/64393)\n", "votes": -17 }, { "id": 399058, "postDate": "2018-10-05T04:53:28.370Z", "content": "I appreciate your courage and integrity in the way you accept responsibility, and as so many said, we all make mistakes. I hope and pray your family is healing, and I am sorry for your loss. I haven't done a Kaggle challenge yet, but I'm looking them over considering jumping in on one. I am impressed with this whole crowd, the support AI folks have for each other and the amazing courses so many people have made available not to mention Kaggle itself.
", "rawMarkdown": "I appreciate your courage and integrity in the way you accept responsibility, and as so many said, we all make mistakes. I hope and pray your family is healing, and I am sorry for your loss. I haven't done a Kaggle challenge yet, but I'm looking them over considering jumping in on one. I am impressed with this whole crowd, the support AI folks have for each other and the amazing courses so many people have made available not to mention Kaggle itself." }, { "id": 393521, "postDate": "2018-09-25T14:10:43.453Z", "content": "Sorry about your loss !! You have exhibited a strong character in detailing out this. hats off !!
", "rawMarkdown": "Sorry about your loss !! You have exhibited a strong character in detailing out this. hats off !!" }, { "id": 393432, "postDate": "2018-09-25T11:07:32.727Z", "content": "Hello, what's up ? We are now 10 days from the official competition deadline and new test data is not available. I'm sure you are working on that, but will you extend the last submission deadline or is the timeline unaffected ?
", "rawMarkdown": "Hello, what's up ? We are now 10 days from the official competition deadline and new test data is not available. I'm sure you are working on that, but will you extend the last submission deadline or is the timeline unaffected ?\n", "replies": [ { "id": 393682, "postDate": "2018-09-25T18:46:43.567Z", "content": "He said, it will take at least 1-2 weeks.
", "rawMarkdown": "He said, it will take at least 1-2 weeks." } ] }, { "id": 391608, "postDate": "2018-09-22T05:15:46.230Z", "content": "Am I mistaken in that Fast.ai is not allowed in the Airbus competition?
", "rawMarkdown": "Am I mistaken in that Fast.ai is not allowed in the Airbus competition?", "replies": [ { "id": 392565, "postDate": "2018-09-24T01:49:04.780Z", "content": "Do you know the reason ?
", "rawMarkdown": "Do you know the reason ?\n" }, { "id": 393542, "postDate": "2018-09-25T14:39:27.957Z", "content": "i dont think it should be.. purpose of competition is find the best solution dsnt matters which framework or libraries used...
", "rawMarkdown": "i dont think it should be.. purpose of competition is find the best solution dsnt matters which framework or libraries used..." } ] }, { "id": 390018, "postDate": "2018-09-19T15:29:49.357Z", "content": "may be this qs could be very naive...\n1) I dint understand the data leak ... please by giving a brief\n2) if we are provided the Test segmentations info .... then what else remains there to predict say what if i take same segmentations file and submit...
", "rawMarkdown": "may be this qs could be very naive...\n1) I dint understand the data leak ... please by giving a brief\n2) if we are provided the Test segmentations info .... then what else remains there to predict say what if i take same segmentations file and submit...\n", "replies": [ { "id": 390350, "postDate": "2018-09-20T04:29:01.900Z", "content": "You get 1.0, but many have already done that if you look at the leaderboard. But if you go through the comments, and other discussions, you would come to know that new test data is being prepared and will be released soon and the leaderboard will be reset again.
", "rawMarkdown": "You get 1.0, but many have already done that if you look at the leaderboard. But if you go through the comments, and other discussions, you would come to know that new test data is being prepared and will be released soon and the leaderboard will be reset again." } ] }, { "id": 389518, "postDate": "2018-09-18T21:12:03.587Z", "content": "Is there anyone who knows the new test set will come out?
", "rawMarkdown": "Is there anyone who knows the new test set will come out?", "replies": [ { "id": 389522, "postDate": "2018-09-18T21:14:55.253Z", "content": "The new data is currently being collected. It will be at least 1-2 weeks to finalize.
", "rawMarkdown": "The new data is currently being collected. It will be at least 1-2 weeks to finalize.", "votes": 2 }, { "id": 389525, "postDate": "2018-09-18T21:19:18.857Z", "content": "That's good news. Thanks for sharing!
", "rawMarkdown": "That's good news. Thanks for sharing!" }, { "id": 389547, "postDate": "2018-09-18T22:10:54.567Z", "content": "Thanks for your reply. That is helpful for planning my stuff. How will the deadline be extended from there after the new testing data is released?
", "rawMarkdown": "Thanks for your reply. That is helpful for planning my stuff. How will the deadline be extended from there after the new testing data is released?" } ] }, { "id": 383400, "postDate": "2018-09-08T15:51:18.767Z", "content": "Had a quick question for tha Kaggle team. Once the new data is released, by what margin will the timeline be extended? Is it by the same amount of time that was lost while we had this data? Or do you guys have a fixed timeline in mind?
", "rawMarkdown": "Had a quick question for tha Kaggle team. Once the new data is released, by what margin will the timeline be extended? Is it by the same amount of time that was lost while we had this data? Or do you guys have a fixed timeline in mind?", "replies": [ { "id": 385262, "postDate": "2018-09-10T16:59:18.020Z", "content": "Great question, our goal is to extend the deadline by 1 month. We will include more details when we know when we will update the new data.
", "rawMarkdown": "Great question, our goal is to extend the deadline by 1 month. We will include more details when we know when we will update the new data. ", "votes": 1 } ] }, { "id": 380234, "postDate": "2018-09-02T05:46:25.467Z", "content": "Please fix the leak. It is crazy to see so many 100% acc on LB.
", "rawMarkdown": "Please fix the leak. It is crazy to see so many 100% acc on LB.", "replies": [ { "id": 380246, "postDate": "2018-09-02T06:07:30.120Z", "content": "haha, so many 100%.. do not fall on trap of data leak.. like what happened in Santander! :D
", "rawMarkdown": "haha, so many 100%.. do not fall on trap of data leak.. like what happened in Santander! :D", "votes": 17 } ] }, { "id": 377549, "postDate": "2018-08-29T10:47:08.077Z", "content": "Humans are bound to make mistakes, machines don't...
", "rawMarkdown": "Humans are bound to make mistakes, machines don't...", "replies": [ { "id": 377557, "postDate": "2018-08-29T10:55:28.423Z", "content": "\n\n\nmachines don't...
\n
I'd rather say they either make no mistake, or make the same error an absurdly high number of times...
\n\nOK, this is irrelevant for the current topic but I could not resist... ;)
", "rawMarkdown": "> machines don't...\n\nI'd rather say they either make no mistake, or make the same error an absurdly high number of times...\n\nOK, this is irrelevant for the current topic but I could not resist... ;)", "votes": 2 }, { "id": 377573, "postDate": "2018-08-29T11:19:18.533Z", "content": "My bad, i meant the same as you explained...(was lazy enough to type though..)\nThanks for your response! &\n(Sorry for an irrelevant notification!)
", "rawMarkdown": "My bad, i meant the same as you explained...(was lazy enough to type though..)\nThanks for your response! &\n(Sorry for an irrelevant notification!)" } ] }, { "id": 402907, "postDate": "2018-10-12T14:48:27.990Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 377530, "author_name": "CPMP", "author_url": "", "post_date": "2018-08-29T10:26:00.760000", "content": "It takes courage to admit one's error. You have shown a great deal of courage here. Respect.
", "votes": 52, "replies": [] }, { "id": 377408, "author_name": "Cut Onion", "author_url": "", "post_date": "2018-08-29T05:44:09.570000", "content": "Thank you for your fast and meaningful response to this situation. The overall tone of your message and willingness to put yourself in the fire and not shifting the blame left and right sounds reassuring that eventually we will get to the point when competitions will flow uninterrupted and as intended.
\n\nSorry for your family loss, please take some rest, sleep well and relax a bit, so all of us could enjoy more and more flawless competitions in the future.
", "votes": 33, "replies": [] }, { "id": 382220, "author_name": "Maggie", "author_url": "", "post_date": "2018-09-05T22:27:27.440000", "content": "Thank you all for your patience, Airbus has offered new data for the competition. We are assessing the time required to gather and annotate the new data and we should have an update on timing in about a week.
", "votes": 20, "replies": [ { "id": 387443, "author_name": "Telemachus", "author_url": "", "post_date": "2018-09-14T23:05:32.067000", "content": "Thank you
", "votes": 1, "replies": [] }, { "id": 388183, "author_name": "Ashutosh Tiwari", "author_url": "", "post_date": "2018-09-16T13:58:45.920000", "content": "Was this done?
", "votes": 1, "replies": [] }, { "id": 388186, "author_name": "", "author_url": "", "post_date": "2018-09-16T14:02:19.270000", "content": "Any updates?
", "votes": 16, "replies": [] }, { "id": 392378, "author_name": "Keshan Sodimana", "author_url": "", "post_date": "2018-09-23T16:51:34.660000", "content": "Hello Kaggle team,
\n\nHope everything is going well. Any update on this? Could this prompt an extension in the timeline?
\n\nThanks and appreciate all your hard work!
", "votes": 0, "replies": [] } ] }, { "id": 378143, "author_name": "Mohsin hasan", "author_url": "", "post_date": "2018-08-30T02:48:38.923000", "content": "Thank you Inversion! Your honesty and boldness is much appreciated.
\n\nAlso, thanks for all the hard work that you and your team puts in to provide us this platform.
\n\nSo at my work, last week I realized that I had pulled data from wrong sql server and my analysis for whole week was rendered useless. It was quite embarrassing to face my client about that. But, I am quite fortunate I do not have hundreds of people, scrutinizing my mistake and giving me hundreds of suggestions on what I could have done to prevent that error.
\n\nThe point being all of us make mistakes, even very silly ones, however strong counter measures. Yes, it is important to scrutinize mistakes but it is equally important to show compassion when a mistake is accepted.
", "votes": 14, "replies": [] }, { "id": 377908, "author_name": "Leete Skinner", "author_url": "", "post_date": "2018-08-29T23:44:30.810000", "content": "I thought this was the Airbus challenge, not the Titanic one?
", "votes": 7, "replies": [ { "id": 399789, "author_name": "Anton Trainer", "author_url": "", "post_date": "2018-10-06T19:10:47.530000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 377405, "author_name": "Peter", "author_url": "", "post_date": "2018-08-29T05:41:19.307000", "content": "Sounds like a good and fair approach to the problem at hand. And once the new data is released, for people a good change to see how much their current network was actually overfitting ;)
\n\nSo thanks for coming up with this solution so quickly and really sorry to hear about your personal loss.
", "votes": 7, "replies": [] }, { "id": 380089, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2018-09-01T16:54:52.963000", "content": "I appreciate the honesty and the reasonable actions that have been taken so far. \nThanks inversion. :)
", "votes": 7, "replies": [] }, { "id": 382162, "author_name": "Blueview", "author_url": "", "post_date": "2018-09-05T20:14:44.123000", "content": "Any news on when the new test data will be available for download?
", "votes": 5, "replies": [] }, { "id": 377620, "author_name": "AIFORCE", "author_url": "", "post_date": "2018-08-29T13:10:03.797000", "content": "Good members of staff are the ones that take good actions in a good timely manner. You've shown a good example.
", "votes": 6, "replies": [] }, { "id": 380042, "author_name": "Cosmin", "author_url": "", "post_date": "2018-09-01T15:00:27.740000", "content": "Do you have an estimated date for the new test set release?
", "votes": 4, "replies": [] }, { "id": 379108, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-08-30T21:55:20.947000", "content": "I came to join this competition and found @Andres Torrubia's leak post.
\n\n@Inversion, thanks for taking responsibility and I am sorry for the loss in your family.
", "votes": 1, "replies": [] }, { "id": 377652, "author_name": "snapperwiffer", "author_url": "", "post_date": "2018-08-29T14:46:15.133000", "content": "Do the images also have timestamps? It might be possible to track ships, even if the images are not overlapping. You might consider checking the timestamps as well. Or just release the space-time stamps and add the additional twist of tracking as part of the competition. The other reason its important to know space-time info is to make cross-validation easier and more robust. Whether there are ships in an image or not, I don't want images from nearby locations taken at the same or similar timestamp on different folds.
", "votes": 1, "replies": [ { "id": 377722, "author_name": "Alexis Letulier", "author_url": "", "post_date": "2018-08-29T16:47:30.003000", "content": "No they do not have timestamps. My understanding is that the images are chips from one larger image, therefore from the same date, covering something like 60km wide and up to 600km long.
", "votes": 0, "replies": [] }, { "id": 377731, "author_name": "snapperwiffer", "author_url": "", "post_date": "2018-08-29T16:56:19.733000", "content": "Well the images do not have spatial information either. But the data that Inversion has access to DOES have spatial information. I am referring to the data that HE has and uses to slice up the images between train and test sets.
", "votes": 2, "replies": [] }, { "id": 377736, "author_name": "Alexis Letulier", "author_url": "", "post_date": "2018-08-29T17:02:03.673000", "content": "You´re right.
", "votes": 0, "replies": [] } ] }, { "id": 377543, "author_name": "zarak", "author_url": "", "post_date": "2018-08-29T10:42:34.543000", "content": "Don't sweat it too much - we all make mistakes. It's the willingness to learn from them that matters.
", "votes": 1, "replies": [] }, { "id": 390287, "author_name": "J.Jones", "author_url": "", "post_date": "2018-09-20T01:25:41.380000", "content": "I am sorry for your loss, Inversion and for your family. Hang in there. This will still be a nice competition when reset. Kaggle is still a great place to learn and get feedback. Thank you for your work.
", "votes": 2, "replies": [] }, { "id": 379447, "author_name": "Kishore M", "author_url": "", "post_date": "2018-08-31T11:18:12.673000", "content": "After all.. we are humans ..!
\nThis one incident can't change the good work you have done so far.. Sorry about the loss in your family. Take care.
@inversion Could you please confirm if the new dataset is released ?
", "votes": 2, "replies": [ { "id": 378962, "author_name": "inversion", "author_url": "", "post_date": "2018-08-30T17:47:42.097000", "content": "The current test mask (not new test images) will be available in a couple of hours. I'll make a forum post when it's ready.
", "votes": 3, "replies": [] }, { "id": 379126, "author_name": "gsadhas", "author_url": "", "post_date": "2018-08-30T23:09:49.080000", "content": "Sure. Np, thank you for taking time to fix this.
", "votes": 1, "replies": [] } ] }, { "id": 377831, "author_name": "Tom Aindow", "author_url": "", "post_date": "2018-08-29T20:13:09.117000", "content": "We all make mistakes, hope you dont stress over it too much inversion and keep up the good work.
", "votes": 2, "replies": [] }, { "id": 383531, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2018-09-08T23:19:51.490000", "content": "I am new to the competition and forgive if sound like I didn't spend enough time investigating or fact checking. Please tell me I understand correctly: due to leakage we now have train.zip and test.zip both annotated as a full training set. On top of this we will be provided a fully new test set. Is it correct interpretation ? Thanks a lot !
\n\nSo now there is no point of making submission for LB probing ?
", "votes": 0, "replies": [ { "id": 383654, "author_name": "AIFORCE", "author_url": "", "post_date": "2018-09-09T07:45:49.353000", "content": "Correct, we are waiting till the new test set is out, the timeline and leaderboard will be reset afterwards
", "votes": 8, "replies": [] }, { "id": 383904, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2018-09-09T22:40:29.033000", "content": "Thank you AIFORCE
", "votes": 0, "replies": [] } ] }, { "id": 377461, "author_name": "Igor Buinyi", "author_url": "", "post_date": "2018-08-29T07:48:43.263000", "content": "Thanks. A new test set might include not only new locations, but also known locations shot at different time.
", "votes": 0, "replies": [] }, { "id": 377859, "author_name": "gavrand", "author_url": "", "post_date": "2018-08-29T21:10:11.280000", "content": "Experience is simply the name we give our mistakes. At least this time you admins did a nice job responding promptly, honestly admitting your mistake and scheduling corrective actions. Damage limitation was the only thing you could do afterwards and you're doing it fair enough.
", "votes": 1, "replies": [] }, { "id": 380507, "author_name": "Mind Rank", "author_url": "", "post_date": "2018-09-02T20:54:46.767000", "content": "take care & respect. @inversion, could you please help, to review the compliance report sent from me? it is about Home-Credit final result been removed. final result was 33rd. I have sent many emails before the deadline to explain the case, but got no response yet. would be appreciate for your attention. thanks in advance.
", "votes": -1, "replies": [] }, { "id": 377413, "author_name": "Alexander Veysov", "author_url": "", "post_date": "2018-08-29T06:00:32.013000", "content": "\n\n\nThat's why I am a strong proponent of standard checklists. In fact, at Kaggle we do independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.\n I welcome and appreciate your feedback (even when it's brutally blunt)!
\n
Please consider some ideas in this thread
", "votes": -17, "replies": [] }, { "id": 399058, "author_name": "Dan Dick", "author_url": "", "post_date": "2018-10-05T04:53:28.370000", "content": "I appreciate your courage and integrity in the way you accept responsibility, and as so many said, we all make mistakes. I hope and pray your family is healing, and I am sorry for your loss. I haven't done a Kaggle challenge yet, but I'm looking them over considering jumping in on one. I am impressed with this whole crowd, the support AI folks have for each other and the amazing courses so many people have made available not to mention Kaggle itself.
", "votes": 0, "replies": [] }, { "id": 393521, "author_name": "Vishy", "author_url": "", "post_date": "2018-09-25T14:10:43.453000", "content": "Sorry about your loss !! You have exhibited a strong character in detailing out this. hats off !!
", "votes": 0, "replies": [] }, { "id": 393432, "author_name": "Ben Farcy", "author_url": "", "post_date": "2018-09-25T11:07:32.727000", "content": "Hello, what's up ? We are now 10 days from the official competition deadline and new test data is not available. I'm sure you are working on that, but will you extend the last submission deadline or is the timeline unaffected ?
", "votes": 0, "replies": [ { "id": 393682, "author_name": "Neo", "author_url": "", "post_date": "2018-09-25T18:46:43.567000", "content": "He said, it will take at least 1-2 weeks.
", "votes": 0, "replies": [] } ] }, { "id": 391608, "author_name": "Ajaxon6255", "author_url": "", "post_date": "2018-09-22T05:15:46.230000", "content": "Am I mistaken in that Fast.ai is not allowed in the Airbus competition?
", "votes": 0, "replies": [ { "id": 392565, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2018-09-24T01:49:04.780000", "content": "Do you know the reason ?
", "votes": 0, "replies": [] }, { "id": 393542, "author_name": "Jaideep", "author_url": "", "post_date": "2018-09-25T14:39:27.957000", "content": "i dont think it should be.. purpose of competition is find the best solution dsnt matters which framework or libraries used...
", "votes": 0, "replies": [] } ] }, { "id": 390018, "author_name": "Jaideep", "author_url": "", "post_date": "2018-09-19T15:29:49.357000", "content": "may be this qs could be very naive...\n1) I dint understand the data leak ... please by giving a brief\n2) if we are provided the Test segmentations info .... then what else remains there to predict say what if i take same segmentations file and submit...
", "votes": 0, "replies": [ { "id": 390350, "author_name": "NVS Abhilash", "author_url": "", "post_date": "2018-09-20T04:29:01.900000", "content": "You get 1.0, but many have already done that if you look at the leaderboard. But if you go through the comments, and other discussions, you would come to know that new test data is being prepared and will be released soon and the leaderboard will be reset again.
", "votes": 0, "replies": [] } ] }, { "id": 389518, "author_name": "XiaokangWang", "author_url": "", "post_date": "2018-09-18T21:12:03.587000", "content": "Is there anyone who knows the new test set will come out?
", "votes": 0, "replies": [ { "id": 389522, "author_name": "inversion", "author_url": "", "post_date": "2018-09-18T21:14:55.253000", "content": "The new data is currently being collected. It will be at least 1-2 weeks to finalize.
", "votes": 2, "replies": [] }, { "id": 389525, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2018-09-18T21:19:18.857000", "content": "That's good news. Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 389547, "author_name": "XiaokangWang", "author_url": "", "post_date": "2018-09-18T22:10:54.567000", "content": "Thanks for your reply. That is helpful for planning my stuff. How will the deadline be extended from there after the new testing data is released?
", "votes": 0, "replies": [] } ] }, { "id": 383400, "author_name": "sandy1112", "author_url": "", "post_date": "2018-09-08T15:51:18.767000", "content": "Had a quick question for tha Kaggle team. Once the new data is released, by what margin will the timeline be extended? Is it by the same amount of time that was lost while we had this data? Or do you guys have a fixed timeline in mind?
", "votes": 0, "replies": [ { "id": 385262, "author_name": "Maggie", "author_url": "", "post_date": "2018-09-10T16:59:18.020000", "content": "Great question, our goal is to extend the deadline by 1 month. We will include more details when we know when we will update the new data.
", "votes": 1, "replies": [] } ] }, { "id": 380234, "author_name": "", "author_url": "", "post_date": "2018-09-02T05:46:25.467000", "content": "", "votes": 0, "replies": [ { "id": 380246, "author_name": "", "author_url": "", "post_date": "2018-09-02T06:07:30.120000", "content": "", "votes": 17, "replies": [] } ] }, { "id": 377549, "author_name": "", "author_url": "", "post_date": "2018-08-29T10:47:08.077000", "content": "", "votes": 0, "replies": [ { "id": 377557, "author_name": "", "author_url": "", "post_date": "2018-08-29T10:55:28.423000", "content": "", "votes": 2, "replies": [] }, { "id": 377573, "author_name": "", "author_url": "", "post_date": "2018-08-29T11:19:18.533000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 402907, "author_name": "", "author_url": "", "post_date": "2018-10-12T14:48:27.990000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "377371": "First and foremost, let me express my sincere apologies for the frustration this situation has caused. I'll provide a full explanation of what happened below, but first want to give a high-level summary of the next steps.\n\n - A new set of Test images will be provided for the competition in a few weeks\n - In the meantime, the competition will still be \"open\" in the sense that the current data will be available and you can submit to the leaderboard\n - To ensure everyone is on an even playing field, the current Test masks will be released (timing: Wed, Aug 29, by EOD)\n - Once new test images are collected, we'll \"reset\" the data by moving current Test images into the Train folder, upload the new Test data, and reset the leaderboard\n - Once the new data has been uploaded, we will extend the competition deadline to account for the delay\n\n\n**Root Cause Analysis and Countermeasures**\n\nI'm sure some of you are scratching your heads over how such an obvious mistake could have been made. It was actually a two-part failure, and I happened to be the direct cause of both.\n\nIt's actually quite embarrassing when I did the initial investigation after reading the report of leakage in the forum. Airbus had provided location information so the Train / Test split would not include overlapping images. This point was literally the first note in my data processing script, \"Split by location.\" Yet, it got missed in the details.\n\nIt is fairly well known that a person's brain can overlook something extremely important as a result of being out-of-routine, in a stressful situation, sleep deprived, or any combination of the three. I happened to be in all three zones when the first mistake happened. \n\nIt is wrong to blame a process breakdown on a stressful week. (It was actually quite bad, and included a death in the family.) Humans have an upper limit of capability, even when they're performing at their best, and that limit is never completely mistake-free. That's why I am a [strong proponent of standard checklists](https://www.kaggle.com/general/19959). In fact, at Kaggle we _do_ independent data checks before the launch of each competition. Unfortunately, this is where I made the second mistake.\n\nThe location information that would have prompted the independent reviewer to ask the obvious question about Train/Test split was not included as part of the competition data (by design), and thus the unintended side effect was that it did not get the scrutiny it should have. Had I provided a list of additional considerations to the reviewer, this would have been caught. \n\nTo keep this type of error from happening again, we are updating our independent check process to explicitly include competition-specific watchouts, so a second set of eyes is properly equipped to catch failures such as this. \n\nAgain, I've very sorry for the frustrations arising from this. As always, I welcome and appreciate your feedback (even when it's brutally blunt)!", "377530": "It takes courage to admit one's error. You have shown a great deal of courage here. Respect.", "377408": "Thank you for your fast and meaningful response to this situation. The overall tone of your message and willingness to put yourself in the fire and not shifting the blame left and right sounds reassuring that eventually we will get to the point when competitions will flow uninterrupted and as intended. \n\nSorry for your family loss, please take some rest, sleep well and relax a bit, so all of us could enjoy more and more flawless competitions in the future. ", "382220": "Thank you all for your patience, Airbus has offered new data for the competition. We are assessing the time required to gather and annotate the new data and we should have an update on timing in about a week. ", "378143": "Thank you Inversion! Your honesty and boldness is much appreciated.\n\nAlso, thanks for all the hard work that you and your team puts in to provide us this platform.\n\nSo at my work, last week I realized that I had pulled data from wrong sql server and my analysis for whole week was rendered useless. It was quite embarrassing to face my client about that. But, I am quite fortunate I do not have hundreds of people, scrutinizing my mistake and giving me hundreds of suggestions on what I could have done to prevent that error.\n\nThe point being all of us make mistakes, even very silly ones, however strong counter measures. Yes, it is important to scrutinize mistakes but it is equally important to show compassion when a mistake is accepted.\n\n\n\n ", "377908": "I thought this was the Airbus challenge, not the Titanic one?", "377405": "Sounds like a good and fair approach to the problem at hand. And once the new data is released, for people a good change to see how much their current network was actually overfitting ;)\n\nSo thanks for coming up with this solution so quickly and really sorry to hear about your personal loss.", "380089": "I appreciate the honesty and the reasonable actions that have been taken so far. \nThanks inversion. :)", "382162": "Any news on when the new test data will be available for download?", "377620": "Good members of staff are the ones that take good actions in a good timely manner. You've shown a good example.", "380042": "Do you have an estimated date for the new test set release?", "379108": "I came to join this competition and found @Andres Torrubia's leak post.\n\n@Inversion, thanks for taking responsibility and I am sorry for the loss in your family.\n", "377652": "Do the images also have timestamps? It might be possible to track ships, even if the images are not overlapping. You might consider checking the timestamps as well. Or just release the space-time stamps and add the additional twist of tracking as part of the competition. The other reason its important to know space-time info is to make cross-validation easier and more robust. Whether there are ships in an image or not, I don't want images from nearby locations taken at the same or similar timestamp on different folds. ", "377543": "Don't sweat it too much - we all make mistakes. It's the willingness to learn from them that matters.", "390287": "I am sorry for your loss, Inversion and for your family. Hang in there. This will still be a nice competition when reset. Kaggle is still a great place to learn and get feedback. Thank you for your work.", "379447": "After all.. we are humans ..!