As was pointed out in different threads, about 1% of the images have some data loss. This issue was introduced during the processing of the images for the competition.
\n\nWe are working on updating the data (both on the data page and in kernels) to include the uncorrupted files, and anticipate the new files to be ready this week. (We want to make sure we understand exactly how the problem was introduced, and ensure that all the files are, indeed, correct before the re-upload.)
\n\nI will use this thread to communicate when the updated files have been uploaded, but wanted to keep everyone posted in the meantime.
\n\nThanks to those who reported the issue and did investigation work. Much appreciated!
", "messages": [ { "id": "369969", "postDate": "08/14/2018 02:56:01", "content": "As was pointed out in different threads, about 1% of the images have some data loss. This issue was introduced during the processing of the images for the competition.
\n\nWe are working on updating the data (both on the data page and in kernels) to include the uncorrupted files, and anticipate the new files to be ready this week. (We want to make sure we understand exactly how the problem was introduced, and ensure that all the files are, indeed, correct before the re-upload.)
\n\nI will use this thread to communicate when the updated files have been uploaded, but wanted to keep everyone posted in the meantime.
\n\nThanks to those who reported the issue and did investigation work. Much appreciated!
", "rawMarkdown": "As was pointed out in [different][1] [threads][2], about 1% of the images have some data loss. This issue was introduced during the processing of the images for the competition.\n\nWe are working on updating the data (both on the data page and in kernels) to include the uncorrupted files, and anticipate the new files to be ready this week. (We want to make sure we understand exactly how the problem was introduced, and ensure that all the files are, indeed, correct before the re-upload.)\n\nI will use this thread to communicate when the updated files have been uploaded, but wanted to keep everyone posted in the meantime.\n\nThanks to those who reported the issue and did investigation work. Much appreciated!\n\n\n [1]: https://www.kaggle.com/c/airbus-ship-detection/discussion/62574\n [2]: https://www.kaggle.com/c/airbus-ship-detection/discussion/62921", "votes": null }, { "id": "371445", "postDate": "08/16/2018 20:05:03", "content": "We've rinsed through the images again. There are two types of problems:
\n\n1) Unreadable Images - There is only a single image in this category, and as was pointed earlier, the image is: 6384c3e78.jpg. This images was corrupted in the transfer process, and should just be ignored. (In other words, there are no plans to re-upload a new dataset with this image removed.)
2) Images with blackout regions - There are a larger number (but overall small percentage) of images that have strips that are blanked out. These issues are contained in the source dataset (for various reasons), and should be considered part of the challenge. It is up to individual teams what is the best way to handle these images.
", "rawMarkdown": "We've rinsed through the images again. There are two types of problems:\n\n1) **Unreadable Images** - There is only a single image in this category, and as was pointed earlier, the image is: `6384c3e78.jpg`. This images was corrupted in the transfer process, and should just be ignored. (In other words, there are no plans to re-upload a new dataset with this image removed.)\n\n2) **Images with blackout regions** - There are a larger number (but overall small percentage) of images that have strips that are blanked out. These issues are contained in the source dataset (for various reasons), and should be considered part of the challenge. It is up to individual teams what is the best way to handle these images.", "votes": null }, { "id": "372652", "postDate": "08/20/2018 01:56:22", "content": "Firstly, thanks for providing the dataset and organizing this competition. It's very inspiring and an awesome learning resource.
\n\nHowever, I must say that this kind of behavior seems to often repeat itself in kaggle competitions. Instead of fixing the dataset, kaggle force thousands of participants to run the same cleanup code, over and over again. How many cpu-hours and development time is going to be wasted across all the teams?
\n\nYou guys could open all images and search for rows/columns with std == 0 or follow the list provided here already: https://www.kaggle.com/c/airbus-ship-detection/discussion/62921
\n\nYou may decide to stick to your decision to not change the data, but I believe many would be very glad if more attention is paid to the data for upcoming challenges. Thanks so much anyway.
", "rawMarkdown": "Firstly, thanks for providing the dataset and organizing this competition. It's very inspiring and an awesome learning resource.\n\nHowever, I must say that this kind of behavior seems to often repeat itself in kaggle competitions. Instead of fixing the dataset, kaggle force thousands of participants to run the same cleanup code, over and over again. How many cpu-hours and development time is going to be wasted across all the teams?\n\nYou guys could open all images and search for rows/columns with std == 0 or follow the list provided here already: https://www.kaggle.com/c/airbus-ship-detection/discussion/62921\n\nYou may decide to stick to your decision to not change the data, but I believe many would be very glad if more attention is paid to the data for upcoming challenges. Thanks so much anyway.", "votes": null } ], "comments": [ { "id": 371445, "author_name": "inversion", "author_url": "", "post_date": "08/16/2018 20:05:03", "content": "We've rinsed through the images again. There are two types of problems:
\n\n1) Unreadable Images - There is only a single image in this category, and as was pointed earlier, the image is: 6384c3e78.jpg. This images was corrupted in the transfer process, and should just be ignored. (In other words, there are no plans to re-upload a new dataset with this image removed.)
2) Images with blackout regions - There are a larger number (but overall small percentage) of images that have strips that are blanked out. These issues are contained in the source dataset (for various reasons), and should be considered part of the challenge. It is up to individual teams what is the best way to handle these images.
", "votes": null, "replies": [ { "id": 372652, "author_name": "hmendonca", "author_url": "", "post_date": "08/20/2018 01:56:22", "content": "Firstly, thanks for providing the dataset and organizing this competition. It's very inspiring and an awesome learning resource.
\n\nHowever, I must say that this kind of behavior seems to often repeat itself in kaggle competitions. Instead of fixing the dataset, kaggle force thousands of participants to run the same cleanup code, over and over again. How many cpu-hours and development time is going to be wasted across all the teams?
\n\nYou guys could open all images and search for rows/columns with std == 0 or follow the list provided here already: https://www.kaggle.com/c/airbus-ship-detection/discussion/62921
\n\nYou may decide to stick to your decision to not change the data, but I believe many would be very glad if more attention is paid to the data for upcoming challenges. Thanks so much anyway.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "369969": "As was pointed out in [different][1] [threads][2], about 1% of the images have some data loss. This issue was introduced during the processing of the images for the competition.\n\nWe are working on updating the data (both on the data page and in kernels) to include the uncorrupted files, and anticipate the new files to be ready this week. (We want to make sure we understand exactly how the problem was introduced, and ensure that all the files are, indeed, correct before the re-upload.)\n\nI will use this thread to communicate when the updated files have been uploaded, but wanted to keep everyone posted in the meantime.\n\nThanks to those who reported the issue and did investigation work. Much appreciated!\n\n\n [1]: https://www.kaggle.com/c/airbus-ship-detection/discussion/62574\n [2]: https://www.kaggle.com/c/airbus-ship-detection/discussion/62921", "371445": "We've rinsed through the images again. There are two types of problems:\n\n1) **Unreadable Images** - There is only a single image in this category, and as was pointed earlier, the image is: `6384c3e78.jpg`. This images was corrupted in the transfer process, and should just be ignored. (In other words, there are no plans to re-upload a new dataset with this image removed.)\n\n2) **Images with blackout regions** - There are a larger number (but overall small percentage) of images that have strips that are blanked out. These issues are contained in the source dataset (for various reasons), and should be considered part of the challenge. It is up to individual teams what is the best way to handle these images.", "372652": "Firstly, thanks for providing the dataset and organizing this competition. It's very inspiring and an awesome learning resource.\n\nHowever, I must say that this kind of behavior seems to often repeat itself in kaggle competitions. Instead of fixing the dataset, kaggle force thousands of participants to run the same cleanup code, over and over again. How many cpu-hours and development time is going to be wasted across all the teams?\n\nYou guys could open all images and search for rows/columns with std == 0 or follow the list provided here already: https://www.kaggle.com/c/airbus-ship-detection/discussion/62921\n\nYou may decide to stick to your decision to not change the data, but I believe many would be very glad if more attention is paid to the data for upcoming challenges. Thanks so much anyway." }, "source": "meta" }