![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3138959%2Fdcce793bf7e28e179a1b9028dab6715b%2FScreen%20Shot%202019-09-09%20at%2008.44.39.png?generation=1568011541396346&alt=media\")
After looking at all the pictures in the training set, I found that each picture had a portion of the black area. How these black areas are generated during data collection?
", "messages": [ { "id": "621854", "postDate": "09/09/2019 03:00:17", "content": "After looking at all the pictures in the training set, I found that each picture had a portion of the black area. How these black areas are generated during data collection?
", "rawMarkdown": "After looking at all the pictures in the training set, I found that each picture had a portion of the black area. How these black areas are generated during data collection?", "votes": null }, { "id": "621890", "postDate": "09/09/2019 04:31:51", "content": "From competition data description:\nDue to the small footprint of the imager (MODIS) on board these satellites, an image might be stitched together from two orbits. The remaining area, which has not been covered by two succeeding orbits, is marked black.
Hope this answers your question!
", "rawMarkdown": "From competition data description:\n`Due to the small footprint of the imager (MODIS) on board these satellites, an image might be stitched together from two orbits. The remaining area, which has not been covered by two succeeding orbits, is marked black.`\n\nHope this answers your question!", "votes": null }, { "id": "621963", "postDate": "09/09/2019 06:02:51", "content": "The satellite orbits in polar orbit at 99 minutes each. The width of the images we are provided are stitched together from two orbits. The black area as stated by Brian Lee is the area not covered because the earth has rotated (99 minutes worth). So we are missing a chuck of land and clouds.
\n\nThe orbit is such that you can get a image without black stripes using two days worth of stitched orbits - but in that length of time the cloud formations will likely have changed a lot.
\n\nWhen you look at the image provided than the right side is 99 minutes older than the left side of the stripe- on the horizontal. I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.
", "rawMarkdown": "The satellite orbits in polar orbit at 99 minutes each. The width of the images we are provided are stitched together from two orbits. The black area as stated by Brian Lee is the area not covered because the earth has rotated (99 minutes worth). So we are missing a chuck of land and clouds.\n\nThe orbit is such that you can get a image without black stripes using two days worth of stitched orbits - but in that length of time the cloud formations will likely have changed a lot.\n\nWhen you look at the image provided than the right side is 99 minutes older than the left side of the stripe- on the horizontal. I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.", "votes": null }, { "id": "621973", "postDate": "09/09/2019 06:11:48", "content": "> PC Jimmmy wrote:\n> \n> I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.
\n\nI’ve determined this information for the 90%+ of images with good stripes with 100% accuracy, and hand labelled the remainder. It hasn’t helped any model whatsoever, or cross validation, or post processing. Score is almost identical across regions. Quite a bit of work for no result.
\n\nThere is an nice animated picture on a nasa blog of the swath and black gaps, but I’ve not been able to re-find it after a few minutes of searching.
", "rawMarkdown": "> **PC Jimmmy wrote:**\n> \n> I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.\n\nI’ve determined this information for the 90%+ of images with good stripes with 100% accuracy, and hand labelled the remainder. It hasn’t helped any model whatsoever, or cross validation, or post processing. Score is almost identical across regions. Quite a bit of work for no result. \n\nThere is an nice animated picture on a nasa blog of the swath and black gaps, but I’ve not been able to re-find it after a few minutes of searching.", "votes": null }, { "id": "622006", "postDate": "09/09/2019 06:49:50", "content": "Thanks to @pcjimmmy, @joonl04 and @robga for jumping in and describing the reason for the black areas very well. I just want to give the visual impression here from NASA Worldview:\n
The orange line shows the times of the satellite's overpass.
\n\nAnd an animation (the helpful part is towards the end): https://svs.gsfc.nasa.gov/3348#
\n\nFeel free to ask further questions,\nHauke
", "rawMarkdown": "Thanks to @pcjimmmy, @joonl04 and @robga for jumping in and describing the reason for the black areas very well. I just want to give the visual impression here from [NASA Worldview](https://worldview.earthdata.nasa.gov):\n\n\nThe orange line shows the times of the satellite's overpass.\n\nAnd an animation (the helpful part is towards the end): https://svs.gsfc.nasa.gov/3348#\n\nFeel free to ask further questions,\nHauke", "votes": null }, { "id": "622024", "postDate": "09/09/2019 07:15:31", "content": "I'm sorry @robga , that this huge effort did not enhance your score. Nevertheless, I want to mention, that your analysis is still of great value towards the understanding of these cloud features.\nDoesn't your analysis infer, that the patterns themselves look similar across the selected regimes? This is nice, because people seemed to have identified the same structural patterns independent of the region. Gravel for example looks everywhere the same than.\nThis is not at all trivial, because factors like wind-speed or sea surface temperature, to name just a few, might influence the structures and cause slight changes in the appearance of the patterns. The spacing for example of Flowers might change. But either this variation is already captured in a single region or your algorithm is good enough to detect them anyway.
", "rawMarkdown": "I'm sorry @robga , that this huge effort did not enhance your score. Nevertheless, I want to mention, that your analysis is still of great value towards the understanding of these cloud features.\nDoesn't your analysis infer, that the patterns themselves look similar across the selected regimes? This is nice, because people seemed to have identified the same structural patterns *independent* of the region. Gravel for example looks everywhere the same than.\nThis is not at all trivial, because factors like wind-speed or sea surface temperature, to name just a few, might influence the structures and cause slight changes in the appearance of the patterns. The spacing for example of Flowers might change. But either this variation is already captured in a single region or your algorithm is good enough to detect them anyway.", "votes": null }, { "id": "622037", "postDate": "09/09/2019 07:49:28", "content": "Thanks！ I think it's a good picture for people of other professional backgrounds to understand it!
", "rawMarkdown": "Thanks！ I think it's a good picture for people of other professional backgrounds to understand it!", "votes": null }, { "id": "622411", "postDate": "09/09/2019 16:09:14", "content": "Nice work robga. Glad you shared. Starting to get frustrated that I cannot make any progress and thinking about putting latitude's back on the todo list.
\n\nOne thing I worried about when deciding not to make the effort was the potential that the model was already smart enough to have incorporated the black strip into its prediction. In the two years I have been playing here at Kaggle I have often been amazed at the power of these tools we are using.
", "rawMarkdown": "Nice work robga. Glad you shared. Starting to get frustrated that I cannot make any progress and thinking about putting latitude's back on the todo list.\n\nOne thing I worried about when deciding not to make the effort was the potential that the model was already smart enough to have incorporated the black strip into its prediction. In the two years I have been playing here at Kaggle I have often been amazed at the power of these tools we are using.", "votes": null } ], "comments": [ { "id": 621890, "author_name": "joonl04", "author_url": "", "post_date": "09/09/2019 04:31:51", "content": "From competition data description:\nDue to the small footprint of the imager (MODIS) on board these satellites, an image might be stitched together from two orbits. The remaining area, which has not been covered by two succeeding orbits, is marked black.
Hope this answers your question!
", "votes": null, "replies": [] }, { "id": 621963, "author_name": "pcjimmmy", "author_url": "", "post_date": "09/09/2019 06:02:51", "content": "The satellite orbits in polar orbit at 99 minutes each. The width of the images we are provided are stitched together from two orbits. The black area as stated by Brian Lee is the area not covered because the earth has rotated (99 minutes worth). So we are missing a chuck of land and clouds.
\n\nThe orbit is such that you can get a image without black stripes using two days worth of stitched orbits - but in that length of time the cloud formations will likely have changed a lot.
\n\nWhen you look at the image provided than the right side is 99 minutes older than the left side of the stripe- on the horizontal. I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.
", "votes": null, "replies": [ { "id": 621973, "author_name": "robga", "author_url": "", "post_date": "09/09/2019 06:11:48", "content": "> PC Jimmmy wrote:\n> \n> I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.
\n\nI’ve determined this information for the 90%+ of images with good stripes with 100% accuracy, and hand labelled the remainder. It hasn’t helped any model whatsoever, or cross validation, or post processing. Score is almost identical across regions. Quite a bit of work for no result.
\n\nThere is an nice animated picture on a nasa blog of the swath and black gaps, but I’ve not been able to re-find it after a few minutes of searching.
", "votes": null, "replies": [] }, { "id": 622024, "author_name": "observingclouds", "author_url": "", "post_date": "09/09/2019 07:15:31", "content": "I'm sorry @robga , that this huge effort did not enhance your score. Nevertheless, I want to mention, that your analysis is still of great value towards the understanding of these cloud features.\nDoesn't your analysis infer, that the patterns themselves look similar across the selected regimes? This is nice, because people seemed to have identified the same structural patterns independent of the region. Gravel for example looks everywhere the same than.\nThis is not at all trivial, because factors like wind-speed or sea surface temperature, to name just a few, might influence the structures and cause slight changes in the appearance of the patterns. The spacing for example of Flowers might change. But either this variation is already captured in a single region or your algorithm is good enough to detect them anyway.
", "votes": null, "replies": [] }, { "id": 622411, "author_name": "pcjimmmy", "author_url": "", "post_date": "09/09/2019 16:09:14", "content": "Nice work robga. Glad you shared. Starting to get frustrated that I cannot make any progress and thinking about putting latitude's back on the todo list.
\n\nOne thing I worried about when deciding not to make the effort was the potential that the model was already smart enough to have incorporated the black strip into its prediction. In the two years I have been playing here at Kaggle I have often been amazed at the power of these tools we are using.
", "votes": null, "replies": [] } ] }, { "id": 622006, "author_name": "observingclouds", "author_url": "", "post_date": "09/09/2019 06:49:50", "content": "Thanks to @pcjimmmy, @joonl04 and @robga for jumping in and describing the reason for the black areas very well. I just want to give the visual impression here from NASA Worldview:\n
The orange line shows the times of the satellite's overpass.
\n\nAnd an animation (the helpful part is towards the end): https://svs.gsfc.nasa.gov/3348#
\n\nFeel free to ask further questions,\nHauke
", "votes": null, "replies": [ { "id": 622037, "author_name": "dongleeovo", "author_url": "", "post_date": "09/09/2019 07:49:28", "content": "Thanks！ I think it's a good picture for people of other professional backgrounds to understand it!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "621854": "After looking at all the pictures in the training set, I found that each picture had a portion of the black area. How these black areas are generated during data collection?", "621890": "From competition data description:\n`Due to the small footprint of the imager (MODIS) on board these satellites, an image might be stitched together from two orbits. The remaining area, which has not been covered by two succeeding orbits, is marked black.`\n\nHope this answers your question!", "621963": "The satellite orbits in polar orbit at 99 minutes each. The width of the images we are provided are stitched together from two orbits. The black area as stated by Brian Lee is the area not covered because the earth has rotated (99 minutes worth). So we are missing a chuck of land and clouds.\n\nThe orbit is such that you can get a image without black stripes using two days worth of stitched orbits - but in that length of time the cloud formations will likely have changed a lot.\n\nWhen you look at the image provided than the right side is 99 minutes older than the left side of the stripe- on the horizontal. I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.", "621973": "> **PC Jimmmy wrote:**\n> \n> I assume that you could spend some time and determine latitude of the image by measurements of the width of the black line at top vs bottom and its slant direction. That might provide a tiny bit of information since the reference paper suggests different percentages of the shapes at different locations and latitudes.\n\nI’ve determined this information for the 90%+ of images with good stripes with 100% accuracy, and hand labelled the remainder. It hasn’t helped any model whatsoever, or cross validation, or post processing. Score is almost identical across regions. Quite a bit of work for no result. \n\nThere is an nice animated picture on a nasa blog of the swath and black gaps, but I’ve not been able to re-find it after a few minutes of searching.", "622006": "Thanks to @pcjimmmy, @joonl04 and @robga for jumping in and describing the reason for the black areas very well. I just want to give the visual impression here from [NASA Worldview](https://worldview.earthdata.nasa.gov):\n\n\nThe orange line shows the times of the satellite's overpass.\n\nAnd an animation (the helpful part is towards the end): https://svs.gsfc.nasa.gov/3348#\n\nFeel free to ask further questions,\nHauke", "622024": "I'm sorry @robga , that this huge effort did not enhance your score. Nevertheless, I want to mention, that your analysis is still of great value towards the understanding of these cloud features.\nDoesn't your analysis infer, that the patterns themselves look similar across the selected regimes? This is nice, because people seemed to have identified the same structural patterns *independent* of the region. Gravel for example looks everywhere the same than.\nThis is not at all trivial, because factors like wind-speed or sea surface temperature, to name just a few, might influence the structures and cause slight changes in the appearance of the patterns. The spacing for example of Flowers might change. But either this variation is already captured in a single region or your algorithm is good enough to detect them anyway.", "622037": "Thanks！ I think it's a good picture for people of other professional backgrounds to understand it!", "622411": "Nice work robga. Glad you shared. Starting to get frustrated that I cannot make any progress and thinking about putting latitude's back on the todo list.\n\nOne thing I worried about when deciding not to make the effort was the potential that the model was already smart enough to have incorporated the black strip into its prediction. In the two years I have been playing here at Kaggle I have often been amazed at the power of these tools we are using." }, "source": "meta" }