Now that the competition entered the final phase, maybe sharing some hints would make it more interesting for everyone. I'll start with one: \nAll images with clouds on them are No3 in order. You can manually label them, or if you don't like manual labeling you can count white pixels over some "whiteness" threshold, the image which have them drastically more then other four is No3.
", "messages": [ { "id": "124696", "postDate": "06/21/2016 11:08:17", "content": "Now that the competition entered the final phase, maybe sharing some hints would make it more interesting for everyone. I'll start with one: \nAll images with clouds on them are No3 in order. You can manually label them, or if you don't like manual labeling you can count white pixels over some "whiteness" threshold, the image which have them drastically more then other four is No3.
", "rawMarkdown": "Now that the competition entered the final phase, maybe sharing some hints would make it more interesting for everyone. I'll start with one: \r\nAll images with clouds on them are No3 in order. You can manually label them, or if you don't like manual labeling you can count white pixels over some \"whiteness\" threshold, the image which have them drastically more then other four is No3.", "votes": null }, { "id": "124702", "postDate": "06/21/2016 12:51:47", "content": "I found it a bit puzzling that some intensities in one image were not present at all, but in the next or previous. "Present in current, but not in previous" seems to yield a blue shift and some interesting psychedelic images...
", "rawMarkdown": "I found it a bit puzzling that some intensities in one image were not present at all, but in the next or previous. \"Present in current, but not in previous\" seems to yield a blue shift and some interesting psychedelic images...", "votes": null }, { "id": "124723", "postDate": "06/21/2016 17:40:38", "content": "The fourth day is a Sunday, it was cloudy on Saturday..!
\n\nThis is fun, reminds me of those logic puzzles from which you can eventually conclude that the butler did it in the pantry with the carving knife!
", "rawMarkdown": "The fourth day is a Sunday, it was cloudy on Saturday..!\r\n\r\nThis is fun, reminds me of those logic puzzles from which you can eventually conclude that the butler did it in the pantry with the carving knife!", "votes": null }, { "id": "124730", "postDate": "06/21/2016 19:33:48", "content": "Please don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.
\n\nJust my two cents, opinions may differ.
\n\n[I'll happily share all my secrets after the contest closes. :^)]
\n\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.
\n\n-Rich
", "rawMarkdown": "Please don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.\r\n\r\nJust my two cents, opinions may differ.\r\n\r\n[I'll happily share all my secrets after the contest closes. :^)]\r\n\r\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.\r\n\r\n-Rich", "votes": null }, { "id": "124736", "postDate": "06/21/2016 21:06:58", "content": "[quote=quadcore;124730]
\n\nPlease don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.
\n\nJust my two cents, opinions may differ.
\n\n[I'll happily share all my secrets after the contest closes. :^)]
\n\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.
\n\n-Rich
\n\n[/quote]
\n\nAbsolutely agree...!
\n\nBut now I'm concerned that people may think there's any useful information in my previous post.
\n\nGuess it's time for a disclaimer...
\n\nThe fourth day is a Sunday does not imply that every Sunday is the fourth day or that every fourth day is a Sunday. Other days of the week may, or may not, appear on the fourth day or any other day come to mention it.
\n\nit was cloudy on Saturday does not imply that the sun didn't shine on Saturday or preclude other clouds from appearing on other days of the week.
\n\nAnd for good measure... Saturday may occur after Sunday, if Sunday preceeded Saturday.
\n\nWell I think I've done a good job of clearing that up. Now where's that butler and the carving knife?
", "rawMarkdown": "[quote=quadcore;124730]\r\n\r\nPlease don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.\r\n\r\nJust my two cents, opinions may differ.\r\n\r\n[I'll happily share all my secrets after the contest closes. :^)]\r\n\r\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.\r\n\r\n-Rich\r\n\r\n[/quote]\r\n\r\nAbsolutely agree...! \r\n\r\nBut now I'm concerned that people may think there's any useful information in my previous post. \r\n\r\nGuess it's time for a disclaimer...\r\n\r\n***The fourth day is a Sunday*** does not imply that every Sunday is the fourth day or that every fourth day is a Sunday. Other days of the week may, or may not, appear on the fourth day or any other day come to mention it. \r\n\r\n***it was cloudy on Saturday*** does not imply that the sun didn't shine on Saturday or preclude other clouds from appearing on other days of the week. \r\n\r\nAnd for good measure... Saturday may occur after Sunday, if Sunday preceeded Saturday.\r\n\r\nWell I think I've done a good job of clearing that up. Now where's that butler and the carving knife?", "votes": null }, { "id": "124749", "postDate": "06/21/2016 22:53:09", "content": "[quote=Chippy;124723]
\n\nthe butler did it in the pantry with the carving knife!
\n\n[/quote]
\n\nThe professor did it in the orthogonal vector space with a support vector machine.
", "rawMarkdown": "[quote=Chippy;124723]\r\n\r\nthe butler did it in the pantry with the carving knife!\r\n\r\n[/quote]\r\n\r\n\r\nThe professor did it in the orthogonal vector space with a support vector machine.", "votes": null }, { "id": "125214", "postDate": "06/27/2016 17:22:32", "content": "I found that it's possible to predict correct order using "puddles" which is in process of drying day by day. There are pretty many tests which have puddles. )
", "rawMarkdown": "I found that it's possible to predict correct order using \"puddles\" which is in process of drying day by day. There are pretty many tests which have puddles. )", "votes": null }, { "id": "125254", "postDate": "06/28/2016 05:47:03", "content": "Congratulations to winners!
\n\nIt's been a tough challenge. This is how I tried doing. It involved more human effort than ML.
\n\nFirst, georeferenced all the images in a set.
\n\nUsually, when they take aerial photographs, it involves "flight planning". From the KML shared on this forum, I could make out that there are "group" of photos which lie in a line. All photos taken on a particular day will have a similar orientation. I did grouping of the jumbled photographs based on the orientation. Each group would represent all photos (from different sets) which are taken on a particular day.
\n\nNext, I did the analysis of shadows. I noticed from training data that shadow of any object followed anti-clockwise pattern. This helped to figure out which photo was taken on Day-1 and which is on Day-5.
\n\nThere were, of course, conflicts between set of photographs which had similar orientation or overlapping shadows. The only way to improve the score was to swap them and try.
\n\nAll these involved more human effort. So I gave up in the middle :(
", "rawMarkdown": "Congratulations to winners!\r\n\r\nIt's been a tough challenge. This is how I tried doing. It involved more human effort than ML.\r\n\r\nFirst, georeferenced all the images in a set.\r\n\r\nUsually, when they take aerial photographs, it involves \"flight planning\". From the KML shared on this forum, I could make out that there are \"group\" of photos which lie in a line. All photos taken on a particular day will have a similar orientation. I did grouping of the jumbled photographs based on the orientation. Each group would represent all photos (from different sets) which are taken on a particular day.\r\n\r\nNext, I did the analysis of shadows. I noticed from training data that shadow of any object followed anti-clockwise pattern. This helped to figure out which photo was taken on Day-1 and which is on Day-5.\r\n\r\nThere were, of course, conflicts between set of photographs which had similar orientation or overlapping shadows. The only way to improve the score was to swap them and try.\r\n\r\nAll these involved more human effort. So I gave up in the middle :(", "votes": null }, { "id": "125279", "postDate": "06/28/2016 11:35:56", "content": "Congrats to the three winners, all three achieved 1.0 score, deserved.
\n\nNow I suppose it's OK to reveal some stuff: I did a ton of manual labeling, and all that effort went thrown away since my best result was achieved with simplest computer vision based method:
\n\nThis simple solution got me in the top 10% (37th) in private leaderbord, with 220+ positions jump from public to private leaderbord. It's disturbing that all those weeks of feature engineering gave me nothing, but I've learned so much from this competition that I can't complain.
\n\nCan't wait to see the "secret sauce" of the winning solutions.
", "rawMarkdown": "Congrats to the three winners, all three achieved 1.0 score, deserved.\r\n\r\nNow I suppose it's OK to reveal some stuff: I did a ton of manual labeling, and all that effort went thrown away since my best result was achieved with simplest computer vision based method:\r\n\r\n- Compose all five images together\r\n- Calculate relative position and size of each of the five images regarding the composition\r\n- Image with most rotation regarding the average rotation of all five images is ordered No3\r\n- two images with smallest average size regarding the composition are order No1 & No2 in random order\r\n- remaining two images (in random order) are ordered No4 & No5\r\n\r\nThis simple solution got me in the top 10% (37th) in private leaderbord, with 220+ positions jump from public to private leaderbord. It's disturbing that all those weeks of feature engineering gave me nothing, but I've learned so much from this competition that I can't complain.\r\n\r\nCan't wait to see the \"secret sauce\" of the winning solutions.", "votes": null }, { "id": "125285", "postDate": "06/28/2016 12:39:17", "content": "Here's a quick outline of our approach which almost got us a perfect score. It sounds similar to that of others that accepted an approach that included handlabelling.
\n\nWe split the task into four parts
\n\nHandlabelling featured in all parts of the above but we found python and R scripts helped greatly with the image registration and identifying cases where humans had made mistakes in steps 2 and 3.
\n\nParticular learnings included
\n\nWhen it came to ordering days, we could usually find a number of image sets in each cluster that involved some sort of construction which physically progressed in each day, hence giving away the image order. We found that cars in carparks, puddles, containers and rubbish in sorting centres etc were generally less reliable.
\n\nWhat made this competition "easier" was the fact that relative few distinct flights were made and so images overlapped, spatially and temporally. This meant it was possible to register images across sets to to the same days. Once you had determined the order of one set you then knew the order of all images in associated sets.
", "rawMarkdown": "Here's a quick outline of our approach which almost got us a perfect score. It sounds similar to that of others that accepted an approach that included handlabelling.\r\n\r\nWe split the task into four parts\r\n\r\n 1. Geocode images\r\n 2. Identify common days\r\n 3. Order days\r\n 4. Manually peer review and correct submission\r\n\r\nHandlabelling featured in all parts of the above but we found python and R scripts helped greatly with the image registration and identifying cases where humans had made mistakes in steps 2 and 3.\r\n\r\nParticular learnings included\r\n\r\n - we used shadow angles and intensities for our first pass of allocating images to common days\r\n - this worked particularly well for clusters where flights were at distinct and well separated times of day\r\n - where images in a set shared shadow angle and intensity we sometimes could differentiate and allocate to common days by comparing shadow lengths\r\n - for the really difficult images (ie 2 images from each set showing scrubland north of the landfill) where shadow angles and lengths could not be determined we found that images could be allocated to common days according to the rotation of the image itself relative to true north.\r\n\r\nWhen it came to ordering days, we could usually find a number of image sets in each cluster that involved some sort of construction which physically progressed in each day, hence giving away the image order. We found that cars in carparks, puddles, containers and rubbish in sorting centres etc were generally less reliable.\r\n\r\nWhat made this competition \"easier\" was the fact that relative few distinct flights were made and so images overlapped, spatially and temporally. This meant it was possible to register images across sets to to the same days. Once you had determined the order of one set you then knew the order of all images in associated sets.", "votes": null }, { "id": "125311", "postDate": "06/28/2016 15:04:24", "content": "For sets that overlapped, the easiest way to match up two sets was looking at parked cars (assuming there were any). Parked cars usually varied day to day, so if two images from different sets had cars in the same places, they were from the same day.
\n\nSchools typically had empty parking lots on Saturdays and Sundays (but watch out for sports activities).
\n\nChurches tended to have full parking lots on Sundays, nearly empty lots other days.
\n\nGoogle Maps was useful to figure out what some buildings were (school, church, etc).
\n\nI think Saturday was the third day in all sets and I think the images were five days in a row (although I don't know if that is proven). As mentioned earlier, at least in some areas, it rained Saturday, with a few puddles left on Sunday.
\n\nSets with construction / dirt moving activity that helped figure out the order:
\n\n22 and 123 - Construction activity
\n\n236 - Landfill activity
\n\n335 - Pipes being added
\n\n52 - farm being plowed, path being built
\n\n172 - some activity at a house, things being moved around
\n\n148 - construction activity
\n\n326 - dirt piles being added
\n\n189 - puddles
\n\n212 - puddles
\n\n156 - some house construction
\n\n230 - road construction lower left hand corner
\n\n14 - garbage processing - changes every day, but difficult to know the order
\n\n259 - upper right corner, activity with trucks and dirt
\n\n121 - baseball field grooming
\n\n177 - house construction, lower center
\n\n145 - truck/construction movement
\n\n11 - puddles. construction somewhere (cannot find now)
\n\n263 - house construction (lower left corner)
\n\n80 - roads being paved
\n\n141 - roof has white dots added (right side of image)
\n\n344 - pipes being placed in dirt lot/new road
\n\n119 - Farmer's market (only open Sunday)
\n\n278 - truck/movement at stadium
\n\n341 - house construction (center)
\n\n81 - some trucks/dirt motion (left side, center)
\n\n215 - puddles
\n\n149 - roof being finished (center) - only helped with first day
\n\n136 - puddles
\n\n-Rich
", "rawMarkdown": "For sets that overlapped, the easiest way to match up two sets was looking at parked cars (assuming there were any). Parked cars usually varied day to day, so if two images from different sets had cars in the same places, they were from the same day.\r\n\r\nSchools typically had empty parking lots on Saturdays and Sundays (but watch out for sports activities).\r\n\r\nChurches tended to have full parking lots on Sundays, nearly empty lots other days.\r\n\r\nGoogle Maps was useful to figure out what some buildings were (school, church, etc).\r\n\r\nI think Saturday was the third day in all sets and I think the images were five days in a row (although I don't know if that is proven). As mentioned earlier, at least in some areas, it rained Saturday, with a few puddles left on Sunday.\r\n\r\nSets with construction / dirt moving activity that helped figure out the order:\r\n\r\n22 and 123 - Construction activity\r\n\r\n236 - Landfill activity\r\n\r\n335 - Pipes being added\r\n\r\n52 - farm being plowed, path being built\r\n\r\n172 - some activity at a house, things being moved around\r\n\r\n148 - construction activity\r\n\r\n326 - dirt piles being added\r\n\r\n189 - puddles\r\n\r\n212 - puddles\r\n\r\n156 - some house construction\r\n\r\n230 - road construction lower left hand corner\r\n\r\n14 - garbage processing - changes every day, but difficult to know the order\r\n\r\n259 - upper right corner, activity with trucks and dirt\r\n\r\n121 - baseball field grooming\r\n\r\n177 - house construction, lower center\r\n\r\n145 - truck/construction movement\r\n\r\n11 - puddles. construction somewhere (cannot find now)\r\n\r\n263 - house construction (lower left corner)\r\n\r\n80 - roads being paved\r\n\r\n141 - roof has white dots added (right side of image)\r\n\r\n344 - pipes being placed in dirt lot/new road\r\n\r\n119 - Farmer's market (only open Sunday)\r\n\r\n278 - truck/movement at stadium\r\n\r\n341 - house construction (center)\r\n\r\n81 - some trucks/dirt motion (left side, center)\r\n\r\n215 - puddles\r\n\r\n149 - roof being finished (center) - only helped with first day\r\n\r\n136 - puddles\r\n\r\n-Rich", "votes": null }, { "id": "125319", "postDate": "06/28/2016 16:05:25", "content": "I initially started hand labelling to get a better understanding on how a machine learning approach might work but then got a bit hooked and didn’t really do much with ML.
\n\nMy approach was similar to those above, but obviously not as successful :)
\n\nGeoreferenced images (thanks for the starting point @kes367):
\n\nFind ‘reference’ sets to determine overall order. I like to think of these as areas whose rate of change is similar to or less frequent than the interval between flights. Good candidates were:
\n\nI also used shadow length in some cases where the shadow angles were indistinguishable. (I was surprised how much shadow length / sun elevation does change even day to day. I now wonder if one could determine true north, estimate time of day from shadow angle and then calculate sun elevation from the shadow length just how far one could get on that alone.)
\n\nFor most of the time I did not assume that the images were taken on consecutive days or that the different geographical areas (within San Diego) were photographed on the same days but am much more convinced of this now.
\n\nI knew that in some cases the way that I used shadow angle was likely to have some errors in assignment of images to flights but ran out of time go back and look for/correct flight path assignment. (I assume I could have made more use of overlapping images as one way to check.)
\n\nOne interesting example of traffic movement that I did not get to use was comparing 2_3 to 183_5. It looks rather like a vehicle has moved into an intersection between these two images. There were similar patterns on highways too from which I think one can determine whether some flights were North to South or South to North. I suspect it might be possible to use this kind of information and changes in time of day estimates to reconstruct the overall sequence/path of each flight.
", "rawMarkdown": "I initially started hand labelling to get a better understanding on how a machine learning approach might work but then got a bit hooked and didn’t really do much with ML.\r\n\r\nMy approach was similar to those above, but obviously not as successful :) \r\n\r\n Georeferenced images (thanks for the starting point @kes367):\r\n\r\n - Came to conclusion that images were taken on a series of North-South flights\r\n - Assign images to flight passes\r\n - Observed consistent image angles for some sets of images\r\n - Observed consistent shadow angles (or lack or shadows) for some sets\r\n - I generally did not use overlapping image features but when I did car parks or car parking were the most helpful \r\n\r\nFind ‘reference’ sets to determine overall order. I like to think of these as areas whose rate of change is similar to or less frequent than the interval between flights. Good candidates were:\r\n\r\n - Building sites\r\n - Scrap yards (gradual dismantling of vehicles)\r\n - Baseball fields (raking of diamond)\r\n - School/church car parks/School sports for weekend patterns\r\n - There was one baseball field with a new scoreboard being installed.\r\n - Use traffic movement in some cases of overlapping images\r\n\r\nI also used shadow length in some cases where the shadow angles were indistinguishable. (I was surprised how much shadow length / sun elevation does change even day to day. I now wonder if one could determine true north, estimate time of day from shadow angle and then calculate sun elevation from the shadow length just how far one could get on that alone.)\r\n\r\nFor most of the time I did not assume that the images were taken on consecutive days or that the different geographical areas (within San Diego) were photographed on the same days but am much more convinced of this now. \r\n\r\nI knew that in some cases the way that I used shadow angle was likely to have some errors in assignment of images to flights but ran out of time go back and look for/correct flight path assignment. (I assume I could have made more use of overlapping images as one way to check.) \r\n\r\nOne interesting example of traffic movement that I did not get to use was comparing 2_3 to 183_5. It looks rather like a vehicle has moved into an intersection between these two images. There were similar patterns on highways too from which I think one can determine whether some flights were North to South or South to North. I suspect it might be possible to use this kind of information and changes in time of day estimates to reconstruct the overall sequence/path of each flight.", "votes": null }, { "id": "126701", "postDate": "07/11/2016 22:26:36", "content": "I found my mistake. I had a typo in set 225. The effect of said typo was a -1.6 difference in spearman r correlation for what I actually entered for set 225 which when divided by 139 results in a private leaderboard delta of 0.01151. That difference meant that I went from 6th on the public leaderboard to 7th on the private; 0.98849 on the private leaderboard instead of 1.0000. So here's my first hint. Beware of typos in a labeling competition... I suppose this is where I should be grateful to all of the people who were faster than me; even without the typo I wouldn't have been in the money. (I imagine 4th to be the most uncomfortable position in many Kaggle competitions.)
\n\nHere's my next hint. There were 28 sets in the public leaderboard and 139 in the private. I never did managed to order a set manually. I did all of my ordering via the public leaderboard. The sets in the public leaderboard are
\n\nThe possible values for Spearman's correlation coefficient between two orderings of 1,2,3,4,5 are
\n\n[1 - .1*x for x in range(21)] = [1.0, 0.9, 0.8, ..., -.8, -.9, -1.0]\nIf you know that a set is in the public leaderboard, you can find it's ordering with two reasonably chosen entries. While I did have the dubious distinction of the most entries of any team in the competition, that is still not enough entries to isolate the leaderboard individually. However, as mentioned in many places [see the winner's post], many of the pictures overlap. So by finding groupings of pictures that were taken on the same pass of the same flight, you can triangulate the group if there is at least one member of the group in the public leaderboard using a combination of linear programming and logic. For example
\n\nFlight: 53, Tree like branching rds south of Spring Canyon rd through dome bldg to gravel pit setIds: [284, 231, 88, 208, 333, 48, 233, 138, 172, 23]\n Day set284 set231 set88 set208 set333 set48 set233 set138 set172 set23\n2 c 3 2 1 2 3 1 3 1 3 4\n4 e 5 4 2 4 4 4 5 3 5 3\n0 a 1 5 3 5 5 3 4 5 4 1\n3 d 4 1 5 3 1 5 2 4 1 2\n1 b 2 3 4 1 2 2 1 2 2 5\nis one of the groupings through Scripps Ranch with pictures that are adjacent and overlapping and it happens to contain set 208 from the public leaderboard. By entering groups of approximately 10 permutation matched sets at at time, I was able to use the public leaderboard to find the correct ordering for most groupings.
\n\nI did not match all of these by hand. This leads to my next hint: template matching. Take two sets that overlap and for each image in the first crop out a reasonably sized part of the overlap and use that as a template to match with each of the 5 images in the second set. For each pairing let the score be the maximum correlation coefficient. For example looking at the scores from matching set333 to set48 using a crop window of (1000,0,2000,500)
set333_1, set48_1 0.38268 \nset333_1, set48_2 0.40771 \nset333_1, set48_3 0.32304 \nset333_1, set48_4 0.42695 \nset333_1, set48_5 0.96155 \n\nset333_2, set48_1 0.28035 \nset333_2, set48_2 0.70019 \nset333_2, set48_3 0.30732 \nset333_2, set48_4 0.32623 \nset333_2, set48_5 0.31640 \n\nset333_3, set48_1 0.73081 \nset333_3, set48_2 0.51668 \nset333_3, set48_3 0.43541 \nset333_3, set48_4 0.45769 \nset333_3, set48_5 0.50292 \n\nset333_4, set48_1 0.19761 \nset333_4, set48_2 0.19939 \nset333_4, set48_3 0.17685 \nset333_4, set48_4 0.60496 \nset333_4, set48_5 0.21628 \n\nset333_5, set48_1 0.38741 \nset333_5, set48_2 0.49584 \nset333_5, set48_3 0.96844 \nset333_5, set48_4 0.46904 \nset333_5, set48_5 0.47248 \nwe see that set333_1 and set48_5 were probably taken on the same pass, etc. Template matching does lead to weird discoveries like the fact that even though sets 30 and 145 look like they overlap, they are not taken in the same pass of the airplane.
\n\nTemplate matching breaks down because not all of the pictures taken in the same pass overlap another picture in that pass. Also water exposes a bug/feature in scikit image's match_template code. So as a second method and my next hint is shadows. Shadows were very useful for connecting both sets that didn't overlap other sets and flight groups that didn't have any public leaderboard presence to flight groups that did.
\n\nAs a final hint, beware of rotation matching. It works much better on the training set than the test set.
\n\nBy treating this whole competition as a crazy combination of Where's Waldo and a jig saw puzzle, I didn't actually have to resort to external data although I must admit to using google maps to help confirm which pictures belonged in which area. (It was the picture on the box of the jigsaw puzzle...) In post competition checks I've confirmed that none of the LA pictures were in the private leaderboard. I started this as an exercise in trying to increase the amount of training data to something that would let me build a model... and never did get to building a model.
", "rawMarkdown": "I found my mistake. I had a typo in set 225. The effect of said typo was a -1.6 difference in spearman r correlation for what I actually entered for set 225 which when divided by 139 results in a private leaderboard delta of 0.01151. That difference meant that I went from 6th on the public leaderboard to 7th on the private; 0.98849 on the private leaderboard instead of 1.0000. So here's my first hint. Beware of typos in a labeling competition... I suppose this is where I should be grateful to all of the people who were faster than me; even without the typo I wouldn't have been in the money. (I imagine 4th to be the most uncomfortable position in many Kaggle competitions.)\r\n\r\nHere's my next hint. There were 28 sets in the public leaderboard and 139 in the private. I never did managed to order a set manually. I did all of my ordering via the public leaderboard. The sets in the public leaderboard are \r\n\r\n - 330, 328, 215, 119, 275, 72 from La Jolla\r\n - 13, 259, 179, 38 from Logan Heights\r\n - 83, 145, 39, 234, 195, 7, 266, 240 from the Point Loma/San Diego Bay/Airport/Old Town area\r\n - 1, 111, 208, 254, 187 from Scripps Ranch \r\n - 253, 152, 200, 67, 54 from Sycamore Landfill and scrub east of landfill\r\n\r\nThe possible values for Spearman's correlation coefficient between two orderings of 1,2,3,4,5 are \r\n\r\n [1 - .1*x for x in range(21)] = [1.0, 0.9, 0.8, ..., -.8, -.9, -1.0]\r\n\r\nIf you know that a set is in the public leaderboard, you can find it's ordering with two reasonably chosen entries. While I did have the dubious distinction of the most entries of any team in the competition, that is still not enough entries to isolate the leaderboard individually. However, as mentioned in many places [\\[see the winner's post\\]][1], many of the pictures overlap. So by finding groupings of pictures that were taken on the same pass of the same flight, you can triangulate the group if there is at least one member of the group in the public leaderboard using a combination of linear programming and logic. For example\r\n\r\n Flight: 53, Tree like branching rds south of Spring Canyon rd through dome bldg to gravel pit setIds: [284, 231, 88, 208, 333, 48, 233, 138, 172, 23]\r\n Day set284 set231 set88 set208 set333 set48 set233 set138 set172 set23\r\n 2 c 3 2 1 2 3 1 3 1 3 4\r\n 4 e 5 4 2 4 4 4 5 3 5 3\r\n 0 a 1 5 3 5 5 3 4 5 4 1\r\n 3 d 4 1 5 3 1 5 2 4 1 2\r\n 1 b 2 3 4 1 2 2 1 2 2 5\r\n\r\nis one of the groupings through Scripps Ranch with pictures that are adjacent and overlapping and it happens to contain set 208 from the public leaderboard. By entering groups of approximately 10 permutation matched sets at at time, I was able to use the public leaderboard to find the correct ordering for most groupings.\r\n\r\nI did not match all of these by hand. This leads to my next hint: [template matching][2]. Take two sets that overlap and for each image in the first crop out a reasonably sized part of the overlap and use that as a template to match with each of the 5 images in the second set. For each pairing let the score be the maximum correlation coefficient. For example looking at the scores from matching set333 to set48 using a crop window of `(1000,0,2000,500)`\r\n\r\n set333_1, set48_1 0.38268 \r\n set333_1, set48_2 0.40771 \r\n set333_1, set48_3 0.32304 \r\n set333_1, set48_4 0.42695 \r\n set333_1, set48_5 0.96155 \r\n \r\n set333_2, set48_1 0.28035 \r\n set333_2, set48_2 0.70019 \r\n set333_2, set48_3 0.30732 \r\n set333_2, set48_4 0.32623 \r\n set333_2, set48_5 0.31640 \r\n \r\n set333_3, set48_1 0.73081 \r\n set333_3, set48_2 0.51668 \r\n set333_3, set48_3 0.43541 \r\n set333_3, set48_4 0.45769 \r\n set333_3, set48_5 0.50292 \r\n \r\n set333_4, set48_1 0.19761 \r\n set333_4, set48_2 0.19939 \r\n set333_4, set48_3 0.17685 \r\n set333_4, set48_4 0.60496 \r\n set333_4, set48_5 0.21628 \r\n \r\n set333_5, set48_1 0.38741 \r\n set333_5, set48_2 0.49584 \r\n set333_5, set48_3 0.96844 \r\n set333_5, set48_4 0.46904 \r\n set333_5, set48_5 0.47248 \r\n\r\nwe see that set333_1 and set48_5 were probably taken on the same pass, etc. Template matching does lead to weird discoveries like the fact that even though sets 30 and 145 look like they overlap, they are not taken in the same pass of the airplane. \r\n\r\nTemplate matching breaks down because not all of the pictures taken in the same pass overlap another picture in that pass. Also water exposes a bug/feature in scikit image's match_template code. So as a second method and my next hint is shadows. Shadows were very useful for connecting both sets that didn't overlap other sets and flight groups that didn't have any public leaderboard presence to flight groups that did. \r\n\r\nAs a final hint, beware of rotation matching. It works much better on the training set than the test set.\r\n\r\nBy treating this whole competition as a crazy combination of Where's Waldo and a jig saw puzzle, I didn't actually have to resort to external data although I must admit to using google maps to help confirm which pictures belonged in which area. (It was the picture on the box of the jigsaw puzzle...) In post competition checks I've confirmed that none of the LA pictures were in the private leaderboard. I started this as an exercise in trying to increase the amount of training data to something that would let me build a model... and never did get to building a model.\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/draper-satellite-image-chronology/forums/t/21936/1st-place-how-to-win-the-competition-if-you-know-nothing-about-image-processing/125346#post125346\r\n [2]: http://scikit-image.org/docs/dev/api/skimage.feature.html?highlight=match_template#skimage.feature.match_template", "votes": null }, { "id": "126702", "postDate": "07/11/2016 22:52:34", "content": "What a nice idea to use a public LB to order groups of sets.
\n\nIt turned out that this competition was ritch in very different solutions, and was much more interesting then initially looked like.
", "rawMarkdown": "What a nice idea to use a public LB to order groups of sets. \r\n\r\nIt turned out that this competition was ritch in very different solutions, and was much more interesting then initially looked like.", "votes": null } ], "comments": [ { "id": 124702, "author_name": "glimmung", "author_url": "", "post_date": "06/21/2016 12:51:47", "content": "I found it a bit puzzling that some intensities in one image were not present at all, but in the next or previous. "Present in current, but not in previous" seems to yield a blue shift and some interesting psychedelic images...
", "votes": null, "replies": [] }, { "id": 124723, "author_name": "nigelcarpenter", "author_url": "", "post_date": "06/21/2016 17:40:38", "content": "The fourth day is a Sunday, it was cloudy on Saturday..!
\n\nThis is fun, reminds me of those logic puzzles from which you can eventually conclude that the butler did it in the pantry with the carving knife!
", "votes": null, "replies": [] }, { "id": 124730, "author_name": "richardepstein", "author_url": "", "post_date": "06/21/2016 19:33:48", "content": "Please don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.
\n\nJust my two cents, opinions may differ.
\n\n[I'll happily share all my secrets after the contest closes. :^)]
\n\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.
\n\n-Rich
", "votes": null, "replies": [] }, { "id": 124736, "author_name": "nigelcarpenter", "author_url": "", "post_date": "06/21/2016 21:06:58", "content": "[quote=quadcore;124730]
\n\nPlease don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.
\n\nJust my two cents, opinions may differ.
\n\n[I'll happily share all my secrets after the contest closes. :^)]
\n\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.
\n\n-Rich
\n\n[/quote]
\n\nAbsolutely agree...!
\n\nBut now I'm concerned that people may think there's any useful information in my previous post.
\n\nGuess it's time for a disclaimer...
\n\nThe fourth day is a Sunday does not imply that every Sunday is the fourth day or that every fourth day is a Sunday. Other days of the week may, or may not, appear on the fourth day or any other day come to mention it.
\n\nit was cloudy on Saturday does not imply that the sun didn't shine on Saturday or preclude other clouds from appearing on other days of the week.
\n\nAnd for good measure... Saturday may occur after Sunday, if Sunday preceeded Saturday.
\n\nWell I think I've done a good job of clearing that up. Now where's that butler and the carving knife?
", "votes": null, "replies": [] }, { "id": 124749, "author_name": "jeffhebert", "author_url": "", "post_date": "06/21/2016 22:53:09", "content": "[quote=Chippy;124723]
\n\nthe butler did it in the pantry with the carving knife!
\n\n[/quote]
\n\nThe professor did it in the orthogonal vector space with a support vector machine.
", "votes": null, "replies": [] }, { "id": 125214, "author_name": "zfturbo", "author_url": "", "post_date": "06/27/2016 17:22:32", "content": "I found that it's possible to predict correct order using "puddles" which is in process of drying day by day. There are pretty many tests which have puddles. )
", "votes": null, "replies": [] }, { "id": 125254, "author_name": "praalankar", "author_url": "", "post_date": "06/28/2016 05:47:03", "content": "Congratulations to winners!
\n\nIt's been a tough challenge. This is how I tried doing. It involved more human effort than ML.
\n\nFirst, georeferenced all the images in a set.
\n\nUsually, when they take aerial photographs, it involves "flight planning". From the KML shared on this forum, I could make out that there are "group" of photos which lie in a line. All photos taken on a particular day will have a similar orientation. I did grouping of the jumbled photographs based on the orientation. Each group would represent all photos (from different sets) which are taken on a particular day.
\n\nNext, I did the analysis of shadows. I noticed from training data that shadow of any object followed anti-clockwise pattern. This helped to figure out which photo was taken on Day-1 and which is on Day-5.
\n\nThere were, of course, conflicts between set of photographs which had similar orientation or overlapping shadows. The only way to improve the score was to swap them and try.
\n\nAll these involved more human effort. So I gave up in the middle :(
", "votes": null, "replies": [] }, { "id": 125279, "author_name": "itodorovic", "author_url": "", "post_date": "06/28/2016 11:35:56", "content": "Congrats to the three winners, all three achieved 1.0 score, deserved.
\n\nNow I suppose it's OK to reveal some stuff: I did a ton of manual labeling, and all that effort went thrown away since my best result was achieved with simplest computer vision based method:
\n\nThis simple solution got me in the top 10% (37th) in private leaderbord, with 220+ positions jump from public to private leaderbord. It's disturbing that all those weeks of feature engineering gave me nothing, but I've learned so much from this competition that I can't complain.
\n\nCan't wait to see the "secret sauce" of the winning solutions.
", "votes": null, "replies": [] }, { "id": 125285, "author_name": "nigelcarpenter", "author_url": "", "post_date": "06/28/2016 12:39:17", "content": "Here's a quick outline of our approach which almost got us a perfect score. It sounds similar to that of others that accepted an approach that included handlabelling.
\n\nWe split the task into four parts
\n\nHandlabelling featured in all parts of the above but we found python and R scripts helped greatly with the image registration and identifying cases where humans had made mistakes in steps 2 and 3.
\n\nParticular learnings included
\n\nWhen it came to ordering days, we could usually find a number of image sets in each cluster that involved some sort of construction which physically progressed in each day, hence giving away the image order. We found that cars in carparks, puddles, containers and rubbish in sorting centres etc were generally less reliable.
\n\nWhat made this competition "easier" was the fact that relative few distinct flights were made and so images overlapped, spatially and temporally. This meant it was possible to register images across sets to to the same days. Once you had determined the order of one set you then knew the order of all images in associated sets.
", "votes": null, "replies": [] }, { "id": 125311, "author_name": "richardepstein", "author_url": "", "post_date": "06/28/2016 15:04:24", "content": "For sets that overlapped, the easiest way to match up two sets was looking at parked cars (assuming there were any). Parked cars usually varied day to day, so if two images from different sets had cars in the same places, they were from the same day.
\n\nSchools typically had empty parking lots on Saturdays and Sundays (but watch out for sports activities).
\n\nChurches tended to have full parking lots on Sundays, nearly empty lots other days.
\n\nGoogle Maps was useful to figure out what some buildings were (school, church, etc).
\n\nI think Saturday was the third day in all sets and I think the images were five days in a row (although I don't know if that is proven). As mentioned earlier, at least in some areas, it rained Saturday, with a few puddles left on Sunday.
\n\nSets with construction / dirt moving activity that helped figure out the order:
\n\n22 and 123 - Construction activity
\n\n236 - Landfill activity
\n\n335 - Pipes being added
\n\n52 - farm being plowed, path being built
\n\n172 - some activity at a house, things being moved around
\n\n148 - construction activity
\n\n326 - dirt piles being added
\n\n189 - puddles
\n\n212 - puddles
\n\n156 - some house construction
\n\n230 - road construction lower left hand corner
\n\n14 - garbage processing - changes every day, but difficult to know the order
\n\n259 - upper right corner, activity with trucks and dirt
\n\n121 - baseball field grooming
\n\n177 - house construction, lower center
\n\n145 - truck/construction movement
\n\n11 - puddles. construction somewhere (cannot find now)
\n\n263 - house construction (lower left corner)
\n\n80 - roads being paved
\n\n141 - roof has white dots added (right side of image)
\n\n344 - pipes being placed in dirt lot/new road
\n\n119 - Farmer's market (only open Sunday)
\n\n278 - truck/movement at stadium
\n\n341 - house construction (center)
\n\n81 - some trucks/dirt motion (left side, center)
\n\n215 - puddles
\n\n149 - roof being finished (center) - only helped with first day
\n\n136 - puddles
\n\n-Rich
", "votes": null, "replies": [] }, { "id": 125319, "author_name": "chrisw1", "author_url": "", "post_date": "06/28/2016 16:05:25", "content": "I initially started hand labelling to get a better understanding on how a machine learning approach might work but then got a bit hooked and didn’t really do much with ML.
\n\nMy approach was similar to those above, but obviously not as successful :)
\n\nGeoreferenced images (thanks for the starting point @kes367):
\n\nFind ‘reference’ sets to determine overall order. I like to think of these as areas whose rate of change is similar to or less frequent than the interval between flights. Good candidates were:
\n\nI also used shadow length in some cases where the shadow angles were indistinguishable. (I was surprised how much shadow length / sun elevation does change even day to day. I now wonder if one could determine true north, estimate time of day from shadow angle and then calculate sun elevation from the shadow length just how far one could get on that alone.)
\n\nFor most of the time I did not assume that the images were taken on consecutive days or that the different geographical areas (within San Diego) were photographed on the same days but am much more convinced of this now.
\n\nI knew that in some cases the way that I used shadow angle was likely to have some errors in assignment of images to flights but ran out of time go back and look for/correct flight path assignment. (I assume I could have made more use of overlapping images as one way to check.)
\n\nOne interesting example of traffic movement that I did not get to use was comparing 2_3 to 183_5. It looks rather like a vehicle has moved into an intersection between these two images. There were similar patterns on highways too from which I think one can determine whether some flights were North to South or South to North. I suspect it might be possible to use this kind of information and changes in time of day estimates to reconstruct the overall sequence/path of each flight.
", "votes": null, "replies": [] }, { "id": 126701, "author_name": "hurlburt", "author_url": "", "post_date": "07/11/2016 22:26:36", "content": "I found my mistake. I had a typo in set 225. The effect of said typo was a -1.6 difference in spearman r correlation for what I actually entered for set 225 which when divided by 139 results in a private leaderboard delta of 0.01151. That difference meant that I went from 6th on the public leaderboard to 7th on the private; 0.98849 on the private leaderboard instead of 1.0000. So here's my first hint. Beware of typos in a labeling competition... I suppose this is where I should be grateful to all of the people who were faster than me; even without the typo I wouldn't have been in the money. (I imagine 4th to be the most uncomfortable position in many Kaggle competitions.)
\n\nHere's my next hint. There were 28 sets in the public leaderboard and 139 in the private. I never did managed to order a set manually. I did all of my ordering via the public leaderboard. The sets in the public leaderboard are
\n\nThe possible values for Spearman's correlation coefficient between two orderings of 1,2,3,4,5 are
\n\n[1 - .1*x for x in range(21)] = [1.0, 0.9, 0.8, ..., -.8, -.9, -1.0]\nIf you know that a set is in the public leaderboard, you can find it's ordering with two reasonably chosen entries. While I did have the dubious distinction of the most entries of any team in the competition, that is still not enough entries to isolate the leaderboard individually. However, as mentioned in many places [see the winner's post], many of the pictures overlap. So by finding groupings of pictures that were taken on the same pass of the same flight, you can triangulate the group if there is at least one member of the group in the public leaderboard using a combination of linear programming and logic. For example
\n\nFlight: 53, Tree like branching rds south of Spring Canyon rd through dome bldg to gravel pit setIds: [284, 231, 88, 208, 333, 48, 233, 138, 172, 23]\n Day set284 set231 set88 set208 set333 set48 set233 set138 set172 set23\n2 c 3 2 1 2 3 1 3 1 3 4\n4 e 5 4 2 4 4 4 5 3 5 3\n0 a 1 5 3 5 5 3 4 5 4 1\n3 d 4 1 5 3 1 5 2 4 1 2\n1 b 2 3 4 1 2 2 1 2 2 5\nis one of the groupings through Scripps Ranch with pictures that are adjacent and overlapping and it happens to contain set 208 from the public leaderboard. By entering groups of approximately 10 permutation matched sets at at time, I was able to use the public leaderboard to find the correct ordering for most groupings.
\n\nI did not match all of these by hand. This leads to my next hint: template matching. Take two sets that overlap and for each image in the first crop out a reasonably sized part of the overlap and use that as a template to match with each of the 5 images in the second set. For each pairing let the score be the maximum correlation coefficient. For example looking at the scores from matching set333 to set48 using a crop window of (1000,0,2000,500)
set333_1, set48_1 0.38268 \nset333_1, set48_2 0.40771 \nset333_1, set48_3 0.32304 \nset333_1, set48_4 0.42695 \nset333_1, set48_5 0.96155 \n\nset333_2, set48_1 0.28035 \nset333_2, set48_2 0.70019 \nset333_2, set48_3 0.30732 \nset333_2, set48_4 0.32623 \nset333_2, set48_5 0.31640 \n\nset333_3, set48_1 0.73081 \nset333_3, set48_2 0.51668 \nset333_3, set48_3 0.43541 \nset333_3, set48_4 0.45769 \nset333_3, set48_5 0.50292 \n\nset333_4, set48_1 0.19761 \nset333_4, set48_2 0.19939 \nset333_4, set48_3 0.17685 \nset333_4, set48_4 0.60496 \nset333_4, set48_5 0.21628 \n\nset333_5, set48_1 0.38741 \nset333_5, set48_2 0.49584 \nset333_5, set48_3 0.96844 \nset333_5, set48_4 0.46904 \nset333_5, set48_5 0.47248 \nwe see that set333_1 and set48_5 were probably taken on the same pass, etc. Template matching does lead to weird discoveries like the fact that even though sets 30 and 145 look like they overlap, they are not taken in the same pass of the airplane.
\n\nTemplate matching breaks down because not all of the pictures taken in the same pass overlap another picture in that pass. Also water exposes a bug/feature in scikit image's match_template code. So as a second method and my next hint is shadows. Shadows were very useful for connecting both sets that didn't overlap other sets and flight groups that didn't have any public leaderboard presence to flight groups that did.
\n\nAs a final hint, beware of rotation matching. It works much better on the training set than the test set.
\n\nBy treating this whole competition as a crazy combination of Where's Waldo and a jig saw puzzle, I didn't actually have to resort to external data although I must admit to using google maps to help confirm which pictures belonged in which area. (It was the picture on the box of the jigsaw puzzle...) In post competition checks I've confirmed that none of the LA pictures were in the private leaderboard. I started this as an exercise in trying to increase the amount of training data to something that would let me build a model... and never did get to building a model.
", "votes": null, "replies": [] }, { "id": 126702, "author_name": "itodorovic", "author_url": "", "post_date": "07/11/2016 22:52:34", "content": "What a nice idea to use a public LB to order groups of sets.
\n\nIt turned out that this competition was ritch in very different solutions, and was much more interesting then initially looked like.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "124696": "Now that the competition entered the final phase, maybe sharing some hints would make it more interesting for everyone. I'll start with one: \r\nAll images with clouds on them are No3 in order. You can manually label them, or if you don't like manual labeling you can count white pixels over some \"whiteness\" threshold, the image which have them drastically more then other four is No3.", "124702": "I found it a bit puzzling that some intensities in one image were not present at all, but in the next or previous. \"Present in current, but not in previous\" seems to yield a blue shift and some interesting psychedelic images...", "124723": "The fourth day is a Sunday, it was cloudy on Saturday..!\r\n\r\nThis is fun, reminds me of those logic puzzles from which you can eventually conclude that the butler did it in the pantry with the carving knife!", "124730": "Please don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.\r\n\r\nJust my two cents, opinions may differ.\r\n\r\n[I'll happily share all my secrets after the contest closes. :^)]\r\n\r\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.\r\n\r\n-Rich", "124736": "[quote=quadcore;124730]\r\n\r\nPlease don't give away too much info; this is a competition, after all. Because this competition lends itself to manual labeling, most hints are basically giving away the answers.\r\n\r\nJust my two cents, opinions may differ.\r\n\r\n[I'll happily share all my secrets after the contest closes. :^)]\r\n\r\nAs an aside, there is no guarantee that all sets have Sunday as the fourth day. I don't think there's a guarantee that the sets are from five days in a row. And since multiple areas were imaged, the weather/cloud cover was not the same everywhere.\r\n\r\n-Rich\r\n\r\n[/quote]\r\n\r\nAbsolutely agree...! \r\n\r\nBut now I'm concerned that people may think there's any useful information in my previous post. \r\n\r\nGuess it's time for a disclaimer...\r\n\r\n***The fourth day is a Sunday*** does not imply that every Sunday is the fourth day or that every fourth day is a Sunday. Other days of the week may, or may not, appear on the fourth day or any other day come to mention it. \r\n\r\n***it was cloudy on Saturday*** does not imply that the sun didn't shine on Saturday or preclude other clouds from appearing on other days of the week. \r\n\r\nAnd for good measure... Saturday may occur after Sunday, if Sunday preceeded Saturday.\r\n\r\nWell I think I've done a good job of clearing that up. Now where's that butler and the carving knife?", "124749": "[quote=Chippy;124723]\r\n\r\nthe butler did it in the pantry with the carving knife!\r\n\r\n[/quote]\r\n\r\n\r\nThe professor did it in the orthogonal vector space with a support vector machine.", "125214": "I found that it's possible to predict correct order using \"puddles\" which is in process of drying day by day. There are pretty many tests which have puddles. )", "125254": "Congratulations to winners!\r\n\r\nIt's been a tough challenge. This is how I tried doing. It involved more human effort than ML.\r\n\r\nFirst, georeferenced all the images in a set.\r\n\r\nUsually, when they take aerial photographs, it involves \"flight planning\". From the KML shared on this forum, I could make out that there are \"group\" of photos which lie in a line. All photos taken on a particular day will have a similar orientation. I did grouping of the jumbled photographs based on the orientation. Each group would represent all photos (from different sets) which are taken on a particular day.\r\n\r\nNext, I did the analysis of shadows. I noticed from training data that shadow of any object followed anti-clockwise pattern. This helped to figure out which photo was taken on Day-1 and which is on Day-5.\r\n\r\nThere were, of course, conflicts between set of photographs which had similar orientation or overlapping shadows. The only way to improve the score was to swap them and try.\r\n\r\nAll these involved more human effort. So I gave up in the middle :(", "125279": "Congrats to the three winners, all three achieved 1.0 score, deserved.\r\n\r\nNow I suppose it's OK to reveal some stuff: I did a ton of manual labeling, and all that effort went thrown away since my best result was achieved with simplest computer vision based method:\r\n\r\n- Compose all five images together\r\n- Calculate relative position and size of each of the five images regarding the composition\r\n- Image with most rotation regarding the average rotation of all five images is ordered No3\r\n- two images with smallest average size regarding the composition are order No1 & No2 in random order\r\n- remaining two images (in random order) are ordered No4 & No5\r\n\r\nThis simple solution got me in the top 10% (37th) in private leaderbord, with 220+ positions jump from public to private leaderbord. It's disturbing that all those weeks of feature engineering gave me nothing, but I've learned so much from this competition that I can't complain.\r\n\r\nCan't wait to see the \"secret sauce\" of the winning solutions.", "125285": "Here's a quick outline of our approach which almost got us a perfect score. It sounds similar to that of others that accepted an approach that included handlabelling.\r\n\r\nWe split the task into four parts\r\n\r\n 1. Geocode images\r\n 2. Identify common days\r\n 3. Order days\r\n 4. Manually peer review and correct submission\r\n\r\nHandlabelling featured in all parts of the above but we found python and R scripts helped greatly with the image registration and identifying cases where humans had made mistakes in steps 2 and 3.\r\n\r\nParticular learnings included\r\n\r\n - we used shadow angles and intensities for our first pass of allocating images to common days\r\n - this worked particularly well for clusters where flights were at distinct and well separated times of day\r\n - where images in a set shared shadow angle and intensity we sometimes could differentiate and allocate to common days by comparing shadow lengths\r\n - for the really difficult images (ie 2 images from each set showing scrubland north of the landfill) where shadow angles and lengths could not be determined we found that images could be allocated to common days according to the rotation of the image itself relative to true north.\r\n\r\nWhen it came to ordering days, we could usually find a number of image sets in each cluster that involved some sort of construction which physically progressed in each day, hence giving away the image order. We found that cars in carparks, puddles, containers and rubbish in sorting centres etc were generally less reliable.\r\n\r\nWhat made this competition \"easier\" was the fact that relative few distinct flights were made and so images overlapped, spatially and temporally. This meant it was possible to register images across sets to to the same days. Once you had determined the order of one set you then knew the order of all images in associated sets.", "125311": "For sets that overlapped, the easiest way to match up two sets was looking at parked cars (assuming there were any). Parked cars usually varied day to day, so if two images from different sets had cars in the same places, they were from the same day.\r\n\r\nSchools typically had empty parking lots on Saturdays and Sundays (but watch out for sports activities).\r\n\r\nChurches tended to have full parking lots on Sundays, nearly empty lots other days.\r\n\r\nGoogle Maps was useful to figure out what some buildings were (school, church, etc).\r\n\r\nI think Saturday was the third day in all sets and I think the images were five days in a row (although I don't know if that is proven). As mentioned earlier, at least in some areas, it rained Saturday, with a few puddles left on Sunday.\r\n\r\nSets with construction / dirt moving activity that helped figure out the order:\r\n\r\n22 and 123 - Construction activity\r\n\r\n236 - Landfill activity\r\n\r\n335 - Pipes being added\r\n\r\n52 - farm being plowed, path being built\r\n\r\n172 - some activity at a house, things being moved around\r\n\r\n148 - construction activity\r\n\r\n326 - dirt piles being added\r\n\r\n189 - puddles\r\n\r\n212 - puddles\r\n\r\n156 - some house construction\r\n\r\n230 - road construction lower left hand corner\r\n\r\n14 - garbage processing - changes every day, but difficult to know the order\r\n\r\n259 - upper right corner, activity with trucks and dirt\r\n\r\n121 - baseball field grooming\r\n\r\n177 - house construction, lower center\r\n\r\n145 - truck/construction movement\r\n\r\n11 - puddles. construction somewhere (cannot find now)\r\n\r\n263 - house construction (lower left corner)\r\n\r\n80 - roads being paved\r\n\r\n141 - roof has white dots added (right side of image)\r\n\r\n344 - pipes being placed in dirt lot/new road\r\n\r\n119 - Farmer's market (only open Sunday)\r\n\r\n278 - truck/movement at stadium\r\n\r\n341 - house construction (center)\r\n\r\n81 - some trucks/dirt motion (left side, center)\r\n\r\n215 - puddles\r\n\r\n149 - roof being finished (center) - only helped with first day\r\n\r\n136 - puddles\r\n\r\n-Rich", "125319": "I initially started hand labelling to get a better understanding on how a machine learning approach might work but then got a bit hooked and didn’t really do much with ML.\r\n\r\nMy approach was similar to those above, but obviously not as successful :) \r\n\r\n Georeferenced images (thanks for the starting point @kes367):\r\n\r\n - Came to conclusion that images were taken on a series of North-South flights\r\n - Assign images to flight passes\r\n - Observed consistent image angles for some sets of images\r\n - Observed consistent shadow angles (or lack or shadows) for some sets\r\n - I generally did not use overlapping image features but when I did car parks or car parking were the most helpful \r\n\r\nFind ‘reference’ sets to determine overall order. I like to think of these as areas whose rate of change is similar to or less frequent than the interval between flights. Good candidates were:\r\n\r\n - Building sites\r\n - Scrap yards (gradual dismantling of vehicles)\r\n - Baseball fields (raking of diamond)\r\n - School/church car parks/School sports for weekend patterns\r\n - There was one baseball field with a new scoreboard being installed.\r\n - Use traffic movement in some cases of overlapping images\r\n\r\nI also used shadow length in some cases where the shadow angles were indistinguishable. (I was surprised how much shadow length / sun elevation does change even day to day. I now wonder if one could determine true north, estimate time of day from shadow angle and then calculate sun elevation from the shadow length just how far one could get on that alone.)\r\n\r\nFor most of the time I did not assume that the images were taken on consecutive days or that the different geographical areas (within San Diego) were photographed on the same days but am much more convinced of this now. \r\n\r\nI knew that in some cases the way that I used shadow angle was likely to have some errors in assignment of images to flights but ran out of time go back and look for/correct flight path assignment. (I assume I could have made more use of overlapping images as one way to check.) \r\n\r\nOne interesting example of traffic movement that I did not get to use was comparing 2_3 to 183_5. It looks rather like a vehicle has moved into an intersection between these two images. There were similar patterns on highways too from which I think one can determine whether some flights were North to South or South to North. I suspect it might be possible to use this kind of information and changes in time of day estimates to reconstruct the overall sequence/path of each flight.", "126701": "I found my mistake. I had a typo in set 225. The effect of said typo was a -1.6 difference in spearman r correlation for what I actually entered for set 225 which when divided by 139 results in a private leaderboard delta of 0.01151. That difference meant that I went from 6th on the public leaderboard to 7th on the private; 0.98849 on the private leaderboard instead of 1.0000. So here's my first hint. Beware of typos in a labeling competition... I suppose this is where I should be grateful to all of the people who were faster than me; even without the typo I wouldn't have been in the money. (I imagine 4th to be the most uncomfortable position in many Kaggle competitions.)\r\n\r\nHere's my next hint. There were 28 sets in the public leaderboard and 139 in the private. I never did managed to order a set manually. I did all of my ordering via the public leaderboard. The sets in the public leaderboard are \r\n\r\n - 330, 328, 215, 119, 275, 72 from La Jolla\r\n - 13, 259, 179, 38 from Logan Heights\r\n - 83, 145, 39, 234, 195, 7, 266, 240 from the Point Loma/San Diego Bay/Airport/Old Town area\r\n - 1, 111, 208, 254, 187 from Scripps Ranch \r\n - 253, 152, 200, 67, 54 from Sycamore Landfill and scrub east of landfill\r\n\r\nThe possible values for Spearman's correlation coefficient between two orderings of 1,2,3,4,5 are \r\n\r\n [1 - .1*x for x in range(21)] = [1.0, 0.9, 0.8, ..., -.8, -.9, -1.0]\r\n\r\nIf you know that a set is in the public leaderboard, you can find it's ordering with two reasonably chosen entries. While I did have the dubious distinction of the most entries of any team in the competition, that is still not enough entries to isolate the leaderboard individually. However, as mentioned in many places [\\[see the winner's post\\]][1], many of the pictures overlap. So by finding groupings of pictures that were taken on the same pass of the same flight, you can triangulate the group if there is at least one member of the group in the public leaderboard using a combination of linear programming and logic. For example\r\n\r\n Flight: 53, Tree like branching rds south of Spring Canyon rd through dome bldg to gravel pit setIds: [284, 231, 88, 208, 333, 48, 233, 138, 172, 23]\r\n Day set284 set231 set88 set208 set333 set48 set233 set138 set172 set23\r\n 2 c 3 2 1 2 3 1 3 1 3 4\r\n 4 e 5 4 2 4 4 4 5 3 5 3\r\n 0 a 1 5 3 5 5 3 4 5 4 1\r\n 3 d 4 1 5 3 1 5 2 4 1 2\r\n 1 b 2 3 4 1 2 2 1 2 2 5\r\n\r\nis one of the groupings through Scripps Ranch with pictures that are adjacent and overlapping and it happens to contain set 208 from the public leaderboard. By entering groups of approximately 10 permutation matched sets at at time, I was able to use the public leaderboard to find the correct ordering for most groupings.\r\n\r\nI did not match all of these by hand. This leads to my next hint: [template matching][2]. Take two sets that overlap and for each image in the first crop out a reasonably sized part of the overlap and use that as a template to match with each of the 5 images in the second set. For each pairing let the score be the maximum correlation coefficient. For example looking at the scores from matching set333 to set48 using a crop window of `(1000,0,2000,500)`\r\n\r\n set333_1, set48_1 0.38268 \r\n set333_1, set48_2 0.40771 \r\n set333_1, set48_3 0.32304 \r\n set333_1, set48_4 0.42695 \r\n set333_1, set48_5 0.96155 \r\n \r\n set333_2, set48_1 0.28035 \r\n set333_2, set48_2 0.70019 \r\n set333_2, set48_3 0.30732 \r\n set333_2, set48_4 0.32623 \r\n set333_2, set48_5 0.31640 \r\n \r\n set333_3, set48_1 0.73081 \r\n set333_3, set48_2 0.51668 \r\n set333_3, set48_3 0.43541 \r\n set333_3, set48_4 0.45769 \r\n set333_3, set48_5 0.50292 \r\n \r\n set333_4, set48_1 0.19761 \r\n set333_4, set48_2 0.19939 \r\n set333_4, set48_3 0.17685 \r\n set333_4, set48_4 0.60496 \r\n set333_4, set48_5 0.21628 \r\n \r\n set333_5, set48_1 0.38741 \r\n set333_5, set48_2 0.49584 \r\n set333_5, set48_3 0.96844 \r\n set333_5, set48_4 0.46904 \r\n set333_5, set48_5 0.47248 \r\n\r\nwe see that set333_1 and set48_5 were probably taken on the same pass, etc. Template matching does lead to weird discoveries like the fact that even though sets 30 and 145 look like they overlap, they are not taken in the same pass of the airplane. \r\n\r\nTemplate matching breaks down because not all of the pictures taken in the same pass overlap another picture in that pass. Also water exposes a bug/feature in scikit image's match_template code. So as a second method and my next hint is shadows. Shadows were very useful for connecting both sets that didn't overlap other sets and flight groups that didn't have any public leaderboard presence to flight groups that did. \r\n\r\nAs a final hint, beware of rotation matching. It works much better on the training set than the test set.\r\n\r\nBy treating this whole competition as a crazy combination of Where's Waldo and a jig saw puzzle, I didn't actually have to resort to external data although I must admit to using google maps to help confirm which pictures belonged in which area. (It was the picture on the box of the jigsaw puzzle...) In post competition checks I've confirmed that none of the LA pictures were in the private leaderboard. I started this as an exercise in trying to increase the amount of training data to something that would let me build a model... and never did get to building a model.\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/draper-satellite-image-chronology/forums/t/21936/1st-place-how-to-win-the-competition-if-you-know-nothing-about-image-processing/125346#post125346\r\n [2]: http://scikit-image.org/docs/dev/api/skimage.feature.html?highlight=match_template#skimage.feature.match_template", "126702": "What a nice idea to use a public LB to order groups of sets. \r\n\r\nIt turned out that this competition was ritch in very different solutions, and was much more interesting then initially looked like." }, "source": "meta" }