What a strange competition! Now three top places are taken by 3 players. Great job! I wonder if that is something Draper really wanted: if I understand correctly, in most cases the submission were hand labelled, so not much ML.
\n\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it.
\n\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling.
", "messages": [ { "id": 120586, "postDate": "2016-05-19T10:21:57.227Z", "content": "What a strange competition! Now three top places are taken by 3 players. Great job! I wonder if that is something Draper really wanted: if I understand correctly, in most cases the submission were hand labelled, so not much ML.
\n\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it.
\n\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling.
", "rawMarkdown": "What a strange competition! Now three top places are taken by 3 players. Great job! I wonder if that is something Draper really wanted: if I understand correctly, in most cases the submission were hand labelled, so not much ML. \r\n\r\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it. \r\n\r\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling. \r\n\r\n\r\n", "votes": 6 }, { "id": 120654, "postDate": "2016-05-19T19:08:11.783Z", "content": "Hurry up, guys. You can get your master title or top 100 (overall) by hand annotations :).
", "rawMarkdown": "Hurry up, guys. You can get your master title or top 100 (overall) by hand annotations :).\r\n", "votes": 5 }, { "id": 120950, "postDate": "2016-05-22T03:18:04.213Z", "content": "\n\n\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?
\n
Both the sets are included in the test files. the current lb scores are on 17% of those images.
", "rawMarkdown": "> \r\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?\r\n\r\nBoth the sets are included in the test files. the current lb scores are on 17% of those images.", "votes": 1 }, { "id": 120718, "postDate": "2016-05-20T03:10:31.953Z", "content": "[quote=FangzouLiao;120710]
\n\nwhat if there are 4 team reaching 1.00 in private LB?
\n\n[/quote]
\n\nTies are broken by submission timestamp (earlier being better).
", "rawMarkdown": "[quote=FangzouLiao;120710]\r\n\r\nwhat if there are 4 team reaching 1.00 in private LB?\r\n\r\n[/quote]\r\n\r\nTies are broken by submission timestamp (earlier being better).", "votes": 1 }, { "id": 122814, "postDate": "2016-06-07T14:12:05.083Z", "content": "I was quite excited to take part but seeing this thread is a bit of a downer.
\n\nAlso I think there are many other much more interesting properties of satellite images, where ML is actually required, such as identifying objects, identifying certain types of change (deforestation) etc.
\n\nTimestamps are usually available as metadata, not sure why we would need ML to infer those?
", "rawMarkdown": "I was quite excited to take part but seeing this thread is a bit of a downer.\r\n\r\nAlso I think there are many other much more interesting properties of satellite images, where ML is actually required, such as identifying objects, identifying certain types of change (deforestation) etc.\r\n\r\nTimestamps are usually available as metadata, not sure why we would need ML to infer those?", "votes": 2 }, { "id": 121369, "postDate": "2016-05-25T23:40:32.170Z", "content": "I have an updated theory on what Draper is looking for.
\n\nThey are looking for people who are keen/willing/predisposed/good at/OCD/observant to go through many many images. They'll identify those people and hire them as image analysts!
\n\nNot exactly ML, but maybe cheaper than a headhunter.
", "rawMarkdown": "I have an updated theory on what Draper is looking for. \r\n\r\nThey are looking for people who are keen/willing/predisposed/good at/OCD/observant to go through many many images. They'll identify those people and hire them as image analysts!\r\n\r\nNot exactly ML, but maybe cheaper than a headhunter.", "votes": 2 }, { "id": 121257, "postDate": "2016-05-25T07:22:49.630Z", "content": "https://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak
", "rawMarkdown": "https://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak", "votes": 1 }, { "id": 125233, "postDate": "2016-06-28T00:01:33.637Z", "content": "now we know why the competition did not finish before. :O
", "rawMarkdown": "now we know why the competition did not finish before. :O" }, { "id": 123298, "postDate": "2016-06-10T23:07:41.227Z", "content": "@zero zero Your theory makes more sense than competition description itself.
", "rawMarkdown": "@zero zero Your theory makes more sense than competition description itself. " }, { "id": 121540, "postDate": "2016-05-27T05:27:50.567Z", "content": "Indeed this task needs some domain knowledge even for manual labeling. I look at some of the image set, and just cannot tell which comes first...
", "rawMarkdown": "Indeed this task needs some domain knowledge even for manual labeling. I look at some of the image set, and just cannot tell which comes first..." }, { "id": 121309, "postDate": "2016-05-25T14:27:33.713Z", "content": "In theory, the method underlying the hand labeling is visual interpretation. Visual interpretation is still widely used in geospatial intelligence, but requires domain knowledge. It can be reproducible in similar contexts.
\n\nThe bigger problem with this competition is that the purpose is unclear. Is this issue lost metadata and Draper wants to date the images? or, is Draper interested in assessing changes in the information contained within the images (as they say in the competition description)?
\n\nFor example, the way this data was constructed, the images can be grouped based on being part of the same flight path. This is just one type of dependency in the dataset that can be exploited. However, exploiting these sorts of dependencies will not give you a prediction method for ordering one randomly chosen set of images outside of this dataset (i.e., there is no external validity).
", "rawMarkdown": "In theory, the method underlying the hand labeling is visual interpretation. Visual interpretation is still widely used in geospatial intelligence, but requires domain knowledge. It can be reproducible in similar contexts.\r\n\r\nThe bigger problem with this competition is that the purpose is unclear. Is this issue lost metadata and Draper wants to date the images? or, is Draper interested in assessing changes in the information contained within the images (as they say in the competition description)? \r\n\r\nFor example, the way this data was constructed, the images can be grouped based on being part of the same flight path. This is just one type of dependency in the dataset that can be exploited. However, exploiting these sorts of dependencies will not give you a prediction method for ordering one randomly chosen set of images outside of this dataset (i.e., there is no external validity). " }, { "id": 121299, "postDate": "2016-05-25T12:56:46.120Z", "content": "[quote=Laurae;121280]
\n\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons "the steps used in set XYZ does not apply to the set ABC because DEF"?) - otherwise it would not be reproducible.
\n\n[/quote]
\n\n@Laurae The documentation doesn't have a fixed required format (it could be annotated images, a text narrative, code, equations, or any combination thereof), but reproducibility is the operative word. The host should be able to follow and understand your methodology based on the report. You may rely on commonalities in your methodology to avoid having to document every single image. However, if your approach is entirely manual and different for every image, you'll end up needing more documentation to make it reproducible.
", "rawMarkdown": "[quote=Laurae;121280]\r\n\r\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons \"the steps used in set XYZ does not apply to the set ABC because DEF\"?) - otherwise it would not be reproducible.\r\n\r\n[/quote]\r\n\r\n@Laurae The documentation doesn't have a fixed required format (it could be annotated images, a text narrative, code, equations, or any combination thereof), but reproducibility is the operative word. The host should be able to follow and understand your methodology based on the report. You may rely on commonalities in your methodology to avoid having to document every single image. However, if your approach is entirely manual and different for every image, you'll end up needing more documentation to make it reproducible." }, { "id": 121280, "postDate": "2016-05-25T10:03:04.123Z", "content": "[quote=gk43;120940]
\n\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?
\n\n[/quote]
\n\nAs the prize winners will have to hand out documentation to reproduce the whole steps they did, I guess if there is a conflict during reproduction of the steps they will be deleted from the LB.
\n\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons "the steps used in set XYZ does not apply to the set ABC because DEF"?) - otherwise it would not be reproducible.
\n\n[quote=Abhimanyu Dikshit;120886]
\n\nIs there any weightage to number of submissions while breaking ties?
\n\n[/quote]
\n\nNo, the rules determine the winners by the submission time. For instance, if you selected the earliest 1.000 Private LB submission among all the 1.000 Private LB submissions of others, you would win.
\n\n[quote=Humberto Brandão;121257]
\n\nhttps://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak
\n\n[/quote]
\n\nThere is no data leak... the way to get 1.00 was already explained numerous times (hand labeling).
", "rawMarkdown": "[quote=gk43;120940]\r\n\r\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed? \r\n\r\n[/quote]\r\n\r\nAs the prize winners will have to hand out documentation to reproduce the whole steps they did, I guess if there is a conflict during reproduction of the steps they will be deleted from the LB.\r\n\r\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons \"the steps used in set XYZ does not apply to the set ABC because DEF\"?) - otherwise it would not be reproducible.\r\n\r\n[quote=Abhimanyu Dikshit;120886]\r\n\r\nIs there any weightage to number of submissions while breaking ties?\r\n\r\n[/quote]\r\n\r\nNo, the rules determine the winners by the submission time. For instance, if you selected the earliest 1.000 Private LB submission among all the 1.000 Private LB submissions of others, you would win.\r\n\r\n[quote=Humberto Brandão;121257]\r\n\r\nhttps://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak\r\n\r\n[/quote]\r\n\r\nThere is no data leak... the way to get 1.00 was already explained numerous times (hand labeling)." }, { "id": 120940, "postDate": "2016-05-21T21:22:53.447Z", "content": "Are you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?
", "rawMarkdown": "Are you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed? \r\n" }, { "id": 120886, "postDate": "2016-05-21T11:42:39.583Z", "content": "Is there any weightage to number of submissions while breaking ties?
", "rawMarkdown": "Is there any weightage to number of submissions while breaking ties?" }, { "id": 120710, "postDate": "2016-05-20T02:18:48.640Z", "content": "what if there are 4 team reaching 1.00 in private LB?
\n\n[quote=William Cukierski;120648]
\n\nWe do not intend to close or alter the competition at this point*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).
\n\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods.
\n\nCongrats on the progress so far!
\n\n* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.
\n\n[/quote]
", "rawMarkdown": "what if there are 4 team reaching 1.00 in private LB?\r\n\r\n[quote=William Cukierski;120648]\r\n\r\nWe do not intend to close or alter the competition at this point\\*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).\r\n\r\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods. \r\n\r\nCongrats on the progress so far!\r\n\r\n\\* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.\r\n\r\n[/quote]\r\n" }, { "id": 120648, "postDate": "2016-05-19T18:55:19.107Z", "content": "We do not intend to close or alter the competition at this point*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).
\n\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods.
\n\nCongrats on the progress so far!
\n\n* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.
", "rawMarkdown": "We do not intend to close or alter the competition at this point\\*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).\r\n\r\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods. \r\n\r\nCongrats on the progress so far!\r\n\r\n\\* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition." }, { "id": 120620, "postDate": "2016-05-19T14:37:54.660Z", "content": "I agree with @the1owl and @smota, I would prefer if they left it up. I am trying a pure ML solution and even though not in the running for the prize its of value to be able to try and improve my method.
", "rawMarkdown": "I agree with @the1owl and @smota, I would prefer if they left it up. I am trying a pure ML solution and even though not in the running for the prize its of value to be able to try and improve my method." }, { "id": 120608, "postDate": "2016-05-19T12:46:06.683Z", "content": "Yeah. The majority of the test data is in a totally separate area, and the test sets that you can actually solve with overlapping training sets are not included in the public LB.
", "rawMarkdown": "Yeah. The majority of the test data is in a totally separate area, and the test sets that you can actually solve with overlapping training sets are not included in the public LB." }, { "id": 120607, "postDate": "2016-05-19T12:40:46.727Z", "content": "I am not agree to close competition. Maybe admins can notice us when the prizes are taken, but I am learning a lot and it is very interesting to follow a competition so different like this.
\n\nAnd, by the way, congrats to all 1 scores.
", "rawMarkdown": "I am not agree to close competition. Maybe admins can notice us when the prizes are taken, but I am learning a lot and it is very interesting to follow a competition so different like this. \r\n\r\nAnd, by the way, congrats to all 1 scores." }, { "id": 120601, "postDate": "2016-05-19T12:12:07.257Z", "content": "I guess so too, the admin can check their private score and if they are all 1.0 now (I bet it should be highly close since there is not much overfitting or anything like that in this competition) then the admin can notice us that the prize are already taken. :D
", "rawMarkdown": "I guess so too, the admin can check their private score and if they are all 1.0 now (I bet it should be highly close since there is not much overfitting or anything like that in this competition) then the admin can notice us that the prize are already taken. :D" }, { "id": 120596, "postDate": "2016-05-19T11:15:11.983Z", "content": "If there are three perfect private scores, sure.
\n\nThe train data only overlaps with the Long Beach portion of the test data, which is not a high percentage.
", "rawMarkdown": "If there are three perfect private scores, sure.\r\n\r\nThe train data only overlaps with the Long Beach portion of the test data, which is not a high percentage." }, { "id": 120594, "postDate": "2016-05-19T11:13:19.927Z", "content": "[quote=olegpolivin;120586]
\n\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it.
\n\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling.
\n\n[/quote]
\n\nIf they allowed hand labeling, the reason was because it might be too difficult for an algorithm to solve the problem automatically (else you would have made a breakthrough in research for that specific domain).
\n\nThe most probable hypotheses for allowing hand labeling are:
\n\nI suppose the three top teams with 1.00 score have already documented in depth all the manual steps they used (I can't imagine the size of the PDF :p). As time goes there could be satellite images that may not be accessible publicly anymore (ex: MapBox, etc.), therefore the steps might not be reproducible.
\n\nAnd not only all pictures can be found online (search for "GIS aerial imagery" and you should find it easily), but not every picture is following the same framework to be localized and analyzed to make a chronological order.
\n\n\n", "rawMarkdown": "[quote=olegpolivin;120586]\r\n\r\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it. \r\n\r\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling. \r\n\r\n[/quote]\r\n\r\nIf they allowed hand labeling, the reason was because it might be too difficult for an algorithm to solve the problem automatically (else you would have made a breakthrough in research for that specific domain).\r\n\r\nThe most probable hypotheses for allowing hand labeling are:\r\n\r\n* The need for a proper framework on how to order chronologically aerial imagery (this is why you need to provide **the whole documentation for each hand labeled image so it can be reproduced**)\r\n* The need of compare potentially contradictory frameworks on how to order chronologically aerial imagery (comparison of researched frameworks, etc.)\r\n* And potentially turning a working framework into a ML code by themselves\r\n\r\nI suppose the three top teams with 1.00 score have already documented in depth all the manual steps they used (I can't imagine the size of the PDF :p). As time goes there could be satellite images that may not be accessible publicly anymore (ex: MapBox, etc.), therefore the steps might not be reproducible.\r\n\r\nAnd not only all pictures can be found online (search for \"GIS aerial imagery\" and you should find it easily), but not every picture is following the same framework to be localized and analyzed to make a chronological order.\r\n\r\n> Given the unique nature of this competition design, please note a\r\n> couple key changes to Kaggle's standard rules, listed below: \r\n> \r\n> * Hand labeling and human prediction are permitted.\r\n> * Winners must deliver documentation that clearly outlines the manual and automatic steps required to create the winning submission.\r\n> * Use of external data is permitted." }, { "id": 125252, "postDate": "2016-06-28T05:29:12.517Z", "rawMarkdown": "", "isDeleted": true }, { "id": 125144, "postDate": "2016-06-26T18:30:25.710Z", "rawMarkdown": "", "isDeleted": true }, { "id": 120618, "postDate": "2016-05-19T14:21:31.450Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 120654, "author_name": "Artem Golubin", "author_url": "", "post_date": "2016-05-19T19:08:11.783000", "content": "Given the unique nature of this competition design, please note a\n couple key changes to Kaggle's standard rules, listed below:
\n \n\n
\n- Hand labeling and human prediction are permitted.
\n- Winners must deliver documentation that clearly outlines the manual and automatic steps required to create the winning submission.
\n- Use of external data is permitted.
\n
Hurry up, guys. You can get your master title or top 100 (overall) by hand annotations :).
", "votes": 5, "replies": [] }, { "id": 120950, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2016-05-22T03:18:04.213000", "content": "\n\n\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?
\n
Both the sets are included in the test files. the current lb scores are on 17% of those images.
", "votes": 1, "replies": [] }, { "id": 120718, "author_name": "Will Cukierski", "author_url": "", "post_date": "2016-05-20T03:10:31.953000", "content": "[quote=FangzouLiao;120710]
\n\nwhat if there are 4 team reaching 1.00 in private LB?
\n\n[/quote]
\n\nTies are broken by submission timestamp (earlier being better).
", "votes": 1, "replies": [] }, { "id": 122814, "author_name": "dharma1", "author_url": "", "post_date": "2016-06-07T14:12:05.083000", "content": "I was quite excited to take part but seeing this thread is a bit of a downer.
\n\nAlso I think there are many other much more interesting properties of satellite images, where ML is actually required, such as identifying objects, identifying certain types of change (deforestation) etc.
\n\nTimestamps are usually available as metadata, not sure why we would need ML to infer those?
", "votes": 2, "replies": [] }, { "id": 121369, "author_name": "zero zero", "author_url": "", "post_date": "2016-05-25T23:40:32.170000", "content": "I have an updated theory on what Draper is looking for.
\n\nThey are looking for people who are keen/willing/predisposed/good at/OCD/observant to go through many many images. They'll identify those people and hire them as image analysts!
\n\nNot exactly ML, but maybe cheaper than a headhunter.
", "votes": 2, "replies": [] }, { "id": 121257, "author_name": "Humberto Brandão, Ph.D.", "author_url": "", "post_date": "2016-05-25T07:22:49.630000", "content": "https://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak
", "votes": 1, "replies": [] }, { "id": 125233, "author_name": "Humberto Brandão, Ph.D.", "author_url": "", "post_date": "2016-06-28T00:01:33.637000", "content": "now we know why the competition did not finish before. :O
", "votes": 0, "replies": [] }, { "id": 123298, "author_name": "Shahnawaz Akhtar", "author_url": "", "post_date": "2016-06-10T23:07:41.227000", "content": "@zero zero Your theory makes more sense than competition description itself.
", "votes": 0, "replies": [] }, { "id": 121540, "author_name": "phph", "author_url": "", "post_date": "2016-05-27T05:27:50.567000", "content": "Indeed this task needs some domain knowledge even for manual labeling. I look at some of the image set, and just cannot tell which comes first...
", "votes": 0, "replies": [] }, { "id": 121309, "author_name": "applied", "author_url": "", "post_date": "2016-05-25T14:27:33.713000", "content": "In theory, the method underlying the hand labeling is visual interpretation. Visual interpretation is still widely used in geospatial intelligence, but requires domain knowledge. It can be reproducible in similar contexts.
\n\nThe bigger problem with this competition is that the purpose is unclear. Is this issue lost metadata and Draper wants to date the images? or, is Draper interested in assessing changes in the information contained within the images (as they say in the competition description)?
\n\nFor example, the way this data was constructed, the images can be grouped based on being part of the same flight path. This is just one type of dependency in the dataset that can be exploited. However, exploiting these sorts of dependencies will not give you a prediction method for ordering one randomly chosen set of images outside of this dataset (i.e., there is no external validity).
", "votes": 0, "replies": [] }, { "id": 121299, "author_name": "Will Cukierski", "author_url": "", "post_date": "2016-05-25T12:56:46.120000", "content": "[quote=Laurae;121280]
\n\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons "the steps used in set XYZ does not apply to the set ABC because DEF"?) - otherwise it would not be reproducible.
\n\n[/quote]
\n\n@Laurae The documentation doesn't have a fixed required format (it could be annotated images, a text narrative, code, equations, or any combination thereof), but reproducibility is the operative word. The host should be able to follow and understand your methodology based on the report. You may rely on commonalities in your methodology to avoid having to document every single image. However, if your approach is entirely manual and different for every image, you'll end up needing more documentation to make it reproducible.
", "votes": 0, "replies": [] }, { "id": 121280, "author_name": "Laurae", "author_url": "", "post_date": "2016-05-25T10:03:04.123000", "content": "[quote=gk43;120940]
\n\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?
\n\n[/quote]
\n\nAs the prize winners will have to hand out documentation to reproduce the whole steps they did, I guess if there is a conflict during reproduction of the steps they will be deleted from the LB.
\n\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons "the steps used in set XYZ does not apply to the set ABC because DEF"?) - otherwise it would not be reproducible.
\n\n[quote=Abhimanyu Dikshit;120886]
\n\nIs there any weightage to number of submissions while breaking ties?
\n\n[/quote]
\n\nNo, the rules determine the winners by the submission time. For instance, if you selected the earliest 1.000 Private LB submission among all the 1.000 Private LB submissions of others, you would win.
\n\n[quote=Humberto Brandão;121257]
\n\nhttps://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak
\n\n[/quote]
\n\nThere is no data leak... the way to get 1.00 was already explained numerous times (hand labeling).
", "votes": 0, "replies": [] }, { "id": 120940, "author_name": "gk43", "author_url": "", "post_date": "2016-05-21T21:22:53.447000", "content": "Are you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?
", "votes": 0, "replies": [] }, { "id": 120886, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2016-05-21T11:42:39.583000", "content": "Is there any weightage to number of submissions while breaking ties?
", "votes": 0, "replies": [] }, { "id": 120710, "author_name": "Fangzhou Liao", "author_url": "", "post_date": "2016-05-20T02:18:48.640000", "content": "what if there are 4 team reaching 1.00 in private LB?
\n\n[quote=William Cukierski;120648]
\n\nWe do not intend to close or alter the competition at this point*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).
\n\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods.
\n\nCongrats on the progress so far!
\n\n* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.
\n\n[/quote]
", "votes": 0, "replies": [] }, { "id": 120648, "author_name": "Will Cukierski", "author_url": "", "post_date": "2016-05-19T18:55:19.107000", "content": "We do not intend to close or alter the competition at this point*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).
\n\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods.
\n\nCongrats on the progress so far!
\n\n* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.
", "votes": 0, "replies": [] }, { "id": 120620, "author_name": "thakadu", "author_url": "", "post_date": "2016-05-19T14:37:54.660000", "content": "I agree with @the1owl and @smota, I would prefer if they left it up. I am trying a pure ML solution and even though not in the running for the prize its of value to be able to try and improve my method.
", "votes": 0, "replies": [] }, { "id": 120608, "author_name": "kes367", "author_url": "", "post_date": "2016-05-19T12:46:06.683000", "content": "Yeah. The majority of the test data is in a totally separate area, and the test sets that you can actually solve with overlapping training sets are not included in the public LB.
", "votes": 0, "replies": [] }, { "id": 120607, "author_name": "Santiago Mota", "author_url": "", "post_date": "2016-05-19T12:40:46.727000", "content": "I am not agree to close competition. Maybe admins can notice us when the prizes are taken, but I am learning a lot and it is very interesting to follow a competition so different like this.
\n\nAnd, by the way, congrats to all 1 scores.
", "votes": 0, "replies": [] }, { "id": 120601, "author_name": "woshialex", "author_url": "", "post_date": "2016-05-19T12:12:07.257000", "content": "I guess so too, the admin can check their private score and if they are all 1.0 now (I bet it should be highly close since there is not much overfitting or anything like that in this competition) then the admin can notice us that the prize are already taken. :D
", "votes": 0, "replies": [] }, { "id": 120596, "author_name": "Tyler Vigen", "author_url": "", "post_date": "2016-05-19T11:15:11.983000", "content": "If there are three perfect private scores, sure.
\n\nThe train data only overlaps with the Long Beach portion of the test data, which is not a high percentage.
", "votes": 0, "replies": [] }, { "id": 120594, "author_name": "Laurae", "author_url": "", "post_date": "2016-05-19T11:13:19.927000", "content": "[quote=olegpolivin;120586]
\n\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it.
\n\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling.
\n\n[/quote]
\n\nIf they allowed hand labeling, the reason was because it might be too difficult for an algorithm to solve the problem automatically (else you would have made a breakthrough in research for that specific domain).
\n\nThe most probable hypotheses for allowing hand labeling are:
\n\nI suppose the three top teams with 1.00 score have already documented in depth all the manual steps they used (I can't imagine the size of the PDF :p). As time goes there could be satellite images that may not be accessible publicly anymore (ex: MapBox, etc.), therefore the steps might not be reproducible.
\n\nAnd not only all pictures can be found online (search for "GIS aerial imagery" and you should find it easily), but not every picture is following the same framework to be localized and analyzed to make a chronological order.
\n\n\n", "votes": 0, "replies": [] }, { "id": 125252, "author_name": "", "author_url": "", "post_date": "2016-06-28T05:29:12.517000", "content": "", "votes": 0, "replies": [] }, { "id": 125144, "author_name": "", "author_url": "", "post_date": "2016-06-26T18:30:25.710000", "content": "", "votes": 0, "replies": [] }, { "id": 120618, "author_name": "", "author_url": "", "post_date": "2016-05-19T14:21:31.450000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "120586": "What a strange competition! Now three top places are taken by 3 players. Great job! I wonder if that is something Draper really wanted: if I understand correctly, in most cases the submission were hand labelled, so not much ML. \r\n\r\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it. \r\n\r\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling. \r\n\r\n\r\n", "120654": "Hurry up, guys. You can get your master title or top 100 (overall) by hand annotations :).\r\n", "120950": "> \r\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed?\r\n\r\nBoth the sets are included in the test files. the current lb scores are on 17% of those images.", "120718": "[quote=FangzouLiao;120710]\r\n\r\nwhat if there are 4 team reaching 1.00 in private LB?\r\n\r\n[/quote]\r\n\r\nTies are broken by submission timestamp (earlier being better).", "122814": "I was quite excited to take part but seeing this thread is a bit of a downer.\r\n\r\nAlso I think there are many other much more interesting properties of satellite images, where ML is actually required, such as identifying objects, identifying certain types of change (deforestation) etc.\r\n\r\nTimestamps are usually available as metadata, not sure why we would need ML to infer those?", "121369": "I have an updated theory on what Draper is looking for. \r\n\r\nThey are looking for people who are keen/willing/predisposed/good at/OCD/observant to go through many many images. They'll identify those people and hire them as image analysts!\r\n\r\nNot exactly ML, but maybe cheaper than a headhunter.", "121257": "https://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak", "125233": "now we know why the competition did not finish before. :O", "123298": "@zero zero Your theory makes more sense than competition description itself. ", "121540": "Indeed this task needs some domain knowledge even for manual labeling. I look at some of the image set, and just cannot tell which comes first...", "121309": "In theory, the method underlying the hand labeling is visual interpretation. Visual interpretation is still widely used in geospatial intelligence, but requires domain knowledge. It can be reproducible in similar contexts.\r\n\r\nThe bigger problem with this competition is that the purpose is unclear. Is this issue lost metadata and Draper wants to date the images? or, is Draper interested in assessing changes in the information contained within the images (as they say in the competition description)? \r\n\r\nFor example, the way this data was constructed, the images can be grouped based on being part of the same flight path. This is just one type of dependency in the dataset that can be exploited. However, exploiting these sorts of dependencies will not give you a prediction method for ordering one randomly chosen set of images outside of this dataset (i.e., there is no external validity). ", "121299": "[quote=Laurae;121280]\r\n\r\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons \"the steps used in set XYZ does not apply to the set ABC because DEF\"?) - otherwise it would not be reproducible.\r\n\r\n[/quote]\r\n\r\n@Laurae The documentation doesn't have a fixed required format (it could be annotated images, a text narrative, code, equations, or any combination thereof), but reproducibility is the operative word. The host should be able to follow and understand your methodology based on the report. You may rely on commonalities in your methodology to avoid having to document every single image. However, if your approach is entirely manual and different for every image, you'll end up needing more documentation to make it reproducible.", "121280": "[quote=gk43;120940]\r\n\r\nAre you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed? \r\n\r\n[/quote]\r\n\r\nAs the prize winners will have to hand out documentation to reproduce the whole steps they did, I guess if there is a conflict during reproduction of the steps they will be deleted from the LB.\r\n\r\n@William Cukierski: are you able to clarify how the manual steps require to be documented? I guess it should be per set of pictures, and not globally? (including all the exact steps used, along all the reasons \"the steps used in set XYZ does not apply to the set ABC because DEF\"?) - otherwise it would not be reproducible.\r\n\r\n[quote=Abhimanyu Dikshit;120886]\r\n\r\nIs there any weightage to number of submissions while breaking ties?\r\n\r\n[/quote]\r\n\r\nNo, the rules determine the winners by the submission time. For instance, if you selected the earliest 1.000 Private LB submission among all the 1.000 Private LB submissions of others, you would win.\r\n\r\n[quote=Humberto Brandão;121257]\r\n\r\nhttps://www.kaggle.com/forums/f/15/kaggle-forum/t/21203/we-need-to-talk-about-data-leak\r\n\r\n[/quote]\r\n\r\nThere is no data leak... the way to get 1.00 was already explained numerous times (hand labeling).", "120940": "Are you saying that the leaders with scores of 1.0 visually determined and manually labeled the > 1000 test images? How can that process be applied to the withheld part of a test image set? Doesn't the competition evaluate the technique by applying it to more images (a withheld set)? If so, how can manual labeling be evaluated outside of the image set that the analysts reviewed? \r\n", "120886": "Is there any weightage to number of submissions while breaking ties?", "120710": "what if there are 4 team reaching 1.00 in private LB?\r\n\r\n[quote=William Cukierski;120648]\r\n\r\nWe do not intend to close or alter the competition at this point\\*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).\r\n\r\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods. \r\n\r\nCongrats on the progress so far!\r\n\r\n\\* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.\r\n\r\n[/quote]\r\n", "120648": "We do not intend to close or alter the competition at this point\\*. There may be three perfect scores on the public leaderboard, but that does not imply perfection on the private leaderboard or in submission selections! You should continue to work on the competition (this is not a statement on the private scores of the top teams; I would say the same thing if 20 people had a perfect private score).\r\n\r\nAlso, please keep in mind that, although we’ve allowed hand annotation and the final determinate is the private leaderboard score, we will still have to ensure that people implemented a repeatable and documented process using freely available methods. \r\n\r\nCongrats on the progress so far!\r\n\r\n\\* usual disclaimer: competitions are subject to change if we receive evidence of leakage or behavior that isn't in the spirit of the competition.", "120620": "I agree with @the1owl and @smota, I would prefer if they left it up. I am trying a pure ML solution and even though not in the running for the prize its of value to be able to try and improve my method.", "120608": "Yeah. The majority of the test data is in a totally separate area, and the test sets that you can actually solve with overlapping training sets are not included in the public LB.", "120607": "I am not agree to close competition. Maybe admins can notice us when the prizes are taken, but I am learning a lot and it is very interesting to follow a competition so different like this. \r\n\r\nAnd, by the way, congrats to all 1 scores.", "120601": "I guess so too, the admin can check their private score and if they are all 1.0 now (I bet it should be highly close since there is not much overfitting or anything like that in this competition) then the admin can notice us that the prize are already taken. :D", "120596": "If there are three perfect private scores, sure.\r\n\r\nThe train data only overlaps with the Long Beach portion of the test data, which is not a high percentage.", "120594": "[quote=olegpolivin;120586]\r\n\r\nAlso, has someone tried hand labelling using the train data? Actually, many pictures in train and test data intersect, so that you can put the order to test data being sure of it. \r\n\r\nAnyway, I am very eager to know the results of the private LB, and to have some reaction from Draper: what did they expect, what was the objective of allowing hand labelling. \r\n\r\n[/quote]\r\n\r\nIf they allowed hand labeling, the reason was because it might be too difficult for an algorithm to solve the problem automatically (else you would have made a breakthrough in research for that specific domain).\r\n\r\nThe most probable hypotheses for allowing hand labeling are:\r\n\r\n* The need for a proper framework on how to order chronologically aerial imagery (this is why you need to provide **the whole documentation for each hand labeled image so it can be reproduced**)\r\n* The need of compare potentially contradictory frameworks on how to order chronologically aerial imagery (comparison of researched frameworks, etc.)\r\n* And potentially turning a working framework into a ML code by themselves\r\n\r\nI suppose the three top teams with 1.00 score have already documented in depth all the manual steps they used (I can't imagine the size of the PDF :p). As time goes there could be satellite images that may not be accessible publicly anymore (ex: MapBox, etc.), therefore the steps might not be reproducible.\r\n\r\nAnd not only all pictures can be found online (search for \"GIS aerial imagery\" and you should find it easily), but not every picture is following the same framework to be localized and analyzed to make a chronological order.\r\n\r\n> Given the unique nature of this competition design, please note a\r\n> couple key changes to Kaggle's standard rules, listed below: \r\n> \r\n> * Hand labeling and human prediction are permitted.\r\n> * Winners must deliver documentation that clearly outlines the manual and automatic steps required to create the winning submission.\r\n> * Use of external data is permitted.", "125252": "", "125144": "", "120618": "" } }Given the unique nature of this competition design, please note a\n couple key changes to Kaggle's standard rules, listed below:
\n \n\n
\n- Hand labeling and human prediction are permitted.
\n- Winners must deliver documentation that clearly outlines the manual and automatic steps required to create the winning submission.
\n- Use of external data is permitted.
\n