I've been focusing on getting more data for the competition:
\n\n$ du -h\n9,1G ./samsung_s4\n1,7G ./nexus_5x\n9,0G ./nexus_6\n5,7G ./iphone_4s\n6,1G ./moto_x\n13G ./moto_maxx\n8,4G ./samsung_note3\n16G ./sony_nex7\n6,6G ./iphone_6\n4,6G ./htc_m7\n79G .\nAlthough it may seem relatively big in size you need to filter out images with low compression quality (I discard all those whose JPG compression quality is <=93) as well as camera models which to do not match the camera models (as reported in EXIF data) in the training set.
\n\nAt the end:
\n\nTraining set distribution:\n HTC-1-M7: 4705 (15.2%)\n iPhone-6: 3088 (10.0%)\n Motorola-Droid-Maxx: 4685 (15.2%)\n Motorola-X: 252 (00.8%)\n Samsung-Galaxy-S4: 4311 (14.0%)\n iPhone-4s: 3205 (10.4%)\n LG-Nexus-5x: 1072 (03.5%)\n Motorola-Nexus-6: 2941 (09.5%)\n Samsung-Galaxy-Note3: 3324 (10.8%)\n Sony-NEX-7: 3319 (10.7%)\nValidation set distribution:\n HTC-1-M7: 56 (10.0%)\n iPhone-6: 56 (10.0%)\n Motorola-Droid-Maxx: 56 (10.0%)\n Motorola-X: 56 (10.0%)\n Samsung-Galaxy-S4: 56 (10.0%)\n iPhone-4s: 56 (10.0%)\n LG-Nexus-5x: 56 (10.0%)\n Motorola-Nexus-6: 56 (10.0%)\n Samsung-Galaxy-Note3: 56 (10.0%)\n Sony-NEX-7: 56 (10.0%)\nTotal training items: 30902\nTotal validation items: 560\nAs you can see the training set size is relatively decent for all classes EXCEPT Motorola-X. Since I want to have a balanced validation set it makes the size of the validation very small (1.8% of total samples... kind of a joke).
The problem with Motorola-X is that the training set model is XT1096 (2nd gen moto-x) however most of the images you can get online (at scale) are for XT1060 (1st gen moto-x).
Have you found a good source for XT1096 images?
I've been focusing on getting more data for the competition:
\n\n$ du -h\n9,1G ./samsung_s4\n1,7G ./nexus_5x\n9,0G ./nexus_6\n5,7G ./iphone_4s\n6,1G ./moto_x\n13G ./moto_maxx\n8,4G ./samsung_note3\n16G ./sony_nex7\n6,6G ./iphone_6\n4,6G ./htc_m7\n79G .\nAlthough it may seem relatively big in size you need to filter out images with low compression quality (I discard all those whose JPG compression quality is <=93) as well as camera models which to do not match the camera models (as reported in EXIF data) in the training set.
\n\nAt the end:
\n\nTraining set distribution:\n HTC-1-M7: 4705 (15.2%)\n iPhone-6: 3088 (10.0%)\n Motorola-Droid-Maxx: 4685 (15.2%)\n Motorola-X: 252 (00.8%)\n Samsung-Galaxy-S4: 4311 (14.0%)\n iPhone-4s: 3205 (10.4%)\n LG-Nexus-5x: 1072 (03.5%)\n Motorola-Nexus-6: 2941 (09.5%)\n Samsung-Galaxy-Note3: 3324 (10.8%)\n Sony-NEX-7: 3319 (10.7%)\nValidation set distribution:\n HTC-1-M7: 56 (10.0%)\n iPhone-6: 56 (10.0%)\n Motorola-Droid-Maxx: 56 (10.0%)\n Motorola-X: 56 (10.0%)\n Samsung-Galaxy-S4: 56 (10.0%)\n iPhone-4s: 56 (10.0%)\n LG-Nexus-5x: 56 (10.0%)\n Motorola-Nexus-6: 56 (10.0%)\n Samsung-Galaxy-Note3: 56 (10.0%)\n Sony-NEX-7: 56 (10.0%)\nTotal training items: 30902\nTotal validation items: 560\nAs you can see the training set size is relatively decent for all classes EXCEPT Motorola-X. Since I want to have a balanced validation set it makes the size of the validation very small (1.8% of total samples... kind of a joke).
The problem with Motorola-X is that the training set model is XT1096 (2nd gen moto-x) however most of the images you can get online (at scale) are for XT1060 (1st gen moto-x).
Have you found a good source for XT1096 images?
I'd suggest broadening search to include any Moto X variant: XT109*. There is no good source of such photos but long-long nights of requests to flickr and filtering results by above criteria are able to get some amount.
", "rawMarkdown": "I'd suggest broadening search to include any Moto X variant: XT109*. There is no good source of such photos but long-long nights of requests to flickr and filtering results by above criteria are able to get some amount.", "votes": null }, { "id": "278507", "postDate": "02/06/2018 10:50:50", "content": "Once comp is over I will open source my dataset building script... which uses a few undocumented API calls to help pull all images automatically, EXCEPT the freaking exact model number (Flickr aliases many different models into moto_x). My moto_x folder is crap:
$ find * -exec sh -c 'exif \"{}\" 2>/dev/null' \\; | grep Model | sort | uniq -c\n 2027 Model |XT1060\n 1 Model |XT1095
\n\nSo totally useless for now.
", "rawMarkdown": "Once comp is over I will open source my dataset building script... which uses a few undocumented API calls to help pull all images automatically, EXCEPT the freaking exact model number (Flickr aliases many different models into `moto_x`). My moto_x folder is crap:\n\n$ find * -exec sh -c 'exif \"{}\" 2>/dev/null' \\; | grep Model | sort | uniq -c\n 2027 Model |XT1060\n 1 Model |XT1095\n\nSo totally useless for now.", "votes": null }, { "id": "278531", "postDate": "02/06/2018 12:06:42", "content": "You may take a look at Yandex.Fotki photo service ;)
", "rawMarkdown": "You may take a look at Yandex.Fotki photo service ;)", "votes": null }, { "id": "278540", "postDate": "02/06/2018 12:29:16", "content": "Just FYI, Kaggle usually requires that you disclose ALL of your external models and data BEFORE the competition is over.
", "rawMarkdown": "Just FYI, Kaggle usually requires that you disclose ALL of your external models and data BEFORE the competition is over.", "votes": null }, { "id": "278541", "postDate": "02/06/2018 12:43:11", "content": "Thanks man! \nI knew the day I had to start thinking about learning Russian will come :-)
", "rawMarkdown": "Thanks man! \nI knew the day I had to start thinking about learning Russian will come :-)", "votes": null }, { "id": "278542", "postDate": "02/06/2018 12:44:21", "content": "Is this the case for this one? It says:
\n\nExternal Data Use: The following provision supersedes General Rules Section 7.B. below: “You may use data, other than the Competition Data, as allowed on the Competition Website to develop and test your models and Submissions; provided, you have the right and authority to use such external data for the purposes of the Competition, and to share such data with Sponsor and Kaggle as may be required.\nReading that I understand CC licenses are OK, there's been some questions for @inversion but he seems MIA :-(
", "rawMarkdown": "Is this the case for this one? It says:\n\n External Data Use: The following provision supersedes General Rules Section 7.B. below: “You may use data, other than the Competition Data, as allowed on the Competition Website to develop and test your models and Submissions; provided, you have the right and authority to use such external data for the purposes of the Competition, and to share such data with Sponsor and Kaggle as may be required.\n\nReading that I understand CC licenses are OK, there's been some questions for @inversion but he seems MIA :-(", "votes": null }, { "id": "278545", "postDate": "02/06/2018 12:50:55", "content": "From what I understand the issue there concerns CC license, and NOT the general Kaggle rule of sharing all of your external data before the competition is over.
", "rawMarkdown": "From what I understand the issue there concerns CC license, and NOT the general Kaggle rule of sharing all of your external data before the competition is over.", "votes": null }, { "id": "278549", "postDate": "02/06/2018 13:00:45", "content": "Where is that general Kaggle rule? I understand the competition rules are described under the tab \"Rules\". While typically in Kaggle competitions require sharing of external data, I do not see it written anywhere for this specific one.
\n\n(btw, there seems to be a mistake in the specific External Data Use above as it supersedes General Rules Section 7.B, and I believe it should be General Rules Section 7.C).
I agree with Bojan. We need to share in the forum external resources to other teams earlier in the competition like what Gleb did. Kaggle competitions are modeling competitions not data collection competitions. I personally follow this rule for all the competitions anyway. Only the Admins have the right to explain but they are not very active in this one.
", "rawMarkdown": "I agree with Bojan. We need to share in the forum external resources to other teams earlier in the competition like what Gleb did. Kaggle competitions are modeling competitions not data collection competitions. I personally follow this rule for all the competitions anyway. Only the Admins have the right to explain but they are not very active in this one.", "votes": null }, { "id": "278593", "postDate": "02/06/2018 14:37:54", "content": "OK guys, I'll share in the spirit of the competition! :-) And... if you get moto_x samples... please share too! :-)
", "rawMarkdown": "OK guys, I'll share in the spirit of the competition! :-) And... if you get moto_x samples... please share too! :-)", "votes": null }, { "id": "278613", "postDate": "02/06/2018 15:31:48", "content": "Awesome! How to get so much data?
", "rawMarkdown": "Awesome! How to get so much data?", "votes": null }, { "id": "278628", "postDate": "02/06/2018 16:06:50", "content": "Hi Everyone! Sorry for missing this thread. When I competed on Kaggle, I used to think it was crazy being in 3 competitions at once. Now, I get the pleasure of focusing on 7-10 competitions concurrently (which includes those coming down the pipeline.)
\n\nHere's the deal . . . the requirement that external data be shared on the forums should have been included in the rules, but somehow got dropped. As such, we cannot require anyone to share external data, especially with just a couple of days left in the competition. On the other hand, we always require this, and it is very much in the spirit of the intent of the competition. So, we strongly urge individuals to share their data openly.
\n\nSorry for the confusion this caused.
", "rawMarkdown": "Hi Everyone! Sorry for missing this thread. When I competed on Kaggle, I used to think it was crazy being in 3 competitions at once. Now, I get the pleasure of focusing on 7-10 competitions concurrently (which includes those coming down the pipeline.)\n\nHere's the deal . . . the requirement that external data be shared on the forums _should have been_ included in the rules, but somehow got dropped. As such, we cannot require anyone to share external data, especially with just a couple of days left in the competition. On the other hand, we _always_ require this, and it is very much in the spirit of the intent of the competition. So, we strongly urge individuals to share their data openly.\n\nSorry for the confusion this caused.", "votes": null }, { "id": "278634", "postDate": "02/06/2018 16:23:31", "content": "\n\n\nsomehow got dropped
\n
wow, that sucks
", "rawMarkdown": "> somehow got dropped\n\nwow, that sucks", "votes": null } ], "comments": [ { "id": 278506, "author_name": "ceperaang", "author_url": "", "post_date": "02/06/2018 10:40:46", "content": "I'd suggest broadening search to include any Moto X variant: XT109*. There is no good source of such photos but long-long nights of requests to flickr and filtering results by above criteria are able to get some amount.
", "votes": null, "replies": [ { "id": 278507, "author_name": "antorsae", "author_url": "", "post_date": "02/06/2018 10:50:50", "content": "Once comp is over I will open source my dataset building script... which uses a few undocumented API calls to help pull all images automatically, EXCEPT the freaking exact model number (Flickr aliases many different models into moto_x). My moto_x folder is crap:
$ find * -exec sh -c 'exif \"{}\" 2>/dev/null' \\; | grep Model | sort | uniq -c\n 2027 Model |XT1060\n 1 Model |XT1095
\n\nSo totally useless for now.
", "votes": null, "replies": [] }, { "id": 278531, "author_name": "ceperaang", "author_url": "", "post_date": "02/06/2018 12:06:42", "content": "You may take a look at Yandex.Fotki photo service ;)
", "votes": null, "replies": [] }, { "id": 278541, "author_name": "antorsae", "author_url": "", "post_date": "02/06/2018 12:43:11", "content": "Thanks man! \nI knew the day I had to start thinking about learning Russian will come :-)
", "votes": null, "replies": [] } ] }, { "id": 278540, "author_name": "tunguz", "author_url": "", "post_date": "02/06/2018 12:29:16", "content": "Just FYI, Kaggle usually requires that you disclose ALL of your external models and data BEFORE the competition is over.
", "votes": null, "replies": [ { "id": 278542, "author_name": "antorsae", "author_url": "", "post_date": "02/06/2018 12:44:21", "content": "Is this the case for this one? It says:
\n\nExternal Data Use: The following provision supersedes General Rules Section 7.B. below: “You may use data, other than the Competition Data, as allowed on the Competition Website to develop and test your models and Submissions; provided, you have the right and authority to use such external data for the purposes of the Competition, and to share such data with Sponsor and Kaggle as may be required.\nReading that I understand CC licenses are OK, there's been some questions for @inversion but he seems MIA :-(
", "votes": null, "replies": [] }, { "id": 278545, "author_name": "tunguz", "author_url": "", "post_date": "02/06/2018 12:50:55", "content": "From what I understand the issue there concerns CC license, and NOT the general Kaggle rule of sharing all of your external data before the competition is over.
", "votes": null, "replies": [] }, { "id": 278549, "author_name": "antorsae", "author_url": "", "post_date": "02/06/2018 13:00:45", "content": "Where is that general Kaggle rule? I understand the competition rules are described under the tab \"Rules\". While typically in Kaggle competitions require sharing of external data, I do not see it written anywhere for this specific one.
\n\n(btw, there seems to be a mistake in the specific External Data Use above as it supersedes General Rules Section 7.B, and I believe it should be General Rules Section 7.C).
I agree with Bojan. We need to share in the forum external resources to other teams earlier in the competition like what Gleb did. Kaggle competitions are modeling competitions not data collection competitions. I personally follow this rule for all the competitions anyway. Only the Admins have the right to explain but they are not very active in this one.
", "votes": null, "replies": [] }, { "id": 278593, "author_name": "antorsae", "author_url": "", "post_date": "02/06/2018 14:37:54", "content": "OK guys, I'll share in the spirit of the competition! :-) And... if you get moto_x samples... please share too! :-)
", "votes": null, "replies": [] }, { "id": 278628, "author_name": "inversion", "author_url": "", "post_date": "02/06/2018 16:06:50", "content": "Hi Everyone! Sorry for missing this thread. When I competed on Kaggle, I used to think it was crazy being in 3 competitions at once. Now, I get the pleasure of focusing on 7-10 competitions concurrently (which includes those coming down the pipeline.)
\n\nHere's the deal . . . the requirement that external data be shared on the forums should have been included in the rules, but somehow got dropped. As such, we cannot require anyone to share external data, especially with just a couple of days left in the competition. On the other hand, we always require this, and it is very much in the spirit of the intent of the competition. So, we strongly urge individuals to share their data openly.
\n\nSorry for the confusion this caused.
", "votes": null, "replies": [] }, { "id": 278634, "author_name": "", "author_url": "", "post_date": "02/06/2018 16:23:31", "content": "\n\n\nsomehow got dropped
\n
wow, that sucks
", "votes": null, "replies": [] } ] }, { "id": 278613, "author_name": "vivalavida1989", "author_url": "", "post_date": "02/06/2018 15:31:48", "content": "Awesome! How to get so much data?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "278485": "I've been focusing on getting more data for the competition:\n\n $ du -h\n 9,1G ./samsung_s4\n 1,7G ./nexus_5x\n 9,0G ./nexus_6\n 5,7G ./iphone_4s\n 6,1G ./moto_x\n 13G ./moto_maxx\n 8,4G ./samsung_note3\n 16G ./sony_nex7\n 6,6G ./iphone_6\n 4,6G ./htc_m7\n 79G .\n\nAlthough it may seem relatively big in size you need to filter out images with low compression quality (I discard all those whose JPG compression quality is <=93) as well as camera models which to do not match the camera models (as reported in EXIF data) in the training set.\n\nAt the end:\n\n Training set distribution:\n HTC-1-M7: 4705 (15.2%)\n iPhone-6: 3088 (10.0%)\n Motorola-Droid-Maxx: 4685 (15.2%)\n Motorola-X: 252 (00.8%)\n Samsung-Galaxy-S4: 4311 (14.0%)\n iPhone-4s: 3205 (10.4%)\n LG-Nexus-5x: 1072 (03.5%)\n Motorola-Nexus-6: 2941 (09.5%)\n Samsung-Galaxy-Note3: 3324 (10.8%)\n Sony-NEX-7: 3319 (10.7%)\n Validation set distribution:\n HTC-1-M7: 56 (10.0%)\n iPhone-6: 56 (10.0%)\n Motorola-Droid-Maxx: 56 (10.0%)\n Motorola-X: 56 (10.0%)\n Samsung-Galaxy-S4: 56 (10.0%)\n iPhone-4s: 56 (10.0%)\n LG-Nexus-5x: 56 (10.0%)\n Motorola-Nexus-6: 56 (10.0%)\n Samsung-Galaxy-Note3: 56 (10.0%)\n Sony-NEX-7: 56 (10.0%)\n Total training items: 30902\n Total validation items: 560\n\nAs you can see the training set size is relatively decent for all classes EXCEPT `Motorola-X`. Since I want to have a balanced validation set it makes the size of the validation very small (1.8% of total samples... kind of a joke).\n\nThe problem with `Motorola-X` is that the training set model is `XT1096` (2nd gen moto-x) however most of the images you can get online (at scale) are for `XT1060` (1st gen moto-x).\n\nHave you found a good source for `XT1096` images?", "278506": "I'd suggest broadening search to include any Moto X variant: XT109*. There is no good source of such photos but long-long nights of requests to flickr and filtering results by above criteria are able to get some amount.", "278507": "Once comp is over I will open source my dataset building script... which uses a few undocumented API calls to help pull all images automatically, EXCEPT the freaking exact model number (Flickr aliases many different models into `moto_x`). My moto_x folder is crap:\n\n$ find * -exec sh -c 'exif \"{}\" 2>/dev/null' \\; | grep Model | sort | uniq -c\n 2027 Model |XT1060\n 1 Model |XT1095\n\nSo totally useless for now.", "278531": "You may take a look at Yandex.Fotki photo service ;)", "278540": "Just FYI, Kaggle usually requires that you disclose ALL of your external models and data BEFORE the competition is over.", "278541": "Thanks man! \nI knew the day I had to start thinking about learning Russian will come :-)", "278542": "Is this the case for this one? It says:\n\n External Data Use: The following provision supersedes General Rules Section 7.B. below: “You may use data, other than the Competition Data, as allowed on the Competition Website to develop and test your models and Submissions; provided, you have the right and authority to use such external data for the purposes of the Competition, and to share such data with Sponsor and Kaggle as may be required.\n\nReading that I understand CC licenses are OK, there's been some questions for @inversion but he seems MIA :-(", "278545": "From what I understand the issue there concerns CC license, and NOT the general Kaggle rule of sharing all of your external data before the competition is over.", "278549": "Where is that general Kaggle rule? I understand the competition rules are described under the tab \"Rules\". While typically in Kaggle competitions require sharing of external data, I do not see it written anywhere for this specific one.\n\n(btw, there seems to be a mistake in the specific `External Data Use` above as it supersedes General Rules Section 7.B, and I believe it should be General Rules Section 7.C).", "278576": "I agree with Bojan. We need to share in the forum external resources to other teams earlier in the competition like what Gleb did. Kaggle competitions are modeling competitions not data collection competitions. I personally follow this rule for all the competitions anyway. Only the Admins have the right to explain but they are not very active in this one.", "278593": "OK guys, I'll share in the spirit of the competition! :-) And... if you get moto_x samples... please share too! :-)", "278613": "Awesome! How to get so much data?", "278628": "Hi Everyone! Sorry for missing this thread. When I competed on Kaggle, I used to think it was crazy being in 3 competitions at once. Now, I get the pleasure of focusing on 7-10 competitions concurrently (which includes those coming down the pipeline.)\n\nHere's the deal . . . the requirement that external data be shared on the forums _should have been_ included in the rules, but somehow got dropped. As such, we cannot require anyone to share external data, especially with just a couple of days left in the competition. On the other hand, we _always_ require this, and it is very much in the spirit of the intent of the competition. So, we strongly urge individuals to share their data openly.\n\nSorry for the confusion this caused.", "278634": "> somehow got dropped\n\nwow, that sucks" }, "source": "meta" }