{
  "id": 49299,
  "title": "2nd place solution with GPU muscles",
  "url": "/competitions/sp-society-camera-model-identification/writeups/ods-ai-gpu-muscles-spcup-eligible-2nd-place-soluti",
  "author_name": "",
  "post_date": "2018-02-09T02:15:02.902071500Z",
  "votes": 71,
  "comment_count": 25,
  "views": 0,
  "content": "<p><strong>tldr</strong>: \nWe downloaded a about 500+ GB photos, trained 9+ imagenet-like models with 3 version of the pipeline. Finally, we averaged 27 checkpoints with geometric mean.</p>\n\n<p><strong>Key components of a good solution, by priorities:</strong></p>\n\n<ol>\n<li>Large and clean external dataset </li>\n<li>Consistant local validation</li>\n<li>Classic competitive approach to learning models</li>\n<li>Diverse models</li>\n</ol>\n\n<p><strong>Data mining</strong>:  We downloaded 500+ Gb photos from various resources: Flickr, Yandex.Fotki, Wikipedia Commons, mobile reviews. In addition, on the last night we downloaded 22k more photos with urls from Flickr gathered by <a href=\"https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49064\">Andres Torrubia</a>. </p>\n\n<p><strong>Filtering data</strong>: Lightroom/photoshop/etc processing could eliminate all the information about the camera. We filtered on: model, resolution, quality of jpeg compression, software of processing.\nAfter filtering of the training and validation, the datasets looked like this:\n<img src=\"https://pp.userapi.com/c824701/v824701072/a95a8/Hgl68WyLzyU.jpg\" alt=\"enter image description here\">\nWe took the validation from <a href=\"https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/47235\">Gleb’s post</a>, but replaced iPhone 6 plus pictures in it by iPhone 6 ones.</p>\n\n<p><strong>Training models</strong>: Our code is based on pytorch version of <a href=\"https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/48679\">Andres solution</a>.\nFor all the models, the binary flag is_manip was used as an additional feature for the classifier of a net. All models had an input of 480. For the majority of our models, five crops + flip photo orientation (10TTA) was applied to the pictures, for D4 models five crops + the whole group of D4 were applied (40TTA); then geometric mean was applied to the predictions.\nUseful tricks:\n1. Adam, reducing LR on plateau with patience 2-4\n2. Cyclic LR with SGD\n3. Pseudo-labeling\n4. Averaging 3 checkpoints with the best loss for validation</p>\n\n<p>Also on the last day we trained several models with D4 augmentations, finetune from best checkpoint of previous pipeline. We didn’t have submit to check all models on LB, but the result on one was impressive.</p>\n\n<p>The final ensemble of models looked like this:\n<img src=\"https://pp.userapi.com/c824701/v824701072/a95b9/dg36oZu6_B8.jpg\" alt=\"enter image description here\">\nLogging models:\n <img src=\"https://pp.userapi.com/c824701/v824701072/a95b2/aMQ2rKhWAI4.jpg\" alt=\"enter image description here\">\n<strong>Averaging</strong>: We tried different approaches with class balancing and dropping the missing classes according to the probabilities. But in the end geometric average of 27 checkpoints of different models is the best.</p>\n\n<p><strong>What did not work?</strong>\nDemosizing\n<img src=\"https://pp.userapi.com/c824701/v824701072/a95c0/sJpJNsYtDcg.jpg\" alt=\"enter image description here\">\nCameras fix on the matrix in a particular pixel the intensity of only one color, and the rest of the colors are restored to neighboring pixels, and the recovery algorithm for different camera manufacturers is different.\nThe idea was to calculate the difference between the original image and the image obtained by one of the standard demosayzingov.\nThe inertia of the differences is to train the neural network.</p>\n\n<p><strong>Hardware:</strong></p>\n\n<ul>\n<li>i7 7700k, 64gb, 2x 1080 (just for development) </li>\n<li>i7 6700k, 32gb, 2x    Titan X (Maxwell)</li>\n<li>i7 5930K, 32GB, 3x1080Ti</li>\n<li>Xeon 2696v3, 64gb, 4x1080Ti</li>\n<li>i7 3770k, 16gb, 2x 1080Ti</li>\n</ul>",
  "messages": [
    {
      "id": "279968",
      "postDate": "02/09/2018 02:15:02",
      "content": "<p><strong>tldr</strong>: \nWe downloaded a about 500+ GB photos, trained 9+ imagenet-like models with 3 version of the pipeline. Finally, we averaged 27 checkpoints with geometric mean.</p>\n\n<p><strong>Key components of a good solution, by priorities:</strong></p>\n\n<ol>\n<li>Large and clean external dataset </li>\n<li>Consistant local validation</li>\n<li>Classic competitive approach to learning models</li>\n<li>Diverse models</li>\n</ol>\n\n<p><strong>Data mining</strong>:  We downloaded 500+ Gb photos from various resources: Flickr, Yandex.Fotki, Wikipedia Commons, mobile reviews. In addition, on the last night we downloaded 22k more photos with urls from Flickr gathered by <a href=\"https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49064\">Andres Torrubia</a>. </p>\n\n<p><strong>Filtering data</strong>: Lightroom/photoshop/etc processing could eliminate all the information about the camera. We filtered on: model, resolution, quality of jpeg compression, software of processing.\nAfter filtering of the training and validation, the datasets looked like this:\n<img src=\"https://pp.userapi.com/c824701/v824701072/a95a8/Hgl68WyLzyU.jpg\" alt=\"enter image description here\">\nWe took the validation from <a href=\"https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/47235\">Gleb’s post</a>, but replaced iPhone 6 plus pictures in it by iPhone 6 ones.</p>\n\n<p><strong>Training models</strong>: Our code is based on pytorch version of <a href=\"https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/48679\">Andres solution</a>.\nFor all the models, the binary flag is_manip was used as an additional feature for the classifier of a net. All models had an input of 480. For the majority of our models, five crops + flip photo orientation (10TTA) was applied to the pictures, for D4 models five crops + the whole group of D4 were applied (40TTA); then geometric mean was applied to the predictions.\nUseful tricks:\n1. Adam, reducing LR on plateau with patience 2-4\n2. Cyclic LR with SGD\n3. Pseudo-labeling\n4. Averaging 3 checkpoints with the best loss for validation</p>\n\n<p>Also on the last day we trained several models with D4 augmentations, finetune from best checkpoint of previous pipeline. We didn’t have submit to check all models on LB, but the result on one was impressive.</p>\n\n<p>The final ensemble of models looked like this:\n<img src=\"https://pp.userapi.com/c824701/v824701072/a95b9/dg36oZu6_B8.jpg\" alt=\"enter image description here\">\nLogging models:\n <img src=\"https://pp.userapi.com/c824701/v824701072/a95b2/aMQ2rKhWAI4.jpg\" alt=\"enter image description here\">\n<strong>Averaging</strong>: We tried different approaches with class balancing and dropping the missing classes according to the probabilities. But in the end geometric average of 27 checkpoints of different models is the best.</p>\n\n<p><strong>What did not work?</strong>\nDemosizing\n<img src=\"https://pp.userapi.com/c824701/v824701072/a95c0/sJpJNsYtDcg.jpg\" alt=\"enter image description here\">\nCameras fix on the matrix in a particular pixel the intensity of only one color, and the rest of the colors are restored to neighboring pixels, and the recovery algorithm for different camera manufacturers is different.\nThe idea was to calculate the difference between the original image and the image obtained by one of the standard demosayzingov.\nThe inertia of the differences is to train the neural network.</p>\n\n<p><strong>Hardware:</strong></p>\n\n<ul>\n<li>i7 7700k, 64gb, 2x 1080 (just for development) </li>\n<li>i7 6700k, 32gb, 2x    Titan X (Maxwell)</li>\n<li>i7 5930K, 32GB, 3x1080Ti</li>\n<li>Xeon 2696v3, 64gb, 4x1080Ti</li>\n<li>i7 3770k, 16gb, 2x 1080Ti</li>\n</ul>",
      "rawMarkdown": "**tldr**: \nWe downloaded a about 500+ GB photos, trained 9+ imagenet-like models with 3 version of the pipeline. Finally, we averaged 27 checkpoints with geometric mean.\n\n**Key components of a good solution, by priorities:**\n\n 1. Large and clean external dataset \n 2. Consistant local validation\n 3. Classic competitive approach to learning models\n 4. Diverse models\n\n**Data mining**:  We downloaded 500+ Gb photos from various resources: Flickr, Yandex.Fotki, Wikipedia Commons, mobile reviews. In addition, on the last night we downloaded 22k more photos with urls from Flickr gathered by [Andres Torrubia][1]. \n\n**Filtering data**: Lightroom/photoshop/etc processing could eliminate all the information about the camera. We filtered on: model, resolution, quality of jpeg compression, software of processing.\nAfter filtering of the training and validation, the datasets looked like this:\n![enter image description here][2]\nWe took the validation from [Gleb’s post][3], but replaced iPhone 6 plus pictures in it by iPhone 6 ones.\n\n**Training models**: Our code is based on pytorch version of [Andres solution][4].\nFor all the models, the binary flag is_manip was used as an additional feature for the classifier of a net. All models had an input of 480. For the majority of our models, five crops + flip photo orientation (10TTA) was applied to the pictures, for D4 models five crops + the whole group of D4 were applied (40TTA); then geometric mean was applied to the predictions.\nUseful tricks:\n1. Adam, reducing LR on plateau with patience 2-4\n2. Cyclic LR with SGD\n3. Pseudo-labeling\n4. Averaging 3 checkpoints with the best loss for validation\n\nAlso on the last day we trained several models with D4 augmentations, finetune from best checkpoint of previous pipeline. We didn’t have submit to check all models on LB, but the result on one was impressive.\n\nThe final ensemble of models looked like this:\n![enter image description here][5]\nLogging models:\n ![enter image description here][6]\n**Averaging**: We tried different approaches with class balancing and dropping the missing classes according to the probabilities. But in the end geometric average of 27 checkpoints of different models is the best.\n\n**What did not work?**\nDemosizing\n![enter image description here][7]\nCameras fix on the matrix in a particular pixel the intensity of only one color, and the rest of the colors are restored to neighboring pixels, and the recovery algorithm for different camera manufacturers is different.\nThe idea was to calculate the difference between the original image and the image obtained by one of the standard demosayzingov.\nThe inertia of the differences is to train the neural network.\n\n**Hardware:**\n\n - i7 7700k, 64gb, 2x 1080 (just for development) \n - i7 6700k, 32gb, 2x    Titan X (Maxwell)\n - i7 5930K, 32GB, 3x1080Ti\n - Xeon 2696v3, 64gb, 4x1080Ti\n - i7 3770k, 16gb, 2x 1080Ti\n\n  [1]: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49064\n  [2]: https://pp.userapi.com/c824701/v824701072/a95a8/Hgl68WyLzyU.jpg\n  [3]: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/47235\n  [4]: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/48679\n  [5]: https://pp.userapi.com/c824701/v824701072/a95b9/dg36oZu6_B8.jpg\n  [6]: https://pp.userapi.com/c824701/v824701072/a95b2/aMQ2rKhWAI4.jpg\n  [7]: https://pp.userapi.com/c824701/v824701072/a95c0/sJpJNsYtDcg.jpg",
      "votes": null
    },
    {
      "id": "279974",
      "postDate": "02/09/2018 02:27:27",
      "content": "<p>Amazing work!</p>",
      "rawMarkdown": "Amazing work!",
      "votes": null
    },
    {
      "id": "279978",
      "postDate": "02/09/2018 02:44:00",
      "content": "<p>Thank you. This was fun to read as someone new to Kaggle.</p>\n\n<blockquote>\n  <p>All models had an input of 480.</p>\n</blockquote>\n\n<p>Did you crop from the 512x512 center region or from the entire image?</p>\n\n<blockquote>\n  <p>D4 augmentations</p>\n</blockquote>\n\n<p>Does this refer to compression, resizing, gamma correction, and transpose? Where does the term \"D4\" come from?</p>",
      "rawMarkdown": "Thank you. This was fun to read as someone new to Kaggle.\n\n&gt; All models had an input of 480.\n\nDid you crop from the 512x512 center region or from the entire image?\n\n&gt; D4 augmentations\n\nDoes this refer to compression, resizing, gamma correction, and transpose? Where does the term \"D4\" come from?",
      "votes": null
    },
    {
      "id": "279987",
      "postDate": "02/09/2018 03:10:03",
      "content": "<p>Wow! Congratulation!</p>",
      "rawMarkdown": "Wow! Congratulation!",
      "votes": null
    },
    {
      "id": "280110",
      "postDate": "02/09/2018 11:48:57",
      "content": "<p>Congratulations!</p>\n\n<p>The hardware is powerful!  </p>\n\n<p>Why most the loss curves are periodically jumping very high? </p>\n\n<p>The validation score is not too high. Is it because the curve are only part of all? But the epic seems already goes beyond 60.</p>\n\n<p>For DenseNet201, what is the batch size used? It does not run in my computer even with a batch size of 1. :(</p>\n\n<p>Have you tried other ensembling methods instead of only geometric mean?</p>\n\n<p>How is dual path network in terms of accuracy, performance and memory consumption comparing to DenseNet or other networks?</p>\n\n<p>Two more questions: what is D4 and what is demosizing?</p>\n\n<p>Quite a few questions. But these are something I am curious to know. :)</p>",
      "rawMarkdown": "Congratulations!\n\nThe hardware is powerful!  \n\nWhy most the loss curves are periodically jumping very high? \n\nThe validation score is not too high. Is it because the curve are only part of all? But the epic seems already goes beyond 60.\n\nFor DenseNet201, what is the batch size used? It does not run in my computer even with a batch size of 1. :(\n\nHave you tried other ensembling methods instead of only geometric mean?\n\nHow is dual path network in terms of accuracy, performance and memory consumption comparing to DenseNet or other networks?\n\nTwo more questions: what is D4 and what is demosizing?\n\nQuite a few questions. But these are something I am curious to know. :)",
      "votes": null
    },
    {
      "id": "280117",
      "postDate": "02/09/2018 11:55:36",
      "content": "<p>We crop from  entire image. Term D4 comes from group theory. It means all rotation on 90 and flips.</p>",
      "rawMarkdown": "We crop from  entire image. Term D4 comes from group theory. It means all rotation on 90 and flips.",
      "votes": null
    },
    {
      "id": "280141",
      "postDate": "02/09/2018 13:13:43",
      "content": "<p>About D4: <a href=\"https://en.wikipedia.org/wiki/Dihedral_group\">https://en.wikipedia.org/wiki/Dihedral_group</a></p>",
      "rawMarkdown": "About D4: https://en.wikipedia.org/wiki/Dihedral_group",
      "votes": null
    },
    {
      "id": "280145",
      "postDate": "02/09/2018 13:23:39",
      "content": "<ol>\n<li>Loss curves seems jumpy due to scale and little amount of sample in\nvalidation set.  </li>\n<li>Almost all validation accuracy is about 0.98. Pretty\n    high for me.</li>\n<li>As far as I remember typically batch size is 24-40 for 4x\n    1080ti.</li>\n<li>We tried weighted mean, small class sorting. Doesn't work for\n    us.</li>\n<li>DPN is fantastic! Our best single model: dpn92 with pseudo\n    labeling, TTA, 3 checkpoints TTA: 0.987 (private) and 0.984 (public).</li>\n<li>About D4 answered above</li>\n</ol>",
      "rawMarkdown": "1. Loss curves seems jumpy due to scale and little amount of sample in\n    validation set.  \n 2. Almost all validation accuracy is about 0.98. Pretty\n        high for me.\n 3. As far as I remember typically batch size is 24-40 for 4x\n        1080ti.\n 4. We tried weighted mean, small class sorting. Doesn't work for\n        us.\n 5. DPN is fantastic! Our best single model: dpn92 with pseudo\n        labeling, TTA, 3 checkpoints TTA: 0.987 (private) and 0.984 (public).\n 6. About D4 answered above",
      "votes": null
    },
    {
      "id": "280170",
      "postDate": "02/09/2018 14:01:24",
      "content": "<p>What is DPN?</p>",
      "rawMarkdown": "What is DPN?",
      "votes": null
    },
    {
      "id": "280179",
      "postDate": "02/09/2018 14:17:13",
      "content": "<p>I would guess that demosizing is demosaicing (debayering) algorithm to remove Bayer pattern.</p>",
      "rawMarkdown": "I would guess that demosizing is demosaicing (debayering) algorithm to remove Bayer pattern.",
      "votes": null
    },
    {
      "id": "280181",
      "postDate": "02/09/2018 14:21:41",
      "content": "<p>DPN is dual path network. </p>",
      "rawMarkdown": "DPN is dual path network.",
      "votes": null
    },
    {
      "id": "280185",
      "postDate": "02/09/2018 14:28:48",
      "content": "<p>@n01z3 Thanks a bunch!</p>",
      "rawMarkdown": "n01z3 Thanks a bunch!",
      "votes": null
    },
    {
      "id": "280204",
      "postDate": "02/09/2018 14:59:17",
      "content": "<p>It's fantastic!</p>",
      "rawMarkdown": "It's fantastic!",
      "votes": null
    },
    {
      "id": "280336",
      "postDate": "02/09/2018 18:26:17",
      "content": "<p>Much data! so Wow!</p>",
      "rawMarkdown": "Much data! so Wow!",
      "votes": null
    },
    {
      "id": "280411",
      "postDate": "02/09/2018 23:09:17",
      "content": "<p>Thank you. Here's an image from wikipedia that explains D8 (not D4) visually:</p>\n\n<p><a href=\"https://en.wikipedia.org/wiki/Dihedral_group#/media/File:Dihedral8.png\">https://en.wikipedia.org/wiki/Dihedral_group#/media/File:Dihedral8.png</a></p>",
      "rawMarkdown": "Thank you. Here's an image from wikipedia that explains D8 (not D4) visually:\n\nhttps://en.wikipedia.org/wiki/Dihedral_group#/media/File:Dihedral8.png",
      "votes": null
    },
    {
      "id": "280421",
      "postDate": "02/09/2018 23:42:34",
      "content": "<p>DPN: <a href=\"https://arxiv.org/abs/1707.01629\">https://arxiv.org/abs/1707.01629</a></p>",
      "rawMarkdown": "DPN: https://arxiv.org/abs/1707.01629",
      "votes": null
    },
    {
      "id": "280464",
      "postDate": "02/10/2018 03:48:07",
      "content": "<p>Congratulations! \nwhat is your input of your reported models? the original image with 480x480 or the difference between the original image and the image obtained by one of the standard demosayzingov? If the difference was used, which one standard demosayzingov?</p>",
      "rawMarkdown": "Congratulations! \nwhat is your input of your reported models? the original image with 480x480 or the difference between the original image and the image obtained by one of the standard demosayzingov? If the difference was used, which one standard demosayzingov?",
      "votes": null
    },
    {
      "id": "280519",
      "postDate": "02/10/2018 07:00:00",
      "content": "<p>n01z3 said demosaicing didn't work for them.</p>",
      "rawMarkdown": "n01z3 said demosaicing didn't work for them.",
      "votes": null
    },
    {
      "id": "280523",
      "postDate": "02/10/2018 07:11:11",
      "content": "<p>Did you use the official manipulations (compression, gamma correction, and resizing) for training? I would guess yes, so you could bring the training set closer to the test set, and because you mentioned using Ivan's PyTorch version of Andres's code which has them turned on by default.</p>",
      "rawMarkdown": "Did you use the official manipulations (compression, gamma correction, and resizing) for training? I would guess yes, so you could bring the training set closer to the test set, and because you mentioned using Ivan's PyTorch version of Andres's code which has them turned on by default.",
      "votes": null
    },
    {
      "id": "280533",
      "postDate": "02/10/2018 07:18:34",
      "content": "<p>Amazing .. Congrats !!</p>",
      "rawMarkdown": "Amazing .. Congrats !!",
      "votes": null
    },
    {
      "id": "280571",
      "postDate": "02/10/2018 09:46:13",
      "content": "<p>nice</p>",
      "rawMarkdown": "nice",
      "votes": null
    },
    {
      "id": "280592",
      "postDate": "02/10/2018 11:46:03",
      "content": "<p>Yes. We did not change augmentations from Ivan's code, except adding D4 transforms after them for some models.</p>",
      "rawMarkdown": "Yes. We did not change augmentations from Ivan's code, except adding D4 transforms after them for some models.",
      "votes": null
    },
    {
      "id": "328066",
      "postDate": "05/13/2018 09:13:07",
      "content": "<p>We have published a video with 1st and 2nd places solutions with English subtitles from Moscow ML trainings meetup. \n<a href=\"https://youtu.be/ETh8bJ_xKGA\">https://youtu.be/ETh8bJ_xKGA</a></p>",
      "rawMarkdown": "We have published a video with 1st and 2nd places solutions with English subtitles from Moscow ML trainings meetup. \nhttps://youtu.be/ETh8bJ_xKGA",
      "votes": null
    },
    {
      "id": "328078",
      "postDate": "05/13/2018 09:43:23",
      "content": "<p>cool !</p>",
      "rawMarkdown": "cool !",
      "votes": null
    },
    {
      "id": "509738",
      "postDate": "04/08/2019 08:15:43",
      "content": "<p>Why some images in HTC directory appears to be of Nexus  o(╥﹏╥)o</p>",
      "rawMarkdown": "Why some images in HTC directory appears to be of Nexus  o(╥﹏╥)o",
      "votes": null
    },
    {
      "id": "509739",
      "postDate": "04/08/2019 08:18:44",
      "content": "<p>anyway thanks for sharing QAQ</p>",
      "rawMarkdown": "anyway thanks for sharing QAQ",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 279974,
      "author_name": "golubev",
      "author_url": "",
      "post_date": "02/09/2018 02:27:27",
      "content": "<p>Amazing work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 279978,
      "author_name": "kleinsmith",
      "author_url": "",
      "post_date": "02/09/2018 02:44:00",
      "content": "<p>Thank you. This was fun to read as someone new to Kaggle.</p>\n\n<blockquote>\n  <p>All models had an input of 480.</p>\n</blockquote>\n\n<p>Did you crop from the 512x512 center region or from the entire image?</p>\n\n<blockquote>\n  <p>D4 augmentations</p>\n</blockquote>\n\n<p>Does this refer to compression, resizing, gamma correction, and transpose? Where does the term \"D4\" come from?</p>",
      "votes": null,
      "replies": [
        {
          "id": 280117,
          "author_name": "fartuk1",
          "author_url": "",
          "post_date": "02/09/2018 11:55:36",
          "content": "<p>We crop from  entire image. Term D4 comes from group theory. It means all rotation on 90 and flips.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280141,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "02/09/2018 13:13:43",
          "content": "<p>About D4: <a href=\"https://en.wikipedia.org/wiki/Dihedral_group\">https://en.wikipedia.org/wiki/Dihedral_group</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280411,
          "author_name": "kleinsmith",
          "author_url": "",
          "post_date": "02/09/2018 23:09:17",
          "content": "<p>Thank you. Here's an image from wikipedia that explains D8 (not D4) visually:</p>\n\n<p><a href=\"https://en.wikipedia.org/wiki/Dihedral_group#/media/File:Dihedral8.png\">https://en.wikipedia.org/wiki/Dihedral_group#/media/File:Dihedral8.png</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 279987,
      "author_name": "yyll008",
      "author_url": "",
      "post_date": "02/09/2018 03:10:03",
      "content": "<p>Wow! Congratulation!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 280110,
      "author_name": "zhaoyangma",
      "author_url": "",
      "post_date": "02/09/2018 11:48:57",
      "content": "<p>Congratulations!</p>\n\n<p>The hardware is powerful!  </p>\n\n<p>Why most the loss curves are periodically jumping very high? </p>\n\n<p>The validation score is not too high. Is it because the curve are only part of all? But the epic seems already goes beyond 60.</p>\n\n<p>For DenseNet201, what is the batch size used? It does not run in my computer even with a batch size of 1. :(</p>\n\n<p>Have you tried other ensembling methods instead of only geometric mean?</p>\n\n<p>How is dual path network in terms of accuracy, performance and memory consumption comparing to DenseNet or other networks?</p>\n\n<p>Two more questions: what is D4 and what is demosizing?</p>\n\n<p>Quite a few questions. But these are something I am curious to know. :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 280145,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "02/09/2018 13:23:39",
          "content": "<ol>\n<li>Loss curves seems jumpy due to scale and little amount of sample in\nvalidation set.  </li>\n<li>Almost all validation accuracy is about 0.98. Pretty\n    high for me.</li>\n<li>As far as I remember typically batch size is 24-40 for 4x\n    1080ti.</li>\n<li>We tried weighted mean, small class sorting. Doesn't work for\n    us.</li>\n<li>DPN is fantastic! Our best single model: dpn92 with pseudo\n    labeling, TTA, 3 checkpoints TTA: 0.987 (private) and 0.984 (public).</li>\n<li>About D4 answered above</li>\n</ol>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280170,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "02/09/2018 14:01:24",
          "content": "<p>What is DPN?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280179,
          "author_name": "sakvaua",
          "author_url": "",
          "post_date": "02/09/2018 14:17:13",
          "content": "<p>I would guess that demosizing is demosaicing (debayering) algorithm to remove Bayer pattern.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280181,
          "author_name": "zhaoyangma",
          "author_url": "",
          "post_date": "02/09/2018 14:21:41",
          "content": "<p>DPN is dual path network. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280185,
          "author_name": "zhaoyangma",
          "author_url": "",
          "post_date": "02/09/2018 14:28:48",
          "content": "<p>@n01z3 Thanks a bunch!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 280421,
          "author_name": "kleinsmith",
          "author_url": "",
          "post_date": "02/09/2018 23:42:34",
          "content": "<p>DPN: <a href=\"https://arxiv.org/abs/1707.01629\">https://arxiv.org/abs/1707.01629</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 280204,
      "author_name": "staniszap",
      "author_url": "",
      "post_date": "02/09/2018 14:59:17",
      "content": "<p>It's fantastic!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 280336,
      "author_name": "alside",
      "author_url": "",
      "post_date": "02/09/2018 18:26:17",
      "content": "<p>Much data! so Wow!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 280464,
      "author_name": "meprobjtu",
      "author_url": "",
      "post_date": "02/10/2018 03:48:07",
      "content": "<p>Congratulations! \nwhat is your input of your reported models? the original image with 480x480 or the difference between the original image and the image obtained by one of the standard demosayzingov? If the difference was used, which one standard demosayzingov?</p>",
      "votes": null,
      "replies": [
        {
          "id": 280519,
          "author_name": "kleinsmith",
          "author_url": "",
          "post_date": "02/10/2018 07:00:00",
          "content": "<p>n01z3 said demosaicing didn't work for them.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 280523,
      "author_name": "kleinsmith",
      "author_url": "",
      "post_date": "02/10/2018 07:11:11",
      "content": "<p>Did you use the official manipulations (compression, gamma correction, and resizing) for training? I would guess yes, so you could bring the training set closer to the test set, and because you mentioned using Ivan's PyTorch version of Andres's code which has them turned on by default.</p>",
      "votes": null,
      "replies": [
        {
          "id": 280592,
          "author_name": "ikibardin",
          "author_url": "",
          "post_date": "02/10/2018 11:46:03",
          "content": "<p>Yes. We did not change augmentations from Ivan's code, except adding D4 transforms after them for some models.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 280533,
      "author_name": "pratipk",
      "author_url": "",
      "post_date": "02/10/2018 07:18:34",
      "content": "<p>Amazing .. Congrats !!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 280571,
      "author_name": "jjzzxx000",
      "author_url": "",
      "post_date": "02/10/2018 09:46:13",
      "content": "<p>nice</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 328066,
      "author_name": "emilkayumov",
      "author_url": "",
      "post_date": "05/13/2018 09:13:07",
      "content": "<p>We have published a video with 1st and 2nd places solutions with English subtitles from Moscow ML trainings meetup. \n<a href=\"https://youtu.be/ETh8bJ_xKGA\">https://youtu.be/ETh8bJ_xKGA</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 328078,
          "author_name": "iddy93",
          "author_url": "",
          "post_date": "05/13/2018 09:43:23",
          "content": "<p>cool !</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 509738,
      "author_name": "mao1998gun",
      "author_url": "",
      "post_date": "04/08/2019 08:15:43",
      "content": "<p>Why some images in HTC directory appears to be of Nexus  o(╥﹏╥)o</p>",
      "votes": null,
      "replies": [
        {
          "id": 509739,
          "author_name": "mao1998gun",
          "author_url": "",
          "post_date": "04/08/2019 08:18:44",
          "content": "<p>anyway thanks for sharing QAQ</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "279968": "**tldr**: \nWe downloaded a about 500+ GB photos, trained 9+ imagenet-like models with 3 version of the pipeline. Finally, we averaged 27 checkpoints with geometric mean.\n\n**Key components of a good solution, by priorities:**\n\n 1. Large and clean external dataset \n 2. Consistant local validation\n 3. Classic competitive approach to learning models\n 4. Diverse models\n\n**Data mining**:  We downloaded 500+ Gb photos from various resources: Flickr, Yandex.Fotki, Wikipedia Commons, mobile reviews. In addition, on the last night we downloaded 22k more photos with urls from Flickr gathered by [Andres Torrubia][1]. \n\n**Filtering data**: Lightroom/photoshop/etc processing could eliminate all the information about the camera. We filtered on: model, resolution, quality of jpeg compression, software of processing.\nAfter filtering of the training and validation, the datasets looked like this:\n![enter image description here][2]\nWe took the validation from [Gleb’s post][3], but replaced iPhone 6 plus pictures in it by iPhone 6 ones.\n\n**Training models**: Our code is based on pytorch version of [Andres solution][4].\nFor all the models, the binary flag is_manip was used as an additional feature for the classifier of a net. All models had an input of 480. For the majority of our models, five crops + flip photo orientation (10TTA) was applied to the pictures, for D4 models five crops + the whole group of D4 were applied (40TTA); then geometric mean was applied to the predictions.\nUseful tricks:\n1. Adam, reducing LR on plateau with patience 2-4\n2. Cyclic LR with SGD\n3. Pseudo-labeling\n4. Averaging 3 checkpoints with the best loss for validation\n\nAlso on the last day we trained several models with D4 augmentations, finetune from best checkpoint of previous pipeline. We didn’t have submit to check all models on LB, but the result on one was impressive.\n\nThe final ensemble of models looked like this:\n![enter image description here][5]\nLogging models:\n ![enter image description here][6]\n**Averaging**: We tried different approaches with class balancing and dropping the missing classes according to the probabilities. But in the end geometric average of 27 checkpoints of different models is the best.\n\n**What did not work?**\nDemosizing\n![enter image description here][7]\nCameras fix on the matrix in a particular pixel the intensity of only one color, and the rest of the colors are restored to neighboring pixels, and the recovery algorithm for different camera manufacturers is different.\nThe idea was to calculate the difference between the original image and the image obtained by one of the standard demosayzingov.\nThe inertia of the differences is to train the neural network.\n\n**Hardware:**\n\n - i7 7700k, 64gb, 2x 1080 (just for development) \n - i7 6700k, 32gb, 2x    Titan X (Maxwell)\n - i7 5930K, 32GB, 3x1080Ti\n - Xeon 2696v3, 64gb, 4x1080Ti\n - i7 3770k, 16gb, 2x 1080Ti\n\n  [1]: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/49064\n  [2]: https://pp.userapi.com/c824701/v824701072/a95a8/Hgl68WyLzyU.jpg\n  [3]: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/47235\n  [4]: https://www.kaggle.com/c/sp-society-camera-model-identification/discussion/48679\n  [5]: https://pp.userapi.com/c824701/v824701072/a95b9/dg36oZu6_B8.jpg\n  [6]: https://pp.userapi.com/c824701/v824701072/a95b2/aMQ2rKhWAI4.jpg\n  [7]: https://pp.userapi.com/c824701/v824701072/a95c0/sJpJNsYtDcg.jpg",
    "279974": "Amazing work!",
    "279978": "Thank you. This was fun to read as someone new to Kaggle.\n\n&gt; All models had an input of 480.\n\nDid you crop from the 512x512 center region or from the entire image?\n\n&gt; D4 augmentations\n\nDoes this refer to compression, resizing, gamma correction, and transpose? Where does the term \"D4\" come from?",
    "279987": "Wow! Congratulation!",
    "280110": "Congratulations!\n\nThe hardware is powerful!  \n\nWhy most the loss curves are periodically jumping very high? \n\nThe validation score is not too high. Is it because the curve are only part of all? But the epic seems already goes beyond 60.\n\nFor DenseNet201, what is the batch size used? It does not run in my computer even with a batch size of 1. :(\n\nHave you tried other ensembling methods instead of only geometric mean?\n\nHow is dual path network in terms of accuracy, performance and memory consumption comparing to DenseNet or other networks?\n\nTwo more questions: what is D4 and what is demosizing?\n\nQuite a few questions. But these are something I am curious to know. :)",
    "280117": "We crop from  entire image. Term D4 comes from group theory. It means all rotation on 90 and flips.",
    "280141": "About D4: https://en.wikipedia.org/wiki/Dihedral_group",
    "280145": "1. Loss curves seems jumpy due to scale and little amount of sample in\n    validation set.  \n 2. Almost all validation accuracy is about 0.98. Pretty\n        high for me.\n 3. As far as I remember typically batch size is 24-40 for 4x\n        1080ti.\n 4. We tried weighted mean, small class sorting. Doesn't work for\n        us.\n 5. DPN is fantastic! Our best single model: dpn92 with pseudo\n        labeling, TTA, 3 checkpoints TTA: 0.987 (private) and 0.984 (public).\n 6. About D4 answered above",
    "280170": "What is DPN?",
    "280179": "I would guess that demosizing is demosaicing (debayering) algorithm to remove Bayer pattern.",
    "280181": "DPN is dual path network.",
    "280185": "n01z3 Thanks a bunch!",
    "280204": "It's fantastic!",
    "280336": "Much data! so Wow!",
    "280411": "Thank you. Here's an image from wikipedia that explains D8 (not D4) visually:\n\nhttps://en.wikipedia.org/wiki/Dihedral_group#/media/File:Dihedral8.png",
    "280421": "DPN: https://arxiv.org/abs/1707.01629",
    "280464": "Congratulations! \nwhat is your input of your reported models? the original image with 480x480 or the difference between the original image and the image obtained by one of the standard demosayzingov? If the difference was used, which one standard demosayzingov?",
    "280519": "n01z3 said demosaicing didn't work for them.",
    "280523": "Did you use the official manipulations (compression, gamma correction, and resizing) for training? I would guess yes, so you could bring the training set closer to the test set, and because you mentioned using Ivan's PyTorch version of Andres's code which has them turned on by default.",
    "280533": "Amazing .. Congrats !!",
    "280571": "nice",
    "280592": "Yes. We did not change augmentations from Ivan's code, except adding D4 transforms after them for some models.",
    "328066": "We have published a video with 1st and 2nd places solutions with English subtitles from Moscow ML trainings meetup. \nhttps://youtu.be/ETh8bJ_xKGA",
    "328078": "cool !",
    "509738": "Why some images in HTC directory appears to be of Nexus  o(╥﹏╥)o",
    "509739": "anyway thanks for sharing QAQ"
  },
  "source": "meta"
}