{
  "id": 77320,
  "title": "3rd place solution with code.",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/77320",
  "author_name": "pudae",
  "post_date": "2019-01-11T14:16:13.048000",
  "votes": 124,
  "comment_count": 18,
  "views": 0,
  "content": "<h2>UPDATE: code available on github</h2>\n\n<p><a href=\"https://github.com/pudae/kaggle-hpa\">https://github.com/pudae/kaggle-hpa</a></p>\n\n<hr>\n\n<p>Congrats to all the winners and thanks to all kagglers who posted great discussions. It was very helpful to me.</p>\n\n<p>Thanks to Kaggle and HPA team for an interesting competition.</p>\n\n<p>Here is overview of my solution.</p>\n\n<h2>Dataset Preparation</h2>\n\n<p>Like almost the other competitors, I also used official + <a href=\"http://www.proteinatlas.org\">external</a> data. \n(Thanks to <a href=\"https://www.kaggle.com/tomomimoriyama\">TomomiMoriyama</a> and <a href=\"https://www.kaggle.com/dr1t10\">David Silva</a>)</p>\n\n<p>I splited dataset as following:</p>\n\n<ul>\n<li>1/10 holdout set for ensemble.</li>\n<li>building 5 folds cross validation sets using rest of 9/10.</li>\n<li>using phash and ahash to prevent duplicate images in difference splits. If the labels are not matched between official and external, I used official one. (Thanks to <a href=\"https://www.kaggle.com/tilii7\">Tilii</a>)</li>\n</ul>\n\n<h2>Input Preprocessing</h2>\n\n<p>I found that the distributions of image mean and stddev are very difference between official and external. So, I used mean and stddev of individual images for input normalization.</p>\n\n<h2>Augmentation</h2>\n\n<p>I searched suitable data augmentation as following <a href=\"https://arxiv.org/pdf/1805.09501.pdf\">AutoAugment</a>. For simplicity, I used random search instead of RL.</p>\n\n<h2>Loss</h2>\n\n<p>Focal loss with gamma 2.</p>\n\n<h2>Training</h2>\n\n<ul>\n<li>Adam optimizer, learning rate 0.0005.</li>\n<li>no learning rate scheduling.</li>\n<li>For the large model with 1024x1024 images, I used gradient accumulation so that weights are updated every 32 examples.</li>\n<li>Early stopping\n<ul><li>If I choose checkpoints that record best macro F1 score for the validation set, LB scores are poor.</li>\n<li>After analyzing F1 scores of each classes, I found that while macro F1 score is increasing, F1 scores of high-proportion classes (like 0, 1) are decreasing. Because of relying on rare class score is risky, I decided to stop training when F1 score of 0 class is decreasing.</li></ul></li>\n</ul>\n\n<h2>Inference</h2>\n\n<ul>\n<li>Averaging the weights of last 10 checkpoints.</li>\n<li>8 test time augmentation</li>\n<li>weighted averaging ensemble</li>\n</ul>\n\n<h2>Thresholds</h2>\n\n<p>Because of rare classes, macro F1 score is very sensitive to thresholds. I tested various method for finding good threshold, but almost tries are failed.</p>\n\n<p>My final method is following:</p>\n\n<ul>\n<li>For each classes, I choose the thresholds that make the proportion of positive predictions in validation set are closed to the proportion of positive examples. (Thanks to <a href=\"https://www.kaggle.com/iafoss\">lafoss</a> for the LB probing)</li>\n</ul>\n\n<h2>Models</h2>\n\n<p><strong>512x512</strong></p>\n\n<ul>\n<li>resnet34: 5 fold ensemble with TTA: Public LB 0.574 / Private LB 0.500</li>\n</ul>\n\n<p><strong>1024x1024</strong></p>\n\n<ul>\n<li>inceptionv3: single fold with TTA: Public LB 0.583 / Private LB 0.549</li>\n<li>se_resnext50: single fold with TTA: Public LB 0.601 / Private LB 0.531</li>\n<li>From 1024x1024, the mean and stddev of individual images are used.</li>\n<li>In case of 1024x1024 input, using global average pooling is not good performance in my case.. maybe... So, I modified last layer following:\n<ul><li>remove global average pooling.</li>\n<li>compute MxM logits using 1x1 convolution.</li>\n<li>compute weight maps using 1x1 convolution followed by softmax.</li>\n<li>using weight maps, compute weighted averaged logits.</li></ul></li>\n<li>Final submission is ensemble of above three predictions.</li>\n<li>Additional models are trained, but the ensemble results were not good.</li>\n</ul>\n\n<p>Because I failed to make stable CV, I can't be sure that methods described above were effective. Finding good methods without stable CV was painful process. So, I hope to learn from the <a href=\"https://www.kaggle.com/bestfitting\">bestfitting</a>'s solution that produce stable results always.</p>",
  "messages": [
    {
      "id": 454363,
      "postDate": "2019-01-11T14:16:13.047Z",
      "content": "<h2>UPDATE: code available on github</h2>\n\n<p><a href=\"https://github.com/pudae/kaggle-hpa\">https://github.com/pudae/kaggle-hpa</a></p>\n\n<hr>\n\n<p>Congrats to all the winners and thanks to all kagglers who posted great discussions. It was very helpful to me.</p>\n\n<p>Thanks to Kaggle and HPA team for an interesting competition.</p>\n\n<p>Here is overview of my solution.</p>\n\n<h2>Dataset Preparation</h2>\n\n<p>Like almost the other competitors, I also used official + <a href=\"http://www.proteinatlas.org\">external</a> data. \n(Thanks to <a href=\"https://www.kaggle.com/tomomimoriyama\">TomomiMoriyama</a> and <a href=\"https://www.kaggle.com/dr1t10\">David Silva</a>)</p>\n\n<p>I splited dataset as following:</p>\n\n<ul>\n<li>1/10 holdout set for ensemble.</li>\n<li>building 5 folds cross validation sets using rest of 9/10.</li>\n<li>using phash and ahash to prevent duplicate images in difference splits. If the labels are not matched between official and external, I used official one. (Thanks to <a href=\"https://www.kaggle.com/tilii7\">Tilii</a>)</li>\n</ul>\n\n<h2>Input Preprocessing</h2>\n\n<p>I found that the distributions of image mean and stddev are very difference between official and external. So, I used mean and stddev of individual images for input normalization.</p>\n\n<h2>Augmentation</h2>\n\n<p>I searched suitable data augmentation as following <a href=\"https://arxiv.org/pdf/1805.09501.pdf\">AutoAugment</a>. For simplicity, I used random search instead of RL.</p>\n\n<h2>Loss</h2>\n\n<p>Focal loss with gamma 2.</p>\n\n<h2>Training</h2>\n\n<ul>\n<li>Adam optimizer, learning rate 0.0005.</li>\n<li>no learning rate scheduling.</li>\n<li>For the large model with 1024x1024 images, I used gradient accumulation so that weights are updated every 32 examples.</li>\n<li>Early stopping\n<ul><li>If I choose checkpoints that record best macro F1 score for the validation set, LB scores are poor.</li>\n<li>After analyzing F1 scores of each classes, I found that while macro F1 score is increasing, F1 scores of high-proportion classes (like 0, 1) are decreasing. Because of relying on rare class score is risky, I decided to stop training when F1 score of 0 class is decreasing.</li></ul></li>\n</ul>\n\n<h2>Inference</h2>\n\n<ul>\n<li>Averaging the weights of last 10 checkpoints.</li>\n<li>8 test time augmentation</li>\n<li>weighted averaging ensemble</li>\n</ul>\n\n<h2>Thresholds</h2>\n\n<p>Because of rare classes, macro F1 score is very sensitive to thresholds. I tested various method for finding good threshold, but almost tries are failed.</p>\n\n<p>My final method is following:</p>\n\n<ul>\n<li>For each classes, I choose the thresholds that make the proportion of positive predictions in validation set are closed to the proportion of positive examples. (Thanks to <a href=\"https://www.kaggle.com/iafoss\">lafoss</a> for the LB probing)</li>\n</ul>\n\n<h2>Models</h2>\n\n<p><strong>512x512</strong></p>\n\n<ul>\n<li>resnet34: 5 fold ensemble with TTA: Public LB 0.574 / Private LB 0.500</li>\n</ul>\n\n<p><strong>1024x1024</strong></p>\n\n<ul>\n<li>inceptionv3: single fold with TTA: Public LB 0.583 / Private LB 0.549</li>\n<li>se_resnext50: single fold with TTA: Public LB 0.601 / Private LB 0.531</li>\n<li>From 1024x1024, the mean and stddev of individual images are used.</li>\n<li>In case of 1024x1024 input, using global average pooling is not good performance in my case.. maybe... So, I modified last layer following:\n<ul><li>remove global average pooling.</li>\n<li>compute MxM logits using 1x1 convolution.</li>\n<li>compute weight maps using 1x1 convolution followed by softmax.</li>\n<li>using weight maps, compute weighted averaged logits.</li></ul></li>\n<li>Final submission is ensemble of above three predictions.</li>\n<li>Additional models are trained, but the ensemble results were not good.</li>\n</ul>\n\n<p>Because I failed to make stable CV, I can't be sure that methods described above were effective. Finding good methods without stable CV was painful process. So, I hope to learn from the <a href=\"https://www.kaggle.com/bestfitting\">bestfitting</a>'s solution that produce stable results always.</p>",
      "rawMarkdown": "## UPDATE: code available on github\nhttps://github.com/pudae/kaggle-hpa\n\n---\n\nCongrats to all the winners and thanks to all kagglers who posted great discussions. It was very helpful to me.\n\nThanks to Kaggle and HPA team for an interesting competition.\n\nHere is overview of my solution.\n\n## Dataset Preparation\nLike almost the other competitors, I also used official + [external](http://www.proteinatlas.org) data. \n(Thanks to [TomomiMoriyama](https://www.kaggle.com/tomomimoriyama) and [David Silva](https://www.kaggle.com/dr1t10))\n\nI splited dataset as following:\n\n* 1/10 holdout set for ensemble.\n* building 5 folds cross validation sets using rest of 9/10.\n* using phash and ahash to prevent duplicate images in difference splits. If the labels are not matched between official and external, I used official one. (Thanks to [Tilii](https://www.kaggle.com/tilii7))\n\n## Input Preprocessing\nI found that the distributions of image mean and stddev are very difference between official and external. So, I used mean and stddev of individual images for input normalization.\n\n## Augmentation\nI searched suitable data augmentation as following [AutoAugment](https://arxiv.org/pdf/1805.09501.pdf). For simplicity, I used random search instead of RL.\n\n## Loss\nFocal loss with gamma 2.\n\n## Training\n* Adam optimizer, learning rate 0.0005.\n* no learning rate scheduling.\n* For the large model with 1024x1024 images, I used gradient accumulation so that weights are updated every 32 examples.\n* Early stopping\n  * If I choose checkpoints that record best macro F1 score for the validation set, LB scores are poor.\n  * After analyzing F1 scores of each classes, I found that while macro F1 score is increasing, F1 scores of high-proportion classes (like 0, 1) are decreasing. Because of relying on rare class score is risky, I decided to stop training when F1 score of 0 class is decreasing.\n\n## Inference\n* Averaging the weights of last 10 checkpoints.\n* 8 test time augmentation\n* weighted averaging ensemble\n\n## Thresholds\nBecause of rare classes, macro F1 score is very sensitive to thresholds. I tested various method for finding good threshold, but almost tries are failed.\n\nMy final method is following:\n\n* For each classes, I choose the thresholds that make the proportion of positive predictions in validation set are closed to the proportion of positive examples. (Thanks to [lafoss](https://www.kaggle.com/iafoss) for the LB probing)\n\n## Models\n**512x512**\n\n* resnet34: 5 fold ensemble with TTA: Public LB 0.574 / Private LB 0.500\n\n**1024x1024**\n\n* inceptionv3: single fold with TTA: Public LB 0.583 / Private LB 0.549\n* se_resnext50: single fold with TTA: Public LB 0.601 / Private LB 0.531\n* From 1024x1024, the mean and stddev of individual images are used.\n* In case of 1024x1024 input, using global average pooling is not good performance in my case.. maybe... So, I modified last layer following:\n  * remove global average pooling.\n  * compute MxM logits using 1x1 convolution.\n  * compute weight maps using 1x1 convolution followed by softmax.\n  * using weight maps, compute weighted averaged logits.\n* Final submission is ensemble of above three predictions.\n* Additional models are trained, but the ensemble results were not good.\n\nBecause I failed to make stable CV, I can't be sure that methods described above were effective. Finding good methods without stable CV was painful process. So, I hope to learn from the [bestfitting](https://www.kaggle.com/bestfitting)'s solution that produce stable results always.",
      "votes": 123
    },
    {
      "id": 454457,
      "postDate": "2019-01-11T16:59:23.800Z",
      "content": "<p><a href=\"/pudae81\">@pudae81</a> Thanks for the write up and congrats! That AutoAugment paper is interesting. Can you post the policy your random search produced? Rotations were obviously effective, another write up said Brightness was a key augmentation, I'm curious what the algorithm ended up finding. </p>",
      "rawMarkdown": "@pudae81 Thanks for the write up and congrats! That AutoAugment paper is interesting. Can you post the policy your random search produced? Rotations were obviously effective, another write up said Brightness was a key augmentation, I'm curious what the algorithm ended up finding. ",
      "votes": 3,
      "replies": [
        {
          "id": 454668,
          "postDate": "2019-01-12T00:31:46.260Z",
          "content": "<p>Congrats and thanks for a interesting AutoAugment approach. Would be very interested to find out the parameters algorithm produced as well.</p>",
          "rawMarkdown": "Congrats and thanks for a interesting AutoAugment approach. Would be very interested to find out the parameters algorithm produced as well."
        },
        {
          "id": 455574,
          "postDate": "2019-01-14T07:14:43.093Z",
          "content": "<p>Congratulations! The autoaugment part is really interesting... would you be able to post the code or at least detail your approach a bit more? </p>",
          "rawMarkdown": "Congratulations! The autoaugment part is really interesting... would you be able to post the code or at least detail your approach a bit more? "
        },
        {
          "id": 555062,
          "postDate": "2019-06-18T11:42:19.793Z",
          "content": "<p>Congrats! I am very interesting in your implement of AutoAugment. How do you use random search?</p>",
          "rawMarkdown": "Congrats! I am very interesting in your implement of AutoAugment. How do you use random search?"
        }
      ]
    },
    {
      "id": 454426,
      "postDate": "2019-01-11T16:16:02.217Z",
      "content": "<p>Congratilations. Is it possible to share your code.</p>",
      "rawMarkdown": "Congratilations. Is it possible to share your code.",
      "votes": 3
    },
    {
      "id": 469487,
      "postDate": "2019-02-11T10:06:26.157Z",
      "content": "<p>Pudae,</p>\n\n<p>Excellent work.  Thank you.</p>\n\n<p>I looked at your code, which is well-written and concise.  My only wish is that you would add comments because it would help hasten my understanding.</p>\n\n<p>I did not see where in your code you fine-tuned the pretrained model.  Did you allow your model to continue training any of its pretrained layers?</p>\n\n<p>--Scott</p>",
      "rawMarkdown": "Pudae,\n\nExcellent work.  Thank you.\n\nI looked at your code, which is well-written and concise.  My only wish is that you would add comments because it would help hasten my understanding.\n\nI did not see where in your code you fine-tuned the pretrained model.  Did you allow your model to continue training any of its pretrained layers?\n\n--Scott"
    },
    {
      "id": 457886,
      "postDate": "2019-01-18T09:13:25.450Z",
      "content": "<p>Thanks for sharing. One question, why did you set 'eval.batch_size' to 2  when inferring the inceptionv3 model? </p>",
      "rawMarkdown": "Thanks for sharing. One question, why did you set 'eval.batch_size' to 2  when inferring the inceptionv3 model? "
    },
    {
      "id": 456760,
      "postDate": "2019-01-16T13:33:21.763Z",
      "content": "<p>Congratulation</p>",
      "rawMarkdown": "Congratulation"
    },
    {
      "id": 454685,
      "postDate": "2019-01-12T01:59:39.160Z",
      "content": "<p>thanks for sharing. \"1/10 holdout set for ensemble\" is really a smart idea. \nHow much LB will decrease if you drop this step and just use validation set for ensemble?</p>",
      "rawMarkdown": "thanks for sharing. \"1/10 holdout set for ensemble\" is really a smart idea. \nHow much LB will decrease if you drop this step and just use validation set for ensemble?"
    },
    {
      "id": 454578,
      "postDate": "2019-01-11T21:01:29.530Z",
      "content": "<p>Yikes, being a novice, this competition was super grounding :).  Anyways, awesome work! Thanks for sharing!! Can;t wait to reproduce your results.</p>",
      "rawMarkdown": "Yikes, being a novice, this competition was super grounding :).  Anyways, awesome work! Thanks for sharing!! Can;t wait to reproduce your results."
    },
    {
      "id": 454488,
      "postDate": "2019-01-11T18:05:06.397Z",
      "content": "<p>congrats! I did almost the same thing as your resnet34 model , but the result is poor.  did you oversample some classes? or use different class weight? Is that possible to share your code ?</p>",
      "rawMarkdown": "congrats! I did almost the same thing as your resnet34 model , but the result is poor.  did you oversample some classes? or use different class weight? Is that possible to share your code ?"
    },
    {
      "id": 454435,
      "postDate": "2019-01-11T16:25:56.677Z",
      "content": "<p>Thank you for the explanation and wonderful paper for data augmentation and I also experienced ensemble method is not good. I was consistently impressed how you efficiently climb up the leaderboard with few submissions and learned a lot from your solution overview above. Thank you!</p>",
      "rawMarkdown": "Thank you for the explanation and wonderful paper for data augmentation and I also experienced ensemble method is not good. I was consistently impressed how you efficiently climb up the leaderboard with few submissions and learned a lot from your solution overview above. Thank you!"
    },
    {
      "id": 456764,
      "postDate": "2019-01-16T13:47:31.363Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 574589,
      "postDate": "2019-07-14T06:31:56.543Z",
      "content": "<p>Congratulations. Thanks for sharing.</p>",
      "rawMarkdown": "Congratulations. Thanks for sharing."
    },
    {
      "id": 457858,
      "postDate": "2019-01-18T08:07:36.393Z",
      "content": "<p>selfless work！thanks</p>",
      "rawMarkdown": "selfless work！thanks"
    },
    {
      "id": 456779,
      "postDate": "2019-01-16T14:38:53.397Z",
      "content": "<p>Thank you for sharing code</p>",
      "rawMarkdown": "Thank you for sharing code"
    },
    {
      "id": 454887,
      "postDate": "2019-01-12T12:34:55.847Z",
      "content": "<p>Congratulations. Thanks for sharing.</p>",
      "rawMarkdown": "Congratulations. Thanks for sharing."
    },
    {
      "id": 454638,
      "postDate": "2019-01-11T22:59:49.963Z",
      "content": "<p>Congrats!. Thank for sharing </p>",
      "rawMarkdown": "Congrats!. Thank for sharing "
    }
  ],
  "comments": [
    {
      "id": 454457,
      "author_name": "David Wagner",
      "author_url": "",
      "post_date": "2019-01-11T16:59:23.800000",
      "content": "<p><a href=\"/pudae81\">@pudae81</a> Thanks for the write up and congrats! That AutoAugment paper is interesting. Can you post the policy your random search produced? Rotations were obviously effective, another write up said Brightness was a key augmentation, I'm curious what the algorithm ended up finding. </p>",
      "votes": 3,
      "replies": [
        {
          "id": 454668,
          "author_name": "Dmytro Poplavskiy",
          "author_url": "",
          "post_date": "2019-01-12T00:31:46.260000",
          "content": "<p>Congrats and thanks for a interesting AutoAugment approach. Would be very interested to find out the parameters algorithm produced as well.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 455574,
          "author_name": "Moshel",
          "author_url": "",
          "post_date": "2019-01-14T07:14:43.093000",
          "content": "<p>Congratulations! The autoaugment part is really interesting... would you be able to post the code or at least detail your approach a bit more? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 555062,
          "author_name": "seefun",
          "author_url": "",
          "post_date": "2019-06-18T11:42:19.793000",
          "content": "<p>Congrats! I am very interesting in your implement of AutoAugment. How do you use random search?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 454426,
      "author_name": "JM100",
      "author_url": "",
      "post_date": "2019-01-11T16:16:02.217000",
      "content": "<p>Congratilations. Is it possible to share your code.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 469487,
      "author_name": "Scott Miller",
      "author_url": "",
      "post_date": "2019-02-11T10:06:26.157000",
      "content": "<p>Pudae,</p>\n\n<p>Excellent work.  Thank you.</p>\n\n<p>I looked at your code, which is well-written and concise.  My only wish is that you would add comments because it would help hasten my understanding.</p>\n\n<p>I did not see where in your code you fine-tuned the pretrained model.  Did you allow your model to continue training any of its pretrained layers?</p>\n\n<p>--Scott</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 457886,
      "author_name": "vikeezhou",
      "author_url": "",
      "post_date": "2019-01-18T09:13:25.450000",
      "content": "<p>Thanks for sharing. One question, why did you set 'eval.batch_size' to 2  when inferring the inceptionv3 model? </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 456760,
      "author_name": "Sehank",
      "author_url": "",
      "post_date": "2019-01-16T13:33:21.763000",
      "content": "<p>Congratulation</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 454685,
      "author_name": "good good study",
      "author_url": "",
      "post_date": "2019-01-12T01:59:39.160000",
      "content": "<p>thanks for sharing. \"1/10 holdout set for ensemble\" is really a smart idea. \nHow much LB will decrease if you drop this step and just use validation set for ensemble?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 454578,
      "author_name": "Harsha Lokavarapu",
      "author_url": "",
      "post_date": "2019-01-11T21:01:29.530000",
      "content": "<p>Yikes, being a novice, this competition was super grounding :).  Anyways, awesome work! Thanks for sharing!! Can;t wait to reproduce your results.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 454488,
      "author_name": "soywu",
      "author_url": "",
      "post_date": "2019-01-11T18:05:06.397000",
      "content": "<p>congrats! I did almost the same thing as your resnet34 model , but the result is poor.  did you oversample some classes? or use different class weight? Is that possible to share your code ?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 454435,
      "author_name": "Soonhwan Kwon",
      "author_url": "",
      "post_date": "2019-01-11T16:25:56.677000",
      "content": "<p>Thank you for the explanation and wonderful paper for data augmentation and I also experienced ensemble method is not good. I was consistently impressed how you efficiently climb up the leaderboard with few submissions and learned a lot from your solution overview above. Thank you!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 456764,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-01-16T13:47:31.363000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 574589,
      "author_name": "Mohammed MASSOUNE",
      "author_url": "",
      "post_date": "2019-07-14T06:31:56.543000",
      "content": "<p>Congratulations. Thanks for sharing.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 457858,
      "author_name": "nopro",
      "author_url": "",
      "post_date": "2019-01-18T08:07:36.393000",
      "content": "<p>selfless work！thanks</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 456779,
      "author_name": "Soonhwan Kwon",
      "author_url": "",
      "post_date": "2019-01-16T14:38:53.397000",
      "content": "<p>Thank you for sharing code</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 454887,
      "author_name": "klickmal",
      "author_url": "",
      "post_date": "2019-01-12T12:34:55.847000",
      "content": "<p>Congratulations. Thanks for sharing.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 454638,
      "author_name": "cab",
      "author_url": "",
      "post_date": "2019-01-11T22:59:49.963000",
      "content": "<p>Congrats!. Thank for sharing </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "454363": "## UPDATE: code available on github\nhttps://github.com/pudae/kaggle-hpa\n\n---\n\nCongrats to all the winners and thanks to all kagglers who posted great discussions. It was very helpful to me.\n\nThanks to Kaggle and HPA team for an interesting competition.\n\nHere is overview of my solution.\n\n## Dataset Preparation\nLike almost the other competitors, I also used official + [external](http://www.proteinatlas.org) data. \n(Thanks to [TomomiMoriyama](https://www.kaggle.com/tomomimoriyama) and [David Silva](https://www.kaggle.com/dr1t10))\n\nI splited dataset as following:\n\n* 1/10 holdout set for ensemble.\n* building 5 folds cross validation sets using rest of 9/10.\n* using phash and ahash to prevent duplicate images in difference splits. If the labels are not matched between official and external, I used official one. (Thanks to [Tilii](https://www.kaggle.com/tilii7))\n\n## Input Preprocessing\nI found that the distributions of image mean and stddev are very difference between official and external. So, I used mean and stddev of individual images for input normalization.\n\n## Augmentation\nI searched suitable data augmentation as following [AutoAugment](https://arxiv.org/pdf/1805.09501.pdf). For simplicity, I used random search instead of RL.\n\n## Loss\nFocal loss with gamma 2.\n\n## Training\n* Adam optimizer, learning rate 0.0005.\n* no learning rate scheduling.\n* For the large model with 1024x1024 images, I used gradient accumulation so that weights are updated every 32 examples.\n* Early stopping\n  * If I choose checkpoints that record best macro F1 score for the validation set, LB scores are poor.\n  * After analyzing F1 scores of each classes, I found that while macro F1 score is increasing, F1 scores of high-proportion classes (like 0, 1) are decreasing. Because of relying on rare class score is risky, I decided to stop training when F1 score of 0 class is decreasing.\n\n## Inference\n* Averaging the weights of last 10 checkpoints.\n* 8 test time augmentation\n* weighted averaging ensemble\n\n## Thresholds\nBecause of rare classes, macro F1 score is very sensitive to thresholds. I tested various method for finding good threshold, but almost tries are failed.\n\nMy final method is following:\n\n* For each classes, I choose the thresholds that make the proportion of positive predictions in validation set are closed to the proportion of positive examples. (Thanks to [lafoss](https://www.kaggle.com/iafoss) for the LB probing)\n\n## Models\n**512x512**\n\n* resnet34: 5 fold ensemble with TTA: Public LB 0.574 / Private LB 0.500\n\n**1024x1024**\n\n* inceptionv3: single fold with TTA: Public LB 0.583 / Private LB 0.549\n* se_resnext50: single fold with TTA: Public LB 0.601 / Private LB 0.531\n* From 1024x1024, the mean and stddev of individual images are used.\n* In case of 1024x1024 input, using global average pooling is not good performance in my case.. maybe... So, I modified last layer following:\n  * remove global average pooling.\n  * compute MxM logits using 1x1 convolution.\n  * compute weight maps using 1x1 convolution followed by softmax.\n  * using weight maps, compute weighted averaged logits.\n* Final submission is ensemble of above three predictions.\n* Additional models are trained, but the ensemble results were not good.\n\nBecause I failed to make stable CV, I can't be sure that methods described above were effective. Finding good methods without stable CV was painful process. So, I hope to learn from the [bestfitting](https://www.kaggle.com/bestfitting)'s solution that produce stable results always.",
    "454457": "@pudae81 Thanks for the write up and congrats! That AutoAugment paper is interesting. Can you post the policy your random search produced? Rotations were obviously effective, another write up said Brightness was a key augmentation, I'm curious what the algorithm ended up finding. ",
    "454426": "Congratilations. Is it possible to share your code.",
    "469487": "Pudae,\n\nExcellent work.  Thank you.\n\nI looked at your code, which is well-written and concise.  My only wish is that you would add comments because it would help hasten my understanding.\n\nI did not see where in your code you fine-tuned the pretrained model.  Did you allow your model to continue training any of its pretrained layers?\n\n--Scott",
    "457886": "Thanks for sharing. One question, why did you set 'eval.batch_size' to 2  when inferring the inceptionv3 model? ",
    "456760": "Congratulation",
    "454685": "thanks for sharing. \"1/10 holdout set for ensemble\" is really a smart idea. \nHow much LB will decrease if you drop this step and just use validation set for ensemble?",
    "454578": "Yikes, being a novice, this competition was super grounding :).  Anyways, awesome work! Thanks for sharing!! Can;t wait to reproduce your results.",
    "454488": "congrats! I did almost the same thing as your resnet34 model , but the result is poor.  did you oversample some classes? or use different class weight? Is that possible to share your code ?",
    "454435": "Thank you for the explanation and wonderful paper for data augmentation and I also experienced ensemble method is not good. I was consistently impressed how you efficiently climb up the leaderboard with few submissions and learned a lot from your solution overview above. Thank you!",
    "456764": "",
    "574589": "Congratulations. Thanks for sharing.",
    "457858": "selfless work！thanks",
    "456779": "Thank you for sharing code",
    "454887": "Congratulations. Thanks for sharing.",
    "454638": "Congrats!. Thank for sharing "
  }
}