{
  "id": 81259,
  "title": "Is there anyone who achieve a >0.9 single model?",
  "url": "/competitions/humpback-whale-identification/discussion/81259",
  "author_name": "",
  "post_date": "2019-02-20T09:24:51.488143100Z",
  "votes": 22,
  "comment_count": 40,
  "views": 0,
  "content": "<p>Hi all, Is there anyone who achieve a single model with score of &gt; 0.9? I would like to request your input for this. Is it made in keras, pytorch or tensorflow? what type of model?configuration?hyperparameters? Many Thanks! :-)</p>",
  "messages": [
    {
      "id": "475111",
      "postDate": "02/20/2019 09:24:51",
      "content": "<p>Hi all, Is there anyone who achieve a single model with score of &gt; 0.9? I would like to request your input for this. Is it made in keras, pytorch or tensorflow? what type of model?configuration?hyperparameters? Many Thanks! :-)</p>",
      "rawMarkdown": "Hi all, Is there anyone who achieve a single model with score of &gt; 0.9? I would like to request your input for this. Is it made in keras, pytorch or tensorflow? what type of model?configuration?hyperparameters? Many Thanks! :-)",
      "votes": null
    },
    {
      "id": "475112",
      "postDate": "02/20/2019 09:29:26",
      "content": "<p>I am fairly close to 0.9 with a single model :)\nKeras, Siamese SEResNet18, one-cycle LR</p>",
      "rawMarkdown": "I am fairly close to 0.9 with a single model :)\nKeras, Siamese SEResNet18, one-cycle LR",
      "votes": null
    },
    {
      "id": "475136",
      "postDate": "02/20/2019 10:20:58",
      "content": "<p>That's really nice! Are you using the public kernel Siamese Network or it is your own Siamese keras-based design?</p>",
      "rawMarkdown": "That's really nice! Are you using the public kernel Siamese Network or it is your own Siamese keras-based design?",
      "votes": null
    },
    {
      "id": "475229",
      "postDate": "02/20/2019 13:26:53",
      "content": "<p>\"What type of model? configuration? hyper parameters\"\nYou ask some pretty intimate questions, sir :)</p>",
      "rawMarkdown": "\"What type of model? configuration? hyper parameters\"\nYou ask some pretty intimate questions, sir :)",
      "votes": null
    },
    {
      "id": "475231",
      "postDate": "02/20/2019 13:28:59",
      "content": "<p>i can get &gt;0.90 using classification model + semi supervised learning (LB test set is treated as unlabeled train set) + combined \"softmax loss + metric loss\". Single model without TTA or ensemble</p>",
      "rawMarkdown": "i can get &gt;0.90 using classification model + semi supervised learning (LB test set is treated as unlabeled train set) + combined \"softmax loss + metric loss\". Single model without TTA or ensemble",
      "votes": null
    },
    {
      "id": "475242",
      "postDate": "02/20/2019 13:47:18",
      "content": "<p>Thanks for the info Heng!</p>",
      "rawMarkdown": "Thanks for the info Heng!",
      "votes": null
    },
    {
      "id": "475272",
      "postDate": "02/20/2019 14:49:59",
      "content": "<p>I can get to 0.884+ using a single Siamese network, It looks like it can get me upto 0.89+ Lb score independently.</p>",
      "rawMarkdown": "I can get to 0.884+ using a single Siamese network, It looks like it can get me upto 0.89+ Lb score independently.",
      "votes": null
    },
    {
      "id": "475332",
      "postDate": "02/20/2019 16:12:34",
      "content": "<p>Classification, 0.95+</p>",
      "rawMarkdown": "Classification, 0.95+",
      "votes": null
    },
    {
      "id": "475349",
      "postDate": "02/20/2019 16:45:36",
      "content": "<p>Insanity :D\nWaiting for your report after the end!</p>",
      "rawMarkdown": "Insanity :D\nWaiting for your report after the end!",
      "votes": null
    },
    {
      "id": "475420",
      "postDate": "02/20/2019 18:22:42",
      "content": "<p>Siamese, 0.91</p>",
      "rawMarkdown": "Siamese, 0.91",
      "votes": null
    },
    {
      "id": "475458",
      "postDate": "02/20/2019 19:18:18",
      "content": "<p>Do you mean metric learning loss on one or more deep layers + classification loss at the end ?</p>",
      "rawMarkdown": "Do you mean metric learning loss on one or more deep layers + classification loss at the end ?",
      "votes": null
    },
    {
      "id": "475614",
      "postDate": "02/21/2019 01:43:50",
      "content": "<p><a href=\"/jeandebleau\">@jeandebleau</a></p>\n\n<p>yes. something like that. you can also refer to the papers:\n<a href=\"https://arxiv.org/pdf/1901.08616.pdf\">https://arxiv.org/pdf/1901.08616.pdf</a>\n<a href=\"https://arxiv.org/pdf/1902.05509.pdf\">https://arxiv.org/pdf/1902.05509.pdf</a>\n<a href=\"https://arxiv.org/pdf/1811.12649.pdf\">https://arxiv.org/pdf/1811.12649.pdf</a>\n<a href=\"https://www.ijcai.org/proceedings/2017/0308.pdf\">https://www.ijcai.org/proceedings/2017/0308.pdf</a></p>",
      "rawMarkdown": "jeandebleau\n\n\nyes. something like that. you can also refer to the papers:\nhttps://arxiv.org/pdf/1901.08616.pdf\nhttps://arxiv.org/pdf/1902.05509.pdf\nhttps://arxiv.org/pdf/1811.12649.pdf\nhttps://www.ijcai.org/proceedings/2017/0308.pdf",
      "votes": null
    },
    {
      "id": "475661",
      "postDate": "02/21/2019 03:42:05",
      "content": "<p>0.901 using Siamese with TTA</p>",
      "rawMarkdown": "0.901 using Siamese with TTA",
      "votes": null
    },
    {
      "id": "475702",
      "postDate": "02/21/2019 04:51:09",
      "content": "<p>Wow that's great! Can't wait your competition solution report! :-)</p>",
      "rawMarkdown": "Wow that's great! Can't wait your competition solution report! :-)",
      "votes": null
    },
    {
      "id": "475703",
      "postDate": "02/21/2019 04:53:41",
      "content": "<p>My best single model on public lb is 0.943.</p>",
      "rawMarkdown": "My best single model on public lb is 0.943.",
      "votes": null
    },
    {
      "id": "475801",
      "postDate": "02/21/2019 08:06:37",
      "content": "<p>Siamese+resnet50, 0.897</p>",
      "rawMarkdown": "Siamese+resnet50, 0.897",
      "votes": null
    },
    {
      "id": "476340",
      "postDate": "02/22/2019 01:55:39",
      "content": "<p>May I ask how many epochs was your network trained?</p>",
      "rawMarkdown": "May I ask how many epochs was your network trained?",
      "votes": null
    },
    {
      "id": "476535",
      "postDate": "02/22/2019 10:02:18",
      "content": "<p>Siamese from public kernel+some modification = 0.90+\nIt is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.\nIt is amazing that only Classification can get 0.95+ ( from A.L.)....</p>",
      "rawMarkdown": "Siamese from public kernel+some modification = 0.90+\nIt is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.\nIt is amazing that only Classification can get 0.95+ ( from A.L.)....",
      "votes": null
    },
    {
      "id": "476542",
      "postDate": "02/22/2019 10:12:27",
      "content": "<p>Siamese, 0.92+</p>",
      "rawMarkdown": "Siamese, 0.92+",
      "votes": null
    },
    {
      "id": "476632",
      "postDate": "02/22/2019 12:59:05",
      "content": "<p>0.918 using Siamese with Lap based on Resnet34 in Pytorch</p>",
      "rawMarkdown": "0.918 using Siamese with Lap based on Resnet34 in Pytorch",
      "votes": null
    },
    {
      "id": "476690",
      "postDate": "02/22/2019 14:49:59",
      "content": "<p>Siamese in public kernel : 0.906\nwith modified augmentation and TTA : 0.921</p>",
      "rawMarkdown": "Siamese in public kernel : 0.906\nwith modified augmentation and TTA : 0.921",
      "votes": null
    },
    {
      "id": "476748",
      "postDate": "02/22/2019 16:40:27",
      "content": "<p>Yes, I got the current result (public LB 0.951) using single model. Siamese network.</p>",
      "rawMarkdown": "Yes, I got the current result (public LB 0.951) using single model. Siamese network.",
      "votes": null
    },
    {
      "id": "477265",
      "postDate": "02/24/2019 09:02:10",
      "content": "<p>0.938 by single ResNet model with metric loss</p>",
      "rawMarkdown": "0.938 by single ResNet model with metric loss",
      "votes": null
    },
    {
      "id": "480289",
      "postDate": "02/28/2019 02:31:53",
      "content": "<p>check my comment at:\n<a href=\"https://www.kaggle.com/c/humpback-whale-identification/discussion/81085\">https://www.kaggle.com/c/humpback-whale-identification/discussion/81085</a></p>\n\n<p>LB 0.80 for resnet18 using 224 input. (threshold at 30% new-whale)\nLB 0.85 for resnet18 using 384 input. (threshold at 30% new-whale)\nLB 0.88 for resnet18 using 640 input. (threshold at 30% new-whale)\nLB 0.91 for resnet18 using 800 input. (threshold at 30% new-whale)</p>\n\n<p>all results are single model without TTA,without ensemble</p>",
      "rawMarkdown": "check my comment at:\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/81085\n\nLB 0.80 for resnet18 using 224 input. (threshold at 30% new-whale)\nLB 0.85 for resnet18 using 384 input. (threshold at 30% new-whale)\nLB 0.88 for resnet18 using 640 input. (threshold at 30% new-whale)\nLB 0.91 for resnet18 using 800 input. (threshold at 30% new-whale)\n\nall results are single model without TTA,without ensemble",
      "votes": null
    },
    {
      "id": "480574",
      "postDate": "02/28/2019 11:20:19",
      "content": "<p>May I ask on how many epochs did you train your models Heng? Thanks</p>",
      "rawMarkdown": "May I ask on how many epochs did you train your models Heng? Thanks",
      "votes": null
    },
    {
      "id": "480581",
      "postDate": "02/28/2019 11:28:31",
      "content": "<p>it varies. typically 100 to 200. train loss practically reaches zero and train accuracy reaches 100%. Sicne i am using very heavy augmentation (which induces label noise) i just let it run. it will not overfit that easily if there is non-static train label noise </p>\n\n<p>here is the trick. you can download some of the released results in the kernel page. e.g. i saw some results with LB 0.855 to 0.847.</p>\n\n<p>extract the top1 results that is common to these results. use this as validation for id whales. \nfor new-whale, you can just flip the test. (there are label error, so we do not expect 100% accuracy in validation but we do expect the validation accuracy to increase as iterations improve. we also know label accuracy is about 80% from the lb score) </p>\n\n<p>this can help you to decide the num of iterations you need for trainining</p>",
      "rawMarkdown": "it varies. typically 100 to 200. train loss practically reaches zero and train accuracy reaches 100%. Sicne i am using very heavy augmentation (which induces label noise) i just let it run. it will not overfit that easily if there is non-static train label noise \n\nhere is the trick. you can download some of the released results in the kernel page. e.g. i saw some results with LB 0.855 to 0.847.\n\nextract the top1 results that is common to these results. use this as validation for id whales. \nfor new-whale, you can just flip the test. (there are label error, so we do not expect 100% accuracy in validation but we do expect the validation accuracy to increase as iterations improve. we also know label accuracy is about 80% from the lb score) \n\nthis can help you to decide the num of iterations you need for trainining",
      "votes": null
    },
    {
      "id": "480594",
      "postDate": "02/28/2019 11:59:49",
      "content": "<p>Thanks for the tips. </p>\n\n<p>I ask because I feel that my models are underfitting. I'm using my local validation to decide whether to stop the training and I've been observing that it follows public LB until around 0.85LB, after that it seems that local validation reaches a top limit and therefore I'm stopping training too early, usually around 100 epochs.</p>\n\n<p>When you say 100 to 200 epochs, do you mean that even for the highest resolutions you train (finetune) 100~200 epochs on top of the pretrained on lower resolution? E.g. For 800x800 you trained ~100 epochs on top of the 640x640 pretrained model?</p>",
      "rawMarkdown": "Thanks for the tips. \n\nI ask because I feel that my models are underfitting. I'm using my local validation to decide whether to stop the training and I've been observing that it follows public LB until around 0.85LB, after that it seems that local validation reaches a top limit and therefore I'm stopping training too early, usually around 100 epochs.\n\nWhen you say 100 to 200 epochs, do you mean that even for the highest resolutions you train (finetune) 100~200 epochs on top of the pretrained on lower resolution? E.g. For 800x800 you trained ~100 epochs on top of the 640x640 pretrained model?",
      "votes": null
    },
    {
      "id": "480597",
      "postDate": "02/28/2019 12:02:35",
      "content": "<p>for 224, it is about 200.  then the 224 model is used to initialize 384 and train for another 100 epoch, etc ...</p>",
      "rawMarkdown": "for 224, it is about 200.  then the 224 model is used to initialize 384 and train for another 100 epoch, etc ...",
      "votes": null
    },
    {
      "id": "480605",
      "postDate": "02/28/2019 12:12:51",
      "content": "<p>Thank you</p>",
      "rawMarkdown": "Thank you",
      "votes": null
    },
    {
      "id": "480688",
      "postDate": "02/28/2019 14:20:45",
      "content": "<p>0.937 with a siamese, let's hope we won't drop to hell ;)</p>",
      "rawMarkdown": "0.937 with a siamese, let's hope we won't drop to hell ;)",
      "votes": null
    },
    {
      "id": "480935",
      "postDate": "02/28/2019 22:19:30",
      "content": "<p>Sorry for the late reply, I did 150 epochs just using the public kernel siamese network.  Can't wait to see other people's solution. Curious about what they have done to achieve such a high score on a single model. </p>",
      "rawMarkdown": "Sorry for the late reply, I did 150 epochs just using the public kernel siamese network.  Can't wait to see other people's solution. Curious about what they have done to achieve such a high score on a single model.",
      "votes": null
    },
    {
      "id": "480940",
      "postDate": "02/28/2019 22:34:25",
      "content": "<blockquote>\n  <p>It is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.</p>\n</blockquote>\n\n<p>Same here.  There's not enough time when I finally realized this. Should try different ways earlier in the next one. </p>",
      "rawMarkdown": "&gt;It is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.\n\nSame here.  There's not enough time when I finally realized this. Should try different ways earlier in the next one.",
      "votes": null
    },
    {
      "id": "480950",
      "postDate": "02/28/2019 23:09:36",
      "content": "<p>0.899 prototypical net seresnext50 RGB image size 384 /no tta /with out ensemble\n0.901 prototypical net seresnext50 RGB image size 384 with bootstrap/no tta /with out ensemble</p>",
      "rawMarkdown": "0.899 prototypical net seresnext50 RGB image size 384 /no tta /with out ensemble\n0.901 prototypical net seresnext50 RGB image size 384 with bootstrap/no tta /with out ensemble",
      "votes": null
    },
    {
      "id": "480994",
      "postDate": "03/01/2019 00:26:59",
      "content": "<p>0.959 (Both public and private) - Single 4Kfold Siamese model based on DenseNet121 trained on 512x512 px input (no TTA).</p>",
      "rawMarkdown": "0.959 (Both public and private) - Single 4Kfold Siamese model based on DenseNet121 trained on 512x512 px input (no TTA).",
      "votes": null
    },
    {
      "id": "481009",
      "postDate": "03/01/2019 00:58:04",
      "content": "<p>public LB 0.966 ~ 0.971\n - arcface model based on densenet121\n - single model with TTA</p>",
      "rawMarkdown": "public LB 0.966 ~ 0.971\n - arcface model based on densenet121\n - single model with TTA",
      "votes": null
    },
    {
      "id": "481012",
      "postDate": "03/01/2019 00:59:38",
      "content": "<p>How did you make arcface work? :)</p>",
      "rawMarkdown": "How did you make arcface work? :)",
      "votes": null
    },
    {
      "id": "481016",
      "postDate": "03/01/2019 01:10:34",
      "content": "<p>Great work, congratulations! What input size and pooling? \nIt would be very nice if you could tell us the tricks to make it work! :)</p>",
      "rawMarkdown": "Great work, congratulations! What input size and pooling? \nIt would be very nice if you could tell us the tricks to make it work! :)",
      "votes": null
    },
    {
      "id": "481024",
      "postDate": "03/01/2019 01:17:18",
      "content": "<p>the input size is 320x320 and I didn't use pooling but concatenation.\nI'll share detailed solution. :)</p>",
      "rawMarkdown": "the input size is 320x320 and I didn't use pooling but concatenation.\nI'll share detailed solution. :)",
      "votes": null
    },
    {
      "id": "481065",
      "postDate": "03/01/2019 02:37:05",
      "content": "<p>hi pudae, How did you deal with new_whale?</p>",
      "rawMarkdown": "hi pudae, How did you deal with new_whale?",
      "votes": null
    },
    {
      "id": "481080",
      "postDate": "03/01/2019 03:01:24",
      "content": "<p>I didn't use new whales on training. \nFor inference, the threshold was selected so that the proportion of new whales are about 27%. </p>",
      "rawMarkdown": "I didn't use new whales on training. \nFor inference, the threshold was selected so that the proportion of new whales are about 27%.",
      "votes": null
    },
    {
      "id": "481238",
      "postDate": "03/01/2019 07:27:47",
      "content": "<p>Good to know. I as approching the similar idea for this competition, but I was able just to achive 0.935 LB using single model. Hope I would learn new stuff from your solution.</p>",
      "rawMarkdown": "Good to know. I as approching the similar idea for this competition, but I was able just to achive 0.935 LB using single model. Hope I would learn new stuff from your solution.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 475112,
      "author_name": "spsancti",
      "author_url": "",
      "post_date": "02/20/2019 09:29:26",
      "content": "<p>I am fairly close to 0.9 with a single model :)\nKeras, Siamese SEResNet18, one-cycle LR</p>",
      "votes": null,
      "replies": [
        {
          "id": 475136,
          "author_name": "",
          "author_url": "",
          "post_date": "02/20/2019 10:20:58",
          "content": "<p>That's really nice! Are you using the public kernel Siamese Network or it is your own Siamese keras-based design?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 475229,
      "author_name": "vshakhray",
      "author_url": "",
      "post_date": "02/20/2019 13:26:53",
      "content": "<p>\"What type of model? configuration? hyper parameters\"\nYou ask some pretty intimate questions, sir :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 475231,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "02/20/2019 13:28:59",
      "content": "<p>i can get &gt;0.90 using classification model + semi supervised learning (LB test set is treated as unlabeled train set) + combined \"softmax loss + metric loss\". Single model without TTA or ensemble</p>",
      "votes": null,
      "replies": [
        {
          "id": 475242,
          "author_name": "",
          "author_url": "",
          "post_date": "02/20/2019 13:47:18",
          "content": "<p>Thanks for the info Heng!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 475458,
          "author_name": "jeandebleau",
          "author_url": "",
          "post_date": "02/20/2019 19:18:18",
          "content": "<p>Do you mean metric learning loss on one or more deep layers + classification loss at the end ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 475614,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "02/21/2019 01:43:50",
          "content": "<p><a href=\"/jeandebleau\">@jeandebleau</a></p>\n\n<p>yes. something like that. you can also refer to the papers:\n<a href=\"https://arxiv.org/pdf/1901.08616.pdf\">https://arxiv.org/pdf/1901.08616.pdf</a>\n<a href=\"https://arxiv.org/pdf/1902.05509.pdf\">https://arxiv.org/pdf/1902.05509.pdf</a>\n<a href=\"https://arxiv.org/pdf/1811.12649.pdf\">https://arxiv.org/pdf/1811.12649.pdf</a>\n<a href=\"https://www.ijcai.org/proceedings/2017/0308.pdf\">https://www.ijcai.org/proceedings/2017/0308.pdf</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 475272,
      "author_name": "axel81",
      "author_url": "",
      "post_date": "02/20/2019 14:49:59",
      "content": "<p>I can get to 0.884+ using a single Siamese network, It looks like it can get me upto 0.89+ Lb score independently.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 475332,
      "author_name": "alexanderliao",
      "author_url": "",
      "post_date": "02/20/2019 16:12:34",
      "content": "<p>Classification, 0.95+</p>",
      "votes": null,
      "replies": [
        {
          "id": 475349,
          "author_name": "spsancti",
          "author_url": "",
          "post_date": "02/20/2019 16:45:36",
          "content": "<p>Insanity :D\nWaiting for your report after the end!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 475702,
          "author_name": "",
          "author_url": "",
          "post_date": "02/21/2019 04:51:09",
          "content": "<p>Wow that's great! Can't wait your competition solution report! :-)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 475420,
      "author_name": "interneuron",
      "author_url": "",
      "post_date": "02/20/2019 18:22:42",
      "content": "<p>Siamese, 0.91</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 475661,
      "author_name": "cooleel",
      "author_url": "",
      "post_date": "02/21/2019 03:42:05",
      "content": "<p>0.901 using Siamese with TTA</p>",
      "votes": null,
      "replies": [
        {
          "id": 476340,
          "author_name": "niuddd",
          "author_url": "",
          "post_date": "02/22/2019 01:55:39",
          "content": "<p>May I ask how many epochs was your network trained?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480935,
          "author_name": "cooleel",
          "author_url": "",
          "post_date": "02/28/2019 22:19:30",
          "content": "<p>Sorry for the late reply, I did 150 epochs just using the public kernel siamese network.  Can't wait to see other people's solution. Curious about what they have done to achieve such a high score on a single model. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 475703,
      "author_name": "xf1994",
      "author_url": "",
      "post_date": "02/21/2019 04:53:41",
      "content": "<p>My best single model on public lb is 0.943.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 475801,
      "author_name": "yayalee",
      "author_url": "",
      "post_date": "02/21/2019 08:06:37",
      "content": "<p>Siamese+resnet50, 0.897</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 476535,
      "author_name": "zjucor",
      "author_url": "",
      "post_date": "02/22/2019 10:02:18",
      "content": "<p>Siamese from public kernel+some modification = 0.90+\nIt is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.\nIt is amazing that only Classification can get 0.95+ ( from A.L.)....</p>",
      "votes": null,
      "replies": [
        {
          "id": 480940,
          "author_name": "cooleel",
          "author_url": "",
          "post_date": "02/28/2019 22:34:25",
          "content": "<blockquote>\n  <p>It is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.</p>\n</blockquote>\n\n<p>Same here.  There's not enough time when I finally realized this. Should try different ways earlier in the next one. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 476542,
      "author_name": "thuongdinh",
      "author_url": "",
      "post_date": "02/22/2019 10:12:27",
      "content": "<p>Siamese, 0.92+</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 476632,
      "author_name": "klickmal",
      "author_url": "",
      "post_date": "02/22/2019 12:59:05",
      "content": "<p>0.918 using Siamese with Lap based on Resnet34 in Pytorch</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 476690,
      "author_name": "a2015003713",
      "author_url": "",
      "post_date": "02/22/2019 14:49:59",
      "content": "<p>Siamese in public kernel : 0.906\nwith modified augmentation and TTA : 0.921</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 476748,
      "author_name": "abmokin",
      "author_url": "",
      "post_date": "02/22/2019 16:40:27",
      "content": "<p>Yes, I got the current result (public LB 0.951) using single model. Siamese network.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 477265,
      "author_name": "pinullmezon",
      "author_url": "",
      "post_date": "02/24/2019 09:02:10",
      "content": "<p>0.938 by single ResNet model with metric loss</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480289,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "02/28/2019 02:31:53",
      "content": "<p>check my comment at:\n<a href=\"https://www.kaggle.com/c/humpback-whale-identification/discussion/81085\">https://www.kaggle.com/c/humpback-whale-identification/discussion/81085</a></p>\n\n<p>LB 0.80 for resnet18 using 224 input. (threshold at 30% new-whale)\nLB 0.85 for resnet18 using 384 input. (threshold at 30% new-whale)\nLB 0.88 for resnet18 using 640 input. (threshold at 30% new-whale)\nLB 0.91 for resnet18 using 800 input. (threshold at 30% new-whale)</p>\n\n<p>all results are single model without TTA,without ensemble</p>",
      "votes": null,
      "replies": [
        {
          "id": 480574,
          "author_name": "arc144",
          "author_url": "",
          "post_date": "02/28/2019 11:20:19",
          "content": "<p>May I ask on how many epochs did you train your models Heng? Thanks</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480581,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "02/28/2019 11:28:31",
          "content": "<p>it varies. typically 100 to 200. train loss practically reaches zero and train accuracy reaches 100%. Sicne i am using very heavy augmentation (which induces label noise) i just let it run. it will not overfit that easily if there is non-static train label noise </p>\n\n<p>here is the trick. you can download some of the released results in the kernel page. e.g. i saw some results with LB 0.855 to 0.847.</p>\n\n<p>extract the top1 results that is common to these results. use this as validation for id whales. \nfor new-whale, you can just flip the test. (there are label error, so we do not expect 100% accuracy in validation but we do expect the validation accuracy to increase as iterations improve. we also know label accuracy is about 80% from the lb score) </p>\n\n<p>this can help you to decide the num of iterations you need for trainining</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480594,
          "author_name": "arc144",
          "author_url": "",
          "post_date": "02/28/2019 11:59:49",
          "content": "<p>Thanks for the tips. </p>\n\n<p>I ask because I feel that my models are underfitting. I'm using my local validation to decide whether to stop the training and I've been observing that it follows public LB until around 0.85LB, after that it seems that local validation reaches a top limit and therefore I'm stopping training too early, usually around 100 epochs.</p>\n\n<p>When you say 100 to 200 epochs, do you mean that even for the highest resolutions you train (finetune) 100~200 epochs on top of the pretrained on lower resolution? E.g. For 800x800 you trained ~100 epochs on top of the 640x640 pretrained model?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480597,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "02/28/2019 12:02:35",
          "content": "<p>for 224, it is about 200.  then the 224 model is used to initialize 384 and train for another 100 epoch, etc ...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480605,
          "author_name": "arc144",
          "author_url": "",
          "post_date": "02/28/2019 12:12:51",
          "content": "<p>Thank you</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480688,
      "author_name": "suicaokhoailang",
      "author_url": "",
      "post_date": "02/28/2019 14:20:45",
      "content": "<p>0.937 with a siamese, let's hope we won't drop to hell ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480950,
      "author_name": "soonhwankwon",
      "author_url": "",
      "post_date": "02/28/2019 23:09:36",
      "content": "<p>0.899 prototypical net seresnext50 RGB image size 384 /no tta /with out ensemble\n0.901 prototypical net seresnext50 RGB image size 384 with bootstrap/no tta /with out ensemble</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480994,
      "author_name": "zfturbo",
      "author_url": "",
      "post_date": "03/01/2019 00:26:59",
      "content": "<p>0.959 (Both public and private) - Single 4Kfold Siamese model based on DenseNet121 trained on 512x512 px input (no TTA).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 481009,
      "author_name": "pudae81",
      "author_url": "",
      "post_date": "03/01/2019 00:58:04",
      "content": "<p>public LB 0.966 ~ 0.971\n - arcface model based on densenet121\n - single model with TTA</p>",
      "votes": null,
      "replies": [
        {
          "id": 481012,
          "author_name": "oldufo",
          "author_url": "",
          "post_date": "03/01/2019 00:59:38",
          "content": "<p>How did you make arcface work? :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481016,
          "author_name": "arc144",
          "author_url": "",
          "post_date": "03/01/2019 01:10:34",
          "content": "<p>Great work, congratulations! What input size and pooling? \nIt would be very nice if you could tell us the tricks to make it work! :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481024,
          "author_name": "pudae81",
          "author_url": "",
          "post_date": "03/01/2019 01:17:18",
          "content": "<p>the input size is 320x320 and I didn't use pooling but concatenation.\nI'll share detailed solution. :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481065,
          "author_name": "qiaojian",
          "author_url": "",
          "post_date": "03/01/2019 02:37:05",
          "content": "<p>hi pudae, How did you deal with new_whale?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481080,
          "author_name": "pudae81",
          "author_url": "",
          "post_date": "03/01/2019 03:01:24",
          "content": "<p>I didn't use new whales on training. \nFor inference, the threshold was selected so that the proportion of new whales are about 27%. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481238,
          "author_name": "melgor",
          "author_url": "",
          "post_date": "03/01/2019 07:27:47",
          "content": "<p>Good to know. I as approching the similar idea for this competition, but I was able just to achive 0.935 LB using single model. Hope I would learn new stuff from your solution.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "475111": "Hi all, Is there anyone who achieve a single model with score of &gt; 0.9? I would like to request your input for this. Is it made in keras, pytorch or tensorflow? what type of model?configuration?hyperparameters? Many Thanks! :-)",
    "475112": "I am fairly close to 0.9 with a single model :)\nKeras, Siamese SEResNet18, one-cycle LR",
    "475136": "That's really nice! Are you using the public kernel Siamese Network or it is your own Siamese keras-based design?",
    "475229": "\"What type of model? configuration? hyper parameters\"\nYou ask some pretty intimate questions, sir :)",
    "475231": "i can get &gt;0.90 using classification model + semi supervised learning (LB test set is treated as unlabeled train set) + combined \"softmax loss + metric loss\". Single model without TTA or ensemble",
    "475242": "Thanks for the info Heng!",
    "475272": "I can get to 0.884+ using a single Siamese network, It looks like it can get me upto 0.89+ Lb score independently.",
    "475332": "Classification, 0.95+",
    "475349": "Insanity :D\nWaiting for your report after the end!",
    "475420": "Siamese, 0.91",
    "475458": "Do you mean metric learning loss on one or more deep layers + classification loss at the end ?",
    "475614": "jeandebleau\n\n\nyes. something like that. you can also refer to the papers:\nhttps://arxiv.org/pdf/1901.08616.pdf\nhttps://arxiv.org/pdf/1902.05509.pdf\nhttps://arxiv.org/pdf/1811.12649.pdf\nhttps://www.ijcai.org/proceedings/2017/0308.pdf",
    "475661": "0.901 using Siamese with TTA",
    "475702": "Wow that's great! Can't wait your competition solution report! :-)",
    "475703": "My best single model on public lb is 0.943.",
    "475801": "Siamese+resnet50, 0.897",
    "476340": "May I ask how many epochs was your network trained?",
    "476535": "Siamese from public kernel+some modification = 0.90+\nIt is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.\nIt is amazing that only Classification can get 0.95+ ( from A.L.)....",
    "476542": "Siamese, 0.92+",
    "476632": "0.918 using Siamese with Lap based on Resnet34 in Pytorch",
    "476690": "Siamese in public kernel : 0.906\nwith modified augmentation and TTA : 0.921",
    "476748": "Yes, I got the current result (public LB 0.951) using single model. Siamese network.",
    "477265": "0.938 by single ResNet model with metric loss",
    "480289": "check my comment at:\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/81085\n\nLB 0.80 for resnet18 using 224 input. (threshold at 30% new-whale)\nLB 0.85 for resnet18 using 384 input. (threshold at 30% new-whale)\nLB 0.88 for resnet18 using 640 input. (threshold at 30% new-whale)\nLB 0.91 for resnet18 using 800 input. (threshold at 30% new-whale)\n\nall results are single model without TTA,without ensemble",
    "480574": "May I ask on how many epochs did you train your models Heng? Thanks",
    "480581": "it varies. typically 100 to 200. train loss practically reaches zero and train accuracy reaches 100%. Sicne i am using very heavy augmentation (which induces label noise) i just let it run. it will not overfit that easily if there is non-static train label noise \n\nhere is the trick. you can download some of the released results in the kernel page. e.g. i saw some results with LB 0.855 to 0.847.\n\nextract the top1 results that is common to these results. use this as validation for id whales. \nfor new-whale, you can just flip the test. (there are label error, so we do not expect 100% accuracy in validation but we do expect the validation accuracy to increase as iterations improve. we also know label accuracy is about 80% from the lb score) \n\nthis can help you to decide the num of iterations you need for trainining",
    "480594": "Thanks for the tips. \n\nI ask because I feel that my models are underfitting. I'm using my local validation to decide whether to stop the training and I've been observing that it follows public LB until around 0.85LB, after that it seems that local validation reaches a top limit and therefore I'm stopping training too early, usually around 100 epochs.\n\nWhen you say 100 to 200 epochs, do you mean that even for the highest resolutions you train (finetune) 100~200 epochs on top of the pretrained on lower resolution? E.g. For 800x800 you trained ~100 epochs on top of the 640x640 pretrained model?",
    "480597": "for 224, it is about 200.  then the 224 model is used to initialize 384 and train for another 100 epoch, etc ...",
    "480605": "Thank you",
    "480688": "0.937 with a siamese, let's hope we won't drop to hell ;)",
    "480935": "Sorry for the late reply, I did 150 epochs just using the public kernel siamese network.  Can't wait to see other people's solution. Curious about what they have done to achieve such a high score on a single model.",
    "480940": "&gt;It is a pity that I spend too much time on mattiport's solution, it seems that there exists better method.\n\nSame here.  There's not enough time when I finally realized this. Should try different ways earlier in the next one.",
    "480950": "0.899 prototypical net seresnext50 RGB image size 384 /no tta /with out ensemble\n0.901 prototypical net seresnext50 RGB image size 384 with bootstrap/no tta /with out ensemble",
    "480994": "0.959 (Both public and private) - Single 4Kfold Siamese model based on DenseNet121 trained on 512x512 px input (no TTA).",
    "481009": "public LB 0.966 ~ 0.971\n - arcface model based on densenet121\n - single model with TTA",
    "481012": "How did you make arcface work? :)",
    "481016": "Great work, congratulations! What input size and pooling? \nIt would be very nice if you could tell us the tricks to make it work! :)",
    "481024": "the input size is 320x320 and I didn't use pooling but concatenation.\nI'll share detailed solution. :)",
    "481065": "hi pudae, How did you deal with new_whale?",
    "481080": "I didn't use new whales on training. \nFor inference, the threshold was selected so that the proportion of new whales are about 27%.",
    "481238": "Good to know. I as approching the similar idea for this competition, but I was able just to achive 0.935 LB using single model. Hope I would learn new stuff from your solution."
  },
  "source": "meta"
}