{
  "id": 71584,
  "title": "Question why my Unet resent 34 model doesn't improve",
  "url": "/competitions/airbus-ship-detection/discussion/71584",
  "author_name": "",
  "post_date": "2018-11-15T00:01:52.311614100Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>First, congrats everyone finished the competition! I would like to ask why my unet_resnet34 model does not improve beyond LB score of ~0.710. I used binary cross entropy loss and started with all layers unfrozen. The initial learning rate was 1e-5 with Adam. Then after 10 epochs, I froze the resnet layers, and tuned down the learning rate to 1e-6, and trained for 20 more epochs. I also added some dropout layers to prevent overfitting. However, it didn't seem to improve my LB score, albeit the loss of validation set was decreasing. Could some people got &gt;0.74 LB score with Unet resnet 34 share about how they made it? Thanks! </p>",
  "messages": [
    {
      "id": "421359",
      "postDate": "11/15/2018 00:01:52",
      "content": "<p>First, congrats everyone finished the competition! I would like to ask why my unet_resnet34 model does not improve beyond LB score of ~0.710. I used binary cross entropy loss and started with all layers unfrozen. The initial learning rate was 1e-5 with Adam. Then after 10 epochs, I froze the resnet layers, and tuned down the learning rate to 1e-6, and trained for 20 more epochs. I also added some dropout layers to prevent overfitting. However, it didn't seem to improve my LB score, albeit the loss of validation set was decreasing. Could some people got &gt;0.74 LB score with Unet resnet 34 share about how they made it? Thanks! </p>",
      "rawMarkdown": "First, congrats everyone finished the competition! I would like to ask why my unet_resnet34 model does not improve beyond LB score of ~0.710. I used binary cross entropy loss and started with all layers unfrozen. The initial learning rate was 1e-5 with Adam. Then after 10 epochs, I froze the resnet layers, and tuned down the learning rate to 1e-6, and trained for 20 more epochs. I also added some dropout layers to prevent overfitting. However, it didn't seem to improve my LB score, albeit the loss of validation set was decreasing. Could some people got &gt;0.74 LB score with Unet resnet 34 share about how they made it? Thanks!",
      "votes": null
    },
    {
      "id": "421368",
      "postDate": "11/15/2018 00:08:33",
      "content": "<p>You should try a different loss. Binary crossentropy is quite bad when we have high unbalancing (much more non-ship pixels than ship). You could try a combination of BCE + dice, focal loss or lovasz.</p>\n\n<p>Another thing that usually helps a lot is adding  <a href=\"https://arxiv.org/abs/1411.5752\">hypercolumns</a> to your decoder. You could also try <a href=\"https://arxiv.org/abs/1803.02579\">squeeze and excitation blocks</a>.</p>\n\n<p>Finally, ensembling usually helps. Try training more folds and average them. </p>",
      "rawMarkdown": "You should try a different loss. Binary crossentropy is quite bad when we have high unbalancing (much more non-ship pixels than ship). You could try a combination of BCE + dice, focal loss or lovasz.\n\nAnother thing that usually helps a lot is adding  [hypercolumns](https://arxiv.org/abs/1411.5752) to your decoder. You could also try [squeeze and excitation blocks](https://arxiv.org/abs/1803.02579).\n\nFinally, ensembling usually helps. Try training more folds and average them.",
      "votes": null
    },
    {
      "id": "421371",
      "postDate": "11/15/2018 00:15:13",
      "content": "<p>Thanks for your suggestions! I'm wondering would you mind also share some idea how to prevent overfitting? Since I noticed I had a pretty significant shakeup on the leaderboard.</p>",
      "rawMarkdown": "Thanks for your suggestions! I'm wondering would you mind also share some idea how to prevent overfitting? Since I noticed I had a pretty significant shakeup on the leaderboard.",
      "votes": null
    },
    {
      "id": "421393",
      "postDate": "11/15/2018 00:35:54",
      "content": "<p>That's also one of my particular issues.. But I guess a good CV is mandatory, ensembling also helps.</p>",
      "rawMarkdown": "That's also one of my particular issues.. But I guess a good CV is mandatory, ensembling also helps.",
      "votes": null
    },
    {
      "id": "421394",
      "postDate": "11/15/2018 00:37:40",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 421368,
      "author_name": "arc144",
      "author_url": "",
      "post_date": "11/15/2018 00:08:33",
      "content": "<p>You should try a different loss. Binary crossentropy is quite bad when we have high unbalancing (much more non-ship pixels than ship). You could try a combination of BCE + dice, focal loss or lovasz.</p>\n\n<p>Another thing that usually helps a lot is adding  <a href=\"https://arxiv.org/abs/1411.5752\">hypercolumns</a> to your decoder. You could also try <a href=\"https://arxiv.org/abs/1803.02579\">squeeze and excitation blocks</a>.</p>\n\n<p>Finally, ensembling usually helps. Try training more folds and average them. </p>",
      "votes": null,
      "replies": [
        {
          "id": 421371,
          "author_name": "zli117",
          "author_url": "",
          "post_date": "11/15/2018 00:15:13",
          "content": "<p>Thanks for your suggestions! I'm wondering would you mind also share some idea how to prevent overfitting? Since I noticed I had a pretty significant shakeup on the leaderboard.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 421393,
          "author_name": "arc144",
          "author_url": "",
          "post_date": "11/15/2018 00:35:54",
          "content": "<p>That's also one of my particular issues.. But I guess a good CV is mandatory, ensembling also helps.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 421394,
          "author_name": "zli117",
          "author_url": "",
          "post_date": "11/15/2018 00:37:40",
          "content": "<p>Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "421359": "First, congrats everyone finished the competition! I would like to ask why my unet_resnet34 model does not improve beyond LB score of ~0.710. I used binary cross entropy loss and started with all layers unfrozen. The initial learning rate was 1e-5 with Adam. Then after 10 epochs, I froze the resnet layers, and tuned down the learning rate to 1e-6, and trained for 20 more epochs. I also added some dropout layers to prevent overfitting. However, it didn't seem to improve my LB score, albeit the loss of validation set was decreasing. Could some people got &gt;0.74 LB score with Unet resnet 34 share about how they made it? Thanks!",
    "421368": "You should try a different loss. Binary crossentropy is quite bad when we have high unbalancing (much more non-ship pixels than ship). You could try a combination of BCE + dice, focal loss or lovasz.\n\nAnother thing that usually helps a lot is adding  [hypercolumns](https://arxiv.org/abs/1411.5752) to your decoder. You could also try [squeeze and excitation blocks](https://arxiv.org/abs/1803.02579).\n\nFinally, ensembling usually helps. Try training more folds and average them.",
    "421371": "Thanks for your suggestions! I'm wondering would you mind also share some idea how to prevent overfitting? Since I noticed I had a pretty significant shakeup on the leaderboard.",
    "421393": "That's also one of my particular issues.. But I guess a good CV is mandatory, ensembling also helps.",
    "421394": "Thank you!"
  },
  "source": "meta"
}