{
  "id": 256178,
  "title": "How to train efficientnetv2 ",
  "url": "/competitions/siim-covid19-detection/discussion/256178",
  "author_name": "Drzhuzhe",
  "post_date": "2021-07-31T08:45:00.330000",
  "votes": 2,
  "comment_count": 38,
  "views": 0,
  "content": "<p>I mess with efficientnetv2 a few weeks ,<br>\nBut seems still not come up with a good result  </p>\n<p>it seem even lower than efficinentB3 baselinemI<br>\n can only get some cv between 3.6 - 3.7</p>\n<p>Base on my baseline 512 image with a mask detector head on block 5<br>\nAnd add some addition method </p>\n<ol>\n<li><p>0.5 dropout </p></li>\n<li><p>aux loss change from BCE to 1 x BCE + 1 x lovsaz loss</p></li>\n<li><p>40 epoch with reduce learning rate on plateau , start laerning rate 1e-3</p></li>\n<li><p>batch size 18 </p></li>\n<li><p>freeze layer finetune </p></li>\n</ol>\n<p>Maybe someone can give me some suggestion for some methods to try ?</p>",
  "messages": [
    {
      "id": 1405813,
      "postDate": "2021-07-31T08:45:00.330Z",
      "content": "<p>I mess with efficientnetv2 a few weeks ,<br>\nBut seems still not come up with a good result  </p>\n<p>it seem even lower than efficinentB3 baselinemI<br>\n can only get some cv between 3.6 - 3.7</p>\n<p>Base on my baseline 512 image with a mask detector head on block 5<br>\nAnd add some addition method </p>\n<ol>\n<li><p>0.5 dropout </p></li>\n<li><p>aux loss change from BCE to 1 x BCE + 1 x lovsaz loss</p></li>\n<li><p>40 epoch with reduce learning rate on plateau , start laerning rate 1e-3</p></li>\n<li><p>batch size 18 </p></li>\n<li><p>freeze layer finetune </p></li>\n</ol>\n<p>Maybe someone can give me some suggestion for some methods to try ?</p>",
      "rawMarkdown": "I mess with efficientnetv2 a few weeks ,\nBut seems still not come up with a good result  \n\nit seem even lower than efficinentB3 baselinemI\n can only get some cv between 3.6 - 3.7\n\n\nBase on my baseline 512 image with a mask detector head on block 5\nAnd add some addition method \n\n1. 0.5 dropout \n\n2. aux loss change from BCE to 1 x BCE + 1 x lovsaz loss\n\n3. 40 epoch with reduce learning rate on plateau , start laerning rate 1e-3\n\n4. batch size 18 \n\n5. freeze layer finetune \n\nMaybe someone can give me some suggestion for some methods to try ?",
      "votes": 2
    },
    {
      "id": 1454011,
      "postDate": "2021-08-06T04:29:09.017Z",
      "content": "<p>One reminder, never use batch size that isn't a power of 2.</p>",
      "rawMarkdown": "One reminder, never use batch size that isn't a power of 2.",
      "replies": [
        {
          "id": 1456901,
          "postDate": "2021-08-07T06:25:11.883Z",
          "content": "<p>Why?          </p>",
          "rawMarkdown": "Why?          "
        },
        {
          "id": 1457843,
          "postDate": "2021-08-07T15:34:45.170Z",
          "content": "<p>In experience, in old models batch sizes which were not the power of 2 were rarely used and were unlikely to give good performance, actually, it is not the power of 2 but the multiple of 4 which should be said, I dont exactly know the reason for their low performance in models back in the day, but right now models tends to use batchnorm layers which are a big help to them, if you dont give enough batch size  or give out too big of a batch size, batchnorm layers tends to fail causing a huge loss in performance…</p>",
          "rawMarkdown": "In experience, in old models batch sizes which were not the power of 2 were rarely used and were unlikely to give good performance, actually, it is not the power of 2 but the multiple of 4 which should be said, I dont exactly know the reason for their low performance in models back in the day, but right now models tends to use batchnorm layers which are a big help to them, if you dont give enough batch size  or give out too big of a batch size, batchnorm layers tends to fail causing a huge loss in performance..."
        }
      ]
    },
    {
      "id": 1406106,
      "postDate": "2021-07-31T14:00:46.210Z",
      "content": "<p>Up your batch size to 32, batch norm can be incredibly important for efficientnetv2…</p>",
      "rawMarkdown": "Up your batch size to 32, batch norm can be incredibly important for efficientnetv2...",
      "replies": [
        {
          "id": 1406886,
          "postDate": "2021-08-01T09:25:05.873Z",
          "content": "<p>I only use a simple aux head on block 5 , I wonder what if I use a  down sampling aux head attach to efficinentnetv5 block1 - 5</p>",
          "rawMarkdown": "I only use a simple aux head on block 5 , I wonder what if I use a  down sampling aux head attach to efficinentnetv5 block1 - 5"
        },
        {
          "id": 1406889,
          "postDate": "2021-08-01T09:28:42.167Z",
          "content": "<p>ennn , Thanks I decide to try to use gradiant accumlate to increase batch norm</p>",
          "rawMarkdown": "ennn , Thanks I decide to try to use gradiant accumlate to increase batch norm"
        },
        {
          "id": 1406909,
          "postDate": "2021-08-01T10:05:14.167Z",
          "content": "<p><a href=\"https://www.kaggle.com/harshitsheoran\" target=\"_blank\">@harshitsheoran</a> batch size 16 with mixed precision equal batch size 32 according to your comment? Or difference?</p>",
          "rawMarkdown": "@harshitsheoran batch size 16 with mixed precision equal batch size 32 according to your comment? Or difference?"
        },
        {
          "id": 1406922,
          "postDate": "2021-08-01T10:24:44.130Z",
          "content": "<p><a href=\"https://www.kaggle.com/magiccard\" target=\"_blank\">@magiccard</a> Different</p>",
          "rawMarkdown": "@magiccard Different",
          "votes": 1
        },
        {
          "id": 1406929,
          "postDate": "2021-08-01T10:35:32.307Z",
          "content": "<p><a href=\"https://www.kaggle.com/harshitsheoran\" target=\"_blank\">@harshitsheoran</a> Thank for your info, I just have a GPU with 16gb Ram so I can not change batch size to 32. Do you have any idea?</p>",
          "rawMarkdown": "@harshitsheoran Thank for your info, I just have a GPU with 16gb Ram so I can not change batch size to 32. Do you have any idea?"
        },
        {
          "id": 1453598,
          "postDate": "2021-08-05T23:42:29.133Z",
          "content": "<p>Use TPU <a href=\"https://www.kaggle.com/sreevishnudamodaran/siim-effnetv2-keras-study-train-tpu-cv0-805\" target=\"_blank\">https://www.kaggle.com/sreevishnudamodaran/siim-effnetv2-keras-study-train-tpu-cv0-805</a></p>",
          "rawMarkdown": "Use TPU https://www.kaggle.com/sreevishnudamodaran/siim-effnetv2-keras-study-train-tpu-cv0-805",
          "votes": -1
        }
      ]
    },
    {
      "id": 1406075,
      "postDate": "2021-07-31T13:21:40.903Z",
      "content": "<p>how do you control your learning rate ? val_loss? mAP ?</p>",
      "rawMarkdown": "how do you control your learning rate ? val_loss? mAP ?",
      "replies": [
        {
          "id": 1406873,
          "postDate": "2021-08-01T09:17:15.727Z",
          "content": "<p>CE valid_loss </p>",
          "rawMarkdown": " CE valid_loss "
        },
        {
          "id": 1406926,
          "postDate": "2021-08-01T10:29:24.610Z",
          "content": "<p>Do you train efficientnetv2_m with pretrained weights?<br>\nIf you would like to train efficientnetv2_m from scratch with heng's baseline solution,<br>\nit might be required to  add more regularization.</p>",
          "rawMarkdown": "Do you train efficientnetv2_m with pretrained weights?\nIf you would like to train efficientnetv2_m from scratch with heng's baseline solution,\nit might be required to  add more regularization.\n"
        },
        {
          "id": 1406928,
          "postDate": "2021-08-01T10:33:06.420Z",
          "content": "<p>I use tf_efficientnetv2_m_im21 weight</p>",
          "rawMarkdown": "I use tf_efficientnetv2_m_im21 weight"
        },
        {
          "id": 1406980,
          "postDate": "2021-08-01T11:44:47.547Z",
          "content": "<p>I keep heng's baseline code and transfer to tf_efficientnetv2_m/efficientnetv2_rw_m without any problem</p>",
          "rawMarkdown": "I keep heng's baseline code and transfer to tf_efficientnetv2_m/efficientnetv2_rw_m without any problem"
        },
        {
          "id": 1407018,
          "postDate": "2021-08-01T12:38:30.203Z",
          "content": "<p>Which batch size did you use? <a href=\"https://www.kaggle.com/atom1231\" target=\"_blank\">@atom1231</a> </p>",
          "rawMarkdown": "Which batch size did you use? @atom1231 "
        },
        {
          "id": 1407070,
          "postDate": "2021-08-01T13:43:26.860Z",
          "content": "<p>8  or 16 with  mixed precision</p>",
          "rawMarkdown": "8  or 16 with  mixed precision",
          "votes": 1
        },
        {
          "id": 1407100,
          "postDate": "2021-08-01T14:22:18.160Z",
          "content": "<p>Thank, but i tried batch size 8 or 16 with imgsz 512 on tf_effcienet_v2_m CV about 0.36~0.37.<br>\nAm i wrong somewhere?</p>",
          "rawMarkdown": "Thank, but i tried batch size 8 or 16 with imgsz 512 on tf_effcienet_v2_m CV about 0.36~0.37.\nAm i wrong somewhere?"
        },
        {
          "id": 1407133,
          "postDate": "2021-08-01T14:50:07.723Z",
          "content": "<p>finetune the network for each fold…</p>",
          "rawMarkdown": "finetune the network for each fold..."
        },
        {
          "id": 1407148,
          "postDate": "2021-08-01T15:02:15.903Z",
          "content": "<p>thank for your info! Can i ask you what is the best mean CV that you gain?</p>",
          "rawMarkdown": "thank for your info! Can i ask you what is the best mean CV that you gain?"
        },
        {
          "id": 1407162,
          "postDate": "2021-08-01T15:20:26.873Z",
          "content": "<p>I got cv 0.383.x in tf_effcienet_v2_m . Possibly up to 0.385+ with more fine-tuning</p>\n<p>also ref to <br>\n<a href=\"https://www.kaggle.com/c/siim-covid19-detection/discussion/248442\" target=\"_blank\">https://www.kaggle.com/c/siim-covid19-detection/discussion/248442</a></p>",
          "rawMarkdown": "I got cv 0.383.x in tf_effcienet_v2_m . Possibly up to 0.385+ with more fine-tuning\n\nalso ref to \nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/248442\n",
          "votes": 1
        },
        {
          "id": 1407167,
          "postDate": "2021-08-01T15:26:20.073Z",
          "content": "<p>thank!! I will try finetune the model</p>",
          "rawMarkdown": "thank!! I will try finetune the model"
        },
        {
          "id": 1407539,
          "postDate": "2021-08-02T01:52:50.857Z",
          "content": "<p>which method for finetune？</p>",
          "rawMarkdown": "which method for finetune？\n"
        },
        {
          "id": 1407587,
          "postDate": "2021-08-02T03:09:25.657Z",
          "content": "<p>May be add more regularization? I tried gradient accum but still out of CUDA :(( Did you try it? <a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a> </p>",
          "rawMarkdown": "May be add more regularization? I tried gradient accum but still out of CUDA :(( Did you try it? @drzhuzhe "
        },
        {
          "id": 1407635,
          "postDate": "2021-08-02T04:16:50.807Z",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a>  nothing special  , just decrease the learning rate (similar to what you did in effb3a)</p>",
          "rawMarkdown": "@drzhuzhe  nothing special  , just decrease the learning rate (similar to what you did in effb3a)\n",
          "votes": 1
        },
        {
          "id": 1408546,
          "postDate": "2021-08-02T13:50:46.777Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 1444455,
          "postDate": "2021-08-04T06:00:44.717Z",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a>  I suggest to review whole flow again and try other models first<br>\nAfter the competition ends I can share the source code If you still do not solve the problem  .</p>",
          "rawMarkdown": "@drzhuzhe  I suggest to review whole flow again and try other models first\nAfter the competition ends I can share the source code If you still do not solve the problem  ."
        },
        {
          "id": 1444911,
          "postDate": "2021-08-04T07:19:31.360Z",
          "content": "<p>Thanks, I will very grad to read you code  after this match end as a review</p>",
          "rawMarkdown": "Thanks, I will very grad to read you code  after this match end as a review"
        },
        {
          "id": 1446834,
          "postDate": "2021-08-04T13:15:41.437Z",
          "content": "<p><a href=\"https://www.kaggle.com/atom1231\" target=\"_blank\">@atom1231</a> heyy, Did you change the data (according to the baseline of heng, he set StratifiedKFold), i guess you changed to GroupKFold?</p>",
          "rawMarkdown": "@atom1231 heyy, Did you change the data (according to the baseline of heng, he set StratifiedKFold), i guess you changed to GroupKFold?"
        },
        {
          "id": 1447236,
          "postDate": "2021-08-04T14:26:40.697Z",
          "content": "<p>no , I use default \"df_fold_rand830.csv\" in this case.</p>",
          "rawMarkdown": "no , I use default \"df_fold_rand830.csv\" in this case.",
          "votes": 1
        },
        {
          "id": 1447829,
          "postDate": "2021-08-04T16:00:41.940Z",
          "content": "<p>Thank for your reply, i will try again, hope that increase my LB before compettition end</p>",
          "rawMarkdown": "Thank for your reply, i will try again, hope that increase my LB before compettition end"
        },
        {
          "id": 1450359,
          "postDate": "2021-08-05T05:03:36.473Z",
          "content": "<p>To me <br>\nmy mistake is data augment is not enough <br>\nwhen i use data augment like this <a href=\"https://www.kaggle.com/southsakura/kaggle\" target=\"_blank\">https://www.kaggle.com/southsakura/kaggle</a><br>\nI got 3.9+ 3.9+ 3.8 3.8 3.76 <br>\nand training become more stable <br>\nBTW I use learning rate only 5 patience</p>",
          "rawMarkdown": "To me \nmy mistake is data augment is not enough \nwhen i use data augment like this https://www.kaggle.com/southsakura/kaggle\nI got 3.9+ 3.9+ 3.8 3.8 3.76 \nand training become more stable \nBTW I use learning rate only 5 patience",
          "votes": 2
        },
        {
          "id": 1451161,
          "postDate": "2021-08-05T09:18:34.517Z",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a>  Oh really impressive score, i have a question you got this scores at epoch 10-20 or 20-40? </p>",
          "rawMarkdown": "@drzhuzhe  Oh really impressive score, i have a question you got this scores at epoch 10-20 or 20-40? "
        },
        {
          "id": 1451299,
          "postDate": "2021-08-05T10:03:26.033Z",
          "content": "<p>shortest is 20 ep. longest is about 35 ep</p>",
          "rawMarkdown": "shortest is 20 ep. longest is about 35 ep",
          "votes": 1
        },
        {
          "id": 1451317,
          "postDate": "2021-08-05T10:11:38.987Z",
          "content": "<p>thanks i guess that your new LB ~ 0.456 with study only</p>",
          "rawMarkdown": "thanks i guess that your new LB ~ 0.456 with study only"
        },
        {
          "id": 1456852,
          "postDate": "2021-08-07T05:51:18.287Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 1457541,
          "postDate": "2021-08-07T13:13:51.567Z",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a> Did you change anything other? Which model did you use?</p>",
          "rawMarkdown": "@drzhuzhe Did you change anything other? Which model did you use?"
        },
        {
          "id": 1457575,
          "postDate": "2021-08-07T13:28:23.203Z",
          "content": "<p>Ennn, I will make it public after this match end</p>",
          "rawMarkdown": "Ennn, I will make it public after this match end",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1454011,
      "author_name": "Logic",
      "author_url": "",
      "post_date": "2021-08-06T04:29:09.017000",
      "content": "<p>One reminder, never use batch size that isn't a power of 2.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1456901,
          "author_name": "Dieter",
          "author_url": "",
          "post_date": "2021-08-07T06:25:11.883000",
          "content": "<p>Why?          </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1457843,
          "author_name": "Harshit Sheoran",
          "author_url": "",
          "post_date": "2021-08-07T15:34:45.170000",
          "content": "<p>In experience, in old models batch sizes which were not the power of 2 were rarely used and were unlikely to give good performance, actually, it is not the power of 2 but the multiple of 4 which should be said, I dont exactly know the reason for their low performance in models back in the day, but right now models tends to use batchnorm layers which are a big help to them, if you dont give enough batch size  or give out too big of a batch size, batchnorm layers tends to fail causing a huge loss in performance…</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1406106,
      "author_name": "Harshit Sheoran",
      "author_url": "",
      "post_date": "2021-07-31T14:00:46.210000",
      "content": "<p>Up your batch size to 32, batch norm can be incredibly important for efficientnetv2…</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1406886,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-01T09:25:05.873000",
          "content": "<p>I only use a simple aux head on block 5 , I wonder what if I use a  down sampling aux head attach to efficinentnetv5 block1 - 5</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1406889,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-01T09:28:42.167000",
          "content": "<p>ennn , Thanks I decide to try to use gradiant accumlate to increase batch norm</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1406909,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-01T10:05:14.167000",
          "content": "<p><a href=\"https://www.kaggle.com/harshitsheoran\" target=\"_blank\">@harshitsheoran</a> batch size 16 with mixed precision equal batch size 32 according to your comment? Or difference?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1406922,
          "author_name": "Harshit Sheoran",
          "author_url": "",
          "post_date": "2021-08-01T10:24:44.130000",
          "content": "<p><a href=\"https://www.kaggle.com/magiccard\" target=\"_blank\">@magiccard</a> Different</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1406929,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-01T10:35:32.307000",
          "content": "<p><a href=\"https://www.kaggle.com/harshitsheoran\" target=\"_blank\">@harshitsheoran</a> Thank for your info, I just have a GPU with 16gb Ram so I can not change batch size to 32. Do you have any idea?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1453598,
          "author_name": "Logic",
          "author_url": "",
          "post_date": "2021-08-05T23:42:29.133000",
          "content": "<p>Use TPU <a href=\"https://www.kaggle.com/sreevishnudamodaran/siim-effnetv2-keras-study-train-tpu-cv0-805\" target=\"_blank\">https://www.kaggle.com/sreevishnudamodaran/siim-effnetv2-keras-study-train-tpu-cv0-805</a></p>",
          "votes": -1,
          "replies": []
        }
      ]
    },
    {
      "id": 1406075,
      "author_name": "atom1231",
      "author_url": "",
      "post_date": "2021-07-31T13:21:40.903000",
      "content": "<p>how do you control your learning rate ? val_loss? mAP ?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1406873,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-01T09:17:15.727000",
          "content": "<p>CE valid_loss </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1406926,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-01T10:29:24.610000",
          "content": "<p>Do you train efficientnetv2_m with pretrained weights?<br>\nIf you would like to train efficientnetv2_m from scratch with heng's baseline solution,<br>\nit might be required to  add more regularization.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1406928,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-01T10:33:06.420000",
          "content": "<p>I use tf_efficientnetv2_m_im21 weight</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1406980,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-01T11:44:47.547000",
          "content": "<p>I keep heng's baseline code and transfer to tf_efficientnetv2_m/efficientnetv2_rw_m without any problem</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407018,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-01T12:38:30.203000",
          "content": "<p>Which batch size did you use? <a href=\"https://www.kaggle.com/atom1231\" target=\"_blank\">@atom1231</a> </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407070,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-01T13:43:26.860000",
          "content": "<p>8  or 16 with  mixed precision</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1407100,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-01T14:22:18.160000",
          "content": "<p>Thank, but i tried batch size 8 or 16 with imgsz 512 on tf_effcienet_v2_m CV about 0.36~0.37.<br>\nAm i wrong somewhere?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407133,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-01T14:50:07.723000",
          "content": "<p>finetune the network for each fold…</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407148,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-01T15:02:15.903000",
          "content": "<p>thank for your info! Can i ask you what is the best mean CV that you gain?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407162,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-01T15:20:26.873000",
          "content": "<p>I got cv 0.383.x in tf_effcienet_v2_m . Possibly up to 0.385+ with more fine-tuning</p>\n<p>also ref to <br>\n<a href=\"https://www.kaggle.com/c/siim-covid19-detection/discussion/248442\" target=\"_blank\">https://www.kaggle.com/c/siim-covid19-detection/discussion/248442</a></p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1407167,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-01T15:26:20.073000",
          "content": "<p>thank!! I will try finetune the model</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407539,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-02T01:52:50.857000",
          "content": "<p>which method for finetune？</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407587,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-02T03:09:25.657000",
          "content": "<p>May be add more regularization? I tried gradient accum but still out of CUDA :(( Did you try it? <a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a> </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1407635,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-02T04:16:50.807000",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a>  nothing special  , just decrease the learning rate (similar to what you did in effb3a)</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1408546,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-08-02T13:50:46.777000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1444455,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-04T06:00:44.717000",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a>  I suggest to review whole flow again and try other models first<br>\nAfter the competition ends I can share the source code If you still do not solve the problem  .</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1444911,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-04T07:19:31.360000",
          "content": "<p>Thanks, I will very grad to read you code  after this match end as a review</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1446834,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-04T13:15:41.437000",
          "content": "<p><a href=\"https://www.kaggle.com/atom1231\" target=\"_blank\">@atom1231</a> heyy, Did you change the data (according to the baseline of heng, he set StratifiedKFold), i guess you changed to GroupKFold?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1447236,
          "author_name": "atom1231",
          "author_url": "",
          "post_date": "2021-08-04T14:26:40.697000",
          "content": "<p>no , I use default \"df_fold_rand830.csv\" in this case.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1447829,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-04T16:00:41.940000",
          "content": "<p>Thank for your reply, i will try again, hope that increase my LB before compettition end</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1450359,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-05T05:03:36.473000",
          "content": "<p>To me <br>\nmy mistake is data augment is not enough <br>\nwhen i use data augment like this <a href=\"https://www.kaggle.com/southsakura/kaggle\" target=\"_blank\">https://www.kaggle.com/southsakura/kaggle</a><br>\nI got 3.9+ 3.9+ 3.8 3.8 3.76 <br>\nand training become more stable <br>\nBTW I use learning rate only 5 patience</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1451161,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-05T09:18:34.517000",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a>  Oh really impressive score, i have a question you got this scores at epoch 10-20 or 20-40? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1451299,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-05T10:03:26.033000",
          "content": "<p>shortest is 20 ep. longest is about 35 ep</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1451317,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-05T10:11:38.987000",
          "content": "<p>thanks i guess that your new LB ~ 0.456 with study only</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1456852,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-08-07T05:51:18.287000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1457541,
          "author_name": "Cloudyy",
          "author_url": "",
          "post_date": "2021-08-07T13:13:51.567000",
          "content": "<p><a href=\"https://www.kaggle.com/drzhuzhe\" target=\"_blank\">@drzhuzhe</a> Did you change anything other? Which model did you use?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1457575,
          "author_name": "Drzhuzhe",
          "author_url": "",
          "post_date": "2021-08-07T13:28:23.203000",
          "content": "<p>Ennn, I will make it public after this match end</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1405813": "I mess with efficientnetv2 a few weeks ,\nBut seems still not come up with a good result  \n\nit seem even lower than efficinentB3 baselinemI\n can only get some cv between 3.6 - 3.7\n\n\nBase on my baseline 512 image with a mask detector head on block 5\nAnd add some addition method \n\n1. 0.5 dropout \n\n2. aux loss change from BCE to 1 x BCE + 1 x lovsaz loss\n\n3. 40 epoch with reduce learning rate on plateau , start laerning rate 1e-3\n\n4. batch size 18 \n\n5. freeze layer finetune \n\nMaybe someone can give me some suggestion for some methods to try ?",
    "1454011": "One reminder, never use batch size that isn't a power of 2.",
    "1406106": "Up your batch size to 32, batch norm can be incredibly important for efficientnetv2...",
    "1406075": "how do you control your learning rate ? val_loss? mAP ?"
  }
}