{
  "id": 254177,
  "title": "Tf Models in TIMM converging faster than pytorch version",
  "url": "/competitions/siim-covid19-detection/discussion/254177",
  "author_name": "",
  "post_date": "2021-07-20T12:54:24.494082600Z",
  "votes": 4,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi folks,</p>\n<p>Since the beginning of this competition, I've been using Pytorch/TIMM but using TF models because for me the pytorch version takes too long to converge and leads to worse results.</p>\n<p>Here is some logs:</p>\n<p><strong>tf_efficientnetv2_m</strong></p>\n<p>Epoch 2: 100%|##########| 529/529 [03:09&lt;00:00,  2.80it/s, loss=1.66, v_num=0_0, val_loss=1.670, val_auc=0.768, <strong>val_map=0.327</strong>, train_auc=0.740, train_map=0.307]<br>\nEpoch 3: 100%|#########9| 528/529 [03:04&lt;00:00,  2.86it/s,., val_map=0.340, train_auc=0.766, train_map=0.330]<br>\nEpoch 4: 100%|#########9| 528/529 [03:13&lt;00:00,  2.73it/s, 1.1895e-04. <strong>val_map=0.348</strong>, train_auc=0.794, train_map=0.350]<br>\nEpoch 5: 100%|##########| 529/529 [03:17&lt;00:00,  2.68it/s, loss=1.48, v_num=0_0, val_loss=1.610, val_auc=0.795, <strong>val_map=0.352</strong>, train_auc=0.812, train_map=0.368]</p>\n<p><strong>efficientnetv2_m</strong></p>\n<p>Epoch 2: 100%|#########9| 528/529 [03:06&lt;00:00,  2.83it/s, loAdjusting learning rate of group 0 to 1.1988e-04., val_map=0.192, train_auc=0.510, train_map=0.170][<br>\nEpoch 3: 100%|##########| 529/529 [03:06&lt;00:00,  2.84it/s, loss=1.92, v_num=0_0, val_loss=1.950, val_auc=0.544, <strong>val_map=0.188</strong>, train_auc=0.522, train_map=0.179]<br>\nEpoch 4: 100%|##########| 529/529 [03:06&lt;00:00,  2.83it/s, loss=1.87, v_num=0_0, val_loss=2.070, val_auc=0.548, <strong>val_map=0.195,</strong> train_auc=0.543, train_map=0.188]<br>\nEpoch 5: 100%|##########| 529/529 [03:06&lt;00:00,  2.84it/s, loss=1.79, v_num=0_0, val_loss=1.860, val_auc=0.600, <strong>val_map=0.232</strong>, train_auc=0.575, train_map=0.205]</p>\n<p><strong>tf_efficientnet_b5</strong></p>\n<p>Epoch 2: 100%|#########9| 528/529 [03:39&lt;00:00,  2.40it/s, lo.,** val_map=0.335**, train_auc=0.744, train_map=0.311]<br>\nEpoch 3: 100%|#########9| 528/529 [03:39&lt;00:00,  2.41it/s, , *<em>val_map=0.341</em>*, train_auc=0.777, train_map=0.341]<br>\nEpoch 4: 100%|##########| 529/529 [03:41&lt;00:00,  2.39it/s,  *<em>val_map=0.340</em>*, train_auc=0.811, train_map=0.368]</p>\n<p><strong>efficientnet_b5</strong></p>\n<p>Epoch 2: 100%|#########9| 528/529 [03:38&lt;00:00,  2.42it/s,  <strong>val_map=0.198</strong>, train_auc=0.525, train_map=0.178]<br>\nEpoch 3: 100%|#########9| 528/529 [03:42&lt;00:00,  2.37it/s,  <strong>val_map=0.211</strong>, train_auc=0.536, train_map=0.184]<br>\nEpoch 4: 100%|#########9| 528/529 [03:44&lt;00:00,  2.35it/s,<strong>val_map=0.217</strong>, train_auc=0.555, train_map=0.196]<br>\nEpoch 5: 100%|#########9| 528/529 [03:44&lt;00:00,  2.36it/s,  <strong>val_map=0.217</strong>, train_auc=0.571, train_map=0.205]</p>\n<p>All those experiments were performed with the same parameters, changing only the model. <br>\n<strong>I create a model like that:</strong></p>\n<p>model_params = {<br>\n            \"model_name\": self.model_config.model_arch,<br>\n            \"pretrained\": self.model_config.pretrained,<br>\n            \"num_classes\": 0<br>\n            if self.model_config.custom_classifier<br>\n            else self.model_config.num_classes,<br>\n            \"global_pool\": None<br>\n            if self.model_config.custom_classifier<br>\n            else self.model_config.global_pool,<br>\n            \"drop_rate\": None<br>\n            if self.model_config.custom_classifier<br>\n            else self.model_config.drop_rate,<br>\n            \"drop_path_rate\": self.model_config.drop_path_rate,<br>\n        }</p>\n<p>self.m = timm.create_model(**model_params)</p>\n<p>does anyone else go through the same thing? any tips? </p>\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "1394583",
      "postDate": "07/20/2021 12:54:24",
      "content": "<p>Hi folks,</p>\n<p>Since the beginning of this competition, I've been using Pytorch/TIMM but using TF models because for me the pytorch version takes too long to converge and leads to worse results.</p>\n<p>Here is some logs:</p>\n<p><strong>tf_efficientnetv2_m</strong></p>\n<p>Epoch 2: 100%|##########| 529/529 [03:09&lt;00:00,  2.80it/s, loss=1.66, v_num=0_0, val_loss=1.670, val_auc=0.768, <strong>val_map=0.327</strong>, train_auc=0.740, train_map=0.307]<br>\nEpoch 3: 100%|#########9| 528/529 [03:04&lt;00:00,  2.86it/s,., val_map=0.340, train_auc=0.766, train_map=0.330]<br>\nEpoch 4: 100%|#########9| 528/529 [03:13&lt;00:00,  2.73it/s, 1.1895e-04. <strong>val_map=0.348</strong>, train_auc=0.794, train_map=0.350]<br>\nEpoch 5: 100%|##########| 529/529 [03:17&lt;00:00,  2.68it/s, loss=1.48, v_num=0_0, val_loss=1.610, val_auc=0.795, <strong>val_map=0.352</strong>, train_auc=0.812, train_map=0.368]</p>\n<p><strong>efficientnetv2_m</strong></p>\n<p>Epoch 2: 100%|#########9| 528/529 [03:06&lt;00:00,  2.83it/s, loAdjusting learning rate of group 0 to 1.1988e-04., val_map=0.192, train_auc=0.510, train_map=0.170][<br>\nEpoch 3: 100%|##########| 529/529 [03:06&lt;00:00,  2.84it/s, loss=1.92, v_num=0_0, val_loss=1.950, val_auc=0.544, <strong>val_map=0.188</strong>, train_auc=0.522, train_map=0.179]<br>\nEpoch 4: 100%|##########| 529/529 [03:06&lt;00:00,  2.83it/s, loss=1.87, v_num=0_0, val_loss=2.070, val_auc=0.548, <strong>val_map=0.195,</strong> train_auc=0.543, train_map=0.188]<br>\nEpoch 5: 100%|##########| 529/529 [03:06&lt;00:00,  2.84it/s, loss=1.79, v_num=0_0, val_loss=1.860, val_auc=0.600, <strong>val_map=0.232</strong>, train_auc=0.575, train_map=0.205]</p>\n<p><strong>tf_efficientnet_b5</strong></p>\n<p>Epoch 2: 100%|#########9| 528/529 [03:39&lt;00:00,  2.40it/s, lo.,** val_map=0.335**, train_auc=0.744, train_map=0.311]<br>\nEpoch 3: 100%|#########9| 528/529 [03:39&lt;00:00,  2.41it/s, , *<em>val_map=0.341</em>*, train_auc=0.777, train_map=0.341]<br>\nEpoch 4: 100%|##########| 529/529 [03:41&lt;00:00,  2.39it/s,  *<em>val_map=0.340</em>*, train_auc=0.811, train_map=0.368]</p>\n<p><strong>efficientnet_b5</strong></p>\n<p>Epoch 2: 100%|#########9| 528/529 [03:38&lt;00:00,  2.42it/s,  <strong>val_map=0.198</strong>, train_auc=0.525, train_map=0.178]<br>\nEpoch 3: 100%|#########9| 528/529 [03:42&lt;00:00,  2.37it/s,  <strong>val_map=0.211</strong>, train_auc=0.536, train_map=0.184]<br>\nEpoch 4: 100%|#########9| 528/529 [03:44&lt;00:00,  2.35it/s,<strong>val_map=0.217</strong>, train_auc=0.555, train_map=0.196]<br>\nEpoch 5: 100%|#########9| 528/529 [03:44&lt;00:00,  2.36it/s,  <strong>val_map=0.217</strong>, train_auc=0.571, train_map=0.205]</p>\n<p>All those experiments were performed with the same parameters, changing only the model. <br>\n<strong>I create a model like that:</strong></p>\n<p>model_params = {<br>\n            \"model_name\": self.model_config.model_arch,<br>\n            \"pretrained\": self.model_config.pretrained,<br>\n            \"num_classes\": 0<br>\n            if self.model_config.custom_classifier<br>\n            else self.model_config.num_classes,<br>\n            \"global_pool\": None<br>\n            if self.model_config.custom_classifier<br>\n            else self.model_config.global_pool,<br>\n            \"drop_rate\": None<br>\n            if self.model_config.custom_classifier<br>\n            else self.model_config.drop_rate,<br>\n            \"drop_path_rate\": self.model_config.drop_path_rate,<br>\n        }</p>\n<p>self.m = timm.create_model(**model_params)</p>\n<p>does anyone else go through the same thing? any tips? </p>\n<p>Thanks!</p>",
      "rawMarkdown": "Hi folks,\n\nSince the beginning of this competition, I've been using Pytorch/TIMM but using TF models because for me the pytorch version takes too long to converge and leads to worse results.\n\nHere is some logs:\n\n**tf_efficientnetv2_m**\n\nEpoch 2: 100%|##########| 529/529 [03:09<00:00,  2.80it/s, loss=1.66, v_num=0_0, val_loss=1.670, val_auc=0.768, **val_map=0.327**, train_auc=0.740, train_map=0.307]\nEpoch 3: 100%|#########9| 528/529 [03:04<00:00,  2.86it/s,., val_map=0.340, train_auc=0.766, train_map=0.330]\nEpoch 4: 100%|#########9| 528/529 [03:13<00:00,  2.73it/s, 1.1895e-04. **val_map=0.348**, train_auc=0.794, train_map=0.350]\nEpoch 5: 100%|##########| 529/529 [03:17<00:00,  2.68it/s, loss=1.48, v_num=0_0, val_loss=1.610, val_auc=0.795, **val_map=0.352**, train_auc=0.812, train_map=0.368]\n\n**efficientnetv2_m**\n\nEpoch 2: 100%|#########9| 528/529 [03:06<00:00,  2.83it/s, loAdjusting learning rate of group 0 to 1.1988e-04., val_map=0.192, train_auc=0.510, train_map=0.170][\nEpoch 3: 100%|##########| 529/529 [03:06<00:00,  2.84it/s, loss=1.92, v_num=0_0, val_loss=1.950, val_auc=0.544, **val_map=0.188**, train_auc=0.522, train_map=0.179]\nEpoch 4: 100%|##########| 529/529 [03:06<00:00,  2.83it/s, loss=1.87, v_num=0_0, val_loss=2.070, val_auc=0.548, **val_map=0.195,** train_auc=0.543, train_map=0.188]\nEpoch 5: 100%|##########| 529/529 [03:06<00:00,  2.84it/s, loss=1.79, v_num=0_0, val_loss=1.860, val_auc=0.600, **val_map=0.232**, train_auc=0.575, train_map=0.205]\n\n\n**tf_efficientnet_b5**\n\nEpoch 2: 100%|#########9| 528/529 [03:39<00:00,  2.40it/s, lo.,** val_map=0.335**, train_auc=0.744, train_map=0.311]\nEpoch 3: 100%|#########9| 528/529 [03:39<00:00,  2.41it/s, , **val_map=0.341**, train_auc=0.777, train_map=0.341]\nEpoch 4: 100%|##########| 529/529 [03:41<00:00,  2.39it/s,  **val_map=0.340**, train_auc=0.811, train_map=0.368]\n\n\n**efficientnet_b5**\n\nEpoch 2: 100%|#########9| 528/529 [03:38<00:00,  2.42it/s,  **val_map=0.198**, train_auc=0.525, train_map=0.178]\nEpoch 3: 100%|#########9| 528/529 [03:42<00:00,  2.37it/s,  **val_map=0.211**, train_auc=0.536, train_map=0.184]\nEpoch 4: 100%|#########9| 528/529 [03:44<00:00,  2.35it/s,**val_map=0.217**, train_auc=0.555, train_map=0.196]\nEpoch 5: 100%|#########9| 528/529 [03:44<00:00,  2.36it/s,  **val_map=0.217**, train_auc=0.571, train_map=0.205]\n\n\nAll those experiments were performed with the same parameters, changing only the model. \n**I create a model like that:**\n\nmodel_params = {\n            \"model_name\": self.model_config.model_arch,\n            \"pretrained\": self.model_config.pretrained,\n            \"num_classes\": 0\n            if self.model_config.custom_classifier\n            else self.model_config.num_classes,\n            \"global_pool\": None\n            if self.model_config.custom_classifier\n            else self.model_config.global_pool,\n            \"drop_rate\": None\n            if self.model_config.custom_classifier\n            else self.model_config.drop_rate,\n            \"drop_path_rate\": self.model_config.drop_path_rate,\n        }\n\nself.m = timm.create_model(**model_params)\n\ndoes anyone else go through the same thing? any tips? \n\nThanks!",
      "votes": null
    },
    {
      "id": "1395612",
      "postDate": "07/21/2021 11:35:31",
      "content": "<p>tf_efficientnetv2_m has pretrained weights, but efficientnetv2_m Not.</p>",
      "rawMarkdown": "tf_efficientnetv2_m has pretrained weights, but efficientnetv2_m Not.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1395612,
      "author_name": "blueboy97",
      "author_url": "",
      "post_date": "07/21/2021 11:35:31",
      "content": "<p>tf_efficientnetv2_m has pretrained weights, but efficientnetv2_m Not.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1394583": "Hi folks,\n\nSince the beginning of this competition, I've been using Pytorch/TIMM but using TF models because for me the pytorch version takes too long to converge and leads to worse results.\n\nHere is some logs:\n\n**tf_efficientnetv2_m**\n\nEpoch 2: 100%|##########| 529/529 [03:09<00:00,  2.80it/s, loss=1.66, v_num=0_0, val_loss=1.670, val_auc=0.768, **val_map=0.327**, train_auc=0.740, train_map=0.307]\nEpoch 3: 100%|#########9| 528/529 [03:04<00:00,  2.86it/s,., val_map=0.340, train_auc=0.766, train_map=0.330]\nEpoch 4: 100%|#########9| 528/529 [03:13<00:00,  2.73it/s, 1.1895e-04. **val_map=0.348**, train_auc=0.794, train_map=0.350]\nEpoch 5: 100%|##########| 529/529 [03:17<00:00,  2.68it/s, loss=1.48, v_num=0_0, val_loss=1.610, val_auc=0.795, **val_map=0.352**, train_auc=0.812, train_map=0.368]\n\n**efficientnetv2_m**\n\nEpoch 2: 100%|#########9| 528/529 [03:06<00:00,  2.83it/s, loAdjusting learning rate of group 0 to 1.1988e-04., val_map=0.192, train_auc=0.510, train_map=0.170][\nEpoch 3: 100%|##########| 529/529 [03:06<00:00,  2.84it/s, loss=1.92, v_num=0_0, val_loss=1.950, val_auc=0.544, **val_map=0.188**, train_auc=0.522, train_map=0.179]\nEpoch 4: 100%|##########| 529/529 [03:06<00:00,  2.83it/s, loss=1.87, v_num=0_0, val_loss=2.070, val_auc=0.548, **val_map=0.195,** train_auc=0.543, train_map=0.188]\nEpoch 5: 100%|##########| 529/529 [03:06<00:00,  2.84it/s, loss=1.79, v_num=0_0, val_loss=1.860, val_auc=0.600, **val_map=0.232**, train_auc=0.575, train_map=0.205]\n\n\n**tf_efficientnet_b5**\n\nEpoch 2: 100%|#########9| 528/529 [03:39<00:00,  2.40it/s, lo.,** val_map=0.335**, train_auc=0.744, train_map=0.311]\nEpoch 3: 100%|#########9| 528/529 [03:39<00:00,  2.41it/s, , **val_map=0.341**, train_auc=0.777, train_map=0.341]\nEpoch 4: 100%|##########| 529/529 [03:41<00:00,  2.39it/s,  **val_map=0.340**, train_auc=0.811, train_map=0.368]\n\n\n**efficientnet_b5**\n\nEpoch 2: 100%|#########9| 528/529 [03:38<00:00,  2.42it/s,  **val_map=0.198**, train_auc=0.525, train_map=0.178]\nEpoch 3: 100%|#########9| 528/529 [03:42<00:00,  2.37it/s,  **val_map=0.211**, train_auc=0.536, train_map=0.184]\nEpoch 4: 100%|#########9| 528/529 [03:44<00:00,  2.35it/s,**val_map=0.217**, train_auc=0.555, train_map=0.196]\nEpoch 5: 100%|#########9| 528/529 [03:44<00:00,  2.36it/s,  **val_map=0.217**, train_auc=0.571, train_map=0.205]\n\n\nAll those experiments were performed with the same parameters, changing only the model. \n**I create a model like that:**\n\nmodel_params = {\n            \"model_name\": self.model_config.model_arch,\n            \"pretrained\": self.model_config.pretrained,\n            \"num_classes\": 0\n            if self.model_config.custom_classifier\n            else self.model_config.num_classes,\n            \"global_pool\": None\n            if self.model_config.custom_classifier\n            else self.model_config.global_pool,\n            \"drop_rate\": None\n            if self.model_config.custom_classifier\n            else self.model_config.drop_rate,\n            \"drop_path_rate\": self.model_config.drop_path_rate,\n        }\n\nself.m = timm.create_model(**model_params)\n\ndoes anyone else go through the same thing? any tips? \n\nThanks!",
    "1395612": "tf_efficientnetv2_m has pretrained weights, but efficientnetv2_m Not."
  },
  "source": "meta"
}