![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1821809%2F215628a5d628946008cbfd3a4768c2d9%2F__results___26_0.png?generation=1595605504331009&alt=media\")
I see many efficient nets getting ~0.95 auc on training. My implementation can only reach 0.85 LB score. And during training it never climbs over 0.65. Validation loss stagnates and the model early stops.
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1821809%2Fb243b5396d63f825273cffe570d598a7%2F__results___26_1.png?generation=1595605593703559&alt=media\")
I have no idea what I'm doing wrong I have an inception V3 version which behaves fine.
\n\nKernel is this if you want to check it out\nhttps://www.kaggle.com/amneves/tensorflow-efficientnetbx-transfer-learning
", "messages": [ { "id": "943820", "postDate": "07/24/2020 15:49:32", "content": "I see many efficient nets getting ~0.95 auc on training. My implementation can only reach 0.85 LB score. And during training it never climbs over 0.65. Validation loss stagnates and the model early stops.
\n\n
I have no idea what I'm doing wrong I have an inception V3 version which behaves fine.
\n\nKernel is this if you want to check it out\nhttps://www.kaggle.com/amneves/tensorflow-efficientnetbx-transfer-learning
", "rawMarkdown": "I see many efficient nets getting ~0.95 auc on training. My implementation can only reach 0.85 LB score. And during training it never climbs over 0.65. Validation loss stagnates and the model early stops.\n\n\n\n\n\nI have no idea what I'm doing wrong I have an inception V3 version which behaves fine.\n\nKernel is this if you want to check it out\n[https://www.kaggle.com/amneves/tensorflow-efficientnetbx-transfer-learning](https://www.kaggle.com/amneves/tensorflow-efficientnetbx-transfer-learning)", "votes": null }, { "id": "943914", "postDate": "07/24/2020 17:05:27", "content": "lr
", "rawMarkdown": "lr", "votes": null }, { "id": "943915", "postDate": "07/24/2020 17:07:17", "content": "That's what I thought. Any tips?
", "rawMarkdown": "That's what I thought. Any tips?", "votes": null }, { "id": "943919", "postDate": "07/24/2020 17:11:19", "content": "On Epoch 6/16 , lr: 0.0205.I think lr is too large.
", "rawMarkdown": "On Epoch 6/16 , lr: 0.0205.I think lr is too large.", "votes": null }, { "id": "943958", "postDate": "07/24/2020 17:59:12", "content": "As commented, it looks like your Learning Rate is too large. You have the code:
\n\nlr_max = 0.000020 * REPLICAS * batch_size
\n\nI don't think you want to multiply by batch_size.
\n\n-Rich
", "rawMarkdown": "As commented, it looks like your Learning Rate is too large. You have the code:\n\n lr_max = 0.000020 * REPLICAS * batch_size\n\nI don't think you want to multiply by batch_size.\n\n-Rich", "votes": null }, { "id": "944526", "postDate": "07/25/2020 07:07:33", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F555ba305717d78a2d6a031f190de9e3f%2FScreenshot%20from%202020-07-25%2010-02-53.png?generation=1595660606549263&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690820%2F9724b20f1b7d0aac873252dedf379cd5%2FScreenshot%20from%202020-07-25%2010-03-05.png?generation=1595660621477968&alt=media\")
Red - train log, Blue - valid log\nAs you can see have the same problem - models starts to overfit after a few epochs.\nSo possible solutions are:\n- Use best model from metric and use ansambling to make it stable\n- Use lower LR\n- Use models that have fewer number of params but overall better accuracy (for example effnets)
", "rawMarkdown": "\n\n\nRed - train log, Blue - valid log\nAs you can see have the same problem - models starts to overfit after a few epochs.\nSo possible solutions are:\n- Use best model from metric and use ansambling to make it stable\n- Use lower LR\n- Use models that have fewer number of params but overall better accuracy (for example effnets)", "votes": null } ], "comments": [ { "id": 943914, "author_name": "chihantsai", "author_url": "", "post_date": "07/24/2020 17:05:27", "content": "lr
", "votes": null, "replies": [ { "id": 943915, "author_name": "amneves", "author_url": "", "post_date": "07/24/2020 17:07:17", "content": "That's what I thought. Any tips?
", "votes": null, "replies": [] }, { "id": 943919, "author_name": "chihantsai", "author_url": "", "post_date": "07/24/2020 17:11:19", "content": "On Epoch 6/16 , lr: 0.0205.I think lr is too large.
", "votes": null, "replies": [] } ] }, { "id": 943958, "author_name": "richardepstein", "author_url": "", "post_date": "07/24/2020 17:59:12", "content": "As commented, it looks like your Learning Rate is too large. You have the code:
\n\nlr_max = 0.000020 * REPLICAS * batch_size
\n\nI don't think you want to multiply by batch_size.
\n\n-Rich
", "votes": null, "replies": [] }, { "id": 944526, "author_name": "vladimirsydor", "author_url": "", "post_date": "07/25/2020 07:07:33", "content": "\n
Red - train log, Blue - valid log\nAs you can see have the same problem - models starts to overfit after a few epochs.\nSo possible solutions are:\n- Use best model from metric and use ansambling to make it stable\n- Use lower LR\n- Use models that have fewer number of params but overall better accuracy (for example effnets)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "943820": "I see many efficient nets getting ~0.95 auc on training. My implementation can only reach 0.85 LB score. And during training it never climbs over 0.65. Validation loss stagnates and the model early stops.\n\n\n\n\n\nI have no idea what I'm doing wrong I have an inception V3 version which behaves fine.\n\nKernel is this if you want to check it out\n[https://www.kaggle.com/amneves/tensorflow-efficientnetbx-transfer-learning](https://www.kaggle.com/amneves/tensorflow-efficientnetbx-transfer-learning)", "943914": "lr", "943915": "That's what I thought. Any tips?", "943919": "On Epoch 6/16 , lr: 0.0205.I think lr is too large.", "943958": "As commented, it looks like your Learning Rate is too large. You have the code:\n\n lr_max = 0.000020 * REPLICAS * batch_size\n\nI don't think you want to multiply by batch_size.\n\n-Rich", "944526": "\n\n\nRed - train log, Blue - valid log\nAs you can see have the same problem - models starts to overfit after a few epochs.\nSo possible solutions are:\n- Use best model from metric and use ansambling to make it stable\n- Use lower LR\n- Use models that have fewer number of params but overall better accuracy (for example effnets)" }, "source": "meta" }