Hello,
\nSo, I am trying out efficient net-B4 architecture, I ran out of GPU quota so I thought of training the same exact notebook on my laptop (with batch size 2 as my VRAM is only 6GB). The problem is during training on kaggle GPU's I am reaching CV accuracy around 90% and it got canceled after 9Hrs, but the same notebook on my laptop is stuck at around 60%.
My specs:
\nI have tensorflow and keras installed perfectly with respective CUDA and CuDNN versions.(No issues there)
\nThanks in Advance.
\nLink to my notebook:
\nThis One
Hello,
\nSo, I am trying out efficient net-B4 architecture, I ran out of GPU quota so I thought of training the same exact notebook on my laptop (with batch size 2 as my VRAM is only 6GB). The problem is during training on kaggle GPU's I am reaching CV accuracy around 90% and it got canceled after 9Hrs, but the same notebook on my laptop is stuck at around 60%.
My specs:
\nI have tensorflow and keras installed perfectly with respective CUDA and CuDNN versions.(No issues there)
\nThanks in Advance.
\nLink to my notebook:
\nThis One
Hi @mohneesh7,
\nGenerally, CV accuracy changes due to some reason,
\nBut then also I had never seen this much of change in results due to only hardware change. Try batch_size=2 in Kaggle and check results.
", "rawMarkdown": "Hi @mohneesh7,\n\nGenerally, CV accuracy changes due to some reason,\n* One of the reasons is the initialization of the network. Due to this, we get different results on every training but it doesn't vary too much.\n* The other reason is batch_size or hyper-parameters.\n\nBut then also I had never seen this much of change in results due to only hardware change. Try batch_size=2 in Kaggle and check results.", "votes": null }, { "id": "1175292", "postDate": "01/29/2021 04:26:23", "content": "Unrelated to This but You should try out mixed precision, since you have a rtx 2060 mixed precision would work well. It will also help you increase your batch size and your training time would be halved. If your are using tensorflow refer To this https://www.tensorflow.org/guide/mixed_precision
", "rawMarkdown": "Unrelated to This but You should try out mixed precision, since you have a rtx 2060 mixed precision would work well. It will also help you increase your batch size and your training time would be halved. If your are using tensorflow refer To this https://www.tensorflow.org/guide/mixed_precision", "votes": null }, { "id": "1175400", "postDate": "01/29/2021 06:21:29", "content": "Thanks for the heads up, I am already using mixed precision on a different private notebook that I am using for the competition. It really is very helpful in training time reduction.
\nAnyways thanks for the suggestion.
I understand that the initialization of the network weights and the random seeds affect the training time and accuracy a little bit.
\nBut the difference b/w both hardware is just too much, that I am positive it isn't the network initialization.
\nanyways thanks for the suggestion. I will now wait for my kaggle quota to come back.
And If you still have problems with your quota, I suggest you try colab. It is a bit slower than kaggle but when you run out of quota Try training on colab instead of training on your Laptop.
", "rawMarkdown": "And If you still have problems with your quota, I suggest you try colab. It is a bit slower than kaggle but when you run out of quota Try training on colab instead of training on your Laptop.", "votes": null }, { "id": "1175640", "postDate": "01/29/2021 09:09:00", "content": "Can you give me a colab URL?
", "rawMarkdown": "Can you give me a colab URL?", "votes": null }, { "id": "1175674", "postDate": "01/29/2021 09:26:27", "content": "@zhangeng https://colab.research.google.com/
", "rawMarkdown": "zhangeng https://colab.research.google.com/", "votes": null } ], "comments": [ { "id": 1174083, "author_name": "vatsalmavani", "author_url": "", "post_date": "01/28/2021 08:55:28", "content": "Hi @mohneesh7,
\nGenerally, CV accuracy changes due to some reason,
\nBut then also I had never seen this much of change in results due to only hardware change. Try batch_size=2 in Kaggle and check results.
", "votes": null, "replies": [ { "id": 1175404, "author_name": "mohneesh7", "author_url": "", "post_date": "01/29/2021 06:25:11", "content": "I understand that the initialization of the network weights and the random seeds affect the training time and accuracy a little bit.
\nBut the difference b/w both hardware is just too much, that I am positive it isn't the network initialization.
\nanyways thanks for the suggestion. I will now wait for my kaggle quota to come back.
Unrelated to This but You should try out mixed precision, since you have a rtx 2060 mixed precision would work well. It will also help you increase your batch size and your training time would be halved. If your are using tensorflow refer To this https://www.tensorflow.org/guide/mixed_precision
", "votes": null, "replies": [ { "id": 1175400, "author_name": "mohneesh7", "author_url": "", "post_date": "01/29/2021 06:21:29", "content": "Thanks for the heads up, I am already using mixed precision on a different private notebook that I am using for the competition. It really is very helpful in training time reduction.
\nAnyways thanks for the suggestion.
And If you still have problems with your quota, I suggest you try colab. It is a bit slower than kaggle but when you run out of quota Try training on colab instead of training on your Laptop.
", "votes": null, "replies": [ { "id": 1175640, "author_name": "zhangeng", "author_url": "", "post_date": "01/29/2021 09:09:00", "content": "Can you give me a colab URL?
", "votes": null, "replies": [] }, { "id": 1175674, "author_name": "mithilsalunkhe", "author_url": "", "post_date": "01/29/2021 09:26:27", "content": "", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1174026": "Hello,\nSo, I am trying out efficient net-B4 architecture, I ran out of GPU quota so I thought of training the same exact notebook on my laptop (with batch size 2 as my VRAM is only 6GB). The problem is during training on kaggle GPU's I am reaching CV accuracy around 90% and it got canceled after 9Hrs, but the same notebook on my laptop is stuck at around 60%.\n\n**My specs:**\n- intel i7 9th gen\n- RTX 2060 6GB VRAM\n- RAM 16 GB\n\nI have tensorflow and keras installed perfectly with respective CUDA and CuDNN versions.(No issues there)\n\nThanks in Advance.\n\n**Link to my notebook:**\n[This One](https://www.kaggle.com/mohneesh7/cassava-leaf-disease-detection-keras)", "1174083": "Hi @mohneesh7,\n\nGenerally, CV accuracy changes due to some reason,\n* One of the reasons is the initialization of the network. Due to this, we get different results on every training but it doesn't vary too much.\n* The other reason is batch_size or hyper-parameters.\n\nBut then also I had never seen this much of change in results due to only hardware change. Try batch_size=2 in Kaggle and check results.", "1175292": "Unrelated to This but You should try out mixed precision, since you have a rtx 2060 mixed precision would work well. It will also help you increase your batch size and your training time would be halved. If your are using tensorflow refer To this https://www.tensorflow.org/guide/mixed_precision", "1175400": "Thanks for the heads up, I am already using mixed precision on a different private notebook that I am using for the competition. It really is very helpful in training time reduction.\nAnyways thanks for the suggestion.", "1175404": "I understand that the initialization of the network weights and the random seeds affect the training time and accuracy a little bit. \nBut the difference b/w both hardware is just too much, that I am positive it isn't the network initialization.\nanyways thanks for the suggestion. I will now wait for my kaggle quota to come back.", "1175590": "And If you still have problems with your quota, I suggest you try colab. It is a bit slower than kaggle but when you run out of quota Try training on colab instead of training on your Laptop.", "1175640": "Can you give me a colab URL?", "1175674": "zhangeng https://colab.research.google.com/" }, "source": "meta" }