This is the most painful yet I've experienced. When I tried to train YOLOv5, my RAM (aka memory) has filled up rapidly, causing the error message saying that \"Your notebook tried to allocate more memory than is available. It has restarted.\" Reducing the batch size works, but I want to keep the batch size to 18 so that my model will detect the COTS accurately. How do I have to keep my memory low while training YOLOv5 with batch size being set to 18 rather than reducing the batch size? (gc didn't work)
", "messages": [ { "id": "1639418", "postDate": "01/05/2022 16:07:11", "content": "This is the most painful yet I've experienced. When I tried to train YOLOv5, my RAM (aka memory) has filled up rapidly, causing the error message saying that \"Your notebook tried to allocate more memory than is available. It has restarted.\" Reducing the batch size works, but I want to keep the batch size to 18 so that my model will detect the COTS accurately. How do I have to keep my memory low while training YOLOv5 with batch size being set to 18 rather than reducing the batch size? (gc didn't work)
", "rawMarkdown": "This is the most painful yet I've experienced. When I tried to train YOLOv5, my RAM (aka memory) has filled up rapidly, causing the error message saying that \"Your notebook tried to allocate more memory than is available. It has restarted.\" Reducing the batch size works, but I want to keep the batch size to 18 so that my model will detect the COTS accurately. How do I have to keep my memory low while training YOLOv5 with batch size being set to 18 rather than reducing the batch size? (gc didn't work)", "votes": null }, { "id": "1639428", "postDate": "01/05/2022 16:14:16", "content": "I think you're using --cache it caches the images into memory. If you are training using huge image size to train then memory will get exhausted hence you'll get OOM
There is no --cache in my code.
in that case, I think you are running your code on CPU instead of GPU. In CPU you'll get the same error. Please check if have turned on GPU?
", "rawMarkdown": "in that case, I think you are running your code on **CPU** instead of **GPU**. In CPU you'll get the same error. Please check if have turned on **GPU**?", "votes": null }, { "id": "1639451", "postDate": "01/05/2022 16:33:57", "content": "Uh, but how?
", "rawMarkdown": "Uh, but how?", "votes": null }, { "id": "1639466", "postDate": "01/05/2022 16:46:20", "content": "check the CPU or GPU drop-down button in the bottom right. You can also check it using !nvidia-smi command in notebook.
I have had this too in my projects. Typically you need to go through your code and try to optimize it. There are many things that makes RAM go up like multiprocessing, many objects creation, etc.
", "rawMarkdown": "I have had this too in my projects. Typically you need to go through your code and try to optimize it. There are many things that makes RAM go up like multiprocessing, many objects creation, etc.", "votes": null } ], "comments": [ { "id": 1639428, "author_name": "awsaf49", "author_url": "", "post_date": "01/05/2022 16:14:16", "content": "I think you're using --cache it caches the images into memory. If you are training using huge image size to train then memory will get exhausted hence you'll get OOM
There is no --cache in my code.
in that case, I think you are running your code on CPU instead of GPU. In CPU you'll get the same error. Please check if have turned on GPU?
", "votes": null, "replies": [] }, { "id": 1639451, "author_name": "dinowun", "author_url": "", "post_date": "01/05/2022 16:33:57", "content": "Uh, but how?
", "votes": null, "replies": [] }, { "id": 1639466, "author_name": "awsaf49", "author_url": "", "post_date": "01/05/2022 16:46:20", "content": "check the CPU or GPU drop-down button in the bottom right. You can also check it using !nvidia-smi command in notebook.
I have had this too in my projects. Typically you need to go through your code and try to optimize it. There are many things that makes RAM go up like multiprocessing, many objects creation, etc.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1639418": "This is the most painful yet I've experienced. When I tried to train YOLOv5, my RAM (aka memory) has filled up rapidly, causing the error message saying that \"Your notebook tried to allocate more memory than is available. It has restarted.\" Reducing the batch size works, but I want to keep the batch size to 18 so that my model will detect the COTS accurately. How do I have to keep my memory low while training YOLOv5 with batch size being set to 18 rather than reducing the batch size? (gc didn't work)", "1639428": "I think you're using `--cache` it caches the images into memory. If you are training using huge image size to train then memory will get exhausted hence you'll get **OOM**", "1639443": "There is no `--cache` in my code.", "1639447": "in that case, I think you are running your code on **CPU** instead of **GPU**. In CPU you'll get the same error. Please check if have turned on **GPU**?", "1639451": "Uh, but how?", "1639466": "check the **CPU** or **GPU** drop-down button in the bottom right. You can also check it using `!nvidia-smi` command in notebook.", "1639931": "I have had this too in my projects. Typically you need to go through your code and try to optimize it. There are many things that makes RAM go up like multiprocessing, many objects creation, etc." }, "source": "meta" }