![\"\"](\"https://i.postimg.cc/x8dcWqch/sartorius.jpg\")
Thank you @slawekbiel for your excellent notebooks. They helped me learn a lot about Detectron2 and its working.
\nI found a way to save the best checkpoints from the Detectron2, This might help out someone in need. So sharing here :)
Import some extra classes
\nfrom detectron2.engine import BestCheckpointer\nfrom detectron2.checkpoint import DetectionCheckpointer\nModify the trainer class to add a BestCheckpointer hook
\nclass Trainer(DefaultTrainer):\n @classmethod\n def build_evaluator(cls, cfg, dataset_name, output_folder=None):\n return MAPIOUEvaluator(dataset_name)\n\n def build_hooks(self):\n\n # copy of cfg\n cfg = self.cfg.clone()\n\n # build the original model hooks\n hooks = super().build_hooks()\n\n # add the best checkpointer hook\n hooks.insert(-1, BestCheckpointer(cfg.TEST.EVAL_PERIOD, \n DetectionCheckpointer(self.model, cfg.OUTPUT_DIR),\n \"MaP IoU\",\n \"max\",\n ))\n return hooks\nThis will save a model_best.pth in the OUTPUT directory. These weights will save the weights with Maximum MaP IoU and you can directly use them for inference.
Thank you @slawekbiel for your excellent notebooks. They helped me learn a lot about Detectron2 and its working.
\nI found a way to save the best checkpoints from the Detectron2, This might help out someone in need. So sharing here :)
Import some extra classes
\nfrom detectron2.engine import BestCheckpointer\nfrom detectron2.checkpoint import DetectionCheckpointer\nModify the trainer class to add a BestCheckpointer hook
\nclass Trainer(DefaultTrainer):\n @classmethod\n def build_evaluator(cls, cfg, dataset_name, output_folder=None):\n return MAPIOUEvaluator(dataset_name)\n\n def build_hooks(self):\n\n # copy of cfg\n cfg = self.cfg.clone()\n\n # build the original model hooks\n hooks = super().build_hooks()\n\n # add the best checkpointer hook\n hooks.insert(-1, BestCheckpointer(cfg.TEST.EVAL_PERIOD, \n DetectionCheckpointer(self.model, cfg.OUTPUT_DIR),\n \"MaP IoU\",\n \"max\",\n ))\n return hooks\nThis will save a model_best.pth in the OUTPUT directory. These weights will save the weights with Maximum MaP IoU and you can directly use them for inference.
And do this to skip saving all weights, which is the default setting (In case you're still using the trainer's DefaultCheckpointer)
\ncfg.SOLVER.CHECKPOINT_PERIOD = cfg.SOLVER.MAX_ITER+1
This one setting can save lots and lots of space.
", "rawMarkdown": "And do this to skip saving all weights, which is the default setting (In case you're still using the trainer's DefaultCheckpointer)\n`cfg.SOLVER.CHECKPOINT_PERIOD = cfg.SOLVER.MAX_ITER+1`\n\nThis one setting can save lots and lots of space.", "votes": 3 }, { "id": 1588791, "postDate": "2021-11-19T16:26:55.763Z", "content": "Hi @yerramvarun thanks for that tip! +1
\n\n\nEDIT: sort it out - works fine!
", "rawMarkdown": "Hi @yerramvarun thanks for that tip! +1\n\n~~Do you have any idea why in my case does not save the `best_model.pth`? \nI can only see `model_final.pth` (and the intermediate chekp if I don't suppress them) ~~\n\n~~anyone with same issue maybe ? ~~\n\nEDIT: sort it out - works fine! ", "votes": 1 }, { "id": 1588298, "postDate": "2021-11-19T10:33:36.910Z", "content": "You are sooooooooooooooooo good!!!!!
", "rawMarkdown": "You are sooooooooooooooooo good!!!!!", "votes": 1 }, { "id": 1587823, "postDate": "2021-11-19T00:46:58.860Z", "content": "Great work. It helps me a lot!
", "rawMarkdown": "Great work. It helps me a lot!", "votes": 1 }, { "id": 1587757, "postDate": "2021-11-18T23:39:52.197Z", "content": "Hi @yerramvarun
\nDo you know why the size of the best model is different from others?
\n
It might be because Bestcheckpointer doesn't save the optimizer states and param groups.
", "rawMarkdown": "It might be because Bestcheckpointer doesn't save the optimizer states and param groups.", "votes": 6 } ] }, { "id": 1580766, "postDate": "2021-11-13T05:21:41.140Z", "content": "Much needed @yerramvarun . Thanks for it.
", "rawMarkdown": "Much needed @yerramvarun . Thanks for it.", "votes": 1 }, { "id": 1579370, "postDate": "2021-11-11T20:15:26.613Z", "content": "Thank you @yerramvarun for sharing, I needed it.
\nto resume training I am just changing this part of the code ( it seems like working)
\ncfg.MODEL.WEIGHTS = '../input/my-Sartorius-v1/output/model_final.pth'\ntrainer.resume_or_load(resume=True)\nYou are welcome ;)
\nThanks for this! @faisalalsrheed
you might need to use this as well for better resume training
\ncfg.SOLVER.WARMUP_ITERS = 0
May i ask a question? Do u know how to apply mutil-gpu training on detectrion2?
", "rawMarkdown": "May i ask a question? Do u know how to apply mutil-gpu training on detectrion2?", "replies": [ { "id": 1587994, "postDate": "2021-11-19T05:43:08.770Z", "content": "@manyuli Hey does multi-gpu argument in train_net.py help?
", "rawMarkdown": "@manyuli Hey does multi-gpu argument in [train_net.py](https://github.com/facebookresearch/detectron2/blob/main/tools/train_net.py) help?" }, { "id": 1587996, "postDate": "2021-11-19T05:44:51.607Z", "content": "I use launch method to solve that problem
Great tip, thanks!!
", "rawMarkdown": "Great tip, thanks!!" } ], "comments": [ { "id": 1581013, "author_name": "Mighty Rains", "author_url": "", "post_date": "2021-11-13T10:19:56.297000", "content": "And do this to skip saving all weights, which is the default setting (In case you're still using the trainer's DefaultCheckpointer)
\ncfg.SOLVER.CHECKPOINT_PERIOD = cfg.SOLVER.MAX_ITER+1
This one setting can save lots and lots of space.
", "votes": 3, "replies": [] }, { "id": 1588791, "author_name": "Ioannis M", "author_url": "", "post_date": "2021-11-19T16:26:55.763000", "content": "Hi @yerramvarun thanks for that tip! +1
\n\n\nEDIT: sort it out - works fine!
", "votes": 1, "replies": [] }, { "id": 1588298, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-19T10:33:36.910000", "content": "You are sooooooooooooooooo good!!!!!
", "votes": 1, "replies": [] }, { "id": 1587823, "author_name": "Roc", "author_url": "", "post_date": "2021-11-19T00:46:58.860000", "content": "Great work. It helps me a lot!
", "votes": 1, "replies": [] }, { "id": 1587757, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-18T23:39:52.197000", "content": "Hi @yerramvarun
\nDo you know why the size of the best model is different from others?
\n
It might be because Bestcheckpointer doesn't save the optimizer states and param groups.
", "votes": 6, "replies": [] } ] }, { "id": 1580766, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2021-11-13T05:21:41.140000", "content": "Much needed @yerramvarun . Thanks for it.
", "votes": 1, "replies": [] }, { "id": 1579370, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-11T20:15:26.613000", "content": "Thank you @yerramvarun for sharing, I needed it.
\nto resume training I am just changing this part of the code ( it seems like working)
\ncfg.MODEL.WEIGHTS = '../input/my-Sartorius-v1/output/model_final.pth'\ntrainer.resume_or_load(resume=True)\nYou are welcome ;)
\nThanks for this! @faisalalsrheed
you might need to use this as well for better resume training
\ncfg.SOLVER.WARMUP_ITERS = 0
May i ask a question? Do u know how to apply mutil-gpu training on detectrion2?
", "votes": 0, "replies": [ { "id": 1587994, "author_name": "Yerram Varun", "author_url": "", "post_date": "2021-11-19T05:43:08.770000", "content": "@manyuli Hey does multi-gpu argument in train_net.py help?
", "votes": 0, "replies": [] }, { "id": 1587996, "author_name": "Manyu Li", "author_url": "", "post_date": "2021-11-19T05:44:51.607000", "content": "I use launch method to solve that problem
Great tip, thanks!!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1579329": "Thank you @slawekbiel for your excellent notebooks. They helped me learn a lot about Detectron2 and its working.\nI found a way to save the best checkpoints from the Detectron2, This might help out someone in need. So sharing here :)\n\nImport some extra classes\n```\nfrom detectron2.engine import BestCheckpointer\nfrom detectron2.checkpoint import DetectionCheckpointer\n```\n\nModify the trainer class to add a BestCheckpointer hook\n\n```\nclass Trainer(DefaultTrainer):\n @classmethod\n def build_evaluator(cls, cfg, dataset_name, output_folder=None):\n return MAPIOUEvaluator(dataset_name)\n \n def build_hooks(self):\n \n # copy of cfg\n cfg = self.cfg.clone()\n \n # build the original model hooks\n hooks = super().build_hooks()\n \n # add the best checkpointer hook\n hooks.insert(-1, BestCheckpointer(cfg.TEST.EVAL_PERIOD, \n DetectionCheckpointer(self.model, cfg.OUTPUT_DIR),\n \"MaP IoU\",\n \"max\",\n ))\n return hooks\n```\n\nThis will save a `model_best.pth` in the OUTPUT directory. These weights will save the weights with Maximum MaP IoU and you can directly use them for inference.\n", "1581013": "And do this to skip saving all weights, which is the default setting (In case you're still using the trainer's DefaultCheckpointer)\n`cfg.SOLVER.CHECKPOINT_PERIOD = cfg.SOLVER.MAX_ITER+1`\n\nThis one setting can save lots and lots of space.", "1588791": "Hi @yerramvarun thanks for that tip! +1\n\n~~Do you have any idea why in my case does not save the `best_model.pth`? \nI can only see `model_final.pth` (and the intermediate chekp if I don't suppress them) ~~\n\n~~anyone with same issue maybe ? ~~\n\nEDIT: sort it out - works fine! ", "1588298": "You are sooooooooooooooooo good!!!!!", "1587823": "Great work. It helps me a lot!", "1587757": "Hi @yerramvarun \n\nDo you know why the size of the best model is different from others?\n\n", "1580766": "Much needed @yerramvarun . Thanks for it.", "1579370": "Thank you @yerramvarun for sharing, I needed it. \n\nto resume training I am just changing this part of the code ( it seems like working)\n```\ncfg.MODEL.WEIGHTS = '../input/my-Sartorius-v1/output/model_final.pth'\ntrainer.resume_or_load(resume=True)\n```\n\n", "1582510": "May i ask a question? Do u know how to apply mutil-gpu training on detectrion2?", "1594883": "", "1594879": "", "1626799": "Great tip, thanks!!" } }