{
  "id": 282788,
  "title": "Displaying validation score in detectron",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/282788",
  "author_name": "",
  "post_date": "2021-10-27T20:15:46.310833600Z",
  "votes": 32,
  "comment_count": 6,
  "views": 0,
  "content": "<p>By default detectron training only shows the training losses, if we want to see validation set metrics there needs to be some custom code added. Here is a way I got it working:</p>\n<ul>\n<li>implement <code>build_evaluator</code> method in trainer to return your custom evaluator</li>\n<li>the evaluator should implement <code>reset</code>, <code>process</code> and <code>evaluate</code></li>\n<li>since by default  the target annotations are not passed to the evaluator I build an index with annotations at the start and keep it in the evaluator.</li>\n<li>for the metric I've based my code on <a href=\"https://www.kaggle.com/theoviel/competition-metric-map-iou\" target=\"_blank\">https://www.kaggle.com/theoviel/competition-metric-map-iou</a> but substituted IoU calculation for one from pycocotools that can handle overlapping targets.</li>\n</ul>\n<p>See the whole implementation in my training notebook: <a href=\"https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training\" target=\"_blank\">https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training</a></p>\n<p>And a sample run displayed by tensorboard: <br>\n<img src=\"https://raw.githubusercontent.com/slawekslex/random/main/mapiou.png\" alt=\"\"></p>",
  "messages": [
    {
      "id": "1561882",
      "postDate": "10/27/2021 20:15:46",
      "content": "<p>By default detectron training only shows the training losses, if we want to see validation set metrics there needs to be some custom code added. Here is a way I got it working:</p>\n<ul>\n<li>implement <code>build_evaluator</code> method in trainer to return your custom evaluator</li>\n<li>the evaluator should implement <code>reset</code>, <code>process</code> and <code>evaluate</code></li>\n<li>since by default  the target annotations are not passed to the evaluator I build an index with annotations at the start and keep it in the evaluator.</li>\n<li>for the metric I've based my code on <a href=\"https://www.kaggle.com/theoviel/competition-metric-map-iou\" target=\"_blank\">https://www.kaggle.com/theoviel/competition-metric-map-iou</a> but substituted IoU calculation for one from pycocotools that can handle overlapping targets.</li>\n</ul>\n<p>See the whole implementation in my training notebook: <a href=\"https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training\" target=\"_blank\">https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training</a></p>\n<p>And a sample run displayed by tensorboard: <br>\n<img src=\"https://raw.githubusercontent.com/slawekslex/random/main/mapiou.png\" alt=\"\"></p>",
      "rawMarkdown": "By default detectron training only shows the training losses, if we want to see validation set metrics there needs to be some custom code added. Here is a way I got it working:\n- implement `build_evaluator` method in trainer to return your custom evaluator\n- the evaluator should implement `reset`, `process` and `evaluate`\n- since by default  the target annotations are not passed to the evaluator I build an index with annotations at the start and keep it in the evaluator.\n- for the metric I've based my code on https://www.kaggle.com/theoviel/competition-metric-map-iou but substituted IoU calculation for one from pycocotools that can handle overlapping targets.\n\nSee the whole implementation in my training notebook: https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training\n\nAnd a sample run displayed by tensorboard: \n![](https://raw.githubusercontent.com/slawekslex/random/main/mapiou.png)",
      "votes": null
    },
    {
      "id": "1564244",
      "postDate": "10/29/2021 02:33:09",
      "content": "<p>Hy I am trying to learn detectron api. I simply played with some parameters in your notebook, <br>\nPart2 training then part 3 inference.  Submission is showing error Notebook Threw Exception. <br>\nWhat is the reason is it because overlaping pixels… </p>",
      "rawMarkdown": "Hy I am trying to learn detectron api. I simply played with some parameters in your notebook, \nPart2 training then part 3 inference.  Submission is showing error Notebook Threw Exception. \nWhat is the reason is it because overlaping pixels...",
      "votes": null
    },
    {
      "id": "1564403",
      "postDate": "10/29/2021 07:20:38",
      "content": "<p>It’s hard to say without knowing the changes you made. I don’t think it’s overlaps as I’m removing them in the inference notebook.</p>\n<p>The first thing I’d try is to run inference on all the train images instead of test and see if it generates the submission file without errors.</p>",
      "rawMarkdown": "It’s hard to say without knowing the changes you made. I don’t think it’s overlaps as I’m removing them in the inference notebook.\n\n\nThe first thing I’d try is to run inference on all the train images instead of test and see if it generates the submission file without errors.",
      "votes": null
    },
    {
      "id": "1566131",
      "postDate": "10/31/2021 12:37:38",
      "content": "<p>Nice work! Could you teach me how to choose the best model from those iterations? It can't be trying them one by one…</p>",
      "rawMarkdown": "Nice work! Could you teach me how to choose the best model from those iterations? It can't be trying them one by one...",
      "votes": null
    },
    {
      "id": "1566173",
      "postDate": "10/31/2021 13:36:57",
      "content": "<p>There are checkpoints written to your outputs directory, you can copy a model from there picking the iteration corresponding to the best score. <br>\nThough what I usually do instead is  to setup hyperparameters in a way that I can just pick the last one.</p>",
      "rawMarkdown": "There are checkpoints written to your outputs directory, you can copy a model from there picking the iteration corresponding to the best score. \nThough what I usually do instead is  to setup hyperparameters in a way that I can just pick the last one.",
      "votes": null
    },
    {
      "id": "1574549",
      "postDate": "11/07/2021 16:36:12",
      "content": "<p>What is the difference between your custom <code>MAPIOUEvaluator</code> and <code>COCOEvaluator</code> (in detectron2) ?</p>",
      "rawMarkdown": "What is the difference between your custom `MAPIOUEvaluator` and `COCOEvaluator` (in detectron2) ?",
      "votes": null
    },
    {
      "id": "1574665",
      "postDate": "11/07/2021 19:14:08",
      "content": "<p>MAPIOUEvaluator is a class I created and it implements the metric as described in this competition. <br>\nThe <code>COCOEvaluator</code> calculates metrics from MS COCO as described here:  <a href=\"https://cocodataset.org/#detection-eval\" target=\"_blank\">https://cocodataset.org/#detection-eval</a></p>",
      "rawMarkdown": "MAPIOUEvaluator is a class I created and it implements the metric as described in this competition. \nThe `COCOEvaluator` calculates metrics from MS COCO as described here:  https://cocodataset.org/#detection-eval",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1564244,
      "author_name": "shivansh002",
      "author_url": "",
      "post_date": "10/29/2021 02:33:09",
      "content": "<p>Hy I am trying to learn detectron api. I simply played with some parameters in your notebook, <br>\nPart2 training then part 3 inference.  Submission is showing error Notebook Threw Exception. <br>\nWhat is the reason is it because overlaping pixels… </p>",
      "votes": null,
      "replies": [
        {
          "id": 1564403,
          "author_name": "slawekbiel",
          "author_url": "",
          "post_date": "10/29/2021 07:20:38",
          "content": "<p>It’s hard to say without knowing the changes you made. I don’t think it’s overlaps as I’m removing them in the inference notebook.</p>\n<p>The first thing I’d try is to run inference on all the train images instead of test and see if it generates the submission file without errors.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1566131,
      "author_name": "xhanxux",
      "author_url": "",
      "post_date": "10/31/2021 12:37:38",
      "content": "<p>Nice work! Could you teach me how to choose the best model from those iterations? It can't be trying them one by one…</p>",
      "votes": null,
      "replies": [
        {
          "id": 1566173,
          "author_name": "slawekbiel",
          "author_url": "",
          "post_date": "10/31/2021 13:36:57",
          "content": "<p>There are checkpoints written to your outputs directory, you can copy a model from there picking the iteration corresponding to the best score. <br>\nThough what I usually do instead is  to setup hyperparameters in a way that I can just pick the last one.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1574549,
      "author_name": "benihime91",
      "author_url": "",
      "post_date": "11/07/2021 16:36:12",
      "content": "<p>What is the difference between your custom <code>MAPIOUEvaluator</code> and <code>COCOEvaluator</code> (in detectron2) ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1574665,
          "author_name": "slawekbiel",
          "author_url": "",
          "post_date": "11/07/2021 19:14:08",
          "content": "<p>MAPIOUEvaluator is a class I created and it implements the metric as described in this competition. <br>\nThe <code>COCOEvaluator</code> calculates metrics from MS COCO as described here:  <a href=\"https://cocodataset.org/#detection-eval\" target=\"_blank\">https://cocodataset.org/#detection-eval</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1561882": "By default detectron training only shows the training losses, if we want to see validation set metrics there needs to be some custom code added. Here is a way I got it working:\n- implement `build_evaluator` method in trainer to return your custom evaluator\n- the evaluator should implement `reset`, `process` and `evaluate`\n- since by default  the target annotations are not passed to the evaluator I build an index with annotations at the start and keep it in the evaluator.\n- for the metric I've based my code on https://www.kaggle.com/theoviel/competition-metric-map-iou but substituted IoU calculation for one from pycocotools that can handle overlapping targets.\n\nSee the whole implementation in my training notebook: https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training\n\nAnd a sample run displayed by tensorboard: \n![](https://raw.githubusercontent.com/slawekslex/random/main/mapiou.png)",
    "1564244": "Hy I am trying to learn detectron api. I simply played with some parameters in your notebook, \nPart2 training then part 3 inference.  Submission is showing error Notebook Threw Exception. \nWhat is the reason is it because overlaping pixels...",
    "1564403": "It’s hard to say without knowing the changes you made. I don’t think it’s overlaps as I’m removing them in the inference notebook.\n\n\nThe first thing I’d try is to run inference on all the train images instead of test and see if it generates the submission file without errors.",
    "1566131": "Nice work! Could you teach me how to choose the best model from those iterations? It can't be trying them one by one...",
    "1566173": "There are checkpoints written to your outputs directory, you can copy a model from there picking the iteration corresponding to the best score. \nThough what I usually do instead is  to setup hyperparameters in a way that I can just pick the last one.",
    "1574549": "What is the difference between your custom `MAPIOUEvaluator` and `COCOEvaluator` (in detectron2) ?",
    "1574665": "MAPIOUEvaluator is a class I created and it implements the metric as described in this competition. \nThe `COCOEvaluator` calculates metrics from MS COCO as described here:  https://cocodataset.org/#detection-eval"
  },
  "source": "meta"
}