{
  "id": 285609,
  "title": "Very slow evaluation",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/285609",
  "author_name": "",
  "post_date": "2021-11-05T12:52:43.867336900Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Does anyone notice that a full-fledged evaluation loop takes too much time? My bottleneck is calculating IoU hit thresholds for every predicted mask and ground-truth mask. That loop's execution time grows quadratically. It takes less than a second to predict and calculate AP for cort images, but shsy5y and astro images with more than 200 masks take 20-30 seconds. I haven't figure out a faster way to do this. Evaluating a single fold takes almost half hour to finish and there 4 more of them. How much does it take for you to do full evaluation?</p>",
  "messages": [
    {
      "id": "1572136",
      "postDate": "11/05/2021 12:52:43",
      "content": "<p>Does anyone notice that a full-fledged evaluation loop takes too much time? My bottleneck is calculating IoU hit thresholds for every predicted mask and ground-truth mask. That loop's execution time grows quadratically. It takes less than a second to predict and calculate AP for cort images, but shsy5y and astro images with more than 200 masks take 20-30 seconds. I haven't figure out a faster way to do this. Evaluating a single fold takes almost half hour to finish and there 4 more of them. How much does it take for you to do full evaluation?</p>",
      "rawMarkdown": "Does anyone notice that a full-fledged evaluation loop takes too much time? My bottleneck is calculating IoU hit thresholds for every predicted mask and ground-truth mask. That loop's execution time grows quadratically. It takes less than a second to predict and calculate AP for cort images, but shsy5y and astro images with more than 200 masks take 20-30 seconds. I haven't figure out a faster way to do this. Evaluating a single fold takes almost half hour to finish and there 4 more of them. How much does it take for you to do full evaluation?",
      "votes": null
    },
    {
      "id": "1572192",
      "postDate": "11/05/2021 13:48:51",
      "content": "<p>In my case it's 37 seconds for a single fold. Out of which 28 seconds is running images through the model - so less than 10s for scoring itself.</p>",
      "rawMarkdown": "In my case it's 37 seconds for a single fold. Out of which 28 seconds is running images through the model - so less than 10s for scoring itself.",
      "votes": null
    },
    {
      "id": "1577401",
      "postDate": "11/10/2021 04:22:03",
      "content": "<p>Since the evaluation involve each pixels,the computation amount just go high.Maybe GPU will not be only for the training.</p>",
      "rawMarkdown": "Since the evaluation involve each pixels,the computation amount just go high.Maybe GPU will not be only for the training.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1572192,
      "author_name": "slawekbiel",
      "author_url": "",
      "post_date": "11/05/2021 13:48:51",
      "content": "<p>In my case it's 37 seconds for a single fold. Out of which 28 seconds is running images through the model - so less than 10s for scoring itself.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1577401,
      "author_name": "yichenwang1988",
      "author_url": "",
      "post_date": "11/10/2021 04:22:03",
      "content": "<p>Since the evaluation involve each pixels,the computation amount just go high.Maybe GPU will not be only for the training.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1572136": "Does anyone notice that a full-fledged evaluation loop takes too much time? My bottleneck is calculating IoU hit thresholds for every predicted mask and ground-truth mask. That loop's execution time grows quadratically. It takes less than a second to predict and calculate AP for cort images, but shsy5y and astro images with more than 200 masks take 20-30 seconds. I haven't figure out a faster way to do this. Evaluating a single fold takes almost half hour to finish and there 4 more of them. How much does it take for you to do full evaluation?",
    "1572192": "In my case it's 37 seconds for a single fold. Out of which 28 seconds is running images through the model - so less than 10s for scoring itself.",
    "1577401": "Since the evaluation involve each pixels,the computation amount just go high.Maybe GPU will not be only for the training."
  },
  "source": "meta"
}