{ "id": 292852, "title": "Evaluation Script for MMDetection", "url": "/competitions/sartorius-cell-instance-segmentation/discussion/292852", "author_name": "", "post_date": "2021-12-03T18:53:42.005616500Z", "votes": 12, "comment_count": 4, "views": 0, "content": "

I created a kernel for a baseline eval script for MMDetection here: https://www.kaggle.com/trushk/mmdetection-custom-eval-script

\n

One main pending issue is that scores seem too low based compared to the LB score. I am hoping others who have tried a similar approach can shed some light on what they have done.

\n

Credit to @theoviel for the custom functions based on the competition eval metric shared here: https://www.kaggle.com/theoviel/competition-metric-map-iou

\n

Also a shout out to my teammate @evilpsycho42 for the work done to create this notebook.

", "messages": [ { "id": "1604837", "postDate": "12/03/2021 18:53:42", "content": "

I created a kernel for a baseline eval script for MMDetection here: https://www.kaggle.com/trushk/mmdetection-custom-eval-script

\n

One main pending issue is that scores seem too low based compared to the LB score. I am hoping others who have tried a similar approach can shed some light on what they have done.

\n

Credit to @theoviel for the custom functions based on the competition eval metric shared here: https://www.kaggle.com/theoviel/competition-metric-map-iou

\n

Also a shout out to my teammate @evilpsycho42 for the work done to create this notebook.

", "rawMarkdown": "I created a kernel for a baseline eval script for MMDetection here: https://www.kaggle.com/trushk/mmdetection-custom-eval-script\n\nOne main pending issue is that scores seem too low based compared to the LB score. I am hoping others who have tried a similar approach can shed some light on what they have done. \n\nCredit to @theoviel for the custom functions based on the competition eval metric shared here: https://www.kaggle.com/theoviel/competition-metric-map-iou\n\nAlso a shout out to my teammate @evilpsycho42 for the work done to create this notebook.", "votes": null }, { "id": "1608270", "postDate": "12/06/2021 10:28:15", "content": "

thanks for sharing @trushk

", "rawMarkdown": "thanks for sharing @trushk", "votes": null }, { "id": "1620966", "postDate": "12/17/2021 09:43:44", "content": "

My CV score also too low compared to LB score,(0.206/0.307) , did you solve this problem? I tried to fix it，but failed

", "rawMarkdown": "My CV score also too low compared to LB score,(0.206/0.307) , did you solve this problem? I tried to fix it，but failed", "votes": null }, { "id": "1621082", "postDate": "12/17/2021 12:34:50", "content": "

@forgakki I used his code block with little modification and got a valid CV. Thanks! to @trushk again.

\n
@torch.no_grad()\ndef calculate_custom_score(model, val_json, root_dir):\n    df = pd.read_csv(root_dir + \"train.csv\")\n    val_ids = [i['id'] for i in val_json['images']]\n    if args.debug:\n        val_ids = val_ids[0:args.debug]\n    df_val = df[df.id.isin(val_ids)].reset_index(drop=True)\n    df_val = df_val.groupby('id').agg(list).reset_index()\n    del df\n    for col in df_val.columns[2:]:\n        df_val[col] = df_val[col].apply(\n            lambda x: np.unique(x)[0] if len(np.unique(x)) == 1 else np.unique(x)\n        )\n    gts = []\n    dts = []\n    scores = []\n   for img_id in df_val.id.tolist():\n        img = mmcv.imread(root_dir + f\"train/{img_id}.png\")\n        result = inference_detector(model, img)\n\n        # dt\n        dt = []\n        for cls, bbs in enumerate(result[0]):\n            if bbs.shape != (0, 5):\n                sgs = result[1][cls]\n                for bb, sg in zip(bbs, sgs):\n                    box = bb[:4]\n                    cnf = bb[4]\n                    if cnf >= confidence_thresholds[cls]:\n                        mask = get_mask_from_result(sg)\n                        mask = remove_overlapping_pixels(mask, dt)\n                        dt.append(mask.astype(bool))\n                        #print(mask)\n        dts.append(combine_mask(dt))\n\n        # gt\n        shape = df_val.loc[df_val.id == img_id, ['height', 'width']].values[0]\n        gt = rles_to_mask(df_val.loc[df_val.id == img_id, \"annotation\"].item(), shape).astype(np.uint16)\n        gts.append(gt)\n        score = iou_map(gts, dts, True)\n        scores.append(score)\n    return scores\n
\n
scores  = calculate_custom_score(model, val_json, root_dir)\nprint(np.mean(scores, axis=0))\n
", "rawMarkdown": "forgakki I used his code block with little modification and got a valid CV. Thanks! to @trushk again.\n\n```\n@torch.no_grad()\ndef calculate_custom_score(model, val_json, root_dir):\n df = pd.read_csv(root_dir + \"train.csv\")\n val_ids = [i['id'] for i in val_json['images']]\n if args.debug:\n val_ids = val_ids[0:args.debug]\n df_val = df[df.id.isin(val_ids)].reset_index(drop=True)\n df_val = df_val.groupby('id').agg(list).reset_index()\n del df\n for col in df_val.columns[2:]:\n df_val[col] = df_val[col].apply(\n lambda x: np.unique(x)[0] if len(np.unique(x)) == 1 else np.unique(x)\n )\n gts = []\n dts = []\n scores = []\n for img_id in df_val.id.tolist():\n img = mmcv.imread(root_dir + f\"train/{img_id}.png\")\n result = inference_detector(model, img)\n\n # dt\n dt = []\n for cls, bbs in enumerate(result[0]):\n if bbs.shape != (0, 5):\n sgs = result[1][cls]\n for bb, sg in zip(bbs, sgs):\n box = bb[:4]\n cnf = bb[4]\n if cnf >= confidence_thresholds[cls]:\n mask = get_mask_from_result(sg)\n mask = remove_overlapping_pixels(mask, dt)\n dt.append(mask.astype(bool))\n #print(mask)\n dts.append(combine_mask(dt))\n\n # gt\n shape = df_val.loc[df_val.id == img_id, ['height', 'width']].values[0]\n gt = rles_to_mask(df_val.loc[df_val.id == img_id, \"annotation\"].item(), shape).astype(np.uint16)\n gts.append(gt)\n score = iou_map(gts, dts, True)\n scores.append(score)\n return scores\n```\n\n```\nscores = calculate_custom_score(model, val_json, root_dir)\nprint(np.mean(scores, axis=0))\n```", "votes": null }, { "id": "1621306", "postDate": "12/17/2021 16:06:06", "content": "

Thanks for your replay! Thanks @trushk's code！

", "rawMarkdown": "Thanks for your replay! Thanks @trushk's code！", "votes": null } ], "comments": [ { "id": 1608270, "author_name": "kanberburak", "author_url": "", "post_date": "12/06/2021 10:28:15", "content": "

thanks for sharing @trushk

", "votes": null, "replies": [] }, { "id": 1620966, "author_name": "forgakki", "author_url": "", "post_date": "12/17/2021 09:43:44", "content": "

My CV score also too low compared to LB score,(0.206/0.307) , did you solve this problem? I tried to fix it，but failed

", "votes": null, "replies": [ { "id": 1621082, "author_name": "anandselvadurai", "author_url": "", "post_date": "12/17/2021 12:34:50", "content": "

@forgakki I used his code block with little modification and got a valid CV. Thanks! to @trushk again.

\n
@torch.no_grad()\ndef calculate_custom_score(model, val_json, root_dir):\n    df = pd.read_csv(root_dir + \"train.csv\")\n    val_ids = [i['id'] for i in val_json['images']]\n    if args.debug:\n        val_ids = val_ids[0:args.debug]\n    df_val = df[df.id.isin(val_ids)].reset_index(drop=True)\n    df_val = df_val.groupby('id').agg(list).reset_index()\n    del df\n    for col in df_val.columns[2:]:\n        df_val[col] = df_val[col].apply(\n            lambda x: np.unique(x)[0] if len(np.unique(x)) == 1 else np.unique(x)\n        )\n    gts = []\n    dts = []\n    scores = []\n   for img_id in df_val.id.tolist():\n        img = mmcv.imread(root_dir + f\"train/{img_id}.png\")\n        result = inference_detector(model, img)\n\n        # dt\n        dt = []\n        for cls, bbs in enumerate(result[0]):\n            if bbs.shape != (0, 5):\n                sgs = result[1][cls]\n                for bb, sg in zip(bbs, sgs):\n                    box = bb[:4]\n                    cnf = bb[4]\n                    if cnf >= confidence_thresholds[cls]:\n                        mask = get_mask_from_result(sg)\n                        mask = remove_overlapping_pixels(mask, dt)\n                        dt.append(mask.astype(bool))\n                        #print(mask)\n        dts.append(combine_mask(dt))\n\n        # gt\n        shape = df_val.loc[df_val.id == img_id, ['height', 'width']].values[0]\n        gt = rles_to_mask(df_val.loc[df_val.id == img_id, \"annotation\"].item(), shape).astype(np.uint16)\n        gts.append(gt)\n        score = iou_map(gts, dts, True)\n        scores.append(score)\n    return scores\n
\n
scores  = calculate_custom_score(model, val_json, root_dir)\nprint(np.mean(scores, axis=0))\n
", "votes": null, "replies": [] }, { "id": 1621306, "author_name": "forgakki", "author_url": "", "post_date": "12/17/2021 16:06:06", "content": "

Thanks for your replay! Thanks @trushk's code！

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1604837": "I created a kernel for a baseline eval script for MMDetection here: https://www.kaggle.com/trushk/mmdetection-custom-eval-script\n\nOne main pending issue is that scores seem too low based compared to the LB score. I am hoping others who have tried a similar approach can shed some light on what they have done. \n\nCredit to @theoviel for the custom functions based on the competition eval metric shared here: https://www.kaggle.com/theoviel/competition-metric-map-iou\n\nAlso a shout out to my teammate @evilpsycho42 for the work done to create this notebook.", "1608270": "thanks for sharing @trushk", "1620966": "My CV score also too low compared to LB score,(0.206/0.307) , did you solve this problem? I tried to fix it，but failed", "1621082": "forgakki I used his code block with little modification and got a valid CV. Thanks! to @trushk again.\n\n```\n@torch.no_grad()\ndef calculate_custom_score(model, val_json, root_dir):\n df = pd.read_csv(root_dir + \"train.csv\")\n val_ids = [i['id'] for i in val_json['images']]\n if args.debug:\n val_ids = val_ids[0:args.debug]\n df_val = df[df.id.isin(val_ids)].reset_index(drop=True)\n df_val = df_val.groupby('id').agg(list).reset_index()\n del df\n for col in df_val.columns[2:]:\n df_val[col] = df_val[col].apply(\n lambda x: np.unique(x)[0] if len(np.unique(x)) == 1 else np.unique(x)\n )\n gts = []\n dts = []\n scores = []\n for img_id in df_val.id.tolist():\n img = mmcv.imread(root_dir + f\"train/{img_id}.png\")\n result = inference_detector(model, img)\n\n # dt\n dt = []\n for cls, bbs in enumerate(result[0]):\n if bbs.shape != (0, 5):\n sgs = result[1][cls]\n for bb, sg in zip(bbs, sgs):\n box = bb[:4]\n cnf = bb[4]\n if cnf >= confidence_thresholds[cls]:\n mask = get_mask_from_result(sg)\n mask = remove_overlapping_pixels(mask, dt)\n dt.append(mask.astype(bool))\n #print(mask)\n dts.append(combine_mask(dt))\n\n # gt\n shape = df_val.loc[df_val.id == img_id, ['height', 'width']].values[0]\n gt = rles_to_mask(df_val.loc[df_val.id == img_id, \"annotation\"].item(), shape).astype(np.uint16)\n gts.append(gt)\n score = iou_map(gts, dts, True)\n scores.append(score)\n return scores\n```\n\n```\nscores = calculate_custom_score(model, val_json, root_dir)\nprint(np.mean(scores, axis=0))\n```", "1621306": "Thanks for your replay! Thanks @trushk's code！" }, "source": "meta" }