{ "id": 292560, "title": "【Valid MAP IOU 0.26】Experiments of using EfficientnetV2 in FPN Mask RCNN ", "url": "/competitions/sartorius-cell-instance-segmentation/discussion/292560", "author_name": "Drzhuzhe", "post_date": "2021-12-02T14:42:54.285000", "votes": 11, "comment_count": 7, "views": 0, "content": "

2021/12/07 update

\n

Ablation Experiments Result

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
namechangedescriptionscorecode
EfficitionnetV2 backbone ( Colab lastest version)add conv_head to every output featureconverge much stable and faster , I use 60 epoch but maybe 35 - 40 epoch is enoughCV: 0.260 heavy overfittinghttps://github.com/mrzhuzhe/Patrick/blob/main/V2_of_Efficientnetv2_hacked_detectron2.ipynb
Resnet backbonereplace detectron backbone with TIMM resnet50 imagenet pretrain------CV: 0.247 heavy overfittinghttps://github.com/mrzhuzhe/Patrick/blob/main/V1_of_Resnet_hacked_detectron2.ipynb
EfficitionnetV2 backbone (previous version)replace detectron backbone with TIMM efficientnetv2s imagenet pretrain----60 epoch CV: 0.25 120 epoch CV: 0.26https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training
Orignal Detectron2 Resnet50-FPN-RCNN----------CV 0.269 LB: 0.296------
\n

Changes in lastest version

\n
    \n
  1. change output layer from b0 b1 b2 b3 b5 to b0 b1 b2 b4 b5 , fix an anchor config bug ( seems no influence in results)
    2. \n
  2. add conv_head to every EdgeConv and InverConv layer , this make network topological structure more consistency to efficientnet original implements , this result in 2 x faster converge than my previous implements
    3. \n
\n

Pros and Cons with alternate Backbone

\n

Cons:

\n
    \n
  1. lack of pretraining
    2. \n
  2. Risk of implement bug cause evaluation score drawback
    3. \n
\n

Pros:

\n
    \n
  1. more diversity in ensemble period
    2. \n
  2. fully customize , can make more improvement
    3. \n
\n

Next Version in several days

\n
    \n
  1. I will make a fully customize RPN and roi heads in detectron2 notebook
    2. \n
  2. add some data aug during training
    3. \n
\n
\n

2021/12/04 update

\n

a dirty code new version with competition data
\nMax map iou 0.251
\nMaybe some bug in it

\n
\n

https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training

\n
\n

Please leave some comment if you find some config wrong or implement bugs

\n
\n

1. An easy way to hack detectron2 Backbone and Trainer in notebook

\n

How to modify Detectron2

\n

Base on this config : COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml

\n

Resnet50 + FPN + Mask RCNN ( StandardROIHeads + FastRCNNConvFCHead + MaskRCNNConvUpsampleHead )

\n

Then I meet a problems : How to make customize to detectron in notebook,

\n

A template to easily modify backbone FPN and Trainer

\n
\n

https://github.com/mrzhuzhe/Patrick/blob/main/v2_of_Detectron2_Modification.ipynb

\n
\n
    \n

  1. With this template we can easily modify backbone FPN and Trainer
    \nlike: add log , inspect model instance

    2. \n

  2. regardless interface namespaces restriction from object oriented (from fvcore)
    \nrun code once and once in notebook

    3. \n
\n

maybe find it usefull

\n

2. A naive way to replace Resnet with EfficientNetV2

\n
\n

Notebook : https://www.kaggle.com/drzhuzhe/wip-pytorch-efficienetv2-fpn

\n
\n

Resnet50

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
namefuncstrideout_channel
stemconv2d + maxpooling2 * 264
res2(layer1)bconv1256
res3(layer2)bconv2512
res4(layer3)bconv21024
res5(layer4)bconv22048
\n

Efficientnetv2s

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
namefuncstrideout_channel
stemconv2d224
b0ConvBnAct124
b1EdgeResidual248
b2EdgeResidual264
b3InvertedResidual2128
b4InvertedResidual1160
b5InvertedResidual2256(original: 272)
\n

After reviews architecture , manually set _out_feature_strides and _out_feature_channel

\n

observation

\n
    \n
  1. It seems take more time to converge , with worse result
    2. \n
  2. does this due to lack of pretraining because I combine pretrain efficientnetV2 with original pretrain FPN ?
    3. \n
  3. this model may bu have some bug in FPN , RPN , and roi head , I will keep debugging to make it better
    4. \n
", "messages": [ { "id": 1603475, "postDate": "2021-12-02T14:42:54.287Z", "content": "

2021/12/07 update

\n

Ablation Experiments Result

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
namechangedescriptionscorecode
EfficitionnetV2 backbone ( Colab lastest version)add conv_head to every output featureconverge much stable and faster , I use 60 epoch but maybe 35 - 40 epoch is enoughCV: 0.260 heavy overfittinghttps://github.com/mrzhuzhe/Patrick/blob/main/V2_of_Efficientnetv2_hacked_detectron2.ipynb
Resnet backbonereplace detectron backbone with TIMM resnet50 imagenet pretrain------CV: 0.247 heavy overfittinghttps://github.com/mrzhuzhe/Patrick/blob/main/V1_of_Resnet_hacked_detectron2.ipynb
EfficitionnetV2 backbone (previous version)replace detectron backbone with TIMM efficientnetv2s imagenet pretrain----60 epoch CV: 0.25 120 epoch CV: 0.26https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training
Orignal Detectron2 Resnet50-FPN-RCNN----------CV 0.269 LB: 0.296------
\n

Changes in lastest version

\n
    \n
  1. change output layer from b0 b1 b2 b3 b5 to b0 b1 b2 b4 b5 , fix an anchor config bug ( seems no influence in results)
    2. \n
  2. add conv_head to every EdgeConv and InverConv layer , this make network topological structure more consistency to efficientnet original implements , this result in 2 x faster converge than my previous implements
    3. \n
\n

Pros and Cons with alternate Backbone

\n

Cons:

\n
    \n
  1. lack of pretraining
    2. \n
  2. Risk of implement bug cause evaluation score drawback
    3. \n
\n

Pros:

\n
    \n
  1. more diversity in ensemble period
    2. \n
  2. fully customize , can make more improvement
    3. \n
\n

Next Version in several days

\n
    \n
  1. I will make a fully customize RPN and roi heads in detectron2 notebook
    2. \n
  2. add some data aug during training
    3. \n
\n
\n

2021/12/04 update

\n

a dirty code new version with competition data
\nMax map iou 0.251
\nMaybe some bug in it

\n
\n

https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training

\n
\n

Please leave some comment if you find some config wrong or implement bugs

\n
\n

1. An easy way to hack detectron2 Backbone and Trainer in notebook

\n

How to modify Detectron2

\n

Base on this config : COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml

\n

Resnet50 + FPN + Mask RCNN ( StandardROIHeads + FastRCNNConvFCHead + MaskRCNNConvUpsampleHead )

\n

Then I meet a problems : How to make customize to detectron in notebook,

\n

A template to easily modify backbone FPN and Trainer

\n
\n

https://github.com/mrzhuzhe/Patrick/blob/main/v2_of_Detectron2_Modification.ipynb

\n
\n
    \n

  1. With this template we can easily modify backbone FPN and Trainer
    \nlike: add log , inspect model instance

    2. \n

  2. regardless interface namespaces restriction from object oriented (from fvcore)
    \nrun code once and once in notebook

    3. \n
\n

maybe find it usefull

\n

2. A naive way to replace Resnet with EfficientNetV2

\n
\n

Notebook : https://www.kaggle.com/drzhuzhe/wip-pytorch-efficienetv2-fpn

\n
\n

Resnet50

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
namefuncstrideout_channel
stemconv2d + maxpooling2 * 264
res2(layer1)bconv1256
res3(layer2)bconv2512
res4(layer3)bconv21024
res5(layer4)bconv22048
\n

Efficientnetv2s

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
namefuncstrideout_channel
stemconv2d224
b0ConvBnAct124
b1EdgeResidual248
b2EdgeResidual264
b3InvertedResidual2128
b4InvertedResidual1160
b5InvertedResidual2256(original: 272)
\n

After reviews architecture , manually set _out_feature_strides and _out_feature_channel

\n

observation

\n
    \n
  1. It seems take more time to converge , with worse result
    2. \n
  2. does this due to lack of pretraining because I combine pretrain efficientnetV2 with original pretrain FPN ?
    3. \n
  3. this model may bu have some bug in FPN , RPN , and roi head , I will keep debugging to make it better
    4. \n
", "rawMarkdown": "#2021/12/07 update\n\n## Ablation Experiments Result\n\n| name | change | description | score | code |\n| ---- | ---- | ------ | ------ | ------ |\n| EfficitionnetV2 backbone ( Colab lastest version) | add conv_head to every output feature | converge much stable and faster , I use 60 epoch but maybe 35 - 40 epoch is enough | CV: 0.260 heavy overfitting | https://github.com/mrzhuzhe/Patrick/blob/main/V2_of_Efficientnetv2_hacked_detectron2.ipynb |\n| Resnet backbone | replace detectron backbone with TIMM resnet50 imagenet pretrain | ------ | CV: 0.247 heavy overfitting | https://github.com/mrzhuzhe/Patrick/blob/main/V1_of_Resnet_hacked_detectron2.ipynb |\n| EfficitionnetV2 backbone (previous version) | replace detectron backbone with TIMM efficientnetv2s imagenet pretrain | ---- | 60 epoch CV: 0.25 120 epoch CV: 0.26 | https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training |\n| Orignal Detectron2 Resnet50-FPN-RCNN | ---- | ------ | CV 0.269 LB: 0.296 | ------ |\n\n\n## Changes in lastest version \n1. change output layer from b0 b1 b2 b3 b5 to b0 b1 b2 b4 b5 , fix an anchor config bug ( seems no influence in results)\n2. add conv_head to every EdgeConv and InverConv layer , this make network topological structure more consistency to efficientnet original implements , this result in 2 x faster converge than my previous implements\n\n# Pros and Cons with alternate Backbone\nCons:\n1. lack of pretraining \n2. Risk of implement bug cause evaluation score drawback\n\nPros:\n1. more diversity in ensemble period\n2. fully customize , can make more improvement\n\n\n\n## Next Version in several days\n1. I will make a fully customize RPN and roi heads in detectron2 notebook \n2. add some data aug during training\n\n\n\n------------------------------------------------------------------------------------------------------------\n\n\n\n\n# 2021/12/04 update \n\na dirty code new version with competition data\nMax map iou 0.251\nMaybe some bug in it \n\n> https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training\n\nPlease leave some comment if you find some config wrong or implement bugs\n\n\n------------------------------------------------------------------------\n\n## 1. An easy way to hack detectron2 Backbone and Trainer in notebook\n\n### How to modify Detectron2 \n\nBase on this config : COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\n\nResnet50 + FPN + Mask RCNN ( StandardROIHeads + FastRCNNConvFCHead + MaskRCNNConvUpsampleHead )\n\nThen I meet a problems : How to make customize to detectron in notebook,\n\n### A template to easily modify backbone FPN and Trainer \n\n> https://github.com/mrzhuzhe/Patrick/blob/main/v2_of_Detectron2_Modification.ipynb\n\n1. With this template we can easily modify backbone FPN and Trainer \nlike: add log , inspect model instance\n\n2. regardless interface namespaces restriction from object oriented (from fvcore) \nrun code once and once in notebook\n\nmaybe find it usefull\n\n## 2. A naive way to replace Resnet with EfficientNetV2 \n\n> Notebook : https://www.kaggle.com/drzhuzhe/wip-pytorch-efficienetv2-fpn\n\n\n### Resnet50\n\n| name | func | stride | out_channel |\n| ---- | ---- | ------ | ------ |\n| stem | conv2d + maxpooling | 2 * 2 | 64 |\n| res2(layer1) | bconv | 1 | 256 |\n| res3(layer2) | bconv | 2 | 512 |\n| res4(layer3) | bconv | 2 | 1024 |\n| res5(layer4) | bconv | 2 | 2048 |\n\n\n### Efficientnetv2s\n\n| name | func | stride | out_channel |\n| ---- | ---- | ------ | ------ |\n| stem | conv2d | 2 | 24 |\n| b0 | ConvBnAct | 1 | 24 |\n| b1 | EdgeResidual | 2 | 48 |\n| b2 | EdgeResidual | 2 | 64 |\n| b3 | InvertedResidual | 2 | 128 |\n| b4 | InvertedResidual | 1 | 160 |\n| b5 | InvertedResidual | 2 | 256(original: 272) |\n\nAfter reviews architecture , manually set _out_feature_strides and _out_feature_channel\n\n\n### observation\n\n1. It seems take more time to converge , with worse result\n2. does this due to lack of pretraining because I combine pretrain efficientnetV2 with original pretrain FPN ?\n3. this model may bu have some bug in FPN , RPN , and roi head , I will keep debugging to make it better \n\n\n", "votes": 11 }, { "id": 1607064, "postDate": "2021-12-05T14:18:04.903Z", "content": "

you can try to remove this line in the output:
\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set a custom testing threshold
\nand use this like detectron 0.294:
\nTHRESHOLDS = [.15, .35, .55]
\nMIN_PIXELS = [75, 150, 75]
\nbut in my experience of learning detectron, even basic mask_rcnn learns better.

", "rawMarkdown": "you can try to remove this line in the output:\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set a custom testing threshold\nand use this like detectron 0.294:\nTHRESHOLDS = [.15, .35, .55]\nMIN_PIXELS = [75, 150, 75]\nbut in my experience of learning detectron, even basic mask_rcnn learns better.", "votes": 1 }, { "id": 1606877, "postDate": "2021-12-05T10:57:08.547Z", "content": "

I find something , I will public some ablation experiments it when training end

", "rawMarkdown": "I find something , I will public some ablation experiments it when training end", "votes": 1 }, { "id": 1606773, "postDate": "2021-12-05T08:46:35.550Z", "content": "

the default backbone outperform customized one.

\n

the better model with worse LB is due to metric/labelling.

", "rawMarkdown": "the default backbone outperform customized one.\n\nthe better model with worse LB is due to metric/labelling.", "replies": [ { "id": 1606813, "postDate": "2021-12-05T09:37:59.370Z", "content": "

Maybe it is due to some implement bug or lack of pretraining

\n

My efficientnetv2 model valid map iou is 0.26

\n

default R50 valid map iou is 0.267 LB: 0.296

\n

It's so close , I'm keeping trying

", "rawMarkdown": "Maybe it is due to some implement bug or lack of pretraining\n\nMy efficientnetv2 model valid map iou is 0.26 \n\ndefault R50 valid map iou is 0.267 LB: 0.296\n\nIt's so close , I'm keeping trying" } ] }, { "id": 1603811, "postDate": "2021-12-02T19:52:47.857Z", "content": "

The link seems to be broken

", "rawMarkdown": "The link seems to be broken", "replies": [ { "id": 1604267, "postDate": "2021-12-03T07:54:06.713Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1605176, "postDate": "2021-12-04T03:11:17.017Z", "content": "

please take a look at this full version:

\n

https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training
\nMAX MaP IoU 0.25

", "rawMarkdown": "please take a look at this full version:\n\nhttps://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training\nMAX MaP IoU 0.25", "votes": 1 } ] } ], "comments": [ { "id": 1607064, "author_name": "Zaakcii Ru", "author_url": "", "post_date": "2021-12-05T14:18:04.903000", "content": "

you can try to remove this line in the output:
\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set a custom testing threshold
\nand use this like detectron 0.294:
\nTHRESHOLDS = [.15, .35, .55]
\nMIN_PIXELS = [75, 150, 75]
\nbut in my experience of learning detectron, even basic mask_rcnn learns better.

", "votes": 1, "replies": [] }, { "id": 1606877, "author_name": "Drzhuzhe", "author_url": "", "post_date": "2021-12-05T10:57:08.547000", "content": "

I find something , I will public some ablation experiments it when training end

", "votes": 1, "replies": [] }, { "id": 1606773, "author_name": "dragon zhang", "author_url": "", "post_date": "2021-12-05T08:46:35.550000", "content": "

the default backbone outperform customized one.

\n

the better model with worse LB is due to metric/labelling.

", "votes": 0, "replies": [ { "id": 1606813, "author_name": "Drzhuzhe", "author_url": "", "post_date": "2021-12-05T09:37:59.370000", "content": "

Maybe it is due to some implement bug or lack of pretraining

\n

My efficientnetv2 model valid map iou is 0.26

\n

default R50 valid map iou is 0.267 LB: 0.296

\n

It's so close , I'm keeping trying

", "votes": 0, "replies": [] } ] }, { "id": 1603811, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2021-12-02T19:52:47.857000", "content": "

The link seems to be broken

", "votes": 0, "replies": [ { "id": 1604267, "author_name": "", "author_url": "", "post_date": "2021-12-03T07:54:06.713000", "content": "", "votes": 0, "replies": [] }, { "id": 1605176, "author_name": "Drzhuzhe", "author_url": "", "post_date": "2021-12-04T03:11:17.017000", "content": "

please take a look at this full version:

\n

https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training
\nMAX MaP IoU 0.25

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1603475": "#2021/12/07 update\n\n## Ablation Experiments Result\n\n| name | change | description | score | code |\n| ---- | ---- | ------ | ------ | ------ |\n| EfficitionnetV2 backbone ( Colab lastest version) | add conv_head to every output feature | converge much stable and faster , I use 60 epoch but maybe 35 - 40 epoch is enough | CV: 0.260 heavy overfitting | https://github.com/mrzhuzhe/Patrick/blob/main/V2_of_Efficientnetv2_hacked_detectron2.ipynb |\n| Resnet backbone | replace detectron backbone with TIMM resnet50 imagenet pretrain | ------ | CV: 0.247 heavy overfitting | https://github.com/mrzhuzhe/Patrick/blob/main/V1_of_Resnet_hacked_detectron2.ipynb |\n| EfficitionnetV2 backbone (previous version) | replace detectron backbone with TIMM efficientnetv2s imagenet pretrain | ---- | 60 epoch CV: 0.25 120 epoch CV: 0.26 | https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training |\n| Orignal Detectron2 Resnet50-FPN-RCNN | ---- | ------ | CV 0.269 LB: 0.296 | ------ |\n\n\n## Changes in lastest version \n1. change output layer from b0 b1 b2 b3 b5 to b0 b1 b2 b4 b5 , fix an anchor config bug ( seems no influence in results)\n2. add conv_head to every EdgeConv and InverConv layer , this make network topological structure more consistency to efficientnet original implements , this result in 2 x faster converge than my previous implements\n\n# Pros and Cons with alternate Backbone\nCons:\n1. lack of pretraining \n2. Risk of implement bug cause evaluation score drawback\n\nPros:\n1. more diversity in ensemble period\n2. fully customize , can make more improvement\n\n\n\n## Next Version in several days\n1. I will make a fully customize RPN and roi heads in detectron2 notebook \n2. add some data aug during training\n\n\n\n------------------------------------------------------------------------------------------------------------\n\n\n\n\n# 2021/12/04 update \n\na dirty code new version with competition data\nMax map iou 0.251\nMaybe some bug in it \n\n> https://www.kaggle.com/drzhuzhe/efficientnetv2-detectron-2-3-training\n\nPlease leave some comment if you find some config wrong or implement bugs\n\n\n------------------------------------------------------------------------\n\n## 1. An easy way to hack detectron2 Backbone and Trainer in notebook\n\n### How to modify Detectron2 \n\nBase on this config : COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\n\nResnet50 + FPN + Mask RCNN ( StandardROIHeads + FastRCNNConvFCHead + MaskRCNNConvUpsampleHead )\n\nThen I meet a problems : How to make customize to detectron in notebook,\n\n### A template to easily modify backbone FPN and Trainer \n\n> https://github.com/mrzhuzhe/Patrick/blob/main/v2_of_Detectron2_Modification.ipynb\n\n1. With this template we can easily modify backbone FPN and Trainer \nlike: add log , inspect model instance\n\n2. regardless interface namespaces restriction from object oriented (from fvcore) \nrun code once and once in notebook\n\nmaybe find it usefull\n\n## 2. A naive way to replace Resnet with EfficientNetV2 \n\n> Notebook : https://www.kaggle.com/drzhuzhe/wip-pytorch-efficienetv2-fpn\n\n\n### Resnet50\n\n| name | func | stride | out_channel |\n| ---- | ---- | ------ | ------ |\n| stem | conv2d + maxpooling | 2 * 2 | 64 |\n| res2(layer1) | bconv | 1 | 256 |\n| res3(layer2) | bconv | 2 | 512 |\n| res4(layer3) | bconv | 2 | 1024 |\n| res5(layer4) | bconv | 2 | 2048 |\n\n\n### Efficientnetv2s\n\n| name | func | stride | out_channel |\n| ---- | ---- | ------ | ------ |\n| stem | conv2d | 2 | 24 |\n| b0 | ConvBnAct | 1 | 24 |\n| b1 | EdgeResidual | 2 | 48 |\n| b2 | EdgeResidual | 2 | 64 |\n| b3 | InvertedResidual | 2 | 128 |\n| b4 | InvertedResidual | 1 | 160 |\n| b5 | InvertedResidual | 2 | 256(original: 272) |\n\nAfter reviews architecture , manually set _out_feature_strides and _out_feature_channel\n\n\n### observation\n\n1. It seems take more time to converge , with worse result\n2. does this due to lack of pretraining because I combine pretrain efficientnetV2 with original pretrain FPN ?\n3. this model may bu have some bug in FPN , RPN , and roi head , I will keep debugging to make it better \n\n\n", "1607064": "you can try to remove this line in the output:\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set a custom testing threshold\nand use this like detectron 0.294:\nTHRESHOLDS = [.15, .35, .55]\nMIN_PIXELS = [75, 150, 75]\nbut in my experience of learning detectron, even basic mask_rcnn learns better.", "1606877": "I find something , I will public some ablation experiments it when training end", "1606773": "the default backbone outperform customized one.\n\nthe better model with worse LB is due to metric/labelling.", "1603811": "The link seems to be broken" } }