Congratulations to all the winners. Here is our solution:
\n\nWe used mmdetection with 4 Tesla T4 during training, 8 Tesla T4 during TTA inference.
\n\nThere are 300 classes, in 3 hierarchical levels: \n- 275 leaf classes\n- 23 parent classes \n- 2 grandparent classes (Carnivore and Reptile)
For the 275 leaf classes, we train models with these 275 classes as labels and use inference results directly.
\n\nFor the 25 parent and grandparent classes, there are two methods to get the predictions:\n- (i). Use the prediction results from the leaf model, and “expand” to the parent and grandparent. For example, if the leaf model predicted a Tortoise mask, we add a Turtle prediction (its parent) and a Reptile prediction (its grandparent) with the same mask and score.
\n- (ii). We train models with only the 23 parent classes as labels, and expand to the 2 grandparent classes. Note that for these models, we need to create hierarchical expansion of the instance segmentation before training (as explained here).
We sampled a small subset from the official validation set for faster inference, 2844 images for the leaf model and 3841 images for the parent model. We used the validation scores to decide when to adjust learning rate, and to choose best checkpoints.
\n\nThe validation scores are significantly higher than LB. For the 275 leaf classes, we have validation score over 0.8. But nevertheless, the delta between validation and LB are stable.
\n\nThe open images dataset is extremely imbalanced between the 300 classes. We rebalanced the training data for the leaf model as follows:\nSort the 275 classes by descending number of images. So class #0 is the largest class.
\n\n| Group | Class | original num of imgs per class | rebalancing |\n|-----|------------|--------------------------------|-------------------------------|\n| 1 | #241 to #274 | 150-13 | oversample x10 |\n| 2 | #64 to #240 | 1500-150 | oversample to 1500 imgs/class |\n| 3 | #24 to #63 | 6k-1500 | no rebalancing |\n| 4 | #0 to #23 | 89k-6k | downsample to 6k* |
\n\nEach epoch has about 450k images after rebalancing.
\n\n*For the downsample, we use different random seeds for each epoch to feed the model as many images as possible. Also, this 6k number is a rough target. In actual sampling, the top few classes ended up having more images because many images have multiple labels, and top classes already have more than 6k images after Group 1-3 sampling are done.
\n\nSimilarly for the parent model, sort the 23 classes
\n\n| Group | Class | rebalancing |\n|-------|----------|------------------------------|\n| 1 | #10 to #22 | upsample to 10k imgs/class |\n| 2 | #5 to #9 | no rebalancing |\n| 3 | #0 to #4 | downsample to 30k imgs/class |
\n\nEach epoch has about 330k images after rebalancing.
\n\nWe trained 3 cascade mrcnn leaf models\n- L1: backbone x101\n- L2: backbone r101 + deformable module\n- L3: backbone x101 + deformable module
\n\nand 1 cascade mrcnn parent model\n- P1: backbone x101
\n\nhyper-paramters:\n\nimgs_per_gpu=1\nnum_gpu=4\nimg_scale=[(1333, 640), (1333, 960)]\nmultiscale_mode='range'\n
lr schedule:\nmodel L1: 0.005 for 760k iterations; 0.005/3 for 60k iterations; 0.005/15 for 152k iterations\nmodel L2: 0.005 for 675k iterations; 0.005/3 for 64k iterations; 0.005/15 for 48k iterations\nmodel L3: 0.005 for 226k iterations; 0.005/5 for 132k iterations; 0.005/50 for 26k iterations\nmodel P1: 0.005 for 148k iterations; 0.005/5 for 62k iterations; 0.005/50 for 8k iterations
\n\nTraining took 0.5-0.65 hour per 1000 iterations with 4 T4. So in total the training the 4 models took about 22, 18, 10 and 5 days respectively.
\n\nPublic/private scores of each (max_per_img=120, thr=0, without TTA):
| model | public score | private score | |\n|-------|--------------|---------------|----------------|\n| L1 | 0.4708 | 0.4354 | on 275 classes |\n| L2 | 0.4705 | 0.4275 | on 275 classes |\n| L3 | 0.4712 | 0.4324 | on 275 classes |\n| P1 | 0.0349 | 0.0344 | on 25 classes |
\n\nWe used the TTA implementation from Miras Amir's winning solution of iMaterialist (Fashion) 2019
\n\nFor the leaf model, it’s the ensemble of 3 single models, 2 scale (1333,800) and (1600,960), and flip. At RPN and BB stages, it’s the NMS ensemble of 12 single models. In the end, it’s the mean of all 12 masks for each instance.
\n\nTTA inference of the leaf model is very slow, which took about 525 T4-hours. We split the 99999 images into 25 chunks and ran it in parallel.
\n\nFor the parent model, it’s the ensemble of 2 scale and flip, but just one single model.
\n\nLastly, for the 25 parent and grandparent classes, we implemented NMS ensemble at mask level (i.e. calculating the mask iou instead of bbox iou when determining which ones to suppress) to ensemble (i) leaf class predications expanded to 25 classes, and (ii) parent class predications expand to 25 classes.
\n\nPublic/private scores of each (max_per_img=120, thr=0):
| model | public score | private score | |\n|--------------------------|--------------|---------------|----------------|\n| leaf model ensemble | 0.5007 | 0.4548 | on 275 classes |\n| leaf ensemble expanded | 0.0325 | 0.0330 | on 25 classes |\n| ensemble of (i) and (ii) | 0.0370 | 0.0370 | on 25 classes |
\n\nAt this point, the total score is public 0.5007+0.0370 = 0.5378, private 0.4548+0.0370 = 0.4918.
\n\nThere’s one last thing we did to boost scores to 0.5383/0.4922: set max_per_img=200 for the leaf ensemble. But we only had time (also restricted by sub file size) to do this for 12/25 of the test images.
Training, inference, pre- and post-process code are available at https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained model weights are also linked in the readme there
", "messages": [ { "id": "639047", "postDate": "10/02/2019 18:00:42", "content": "Congratulations to all the winners. Here is our solution:
\n\nWe used mmdetection with 4 Tesla T4 during training, 8 Tesla T4 during TTA inference.
\n\nThere are 300 classes, in 3 hierarchical levels: \n- 275 leaf classes\n- 23 parent classes \n- 2 grandparent classes (Carnivore and Reptile)
For the 275 leaf classes, we train models with these 275 classes as labels and use inference results directly.
\n\nFor the 25 parent and grandparent classes, there are two methods to get the predictions:\n- (i). Use the prediction results from the leaf model, and “expand” to the parent and grandparent. For example, if the leaf model predicted a Tortoise mask, we add a Turtle prediction (its parent) and a Reptile prediction (its grandparent) with the same mask and score.
\n- (ii). We train models with only the 23 parent classes as labels, and expand to the 2 grandparent classes. Note that for these models, we need to create hierarchical expansion of the instance segmentation before training (as explained here).
We sampled a small subset from the official validation set for faster inference, 2844 images for the leaf model and 3841 images for the parent model. We used the validation scores to decide when to adjust learning rate, and to choose best checkpoints.
\n\nThe validation scores are significantly higher than LB. For the 275 leaf classes, we have validation score over 0.8. But nevertheless, the delta between validation and LB are stable.
\n\nThe open images dataset is extremely imbalanced between the 300 classes. We rebalanced the training data for the leaf model as follows:\nSort the 275 classes by descending number of images. So class #0 is the largest class.
\n\n| Group | Class | original num of imgs per class | rebalancing |\n|-----|------------|--------------------------------|-------------------------------|\n| 1 | #241 to #274 | 150-13 | oversample x10 |\n| 2 | #64 to #240 | 1500-150 | oversample to 1500 imgs/class |\n| 3 | #24 to #63 | 6k-1500 | no rebalancing |\n| 4 | #0 to #23 | 89k-6k | downsample to 6k* |
\n\nEach epoch has about 450k images after rebalancing.
\n\n*For the downsample, we use different random seeds for each epoch to feed the model as many images as possible. Also, this 6k number is a rough target. In actual sampling, the top few classes ended up having more images because many images have multiple labels, and top classes already have more than 6k images after Group 1-3 sampling are done.
\n\nSimilarly for the parent model, sort the 23 classes
\n\n| Group | Class | rebalancing |\n|-------|----------|------------------------------|\n| 1 | #10 to #22 | upsample to 10k imgs/class |\n| 2 | #5 to #9 | no rebalancing |\n| 3 | #0 to #4 | downsample to 30k imgs/class |
\n\nEach epoch has about 330k images after rebalancing.
\n\nWe trained 3 cascade mrcnn leaf models\n- L1: backbone x101\n- L2: backbone r101 + deformable module\n- L3: backbone x101 + deformable module
\n\nand 1 cascade mrcnn parent model\n- P1: backbone x101
\n\nhyper-paramters:\n\nimgs_per_gpu=1\nnum_gpu=4\nimg_scale=[(1333, 640), (1333, 960)]\nmultiscale_mode='range'\n
lr schedule:\nmodel L1: 0.005 for 760k iterations; 0.005/3 for 60k iterations; 0.005/15 for 152k iterations\nmodel L2: 0.005 for 675k iterations; 0.005/3 for 64k iterations; 0.005/15 for 48k iterations\nmodel L3: 0.005 for 226k iterations; 0.005/5 for 132k iterations; 0.005/50 for 26k iterations\nmodel P1: 0.005 for 148k iterations; 0.005/5 for 62k iterations; 0.005/50 for 8k iterations
\n\nTraining took 0.5-0.65 hour per 1000 iterations with 4 T4. So in total the training the 4 models took about 22, 18, 10 and 5 days respectively.
\n\nPublic/private scores of each (max_per_img=120, thr=0, without TTA):
| model | public score | private score | |\n|-------|--------------|---------------|----------------|\n| L1 | 0.4708 | 0.4354 | on 275 classes |\n| L2 | 0.4705 | 0.4275 | on 275 classes |\n| L3 | 0.4712 | 0.4324 | on 275 classes |\n| P1 | 0.0349 | 0.0344 | on 25 classes |
\n\nWe used the TTA implementation from Miras Amir's winning solution of iMaterialist (Fashion) 2019
\n\nFor the leaf model, it’s the ensemble of 3 single models, 2 scale (1333,800) and (1600,960), and flip. At RPN and BB stages, it’s the NMS ensemble of 12 single models. In the end, it’s the mean of all 12 masks for each instance.
\n\nTTA inference of the leaf model is very slow, which took about 525 T4-hours. We split the 99999 images into 25 chunks and ran it in parallel.
\n\nFor the parent model, it’s the ensemble of 2 scale and flip, but just one single model.
\n\nLastly, for the 25 parent and grandparent classes, we implemented NMS ensemble at mask level (i.e. calculating the mask iou instead of bbox iou when determining which ones to suppress) to ensemble (i) leaf class predications expanded to 25 classes, and (ii) parent class predications expand to 25 classes.
\n\nPublic/private scores of each (max_per_img=120, thr=0):
| model | public score | private score | |\n|--------------------------|--------------|---------------|----------------|\n| leaf model ensemble | 0.5007 | 0.4548 | on 275 classes |\n| leaf ensemble expanded | 0.0325 | 0.0330 | on 25 classes |\n| ensemble of (i) and (ii) | 0.0370 | 0.0370 | on 25 classes |
\n\nAt this point, the total score is public 0.5007+0.0370 = 0.5378, private 0.4548+0.0370 = 0.4918.
\n\nThere’s one last thing we did to boost scores to 0.5383/0.4922: set max_per_img=200 for the leaf ensemble. But we only had time (also restricted by sub file size) to do this for 12/25 of the test images.
Training, inference, pre- and post-process code are available at https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained model weights are also linked in the readme there
", "rawMarkdown": "Congratulations to all the winners. Here is our solution:\n\n## Codebase and hardware\n\nWe used [mmdetection](https://github.com/open-mmlab/mmdetection) with 4 Tesla T4 during training, 8 Tesla T4 during TTA inference.\n\n## Model setup\n\nThere are 300 classes, in 3 hierarchical levels: \n- 275 leaf classes\n- 23 parent classes \n- 2 grandparent classes (`Carnivore` and `Reptile`)\n\nFor the 275 leaf classes, we train models with these 275 classes as labels and use inference results directly.\n\nFor the 25 parent and grandparent classes, there are two methods to get the predictions:\n- (i). Use the prediction results from the leaf model, and “expand” to the parent and grandparent. For example, if the leaf model predicted a `Tortoise` mask, we add a `Turtle` prediction (its parent) and a `Reptile` prediction (its grandparent) with the same mask and score. \n- (ii). We train models with only the 23 parent classes as labels, and expand to the 2 grandparent classes. Note that for these models, we need to create hierarchical expansion of the instance segmentation before training (as explained [here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#instance-segmentation-track)).\n\n\n## Validation set\n\nWe sampled a small subset from the official validation set for faster inference, 2844 images for the leaf model and 3841 images for the parent model. We used the validation scores to decide when to adjust learning rate, and to choose best checkpoints. \n\nThe validation scores are significantly higher than LB. For the 275 leaf classes, we have validation score over 0.8. But nevertheless, the delta between validation and LB are stable.\n\n\n## Training set rebalancing\n\nThe open images dataset is extremely imbalanced between the 300 classes. We rebalanced the training data for the leaf model as follows:\nSort the 275 classes by descending number of images. So class #0 is the largest class.\n\n| Group | Class | original num of imgs per class | rebalancing |\n|-----|------------|--------------------------------|-------------------------------|\n| 1 | #241 to #274 | 150-13 | oversample x10 |\n| 2 | #64 to #240 | 1500-150 | oversample to 1500 imgs/class |\n| 3 | #24 to #63 | 6k-1500 | no rebalancing |\n| 4 | #0 to #23 | 89k-6k | downsample to 6k* |\n\nEach epoch has about 450k images after rebalancing. \n\n*For the downsample, we use different random seeds for each epoch to feed the model as many images as possible. Also, this 6k number is a rough target. In actual sampling, the top few classes ended up having more images because many images have multiple labels, and top classes already have more than 6k images after Group 1-3 sampling are done.\n\nSimilarly for the parent model, sort the 23 classes\n\n| Group | Class | rebalancing |\n|-------|----------|------------------------------|\n| 1 | #10 to #22 | upsample to 10k imgs/class |\n| 2 | #5 to #9 | no rebalancing |\n| 3 | #0 to #4 | downsample to 30k imgs/class |\n\nEach epoch has about 330k images after rebalancing.\n\n## Single models\n\nWe trained 3 cascade mrcnn leaf models\n- L1: backbone x101\n- L2: backbone r101 + deformable module\n- L3: backbone x101 + deformable module\n\nand 1 cascade mrcnn parent model\n- P1: backbone x101\n\nhyper-paramters:\n```\nimgs_per_gpu=1\nnum_gpu=4\nimg_scale=[(1333, 640), (1333, 960)]\nmultiscale_mode='range'\n```\n\nlr schedule:\nmodel L1: 0.005 for 760k iterations; 0.005/3 for 60k iterations; 0.005/15 for 152k iterations\nmodel L2: 0.005 for 675k iterations; 0.005/3 for 64k iterations; 0.005/15 for 48k iterations\nmodel L3: 0.005 for 226k iterations; 0.005/5 for 132k iterations; 0.005/50 for 26k iterations\nmodel P1: 0.005 for 148k iterations; 0.005/5 for 62k iterations; 0.005/50 for 8k iterations\n\nTraining took 0.5-0.65 hour per 1000 iterations with 4 T4. So in total the training the 4 models took about 22, 18, 10 and 5 days respectively.\n\nPublic/private scores of each (`max_per_img=120, thr=0`, without TTA):\n\n| model | public score | private score | |\n|-------|--------------|---------------|----------------|\n| L1 | 0.4708 | 0.4354 | on 275 classes |\n| L2 | 0.4705 | 0.4275 | on 275 classes |\n| L3 | 0.4712 | 0.4324 | on 275 classes |\n| P1 | 0.0349 | 0.0344 | on 25 classes |\n\n\n## TTA\n\nWe used the TTA implementation from [Miras Amir's winning solution](https://github.com/amirassov/kaggle-imaterialist) of iMaterialist (Fashion) 2019 \n\nFor the leaf model, it’s the ensemble of 3 single models, 2 scale (1333,800) and (1600,960), and flip. At RPN and BB stages, it’s the NMS ensemble of 12 single models. In the end, it’s the mean of all 12 masks for each instance. \n\nTTA inference of the leaf model is very slow, which took about 525 T4-hours. We split the 99999 images into 25 chunks and ran it in parallel. \n\nFor the parent model, it’s the ensemble of 2 scale and flip, but just one single model.\n\nLastly, for the 25 parent and grandparent classes, we implemented NMS ensemble at mask level (i.e. calculating the mask iou instead of bbox iou when determining which ones to suppress) to ensemble (i) leaf class predications expanded to 25 classes, and (ii) parent class predications expand to 25 classes.\n\nPublic/private scores of each (`max_per_img=120, thr=0`):\n\n| model | public score | private score | |\n|--------------------------|--------------|---------------|----------------|\n| leaf model ensemble | 0.5007 | 0.4548 | on 275 classes |\n| leaf ensemble expanded | 0.0325 | 0.0330 | on 25 classes |\n| ensemble of (i) and (ii) | 0.0370 | 0.0370 | on 25 classes |\n\nAt this point, the total score is public 0.5007+0.0370 = 0.5378, private 0.4548+0.0370 = 0.4918. \n\nThere’s one last thing we did to boost scores to 0.5383/0.4922: set `max_per_img=200` for the leaf ensemble. But we only had time (also restricted by sub file size) to do this for 12/25 of the test images.\n\n\n## Code\n\nTraining, inference, pre- and post-process code are available at https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained model weights are also linked in the readme there", "votes": null }, { "id": "639324", "postDate": "10/03/2019 04:30:35", "content": "great!
", "rawMarkdown": "great!", "votes": null }, { "id": "639343", "postDate": "10/03/2019 05:12:35", "content": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @boliu0
", "rawMarkdown": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @boliu0", "votes": null }, { "id": "639592", "postDate": "10/03/2019 11:32:18", "content": "Congratulations on winning a gold medal! Very nice solution approaches! Looking forward to your code solutions. :-)
", "rawMarkdown": "Congratulations on winning a gold medal! Very nice solution approaches! Looking forward to your code solutions. :-)", "votes": null }, { "id": "642385", "postDate": "10/06/2019 01:36:22", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "642561", "postDate": "10/06/2019 09:38:14", "content": "Congrats, thank you for sharing approach
", "rawMarkdown": "Congrats, thank you for sharing approach", "votes": null }, { "id": "643617", "postDate": "10/07/2019 17:32:23", "content": "Update: We have pushed code (training, inference, pre-process, post-process) to https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained weights are also linked in the repo readme
", "rawMarkdown": "**Update**: We have pushed code (training, inference, pre-process, post-process) to https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained weights are also linked in the repo readme", "votes": null }, { "id": "644584", "postDate": "10/09/2019 03:19:28", "content": "Congrats!\nI wonder how do you do the operation upsample when rebalancing. Just repeat the images?
", "rawMarkdown": "Congrats!\nI wonder how do you do the operation upsample when rebalancing. Just repeat the images?", "votes": null }, { "id": "644897", "postDate": "10/09/2019 13:48:07", "content": "Thanks. Yes, unsample and oversample mean the same thing in my writeup above.
\n\n\"upsample to 10k imgs/class\" was done like this:\nIf a class has 3000 images, then repeat each image 3 times, then randomly choose 1000 (so that they appear 4 times);\nif a class has 7000 images, then randomly choose 3000 (so that these 3000 appear twice, the other 4000 appear once);\n...
\n\nThis was done using different seeds for each epoch, so that a different 3000 in above example was chosen in each epoch.
\n\nImplementation at:\nhttps://github.com/boliu61/open-images-2019-instance-segmentation/blob/master/util/make_rebalanced_train_ann_parent.py#L116
", "rawMarkdown": "Thanks. Yes, unsample and oversample mean the same thing in my writeup above. \n\n\"upsample to 10k imgs/class\" was done like this:\nIf a class has 3000 images, then repeat each image 3 times, then randomly choose 1000 (so that they appear 4 times);\nif a class has 7000 images, then randomly choose 3000 (so that these 3000 appear twice, the other 4000 appear once);\n...\n\nThis was done using different seeds for each epoch, so that a different 3000 in above example was chosen in each epoch.\n\nImplementation at:\nhttps://github.com/boliu61/open-images-2019-instance-segmentation/blob/master/util/make_rebalanced_train_ann_parent.py#L116", "votes": null }, { "id": "645271", "postDate": "10/10/2019 00:26:30", "content": "thx👍
", "rawMarkdown": "thx👍", "votes": null } ], "comments": [ { "id": 639324, "author_name": "ryyghfjk89", "author_url": "", "post_date": "10/03/2019 04:30:35", "content": "great!
", "votes": null, "replies": [] }, { "id": 639343, "author_name": "veeralakrishna", "author_url": "", "post_date": "10/03/2019 05:12:35", "content": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @boliu0
", "votes": null, "replies": [] }, { "id": 639592, "author_name": "projdev", "author_url": "", "post_date": "10/03/2019 11:32:18", "content": "Congratulations on winning a gold medal! Very nice solution approaches! Looking forward to your code solutions. :-)
", "votes": null, "replies": [] }, { "id": 642385, "author_name": "terenceliu4444", "author_url": "", "post_date": "10/06/2019 01:36:22", "content": "Congratulations!
", "votes": null, "replies": [] }, { "id": 642561, "author_name": "corochann", "author_url": "", "post_date": "10/06/2019 09:38:14", "content": "Congrats, thank you for sharing approach
", "votes": null, "replies": [] }, { "id": 643617, "author_name": "boliu0", "author_url": "", "post_date": "10/07/2019 17:32:23", "content": "Update: We have pushed code (training, inference, pre-process, post-process) to https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained weights are also linked in the repo readme
", "votes": null, "replies": [] }, { "id": 644584, "author_name": "wangchuanyuan", "author_url": "", "post_date": "10/09/2019 03:19:28", "content": "Congrats!\nI wonder how do you do the operation upsample when rebalancing. Just repeat the images?
", "votes": null, "replies": [ { "id": 644897, "author_name": "boliu0", "author_url": "", "post_date": "10/09/2019 13:48:07", "content": "Thanks. Yes, unsample and oversample mean the same thing in my writeup above.
\n\n\"upsample to 10k imgs/class\" was done like this:\nIf a class has 3000 images, then repeat each image 3 times, then randomly choose 1000 (so that they appear 4 times);\nif a class has 7000 images, then randomly choose 3000 (so that these 3000 appear twice, the other 4000 appear once);\n...
\n\nThis was done using different seeds for each epoch, so that a different 3000 in above example was chosen in each epoch.
\n\nImplementation at:\nhttps://github.com/boliu61/open-images-2019-instance-segmentation/blob/master/util/make_rebalanced_train_ann_parent.py#L116
", "votes": null, "replies": [] }, { "id": 645271, "author_name": "wangchuanyuan", "author_url": "", "post_date": "10/10/2019 00:26:30", "content": "thx👍
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "639047": "Congratulations to all the winners. Here is our solution:\n\n## Codebase and hardware\n\nWe used [mmdetection](https://github.com/open-mmlab/mmdetection) with 4 Tesla T4 during training, 8 Tesla T4 during TTA inference.\n\n## Model setup\n\nThere are 300 classes, in 3 hierarchical levels: \n- 275 leaf classes\n- 23 parent classes \n- 2 grandparent classes (`Carnivore` and `Reptile`)\n\nFor the 275 leaf classes, we train models with these 275 classes as labels and use inference results directly.\n\nFor the 25 parent and grandparent classes, there are two methods to get the predictions:\n- (i). Use the prediction results from the leaf model, and “expand” to the parent and grandparent. For example, if the leaf model predicted a `Tortoise` mask, we add a `Turtle` prediction (its parent) and a `Reptile` prediction (its grandparent) with the same mask and score. \n- (ii). We train models with only the 23 parent classes as labels, and expand to the 2 grandparent classes. Note that for these models, we need to create hierarchical expansion of the instance segmentation before training (as explained [here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/challenge_evaluation.md#instance-segmentation-track)).\n\n\n## Validation set\n\nWe sampled a small subset from the official validation set for faster inference, 2844 images for the leaf model and 3841 images for the parent model. We used the validation scores to decide when to adjust learning rate, and to choose best checkpoints. \n\nThe validation scores are significantly higher than LB. For the 275 leaf classes, we have validation score over 0.8. But nevertheless, the delta between validation and LB are stable.\n\n\n## Training set rebalancing\n\nThe open images dataset is extremely imbalanced between the 300 classes. We rebalanced the training data for the leaf model as follows:\nSort the 275 classes by descending number of images. So class #0 is the largest class.\n\n| Group | Class | original num of imgs per class | rebalancing |\n|-----|------------|--------------------------------|-------------------------------|\n| 1 | #241 to #274 | 150-13 | oversample x10 |\n| 2 | #64 to #240 | 1500-150 | oversample to 1500 imgs/class |\n| 3 | #24 to #63 | 6k-1500 | no rebalancing |\n| 4 | #0 to #23 | 89k-6k | downsample to 6k* |\n\nEach epoch has about 450k images after rebalancing. \n\n*For the downsample, we use different random seeds for each epoch to feed the model as many images as possible. Also, this 6k number is a rough target. In actual sampling, the top few classes ended up having more images because many images have multiple labels, and top classes already have more than 6k images after Group 1-3 sampling are done.\n\nSimilarly for the parent model, sort the 23 classes\n\n| Group | Class | rebalancing |\n|-------|----------|------------------------------|\n| 1 | #10 to #22 | upsample to 10k imgs/class |\n| 2 | #5 to #9 | no rebalancing |\n| 3 | #0 to #4 | downsample to 30k imgs/class |\n\nEach epoch has about 330k images after rebalancing.\n\n## Single models\n\nWe trained 3 cascade mrcnn leaf models\n- L1: backbone x101\n- L2: backbone r101 + deformable module\n- L3: backbone x101 + deformable module\n\nand 1 cascade mrcnn parent model\n- P1: backbone x101\n\nhyper-paramters:\n```\nimgs_per_gpu=1\nnum_gpu=4\nimg_scale=[(1333, 640), (1333, 960)]\nmultiscale_mode='range'\n```\n\nlr schedule:\nmodel L1: 0.005 for 760k iterations; 0.005/3 for 60k iterations; 0.005/15 for 152k iterations\nmodel L2: 0.005 for 675k iterations; 0.005/3 for 64k iterations; 0.005/15 for 48k iterations\nmodel L3: 0.005 for 226k iterations; 0.005/5 for 132k iterations; 0.005/50 for 26k iterations\nmodel P1: 0.005 for 148k iterations; 0.005/5 for 62k iterations; 0.005/50 for 8k iterations\n\nTraining took 0.5-0.65 hour per 1000 iterations with 4 T4. So in total the training the 4 models took about 22, 18, 10 and 5 days respectively.\n\nPublic/private scores of each (`max_per_img=120, thr=0`, without TTA):\n\n| model | public score | private score | |\n|-------|--------------|---------------|----------------|\n| L1 | 0.4708 | 0.4354 | on 275 classes |\n| L2 | 0.4705 | 0.4275 | on 275 classes |\n| L3 | 0.4712 | 0.4324 | on 275 classes |\n| P1 | 0.0349 | 0.0344 | on 25 classes |\n\n\n## TTA\n\nWe used the TTA implementation from [Miras Amir's winning solution](https://github.com/amirassov/kaggle-imaterialist) of iMaterialist (Fashion) 2019 \n\nFor the leaf model, it’s the ensemble of 3 single models, 2 scale (1333,800) and (1600,960), and flip. At RPN and BB stages, it’s the NMS ensemble of 12 single models. In the end, it’s the mean of all 12 masks for each instance. \n\nTTA inference of the leaf model is very slow, which took about 525 T4-hours. We split the 99999 images into 25 chunks and ran it in parallel. \n\nFor the parent model, it’s the ensemble of 2 scale and flip, but just one single model.\n\nLastly, for the 25 parent and grandparent classes, we implemented NMS ensemble at mask level (i.e. calculating the mask iou instead of bbox iou when determining which ones to suppress) to ensemble (i) leaf class predications expanded to 25 classes, and (ii) parent class predications expand to 25 classes.\n\nPublic/private scores of each (`max_per_img=120, thr=0`):\n\n| model | public score | private score | |\n|--------------------------|--------------|---------------|----------------|\n| leaf model ensemble | 0.5007 | 0.4548 | on 275 classes |\n| leaf ensemble expanded | 0.0325 | 0.0330 | on 25 classes |\n| ensemble of (i) and (ii) | 0.0370 | 0.0370 | on 25 classes |\n\nAt this point, the total score is public 0.5007+0.0370 = 0.5378, private 0.4548+0.0370 = 0.4918. \n\nThere’s one last thing we did to boost scores to 0.5383/0.4922: set `max_per_img=200` for the leaf ensemble. But we only had time (also restricted by sub file size) to do this for 12/25 of the test images.\n\n\n## Code\n\nTraining, inference, pre- and post-process code are available at https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained model weights are also linked in the readme there", "639324": "great!", "639343": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @boliu0", "639592": "Congratulations on winning a gold medal! Very nice solution approaches! Looking forward to your code solutions. :-)", "642385": "Congratulations!", "642561": "Congrats, thank you for sharing approach", "643617": "**Update**: We have pushed code (training, inference, pre-process, post-process) to https://github.com/boliu61/open-images-2019-instance-segmentation\nTrained weights are also linked in the repo readme", "644584": "Congrats!\nI wonder how do you do the operation upsample when rebalancing. Just repeat the images?", "644897": "Thanks. Yes, unsample and oversample mean the same thing in my writeup above. \n\n\"upsample to 10k imgs/class\" was done like this:\nIf a class has 3000 images, then repeat each image 3 times, then randomly choose 1000 (so that they appear 4 times);\nif a class has 7000 images, then randomly choose 3000 (so that these 3000 appear twice, the other 4000 appear once);\n...\n\nThis was done using different seeds for each epoch, so that a different 3000 in above example was chosen in each epoch.\n\nImplementation at:\nhttps://github.com/boliu61/open-images-2019-instance-segmentation/blob/master/util/make_rebalanced_train_ann_parent.py#L116", "645271": "thx👍" }, "source": "meta" }