![\"\"](\"https://i.ibb.co/j5Vkb9h/Selection-999-591.png\")
Hi everyone,
\nThis is doing my head in… not sure if anyone else experienced this or someone may be able to give me some advice.
\nI used this model checkpoint in the public dataset to get score of 0.539 just like many people did. https://www.kaggle.com/dragonzhang/f0-25-yolox-pth
\nThen, I moved on to train my own yolox model with tuned hyperparameters and more epochs. I was able to get better Precision, Recall and mAP at the end of the training loop. However, when I submitted to the leaderboard, I got much lower results than the previous which used the checkpoint in the public dataset mentioned above.
\nLeaderboard score
\nPublic checkpoint - 0.539
\nMy checkpoint - 0.384
Public checkpoint
\nAverage forward time: 46.69 ms, Average NMS time: 3.08 ms, Average inference time: 49.77 ms
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.262
\n Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.547
\n Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.205
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.120
\n Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.300
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.153
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.318
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.318
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.160
\n Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.361
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
2021-12-18 09:57:42.160 | INFO | yolox.core.trainer:save_ckpt:318 - Save weights to ./YOLOX_outputs/cots_config
\n2021-12-18 09:57:43.408 | INFO | yolox.core.trainer:after_train:184 - Training of experiment is done and the best AP is 47.58
My checkpoint
\nAverage forward time: 46.02 ms, Average NMS time: 2.13 ms, Average inference time: 48.15 ms
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.586
\n Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.969
\n Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.655
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.479
\n Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.607
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.281
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.612
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.651
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.563
\n Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.672
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.686
2022-01-02 06:45:52.360 | INFO | yolox.core.trainer:save_ckpt:318 - Save weights to ./YOLOX_outputs/cots_config
\n2022-01-02 06:45:54.322 | INFO | yolox.core.trainer:after_train:184 - Training of experiment is done and the best AP is 58.64
As you can see from the above, the best AP from my checkpoint is 58.64 comparing to 47.58 from the public checkpoint.
\nI understand we are using the average F2 score which is different to AP (Average Precision) as F2 puts more weight on recall. Nevertheless, I should be getting a better result if both of my precision and recall are better right?
\n![https://blog.paperspace.com/content/images/2020/09/Fig04-1.jpg]
\nThe inference notebook I used is very similar to this one:
\nhttps://www.kaggle.com/parapapapam/yolox-inference-tracking-on-cots-lb-0-539?kernelSessionId=83072655
And I change the checkpoint by updating the variable 'CHECKPOINT_FILE'.
\nThe only thing I can think of at the moment… is maybe I need to tune the thresholds as my model is different now? it shouldn't really make too big of difference tho…
\nconfthre = 0.35
\nnmsthre = 0.4
Any comments will be much appreciated!!
", "messages": [ { "id": 1636401, "postDate": "2022-01-02T20:43:23.817Z", "content": "Hi everyone,
\nThis is doing my head in… not sure if anyone else experienced this or someone may be able to give me some advice.
\nI used this model checkpoint in the public dataset to get score of 0.539 just like many people did. https://www.kaggle.com/dragonzhang/f0-25-yolox-pth
\nThen, I moved on to train my own yolox model with tuned hyperparameters and more epochs. I was able to get better Precision, Recall and mAP at the end of the training loop. However, when I submitted to the leaderboard, I got much lower results than the previous which used the checkpoint in the public dataset mentioned above.
\nLeaderboard score
\nPublic checkpoint - 0.539
\nMy checkpoint - 0.384
Public checkpoint
\nAverage forward time: 46.69 ms, Average NMS time: 3.08 ms, Average inference time: 49.77 ms
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.262
\n Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.547
\n Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.205
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.120
\n Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.300
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.153
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.318
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.318
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.160
\n Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.361
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
2021-12-18 09:57:42.160 | INFO | yolox.core.trainer:save_ckpt:318 - Save weights to ./YOLOX_outputs/cots_config
\n2021-12-18 09:57:43.408 | INFO | yolox.core.trainer:after_train:184 - Training of experiment is done and the best AP is 47.58
My checkpoint
\nAverage forward time: 46.02 ms, Average NMS time: 2.13 ms, Average inference time: 48.15 ms
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.586
\n Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.969
\n Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.655
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.479
\n Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.607
\n Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.281
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.612
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.651
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.563
\n Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.672
\n Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.686
2022-01-02 06:45:52.360 | INFO | yolox.core.trainer:save_ckpt:318 - Save weights to ./YOLOX_outputs/cots_config
\n2022-01-02 06:45:54.322 | INFO | yolox.core.trainer:after_train:184 - Training of experiment is done and the best AP is 58.64
As you can see from the above, the best AP from my checkpoint is 58.64 comparing to 47.58 from the public checkpoint.
\nI understand we are using the average F2 score which is different to AP (Average Precision) as F2 puts more weight on recall. Nevertheless, I should be getting a better result if both of my precision and recall are better right?
\n![https://blog.paperspace.com/content/images/2020/09/Fig04-1.jpg]
\nThe inference notebook I used is very similar to this one:
\nhttps://www.kaggle.com/parapapapam/yolox-inference-tracking-on-cots-lb-0-539?kernelSessionId=83072655
And I change the checkpoint by updating the variable 'CHECKPOINT_FILE'.
\nThe only thing I can think of at the moment… is maybe I need to tune the thresholds as my model is different now? it shouldn't really make too big of difference tho…
\nconfthre = 0.35
\nnmsthre = 0.4
Any comments will be much appreciated!!
", "rawMarkdown": "Hi everyone, \n\nThis is doing my head in... not sure if anyone else experienced this or someone may be able to give me some advice. \n\nI used this model checkpoint in the public dataset to get score of 0.539 just like many people did. [https://www.kaggle.com/dragonzhang/f0-25-yolox-pth](url)\n\nThen, I moved on to train my own yolox model with tuned hyperparameters and more epochs. I was able to get better Precision, Recall and mAP at the end of the training loop. However, when I submitted to the leaderboard, I got much lower results than the previous which used the checkpoint in the public dataset mentioned above. \n\nLeaderboard score\nPublic checkpoint - 0.539\nMy checkpoint - 0.384\n\n**Public checkpoint**\nAverage forward time: 46.69 ms, Average NMS time: 3.08 ms, Average inference time: 49.77 ms\n Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.262\n Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.547\n Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.205\n Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.120\n Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.300\n Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.153\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.318\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.318\n Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.160\n Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.361\n Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n\n2021-12-18 09:57:42.160 | INFO | yolox.core.trainer:save_ckpt:318 - Save weights to ./YOLOX_outputs/cots_config\n2021-12-18 09:57:43.408 | INFO | yolox.core.trainer:after_train:184 - Training of experiment is done and the best AP is **47.58**\n\n**My checkpoint**\nAverage forward time: 46.02 ms, Average NMS time: 2.13 ms, Average inference time: 48.15 ms\n Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.586\n Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.969\n Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.655\n Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.479\n Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.607\n Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.281\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.612\n Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.651\n Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.563\n Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.672\n Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.686\n\n2022-01-02 06:45:52.360 | INFO | yolox.core.trainer:save_ckpt:318 - Save weights to ./YOLOX_outputs/cots_config\n2022-01-02 06:45:54.322 | INFO | yolox.core.trainer:after_train:184 - Training of experiment is done and the best AP is **58.64**\n\nAs you can see from the above, the best AP from my checkpoint is 58.64 comparing to 47.58 from the public checkpoint. \n\nI understand we are using the average F2 score which is different to AP (Average Precision) as F2 puts more weight on recall. Nevertheless, I should be getting a better result if both of my precision and recall are better right?\n\n![https://blog.paperspace.com/content/images/2020/09/Fig04-1.jpg]\n\nThe inference notebook I used is very similar to this one:\n[https://www.kaggle.com/parapapapam/yolox-inference-tracking-on-cots-lb-0-539?kernelSessionId=83072655](url)\n\nAnd I change the checkpoint by updating the variable 'CHECKPOINT_FILE'. \n\nThe only thing I can think of at the moment... is maybe I need to tune the thresholds as my model is different now? it shouldn't really make too big of difference tho... \nconfthre = 0.35\nnmsthre = 0.4\n\nAny comments will be much appreciated!!", "votes": 26 }, { "id": 1638482, "postDate": "2022-01-04T18:56:28.563Z", "content": "i think there is going to be video_3, video_4 in the test (while video_0, video_1, video_2 are the given train). according to the COTS dataset paper, there are 5 locations for data collection. each location has about 6 to 8k images
\nyou may want to try these, e.g
\n(we are trying to measure domain difference between the different video id) pay attention if cross domain will increase number of FP and decrease the confidence score of target object.
\nif you are successful, you should be able to e.g. train on video_0 + video_1, and get good results when testing on video_2
", "rawMarkdown": "i think there is going to be video\\_3, video\\_4 in the test (while video\\_0, video\\_1, video\\_2 are the given train). according to the COTS dataset paper, there are 5 locations for data collection. each location has about 6 to 8k images\n\nyou may want to try these, e.g\n1. if I use train = subset of video\\_n only, test = of video\\_n only, what is the metric performance?\n2. if I use train = subset of video\\_n only, test = of video\\_m only, what is the metric performance?\n\n(we are trying to measure domain difference between the different video id) pay attention if cross domain will increase number of FP and decrease the confidence score of target object.\n\n3. what augmentation/tricks can I use so that my model is robust against the change of video id.\n\nif you are successful, you should be able to e.g. train on video\\_0 + video\\_1, and get good results when testing on video\\_2", "votes": 14, "replies": [ { "id": 1639147, "postDate": "2022-01-05T12:30:34.140Z", "content": "here are the experimental results
\n
effect of better augmentation !!!!
\n![\"\"](\"https://i.ibb.co/NFry0MS/Selection-999-592.png\")
Nice ablation experiments, what is the difference between your weak augmentation and the strong augmentation? I have tried some strong augmentation methods such as copy paste but only obtain a little improvement.
", "rawMarkdown": "Nice ablation experiments, what is the difference between your weak augmentation and the strong augmentation? I have tried some strong augmentation methods such as copy paste but only obtain a little improvement." }, { "id": 1639938, "postDate": "2022-01-06T04:31:09.947Z", "content": "Thank you @hengck23! Agree with you about videos 3 and 4 being in the test. To clarify may I ask if one uses the video-id as the train set, wouldn't that also decrease the number of training samples - hence the need for data augmentation? For me, that was an aversion to splitting like that, so I split using sequences. I am currently experimenting with ways to expand/augment the training dataset.
", "rawMarkdown": "Thank you @hengck23! Agree with you about videos 3 and 4 being in the test. To clarify may I ask if one uses the video-id as the train set, wouldn't that also decrease the number of training samples - hence the need for data augmentation? For me, that was an aversion to splitting like that, so I split using sequences. I am currently experimenting with ways to expand/augment the training dataset. " }, { "id": 1639954, "postDate": "2022-01-06T04:55:21.243Z", "content": "\"video-id as the train set, wouldn't that also decrease the number of training samples?\"
\nusing video id as validation split is for me to design augmentation methods. you can see that the initial CV results range from 0.37 to 0.51. My target is to design augmentation so that this difference is minimized, preferably all above 0.55 regardless of video id split (which I think is possible) and the difference between LB and CV should be smaller than 0.04
\nafter that, maybe I will change to split by seq or other cv splitting strategy.
\nfrom my experiments, it seems that the public test data is close and smliar to some subset of video0,1,2.
\nthat is why some fold performs better than others.
But this results can be unreliable because I am not sure about the hidden private test set (i.e. if the hidden private test set is as similar, compared to the public, to the train or not)
", "rawMarkdown": "\"video-id as the train set, wouldn't that also decrease the number of training samples?\"\n\nusing video id as validation split is for me to design augmentation methods. you can see that the initial CV results range from 0.37 to 0.51. My target is to design augmentation so that this difference is minimized, preferably all above 0.55 regardless of video id split (which I think is possible) and the difference between LB and CV should be smaller than 0.04\n\nafter that, maybe I will change to split by seq or other cv splitting strategy.\n\nfrom my experiments, it seems that the public test data is close and smliar to some subset of video0,1,2.\nthat is why some fold performs better than others.\n\nBut this results can be unreliable because I am not sure about the hidden private test set (i.e. if the hidden private test set is as similar, compared to the public, to the train or not)", "votes": 2 }, { "id": 1639964, "postDate": "2022-01-06T05:02:46.197Z", "content": "\"I have tried some strong augmentation methods such as copy paste but only obtain a little improvement.\"
\nyou should make a list of variations based on the observation of the images and understanding of data collection process.
\nfor example:
\n1. camera pose (is the camera directly above or at an angle from COTS object)\n- Augmentation: affine, perspective image transform\n\n2. camera distance from object\n- Augmentation: scale transform\n(how much scale to use depends on the object size distribution)\n\n3. deep of water\n- Augmentation: more bluish, whitish ..., light scattering, ..\n(hsv change and intensity, color shift)\n\n4. motion blur, focus blur\n\n5. context\n- COTs is on coral or on sands, etc\nAugmentation : cut and paste\n\n6. water quality\n- bubbles, small fishes, mirky ...\n\n7. occlusions\n\n8. underwater shadows\n\n9. density of COTS\n- occurs in groups. etc ...\n- relation between location and size of box\n\n10. pose of COTS (are their star legs spread out, etc ...)\n\n11. camera noise \n@hengck23 thank you. I never thought about approaching the data that way as a means to optimize the augmentations so I learned something valuable in your work. Very cool to know you are from NUS too!
", "rawMarkdown": "@hengck23 thank you. I never thought about approaching the data that way as a means to optimize the augmentations so I learned something valuable in your work. Very cool to know you are from NUS too! " }, { "id": 1639989, "postDate": "2022-01-06T05:21:51.430Z", "content": "signs that shows that augmentation is correct:
\nthere is strong evidence that there are many smaller COTS objects the public test set
", "rawMarkdown": "there is strong evidence that there are many smaller COTS objects the public test set", "votes": 1 }, { "id": 1640061, "postDate": "2022-01-06T06:33:11.363Z", "content": "Got it, thanks very much!
", "rawMarkdown": "Got it, thanks very much!" }, { "id": 1640443, "postDate": "2022-01-06T13:43:02Z", "content": "update of results
\n![\"\"](\"https://i.ibb.co/dm5cQCw/Selection-999-593.png\")
@wilbertbhtan Did you use only images with labels in your training or add some without labels ?
", "rawMarkdown": "@wilbertbhtan Did you use only images with labels in your training or add some without labels ?" }, { "id": 1640581, "postDate": "2022-01-06T15:41:39.170Z", "content": "@garvitgarg I used 1% of the training of set with the images without annotations as per Yolov5 documentation. I imagine can add more background images (<10%) to reduce false positives.
\nBackground images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total). No labels are required for background images.
From Yolov5
", "rawMarkdown": "@garvitgarg I used 1% of the training of set with the images without annotations as per Yolov5 documentation. I imagine can add more background images (<10%) to reduce false positives. \n\n`Background images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total). No labels are required for background images.`\n\nFrom [Yolov5 ](https://github.com/ultralytics/yolov5/wiki/Tips-for-Best-Training-Results)", "votes": 1 }, { "id": 1640637, "postDate": "2022-01-06T16:53:35.757Z", "content": "typical miss (i.e. what to augment)
\n![\"\"](\"https://i.ibb.co/F8tP6mf/Selection-999-617.png\")
it is important is ask yourself this question: \"now I have a miss detection, should I handle it or just ignore it?\"
\nanother update:
\nthis will be the final update of the results. i would like to conclude:
\n![\"\"](\"https://i.ibb.co/6yhCsgX/Selection-999-619.png\")
the augmentation used to produce the above results is:
\ndef train_augment6(image, target):\n image, target = do_random_flip(image, target) # hflip, vflip or both\n #image, target = do_random_hflip(image, target)\n\n if np.random.rand() < 0.7:\n for func in np.random.choice([\n lambda image, target: do_random_perspective(image, target, m=0.3),\n lambda image, target: do_random_zoom_small(image, target, m=20), #aug min object size is 20\n lambda image, target: do_random_zoom_big(image, target, m=120), #aug max object size is 120\n lambda image, target: do_random_rotate(image, target, m=30), #up to 30 degree\n ], 1):\n image, target = func(image, target)\n pass\n pass\n\n if np.random.rand() < 0.7:\n for func in np.random.choice([\n lambda image: do_random_hsv(image, h=20, s=50, v=50),\n lambda image: do_random_guassian_blur(image, k=[3, 5], s=[0.1, 2.0]),\n lambda image: do_random_noise(image, m=0.08),\n ], 1):\n image = func(image)\n pass\n\n return image, target\nmore augmentations (e.g. mosaic, cut and paste, light scattering, cutout inside bbox, …) will further improve results but are not yet used and shown here
", "rawMarkdown": "another update:\n\nthis will be the final update of the results. i would like to conclude:\n- augmentation is the key to CV/LB gap\n- better split would help, but just by using split by video id could already get the public kernel results\n- with better and more augmentation, bigger models benefits\n\n\n\nthe augmentation used to produce the above results is:\n\n```\n\ndef train_augment6(image, target):\n image, target = do_random_flip(image, target) # hflip, vflip or both\n #image, target = do_random_hflip(image, target)\n\n if np.random.rand() < 0.7:\n for func in np.random.choice([\n lambda image, target: do_random_perspective(image, target, m=0.3),\n lambda image, target: do_random_zoom_small(image, target, m=20), #aug min object size is 20\n lambda image, target: do_random_zoom_big(image, target, m=120), #aug max object size is 120\n lambda image, target: do_random_rotate(image, target, m=30), #up to 30 degree\n ], 1):\n image, target = func(image, target)\n pass\n pass\n\n if np.random.rand() < 0.7:\n for func in np.random.choice([\n lambda image: do_random_hsv(image, h=20, s=50, v=50),\n lambda image: do_random_guassian_blur(image, k=[3, 5], s=[0.1, 2.0]),\n lambda image: do_random_noise(image, m=0.08),\n ], 1):\n image = func(image)\n pass\n\n return image, target\n\n\n```\n\nmore augmentations (e.g. mosaic, cut and paste, light scattering, cutout inside bbox, ...) will further improve results but are not yet used and shown here", "votes": 12 }, { "id": 1640943, "postDate": "2022-01-07T01:31:15.540Z", "content": "@hengck23 Thank you for sharing a lot of information. I gained much insight out of this discussion.
\nI have a question about your augmentation. Can I ask what do_random_zoom_small and do_random_zoom_big doing? Is this scaling entire image or scaling just for bounding box area?
this is due to the difference in the test and validation data. this is what you can do:
\nthe threshold should be selected to give a good FP per 100 images (note that I use per 100 images, meaning that you need many more images in validation to give a good statistics)
\nthere are 13000 test images, use your train data to estimate num of empty, non-empty images in the test. Also, the distribution of the number of targets per image in test. Then ask yourself: \" do I have enough images in the validation to get a reliable statistics?\"
", "rawMarkdown": "this is due to the difference in the test and validation data. this is what you can do:\n1. try different fold of train and validation set\n2. if you get good validation results by lowering the confidence threshold, e.g. 0.1, it may be likely that test results can be worse. because a low threshold should lead many more FP (it may not happen in your validation set but it can happen to another set)\n\nthe threshold should be selected to give a good FP per 100 images (note that I use per 100 images, meaning that you need many more images in validation to give a good statistics)\n\nthere are 13000 test images, use your train data to estimate num of empty, non-empty images in the test. Also, the distribution of the number of targets per image in test. Then ask yourself: \" do I have enough images in the validation to get a reliable statistics?\"", "votes": 9, "replies": [ { "id": 1636884, "postDate": "2022-01-03T11:01:57.333Z", "content": "tip: in theory, you can reduce fp by training with any images (e.g. imagenet, cityscape, … just do a bootstrapping to collect hard negative)
\nfor me, I am downloading underwater, diving video from youtube, etc
", "rawMarkdown": "tip: in theory, you can reduce fp by training with any images (e.g. imagenet, cityscape, ... just do a bootstrapping to collect hard negative)\n\nfor me, I am downloading underwater, diving video from youtube, etc", "votes": 3 }, { "id": 1641597, "postDate": "2022-01-07T15:13:26.130Z", "content": "Hello, I would like to ask when calculating the F2 score for the validation set, do we have to ensure that the percentage of unannotated images in the validation set is equal to the percentage of unannotated images in the entire dataset(i.e., 4919/23501)?
@shengzheliu F2 score calculated only for the positive dataset is typically higher than that for the entire dataset because of the FPs on the negative dataset. So it tends to cause more CV/LB gaps. To check if you model correctly predicts on the back ground, validating on entire (positive and negative) dataset is necessary.
\nOne point we have to care is, we can’t know the positive-negative ratio of public and private dataset.
\nTherefore, the hyper parameter (like confidence threshold) optimized on your validation data doesn’t necessarily optimal on the test data. E.g. suppose test data has much less annotated data than your validation data and the optimal confidence threshold to your validation set is 0.3. Since test set contain more background data than validation set, your model tend to predict more FPs on test set than validation set. Therefore, the optimal confidence threshold for the test set is typically less than that for validation set.
For that reason, setting the positive negative ratio equal to train dataset doesn’t necessarily cause optimal performance on the test set. I think we have to design robust model to any positive-negative ratio.
", "rawMarkdown": "@shengzheliu F2 score calculated only for the positive dataset is typically higher than that for the entire dataset because of the FPs on the negative dataset. So it tends to cause more CV/LB gaps. To check if you model correctly predicts on the back ground, validating on entire (positive and negative) dataset is necessary.\n\nOne point we have to care is, we can’t know the positive-negative ratio of public and private dataset.\nTherefore, the hyper parameter (like confidence threshold) optimized on your validation data doesn’t necessarily optimal on the test data. E.g. suppose test data has much less annotated data than your validation data and the optimal confidence threshold to your validation set is 0.3. Since test set contain more background data than validation set, your model tend to predict more FPs on test set than validation set. Therefore, the optimal confidence threshold for the test set is typically less than that for validation set.\n\nFor that reason, setting the positive negative ratio equal to train dataset doesn’t necessarily cause optimal performance on the test set. I think we have to design robust model to any positive-negative ratio.", "votes": 1 }, { "id": 1646182, "postDate": "2022-01-11T14:24:00.860Z", "content": "@hengck23 Thanks for sharing many helpful tips. Did you trained unlabeled data too? In my case with Yolov5l, object loss is keep increasing and recall is getting lower. If there any tip to increase true positive rate and reduce false negative rate. Thanks
", "rawMarkdown": "@hengck23 Thanks for sharing many helpful tips. Did you trained unlabeled data too? In my case with Yolov5l, object loss is keep increasing and recall is getting lower. If there any tip to increase true positive rate and reduce false negative rate. Thanks" } ] }, { "id": 1637390, "postDate": "2022-01-03T20:38:24.133Z", "content": "Think I have found the cause for my case guys. For me, it was due to the way I split the train/val dataset.
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/293723
\nThe discussion thread above is worth a read.
\nIn a summary, a few things could attribute to the big gap between val and LB score.
\nIf i'm correct, you trained from other's weight, instead of scratch?
", "rawMarkdown": "If i'm correct, you trained from other's weight, instead of scratch?", "replies": [ { "id": 1636805, "postDate": "2022-01-03T09:35:42.373Z", "content": "used weights pretrained from coco
", "rawMarkdown": "used weights pretrained from coco" }, { "id": 1637613, "postDate": "2022-01-04T04:25:58.770Z", "content": "Dose it matters?
", "rawMarkdown": "Dose it matters?" }, { "id": 1638062, "postDate": "2022-01-04T12:14:14.447Z", "content": "Maybe try from scratch
", "rawMarkdown": "Maybe try from scratch" } ] }, { "id": 1636741, "postDate": "2022-01-03T08:29:20.663Z", "content": "Similar observation here. Trained model with some augmentations and got mAP of 61. But my score was only 0.501. My current best single model with mAP = 42 scored 0.510.
", "rawMarkdown": "Similar observation here. Trained model with some augmentations and got mAP of 61. But my score was only 0.501. My current best single model with mAP = 42 scored 0.510." }, { "id": 1636582, "postDate": "2022-01-03T04:45:12.367Z", "content": "confthre may shift for optimal LB, but not than much difference.
", "rawMarkdown": "confthre may shift for optimal LB, but not than much difference." }, { "id": 1636581, "postDate": "2022-01-03T04:42:53.620Z", "content": "I met same problem. After I bought colab pro, I have more GPU available. several checkpoints seems better, but
\nLB just similar or worse.
I am observing similar results. I trained a yolov5l with some pre-processing and training modifications, higher mAP than public training notebooks using the same fold, but lb is only 0.389. Not really sure what is going on.
", "rawMarkdown": "I am observing similar results. I trained a yolov5l with some pre-processing and training modifications, higher mAP than public training notebooks using the same fold, but lb is only 0.389. Not really sure what is going on.", "replies": [ { "id": 1641852, "postDate": "2022-01-07T19:03:16.343Z", "content": "Hey I have been trying yolov5l too. I was able to get LB 0.425 but I am struggling to go above this. I got 0.416 without tracking and 0.425 with implementation of Norfair tracking.
\nTake a look at video generated this video is generated from 4th fold [Green bounding boxes are yolov5l predictions, Red bounding boxes are original annotations and Black bounding boxes are after tacking]
\nhttps://drive.google.com/file/d/11gp6IDav0X7fNxBYG3C0uL6qOZ6pj0eD/view?usp=sharing
i think there is going to be video_3, video_4 in the test (while video_0, video_1, video_2 are the given train). according to the COTS dataset paper, there are 5 locations for data collection. each location has about 6 to 8k images
\nyou may want to try these, e.g
\n(we are trying to measure domain difference between the different video id) pay attention if cross domain will increase number of FP and decrease the confidence score of target object.
\nif you are successful, you should be able to e.g. train on video_0 + video_1, and get good results when testing on video_2
", "votes": 14, "replies": [ { "id": 1639147, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-05T12:30:34.140000", "content": "here are the experimental results
\n
effect of better augmentation !!!!
\n
Nice ablation experiments, what is the difference between your weak augmentation and the strong augmentation? I have tried some strong augmentation methods such as copy paste but only obtain a little improvement.
", "votes": 0, "replies": [] }, { "id": 1639938, "author_name": "Wilbert Tan", "author_url": "", "post_date": "2022-01-06T04:31:09.947000", "content": "Thank you @hengck23! Agree with you about videos 3 and 4 being in the test. To clarify may I ask if one uses the video-id as the train set, wouldn't that also decrease the number of training samples - hence the need for data augmentation? For me, that was an aversion to splitting like that, so I split using sequences. I am currently experimenting with ways to expand/augment the training dataset.
", "votes": 0, "replies": [] }, { "id": 1639954, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-06T04:55:21.243000", "content": "\"video-id as the train set, wouldn't that also decrease the number of training samples?\"
\nusing video id as validation split is for me to design augmentation methods. you can see that the initial CV results range from 0.37 to 0.51. My target is to design augmentation so that this difference is minimized, preferably all above 0.55 regardless of video id split (which I think is possible) and the difference between LB and CV should be smaller than 0.04
\nafter that, maybe I will change to split by seq or other cv splitting strategy.
\nfrom my experiments, it seems that the public test data is close and smliar to some subset of video0,1,2.
\nthat is why some fold performs better than others.
But this results can be unreliable because I am not sure about the hidden private test set (i.e. if the hidden private test set is as similar, compared to the public, to the train or not)
", "votes": 2, "replies": [] }, { "id": 1639964, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-06T05:02:46.197000", "content": "\"I have tried some strong augmentation methods such as copy paste but only obtain a little improvement.\"
\nyou should make a list of variations based on the observation of the images and understanding of data collection process.
\nfor example:
\n1. camera pose (is the camera directly above or at an angle from COTS object)\n- Augmentation: affine, perspective image transform\n\n2. camera distance from object\n- Augmentation: scale transform\n(how much scale to use depends on the object size distribution)\n\n3. deep of water\n- Augmentation: more bluish, whitish ..., light scattering, ..\n(hsv change and intensity, color shift)\n\n4. motion blur, focus blur\n\n5. context\n- COTs is on coral or on sands, etc\nAugmentation : cut and paste\n\n6. water quality\n- bubbles, small fishes, mirky ...\n\n7. occlusions\n\n8. underwater shadows\n\n9. density of COTS\n- occurs in groups. etc ...\n- relation between location and size of box\n\n10. pose of COTS (are their star legs spread out, etc ...)\n\n11. camera noise \n@hengck23 thank you. I never thought about approaching the data that way as a means to optimize the augmentations so I learned something valuable in your work. Very cool to know you are from NUS too!
", "votes": 0, "replies": [] }, { "id": 1639989, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-06T05:21:51.430000", "content": "signs that shows that augmentation is correct:
\nthere is strong evidence that there are many smaller COTS objects the public test set
", "votes": 1, "replies": [] }, { "id": 1640061, "author_name": "zhengye", "author_url": "", "post_date": "2022-01-06T06:33:11.363000", "content": "Got it, thanks very much!
", "votes": 0, "replies": [] }, { "id": 1640443, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-06T13:43:02", "content": "update of results
\n
@wilbertbhtan Did you use only images with labels in your training or add some without labels ?
", "votes": 0, "replies": [] }, { "id": 1640581, "author_name": "Wilbert Tan", "author_url": "", "post_date": "2022-01-06T15:41:39.170000", "content": "@garvitgarg I used 1% of the training of set with the images without annotations as per Yolov5 documentation. I imagine can add more background images (<10%) to reduce false positives.
\nBackground images. Background images are images with no objects that are added to a dataset to reduce False Positives (FP). We recommend about 0-10% background images to help reduce FPs (COCO has 1000 background images for reference, 1% of the total). No labels are required for background images.
From Yolov5
", "votes": 1, "replies": [] }, { "id": 1640637, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-06T16:53:35.757000", "content": "typical miss (i.e. what to augment)
\n
it is important is ask yourself this question: \"now I have a miss detection, should I handle it or just ignore it?\"
\nanother update:
\nthis will be the final update of the results. i would like to conclude:
\n
the augmentation used to produce the above results is:
\ndef train_augment6(image, target):\n image, target = do_random_flip(image, target) # hflip, vflip or both\n #image, target = do_random_hflip(image, target)\n\n if np.random.rand() < 0.7:\n for func in np.random.choice([\n lambda image, target: do_random_perspective(image, target, m=0.3),\n lambda image, target: do_random_zoom_small(image, target, m=20), #aug min object size is 20\n lambda image, target: do_random_zoom_big(image, target, m=120), #aug max object size is 120\n lambda image, target: do_random_rotate(image, target, m=30), #up to 30 degree\n ], 1):\n image, target = func(image, target)\n pass\n pass\n\n if np.random.rand() < 0.7:\n for func in np.random.choice([\n lambda image: do_random_hsv(image, h=20, s=50, v=50),\n lambda image: do_random_guassian_blur(image, k=[3, 5], s=[0.1, 2.0]),\n lambda image: do_random_noise(image, m=0.08),\n ], 1):\n image = func(image)\n pass\n\n return image, target\nmore augmentations (e.g. mosaic, cut and paste, light scattering, cutout inside bbox, …) will further improve results but are not yet used and shown here
", "votes": 12, "replies": [] }, { "id": 1640943, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-07T01:31:15.540000", "content": "@hengck23 Thank you for sharing a lot of information. I gained much insight out of this discussion.
\nI have a question about your augmentation. Can I ask what do_random_zoom_small and do_random_zoom_big doing? Is this scaling entire image or scaling just for bounding box area?
this is due to the difference in the test and validation data. this is what you can do:
\nthe threshold should be selected to give a good FP per 100 images (note that I use per 100 images, meaning that you need many more images in validation to give a good statistics)
\nthere are 13000 test images, use your train data to estimate num of empty, non-empty images in the test. Also, the distribution of the number of targets per image in test. Then ask yourself: \" do I have enough images in the validation to get a reliable statistics?\"
", "votes": 9, "replies": [ { "id": 1636884, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-03T11:01:57.333000", "content": "tip: in theory, you can reduce fp by training with any images (e.g. imagenet, cityscape, … just do a bootstrapping to collect hard negative)
\nfor me, I am downloading underwater, diving video from youtube, etc
", "votes": 3, "replies": [] }, { "id": 1641597, "author_name": "ShengzheLiu", "author_url": "", "post_date": "2022-01-07T15:13:26.130000", "content": "Hello, I would like to ask when calculating the F2 score for the validation set, do we have to ensure that the percentage of unannotated images in the validation set is equal to the percentage of unannotated images in the entire dataset(i.e., 4919/23501)?
@shengzheliu F2 score calculated only for the positive dataset is typically higher than that for the entire dataset because of the FPs on the negative dataset. So it tends to cause more CV/LB gaps. To check if you model correctly predicts on the back ground, validating on entire (positive and negative) dataset is necessary.
\nOne point we have to care is, we can’t know the positive-negative ratio of public and private dataset.
\nTherefore, the hyper parameter (like confidence threshold) optimized on your validation data doesn’t necessarily optimal on the test data. E.g. suppose test data has much less annotated data than your validation data and the optimal confidence threshold to your validation set is 0.3. Since test set contain more background data than validation set, your model tend to predict more FPs on test set than validation set. Therefore, the optimal confidence threshold for the test set is typically less than that for validation set.
For that reason, setting the positive negative ratio equal to train dataset doesn’t necessarily cause optimal performance on the test set. I think we have to design robust model to any positive-negative ratio.
", "votes": 1, "replies": [] }, { "id": 1646182, "author_name": "Ichimaru Gin", "author_url": "", "post_date": "2022-01-11T14:24:00.860000", "content": "@hengck23 Thanks for sharing many helpful tips. Did you trained unlabeled data too? In my case with Yolov5l, object loss is keep increasing and recall is getting lower. If there any tip to increase true positive rate and reduce false negative rate. Thanks
", "votes": 0, "replies": [] } ] }, { "id": 1637390, "author_name": "Early Bird", "author_url": "", "post_date": "2022-01-03T20:38:24.133000", "content": "Think I have found the cause for my case guys. For me, it was due to the way I split the train/val dataset.
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/293723
\nThe discussion thread above is worth a read.
\nIn a summary, a few things could attribute to the big gap between val and LB score.
\nIf i'm correct, you trained from other's weight, instead of scratch?
", "votes": 0, "replies": [ { "id": 1636805, "author_name": "nymfree", "author_url": "", "post_date": "2022-01-03T09:35:42.373000", "content": "used weights pretrained from coco
", "votes": 0, "replies": [] }, { "id": 1637613, "author_name": "Kevin", "author_url": "", "post_date": "2022-01-04T04:25:58.770000", "content": "Dose it matters?
", "votes": 0, "replies": [] }, { "id": 1638062, "author_name": "Owen Xing", "author_url": "", "post_date": "2022-01-04T12:14:14.447000", "content": "Maybe try from scratch
", "votes": 0, "replies": [] } ] }, { "id": 1636741, "author_name": "nymfree", "author_url": "", "post_date": "2022-01-03T08:29:20.663000", "content": "Similar observation here. Trained model with some augmentations and got mAP of 61. But my score was only 0.501. My current best single model with mAP = 42 scored 0.510.
", "votes": 0, "replies": [] }, { "id": 1636582, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-03T04:45:12.367000", "content": "confthre may shift for optimal LB, but not than much difference.
", "votes": 0, "replies": [] }, { "id": 1636581, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-03T04:42:53.620000", "content": "I met same problem. After I bought colab pro, I have more GPU available. several checkpoints seems better, but
\nLB just similar or worse.
I am observing similar results. I trained a yolov5l with some pre-processing and training modifications, higher mAP than public training notebooks using the same fold, but lb is only 0.389. Not really sure what is going on.
", "votes": 0, "replies": [ { "id": 1641852, "author_name": "manas joshi", "author_url": "", "post_date": "2022-01-07T19:03:16.343000", "content": "Hey I have been trying yolov5l too. I was able to get LB 0.425 but I am struggling to go above this. I got 0.416 without tracking and 0.425 with implementation of Norfair tracking.
\nTake a look at video generated this video is generated from 4th fold [Green bounding boxes are yolov5l predictions, Red bounding boxes are original annotations and Black bounding boxes are after tacking]
\nhttps://drive.google.com/file/d/11gp6IDav0X7fNxBYG3C0uL6qOZ6pj0eD/view?usp=sharing