Share your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB.
", "messages": [ { "id": 129848, "postDate": "2016-08-02T14:59:48.080Z", "content": "@tetmin, @Duc Nguyen We trained googlenet V3 (no pre-training) on about (5 million), yes 5 million synthetically generated images from the original training set. To generate the synthetic images we split each image into right and left halves [240: 400] and randomly recombined right and left halves from the same class until we had 5 million images. We also added noise to some images (blur, Gaussian…e.t.c) all augmentation was done offline, fortunately we had the hardware to train the behemoth in reasonable time. 10 titan X gpu’s many thanks to zeng zeng.
", "rawMarkdown": "@tetmin, @Duc Nguyen We trained googlenet V3 (no pre-training) on about **(5 million),** yes 5 million synthetically generated images from the original training set. To generate the synthetic images we split each image into right and left halves [240: 400] and randomly recombined right and left halves from the same class until we had 5 million images. We also added noise to some images (blur, Gaussian…e.t.c) all augmentation was done offline, fortunately we had the hardware to train the behemoth in reasonable time. 10 titan X gpu’s many thanks to zeng zeng. ", "votes": 5 }, { "id": 129986, "postDate": "2016-08-03T07:54:04.893Z", "content": "[quote=Heng CherKeng;129948]
\n\n@bobutis\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!
\n\n[/quote]
\n\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver.
", "rawMarkdown": "[quote=Heng CherKeng;129948]\r\n\r\n@bobutis\r\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!\r\n\r\n[/quote]\r\n\r\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver. ", "votes": 1 }, { "id": 129948, "postDate": "2016-08-03T02:28:27.053Z", "content": "@bobutis\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!
\n\n[quote=bobutis;129829]
\n\nVGG-16 got 12.5 with dark knowledge, 17.5 without.
\n\n[/quote]
", "rawMarkdown": "@bobutis\r\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!\r\n\r\n[quote=bobutis;129829]\r\n\r\nVGG-16 got 12.5 with dark knowledge, 17.5 without.\r\n\r\n[/quote]\r\n", "votes": 1 }, { "id": 129868, "postDate": "2016-08-02T16:51:04.187Z", "content": "@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion.
", "rawMarkdown": "@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion. ", "votes": 1 }, { "id": 129857, "postDate": "2016-08-02T15:28:13.533Z", "content": "[quote=Guanshuo Xu;129844]
\n\nCan I know a little bit about your dark knowledge?\nappreciate
\n\n[/quote]
\n\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model.
", "rawMarkdown": "[quote=Guanshuo Xu;129844]\r\n\r\nCan I know a little bit about your dark knowledge?\r\nappreciate\r\n\r\n[/quote]\r\n\r\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model. \r\n\r\n", "votes": 1 }, { "id": 129844, "postDate": "2016-08-02T14:36:15.693Z", "content": "[quote=bobutis;129829]
\n\nVGG-16 got 12.5 with dark knowledge.
\n\n[/quote]
\n\nCan I know a little bit about your dark knowledge?\nappreciate
", "rawMarkdown": "[quote=bobutis;129829]\r\n\r\nVGG-16 got 12.5 with dark knowledge.\r\n\r\n[/quote]\r\n\r\nCan I know a little bit about your dark knowledge?\r\nappreciate", "votes": 1 }, { "id": 129817, "postDate": "2016-08-02T13:09:25.327Z", "content": "Thanks for sharing!\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.
", "rawMarkdown": "Thanks for sharing!\r\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.", "votes": 1 }, { "id": 129796, "postDate": "2016-08-02T11:25:01.110Z", "content": "DavidGbodiOdaibo, thanks for sharing your results.\nI mainly worked with VGGnet 16. The best single model I trained achieves ~0.29 on public LB. \nI also tried googlenet once but only got ~0.5 on public LB.\nCan you describe in more details how you trained your googlenetv3 model?
\n\n[quote=DavidGbodiOdaibo;129792]
\n\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB.
\n\n[/quote]
", "rawMarkdown": "DavidGbodiOdaibo, thanks for sharing your results.\r\nI mainly worked with VGGnet 16. The best single model I trained achieves ~0.29 on public LB. \r\nI also tried googlenet once but only got ~0.5 on public LB.\r\nCan you describe in more details how you trained your googlenetv3 model? \r\n\r\n[quote=DavidGbodiOdaibo;129792]\r\n\r\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB. \r\n\r\n[/quote]\r\n", "votes": 1 }, { "id": 129862, "postDate": "2016-08-02T16:10:50.440Z", "content": "@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge, however, these were not trained on the 5 million augmented data they were trained on the original training set with standard flip, shift , rotation augmentation.
", "rawMarkdown": "@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \r\n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge, however, these were not trained on the 5 million augmented data they were trained on the original training set with standard flip, shift , rotation augmentation.\r\n\r\n", "votes": 2 }, { "id": 129792, "postDate": "2016-08-02T11:09:39.020Z", "content": "Share your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB.
", "rawMarkdown": "Share your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB. ", "votes": 2 }, { "id": 130140, "postDate": "2016-08-04T10:00:36.003Z", "content": "@bobutis, this is really an interesting way to use FRCNN, from my knowlege of FRCNN, the classification accuary may benifit the image augmentation in rpn and nms between image patches.
", "rawMarkdown": "@bobutis, this is really an interesting way to use FRCNN, from my knowlege of FRCNN, the classification accuary may benifit the image augmentation in rpn and nms between image patches." }, { "id": 130073, "postDate": "2016-08-03T16:14:52.297Z", "content": "[quote=shenzhenwei;130072]
\n\n@bobutis\nThanks for sharing. Did you annotate bbox before the faster rcnn?
\n\n[/quote]
\n\nI did try learning on annotated boxes for the drivers (one box for every driver/class pair, in total 260 different possible boxes), however, I didn't see any improvement, so mostly I tried "detecting" the whole image.
", "rawMarkdown": "[quote=shenzhenwei;130072]\r\n\r\n@bobutis\r\nThanks for sharing. Did you annotate bbox before the faster rcnn? \r\n\r\n[/quote]\r\n\r\nI did try learning on annotated boxes for the drivers (one box for every driver/class pair, in total 260 different possible boxes), however, I didn't see any improvement, so mostly I tried \"detecting\" the whole image. " }, { "id": 130061, "postDate": "2016-08-03T14:40:13.460Z", "content": "A single wide ResNet using pseudo labels generated from finetuned VGG got me 0.174 / 0.184 (public/private).
", "rawMarkdown": "A single wide ResNet using pseudo labels generated from finetuned VGG got me 0.174 / 0.184 (public/private). " }, { "id": 130019, "postDate": "2016-08-03T11:22:31.330Z", "content": "[quote=AntonMaltsev;129817]
\n\nThanks for sharing!\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.
\n\n[/quote]
\n\nThanks for sharing!\nI have fine-tuned ResNet-100 using modified code from here( to remove horizontal flip for augmentation). I never could get more than 0.3 LB.
\n\n@AntonMaltsev: It'd be great if you can share more details about your training. Even better if you can share your code.
", "rawMarkdown": "[quote=AntonMaltsev;129817]\r\n\r\nThanks for sharing!\r\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.\r\n\r\n[/quote]\r\n\r\nThanks for sharing!\r\nI have fine-tuned ResNet-100 using modified code from [here](https://github.com/facebook/fb.resnet.torch)( to remove horizontal flip for augmentation). I never could get more than 0.3 LB. \r\n\r\n@AntonMaltsev: It'd be great if you can share more details about your training. Even better if you can share your code. " }, { "id": 129990, "postDate": "2016-08-03T08:13:16.993Z", "content": "@bobutis Thank you very much. Very good method!
\n\n[quote=bobutis;129986]
\n\n[quote=Heng CherKeng;129948]
\n\n@bobutis\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!
\n\n[/quote]
\n\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver.
\n\n[/quote]
", "rawMarkdown": "@bobutis Thank you very much. Very good method! \r\n\r\n[quote=bobutis;129986]\r\n\r\n[quote=Heng CherKeng;129948]\r\n\r\n@bobutis\r\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!\r\n\r\n[/quote]\r\n\r\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver. \r\n\r\n[/quote]\r\n" }, { "id": 129876, "postDate": "2016-08-02T17:30:58.597Z", "content": "[quote=DavidGbodiOdaibo;129875]
\n\n[quote=Guanshuo Xu;129871]
\n\nI don't think the first point you mentioned is true because your "break and combine" is not selective.
\n\n[/quote]
\n\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.
\n\n[/quote]
\n\nOh, you are right, now I see your point. I should think harder
", "rawMarkdown": "[quote=DavidGbodiOdaibo;129875]\r\n\r\n[quote=Guanshuo Xu;129871]\r\n\r\n I don't think the first point you mentioned is true because your \"break and combine\" is not selective.\r\n\r\n[/quote]\r\n\r\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.\r\n\r\n[/quote]\r\n\r\nOh, you are right, now I see your point. I should think harder\r\n" }, { "id": 129875, "postDate": "2016-08-02T17:17:31.457Z", "content": "[quote=Guanshuo Xu;129871]
\n\nI don't think the first point you mentioned is true because your "break and combine" is not selective.
\n\n[/quote]
\n\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.
", "rawMarkdown": "[quote=Guanshuo Xu;129871]\r\n\r\n I don't think the first point you mentioned is true because your \"break and combine\" is not selective.\r\n\r\n[/quote]\r\n\r\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these." }, { "id": 129871, "postDate": "2016-08-02T17:10:16.790Z", "content": "[quote=DavidGbodiOdaibo;129868]
\n\n@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion.
\n\n[/quote]
\n\nI'm also thinking of why this approach worked. Your mention of "various skin tones" makes a lot of sense to me. This augmentation makes the exact drivers in the photos noisy, preventing (making harder) the network from focusing on the exact drivers from training set, in the meantime, it generally preserves what the networks should really focus on. I believe this type of augmentation (as least the philosophy behind) as well as what bobutis mentioned could be generalized to other computer vision tasks. BTW, I don't think the first point you mentioned is true because your "break and combine" is not selective.
", "rawMarkdown": "[quote=DavidGbodiOdaibo;129868]\r\n\r\n@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion. \r\n\r\n[/quote]\r\n\r\nI'm also thinking of why this approach worked. Your mention of \"various skin tones\" makes a lot of sense to me. This augmentation makes the exact drivers in the photos noisy, preventing (making harder) the network from focusing on the exact drivers from training set, in the meantime, it generally preserves what the networks should really focus on. I believe this type of augmentation (as least the philosophy behind) as well as what bobutis mentioned could be generalized to other computer vision tasks. BTW, I don't think the first point you mentioned is true because your \"break and combine\" is not selective." }, { "id": 129864, "postDate": "2016-08-02T16:24:02.450Z", "content": "[quote=DavidGbodiOdaibo;129862]
\n\n@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge.
\n\n[/quote]
\n\nI have never saw such type of data augmentation. Apparently it works well in this competition. Thanks!
", "rawMarkdown": "[quote=DavidGbodiOdaibo;129862]\r\n\r\n@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \r\n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge.\r\n\r\n\r\n\r\n[/quote]\r\n\r\nI have never saw such type of data augmentation. Apparently it works well in this competition. Thanks! \r\n" }, { "id": 129861, "postDate": "2016-08-02T15:55:16.800Z", "content": "@ DavidGbodiOdaibo I just noticed that you trained inception v3 from scratch ... That's cool. Did you compare it with a pre-trained inception v3? I am also confused with your "breaking into halves and recombine strategy", what does [240: 400] mean? Thanks
", "rawMarkdown": "@ DavidGbodiOdaibo I just noticed that you trained inception v3 from scratch ... That's cool. Did you compare it with a pre-trained inception v3? I am also confused with your \"breaking into halves and recombine strategy\", what does [240: 400] mean? Thanks" }, { "id": 129858, "postDate": "2016-08-02T15:38:25.127Z", "content": "[quote=bobutis;129857]
\n\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model.
\n\n[/quote]
\n\nThis is a smart way to incorporate the driver information in test set to training process. It can even be made recursive. Learned a lot, thanks
", "rawMarkdown": "[quote=bobutis;129857]\r\n\r\n\r\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model. \r\n\r\n\r\n\r\n[/quote]\r\n\r\nThis is a smart way to incorporate the driver information in test set to training process. It can even be made recursive. Learned a lot, thanks\r\n" }, { "id": 129834, "postDate": "2016-08-02T14:15:57.230Z", "content": "[quote=DavidGbodiOdaibo;129792]
\n\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB.
\n\n[/quote]
\n\nFor googlenetv3 did you just use the bottlenecks and train the classifier or fine-tune the entire network out of interest?
", "rawMarkdown": "[quote=DavidGbodiOdaibo;129792]\r\n\r\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB. \r\n\r\n[/quote]\r\n\r\nFor googlenetv3 did you just use the bottlenecks and train the classifier or fine-tune the entire network out of interest?" }, { "id": 129829, "postDate": "2016-08-02T13:46:15.767Z", "content": "VGG-16 got 12.5 with dark knowledge, 17.5 without.
", "rawMarkdown": "VGG-16 got 12.5 with dark knowledge, 17.5 without." }, { "id": 130113, "postDate": "2016-08-04T00:33:24.233Z", "rawMarkdown": "", "isDeleted": true }, { "id": 130072, "postDate": "2016-08-03T15:42:40.463Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 129848, "author_name": "DavidGbodiOdaibo", "author_url": "", "post_date": "2016-08-02T14:59:48.080000", "content": "@tetmin, @Duc Nguyen We trained googlenet V3 (no pre-training) on about (5 million), yes 5 million synthetically generated images from the original training set. To generate the synthetic images we split each image into right and left halves [240: 400] and randomly recombined right and left halves from the same class until we had 5 million images. We also added noise to some images (blur, Gaussian…e.t.c) all augmentation was done offline, fortunately we had the hardware to train the behemoth in reasonable time. 10 titan X gpu’s many thanks to zeng zeng.
", "votes": 5, "replies": [] }, { "id": 129986, "author_name": "bobutis", "author_url": "", "post_date": "2016-08-03T07:54:04.893000", "content": "[quote=Heng CherKeng;129948]
\n\n@bobutis\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!
\n\n[/quote]
\n\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver.
", "votes": 1, "replies": [] }, { "id": 129948, "author_name": "hengck23", "author_url": "", "post_date": "2016-08-03T02:28:27.053000", "content": "@bobutis\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!
\n\n[quote=bobutis;129829]
\n\nVGG-16 got 12.5 with dark knowledge, 17.5 without.
\n\n[/quote]
", "votes": 1, "replies": [] }, { "id": 129868, "author_name": "DavidGbodiOdaibo", "author_url": "", "post_date": "2016-08-02T16:51:04.187000", "content": "@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion.
", "votes": 1, "replies": [] }, { "id": 129857, "author_name": "bobutis", "author_url": "", "post_date": "2016-08-02T15:28:13.533000", "content": "[quote=Guanshuo Xu;129844]
\n\nCan I know a little bit about your dark knowledge?\nappreciate
\n\n[/quote]
\n\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model.
", "votes": 1, "replies": [] }, { "id": 129844, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2016-08-02T14:36:15.693000", "content": "[quote=bobutis;129829]
\n\nVGG-16 got 12.5 with dark knowledge.
\n\n[/quote]
\n\nCan I know a little bit about your dark knowledge?\nappreciate
", "votes": 1, "replies": [] }, { "id": 129817, "author_name": "AntonMaltsev", "author_url": "", "post_date": "2016-08-02T13:09:25.327000", "content": "Thanks for sharing!\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.
", "votes": 1, "replies": [] }, { "id": 129796, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2016-08-02T11:25:01.110000", "content": "DavidGbodiOdaibo, thanks for sharing your results.\nI mainly worked with VGGnet 16. The best single model I trained achieves ~0.29 on public LB. \nI also tried googlenet once but only got ~0.5 on public LB.\nCan you describe in more details how you trained your googlenetv3 model?
\n\n[quote=DavidGbodiOdaibo;129792]
\n\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB.
\n\n[/quote]
", "votes": 1, "replies": [] }, { "id": 129862, "author_name": "DavidGbodiOdaibo", "author_url": "", "post_date": "2016-08-02T16:10:50.440000", "content": "@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge, however, these were not trained on the 5 million augmented data they were trained on the original training set with standard flip, shift , rotation augmentation.
", "votes": 2, "replies": [] }, { "id": 130140, "author_name": "Argmen", "author_url": "", "post_date": "2016-08-04T10:00:36.003000", "content": "@bobutis, this is really an interesting way to use FRCNN, from my knowlege of FRCNN, the classification accuary may benifit the image augmentation in rpn and nms between image patches.
", "votes": 0, "replies": [] }, { "id": 130073, "author_name": "bobutis", "author_url": "", "post_date": "2016-08-03T16:14:52.297000", "content": "[quote=shenzhenwei;130072]
\n\n@bobutis\nThanks for sharing. Did you annotate bbox before the faster rcnn?
\n\n[/quote]
\n\nI did try learning on annotated boxes for the drivers (one box for every driver/class pair, in total 260 different possible boxes), however, I didn't see any improvement, so mostly I tried "detecting" the whole image.
", "votes": 0, "replies": [] }, { "id": 130061, "author_name": "Florian Muellerklein", "author_url": "", "post_date": "2016-08-03T14:40:13.460000", "content": "A single wide ResNet using pseudo labels generated from finetuned VGG got me 0.174 / 0.184 (public/private).
", "votes": 0, "replies": [] }, { "id": 130019, "author_name": "chsasank", "author_url": "", "post_date": "2016-08-03T11:22:31.330000", "content": "[quote=AntonMaltsev;129817]
\n\nThanks for sharing!\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.
\n\n[/quote]
\n\nThanks for sharing!\nI have fine-tuned ResNet-100 using modified code from here( to remove horizontal flip for augmentation). I never could get more than 0.3 LB.
\n\n@AntonMaltsev: It'd be great if you can share more details about your training. Even better if you can share your code.
", "votes": 0, "replies": [] }, { "id": 129990, "author_name": "hengck23", "author_url": "", "post_date": "2016-08-03T08:13:16.993000", "content": "@bobutis Thank you very much. Very good method!
\n\n[quote=bobutis;129986]
\n\n[quote=Heng CherKeng;129948]
\n\n@bobutis\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!
\n\n[/quote]
\n\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver.
\n\n[/quote]
", "votes": 0, "replies": [] }, { "id": 129876, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2016-08-02T17:30:58.597000", "content": "[quote=DavidGbodiOdaibo;129875]
\n\n[quote=Guanshuo Xu;129871]
\n\nI don't think the first point you mentioned is true because your "break and combine" is not selective.
\n\n[/quote]
\n\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.
\n\n[/quote]
\n\nOh, you are right, now I see your point. I should think harder
", "votes": 0, "replies": [] }, { "id": 129875, "author_name": "DavidGbodiOdaibo", "author_url": "", "post_date": "2016-08-02T17:17:31.457000", "content": "[quote=Guanshuo Xu;129871]
\n\nI don't think the first point you mentioned is true because your "break and combine" is not selective.
\n\n[/quote]
\n\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.
", "votes": 0, "replies": [] }, { "id": 129871, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2016-08-02T17:10:16.790000", "content": "[quote=DavidGbodiOdaibo;129868]
\n\n@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion.
\n\n[/quote]
\n\nI'm also thinking of why this approach worked. Your mention of "various skin tones" makes a lot of sense to me. This augmentation makes the exact drivers in the photos noisy, preventing (making harder) the network from focusing on the exact drivers from training set, in the meantime, it generally preserves what the networks should really focus on. I believe this type of augmentation (as least the philosophy behind) as well as what bobutis mentioned could be generalized to other computer vision tasks. BTW, I don't think the first point you mentioned is true because your "break and combine" is not selective.
", "votes": 0, "replies": [] }, { "id": 129864, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2016-08-02T16:24:02.450000", "content": "[quote=DavidGbodiOdaibo;129862]
\n\n@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge.
\n\n[/quote]
\n\nI have never saw such type of data augmentation. Apparently it works well in this competition. Thanks!
", "votes": 0, "replies": [] }, { "id": 129861, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2016-08-02T15:55:16.800000", "content": "@ DavidGbodiOdaibo I just noticed that you trained inception v3 from scratch ... That's cool. Did you compare it with a pre-trained inception v3? I am also confused with your "breaking into halves and recombine strategy", what does [240: 400] mean? Thanks
", "votes": 0, "replies": [] }, { "id": 129858, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2016-08-02T15:38:25.127000", "content": "[quote=bobutis;129857]
\n\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model.
\n\n[/quote]
\n\nThis is a smart way to incorporate the driver information in test set to training process. It can even be made recursive. Learned a lot, thanks
", "votes": 0, "replies": [] }, { "id": 129834, "author_name": "tetmin", "author_url": "", "post_date": "2016-08-02T14:15:57.230000", "content": "[quote=DavidGbodiOdaibo;129792]
\n\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB.
\n\n[/quote]
\n\nFor googlenetv3 did you just use the bottlenecks and train the classifier or fine-tune the entire network out of interest?
", "votes": 0, "replies": [] }, { "id": 129829, "author_name": "bobutis", "author_url": "", "post_date": "2016-08-02T13:46:15.767000", "content": "VGG-16 got 12.5 with dark knowledge, 17.5 without.
", "votes": 0, "replies": [] }, { "id": 130113, "author_name": "", "author_url": "", "post_date": "2016-08-04T00:33:24.233000", "content": "", "votes": 0, "replies": [] }, { "id": 130072, "author_name": "", "author_url": "", "post_date": "2016-08-03T15:42:40.463000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "129848": "@tetmin, @Duc Nguyen We trained googlenet V3 (no pre-training) on about **(5 million),** yes 5 million synthetically generated images from the original training set. To generate the synthetic images we split each image into right and left halves [240: 400] and randomly recombined right and left halves from the same class until we had 5 million images. We also added noise to some images (blur, Gaussian…e.t.c) all augmentation was done offline, fortunately we had the hardware to train the behemoth in reasonable time. 10 titan X gpu’s many thanks to zeng zeng. ", "129986": "[quote=Heng CherKeng;129948]\r\n\r\n@bobutis\r\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!\r\n\r\n[/quote]\r\n\r\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver. ", "129948": "@bobutis\r\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!\r\n\r\n[quote=bobutis;129829]\r\n\r\nVGG-16 got 12.5 with dark knowledge, 17.5 without.\r\n\r\n[/quote]\r\n", "129868": "@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion. ", "129857": "[quote=Guanshuo Xu;129844]\r\n\r\nCan I know a little bit about your dark knowledge?\r\nappreciate\r\n\r\n[/quote]\r\n\r\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model. \r\n\r\n", "129844": "[quote=bobutis;129829]\r\n\r\nVGG-16 got 12.5 with dark knowledge.\r\n\r\n[/quote]\r\n\r\nCan I know a little bit about your dark knowledge?\r\nappreciate", "129817": "Thanks for sharing!\r\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.", "129796": "DavidGbodiOdaibo, thanks for sharing your results.\r\nI mainly worked with VGGnet 16. The best single model I trained achieves ~0.29 on public LB. \r\nI also tried googlenet once but only got ~0.5 on public LB.\r\nCan you describe in more details how you trained your googlenetv3 model? \r\n\r\n[quote=DavidGbodiOdaibo;129792]\r\n\r\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB. \r\n\r\n[/quote]\r\n", "129862": "@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \r\n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge, however, these were not trained on the 5 million augmented data they were trained on the original training set with standard flip, shift , rotation augmentation.\r\n\r\n", "129792": "Share your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB. ", "130140": "@bobutis, this is really an interesting way to use FRCNN, from my knowlege of FRCNN, the classification accuary may benifit the image augmentation in rpn and nms between image patches.", "130073": "[quote=shenzhenwei;130072]\r\n\r\n@bobutis\r\nThanks for sharing. Did you annotate bbox before the faster rcnn? \r\n\r\n[/quote]\r\n\r\nI did try learning on annotated boxes for the drivers (one box for every driver/class pair, in total 260 different possible boxes), however, I didn't see any improvement, so mostly I tried \"detecting\" the whole image. ", "130061": "A single wide ResNet using pseudo labels generated from finetuned VGG got me 0.174 / 0.184 (public/private). ", "130019": "[quote=AntonMaltsev;129817]\r\n\r\nThanks for sharing!\r\nOur single ResNet-100 give us about 0.21 on public LB (batch 12, agumentation). After semi-supervised learning it get us about 0.17.\r\n\r\n[/quote]\r\n\r\nThanks for sharing!\r\nI have fine-tuned ResNet-100 using modified code from [here](https://github.com/facebook/fb.resnet.torch)( to remove horizontal flip for augmentation). I never could get more than 0.3 LB. \r\n\r\n@AntonMaltsev: It'd be great if you can share more details about your training. Even better if you can share your code. ", "129990": "@bobutis Thank you very much. Very good method! \r\n\r\n[quote=bobutis;129986]\r\n\r\n[quote=Heng CherKeng;129948]\r\n\r\n@bobutis\r\nVGG16 for LB 0.175 is very good. can you give details on how to achieve that. Thanks!\r\n\r\n[/quote]\r\n\r\nI used Faster R-CNN with VGG-16 as its classifier. It is an object detection framework, however, it can be easily used as a multi-crop classifier (up to hundreds or even thousands of different crops while still maintaining efficiency), as it essentially classifies many different patches to perform object detection. Rest is nothing new, just some augmentation (blurring, rescaling) and looking for good Faster R-CNN parameters, as the default ones are suboptimal because they were tuned for an object detection task. For example, lowering the RPN_POSITIVE_OVERLAP threshold from 0.7 to 0.5 gave a large decrease in loss (~0.22 to 0.175) , which is kind of interesting as Regional Proposal Network is not even directly related to the task that we care about. However, it is indirectly related and my intuition for this is that the network learned to correctly classify patches even when they covered only a part (around 50%) of the driver. \r\n\r\n[/quote]\r\n", "129876": "[quote=DavidGbodiOdaibo;129875]\r\n\r\n[quote=Guanshuo Xu;129871]\r\n\r\n I don't think the first point you mentioned is true because your \"break and combine\" is not selective.\r\n\r\n[/quote]\r\n\r\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.\r\n\r\n[/quote]\r\n\r\nOh, you are right, now I see your point. I should think harder\r\n", "129875": "[quote=Guanshuo Xu;129871]\r\n\r\n I don't think the first point you mentioned is true because your \"break and combine\" is not selective.\r\n\r\n[/quote]\r\n\r\nYes, its is not selective, but it creates more training samples with the visor down, legs behind e.t.c to prevent over fitting to these.", "129871": "[quote=DavidGbodiOdaibo;129868]\r\n\r\n@Guanshuo Xu I think the visor, driver side mirror and legs of the passenger behind are major sources of error and this approach helps reduce the impact of those errors also the impact of the class imbalance because of various skin tones in the training set will be reduced as well is my suspicion. \r\n\r\n[/quote]\r\n\r\nI'm also thinking of why this approach worked. Your mention of \"various skin tones\" makes a lot of sense to me. This augmentation makes the exact drivers in the photos noisy, preventing (making harder) the network from focusing on the exact drivers from training set, in the meantime, it generally preserves what the networks should really focus on. I believe this type of augmentation (as least the philosophy behind) as well as what bobutis mentioned could be generalized to other computer vision tasks. BTW, I don't think the first point you mentioned is true because your \"break and combine\" is not selective.", "129864": "[quote=DavidGbodiOdaibo;129862]\r\n\r\n@ Guanshuo Xu, see the attached image, training pics have a width of 640px so we took \r\n0 -240px from pic1 and 240 - 640px from pic 2 and recombined to create new pic combining some parts from pic1 and the other from pic2. We compared with pre-trained our best pre-trained had ~0.31 without dark knowledge, and ~0.26 with dark knowledge.\r\n\r\n\r\n\r\n[/quote]\r\n\r\nI have never saw such type of data augmentation. Apparently it works well in this competition. Thanks! \r\n", "129861": "@ DavidGbodiOdaibo I just noticed that you trained inception v3 from scratch ... That's cool. Did you compare it with a pre-trained inception v3? I am also confused with your \"breaking into halves and recombine strategy\", what does [240: 400] mean? Thanks", "129858": "[quote=bobutis;129857]\r\n\r\n\r\nI used test set predictions generated by some 12ish submission to generate a random sample of 6000-12000 test set images (samples with more confident predictions were more likely to be sampled) with a random label (where probability to get a specific label was equal to the prediction). These randomly sampled randomly labeled test images were then fused with the training set and used to train a VGG-16 model. \r\n\r\n\r\n\r\n[/quote]\r\n\r\nThis is a smart way to incorporate the driver information in test set to training process. It can even be made recursive. Learned a lot, thanks\r\n", "129834": "[quote=DavidGbodiOdaibo;129792]\r\n\r\nShare your best single model score on public LB with no ensembling, if you used kfold averaging like jiao dong's ~0.23 script posted in the forum it does not count. We had a single googlenetv3 (thanks to our team mate zeng zeng) that achieved ~0.15 on public LB. \r\n\r\n[/quote]\r\n\r\nFor googlenetv3 did you just use the bottlenecks and train the classifier or fine-tune the entire network out of interest?", "129829": "VGG-16 got 12.5 with dark knowledge, 17.5 without.", "130113": "", "130072": "" } }