I am not familiar with this type of the train set and this number of target classes. I hope this thread will help to figure out what models perform better and which of them worth considering.
\n\nTo start:
\n\nResnet 101, no bagging, no TTA
\n\nI am not familiar with this type of the train set and this number of target classes. I hope this thread will help to figure out what models perform better and which of them worth considering.
\n\nTo start:
\n\nResnet 101, no bagging, no TTA
\n\nthanks a lot!
", "rawMarkdown": "thanks a lot!", "votes": null }, { "id": "233996", "postDate": "10/22/2017 01:48:17", "content": "That's quite high score for single model! How long did you train for this and which GPU did you use?
\n\nDue to limited resources, I'm using images scaled down to 90x90 and simple Xception, no TTA
\n\nval_acc = .6904 (for each image)
LB= .7092 (averaged softmax output for all images in same product id)
I'm trying many other approaches like using top 2000 frequent items, using category 1,2 information, etc, but nothing beats this baseline yet. Sad to see much better results with basic model, but thanks for the info.
", "rawMarkdown": "That's quite high score for single model! How long did you train for this and which GPU did you use?\n\nDue to limited resources, I'm using images scaled down to 90x90 and simple Xception, no TTA\n\n - val_acc = .6904 (for each image) \n\n - LB= .7092 (averaged softmax output for all images in same product id)\n\nI'm trying many other approaches like using top 2000 frequent items, using category 1,2 information, etc, but nothing beats this baseline yet. Sad to see much better results with basic model, but thanks for the info.", "votes": null }, { "id": "233997", "postDate": "10/22/2017 01:51:12", "content": "4 x GTX 1080 Ti
\n\n18 epochs x 4.5 hours
\n\nI have 0.69 after 7th epoch
", "rawMarkdown": "4 x GTX 1080 Ti\n\n18 epochs x 4.5 hours\n\nI have 0.69 after 7th epoch", "votes": null }, { "id": "234000", "postDate": "10/22/2017 01:58:43", "content": "seems that we need long epoches!
", "rawMarkdown": "seems that we need long epoches!", "votes": null }, { "id": "234005", "postDate": "10/22/2017 02:40:41", "content": "Amazing score. Did you do training without pretrained weight?
\n\nMy results: \ninception_v3, 160 crop, val loss: 1.78~, val acc: 0.66~, lb: 0.673~
", "rawMarkdown": "Amazing score. Did you do training without pretrained weight?\n\nMy results: \ninception_v3, 160 crop, val loss: 1.78~, val acc: 0.66~, lb: 0.673~", "votes": null }, { "id": "234006", "postDate": "10/22/2017 02:41:18", "content": "i am also using 4 x GTX 1080 Ti. For resnext101 (about same flops as your resnet101 ), 1 epoch=12283645 images takes about 1.3 hr. I am using 180x180 as input. I suppose your input size is not 180x180?
", "rawMarkdown": "i am also using 4 x GTX 1080 Ti. For resnext101 (about same flops as your resnet101 ), 1 epoch=12283645 images takes about 1.3 hr. I am using 180x180 as input. I suppose your input size is not 180x180?", "votes": null }, { "id": "234007", "postDate": "10/22/2017 02:43:55", "content": "what does TTA mean?
", "rawMarkdown": "what does TTA mean?", "votes": null }, { "id": "234010", "postDate": "10/22/2017 02:52:06", "content": "I use 160x160 crops.
\n\n1.3hrs per epoch.... It looks like I am doing something wrong.
\n\nIt may happen that my SSD is not fast enough and I need to buy faster one.
", "rawMarkdown": "I use 160x160 crops.\n\n1.3hrs per epoch.... It looks like I am doing something wrong.\n\nIt may happen that my SSD is not fast enough and I need to buy faster one.", "votes": null }, { "id": "234012", "postDate": "10/22/2017 02:53:57", "content": "TTA= test time augmentation (e.g. multi crops, resize in test time)
", "rawMarkdown": "TTA= test time augmentation (e.g. multi crops, resize in test time)", "votes": null }, { "id": "234013", "postDate": "10/22/2017 02:56:01", "content": "Test-time augmentation. In my case, I can get extra 0.01 just with 12-way TTA(flip, shift up/down, etc) with exact same weight - almost free cake :)
", "rawMarkdown": "Test-time augmentation. In my case, I can get extra 0.01 just with 12-way TTA(flip, shift up/down, etc) with exact same weight - almost free cake :)", "votes": null }, { "id": "234019", "postDate": "10/22/2017 03:31:19", "content": "I used pre-trained. I would be also interested how long will it take to get the same result without pre-trained, but it may be too computationally expensive.
", "rawMarkdown": "I used pre-trained. I would be also interested how long will it take to get the same result without pre-trained, but it may be too computationally expensive.", "votes": null }, { "id": "234026", "postDate": "10/22/2017 03:51:23", "content": "I used resnext-101, with image centor croped to 128X128, used pre-trained. It can get 0.59 val _acc after 1 epoches. I used sgd optimizer. Training is going...
", "rawMarkdown": "I used resnext-101, with image centor croped to 128X128, used pre-trained. It can get 0.59 val _acc after 1 epoches. I used sgd optimizer. Training is going...", "votes": null }, { "id": "234032", "postDate": "10/22/2017 04:10:23", "content": "i am sorry that i made a mistake. 1 epoch=12283645 images takes about 13 hr, with 180x180 as input. So your timing may be reasonable.
\n\nAlso speed of resnext and resnet are not the same even if they have the same flops, https://github.com/Cadene/pretrained-models.pytorch/issues/8
\n\nI suspect it is due to group convolution in pytorch implementation ,\nhttps://discuss.pytorch.org/t/does-pytorch-optimize-the-group-parameter-in-convs/3060
\n\nWhat’s New in cuDNN 7?\nGrouped Convolutions for models such as ResNeXt and Xception and CTC (Connectionist Temporal Classification) loss layer for temporal classification
\n\nhttps://developer.nvidia.com/cudnn
", "rawMarkdown": "i am sorry that i made a mistake. 1 epoch=12283645 images takes about 13 hr, with 180x180 as input. So your timing may be reasonable. \n\nAlso speed of resnext and resnet are not the same even if they have the same flops, https://github.com/Cadene/pretrained-models.pytorch/issues/8\n\nI suspect it is due to group convolution in pytorch implementation ,\nhttps://discuss.pytorch.org/t/does-pytorch-optimize-the-group-parameter-in-convs/3060\n\nWhat’s New in cuDNN 7?\nGrouped Convolutions for models such as ResNeXt and Xception and CTC (Connectionist Temporal Classification) loss layer for temporal classification\n\nhttps://developer.nvidia.com/cudnn", "votes": null }, { "id": "234047", "postDate": "10/22/2017 05:45:20", "content": "I'm just wondering, how many layers are actually training in your 101 layers net? For me, training one more block of Xception takes me 2 more hours for one epoch. (single GTX 1080)
", "rawMarkdown": "I'm just wondering, how many layers are actually training in your 101 layers net? For me, training one more block of Xception takes me 2 more hours for one epoch. (single GTX 1080)", "votes": null }, { "id": "234061", "postDate": "10/22/2017 07:06:31", "content": "I use single ResNet 50 and my current LB-score is 0.45813, local validation score is 0.4914.
", "rawMarkdown": "I use single ResNet 50 and my current LB-score is 0.45813, local validation score is 0.4914.", "votes": null }, { "id": "234093", "postDate": "10/22/2017 09:31:06", "content": "SE-ResNet-50:
\n\nthanks for the information! What is the learning rate you used for finetunning?
", "rawMarkdown": "thanks for the information! What is the learning rate you used for finetunning?", "votes": null }, { "id": "234101", "postDate": "10/22/2017 10:10:07", "content": "What's your learning rate schedule/optimize?
", "rawMarkdown": "What's your learning rate schedule/optimize?", "votes": null }, { "id": "234102", "postDate": "10/22/2017 10:11:05", "content": "I am using SGD with Nesterov momentum (0.9) and batch_size=256
\n\nGrouped Convs seem to be supported in the current master of Pytorch: https://github.com/pytorch/pytorch/pull/3057
", "rawMarkdown": "Grouped Convs seem to be supported in the current master of Pytorch: https://github.com/pytorch/pytorch/pull/3057", "votes": null }, { "id": "234111", "postDate": "10/22/2017 10:43:25", "content": "i think this is spatial, where group = in_planes. It improve mobilenet but not Resnext, where group=32 or 64.
", "rawMarkdown": "i think this is spatial, where group = in_planes. It improve mobilenet but not Resnext, where group=32 or 64.", "votes": null }, { "id": "234115", "postDate": "10/22/2017 10:49:21", "content": "@Vladimir Iglovikov
\n\nI rewrite my code an i get the following results:
\n\nSystem: 4x1080Ti/pytorch
input =160x160
network = resnet101
batch = 256
time for 1000 iterations = 1000*256 images = 7 min.
if one epoch = 12283645 images, i am getting time per epoch = 336 min = 5.6 hr.
\n\nhow many images are there in one epoch for your timing of 4.5hr? Thanks a lot!
\n\n(I am trying to see if my system has been setup correctly. This is the first time i use multi-gpu training intensively)
", "rawMarkdown": "Vladimir Iglovikov\n \nI rewrite my code an i get the following results:\n\n - System: 4x1080Ti/pytorch\n\n - input =160x160\n\n - network = resnet101\n\n - batch = 256\n\n - time for 1000 iterations = 1000*256 images = 7 min.\n\nif one epoch = 12283645 images, i am getting time per epoch = 336 min = 5.6 hr.\n\nhow many images are there in one epoch for your timing of 4.5hr? Thanks a lot!\n\n(I am trying to see if my system has been setup correctly. This is the first time i use multi-gpu training intensively)", "votes": null }, { "id": "234137", "postDate": "10/22/2017 11:56:17", "content": "one epoch 4.5 hours? why so fast?
", "rawMarkdown": "one epoch 4.5 hours? why so fast?", "votes": null }, { "id": "234151", "postDate": "10/22/2017 12:49:19", "content": "Resnet-18: val accuracy 0.63 trained for 6 epochs, simple SGD, lr 0.1 -> 0.01 -> 0.001 , change lr after 2 epochs, no data augmentation no crop, 180x180, just change FC layer to get the 5270 classes and train
", "rawMarkdown": "Resnet-18: val accuracy 0.63 trained for 6 epochs, simple SGD, lr 0.1 -> 0.01 -> 0.001 , change lr after 2 epochs, no data augmentation no crop, 180x180, just change FC layer to get the 5270 classes and train", "votes": null }, { "id": "234175", "postDate": "10/22/2017 14:21:12", "content": "One epoch - 11738624 Images. Rest are used for validation.
\n\nBatch_size = 512
", "rawMarkdown": "One epoch - 11738624 Images. Rest are used for validation. \n\nBatch_size = 512", "votes": null }, { "id": "234204", "postDate": "10/22/2017 16:45:04", "content": "Under the same setting (batch=512), i am able to get training iteration of 40,526 images per minute. For an epoch of 11,738,624 it would take 282 min or 4.7 hr, which is close to your results. Thanks again!
", "rawMarkdown": "Under the same setting (batch=512), i am able to get training iteration of 40,526 images per minute. For an epoch of 11,738,624 it would take 282 min or 4.7 hr, which is close to your results. Thanks again!", "votes": null }, { "id": "234294", "postDate": "10/22/2017 22:25:40", "content": "Some useful info in this thread. Thanks all. I'm curious what people are settling on for training augmentation? I adapted some training code used in previous challenges last week and set some models training. I've got one model moving past the mid 60s in validation error and another that got stuck in the mid 50s.
\n\nThe model that got stuck was high capacity but I had colour/saturation augmentation enabled. The one that's still chugging along and doing much better only had random crop/scale and rotation. I'm guessing the prevalence of almost constant black or white backgrounds in most images means that adding variability to that requires much more learning than leaving it be.
\n\nNext angle is to see if disabling the random crop and just leaving h-flip and a bit of rotation on does better... basically assuming that the preprocessing of the competition dataset leaves things relatively well centred and similarly scaled across train and test datasets. Has anyone hit close to .7 or better with almost no augmentation?
", "rawMarkdown": "Some useful info in this thread. Thanks all. I'm curious what people are settling on for training augmentation? I adapted some training code used in previous challenges last week and set some models training. I've got one model moving past the mid 60s in validation error and another that got stuck in the mid 50s. \n\nThe model that got stuck was high capacity but I had colour/saturation augmentation enabled. The one that's still chugging along and doing much better only had random crop/scale and rotation. I'm guessing the prevalence of almost constant black or white backgrounds in most images means that adding variability to that requires much more learning than leaving it be.\n\nNext angle is to see if disabling the random crop and just leaving h-flip and a bit of rotation on does better... basically assuming that the preprocessing of the competition dataset leaves things relatively well centred and similarly scaled across train and test datasets. Has anyone hit close to .7 or better with almost no augmentation?", "votes": null }, { "id": "234309", "postDate": "10/22/2017 23:06:07", "content": "Are you training all params in your net?
", "rawMarkdown": "Are you training all params in your net?", "votes": null }, { "id": "234997", "postDate": "10/24/2017 14:54:34", "content": "here is a possible Cdiscount in-house baseline results:\nhttps://www.linkedin.com/in/zhiwei-li-81b444126/
\n\nApr 2017 – Sep 2017 Employment Duration 6 mos
\n\nLocationBordeaux Area, France
\n\nLarge Scale Image Classification for e-business product categorization
\n\n• Research work on different Deep Learning architectures like Inception, ResNet etc.
\n\n• Worked on 20 million images in roughly 9000 unbalanced class for product categories
\n\n• Implemented transfer learning based on Inception V3 model using TensorFlow and Keras
\n\n• Designed a quick model construction pipeline, raised ~60 times efficacy, achieved 72.3% accuracy
\n\n• This model will be used to improve the categorization ability of current model
", "rawMarkdown": "here is a possible Cdiscount in-house baseline results:\nhttps://www.linkedin.com/in/zhiwei-li-81b444126/\n\nApr 2017 – Sep 2017 Employment Duration 6 mos\n\nLocationBordeaux Area, France\n\nLarge Scale Image Classification for e-business product categorization\n\n• Research work on different Deep Learning architectures like Inception, ResNet etc.\n\n• Worked on 20 million images in roughly 9000 unbalanced class for product categories\n\n• Implemented transfer learning based on Inception V3 model using TensorFlow and Keras\n\n• Designed a quick model construction pipeline, raised ~60 times efficacy, achieved 72.3% accuracy\n\n• This model will be used to improve the categorization ability of current model", "votes": null }, { "id": "235034", "postDate": "10/24/2017 16:40:10", "content": "\n\n\n9000 unbalanced classes
\n
This probably makes the problem harder.
", "rawMarkdown": "> 9000 unbalanced classes \n\nThis probably makes the problem harder.", "votes": null }, { "id": "235081", "postDate": "10/24/2017 17:44:20", "content": "> This probably makes the problem harder.
\n\nOr you just caught a resume 'embellishment' gone wrong..
\nThat's linkedin.
when i test with renet v2 101 one epoch (train:9 val:1)
\n : lr rate 0.01,
\n : aug flip
\n : size 180*180
\n : rmsprop optimizer
\n : init with tensorflow slim renet 101
\n : no weight decay, no batch norm decay
\n : batch size 48
train acc graph movi around 60%, but\nvaldation set give me acc 0.1%
\n\nwhat makes my model so overfitting?\n - 5270 1*1 conv layer?\ni refenrece following model\n - https://github.com/tensorflow/models/tree/master/research/slim
", "rawMarkdown": "when i test with renet v2 101 one epoch (train:9 val:1)you can use sgd, rmsprop is not recommended
", "rawMarkdown": "you can use sgd, rmsprop is not recommended", "votes": null }, { "id": "235896", "postDate": "10/26/2017 08:55:53", "content": "why rmsprop is not recommended?
", "rawMarkdown": "why rmsprop is not recommended?", "votes": null }, { "id": "235902", "postDate": "10/26/2017 09:08:27", "content": "RMSProp should not make much of difference imho. I think you have bug in your code with that much of a difference in train/val!
", "rawMarkdown": "RMSProp should not make much of difference imho. I think you have bug in your code with that much of a difference in train/val!", "votes": null }, { "id": "235941", "postDate": "10/26/2017 10:55:59", "content": "@DaeYoungPark
\n\nrefer to: https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/42069
", "rawMarkdown": "DaeYoungPark\n \nrefer to: https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/42069", "votes": null }, { "id": "236060", "postDate": "10/26/2017 16:20:38", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "236164", "postDate": "10/26/2017 19:11:46", "content": "Its a useful source and am new to kaggle hope to learn something new
", "rawMarkdown": "Its a useful source and am new to kaggle hope to learn something new", "votes": null }, { "id": "236285", "postDate": "10/27/2017 01:27:06", "content": "Thank you for tour refer!!
", "rawMarkdown": "Thank you for tour refer!!", "votes": null }, { "id": "236443", "postDate": "10/27/2017 11:14:47", "content": "Does 'val single crop accuracy: 70.96' mean that you got a validation accuracy of 70.96% after always training with the same 161x161 crop?
\n\nDoes 'val 10 crops accuracy: 71.79' mean that the accuracy went up to 71.79 when you used 10 different crops per image?
\n\nThe conclusion being that augmentation helps.
\n\nSorry to ask. I am still getting used to the shorthand that people use.
\n\nLB - is the score you got on the leader board, right?
\n\nAnd during validation do you take a middle crop or resize from 180 to 161?
", "rawMarkdown": "Does 'val single crop accuracy: 70.96' mean that you got a validation accuracy of 70.96% after always training with the same 161x161 crop?\n\nDoes 'val 10 crops accuracy: 71.79' mean that the accuracy went up to 71.79 when you used 10 different crops per image?\n\nThe conclusion being that augmentation helps.\n\nSorry to ask. I am still getting used to the shorthand that people use.\n\nLB - is the score you got on the leader board, right?\n\nAnd during validation do you take a middle crop or resize from 180 to 161?", "votes": null }, { "id": "236444", "postDate": "10/27/2017 11:15:32", "content": "You must have a bug. What are you doing for pre-processing?
", "rawMarkdown": "You must have a bug. What are you doing for pre-processing?", "votes": null }, { "id": "237360", "postDate": "10/30/2017 06:21:15", "content": "From discussions, it seems that the highest score we get is 0.70 with single model. Did you tried any other tricks?
\n\nDid you make use the other two categories info?
\n\nThanks!
", "rawMarkdown": "From discussions, it seems that the highest score we get is 0.70 with single model. Did you tried any other tricks?\n\nDid you make use the other two categories info? \n\nThanks!", "votes": null }, { "id": "237844", "postDate": "10/31/2017 04:44:58", "content": "How did you adjust your learning rate?\nThanks!
", "rawMarkdown": "How did you adjust your learning rate?\nThanks!", "votes": null }, { "id": "238069", "postDate": "10/31/2017 15:25:41", "content": "Wondering how do you modify resnet101 to accept smaller images. Seems diffrent people has diffrent strategies
", "rawMarkdown": "Wondering how do you modify resnet101 to accept smaller images. Seems diffrent people has diffrent strategies", "votes": null }, { "id": "238242", "postDate": "10/31/2017 23:31:53", "content": "Resnet50, 180x180 input, make use of the other two categories info, 6 epoch, no TTA:\nval_acc: 0.692\nLB: 0.705
", "rawMarkdown": "Resnet50, 180x180 input, make use of the other two categories info, 6 epoch, no TTA:\nval_acc: 0.692\nLB: 0.705", "votes": null }, { "id": "238325", "postDate": "11/01/2017 06:46:30", "content": "What's the meaning of the other two categories info? Did you train multi classifier?
", "rawMarkdown": "What's the meaning of the other two categories info? Did you train multi classifier?", "votes": null }, { "id": "238623", "postDate": "11/01/2017 19:05:37", "content": "In addition to the categories we are interested in, I also train the model to predict the categories of the father levels, like animal - mammal - cat, where cat is our target.
", "rawMarkdown": "In addition to the categories we are interested in, I also train the model to predict the categories of the father levels, like animal - mammal - cat, where cat is our target.", "votes": null }, { "id": "238775", "postDate": "11/02/2017 06:10:18", "content": "Are you using the father levels as auxiliary classifier?
", "rawMarkdown": "Are you using the father levels as auxiliary classifier?", "votes": null }, { "id": "238776", "postDate": "11/02/2017 06:12:05", "content": "regarding hierarchical categorization, one can consider the yolo9000 approach (wordTree): https://arxiv.org/abs/1612.08242
", "rawMarkdown": "regarding hierarchical categorization, one can consider the yolo9000 approach (wordTree): https://arxiv.org/abs/1612.08242", "votes": null }, { "id": "242245", "postDate": "11/10/2017 22:30:23", "content": "Xception, 180x180 \nval acc: 0.7277\naverage images for product LB: 0.73785.
", "rawMarkdown": "Xception, 180x180 \nval acc: 0.7277\naverage images for product LB: 0.73785.", "votes": null }, { "id": "242271", "postDate": "11/11/2017 01:28:21", "content": "average images for product，you predict the prob of every image and average them?
", "rawMarkdown": "average images for product，you predict the prob of every image and average them?", "votes": null }, { "id": "242282", "postDate": "11/11/2017 02:42:55", "content": "Yes, for test data, for each product I predict the prob of every image and average them.
", "rawMarkdown": "Yes, for test data, for each product I predict the prob of every image and average them.", "votes": null }, { "id": "242303", "postDate": "11/11/2017 05:24:16", "content": "I'm also using Xception now. But I only got LB: 0.68 for about 15 epoch. Do you train all parameters in the model? Which augmentation strategy do you use? Thanks a lot!
", "rawMarkdown": "I'm also using Xception now. But I only got LB: 0.68 for about 15 epoch. Do you train all parameters in the model? Which augmentation strategy do you use? Thanks a lot!", "votes": null }, { "id": "242306", "postDate": "11/11/2017 05:32:00", "content": "Do you use pretrained model?
", "rawMarkdown": "Do you use pretrained model?", "votes": null }, { "id": "242335", "postDate": "11/11/2017 09:26:11", "content": "i'm still using a single resnet-18 because of very limited resources. 160x160 random crop in training, from scratch, lr 0.1 -> 0.01 -> 0.001 -> 0.0005 -> 0.0001, changing lr every few epochs, SGD with momentum 0.9, 192 images batches. I have modified the network in the last few conv layers. LB 0.69. inference with only 1 center 160x160 crop.
", "rawMarkdown": "i'm still using a single resnet-18 because of very limited resources. 160x160 random crop in training, from scratch, lr 0.1 -> 0.01 -> 0.001 -> 0.0005 -> 0.0001, changing lr every few epochs, SGD with momentum 0.9, 192 images batches. I have modified the network in the last few conv layers. LB 0.69. inference with only 1 center 160x160 crop.", "votes": null }, { "id": "242363", "postDate": "11/11/2017 10:45:11", "content": "How did you modify the conv layers? Seems like a pretty good score for resnet-18!\nAny data augmentation? We are still looking for the right kind...
", "rawMarkdown": "How did you modify the conv layers? Seems like a pretty good score for resnet-18!\nAny data augmentation? We are still looking for the right kind...", "votes": null }, { "id": "242370", "postDate": "11/11/2017 11:19:51", "content": "it seems to me that there's a kind of flaw in vanilla resnet-18 in the final few stages, if the final classifier has to distinguish among 5270 classes. The final FC layer is just a matrix, it basically can do 2 things: rotate a vector and project or embed the vector in the target space. Embedding in higher dimensional space does not change the dimension of the initial subspace, it cannot give higher resolution. Resnet 18 has a average pooling before the FC, there is too much of a bottlenck before FC. So vanilla resnet 18 cannot get good class resolution beyond a 10-20% of the full set. Unfortunately time is running and i have very old metal.... i'm working on a completely different back-end for my little resnet 18 but not sure if i can make it in time.
", "rawMarkdown": "it seems to me that there's a kind of flaw in vanilla resnet-18 in the final few stages, if the final classifier has to distinguish among 5270 classes. The final FC layer is just a matrix, it basically can do 2 things: rotate a vector and project or embed the vector in the target space. Embedding in higher dimensional space does not change the dimension of the initial subspace, it cannot give higher resolution. Resnet 18 has a average pooling before the FC, there is too much of a bottlenck before FC. So vanilla resnet 18 cannot get good class resolution beyond a 10-20% of the full set. Unfortunately time is running and i have very old metal.... i'm working on a completely different back-end for my little resnet 18 but not sure if i can make it in time.", "votes": null }, { "id": "242525", "postDate": "11/11/2017 19:56:11", "content": "@CSAdu: Yes I use Keras pre-trained model.
", "rawMarkdown": "CSAdu: Yes I use Keras pre-trained model.", "votes": null }, { "id": "242526", "postDate": "11/11/2017 19:58:09", "content": "@Brian Luo: Yes I train all parameters. I use small augmentation to get 0.68 then continue training without augmentation.
", "rawMarkdown": "Brian Luo: Yes I train all parameters. I use small augmentation to get 0.68 then continue training without augmentation.", "votes": null }, { "id": "242761", "postDate": "11/12/2017 14:56:23", "content": "So do you use pretrained model? Where you find your pretrained model. I start to train the SE-Resnet50 but seems after 1 epoch I only got 0.50, it this validation accuracy close to your data after 1 epoch?
", "rawMarkdown": "So do you use pretrained model? Where you find your pretrained model. I start to train the SE-Resnet50 but seems after 1 epoch I only got 0.50, it this validation accuracy close to your data after 1 epoch?", "votes": null }, { "id": "242964", "postDate": "11/13/2017 03:50:10", "content": "Thanks for your reply. How many epoch do you use to get 0.73?
", "rawMarkdown": "Thanks for your reply. How many epoch do you use to get 0.73?", "votes": null }, { "id": "243159", "postDate": "11/13/2017 13:30:15", "content": "It takes me about 60 epochs to get 0.68 and another 100 epochs to get 0.73, each epoch is 1/10 train data.
", "rawMarkdown": "It takes me about 60 epochs to get 0.68 and another 100 epochs to get 0.73, each epoch is 1/10 train data.", "votes": null }, { "id": "245805", "postDate": "11/19/2017 19:16:05", "content": "Are you using the wordTree approach? I'm not sure how to apply this to our task. Do you know how to predict conditional probabilities for each intermediate node?
", "rawMarkdown": "Are you using the wordTree approach? I'm not sure how to apply this to our task. Do you know how to predict conditional probabilities for each intermediate node?", "votes": null }, { "id": "245946", "postDate": "11/20/2017 06:28:59", "content": "My resnet50 only got to about 63% accuracy before it started massively overfitting
\n\nEDIT: went back to an earlier epoch and changed the data augmentation, so it got to 65% (with room for improvement) after a couple of epochs with more overfitting than some of my other models, but not so much as before where the validation accuracy was actually decreasing
", "rawMarkdown": "My resnet50 only got to about 63% accuracy before it started massively overfitting\n\nEDIT: went back to an earlier epoch and changed the data augmentation, so it got to 65% (with room for improvement) after a couple of epochs with more overfitting than some of my other models, but not so much as before where the validation accuracy was actually decreasing", "votes": null }, { "id": "246516", "postDate": "11/21/2017 08:19:04", "content": "Did you use a globalaveragePooling layer just before the last FC layers (which gives 2048-->5270)?\nOr some other poolling layers which gives X>2048 --->5270 at the top?\nThanks
", "rawMarkdown": "Did you use a globalaveragePooling layer just before the last FC layers (which gives 2048-->5270)?\nOr some other poolling layers which gives X>2048 --->5270 at the top?\nThanks", "votes": null }, { "id": "246544", "postDate": "11/21/2017 09:07:49", "content": "Just GlobalAveragePooling.
", "rawMarkdown": "Just GlobalAveragePooling.", "votes": null }, { "id": "247083", "postDate": "11/22/2017 09:25:34", "content": "inception resnet v2, tune keras pretrained model. Train top layers for 2 epoch, then train all parameters for 12 epoch. \n180x180, single image val acc: 0.7305, LB: 0.74564(average prediction for all images of a product.)
", "rawMarkdown": "inception resnet v2, tune keras pretrained model. Train top layers for 2 epoch, then train all parameters for 12 epoch. \n180x180, single image val acc: 0.7305, LB: 0.74564(average prediction for all images of a product.)", "votes": null }, { "id": "247212", "postDate": "11/22/2017 15:45:43", "content": "How long you need to train one epoch for inception resnet v2?
", "rawMarkdown": "How long you need to train one epoch for inception resnet v2?", "votes": null }, { "id": "247386", "postDate": "11/22/2017 23:20:26", "content": "It takes me about 9 hours to train all parameters for each epoch with AMD 1950X, 4 gtx 1080ti and Samsung 960 pro. My GPUs usage is not 100%, I think there is still space to improve speed of I/O of my code.
", "rawMarkdown": "It takes me about 9 hours to train all parameters for each epoch with AMD 1950X, 4 gtx 1080ti and Samsung 960 pro. My GPUs usage is not 100%, I think there is still space to improve speed of I/O of my code.", "votes": null }, { "id": "248763", "postDate": "11/26/2017 22:49:09", "content": "Wondering what is your best single model and its accuracy?
", "rawMarkdown": "Wondering what is your best single model and its accuracy?", "votes": null }, { "id": "248797", "postDate": "11/27/2017 01:08:15", "content": "My best model gets about 74.8% on the leaderboard by itself. I only have two or three real models trained and ensembling currently isn't getting me very big gains.
", "rawMarkdown": "My best model gets about 74.8% on the leaderboard by itself. I only have two or three real models trained and ensembling currently isn't getting me very big gains.", "votes": null }, { "id": "249023", "postDate": "11/27/2017 15:00:58", "content": "Thanks for your share. I have some questions:
\n\nHow did you guys perform the train/val split?
", "rawMarkdown": "How did you guys perform the train/val split?", "votes": null }, { "id": "249228", "postDate": "11/28/2017 01:21:34", "content": "You are welcome.
\n\nthanks for the details; do you have any tips / ideas why it \"doesn't work\" for me? I tried \"Inception_v3\" from Keras, after one epoch (about 80% of Train data) I was getting only 20% accuracy (on Validation only 13%) !!!
\n\nI use my own Generator, 512 Batch size, random shuffle of the data .... ,aybe I need to try with the Generator from the link
", "rawMarkdown": "thanks for the details; do you have any tips / ideas why it \"doesn't work\" for me? I tried \"Inception_v3\" from Keras, after one epoch (about 80% of Train data) I was getting only 20% accuracy (on Validation only 13%) !!!\n\nI use my own Generator, 512 Batch size, random shuffle of the data .... ,aybe I need to try with the Generator from the link", "votes": null }, { "id": "249405", "postDate": "11/28/2017 11:41:04", "content": "My guess:
\n\nDid you use the keras application build in preprocess function? \nfrom keras.applications.inception_v3 import preprocess_input,
What optimizer did you use, can you try 'adam' ?
I didn't use the function itself (think it was throwing an error?) but did:\nimg = img / 255.\nimg = img - 0.5\n(I think there was one more step .. basically I copied the 'tf' mode)
\n\nI used rmsprop, will try with adam in the evening ...
\n\nbtw. I used custom Input shape (180,180,3) - could that also be the problem?
", "rawMarkdown": "I didn't use the function itself (think it was throwing an error?) but did:\nimg = img / 255.\nimg = img - 0.5\n(I think there was one more step .. basically I copied the 'tf' mode)\n\nI used rmsprop, will try with adam in the evening ...\n\nbtw. I used custom Input shape (180,180,3) - could that also be the problem?", "votes": null }, { "id": "249679", "postDate": "11/29/2017 01:14:17", "content": "Usually, my val_acc is higher than acc when acc is lower than 60%. I was using (img/255-0.5)*2.0 and can get 40% with 10% data(with inception resnet v2 and xception). Then I switch to the keras preprocess and can get 50% with 10% data(with resnet101 or resnet 151). I also use (180,180,3). What is your batch size?
", "rawMarkdown": "Usually, my val_acc is higher than acc when acc is lower than 60%. I was using (img/255-0.5)*2.0 and can get 40% with 10% data(with inception resnet v2 and xception). Then I switch to the keras preprocess and can get 50% with 10% data(with resnet101 or resnet 151). I also use (180,180,3). What is your batch size?", "votes": null }, { "id": "249765", "postDate": "11/29/2017 07:05:49", "content": "thank you, with some corrections (guess the preprocessing was wrong) I got 50% after around 1.5 epoch (around 12M Images), an improvement but still no so great
\n\nI'm using batch size = 512, this time it was with adam optimizer (all default settings), will test with Xception or the pretrained weights from Heng CherKeng
", "rawMarkdown": "thank you, with some corrections (guess the preprocessing was wrong) I got 50% after around 1.5 epoch (around 12M Images), an improvement but still no so great\n\nI'm using batch size = 512, this time it was with adam optimizer (all default settings), will test with Xception or the pretrained weights from Heng CherKeng", "votes": null }, { "id": "249807", "postDate": "11/29/2017 08:28:49", "content": "Hi @iFighting, @minu, @Heng CherKeng,\nIt seems like you guys are using MXNet?\nWhat are you impressions comparing to PyTorch?\nDoes MXNet facilitates specific aspects? Were you able to attain better results training with MXNet?
", "rawMarkdown": "Hi @iFighting, @minu, @Heng CherKeng,\nIt seems like you guys are using MXNet?\nWhat are you impressions comparing to PyTorch?\nDoes MXNet facilitates specific aspects? Were you able to attain better results training with MXNet?", "votes": null }, { "id": "249816", "postDate": "11/29/2017 08:50:38", "content": "i am using pytorch only for this competition. I use caffe, mxnet, tensorflow before. I think their results are all about the same (because they use the \"same formula\" and based on nvidia cudnn). Some DL framework are faster. But overall, I think pytorch is fast and easy to debug and experiment with new structure.
", "rawMarkdown": "i am using pytorch only for this competition. I use caffe, mxnet, tensorflow before. I think their results are all about the same (because they use the \"same formula\" and based on nvidia cudnn). Some DL framework are faster. But overall, I think pytorch is fast and easy to debug and experiment with new structure.", "votes": null }, { "id": "250062", "postDate": "11/29/2017 17:43:48", "content": "what is the 'tf' mode? Can you guys point me where to read more about retraining and transfer learning with Xception (or Inception, resnet, etc)? :)
", "rawMarkdown": "what is the 'tf' mode? Can you guys point me where to read more about retraining and transfer learning with Xception (or Inception, resnet, etc)? :)", "votes": null }, { "id": "250103", "postDate": "11/29/2017 19:49:01", "content": "I assume 'tf' stands for tensorflow, but might be wrong :)\nwhen you check the code in https://github.com/fchollet/keras/blob/master/keras/applications/imagenet_utils.py you will see, but basically it's (since 2 days ago, before it was the transformation mentioned by Enhao above):
\nx /= 127.5\nx -= 1.
for a tutorial you can check out this keras blog: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
", "rawMarkdown": "I assume 'tf' stands for tensorflow, but might be wrong :)\nwhen you check the code in https://github.com/fchollet/keras/blob/master/keras/applications/imagenet_utils.py you will see, but basically it's (since 2 days ago, before it was the transformation mentioned by Enhao above): \nx /= 127.5\nx -= 1.\n\nfor a tutorial you can check out this keras blog: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html", "votes": null }, { "id": "250479", "postDate": "11/30/2017 03:38:05", "content": "Update: Resnet50 gets 0.71450 on the leaderboard after 10 epochs
", "rawMarkdown": "Update: Resnet50 gets 0.71450 on the leaderboard after 10 epochs", "votes": null }, { "id": "250497", "postDate": "11/30/2017 04:03:18", "content": "Just wondering what you have done to boost from 65% to 71%?
", "rawMarkdown": "Just wondering what you have done to boost from 65% to 71%?", "votes": null }, { "id": "250520", "postDate": "11/30/2017 04:34:56", "content": "Thanks! And 65% at 2 epoch is quite high I believed. Would you mind if you can share how do you achieve this? My resnet 101 only have about 58% after two epoch with 0.001 lr
", "rawMarkdown": "Thanks! And 65% at 2 epoch is quite high I believed. Would you mind if you can share how do you achieve this? My resnet 101 only have about 58% after two epoch with 0.001 lr", "votes": null }, { "id": "250607", "postDate": "11/30/2017 06:20:52", "content": "yes, i use mxnet
", "rawMarkdown": "yes, i use mxnet", "votes": null }, { "id": "250633", "postDate": "11/30/2017 06:40:21", "content": "I'm just familiar with MXNet. and MXNet is efficient in Multi-GPU.
", "rawMarkdown": "I'm just familiar with MXNet. and MXNet is efficient in Multi-GPU.", "votes": null }, { "id": "252273", "postDate": "12/02/2017 16:49:51", "content": "What's different between (img/255-0.5)*2.0 and keras preprocess (x /= 127.5 x -= 1)? I think they're the same thing, why you get higher result with keras preprocess?
", "rawMarkdown": "What's different between (img/255-0.5)*2.0 and keras preprocess (x /= 127.5 x -= 1)? I think they're the same thing, why you get higher result with keras preprocess?", "votes": null }, { "id": "252279", "postDate": "12/02/2017 16:57:55", "content": "yes, most likely the main difference was coming from the adam optimizer (I did both changes at once)
", "rawMarkdown": "yes, most likely the main difference was coming from the adam optimizer (I did both changes at once)", "votes": null } ], "comments": [ { "id": 233991, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 01:25:43", "content": "thanks a lot!
", "votes": null, "replies": [] }, { "id": 233996, "author_name": "jandjenter", "author_url": "", "post_date": "10/22/2017 01:48:17", "content": "That's quite high score for single model! How long did you train for this and which GPU did you use?
\n\nDue to limited resources, I'm using images scaled down to 90x90 and simple Xception, no TTA
\n\nval_acc = .6904 (for each image)
LB= .7092 (averaged softmax output for all images in same product id)
I'm trying many other approaches like using top 2000 frequent items, using category 1,2 information, etc, but nothing beats this baseline yet. Sad to see much better results with basic model, but thanks for the info.
", "votes": null, "replies": [ { "id": 233997, "author_name": "iglovikov", "author_url": "", "post_date": "10/22/2017 01:51:12", "content": "4 x GTX 1080 Ti
\n\n18 epochs x 4.5 hours
\n\nI have 0.69 after 7th epoch
", "votes": null, "replies": [] }, { "id": 234000, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 01:58:43", "content": "seems that we need long epoches!
", "votes": null, "replies": [] }, { "id": 234006, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 02:41:18", "content": "i am also using 4 x GTX 1080 Ti. For resnext101 (about same flops as your resnet101 ), 1 epoch=12283645 images takes about 1.3 hr. I am using 180x180 as input. I suppose your input size is not 180x180?
", "votes": null, "replies": [] }, { "id": 234007, "author_name": "ifighting", "author_url": "", "post_date": "10/22/2017 02:43:55", "content": "what does TTA mean?
", "votes": null, "replies": [] }, { "id": 234010, "author_name": "iglovikov", "author_url": "", "post_date": "10/22/2017 02:52:06", "content": "I use 160x160 crops.
\n\n1.3hrs per epoch.... It looks like I am doing something wrong.
\n\nIt may happen that my SSD is not fast enough and I need to buy faster one.
", "votes": null, "replies": [] }, { "id": 234012, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 02:53:57", "content": "TTA= test time augmentation (e.g. multi crops, resize in test time)
", "votes": null, "replies": [] }, { "id": 234013, "author_name": "jandjenter", "author_url": "", "post_date": "10/22/2017 02:56:01", "content": "Test-time augmentation. In my case, I can get extra 0.01 just with 12-way TTA(flip, shift up/down, etc) with exact same weight - almost free cake :)
", "votes": null, "replies": [] }, { "id": 234032, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 04:10:23", "content": "i am sorry that i made a mistake. 1 epoch=12283645 images takes about 13 hr, with 180x180 as input. So your timing may be reasonable.
\n\nAlso speed of resnext and resnet are not the same even if they have the same flops, https://github.com/Cadene/pretrained-models.pytorch/issues/8
\n\nI suspect it is due to group convolution in pytorch implementation ,\nhttps://discuss.pytorch.org/t/does-pytorch-optimize-the-group-parameter-in-convs/3060
\n\nWhat’s New in cuDNN 7?\nGrouped Convolutions for models such as ResNeXt and Xception and CTC (Connectionist Temporal Classification) loss layer for temporal classification
\n\nhttps://developer.nvidia.com/cudnn
", "votes": null, "replies": [] }, { "id": 234106, "author_name": "timjoseph", "author_url": "", "post_date": "10/22/2017 10:17:56", "content": "Grouped Convs seem to be supported in the current master of Pytorch: https://github.com/pytorch/pytorch/pull/3057
", "votes": null, "replies": [] }, { "id": 234111, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 10:43:25", "content": "i think this is spatial, where group = in_planes. It improve mobilenet but not Resnext, where group=32 or 64.
", "votes": null, "replies": [] }, { "id": 234115, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 10:49:21", "content": "@Vladimir Iglovikov
\n\nI rewrite my code an i get the following results:
\n\nSystem: 4x1080Ti/pytorch
input =160x160
network = resnet101
batch = 256
time for 1000 iterations = 1000*256 images = 7 min.
if one epoch = 12283645 images, i am getting time per epoch = 336 min = 5.6 hr.
\n\nhow many images are there in one epoch for your timing of 4.5hr? Thanks a lot!
\n\n(I am trying to see if my system has been setup correctly. This is the first time i use multi-gpu training intensively)
", "votes": null, "replies": [] }, { "id": 234137, "author_name": "ifighting", "author_url": "", "post_date": "10/22/2017 11:56:17", "content": "one epoch 4.5 hours? why so fast?
", "votes": null, "replies": [] }, { "id": 234175, "author_name": "iglovikov", "author_url": "", "post_date": "10/22/2017 14:21:12", "content": "One epoch - 11738624 Images. Rest are used for validation.
\n\nBatch_size = 512
", "votes": null, "replies": [] }, { "id": 234204, "author_name": "hengck23", "author_url": "", "post_date": "10/22/2017 16:45:04", "content": "Under the same setting (batch=512), i am able to get training iteration of 40,526 images per minute. For an epoch of 11,738,624 it would take 282 min or 4.7 hr, which is close to your results. Thanks again!
", "votes": null, "replies": [] }, { "id": 234309, "author_name": "brianlzm", "author_url": "", "post_date": "10/22/2017 23:06:07", "content": "Are you training all params in your net?
", "votes": null, "replies": [] } ] }, { "id": 234005, "author_name": "lyakaap", "author_url": "", "post_date": "10/22/2017 02:40:41", "content": "Amazing score. Did you do training without pretrained weight?
\n\nMy results: \ninception_v3, 160 crop, val loss: 1.78~, val acc: 0.66~, lb: 0.673~
", "votes": null, "replies": [ { "id": 234019, "author_name": "iglovikov", "author_url": "", "post_date": "10/22/2017 03:31:19", "content": "I used pre-trained. I would be also interested how long will it take to get the same result without pre-trained, but it may be too computationally expensive.
", "votes": null, "replies": [] }, { "id": 234026, "author_name": "heroxrq", "author_url": "", "post_date": "10/22/2017 03:51:23", "content": "I used resnext-101, with image centor croped to 128X128, used pre-trained. It can get 0.59 val _acc after 1 epoches. I used sgd optimizer. Training is going...
", "votes": null, "replies": [] } ] }, { "id": 234047, "author_name": "brianlzm", "author_url": "", "post_date": "10/22/2017 05:45:20", "content": "I'm just wondering, how many layers are actually training in your 101 layers net? For me, training one more block of Xception takes me 2 more hours for one epoch. (single GTX 1080)
", "votes": null, "replies": [] }, { "id": 234061, "author_name": "toshik", "author_url": "", "post_date": "10/22/2017 07:06:31", "content": "I use single ResNet 50 and my current LB-score is 0.45813, local validation score is 0.4914.
", "votes": null, "replies": [ { "id": 236444, "author_name": "jinkos", "author_url": "", "post_date": "10/27/2017 11:15:32", "content": "You must have a bug. What are you doing for pre-processing?
", "votes": null, "replies": [] } ] }, { "id": 234093, "author_name": "davletag", "author_url": "", "post_date": "10/22/2017 09:31:06", "content": "SE-ResNet-50:
\n\nthanks for the information! What is the learning rate you used for finetunning?
", "votes": null, "replies": [] }, { "id": 234102, "author_name": "davletag", "author_url": "", "post_date": "10/22/2017 10:11:05", "content": "I am using SGD with Nesterov momentum (0.9) and batch_size=256
\n\nDoes 'val single crop accuracy: 70.96' mean that you got a validation accuracy of 70.96% after always training with the same 161x161 crop?
\n\nDoes 'val 10 crops accuracy: 71.79' mean that the accuracy went up to 71.79 when you used 10 different crops per image?
\n\nThe conclusion being that augmentation helps.
\n\nSorry to ask. I am still getting used to the shorthand that people use.
\n\nLB - is the score you got on the leader board, right?
\n\nAnd during validation do you take a middle crop or resize from 180 to 161?
", "votes": null, "replies": [] }, { "id": 242761, "author_name": "strideradu", "author_url": "", "post_date": "11/12/2017 14:56:23", "content": "So do you use pretrained model? Where you find your pretrained model. I start to train the SE-Resnet50 but seems after 1 epoch I only got 0.50, it this validation accuracy close to your data after 1 epoch?
", "votes": null, "replies": [] } ] }, { "id": 234101, "author_name": "timjoseph", "author_url": "", "post_date": "10/22/2017 10:10:07", "content": "What's your learning rate schedule/optimize?
", "votes": null, "replies": [] }, { "id": 234151, "author_name": "andreanobile", "author_url": "", "post_date": "10/22/2017 12:49:19", "content": "Resnet-18: val accuracy 0.63 trained for 6 epochs, simple SGD, lr 0.1 -> 0.01 -> 0.001 , change lr after 2 epochs, no data augmentation no crop, 180x180, just change FC layer to get the 5270 classes and train
", "votes": null, "replies": [] }, { "id": 234294, "author_name": "rwightman", "author_url": "", "post_date": "10/22/2017 22:25:40", "content": "Some useful info in this thread. Thanks all. I'm curious what people are settling on for training augmentation? I adapted some training code used in previous challenges last week and set some models training. I've got one model moving past the mid 60s in validation error and another that got stuck in the mid 50s.
\n\nThe model that got stuck was high capacity but I had colour/saturation augmentation enabled. The one that's still chugging along and doing much better only had random crop/scale and rotation. I'm guessing the prevalence of almost constant black or white backgrounds in most images means that adding variability to that requires much more learning than leaving it be.
\n\nNext angle is to see if disabling the random crop and just leaving h-flip and a bit of rotation on does better... basically assuming that the preprocessing of the competition dataset leaves things relatively well centred and similarly scaled across train and test datasets. Has anyone hit close to .7 or better with almost no augmentation?
", "votes": null, "replies": [] }, { "id": 234997, "author_name": "hengck23", "author_url": "", "post_date": "10/24/2017 14:54:34", "content": "here is a possible Cdiscount in-house baseline results:\nhttps://www.linkedin.com/in/zhiwei-li-81b444126/
\n\nApr 2017 – Sep 2017 Employment Duration 6 mos
\n\nLocationBordeaux Area, France
\n\nLarge Scale Image Classification for e-business product categorization
\n\n• Research work on different Deep Learning architectures like Inception, ResNet etc.
\n\n• Worked on 20 million images in roughly 9000 unbalanced class for product categories
\n\n• Implemented transfer learning based on Inception V3 model using TensorFlow and Keras
\n\n• Designed a quick model construction pipeline, raised ~60 times efficacy, achieved 72.3% accuracy
\n\n• This model will be used to improve the categorization ability of current model
", "votes": null, "replies": [ { "id": 235034, "author_name": "andreaslup", "author_url": "", "post_date": "10/24/2017 16:40:10", "content": "\n\n\n9000 unbalanced classes
\n
This probably makes the problem harder.
", "votes": null, "replies": [] }, { "id": 235081, "author_name": "fernandoconstantino", "author_url": "", "post_date": "10/24/2017 17:44:20", "content": "> This probably makes the problem harder.
\n\nOr you just caught a resume 'embellishment' gone wrong..
\nThat's linkedin.
when i test with renet v2 101 one epoch (train:9 val:1)
\n : lr rate 0.01,
\n : aug flip
\n : size 180*180
\n : rmsprop optimizer
\n : init with tensorflow slim renet 101
\n : no weight decay, no batch norm decay
\n : batch size 48
train acc graph movi around 60%, but\nvaldation set give me acc 0.1%
\n\nwhat makes my model so overfitting?\n - 5270 1*1 conv layer?\ni refenrece following model\n - https://github.com/tensorflow/models/tree/master/research/slim
", "votes": null, "replies": [ { "id": 235892, "author_name": "ifighting", "author_url": "", "post_date": "10/26/2017 08:48:24", "content": "you can use sgd, rmsprop is not recommended
", "votes": null, "replies": [] }, { "id": 235896, "author_name": "dyaprk", "author_url": "", "post_date": "10/26/2017 08:55:53", "content": "why rmsprop is not recommended?
", "votes": null, "replies": [] }, { "id": 235902, "author_name": "timjoseph", "author_url": "", "post_date": "10/26/2017 09:08:27", "content": "RMSProp should not make much of difference imho. I think you have bug in your code with that much of a difference in train/val!
", "votes": null, "replies": [] }, { "id": 235941, "author_name": "hengck23", "author_url": "", "post_date": "10/26/2017 10:55:59", "content": "@DaeYoungPark
\n\nrefer to: https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/42069
", "votes": null, "replies": [] }, { "id": 236285, "author_name": "dyaprk", "author_url": "", "post_date": "10/27/2017 01:27:06", "content": "Thank you for tour refer!!
", "votes": null, "replies": [] } ] }, { "id": 236060, "author_name": "nelsonzhao", "author_url": "", "post_date": "10/26/2017 16:20:38", "content": "Thanks a lot!
", "votes": null, "replies": [] }, { "id": 236164, "author_name": "owaisqayum", "author_url": "", "post_date": "10/26/2017 19:11:46", "content": "Its a useful source and am new to kaggle hope to learn something new
", "votes": null, "replies": [] }, { "id": 237360, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/30/2017 06:21:15", "content": "From discussions, it seems that the highest score we get is 0.70 with single model. Did you tried any other tricks?
\n\nDid you make use the other two categories info?
\n\nThanks!
", "votes": null, "replies": [] }, { "id": 237844, "author_name": "owruby", "author_url": "", "post_date": "10/31/2017 04:44:58", "content": "How did you adjust your learning rate?\nThanks!
", "votes": null, "replies": [] }, { "id": 238069, "author_name": "strideradu", "author_url": "", "post_date": "10/31/2017 15:25:41", "content": "Wondering how do you modify resnet101 to accept smaller images. Seems diffrent people has diffrent strategies
", "votes": null, "replies": [] }, { "id": 238242, "author_name": "ybw9000", "author_url": "", "post_date": "10/31/2017 23:31:53", "content": "Resnet50, 180x180 input, make use of the other two categories info, 6 epoch, no TTA:\nval_acc: 0.692\nLB: 0.705
", "votes": null, "replies": [ { "id": 238325, "author_name": "zhangsongwei", "author_url": "", "post_date": "11/01/2017 06:46:30", "content": "What's the meaning of the other two categories info? Did you train multi classifier?
", "votes": null, "replies": [] }, { "id": 238623, "author_name": "ybw9000", "author_url": "", "post_date": "11/01/2017 19:05:37", "content": "In addition to the categories we are interested in, I also train the model to predict the categories of the father levels, like animal - mammal - cat, where cat is our target.
", "votes": null, "replies": [] }, { "id": 238775, "author_name": "zhangsongwei", "author_url": "", "post_date": "11/02/2017 06:10:18", "content": "Are you using the father levels as auxiliary classifier?
", "votes": null, "replies": [] }, { "id": 238776, "author_name": "hengck23", "author_url": "", "post_date": "11/02/2017 06:12:05", "content": "regarding hierarchical categorization, one can consider the yolo9000 approach (wordTree): https://arxiv.org/abs/1612.08242
", "votes": null, "replies": [] }, { "id": 245805, "author_name": "brianlzm", "author_url": "", "post_date": "11/19/2017 19:16:05", "content": "Are you using the wordTree approach? I'm not sure how to apply this to our task. Do you know how to predict conditional probabilities for each intermediate node?
", "votes": null, "replies": [] } ] }, { "id": 242245, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/10/2017 22:30:23", "content": "Xception, 180x180 \nval acc: 0.7277\naverage images for product LB: 0.73785.
", "votes": null, "replies": [ { "id": 242271, "author_name": "ifighting", "author_url": "", "post_date": "11/11/2017 01:28:21", "content": "average images for product，you predict the prob of every image and average them?
", "votes": null, "replies": [] }, { "id": 242282, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/11/2017 02:42:55", "content": "Yes, for test data, for each product I predict the prob of every image and average them.
", "votes": null, "replies": [] }, { "id": 242303, "author_name": "brianlzm", "author_url": "", "post_date": "11/11/2017 05:24:16", "content": "I'm also using Xception now. But I only got LB: 0.68 for about 15 epoch. Do you train all parameters in the model? Which augmentation strategy do you use? Thanks a lot!
", "votes": null, "replies": [] }, { "id": 242306, "author_name": "strideradu", "author_url": "", "post_date": "11/11/2017 05:32:00", "content": "Do you use pretrained model?
", "votes": null, "replies": [] }, { "id": 242525, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/11/2017 19:56:11", "content": "@CSAdu: Yes I use Keras pre-trained model.
", "votes": null, "replies": [] }, { "id": 242526, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/11/2017 19:58:09", "content": "@Brian Luo: Yes I train all parameters. I use small augmentation to get 0.68 then continue training without augmentation.
", "votes": null, "replies": [] }, { "id": 242964, "author_name": "brianlzm", "author_url": "", "post_date": "11/13/2017 03:50:10", "content": "Thanks for your reply. How many epoch do you use to get 0.73?
", "votes": null, "replies": [] }, { "id": 243159, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/13/2017 13:30:15", "content": "It takes me about 60 epochs to get 0.68 and another 100 epochs to get 0.73, each epoch is 1/10 train data.
", "votes": null, "replies": [] }, { "id": 246516, "author_name": "pierretisseur", "author_url": "", "post_date": "11/21/2017 08:19:04", "content": "Did you use a globalaveragePooling layer just before the last FC layers (which gives 2048-->5270)?\nOr some other poolling layers which gives X>2048 --->5270 at the top?\nThanks
", "votes": null, "replies": [] }, { "id": 246544, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/21/2017 09:07:49", "content": "Just GlobalAveragePooling.
", "votes": null, "replies": [] }, { "id": 247083, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/22/2017 09:25:34", "content": "inception resnet v2, tune keras pretrained model. Train top layers for 2 epoch, then train all parameters for 12 epoch. \n180x180, single image val acc: 0.7305, LB: 0.74564(average prediction for all images of a product.)
", "votes": null, "replies": [] }, { "id": 247212, "author_name": "strideradu", "author_url": "", "post_date": "11/22/2017 15:45:43", "content": "How long you need to train one epoch for inception resnet v2?
", "votes": null, "replies": [] }, { "id": 247386, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/22/2017 23:20:26", "content": "It takes me about 9 hours to train all parameters for each epoch with AMD 1950X, 4 gtx 1080ti and Samsung 960 pro. My GPUs usage is not 100%, I think there is still space to improve speed of I/O of my code.
", "votes": null, "replies": [] }, { "id": 249023, "author_name": "skinish", "author_url": "", "post_date": "11/27/2017 15:00:58", "content": "Thanks for your share. I have some questions:
\n\nYou are welcome.
\n\nthanks for the details; do you have any tips / ideas why it \"doesn't work\" for me? I tried \"Inception_v3\" from Keras, after one epoch (about 80% of Train data) I was getting only 20% accuracy (on Validation only 13%) !!!
\n\nI use my own Generator, 512 Batch size, random shuffle of the data .... ,aybe I need to try with the Generator from the link
", "votes": null, "replies": [] }, { "id": 249405, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/28/2017 11:41:04", "content": "My guess:
\n\nDid you use the keras application build in preprocess function? \nfrom keras.applications.inception_v3 import preprocess_input,
What optimizer did you use, can you try 'adam' ?
I didn't use the function itself (think it was throwing an error?) but did:\nimg = img / 255.\nimg = img - 0.5\n(I think there was one more step .. basically I copied the 'tf' mode)
\n\nI used rmsprop, will try with adam in the evening ...
\n\nbtw. I used custom Input shape (180,180,3) - could that also be the problem?
", "votes": null, "replies": [] }, { "id": 249679, "author_name": "blackarrow3542", "author_url": "", "post_date": "11/29/2017 01:14:17", "content": "Usually, my val_acc is higher than acc when acc is lower than 60%. I was using (img/255-0.5)*2.0 and can get 40% with 10% data(with inception resnet v2 and xception). Then I switch to the keras preprocess and can get 50% with 10% data(with resnet101 or resnet 151). I also use (180,180,3). What is your batch size?
", "votes": null, "replies": [] }, { "id": 249765, "author_name": "steelrose", "author_url": "", "post_date": "11/29/2017 07:05:49", "content": "thank you, with some corrections (guess the preprocessing was wrong) I got 50% after around 1.5 epoch (around 12M Images), an improvement but still no so great
\n\nI'm using batch size = 512, this time it was with adam optimizer (all default settings), will test with Xception or the pretrained weights from Heng CherKeng
", "votes": null, "replies": [] }, { "id": 250062, "author_name": "skinish", "author_url": "", "post_date": "11/29/2017 17:43:48", "content": "what is the 'tf' mode? Can you guys point me where to read more about retraining and transfer learning with Xception (or Inception, resnet, etc)? :)
", "votes": null, "replies": [] }, { "id": 250103, "author_name": "steelrose", "author_url": "", "post_date": "11/29/2017 19:49:01", "content": "I assume 'tf' stands for tensorflow, but might be wrong :)\nwhen you check the code in https://github.com/fchollet/keras/blob/master/keras/applications/imagenet_utils.py you will see, but basically it's (since 2 days ago, before it was the transformation mentioned by Enhao above):
\nx /= 127.5\nx -= 1.
for a tutorial you can check out this keras blog: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
", "votes": null, "replies": [] }, { "id": 252273, "author_name": "brianlzm", "author_url": "", "post_date": "12/02/2017 16:49:51", "content": "What's different between (img/255-0.5)*2.0 and keras preprocess (x /= 127.5 x -= 1)? I think they're the same thing, why you get higher result with keras preprocess?
", "votes": null, "replies": [] }, { "id": 252279, "author_name": "steelrose", "author_url": "", "post_date": "12/02/2017 16:57:55", "content": "yes, most likely the main difference was coming from the adam optimizer (I did both changes at once)
", "votes": null, "replies": [] } ] }, { "id": 242335, "author_name": "andreanobile", "author_url": "", "post_date": "11/11/2017 09:26:11", "content": "i'm still using a single resnet-18 because of very limited resources. 160x160 random crop in training, from scratch, lr 0.1 -> 0.01 -> 0.001 -> 0.0005 -> 0.0001, changing lr every few epochs, SGD with momentum 0.9, 192 images batches. I have modified the network in the last few conv layers. LB 0.69. inference with only 1 center 160x160 crop.
", "votes": null, "replies": [ { "id": 242363, "author_name": "timjoseph", "author_url": "", "post_date": "11/11/2017 10:45:11", "content": "How did you modify the conv layers? Seems like a pretty good score for resnet-18!\nAny data augmentation? We are still looking for the right kind...
", "votes": null, "replies": [] }, { "id": 242370, "author_name": "andreanobile", "author_url": "", "post_date": "11/11/2017 11:19:51", "content": "it seems to me that there's a kind of flaw in vanilla resnet-18 in the final few stages, if the final classifier has to distinguish among 5270 classes. The final FC layer is just a matrix, it basically can do 2 things: rotate a vector and project or embed the vector in the target space. Embedding in higher dimensional space does not change the dimension of the initial subspace, it cannot give higher resolution. Resnet 18 has a average pooling before the FC, there is too much of a bottlenck before FC. So vanilla resnet 18 cannot get good class resolution beyond a 10-20% of the full set. Unfortunately time is running and i have very old metal.... i'm working on a completely different back-end for my little resnet 18 but not sure if i can make it in time.
", "votes": null, "replies": [] } ] }, { "id": 245946, "author_name": "eachshadow", "author_url": "", "post_date": "11/20/2017 06:28:59", "content": "My resnet50 only got to about 63% accuracy before it started massively overfitting
\n\nEDIT: went back to an earlier epoch and changed the data augmentation, so it got to 65% (with room for improvement) after a couple of epochs with more overfitting than some of my other models, but not so much as before where the validation accuracy was actually decreasing
", "votes": null, "replies": [ { "id": 248763, "author_name": "strideradu", "author_url": "", "post_date": "11/26/2017 22:49:09", "content": "Wondering what is your best single model and its accuracy?
", "votes": null, "replies": [] }, { "id": 248797, "author_name": "eachshadow", "author_url": "", "post_date": "11/27/2017 01:08:15", "content": "My best model gets about 74.8% on the leaderboard by itself. I only have two or three real models trained and ensembling currently isn't getting me very big gains.
", "votes": null, "replies": [] }, { "id": 250479, "author_name": "eachshadow", "author_url": "", "post_date": "11/30/2017 03:38:05", "content": "Update: Resnet50 gets 0.71450 on the leaderboard after 10 epochs
", "votes": null, "replies": [] }, { "id": 250497, "author_name": "strideradu", "author_url": "", "post_date": "11/30/2017 04:03:18", "content": "Just wondering what you have done to boost from 65% to 71%?
", "votes": null, "replies": [] }, { "id": 250520, "author_name": "eachshadow", "author_url": "", "post_date": "11/30/2017 04:34:56", "content": "Thanks! And 65% at 2 epoch is quite high I believed. Would you mind if you can share how do you achieve this? My resnet 101 only have about 58% after two epoch with 0.001 lr
", "votes": null, "replies": [] } ] }, { "id": 249027, "author_name": "skinish", "author_url": "", "post_date": "11/27/2017 15:08:40", "content": "How did you guys perform the train/val split?
", "votes": null, "replies": [] }, { "id": 249807, "author_name": "burgalon", "author_url": "", "post_date": "11/29/2017 08:28:49", "content": "Hi @iFighting, @minu, @Heng CherKeng,\nIt seems like you guys are using MXNet?\nWhat are you impressions comparing to PyTorch?\nDoes MXNet facilitates specific aspects? Were you able to attain better results training with MXNet?
", "votes": null, "replies": [ { "id": 249816, "author_name": "hengck23", "author_url": "", "post_date": "11/29/2017 08:50:38", "content": "i am using pytorch only for this competition. I use caffe, mxnet, tensorflow before. I think their results are all about the same (because they use the \"same formula\" and based on nvidia cudnn). Some DL framework are faster. But overall, I think pytorch is fast and easy to debug and experiment with new structure.
", "votes": null, "replies": [] }, { "id": 250607, "author_name": "ifighting", "author_url": "", "post_date": "11/30/2017 06:20:52", "content": "yes, i use mxnet
", "votes": null, "replies": [] }, { "id": 250633, "author_name": "mwbyeon", "author_url": "", "post_date": "11/30/2017 06:40:21", "content": "I'm just familiar with MXNet. and MXNet is efficient in Multi-GPU.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "233986": "I am not familiar with this type of the train set and this number of target classes. I hope this thread will help to figure out what models perform better and which of them worth considering.\n\nTo start:\n\nResnet 101, no bagging, no TTA\n\n - val_loss = 1.18 \n - val_acc = 0.744 \n - LB = 0.7437", "233991": "thanks a lot!", "233996": "That's quite high score for single model! How long did you train for this and which GPU did you use?\n\nDue to limited resources, I'm using images scaled down to 90x90 and simple Xception, no TTA\n\n - val_acc = .6904 (for each image) \n\n - LB= .7092 (averaged softmax output for all images in same product id)\n\nI'm trying many other approaches like using top 2000 frequent items, using category 1,2 information, etc, but nothing beats this baseline yet. Sad to see much better results with basic model, but thanks for the info.", "233997": "4 x GTX 1080 Ti\n\n18 epochs x 4.5 hours\n\nI have 0.69 after 7th epoch", "234000": "seems that we need long epoches!", "234005": "Amazing score. Did you do training without pretrained weight?\n\nMy results: \ninception_v3, 160 crop, val loss: 1.78~, val acc: 0.66~, lb: 0.673~", "234006": "i am also using 4 x GTX 1080 Ti. For resnext101 (about same flops as your resnet101 ), 1 epoch=12283645 images takes about 1.3 hr. I am using 180x180 as input. I suppose your input size is not 180x180?", "234007": "what does TTA mean?", "234010": "I use 160x160 crops.\n\n1.3hrs per epoch.... It looks like I am doing something wrong.\n\nIt may happen that my SSD is not fast enough and I need to buy faster one.", "234012": "TTA= test time augmentation (e.g. multi crops, resize in test time)", "234013": "Test-time augmentation. In my case, I can get extra 0.01 just with 12-way TTA(flip, shift up/down, etc) with exact same weight - almost free cake :)", "234019": "I used pre-trained. I would be also interested how long will it take to get the same result without pre-trained, but it may be too computationally expensive.", "234026": "I used resnext-101, with image centor croped to 128X128, used pre-trained. It can get 0.59 val _acc after 1 epoches. I used sgd optimizer. Training is going...", "234032": "i am sorry that i made a mistake. 1 epoch=12283645 images takes about 13 hr, with 180x180 as input. So your timing may be reasonable. \n\nAlso speed of resnext and resnet are not the same even if they have the same flops, https://github.com/Cadene/pretrained-models.pytorch/issues/8\n\nI suspect it is due to group convolution in pytorch implementation ,\nhttps://discuss.pytorch.org/t/does-pytorch-optimize-the-group-parameter-in-convs/3060\n\nWhat’s New in cuDNN 7?\nGrouped Convolutions for models such as ResNeXt and Xception and CTC (Connectionist Temporal Classification) loss layer for temporal classification\n\nhttps://developer.nvidia.com/cudnn", "234047": "I'm just wondering, how many layers are actually training in your 101 layers net? For me, training one more block of Xception takes me 2 more hours for one epoch. (single GTX 1080)", "234061": "I use single ResNet 50 and my current LB-score is 0.45813, local validation score is 0.4914.", "234093": "SE-ResNet-50:\n\n - finetuning about 7 epochs (about 16 hours per epoch on 2x1080)\n - train augmentation: 161x161 random crops + horizontal flips\n - val single crop accuracy: 70.96\n - val 10 crops accuracy: 71.79\n - LB 10 crops accuracy: 71.67", "234099": "thanks for the information! What is the learning rate you used for finetunning?", "234101": "What's your learning rate schedule/optimize?", "234102": "I am using SGD with Nesterov momentum (0.9) and batch_size=256\n\n - epochs 1..5: 0.001\n - epoch 6: 0.0001\n - epoch 7: 0.00001", "234106": "Grouped Convs seem to be supported in the current master of Pytorch: https://github.com/pytorch/pytorch/pull/3057", "234111": "i think this is spatial, where group = in_planes. It improve mobilenet but not Resnext, where group=32 or 64.", "234115": "Vladimir Iglovikov\n \nI rewrite my code an i get the following results:\n\n - System: 4x1080Ti/pytorch\n\n - input =160x160\n\n - network = resnet101\n\n - batch = 256\n\n - time for 1000 iterations = 1000*256 images = 7 min.\n\nif one epoch = 12283645 images, i am getting time per epoch = 336 min = 5.6 hr.\n\nhow many images are there in one epoch for your timing of 4.5hr? Thanks a lot!\n\n(I am trying to see if my system has been setup correctly. This is the first time i use multi-gpu training intensively)", "234137": "one epoch 4.5 hours? why so fast?", "234151": "Resnet-18: val accuracy 0.63 trained for 6 epochs, simple SGD, lr 0.1 -> 0.01 -> 0.001 , change lr after 2 epochs, no data augmentation no crop, 180x180, just change FC layer to get the 5270 classes and train", "234175": "One epoch - 11738624 Images. Rest are used for validation. \n\nBatch_size = 512", "234204": "Under the same setting (batch=512), i am able to get training iteration of 40,526 images per minute. For an epoch of 11,738,624 it would take 282 min or 4.7 hr, which is close to your results. Thanks again!", "234294": "Some useful info in this thread. Thanks all. I'm curious what people are settling on for training augmentation? I adapted some training code used in previous challenges last week and set some models training. I've got one model moving past the mid 60s in validation error and another that got stuck in the mid 50s. \n\nThe model that got stuck was high capacity but I had colour/saturation augmentation enabled. The one that's still chugging along and doing much better only had random crop/scale and rotation. I'm guessing the prevalence of almost constant black or white backgrounds in most images means that adding variability to that requires much more learning than leaving it be.\n\nNext angle is to see if disabling the random crop and just leaving h-flip and a bit of rotation on does better... basically assuming that the preprocessing of the competition dataset leaves things relatively well centred and similarly scaled across train and test datasets. Has anyone hit close to .7 or better with almost no augmentation?", "234309": "Are you training all params in your net?", "234997": "here is a possible Cdiscount in-house baseline results:\nhttps://www.linkedin.com/in/zhiwei-li-81b444126/\n\nApr 2017 – Sep 2017 Employment Duration 6 mos\n\nLocationBordeaux Area, France\n\nLarge Scale Image Classification for e-business product categorization\n\n• Research work on different Deep Learning architectures like Inception, ResNet etc.\n\n• Worked on 20 million images in roughly 9000 unbalanced class for product categories\n\n• Implemented transfer learning based on Inception V3 model using TensorFlow and Keras\n\n• Designed a quick model construction pipeline, raised ~60 times efficacy, achieved 72.3% accuracy\n\n• This model will be used to improve the categorization ability of current model", "235034": "> 9000 unbalanced classes \n\nThis probably makes the problem harder.", "235081": "> This probably makes the problem harder.\n\n\n\n\nOr you just caught a resume 'embellishment' gone wrong.. \nThat's linkedin.", "235885": "when i test with renet v2 101 one epoch (train:9 val:1)