** important **\nplease refer to @Vladimir Iglovikov at https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/41652
\n\nplease train with long epoch to get better results . My models results are from training only up to 4 to 5 epoch.
\n\nThis is a pytorch starter kit. It is not completed yet. I use ideas from the kernels and discussion contributions. Thanks to kagglers, especially @Bruno G. do Amaral, @Human Analog, @Adam Blazek for code and discussion.
\n\nDownload: \nhttps://drive.google.com/drive/folders/0B_DICebvRE-kb2dFd2FKX1hfRkE?usp=sharing
\n\nversion.10-03:
\n\nversion.10-07:
\n\nLB 0.63358(160x160 single center crop) or 0.63839(180x180) or 0.64356 (180x180 resize to 160x160) for se-resnet50 trained on 160x160 crops.
LB 0.66708 for same se-resnet50 if train=224x224 crops from 256, test = resize to 224
LB 0.68939 for same se-resnet50 if train=180x180, test = 180x180 (weight initialised from 224 se-resnet50 of above)
LB 0.69565 for inception3 if train=180x180, test = 180x180 (be careful on how to control the lr, batch size, momentum, level of augmentation,etc. I will have a writeup on this later)
support various network like resnet, resnext, inception v3, etc. see trainer_xxx.py
support gradient accumulation
note: code is messy and dirty. It may not be backward- compatible.
\n\nversion.10-17 (latest):
\n\nadvance version. implement multi-gpu support in training. see \"trainer_excited_resnext101_32x4d.py\"
LB 0.71064 for se-resnext101_32x4d (single crop 180/180)
support xception, se-xception, resnext101_32x4d, se-resnext101_32x4d, inceptionv4, inception_resnetv2
folder \"10-17/senet_conversion\" contains conversion tools for converting caffe model to pytorch
note:
\n\n1. may not be backward compatible
\n\n2. old training scripts are found in \"dummy-00/temp/old_trainer\"
\nnext version:
focal loss for class imbalance
super large batch size to speed up training
efficient multi-crop for testing
divide test/train data into easy and difficult set for faster testing/training. class balancing
Note:
\n\nMy software and thread will be constantly update. Also, ther great works:
\n\nhttps://www.kaggle.com/blazeka/multi-gpu-tensorflow-convnet-0-65/notebook
https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40715
** important **\nplease refer to @Vladimir Iglovikov at https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/41652
\n\nplease train with long epoch to get better results . My models results are from training only up to 4 to 5 epoch.
\n\nThis is a pytorch starter kit. It is not completed yet. I use ideas from the kernels and discussion contributions. Thanks to kagglers, especially @Bruno G. do Amaral, @Human Analog, @Adam Blazek for code and discussion.
\n\nDownload: \nhttps://drive.google.com/drive/folders/0B_DICebvRE-kb2dFd2FKX1hfRkE?usp=sharing
\n\nversion.10-03:
\n\nversion.10-07:
\n\nLB 0.63358(160x160 single center crop) or 0.63839(180x180) or 0.64356 (180x180 resize to 160x160) for se-resnet50 trained on 160x160 crops.
LB 0.66708 for same se-resnet50 if train=224x224 crops from 256, test = resize to 224
LB 0.68939 for same se-resnet50 if train=180x180, test = 180x180 (weight initialised from 224 se-resnet50 of above)
LB 0.69565 for inception3 if train=180x180, test = 180x180 (be careful on how to control the lr, batch size, momentum, level of augmentation,etc. I will have a writeup on this later)
support various network like resnet, resnext, inception v3, etc. see trainer_xxx.py
support gradient accumulation
note: code is messy and dirty. It may not be backward- compatible.
\n\nversion.10-17 (latest):
\n\nadvance version. implement multi-gpu support in training. see \"trainer_excited_resnext101_32x4d.py\"
LB 0.71064 for se-resnext101_32x4d (single crop 180/180)
support xception, se-xception, resnext101_32x4d, se-resnext101_32x4d, inceptionv4, inception_resnetv2
folder \"10-17/senet_conversion\" contains conversion tools for converting caffe model to pytorch
note:
\n\n1. may not be backward compatible
\n\n2. old training scripts are found in \"dummy-00/temp/old_trainer\"
\nnext version:
focal loss for class imbalance
super large batch size to speed up training
efficient multi-crop for testing
divide test/train data into easy and difficult set for faster testing/training. class balancing
Note:
\n\nMy software and thread will be constantly update. Also, ther great works:
\n\nhttps://www.kaggle.com/blazeka/multi-gpu-tensorflow-convnet-0-65/notebook
https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40715
Welcome Heng in this new competition. I am quite sure that this one is also going to be very tight competitive one because of your generous contributions towards the solution. Keep it up. However, I think, in general, this competition will give an advantage to those who have access to more GPU resources. Otherwise, it will be interesting to see how people come up with ingenious ideas to deal with this large dataset. Good luck for a gold medal this time!
", "rawMarkdown": "Welcome Heng in this new competition. I am quite sure that this one is also going to be very tight competitive one because of your generous contributions towards the solution. Keep it up. However, I think, in general, this competition will give an advantage to those who have access to more GPU resources. Otherwise, it will be interesting to see how people come up with ingenious ideas to deal with this large dataset. Good luck for a gold medal this time!", "votes": null }, { "id": "227387", "postDate": "10/04/2017 08:37:07", "content": "Trained more iterations to get LB 58.6% (12 hr). initial observations:
\n\nToo much layers are frozen. Next step is to retrain/finetune with more parameters.
I estimate the results of single crop resnet50 is around LB = 62 to 65%
I estimate the leader board top score by the kagglers to be near to 82% at the end of the competition.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/227387/7469/loss.png\" "\\"\\"")
** experiments on pretrain/freezing, learning rates. etc **
\n\nbaseline results for resnet50 LB = 0.62978 :
\n\ntrain all layers (no freezing)
rates:
\n\n0.01 for epoch 1,2
0.001 for epoch 3
0.0001 till 3.25
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/227862/7495/loss_full.png\" "\\"\\"")
Hi Heng, good to see you in this competition again.\nIs there a reason you don't use github instead of Google Drive for your code?
", "rawMarkdown": "Hi Heng, good to see you in this competition again.\nIs there a reason you don't use github instead of Google Drive for your code?", "votes": null }, { "id": "227928", "postDate": "10/05/2017 13:41:50", "content": "i have an github account too. But i find it difficult to update because my code organisation is messy and file size limit. Maybe i will try to learn the pycharm github integration or use some friendly github gui tool one day.
", "rawMarkdown": "i have an github account too. But i find it difficult to update because my code organisation is messy and file size limit. Maybe i will try to learn the pycharm github integration or use some friendly github gui tool one day.", "votes": null }, { "id": "227935", "postDate": "10/05/2017 13:57:29", "content": "I think for public projects like yours it would be much better to split your project into source code and files. So you can still host the big files on Google Drive, but everything related to source code on github.\nThere are three major advantages: 1. Everyone can easily read the code without downloading everything. 2. If you use readable commit messages you create an implicit change log. 3. Issue tracker + Pull requests can make this a community effort.
\n\nThat said, I appreciate your effort no matter where all the stuff is hosted ;D
", "rawMarkdown": "I think for public projects like yours it would be much better to split your project into source code and files. So you can still host the big files on Google Drive, but everything related to source code on github.\nThere are three major advantages: 1. Everyone can easily read the code without downloading everything. 2. If you use readable commit messages you create an implicit change log. 3. Issue tracker + Pull requests can make this a community effort.\n\nThat said, I appreciate your effort no matter where all the stuff is hosted ;D", "votes": null }, { "id": "227946", "postDate": "10/05/2017 14:09:20", "content": "Good suggestion! I will get it done when i have time :)
", "rawMarkdown": "Good suggestion! I will get it done when i have time :)", "votes": null }, { "id": "228236", "postDate": "10/06/2017 07:14:11", "content": "** experiments on input sizes, network strides. etc **
\n\n224x224 (bilinear up-scale) input vs 160x160 input for resnet50. \n ![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/228236/7507/Slide2.png\" "\\"\\"")
To get top results, you should take care of the issues:
\n\nbalancing class
small objects
how to train efficiency (extract information efficiency, not all data provide \"useful\" information)
break down the problem into simpler ones (e.g. special handling for minority class)
... to be updated ...
", "rawMarkdown": "To get top results, you should take care of the issues:\n\n 1. balancing class \n\n 2. small objects\n\n 3. how to train efficiency (extract information efficiency, not all data provide \"useful\" information) \n\n 4. break down the problem into simpler ones (e.g. special handling for minority class)\n\n... to be updated ...", "votes": null }, { "id": "228255", "postDate": "10/06/2017 08:08:56", "content": "** experiments on different network structure **
\n\n...this is still in progress ...
\n\nbased on initial results it seems that any of the large network (those that obtained about 20% top1 error on imagenet like resnext, inceptionv3, resnet, etc) can get about 67to 69% for single crop on cdiscount image. With proper\n ensemble, 70 to 72% is easily obtainable. Care has to be taken in augmentation, input resolution, learning rate, batch size, etc. I estimate top 50 ranks results >72% at the end of the competitions if kagglers have enough gpu.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/228255/7517/large_input_nets2.png\" "\\"\\"")
\n ![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/228255/7536/batch_size1.png\" "\\"\\"")
It seems that data augmentation can improve balancing class.
", "rawMarkdown": "It seems that data augmentation can improve balancing class.", "votes": null }, { "id": "228661", "postDate": "10/07/2017 12:05:50", "content": "compare resnet50 and se-resnet50
\n\n(there is a typo mistake in the figure below. the LB results should be flipped, i.e. SE-resnet is better)
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/228661/7515/se-resnet50.png\" "\\"\\"")
** experiments on augmentation **
\n\n... to be updated ...
\n\nseems that crop 160x160 from 180x180 is not a good way to augment. I suspect downsize 180x180 to 160x160 gives better results. But I do not have time or extra gpu to do this experiments. anyone has done this comparison before?
", "rawMarkdown": "** experiments on augmentation **\n\n... to be updated ...\n\nseems that crop 160x160 from 180x180 is not a good way to augment. I suspect downsize 180x180 to 160x160 gives better results. But I do not have time or extra gpu to do this experiments. anyone has done this comparison before?", "votes": null }, { "id": "228735", "postDate": "10/07/2017 17:52:09", "content": "Heng, how did you create pretrain_convert_table.py and what weights are you loading?
", "rawMarkdown": "Heng, how did you create pretrain_convert_table.py and what weights are you loading?", "votes": null }, { "id": "228777", "postDate": "10/07/2017 20:27:58", "content": "this is for resnet50. the weights are actaully the same as the default pytorch model zoo. i reorganize resnet for future experiments and the naming changed. i manually create the mapping for the model zoo keys to my keys. actually you can just use the default pytorch model zoo and ignore my resnet_xx.py
", "rawMarkdown": "this is for resnet50. the weights are actaully the same as the default pytorch model zoo. i reorganize resnet for future experiments and the naming changed. i manually create the mapping for the model zoo keys to my keys. actually you can just use the default pytorch model zoo and ignore my resnet_xx.py", "votes": null }, { "id": "228890", "postDate": "10/08/2017 07:26:43", "content": "Hi, Heng, I use similar way to load data as you, converting bson to files.
\n\nThe loading process is very fast at first(0.3s, CPU 30%, RAM 10% of 30G), but some iterations later, it becomes very slow(6s, CPU 5%, RAM 10% of 30G).
\n\nI set the CPU mode to performance, and try different num_workers(0,4,8,12 …), but the situation stays the same.
\n\nHere is my code:
\n\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\n\n\nclass xiongDataset(Dataset):\n def __init__(self, csv_file, root_dir, transform=None):\n self.train_names=[]\n self.root_dir=root_dir\n self.transform = transform\n train_images = pd.read_csv(csv_file)\n train_ids = list(train_images['product_id'])\n train_idxs = list(train_images['img_idx'])\n self.labels = list(train_images['category_idx'])\n num_train = len(train_images)\n for i in range(num_train):\n train_name = '{}_{}.jpg'.format(train_ids[i],train_idxs[i])\n self.train_names.append(train_name)\n\n def __len__(self):\n return len(self.train_names)\n\n def __getitem__(self, idx):\n img = cv2.imread(self.root_dir + self.train_names[idx])\n label = self.labels[idx]\n if self.transform is not None:\n img = self.transform(img)\n\n return img,label\n\ntrain_data = xiongDataset('../train_images.csv','../train/',transform=transforms.ToTensor())\n\n\n\ndata_loader= DataLoader(train_data,batch_size=256,shuffle=False,num_workers=0)\ni use ssd drive (solid state drive). if you are using regular hard disk, you need a better random access database format like leveldb, hdfs i think.
\n\nwhen i use regular hard disk, i face the same problem you said.
", "rawMarkdown": "i use ssd drive (solid state drive). if you are using regular hard disk, you need a better random access database format like leveldb, hdfs i think.\n\nwhen i use regular hard disk, i face the same problem you said.", "votes": null }, { "id": "228892", "postDate": "10/08/2017 07:51:44", "content": "Thank you :)
", "rawMarkdown": "Thank you :)", "votes": null }, { "id": "228972", "postDate": "10/08/2017 12:42:45", "content": "Thank you for your advice. I do as what you said, and make it. Now, loading data only takes less than 0.08 seconds, and one iteration takes 0.3 seconds(batch size=256, Resnet18, a 1080Ti).
", "rawMarkdown": "Thank you for your advice. I do as what you said, and make it. Now, loading data only takes less than 0.08 seconds, and one iteration takes 0.3 seconds(batch size=256, Resnet18, a 1080Ti).", "votes": null }, { "id": "229345", "postDate": "10/09/2017 12:28:08", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": null }, { "id": "229987", "postDate": "10/11/2017 00:47:45", "content": "Hi, Heng, have you tried multi-gpu? I also train with pytorch, and I set model = nn.DataParallel(model, device_ids=[0, 1]), but there is several errors https://discuss.pytorch.org/t/multi-gpu-error-the-accuracy-of-one-gpu-is-much-lower-than-two-gpus/8492
And then I tried your way os.environ[\"CUDA_VISIBLE_DEVICES\"] = '0,1'(I have 2 GPUs), but only one GPU works.
haven't try yet. multi-gpu development would come later
", "rawMarkdown": "haven't try yet. multi-gpu development would come later", "votes": null }, { "id": "230220", "postDate": "10/11/2017 14:36:31", "content": "I have encountered same problem by using ssd. Probably it occurred due to GC, so adding a line gc.disable() in code is a way (I am scared that it may cause some bad effect...).
How many GPUs do you have? I remember you have 4 Titan X? Just want to compare the running time.
", "rawMarkdown": "How many GPUs do you have? I remember you have 4 Titan X? Just want to compare the running time.", "votes": null }, { "id": "230391", "postDate": "10/11/2017 21:25:24", "content": "Hello Heng,
\n\nThanks a lot for sharing your scripts and models so generously! I am struggling to break through 50% acc so going through your scripts will be very helpful to find the issues in my pipeline.
\n\nI have a few questions on your train & valid split:
\n\nvalidation set is not 5k. it is 50k.
\n\nthe 5k is used in the training iteration is a subset of validation to make \"visualisation\" fast.
\n\nthay are choosen randomly. (not the best way)
\n\nI will come back to data sampling later. now i am still at early stage of experiments
", "rawMarkdown": "validation set is not 5k. it is 50k.\n\nthe 5k is used in the training iteration is a subset of validation to make \"visualisation\" fast.\n\nthay are choosen randomly. (not the best way)\n\nI will come back to data sampling later. now i am still at early stage of experiments", "votes": null }, { "id": "230397", "postDate": "10/11/2017 21:50:04", "content": "i am using 1 pascal titanx and 3 1080 ti. but each gpu is use to train one model. i haven't train across multiple gpu yet. i am still \"exploring\" different models
", "rawMarkdown": "i am using 1 pascal titanx and 3 1080 ti. but each gpu is use to train one model. i haven't train across multiple gpu yet. i am still \"exploring\" different models", "votes": null }, { "id": "230576", "postDate": "10/12/2017 08:54:40", "content": "Can't find pretrained_file: /root/share/data/models/pytorch/imagenet/resenet/resnet50-19c8e357.pth
", "rawMarkdown": "Can't find pretrained_file: /root/share/data/models/pytorch/imagenet/resenet/resnet50-19c8e357.pth", "votes": null }, { "id": "230577", "postDate": "10/12/2017 08:56:13", "content": "", "rawMarkdown": "https://www.google.com.sg/search?q=resnet50-19c8e357.pth&oq=resnet50-19c8e357.pth&aqs=chrome..69i57.1903j0j4&sourceid=chrome&ie=UTF-8", "votes": null }, { "id": "230596", "postDate": "10/12/2017 09:52:49", "content": "How to support multi-gpu ? According to https://discuss.pytorch.org/t/does-it-support-multi-gpu-card-on-a-single-node/75/7 this shouldn't be too difficult ?
", "rawMarkdown": "How to support multi-gpu ? According to [https://discuss.pytorch.org/t/does-it-support-multi-gpu-card-on-a-single-node/75/7][1] this shouldn't be too difficult ?\n\n\n [1]: https://discuss.pytorch.org/t/does-it-support-multi-gpu-card-on-a-single-node/75/7", "votes": null }, { "id": "230603", "postDate": "10/12/2017 10:24:25", "content": "Have you tried multi-gpu? I tried two gpus, but there are several errors https://discuss.pytorch.org/t/multi-gpu-error-the-accuracy-of-one-gpu-is-much-lower-than-two-gpus/8492
", "rawMarkdown": "Have you tried multi-gpu? I tried two gpus, but there are several errors https://discuss.pytorch.org/t/multi-gpu-error-the-accuracy-of-one-gpu-is-much-lower-than-two-gpus/8492", "votes": null }, { "id": "231533", "postDate": "10/15/2017 09:13:01", "content": "Is the 10-7 release the lastest code?
\n\nprobs = F.softmax(logits)\nloss = F.cross_entropy(logits, labels)\nThis seems wrong, since crossentropy already includes log_softmax and NLLLoss!
", "rawMarkdown": "Is the 10-7 release the lastest code?\n\n probs = F.softmax(logits)\n loss = F.cross_entropy(logits, labels)\n\nThis seems wrong, since crossentropy already includes log_softmax and NLLLoss!", "votes": null }, { "id": "231535", "postDate": "10/15/2017 09:16:20", "content": "should be correct
\n\nloss = F.cross_entropy(logits, labels)
\n\nthe input is logits, not probs
\n\nthe loss statement is not dependent on the prob statement.
", "rawMarkdown": "should be correct\n\nloss = F.cross_entropy(logits, labels)\n\nthe input is logits, not probs\n\nthe loss statement is not dependent on the prob statement.", "votes": null }, { "id": "231537", "postDate": "10/15/2017 09:20:23", "content": "Oh wow. You are totally right, so stupid <--
", "rawMarkdown": "Oh wow. You are totally right, so stupid <--", "votes": null }, { "id": "231550", "postDate": "10/15/2017 10:37:48", "content": "I'm curious about your model, how do you train your model, how do you predict the image in one crop
", "rawMarkdown": "I'm curious about your model, how do you train your model, how do you predict the image in one crop", "votes": null }, { "id": "232860", "postDate": "10/18/2017 16:08:33", "content": "results of se-resnext101. This is trained with the latest version 10-17 of the pytorch starter kit above. You need multi-gpu to use this (or it would be really slow). LB 0.71064 for se-resnext101_32x4d (single crop 180/180)
\n\nI try several sets of parameters, etc and finally it works! if the parameters are not correct the loss won't drop. If the parameters are correct, you will see the loss drop slowly over several epoch.
\n\nif se-resnext101 works, senet should be better: https://github.com/hujie-frank/SENet
\n\nmy next plan would be senet or larger (like se-resnext269)
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/232860/7697/se-resnext101.png\" "\\"\\"")
Do you use the weights from resneXt pretrained on imagenet or did you map the weights from the original se-resneXt to your own pytorch implementation?
", "rawMarkdown": "Do you use the weights from resneXt pretrained on imagenet or did you map the weights from the original se-resneXt to your own pytorch implementation?", "votes": null }, { "id": "233048", "postDate": "10/19/2017 03:34:54", "content": "imagenet pretrained weights for resnext and se-resnext can be downloaded. i think there are no difference if you start off with \"resnext imagenet weights --> resnext weights --> se-resnext weights\" or \"se-resnext imagenet weights --> se-resnext weights\" .
\n\nfor the graph above, i am using \"resnext imagenet weights --> resnext weights --> se-resnext weights\".
\n\nbut for the new experiment on senet (which is se-resnext152) i am using \"se-resnext imagenet weights --> se-resnext weights\" directly.
\n\nboth seems to work well
", "rawMarkdown": "imagenet pretrained weights for resnext and se-resnext can be downloaded. i think there are no difference if you start off with \"resnext imagenet weights --> resnext weights --> se-resnext weights\" or \"se-resnext imagenet weights --> se-resnext weights\" .\n\nfor the graph above, i am using \"resnext imagenet weights --> resnext weights --> se-resnext weights\".\n\nbut for the new experiment on senet (which is se-resnext152) i am using \"se-resnext imagenet weights --> se-resnext weights\" directly.\n\nboth seems to work well", "votes": null }, { "id": "233073", "postDate": "10/19/2017 05:23:22", "content": "It seems that pytorch only has resnet imagenet weights. Did you only use that to fine-tune resnext or se-resnext?
", "rawMarkdown": "It seems that pytorch only has resnet imagenet weights. Did you only use that to fine-tune resnext or se-resnext?", "votes": null }, { "id": "233074", "postDate": "10/19/2017 05:34:34", "content": "I write a converter to convert from caffe to pytorch
", "rawMarkdown": "I write a converter to convert from caffe to pytorch", "votes": null }, { "id": "233091", "postDate": "10/19/2017 07:36:15", "content": "i added a conversion to for snet (se-resnext152) in folder \"10-17/senet_conversion\". you can modify this for se-resnext101
", "rawMarkdown": "i added a conversion to for snet (se-resnext152) in folder \"10-17/senet_conversion\". you can modify this for se-resnext101", "votes": null }, { "id": "233106", "postDate": "10/19/2017 08:19:40", "content": "what about the performance of the se-resnext?
", "rawMarkdown": "what about the performance of the se-resnext?", "votes": null }, { "id": "233115", "postDate": "10/19/2017 08:56:28", "content": "how is your optimazer? sgd? or any other?
", "rawMarkdown": "how is your optimazer? sgd? or any other?", "votes": null }, { "id": "234459", "postDate": "10/23/2017 11:37:28", "content": "My linux rig reboots few minutes after I run the script using 2 GPUs. Any idea on the cause and how to resolve it? Thanks!
\n\nAnd thanks again for sharing the scripts!
", "rawMarkdown": "My linux rig reboots few minutes after I run the script using 2 GPUs. Any idea on the cause and how to resolve it? Thanks!\n\nAnd thanks again for sharing the scripts!", "votes": null }, { "id": "234474", "postDate": "10/23/2017 12:22:01", "content": "Maybe it's overheating?
", "rawMarkdown": "Maybe it's overheating?", "votes": null }, { "id": "234518", "postDate": "10/23/2017 14:26:48", "content": "Thanks for your reply Tim. The GPUs are running 2 scripts separately and seems fine. In this case do you think overheating is still a probable cause? Thanks!
", "rawMarkdown": "Thanks for your reply Tim. The GPUs are running 2 scripts separately and seems fine. In this case do you think overheating is still a probable cause? Thanks!", "votes": null }, { "id": "234524", "postDate": "10/23/2017 14:37:24", "content": "Hi Heng,\nFirst thank you for sharing all the knowledge and information. I enjoy reading your posts.\nI was wondering, why in your do_submit() code, do you split the test set?
", "rawMarkdown": "Hi Heng,\nFirst thank you for sharing all the knowledge and information. I enjoy reading your posts.\nI was wondering, why in your do_submit() code, do you split the test set?", "votes": null }, { "id": "234526", "postDate": "10/23/2017 14:43:25", "content": "I can run in 4 separate machines. In case of running in a single machine, i don't have to start from zero if the testing stop due to some error. Lastly, the saved npy files are smaller , making them faster to read and write
", "rawMarkdown": "I can run in 4 separate machines. In case of running in a single machine, i don't have to start from zero if the testing stop due to some error. Lastly, the saved npy files are smaller , making them faster to read and write", "votes": null }, { "id": "234532", "postDate": "10/23/2017 14:57:20", "content": "Just monitor it with nvidia-smi. Or look in your logs what could have caused the error!
", "rawMarkdown": "Just monitor it with nvidia-smi. Or look in your logs what could have caused the error!", "votes": null }, { "id": "237449", "postDate": "10/30/2017 10:44:39", "content": "Hi Heng, any chance that you could further share how to get xception.keras.convert.pth? Thanks!
It works by reducing the batch size. Maybe it's really due to overheating. Thanks!
", "rawMarkdown": "It works by reducing the batch size. Maybe it's really due to overheating. Thanks!", "votes": null }, { "id": "237466", "postDate": "10/30/2017 11:29:21", "content": "Oh, did you also check whether you have a memory leak? Higher batch size means more memory is needed (I am speaking about main memory, not GPU memory here)
", "rawMarkdown": "Oh, did you also check whether you have a memory leak? Higher batch size means more memory is needed (I am speaking about main memory, not GPU memory here)", "votes": null }, { "id": "237987", "postDate": "10/31/2017 13:22:52", "content": "Oh may I know how I can check this?
", "rawMarkdown": "Oh may I know how I can check this?", "votes": null }, { "id": "242305", "postDate": "11/11/2017 05:30:23", "content": "Wondering if you find where to got the xception pretrained model?
", "rawMarkdown": "Wondering if you find where to got the xception pretrained model?", "votes": null }, { "id": "242727", "postDate": "11/12/2017 12:54:58", "content": "Sorry no luck...
", "rawMarkdown": "Sorry no luck...", "votes": null }, { "id": "242733", "postDate": "11/12/2017 13:13:33", "content": "xception.keras.convert.pth is converted by myself. However, conversion from keras to pytorch is not correct. I don't know why, but could be due to difference in padding, ceil/floor in max pooling.
\n\nnevertheless, the wrong converted weights seems to be usable for initialization for cdisount image training.
", "rawMarkdown": "xception.keras.convert.pth is converted by myself. However, conversion from keras to pytorch is not correct. I don't know why, but could be due to difference in padding, ceil/floor in max pooling.\n\nnevertheless, the wrong converted weights seems to be usable for initialization for cdisount image training.", "votes": null }, { "id": "242968", "postDate": "11/13/2017 04:05:20", "content": "Have you check this https://github.com/ysh329/deep-learning-model-convertor
", "rawMarkdown": "Have you check this https://github.com/ysh329/deep-learning-model-convertor", "votes": null }, { "id": "249030", "postDate": "11/27/2017 15:18:08", "content": "How did you guys perform the train/val split?
", "rawMarkdown": "How did you guys perform the train/val split?", "votes": null }, { "id": "251376", "postDate": "12/01/2017 04:29:05", "content": "Has anyone had this error training the Resnet101?
\n\n![\"problem\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/251376/7996/Screen%20Shot%202017-11-30%20at%2010.28.28%20PM.png\" "\\"\\"")
what is src in your code?
", "rawMarkdown": "what is src in your code?", "votes": null }, { "id": "251630", "postDate": "12/01/2017 13:59:19", "content": "Nevermind it looks like some of my images are missing.
", "rawMarkdown": "Nevermind it looks like some of my images are missing.", "votes": null }, { "id": "327935", "postDate": "05/13/2018 01:02:58", "content": "Hi, Heng. Did you save the data(.bson or .jpg)? The data has been removed after the competition, but I still want to use several images(Non-commercial). Could you please share it? Thank you~
", "rawMarkdown": "Hi, Heng. Did you save the data(.bson or .jpg)? The data has been removed after the competition, but I still want to use several images(Non-commercial). Could you please share it? Thank you~", "votes": null }, { "id": "2162194", "postDate": "02/28/2023 03:37:00", "content": "this is very handy, thank you for your work!
", "rawMarkdown": "this is very handy, thank you for your work!", "votes": null } ], "comments": [ { "id": 2162194, "author_name": "ianmoonee", "author_url": "", "post_date": "02/28/2023 03:37:00", "content": "this is very handy, thank you for your work!
", "votes": null, "replies": [] }, { "id": 227081, "author_name": "sgalib", "author_url": "", "post_date": "10/03/2017 17:19:22", "content": "Welcome Heng in this new competition. I am quite sure that this one is also going to be very tight competitive one because of your generous contributions towards the solution. Keep it up. However, I think, in general, this competition will give an advantage to those who have access to more GPU resources. Otherwise, it will be interesting to see how people come up with ingenious ideas to deal with this large dataset. Good luck for a gold medal this time!
", "votes": null, "replies": [] }, { "id": 227387, "author_name": "hengck23", "author_url": "", "post_date": "10/04/2017 08:37:07", "content": "Trained more iterations to get LB 58.6% (12 hr). initial observations:
\n\nToo much layers are frozen. Next step is to retrain/finetune with more parameters.
I estimate the results of single crop resnet50 is around LB = 62 to 65%
I estimate the leader board top score by the kagglers to be near to 82% at the end of the competition.
\n\n
** experiments on pretrain/freezing, learning rates. etc **
\n\nbaseline results for resnet50 LB = 0.62978 :
\n\ntrain all layers (no freezing)
rates:
\n\n0.01 for epoch 1,2
0.001 for epoch 3
0.0001 till 3.25
Hi Heng, good to see you in this competition again.\nIs there a reason you don't use github instead of Google Drive for your code?
", "votes": null, "replies": [ { "id": 227928, "author_name": "hengck23", "author_url": "", "post_date": "10/05/2017 13:41:50", "content": "i have an github account too. But i find it difficult to update because my code organisation is messy and file size limit. Maybe i will try to learn the pycharm github integration or use some friendly github gui tool one day.
", "votes": null, "replies": [] }, { "id": 227935, "author_name": "timjoseph", "author_url": "", "post_date": "10/05/2017 13:57:29", "content": "I think for public projects like yours it would be much better to split your project into source code and files. So you can still host the big files on Google Drive, but everything related to source code on github.\nThere are three major advantages: 1. Everyone can easily read the code without downloading everything. 2. If you use readable commit messages you create an implicit change log. 3. Issue tracker + Pull requests can make this a community effort.
\n\nThat said, I appreciate your effort no matter where all the stuff is hosted ;D
", "votes": null, "replies": [] }, { "id": 227946, "author_name": "hengck23", "author_url": "", "post_date": "10/05/2017 14:09:20", "content": "Good suggestion! I will get it done when i have time :)
", "votes": null, "replies": [] } ] }, { "id": 228236, "author_name": "hengck23", "author_url": "", "post_date": "10/06/2017 07:14:11", "content": "** experiments on input sizes, network strides. etc **
\n\n224x224 (bilinear up-scale) input vs 160x160 input for resnet50. \n
To get top results, you should take care of the issues:
\n\nbalancing class
small objects
how to train efficiency (extract information efficiency, not all data provide \"useful\" information)
break down the problem into simpler ones (e.g. special handling for minority class)
... to be updated ...
", "votes": null, "replies": [ { "id": 228593, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/07/2017 06:42:07", "content": "It seems that data augmentation can improve balancing class.
", "votes": null, "replies": [] } ] }, { "id": 228255, "author_name": "hengck23", "author_url": "", "post_date": "10/06/2017 08:08:56", "content": "** experiments on different network structure **
\n\n...this is still in progress ...
\n\nbased on initial results it seems that any of the large network (those that obtained about 20% top1 error on imagenet like resnext, inceptionv3, resnet, etc) can get about 67to 69% for single crop on cdiscount image. With proper\n ensemble, 70 to 72% is easily obtainable. Care has to be taken in augmentation, input resolution, learning rate, batch size, etc. I estimate top 50 ranks results >72% at the end of the competitions if kagglers have enough gpu.
\n\n\n
compare resnet50 and se-resnet50
\n\n(there is a typo mistake in the figure below. the LB results should be flipped, i.e. SE-resnet is better)
\n\n
** experiments on augmentation **
\n\n... to be updated ...
\n\nseems that crop 160x160 from 180x180 is not a good way to augment. I suspect downsize 180x180 to 160x160 gives better results. But I do not have time or extra gpu to do this experiments. anyone has done this comparison before?
", "votes": null, "replies": [] }, { "id": 228735, "author_name": "timjoseph", "author_url": "", "post_date": "10/07/2017 17:52:09", "content": "Heng, how did you create pretrain_convert_table.py and what weights are you loading?
", "votes": null, "replies": [ { "id": 228777, "author_name": "hengck23", "author_url": "", "post_date": "10/07/2017 20:27:58", "content": "this is for resnet50. the weights are actaully the same as the default pytorch model zoo. i reorganize resnet for future experiments and the naming changed. i manually create the mapping for the model zoo keys to my keys. actually you can just use the default pytorch model zoo and ignore my resnet_xx.py
", "votes": null, "replies": [] }, { "id": 228892, "author_name": "timjoseph", "author_url": "", "post_date": "10/08/2017 07:51:44", "content": "Thank you :)
", "votes": null, "replies": [] } ] }, { "id": 228890, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/08/2017 07:26:43", "content": "Hi, Heng, I use similar way to load data as you, converting bson to files.
\n\nThe loading process is very fast at first(0.3s, CPU 30%, RAM 10% of 30G), but some iterations later, it becomes very slow(6s, CPU 5%, RAM 10% of 30G).
\n\nI set the CPU mode to performance, and try different num_workers(0,4,8,12 …), but the situation stays the same.
\n\nHere is my code:
\n\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\n\n\nclass xiongDataset(Dataset):\n def __init__(self, csv_file, root_dir, transform=None):\n self.train_names=[]\n self.root_dir=root_dir\n self.transform = transform\n train_images = pd.read_csv(csv_file)\n train_ids = list(train_images['product_id'])\n train_idxs = list(train_images['img_idx'])\n self.labels = list(train_images['category_idx'])\n num_train = len(train_images)\n for i in range(num_train):\n train_name = '{}_{}.jpg'.format(train_ids[i],train_idxs[i])\n self.train_names.append(train_name)\n\n def __len__(self):\n return len(self.train_names)\n\n def __getitem__(self, idx):\n img = cv2.imread(self.root_dir + self.train_names[idx])\n label = self.labels[idx]\n if self.transform is not None:\n img = self.transform(img)\n\n return img,label\n\ntrain_data = xiongDataset('../train_images.csv','../train/',transform=transforms.ToTensor())\n\n\n\ndata_loader= DataLoader(train_data,batch_size=256,shuffle=False,num_workers=0)\ni use ssd drive (solid state drive). if you are using regular hard disk, you need a better random access database format like leveldb, hdfs i think.
\n\nwhen i use regular hard disk, i face the same problem you said.
", "votes": null, "replies": [] }, { "id": 228972, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/08/2017 12:42:45", "content": "Thank you for your advice. I do as what you said, and make it. Now, loading data only takes less than 0.08 seconds, and one iteration takes 0.3 seconds(batch size=256, Resnet18, a 1080Ti).
", "votes": null, "replies": [] }, { "id": 230220, "author_name": "lyakaap", "author_url": "", "post_date": "10/11/2017 14:36:31", "content": "I have encountered same problem by using ssd. Probably it occurred due to GC, so adding a line gc.disable() in code is a way (I am scared that it may cause some bad effect...).
Thanks for sharing
", "votes": null, "replies": [] }, { "id": 229987, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/11/2017 00:47:45", "content": "Hi, Heng, have you tried multi-gpu? I also train with pytorch, and I set model = nn.DataParallel(model, device_ids=[0, 1]), but there is several errors https://discuss.pytorch.org/t/multi-gpu-error-the-accuracy-of-one-gpu-is-much-lower-than-two-gpus/8492
And then I tried your way os.environ[\"CUDA_VISIBLE_DEVICES\"] = '0,1'(I have 2 GPUs), but only one GPU works.
haven't try yet. multi-gpu development would come later
", "votes": null, "replies": [] } ] }, { "id": 230350, "author_name": "strideradu", "author_url": "", "post_date": "10/11/2017 19:30:48", "content": "How many GPUs do you have? I remember you have 4 Titan X? Just want to compare the running time.
", "votes": null, "replies": [ { "id": 230397, "author_name": "hengck23", "author_url": "", "post_date": "10/11/2017 21:50:04", "content": "i am using 1 pascal titanx and 3 1080 ti. but each gpu is use to train one model. i haven't train across multiple gpu yet. i am still \"exploring\" different models
", "votes": null, "replies": [] } ] }, { "id": 230391, "author_name": "lamdang", "author_url": "", "post_date": "10/11/2017 21:25:24", "content": "Hello Heng,
\n\nThanks a lot for sharing your scripts and models so generously! I am struggling to break through 50% acc so going through your scripts will be very helpful to find the issues in my pipeline.
\n\nI have a few questions on your train & valid split:
\n\nvalidation set is not 5k. it is 50k.
\n\nthe 5k is used in the training iteration is a subset of validation to make \"visualisation\" fast.
\n\nthay are choosen randomly. (not the best way)
\n\nI will come back to data sampling later. now i am still at early stage of experiments
", "votes": null, "replies": [] } ] }, { "id": 230576, "author_name": "yanchao727", "author_url": "", "post_date": "10/12/2017 08:54:40", "content": "Can't find pretrained_file: /root/share/data/models/pytorch/imagenet/resenet/resnet50-19c8e357.pth
", "votes": null, "replies": [ { "id": 230577, "author_name": "hengck23", "author_url": "", "post_date": "10/12/2017 08:56:13", "content": "", "votes": null, "replies": [] } ] }, { "id": 230596, "author_name": "yanchao727", "author_url": "", "post_date": "10/12/2017 09:52:49", "content": "How to support multi-gpu ? According to https://discuss.pytorch.org/t/does-it-support-multi-gpu-card-on-a-single-node/75/7 this shouldn't be too difficult ?
", "votes": null, "replies": [ { "id": 230603, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/12/2017 10:24:25", "content": "Have you tried multi-gpu? I tried two gpus, but there are several errors https://discuss.pytorch.org/t/multi-gpu-error-the-accuracy-of-one-gpu-is-much-lower-than-two-gpus/8492
", "votes": null, "replies": [] } ] }, { "id": 231533, "author_name": "timjoseph", "author_url": "", "post_date": "10/15/2017 09:13:01", "content": "Is the 10-7 release the lastest code?
\n\nprobs = F.softmax(logits)\nloss = F.cross_entropy(logits, labels)\nThis seems wrong, since crossentropy already includes log_softmax and NLLLoss!
", "votes": null, "replies": [ { "id": 231535, "author_name": "hengck23", "author_url": "", "post_date": "10/15/2017 09:16:20", "content": "should be correct
\n\nloss = F.cross_entropy(logits, labels)
\n\nthe input is logits, not probs
\n\nthe loss statement is not dependent on the prob statement.
", "votes": null, "replies": [] }, { "id": 231537, "author_name": "timjoseph", "author_url": "", "post_date": "10/15/2017 09:20:23", "content": "Oh wow. You are totally right, so stupid <--
", "votes": null, "replies": [] } ] }, { "id": 231550, "author_name": "ifighting", "author_url": "", "post_date": "10/15/2017 10:37:48", "content": "I'm curious about your model, how do you train your model, how do you predict the image in one crop
", "votes": null, "replies": [] }, { "id": 232860, "author_name": "hengck23", "author_url": "", "post_date": "10/18/2017 16:08:33", "content": "results of se-resnext101. This is trained with the latest version 10-17 of the pytorch starter kit above. You need multi-gpu to use this (or it would be really slow). LB 0.71064 for se-resnext101_32x4d (single crop 180/180)
\n\nI try several sets of parameters, etc and finally it works! if the parameters are not correct the loss won't drop. If the parameters are correct, you will see the loss drop slowly over several epoch.
\n\nif se-resnext101 works, senet should be better: https://github.com/hujie-frank/SENet
\n\nmy next plan would be senet or larger (like se-resnext269)
\n\n
Do you use the weights from resneXt pretrained on imagenet or did you map the weights from the original se-resneXt to your own pytorch implementation?
", "votes": null, "replies": [] }, { "id": 233048, "author_name": "hengck23", "author_url": "", "post_date": "10/19/2017 03:34:54", "content": "imagenet pretrained weights for resnext and se-resnext can be downloaded. i think there are no difference if you start off with \"resnext imagenet weights --> resnext weights --> se-resnext weights\" or \"se-resnext imagenet weights --> se-resnext weights\" .
\n\nfor the graph above, i am using \"resnext imagenet weights --> resnext weights --> se-resnext weights\".
\n\nbut for the new experiment on senet (which is se-resnext152) i am using \"se-resnext imagenet weights --> se-resnext weights\" directly.
\n\nboth seems to work well
", "votes": null, "replies": [] }, { "id": 233073, "author_name": "zhangsongwei", "author_url": "", "post_date": "10/19/2017 05:23:22", "content": "It seems that pytorch only has resnet imagenet weights. Did you only use that to fine-tune resnext or se-resnext?
", "votes": null, "replies": [] }, { "id": 233074, "author_name": "hengck23", "author_url": "", "post_date": "10/19/2017 05:34:34", "content": "I write a converter to convert from caffe to pytorch
", "votes": null, "replies": [] }, { "id": 233091, "author_name": "hengck23", "author_url": "", "post_date": "10/19/2017 07:36:15", "content": "i added a conversion to for snet (se-resnext152) in folder \"10-17/senet_conversion\". you can modify this for se-resnext101
", "votes": null, "replies": [] }, { "id": 233106, "author_name": "ifighting", "author_url": "", "post_date": "10/19/2017 08:19:40", "content": "what about the performance of the se-resnext?
", "votes": null, "replies": [] }, { "id": 233115, "author_name": "ifighting", "author_url": "", "post_date": "10/19/2017 08:56:28", "content": "how is your optimazer? sgd? or any other?
", "votes": null, "replies": [] } ] }, { "id": 234459, "author_name": "terryli", "author_url": "", "post_date": "10/23/2017 11:37:28", "content": "My linux rig reboots few minutes after I run the script using 2 GPUs. Any idea on the cause and how to resolve it? Thanks!
\n\nAnd thanks again for sharing the scripts!
", "votes": null, "replies": [ { "id": 234474, "author_name": "timjoseph", "author_url": "", "post_date": "10/23/2017 12:22:01", "content": "Maybe it's overheating?
", "votes": null, "replies": [] }, { "id": 234518, "author_name": "terryli", "author_url": "", "post_date": "10/23/2017 14:26:48", "content": "Thanks for your reply Tim. The GPUs are running 2 scripts separately and seems fine. In this case do you think overheating is still a probable cause? Thanks!
", "votes": null, "replies": [] }, { "id": 234532, "author_name": "timjoseph", "author_url": "", "post_date": "10/23/2017 14:57:20", "content": "Just monitor it with nvidia-smi. Or look in your logs what could have caused the error!
", "votes": null, "replies": [] }, { "id": 237452, "author_name": "terryli", "author_url": "", "post_date": "10/30/2017 10:45:53", "content": "It works by reducing the batch size. Maybe it's really due to overheating. Thanks!
", "votes": null, "replies": [] }, { "id": 237466, "author_name": "timjoseph", "author_url": "", "post_date": "10/30/2017 11:29:21", "content": "Oh, did you also check whether you have a memory leak? Higher batch size means more memory is needed (I am speaking about main memory, not GPU memory here)
", "votes": null, "replies": [] }, { "id": 237987, "author_name": "terryli", "author_url": "", "post_date": "10/31/2017 13:22:52", "content": "Oh may I know how I can check this?
", "votes": null, "replies": [] } ] }, { "id": 234524, "author_name": "burgalon", "author_url": "", "post_date": "10/23/2017 14:37:24", "content": "Hi Heng,\nFirst thank you for sharing all the knowledge and information. I enjoy reading your posts.\nI was wondering, why in your do_submit() code, do you split the test set?
", "votes": null, "replies": [ { "id": 234526, "author_name": "hengck23", "author_url": "", "post_date": "10/23/2017 14:43:25", "content": "I can run in 4 separate machines. In case of running in a single machine, i don't have to start from zero if the testing stop due to some error. Lastly, the saved npy files are smaller , making them faster to read and write
", "votes": null, "replies": [] } ] }, { "id": 237449, "author_name": "terryli", "author_url": "", "post_date": "10/30/2017 10:44:39", "content": "Hi Heng, any chance that you could further share how to get xception.keras.convert.pth? Thanks!
Wondering if you find where to got the xception pretrained model?
", "votes": null, "replies": [] }, { "id": 242727, "author_name": "terryli", "author_url": "", "post_date": "11/12/2017 12:54:58", "content": "Sorry no luck...
", "votes": null, "replies": [] } ] }, { "id": 242733, "author_name": "hengck23", "author_url": "", "post_date": "11/12/2017 13:13:33", "content": "xception.keras.convert.pth is converted by myself. However, conversion from keras to pytorch is not correct. I don't know why, but could be due to difference in padding, ceil/floor in max pooling.
\n\nnevertheless, the wrong converted weights seems to be usable for initialization for cdisount image training.
", "votes": null, "replies": [ { "id": 242968, "author_name": "strideradu", "author_url": "", "post_date": "11/13/2017 04:05:20", "content": "Have you check this https://github.com/ysh329/deep-learning-model-convertor
", "votes": null, "replies": [] } ] }, { "id": 249030, "author_name": "skinish", "author_url": "", "post_date": "11/27/2017 15:18:08", "content": "How did you guys perform the train/val split?
", "votes": null, "replies": [] }, { "id": 251376, "author_name": "stevenknguyen", "author_url": "", "post_date": "12/01/2017 04:29:05", "content": "Has anyone had this error training the Resnet101?
\n\n
what is src in your code?
", "votes": null, "replies": [] }, { "id": 251630, "author_name": "stevenknguyen", "author_url": "", "post_date": "12/01/2017 13:59:19", "content": "Nevermind it looks like some of my images are missing.
", "votes": null, "replies": [] } ] }, { "id": 327935, "author_name": "zhangsongwei", "author_url": "", "post_date": "05/13/2018 01:02:58", "content": "Hi, Heng. Did you save the data(.bson or .jpg)? The data has been removed after the competition, but I still want to use several images(Non-commercial). Could you please share it? Thank you~
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "227035": "** important **\nplease refer to @Vladimir Iglovikov at https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/41652\n\nplease train with long epoch to get better results . My models results are from training only up to 4 to 5 epoch. \n\n------\n\nThis is a pytorch starter kit. It is not completed yet. I use ideas from the kernels and discussion contributions. Thanks to kagglers, especially @Bruno G. do Amaral, @Human Analog, @Adam Blazek for code and discussion.\n\n\nDownload: \nhttps://drive.google.com/drive/folders/0B_DICebvRE-kb2dFd2FKX1hfRkE?usp=sharing\n\n---------------\nversion.10-03:\n\n- initial version for early experiments. see \"trainer.py\"\n\n---------------\n\nversion.10-07:\n\n- LB 0.63358(160x160 single center crop) or 0.63839(180x180) or 0.64356 (180x180 resize to 160x160) for se-resnet50 trained on 160x160 crops.\n\n- LB 0.66708 for same se-resnet50 if train=224x224 crops from 256, test = resize to 224\n\n- LB 0.68939 for same se-resnet50 if train=180x180, test = 180x180 (weight initialised from 224 se-resnet50 of above)\n\n- LB 0.69565 for inception3 if train=180x180, test = 180x180 (be careful on how to control the lr, batch size, momentum, level of augmentation,etc. I will have a writeup on this later)\n\n- support various network like resnet, resnext, inception v3, etc. see trainer_xxx.py\n\n- support gradient accumulation \n\nnote: code is messy and dirty. It may not be backward- compatible.\n\n---------------\n\nversion.10-17 (latest):\n\n- advance version. implement multi-gpu support in training. see \"trainer_excited_resnext101_32x4d.py\"\n\n- LB 0.71064 for se-resnext101_32x4d (single crop 180/180)\n\n- support xception, se-xception, resnext101_32x4d, se-resnext101_32x4d, inceptionv4, inception_resnetv2 \n\n- folder \"10-17/senet_conversion\" contains conversion tools for converting caffe model to pytorch \n\n- note: \n\n *1.* may not be backward compatible \n\n *2.* old training scripts are found in \"dummy-00/__temp__/old_trainer\"\n\n--------------- \n\n