I trained all image and 2 epoch cost about 1.5 day
", "messages": [ { "id": 223135, "postDate": "2017-09-21T07:58:46.657Z", "content": "I trained all image and 2 epoch cost about 1.5 day
", "rawMarkdown": "I trained all image and 2 epoch cost about 1.5 day", "votes": 1 }, { "id": 224967, "postDate": "2017-09-28T01:52:50.160Z", "content": "waiting for your result
", "rawMarkdown": "waiting for your result", "votes": -1 }, { "id": 226344, "postDate": "2017-10-02T00:08:21.650Z", "content": "My solution is based on cxflow framework with Xception model and 128 image size, batch_size=300. Using two 1080ti gpus to train , it is over 10 hours per epoch. Very slow. : (
", "rawMarkdown": "My solution is based on cxflow framework with Xception model and 128 image size, batch_size=300. Using two 1080ti gpus to train , it is over 10 hours per epoch. Very slow. : (" }, { "id": 225946, "postDate": "2017-09-30T13:58:12.287Z", "content": "I've used SE-ResNet-50 with a 5270-neuron classification layer on Keras + TensorFlow on a single 1080 Ti, batch size 256, training/validation split is 80%-20%, input image size 160x160. It takes ~6 hours to do a single epoch (with 8 workers).
\n\nThis uses my BSON generator for Keras that I posted in Kernels, but with additional data augmentation, train.bson stored on SSD drive.
\n\nCurrently I'm creating predictions for the test set using 10 crops per image and this takes about 18 hours... I should try to optimize this a bit. ;-)
", "rawMarkdown": "I've used SE-ResNet-50 with a 5270-neuron classification layer on Keras + TensorFlow on a single 1080 Ti, batch size 256, training/validation split is 80%-20%, input image size 160x160. It takes ~6 hours to do a single epoch (with 8 workers). \n\nThis uses my BSON generator for Keras that I posted in Kernels, but with additional data augmentation, train.bson stored on SSD drive. \n\nCurrently I'm creating predictions for the test set using 10 crops per image and this takes about 18 hours... I should try to optimize this a bit. ;-)", "replies": [ { "id": 226029, "postDate": "2017-09-30T19:19:50.877Z", "content": "Thanks for sharing, your BSON generator is really helpful! Do you train from scratch or fine-tuning on SE-ResNet-50? I tried to fine-tuning on Xception, it took me 6 hours on a single epoch. (1080, 32RAM, I7, batch size 32) Is that normal?
", "rawMarkdown": "Thanks for sharing, your BSON generator is really helpful! Do you train from scratch or fine-tuning on SE-ResNet-50? I tried to fine-tuning on Xception, it took me 6 hours on a single epoch. (1080, 32RAM, I7, batch size 32) Is that normal?" }, { "id": 226030, "postDate": "2017-09-30T19:25:40.793Z", "content": "But from scratch, it will take me more than 30 hours (now it takes me 17 hours on half of the training set)
", "rawMarkdown": "But from scratch, it will take me more than 30 hours (now it takes me 17 hours on half of the training set)" }, { "id": 226144, "postDate": "2017-10-01T06:52:21.337Z", "content": "By the way, how did you do with 8 workers?
", "rawMarkdown": "By the way, how did you do with 8 workers?" }, { "id": 226147, "postDate": "2017-10-01T07:07:57.920Z", "content": "By the way , how did you do with 8 workers?
", "rawMarkdown": "By the way , how did you do with 8 workers?" }, { "id": 226183, "postDate": "2017-10-01T10:00:33.613Z", "content": "I did not train from scratch, fine-tuning only (on the last block of the network). It might help to eventually fine-tune all the layers but with a tiny learning rate, but I'd save that for until the network already gets a good score.
", "rawMarkdown": "I did not train from scratch, fine-tuning only (on the last block of the network). It might help to eventually fine-tune *all* the layers but with a tiny learning rate, but I'd save that for until the network already gets a good score." }, { "id": 226201, "postDate": "2017-10-01T12:31:59.527Z", "content": "Hi Analog, I use your BSONIterator class to fine-tune ResNet18 with the same database(the same machine, the same batch size). It takes about 33 hours for one epoch. I profile the program and find the lock code scope takes about 70% of the total time.
\n\nIs it normal? Is there any way to speed up the training process?
\n\nAnd as a green hand, I don't how to train it on 8 workers as you, could you share more details?
\n\nBest regard.
\n\n with self.lock:\n image_row = self.images_df.iloc[j]\n product_id = image_row[\"product_id\"]\n offset_row = self.offsets_df.loc[product_id]\n # Read this product's data from the BSON file.\n self.file.seek(offset_row[\"offset\"])\n item_data = self.file.read(offset_row[\"length\"])\nIt makes sense that it spends most of the time there, since disk I/O is slow (especially since I think you mentioned you're not using an SSD). If it takes 33 hours for one epoch, then your computer spends most of its time generating the batches, and your GPU is mostly waiting (not computing anything).
\n\nTo use 8 workers, just add the parameter workers=8 to the model.fit_generator() call.
hi @zhangsongwei which program did you use to profile the code, thanks
", "rawMarkdown": "hi @zhangsongwei which program did you use to profile the code, thanks" }, { "id": 226351, "postDate": "2017-10-02T01:01:18.450Z", "content": "Just compute the time.
", "rawMarkdown": "Just compute the time." } ] }, { "id": 225860, "postDate": "2017-09-30T06:44:13.263Z", "content": "Using the random access kernel, I can train 1 epoch in 10-11hours with the normal resolutions and train-time augmentations (random rotate, flip, and transpose). batch size is 96 for training and 32 for validation. 98-2% split match the lb +- 10^-3
", "rawMarkdown": "Using the random access kernel, I can train 1 epoch in 10-11hours with the normal resolutions and train-time augmentations (random rotate, flip, and transpose). batch size is 96 for training and 32 for validation. 98-2% split match the lb +- 10^-3", "replies": [ { "id": 225862, "postDate": "2017-09-30T06:47:38.307Z", "content": "Oh, thank you, and what's your hardware?
", "rawMarkdown": "Oh, thank you, and what's your hardware?" }, { "id": 225867, "postDate": "2017-09-30T07:34:20.207Z", "content": "1080 (unfortunately not Ti), 64 GB of ram, i7, and ~3TB of storage
", "rawMarkdown": "1080 (unfortunately not Ti), 64 GB of ram, i7, and ~3TB of storage" }, { "id": 225868, "postDate": "2017-09-30T07:38:37.580Z", "content": "16GB ram is so sad
", "rawMarkdown": "16GB ram is so sad", "votes": 1 }, { "id": 225879, "postDate": "2017-09-30T08:38:08.203Z", "content": "16GB ram is what I have, too. 32GB would be perfect, but 16GB is just enough!
", "rawMarkdown": "16GB ram is what I have, too. 32GB would be perfect, but 16GB is just enough!", "votes": -1 }, { "id": 226007, "postDate": "2017-09-30T17:05:41.883Z", "content": "Yeah 16 is more than enough for this competition if you're using random access
", "rawMarkdown": "Yeah 16 is more than enough for this competition if you're using random access" }, { "id": 226114, "postDate": "2017-10-01T04:22:57.740Z", "content": "Yes, i just use this method
", "rawMarkdown": "Yes, i just use this method" }, { "id": 230990, "postDate": "2017-10-13T11:17:41.147Z", "content": "SpruceMoose how many images You use for train and validation set?
\n\nYou train all layers?
\n\nThanks
", "rawMarkdown": "SpruceMoose how many images You use for train and validation set?\n\nYou train all layers?\n\nThanks" } ] }, { "id": 225769, "postDate": "2017-09-30T01:27:26.097Z", "content": "My net is ResNet18, and a epoch took about one day.
\n\nMine is 1080TI. So sad!
", "rawMarkdown": "My net is ResNet18, and a epoch took about one day.\n\nMine is 1080TI. So sad!", "replies": [ { "id": 225865, "postDate": "2017-09-30T07:27:07.013Z", "content": "It's ok, I use googlenet, the net is faster.
", "rawMarkdown": "It's ok, I use googlenet, the net is faster." }, { "id": 225880, "postDate": "2017-09-30T08:38:39.060Z", "content": "Something must be wrong with your setup. 1 Epoch takes 2 hours for me on my 1080TI with resnet-18
", "rawMarkdown": "Something must be wrong with your setup. 1 Epoch takes 2 hours for me on my 1080TI with resnet-18" }, { "id": 226087, "postDate": "2017-10-01T00:40:53.557Z", "content": "@Tim Joseph
\n\nI used Human's BSON generator for Keras that posted in his Kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson), but without additional data augmentation. And I put my data into ResNet-18 using pytorch. My batch size is 256.
\n\nI tried to find out what happened, but didn't succeed. Could you please show more details about you setup?
\n\nThank you!
", "rawMarkdown": "@Tim Joseph\n\nI used Human's BSON generator for Keras that posted in his Kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson), but without additional data augmentation. And I put my data into ResNet-18 using pytorch. My batch size is 256.\n\nI tried to find out what happened, but didn't succeed. Could you please show more details about you setup?\n\nThank you!" }, { "id": 226089, "postDate": "2017-10-01T00:55:45.253Z", "content": "@zhangsongwei does that mean you're using decode without decode_images? (the combining part) If so, I recommend combining product images into one and then resizing. It speeds things up and doesn't seem to hurt acc.
", "rawMarkdown": "@zhangsongwei does that mean you're using decode without decode_images? (the combining part) If so, I recommend combining product images into one and then resizing. It speeds things up and doesn't seem to hurt acc." }, { "id": 226095, "postDate": "2017-10-01T02:26:02.630Z", "content": "Hi , @SpruceMoose, thanks for your advice. I think you are right , I am using decode after checking my code. But I am confused about what's the meaning of decode_images, could you please show more details about it?
\n\nIs it like this?
\n\nfor c, d in enumerate(data):\n product_id = d['_id']\n category_id = d['category_id'] \n prod_to_category[product_id] = category_id\n for e, pic in enumerate(d['imgs']):\n picture = imread(io.BytesIO(pic['picture']))\nThank you!
", "rawMarkdown": "Hi , @SpruceMoose, thanks for your advice. I think you are right , I am using decode after checking my code. But I am confused about what's the meaning of decode_images, could you please show more details about it?\n\nIs it like this?\n\n for c, d in enumerate(data):\n product_id = d['_id']\n category_id = d['category_id'] \n prod_to_category[product_id] = category_id\n for e, pic in enumerate(d['imgs']):\n picture = imread(io.BytesIO(pic['picture']))\n\nThank you!", "votes": 1 }, { "id": 226101, "postDate": "2017-10-01T03:04:34.877Z", "content": "decode_images will simplify life for you because it allows for making a single prediction for a given product id (1 image per product id). What it does is merge the images in product dictionaries with 2-4 images. You will have to resize afterwards. I have a working version that predicts the category id given an unmerged image. However, many products have a different prediction for each image. I'll think about doing mini-ensembles when I get a better model.
", "rawMarkdown": "decode_images will simplify life for you because it allows for making a single prediction for a given product id (1 image per product id). What it does is merge the images in product dictionaries with 2-4 images. You will have to resize afterwards. I have a working version that predicts the category id given an unmerged image. However, many products have a different prediction for each image. I'll think about doing mini-ensembles when I get a better model." }, { "id": 226106, "postDate": "2017-10-01T03:35:29.297Z", "content": "Just wondering how do you modify resnet 18 to accept smaller image?
", "rawMarkdown": "Just wondering how do you modify resnet 18 to accept smaller image?" }, { "id": 226107, "postDate": "2017-10-01T03:39:35.570Z", "content": "I resized the input size.
", "rawMarkdown": "I resized the input size." }, { "id": 226112, "postDate": "2017-10-01T04:03:37.637Z", "content": "@zhangsongwei I meant resize the image that is returned by decode_images if needed (was in my case)
\n\n@CSAdu I use cv2.resize. In Keras, maybe just change input_size. In TF, I think you modify the first layer's filters/dimensions to match your input data. In PyTorch, I could be wrong but it seems you just need to care about the number of channels:
\n\nin_channels, height, width = in_shape\n\n self.conv1 = nn.Conv2d(in_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)\n self.bn1 = nn.BatchNorm2d(64)\n@zhangsongwei @SpriceMoose So what you guys mean just increase picture size to 224?
", "rawMarkdown": "@zhangsongwei @SpriceMoose So what you guys mean just increase picture size to 224?" }, { "id": 226124, "postDate": "2017-10-01T04:57:15.970Z", "content": "Yes , it seems that multiple workers could solve it.
", "rawMarkdown": "Yes , it seems that multiple workers could solve it." }, { "id": 226127, "postDate": "2017-10-01T05:04:24.287Z", "content": "Depends on what you're using for transfer learning
", "rawMarkdown": "Depends on what you're using for transfer learning" }, { "id": 226933, "postDate": "2017-10-03T11:51:36.617Z", "content": "@Tim Joseph
\n\nCould you please show more details about your setup? I renew my setup , but it still takes about 6 hours per epoch.(Resnet18 1080Ti all images)
", "rawMarkdown": "@Tim Joseph\n\nCould you please show more details about your setup? I renew my setup , but it still takes about 6 hours per epoch.(Resnet18 1080Ti all images)" }, { "id": 226973, "postDate": "2017-10-03T13:27:06.100Z", "content": "Sorry, I will not discuss my setup here, since it is not based on any kernel and I am not planing to make it public until the competition is over.\nIf I was using any of the kernels I would probably just put the images into folders and then use them like you normally would do.
\n\nBut I think 6 hours is good already. I gave your wrong information in my post above by mistake. I only run 1 epoch on 9 million images which takes 2 hours. If you are using 15 million images it should take 2 hours 30 min. If you want to keep your current setup you have to identify the bottleneck first and then based on this change your setup!
", "rawMarkdown": "Sorry, I will not discuss my setup here, since it is not based on any kernel and I am not planing to make it public until the competition is over.\nIf I was using any of the kernels I would probably just put the images into folders and then use them like you normally would do. \n\nBut I think 6 hours is good already. I gave your wrong information in my post above by mistake. I only run 1 epoch on 9 million images which takes 2 hours. If you are using 15 million images it should take 2 hours 30 min. If you want to keep your current setup you have to identify the bottleneck first and then based on this change your setup!", "votes": -1 } ] }, { "id": 223542, "postDate": "2017-09-22T14:16:49.120Z", "content": "Which architeture are you using? Mine is InceptionV3 2 epochs. It took 48h. GTX 1080TI.
", "rawMarkdown": "Which architeture are you using? Mine is InceptionV3 2 epochs. It took 48h. GTX 1080TI.", "replies": [ { "id": 223548, "postDate": "2017-09-22T14:38:36.867Z", "content": "GTX 1070 16G RAM CPU i7, on my machine. I use googlenet, and after know your time, I feel better in my heart.
", "rawMarkdown": "GTX 1070 16G RAM CPU i7, on my machine. I use googlenet, and after know your time, I feel better in my heart.", "votes": 1 } ] }, { "id": 223350, "postDate": "2017-09-22T01:09:14.493Z", "content": "GTX 1070 16G RAM CPU i7, on my machine
", "rawMarkdown": "GTX 1070 16G RAM CPU i7, on my machine" }, { "id": 223332, "postDate": "2017-09-21T22:35:23.877Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 224967, "author_name": "Joy", "author_url": "", "post_date": "2017-09-28T01:52:50.160000", "content": "waiting for your result
", "votes": -1, "replies": [] }, { "id": 226344, "author_name": "huiqin", "author_url": "", "post_date": "2017-10-02T00:08:21.650000", "content": "My solution is based on cxflow framework with Xception model and 128 image size, batch_size=300. Using two 1080ti gpus to train , it is over 10 hours per epoch. Very slow. : (
", "votes": 0, "replies": [] }, { "id": 225946, "author_name": "Human Analog", "author_url": "", "post_date": "2017-09-30T13:58:12.287000", "content": "I've used SE-ResNet-50 with a 5270-neuron classification layer on Keras + TensorFlow on a single 1080 Ti, batch size 256, training/validation split is 80%-20%, input image size 160x160. It takes ~6 hours to do a single epoch (with 8 workers).
\n\nThis uses my BSON generator for Keras that I posted in Kernels, but with additional data augmentation, train.bson stored on SSD drive.
\n\nCurrently I'm creating predictions for the test set using 10 crops per image and this takes about 18 hours... I should try to optimize this a bit. ;-)
", "votes": 0, "replies": [ { "id": 226029, "author_name": "Brian Luo", "author_url": "", "post_date": "2017-09-30T19:19:50.877000", "content": "Thanks for sharing, your BSON generator is really helpful! Do you train from scratch or fine-tuning on SE-ResNet-50? I tried to fine-tuning on Xception, it took me 6 hours on a single epoch. (1080, 32RAM, I7, batch size 32) Is that normal?
", "votes": 0, "replies": [] }, { "id": 226030, "author_name": "Brian Luo", "author_url": "", "post_date": "2017-09-30T19:25:40.793000", "content": "But from scratch, it will take me more than 30 hours (now it takes me 17 hours on half of the training set)
", "votes": 0, "replies": [] }, { "id": 226144, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T06:52:21.337000", "content": "By the way, how did you do with 8 workers?
", "votes": 0, "replies": [] }, { "id": 226147, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T07:07:57.920000", "content": "By the way , how did you do with 8 workers?
", "votes": 0, "replies": [] }, { "id": 226183, "author_name": "Human Analog", "author_url": "", "post_date": "2017-10-01T10:00:33.613000", "content": "I did not train from scratch, fine-tuning only (on the last block of the network). It might help to eventually fine-tune all the layers but with a tiny learning rate, but I'd save that for until the network already gets a good score.
", "votes": 0, "replies": [] }, { "id": 226201, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T12:31:59.527000", "content": "Hi Analog, I use your BSONIterator class to fine-tune ResNet18 with the same database(the same machine, the same batch size). It takes about 33 hours for one epoch. I profile the program and find the lock code scope takes about 70% of the total time.
\n\nIs it normal? Is there any way to speed up the training process?
\n\nAnd as a green hand, I don't how to train it on 8 workers as you, could you share more details?
\n\nBest regard.
\n\n with self.lock:\n image_row = self.images_df.iloc[j]\n product_id = image_row[\"product_id\"]\n offset_row = self.offsets_df.loc[product_id]\n # Read this product's data from the BSON file.\n self.file.seek(offset_row[\"offset\"])\n item_data = self.file.read(offset_row[\"length\"])\nIt makes sense that it spends most of the time there, since disk I/O is slow (especially since I think you mentioned you're not using an SSD). If it takes 33 hours for one epoch, then your computer spends most of its time generating the batches, and your GPU is mostly waiting (not computing anything).
\n\nTo use 8 workers, just add the parameter workers=8 to the model.fit_generator() call.
hi @zhangsongwei which program did you use to profile the code, thanks
", "votes": 0, "replies": [] }, { "id": 226351, "author_name": "Will", "author_url": "", "post_date": "2017-10-02T01:01:18.450000", "content": "Just compute the time.
", "votes": 0, "replies": [] } ] }, { "id": 225860, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-30T06:44:13.263000", "content": "Using the random access kernel, I can train 1 epoch in 10-11hours with the normal resolutions and train-time augmentations (random rotate, flip, and transpose). batch size is 96 for training and 32 for validation. 98-2% split match the lb +- 10^-3
", "votes": 0, "replies": [ { "id": 225862, "author_name": "Five Years", "author_url": "", "post_date": "2017-09-30T06:47:38.307000", "content": "Oh, thank you, and what's your hardware?
", "votes": 0, "replies": [] }, { "id": 225867, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-30T07:34:20.207000", "content": "1080 (unfortunately not Ti), 64 GB of ram, i7, and ~3TB of storage
", "votes": 0, "replies": [] }, { "id": 225868, "author_name": "Five Years", "author_url": "", "post_date": "2017-09-30T07:38:37.580000", "content": "16GB ram is so sad
", "votes": 1, "replies": [] }, { "id": 225879, "author_name": "Tim Joseph", "author_url": "", "post_date": "2017-09-30T08:38:08.203000", "content": "16GB ram is what I have, too. 32GB would be perfect, but 16GB is just enough!
", "votes": -1, "replies": [] }, { "id": 226007, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-30T17:05:41.883000", "content": "Yeah 16 is more than enough for this competition if you're using random access
", "votes": 0, "replies": [] }, { "id": 226114, "author_name": "Five Years", "author_url": "", "post_date": "2017-10-01T04:22:57.740000", "content": "Yes, i just use this method
", "votes": 0, "replies": [] }, { "id": 230990, "author_name": "Juan Pizarro", "author_url": "", "post_date": "2017-10-13T11:17:41.147000", "content": "SpruceMoose how many images You use for train and validation set?
\n\nYou train all layers?
\n\nThanks
", "votes": 0, "replies": [] } ] }, { "id": 225769, "author_name": "Will", "author_url": "", "post_date": "2017-09-30T01:27:26.097000", "content": "My net is ResNet18, and a epoch took about one day.
\n\nMine is 1080TI. So sad!
", "votes": 0, "replies": [ { "id": 225865, "author_name": "Five Years", "author_url": "", "post_date": "2017-09-30T07:27:07.013000", "content": "It's ok, I use googlenet, the net is faster.
", "votes": 0, "replies": [] }, { "id": 225880, "author_name": "Tim Joseph", "author_url": "", "post_date": "2017-09-30T08:38:39.060000", "content": "Something must be wrong with your setup. 1 Epoch takes 2 hours for me on my 1080TI with resnet-18
", "votes": 0, "replies": [] }, { "id": 226087, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T00:40:53.557000", "content": "@Tim Joseph
\n\nI used Human's BSON generator for Keras that posted in his Kernels(https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson), but without additional data augmentation. And I put my data into ResNet-18 using pytorch. My batch size is 256.
\n\nI tried to find out what happened, but didn't succeed. Could you please show more details about you setup?
\n\nThank you!
", "votes": 0, "replies": [] }, { "id": 226089, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-10-01T00:55:45.253000", "content": "@zhangsongwei does that mean you're using decode without decode_images? (the combining part) If so, I recommend combining product images into one and then resizing. It speeds things up and doesn't seem to hurt acc.
", "votes": 0, "replies": [] }, { "id": 226095, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T02:26:02.630000", "content": "Hi , @SpruceMoose, thanks for your advice. I think you are right , I am using decode after checking my code. But I am confused about what's the meaning of decode_images, could you please show more details about it?
\n\nIs it like this?
\n\nfor c, d in enumerate(data):\n product_id = d['_id']\n category_id = d['category_id'] \n prod_to_category[product_id] = category_id\n for e, pic in enumerate(d['imgs']):\n picture = imread(io.BytesIO(pic['picture']))\nThank you!
", "votes": 1, "replies": [] }, { "id": 226101, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-10-01T03:04:34.877000", "content": "decode_images will simplify life for you because it allows for making a single prediction for a given product id (1 image per product id). What it does is merge the images in product dictionaries with 2-4 images. You will have to resize afterwards. I have a working version that predicts the category id given an unmerged image. However, many products have a different prediction for each image. I'll think about doing mini-ensembles when I get a better model.
", "votes": 0, "replies": [] }, { "id": 226106, "author_name": "Strideradu", "author_url": "", "post_date": "2017-10-01T03:35:29.297000", "content": "Just wondering how do you modify resnet 18 to accept smaller image?
", "votes": 0, "replies": [] }, { "id": 226107, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T03:39:35.570000", "content": "I resized the input size.
", "votes": 0, "replies": [] }, { "id": 226112, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-10-01T04:03:37.637000", "content": "@zhangsongwei I meant resize the image that is returned by decode_images if needed (was in my case)
\n\n@CSAdu I use cv2.resize. In Keras, maybe just change input_size. In TF, I think you modify the first layer's filters/dimensions to match your input data. In PyTorch, I could be wrong but it seems you just need to care about the number of channels:
\n\nin_channels, height, width = in_shape\n\n self.conv1 = nn.Conv2d(in_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)\n self.bn1 = nn.BatchNorm2d(64)\n@zhangsongwei @SpriceMoose So what you guys mean just increase picture size to 224?
", "votes": 0, "replies": [] }, { "id": 226124, "author_name": "Will", "author_url": "", "post_date": "2017-10-01T04:57:15.970000", "content": "Yes , it seems that multiple workers could solve it.
", "votes": 0, "replies": [] }, { "id": 226127, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-10-01T05:04:24.287000", "content": "Depends on what you're using for transfer learning
", "votes": 0, "replies": [] }, { "id": 226933, "author_name": "Will", "author_url": "", "post_date": "2017-10-03T11:51:36.617000", "content": "@Tim Joseph
\n\nCould you please show more details about your setup? I renew my setup , but it still takes about 6 hours per epoch.(Resnet18 1080Ti all images)
", "votes": 0, "replies": [] }, { "id": 226973, "author_name": "Tim Joseph", "author_url": "", "post_date": "2017-10-03T13:27:06.100000", "content": "Sorry, I will not discuss my setup here, since it is not based on any kernel and I am not planing to make it public until the competition is over.\nIf I was using any of the kernels I would probably just put the images into folders and then use them like you normally would do.
\n\nBut I think 6 hours is good already. I gave your wrong information in my post above by mistake. I only run 1 epoch on 9 million images which takes 2 hours. If you are using 15 million images it should take 2 hours 30 min. If you want to keep your current setup you have to identify the bottleneck first and then based on this change your setup!
", "votes": -1, "replies": [] } ] }, { "id": 223542, "author_name": "Aloisio Dourado", "author_url": "", "post_date": "2017-09-22T14:16:49.120000", "content": "Which architeture are you using? Mine is InceptionV3 2 epochs. It took 48h. GTX 1080TI.
", "votes": 0, "replies": [ { "id": 223548, "author_name": "Five Years", "author_url": "", "post_date": "2017-09-22T14:38:36.867000", "content": "GTX 1070 16G RAM CPU i7, on my machine. I use googlenet, and after know your time, I feel better in my heart.
", "votes": 1, "replies": [] } ] }, { "id": 223350, "author_name": "Five Years", "author_url": "", "post_date": "2017-09-22T01:09:14.493000", "content": "GTX 1070 16G RAM CPU i7, on my machine
", "votes": 0, "replies": [] }, { "id": 223332, "author_name": "", "author_url": "", "post_date": "2017-09-21T22:35:23.877000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "223135": "I trained all image and 2 epoch cost about 1.5 day", "224967": "waiting for your result", "226344": "My solution is based on cxflow framework with Xception model and 128 image size, batch_size=300. Using two 1080ti gpus to train , it is over 10 hours per epoch. Very slow. : (", "225946": "I've used SE-ResNet-50 with a 5270-neuron classification layer on Keras + TensorFlow on a single 1080 Ti, batch size 256, training/validation split is 80%-20%, input image size 160x160. It takes ~6 hours to do a single epoch (with 8 workers). \n\nThis uses my BSON generator for Keras that I posted in Kernels, but with additional data augmentation, train.bson stored on SSD drive. \n\nCurrently I'm creating predictions for the test set using 10 crops per image and this takes about 18 hours... I should try to optimize this a bit. ;-)", "225860": "Using the random access kernel, I can train 1 epoch in 10-11hours with the normal resolutions and train-time augmentations (random rotate, flip, and transpose). batch size is 96 for training and 32 for validation. 98-2% split match the lb +- 10^-3", "225769": "My net is ResNet18, and a epoch took about one day.\n\nMine is 1080TI. So sad!", "223542": "Which architeture are you using? Mine is InceptionV3 2 epochs. It took 48h. GTX 1080TI.", "223350": "GTX 1070 16G RAM CPU i7, on my machine", "223332": "" } }