In my experiments, 1024x1024 seems to be sufficient. Beyond that, the gain is not very much. \nCan any kagglers confirmed this?
", "messages": [ { "id": 217664, "postDate": "2017-08-31T14:56:23.423Z", "content": "In my experiments, 1024x1024 seems to be sufficient. Beyond that, the gain is not very much. \nCan any kagglers confirmed this?
", "rawMarkdown": "In my experiments, 1024x1024 seems to be sufficient. Beyond that, the gain is not very much. \nCan any kagglers confirmed this?\n", "votes": 6 }, { "id": 217770, "postDate": "2017-08-31T21:08:47.293Z", "content": "Something Ive noticed, is that given a pixel budget, keeping the width to be length*1.5 (~original proportion) does slightly better. Have you tried maintaining your 1024^2 budget and formulate an experiment with something close to 1260x840?
", "rawMarkdown": "Something Ive noticed, is that given a pixel budget, keeping the width to be length*1.5 (~original proportion) does slightly better. Have you tried maintaining your 1024^2 budget and formulate an experiment with something close to 1260x840? ", "votes": 3, "replies": [ { "id": 217776, "postDate": "2017-08-31T21:48:27.497Z", "content": "Are you using a unet or some other architecture?
", "rawMarkdown": "Are you using a unet or some other architecture?" }, { "id": 217778, "postDate": "2017-08-31T22:00:36.247Z", "content": "using u-net based idea, with input (1104x736 since my model needs both l,w to be 16-divisible)
", "rawMarkdown": "using u-net based idea, with input (1104x736 since my model needs both l,w to be 16-divisible)" }, { "id": 217784, "postDate": "2017-08-31T22:52:32.690Z", "content": "It is interesting to play with the dimensions of the input images. Some of my results from several aspect ratios are comparable (LB 0.996+) in previous low quality data. I forked Peter's keras starter using unet512.
", "rawMarkdown": "It is interesting to play with the dimensions of the input images. Some of my results from several aspect ratios are comparable (LB 0.996+) in previous low quality data. I forked Peter's keras starter using unet512." }, { "id": 217823, "postDate": "2017-09-01T05:09:24.260Z", "content": "Is there any increase in computation speed using the rectangle over the square?
", "rawMarkdown": "Is there any increase in computation speed using the rectangle over the square?", "votes": 2 }, { "id": 217898, "postDate": "2017-09-01T12:58:07.947Z", "content": "I haven't tried square yet. But I guess the computing time would respond to pixel counts. FYI, some random clips of one epoch from logs:
\n\nw0768h0512 (3:2, pixel counts ~0.39M), 642s
\n\nw0960h0640 (3:2 ~0.61M), 975s
\n\nw0768h1024 (3:4 ~0.79M), 1257s
\n\nw1280h0640 (2:1 ~0.82M), 1316s
", "rawMarkdown": "I haven't tried square yet. But I guess the computing time would respond to pixel counts. FYI, some random clips of one epoch from logs:\n\nw0768h0512 (3:2, pixel counts ~0.39M), 642s\n\nw0960h0640 (3:2 ~0.61M), 975s\n\nw0768h1024 (3:4 ~0.79M), 1257s\n\nw1280h0640 (2:1 ~0.82M), 1316s", "votes": 3 } ] }, { "id": 217678, "postDate": "2017-08-31T15:46:03.830Z", "content": "I am training full resolution, I will try to do an experiment comparing 1024x1024 vs. 1918x1280 and report back.
", "rawMarkdown": "I am training full resolution, I will try to do an experiment comparing 1024x1024 vs. 1918x1280 and report back. ", "votes": 3, "replies": [ { "id": 218379, "postDate": "2017-09-04T01:59:17.993Z", "content": "It seems that 1918x1280 performs better. \nWhat is your batch size? Mine is 8.
", "rawMarkdown": "It seems that 1918x1280 performs better. \nWhat is your batch size? Mine is 8.", "votes": 1 }, { "id": 218501, "postDate": "2017-09-04T14:12:18.073Z", "content": "GPU batch size = 1, b/c I cannot fit more in my GPU (11 Gb), but I used the Adam gradient accumulation trick with an effective batch-size (grandient-wise) of 8.
", "rawMarkdown": "GPU batch size = 1, b/c I cannot fit more in my GPU (11 Gb), but I used the Adam gradient accumulation trick with an effective batch-size (grandient-wise) of 8." } ] }, { "id": 218504, "postDate": "2017-09-04T14:18:54.257Z", "content": "for those using gradient accumulative trick, please note that computation of moving statistics like mean and var of BN may not be accumulative. You have to check the code and documentation of your respective deep learning tools like pytorch, tf, mxnet, caffe ... So there is still difference for using large batch size of accumulating small batch size. In our case, the BN mean and var over one image may be similar to that over many images, so the results may still be ok.
", "rawMarkdown": "for those using gradient accumulative trick, please note that computation of moving statistics like mean and var of BN may not be accumulative. You have to check the code and documentation of your respective deep learning tools like pytorch, tf, mxnet, caffe ... So there is still difference for using large batch size of accumulating small batch size. In our case, the BN mean and var over one image may be similar to that over many images, so the results may still be ok.", "votes": 1, "replies": [ { "id": 218537, "postDate": "2017-09-04T17:03:20.657Z", "content": "Yeah I encountered this problem as well in TF. I simply disregarded BN for this reason, but I imagine one could tie the \"with tf.control_dependencies(update_ops)\" to the logits output somehow instead of the training op, but I'm frankly unsure how and when the statistics in BN get updated in Tensorflow\n, so I could be wrong.
\n\nOne can use weight norm though instead: https://arxiv.org/abs/1602.07868
", "rawMarkdown": "Yeah I encountered this problem as well in TF. I simply disregarded BN for this reason, but I imagine one could tie the \"with tf.control_dependencies(update_ops)\" to the logits output somehow instead of the training op, but I'm frankly unsure how and when the statistics in BN get updated in Tensorflow\n, so I could be wrong.\n\nOne can use weight norm though instead: https://arxiv.org/abs/1602.07868" } ] }, { "id": 218149, "postDate": "2017-09-02T14:56:57.747Z", "content": "In my experiments , 1280*1280 got better LB score(I forked Peter's code with little modification)
", "rawMarkdown": "In my experiments , 1280*1280 got better LB score(I forked Peter's code with little modification)", "votes": 1, "replies": [ { "id": 218381, "postDate": "2017-09-04T02:11:03.353Z", "content": "Could you please show your batch size?
", "rawMarkdown": "Could you please show your batch size?" }, { "id": 218391, "postDate": "2017-09-04T03:40:39.967Z", "content": "batch size = 3
", "rawMarkdown": "batch size = 3\n" } ] }, { "id": 217902, "postDate": "2017-09-01T13:16:30.650Z", "content": "In my experience, going rectangle vs square does not change much (0.0001 in some cases, 0 in others).\nSticking to rectangles though, to feel better with myself! Feels more correct!
", "rawMarkdown": "In my experience, going rectangle vs square does not change much (0.0001 in some cases, 0 in others).\nSticking to rectangles though, to feel better with myself! Feels more correct!", "votes": 2 }, { "id": 219298, "postDate": "2017-09-07T16:35:26.813Z", "content": "I get lb score of 0.9966 for 1024 but only 0.9962 for 1280
", "rawMarkdown": "I get lb score of 0.9966 for 1024 but only 0.9962 for 1280", "replies": [ { "id": 219394, "postDate": "2017-09-08T01:15:05.457Z", "content": "What is your optimizer?RMSprop?
", "rawMarkdown": "What is your optimizer?RMSprop?" }, { "id": 219442, "postDate": "2017-09-08T07:37:26.347Z", "content": "i am using sdg with clip gradient
", "rawMarkdown": "i am using sdg with clip gradient" }, { "id": 219452, "postDate": "2017-09-08T08:27:35.627Z", "content": "This is my score, thank you for your advice. In my net, the bigger input size, the better LB score. And my optimizer is RMSprop.\nhttps://www.kaggle.com/c/carvana-image-masking-challenge/discussion/39148
", "rawMarkdown": "This is my score, thank you for your advice. In my net, the bigger input size, the better LB score. And my optimizer is RMSprop.\nhttps://www.kaggle.com/c/carvana-image-masking-challenge/discussion/39148\n" } ] }, { "id": 217703, "postDate": "2017-08-31T17:24:18.953Z", "content": "I've been using full resolution (padding to 1920 though), haven't really tried any other sizes really, although the boost in batch size would be nice.
\n\nOut of curiosity, I guess you are resizing to 1024 not cropping?
", "rawMarkdown": "I've been using full resolution (padding to 1920 though), haven't really tried any other sizes really, although the boost in batch size would be nice. \n\nOut of curiosity, I guess you are resizing to 1024 not cropping? " }, { "id": 218202, "postDate": "2017-09-02T22:14:17.630Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 218212, "postDate": "2017-09-02T23:29:46.700Z", "content": "is that just the input or are you changing the size of the convolutions as well?
", "rawMarkdown": "is that just the input or are you changing the size of the convolutions as well?" }, { "id": 218227, "postDate": "2017-09-03T02:11:15.117Z", "content": "with both. In both case I got better results with 1280*1280
", "rawMarkdown": "with both. In both case I got better results with 1280*1280" } ] } ], "comments": [ { "id": 217770, "author_name": "mshliselberg", "author_url": "", "post_date": "2017-08-31T21:08:47.293000", "content": "Something Ive noticed, is that given a pixel budget, keeping the width to be length*1.5 (~original proportion) does slightly better. Have you tried maintaining your 1024^2 budget and formulate an experiment with something close to 1260x840?
", "votes": 3, "replies": [ { "id": 217776, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2017-08-31T21:48:27.497000", "content": "Are you using a unet or some other architecture?
", "votes": 0, "replies": [] }, { "id": 217778, "author_name": "mshliselberg", "author_url": "", "post_date": "2017-08-31T22:00:36.247000", "content": "using u-net based idea, with input (1104x736 since my model needs both l,w to be 16-divisible)
", "votes": 0, "replies": [] }, { "id": 217784, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2017-08-31T22:52:32.690000", "content": "It is interesting to play with the dimensions of the input images. Some of my results from several aspect ratios are comparable (LB 0.996+) in previous low quality data. I forked Peter's keras starter using unet512.
", "votes": 0, "replies": [] }, { "id": 217823, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2017-09-01T05:09:24.260000", "content": "Is there any increase in computation speed using the rectangle over the square?
", "votes": 2, "replies": [] }, { "id": 217898, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2017-09-01T12:58:07.947000", "content": "I haven't tried square yet. But I guess the computing time would respond to pixel counts. FYI, some random clips of one epoch from logs:
\n\nw0768h0512 (3:2, pixel counts ~0.39M), 642s
\n\nw0960h0640 (3:2 ~0.61M), 975s
\n\nw0768h1024 (3:4 ~0.79M), 1257s
\n\nw1280h0640 (2:1 ~0.82M), 1316s
", "votes": 3, "replies": [] } ] }, { "id": 217678, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2017-08-31T15:46:03.830000", "content": "I am training full resolution, I will try to do an experiment comparing 1024x1024 vs. 1918x1280 and report back.
", "votes": 3, "replies": [ { "id": 218379, "author_name": "Will", "author_url": "", "post_date": "2017-09-04T01:59:17.993000", "content": "It seems that 1918x1280 performs better. \nWhat is your batch size? Mine is 8.
", "votes": 1, "replies": [] }, { "id": 218501, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2017-09-04T14:12:18.073000", "content": "GPU batch size = 1, b/c I cannot fit more in my GPU (11 Gb), but I used the Adam gradient accumulation trick with an effective batch-size (grandient-wise) of 8.
", "votes": 0, "replies": [] } ] }, { "id": 218504, "author_name": "hengck23", "author_url": "", "post_date": "2017-09-04T14:18:54.257000", "content": "for those using gradient accumulative trick, please note that computation of moving statistics like mean and var of BN may not be accumulative. You have to check the code and documentation of your respective deep learning tools like pytorch, tf, mxnet, caffe ... So there is still difference for using large batch size of accumulating small batch size. In our case, the BN mean and var over one image may be similar to that over many images, so the results may still be ok.
", "votes": 1, "replies": [ { "id": 218537, "author_name": "Adam Hart", "author_url": "", "post_date": "2017-09-04T17:03:20.657000", "content": "Yeah I encountered this problem as well in TF. I simply disregarded BN for this reason, but I imagine one could tie the \"with tf.control_dependencies(update_ops)\" to the logits output somehow instead of the training op, but I'm frankly unsure how and when the statistics in BN get updated in Tensorflow\n, so I could be wrong.
\n\nOne can use weight norm though instead: https://arxiv.org/abs/1602.07868
", "votes": 0, "replies": [] } ] }, { "id": 218149, "author_name": "atom1231", "author_url": "", "post_date": "2017-09-02T14:56:57.747000", "content": "In my experiments , 1280*1280 got better LB score(I forked Peter's code with little modification)
", "votes": 1, "replies": [ { "id": 218381, "author_name": "Will", "author_url": "", "post_date": "2017-09-04T02:11:03.353000", "content": "Could you please show your batch size?
", "votes": 0, "replies": [] }, { "id": 218391, "author_name": "atom1231", "author_url": "", "post_date": "2017-09-04T03:40:39.967000", "content": "batch size = 3
", "votes": 0, "replies": [] } ] }, { "id": 217902, "author_name": "Davide Boschetto", "author_url": "", "post_date": "2017-09-01T13:16:30.650000", "content": "In my experience, going rectangle vs square does not change much (0.0001 in some cases, 0 in others).\nSticking to rectangles though, to feel better with myself! Feels more correct!
", "votes": 2, "replies": [] }, { "id": 219298, "author_name": "hengck23", "author_url": "", "post_date": "2017-09-07T16:35:26.813000", "content": "I get lb score of 0.9966 for 1024 but only 0.9962 for 1280
", "votes": 0, "replies": [ { "id": 219394, "author_name": "Will", "author_url": "", "post_date": "2017-09-08T01:15:05.457000", "content": "What is your optimizer?RMSprop?
", "votes": 0, "replies": [] }, { "id": 219442, "author_name": "hengck23", "author_url": "", "post_date": "2017-09-08T07:37:26.347000", "content": "i am using sdg with clip gradient
", "votes": 0, "replies": [] }, { "id": 219452, "author_name": "Will", "author_url": "", "post_date": "2017-09-08T08:27:35.627000", "content": "This is my score, thank you for your advice. In my net, the bigger input size, the better LB score. And my optimizer is RMSprop.\nhttps://www.kaggle.com/c/carvana-image-masking-challenge/discussion/39148
", "votes": 0, "replies": [] } ] }, { "id": 217703, "author_name": "Adam Hart", "author_url": "", "post_date": "2017-08-31T17:24:18.953000", "content": "I've been using full resolution (padding to 1920 though), haven't really tried any other sizes really, although the boost in batch size would be nice.
\n\nOut of curiosity, I guess you are resizing to 1024 not cropping?
", "votes": 0, "replies": [] }, { "id": 218202, "author_name": "", "author_url": "", "post_date": "2017-09-02T22:14:17.630000", "content": "", "votes": 2, "replies": [ { "id": 218212, "author_name": "Alan Khoa Nguyen", "author_url": "", "post_date": "2017-09-02T23:29:46.700000", "content": "is that just the input or are you changing the size of the convolutions as well?
", "votes": 0, "replies": [] }, { "id": 218227, "author_name": "Charles Jansen", "author_url": "", "post_date": "2017-09-03T02:11:15.117000", "content": "with both. In both case I got better results with 1280*1280
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "217664": "In my experiments, 1024x1024 seems to be sufficient. Beyond that, the gain is not very much. \nCan any kagglers confirmed this?\n", "217770": "Something Ive noticed, is that given a pixel budget, keeping the width to be length*1.5 (~original proportion) does slightly better. Have you tried maintaining your 1024^2 budget and formulate an experiment with something close to 1260x840? ", "217678": "I am training full resolution, I will try to do an experiment comparing 1024x1024 vs. 1918x1280 and report back. ", "218504": "for those using gradient accumulative trick, please note that computation of moving statistics like mean and var of BN may not be accumulative. You have to check the code and documentation of your respective deep learning tools like pytorch, tf, mxnet, caffe ... So there is still difference for using large batch size of accumulating small batch size. In our case, the BN mean and var over one image may be similar to that over many images, so the results may still be ok.", "218149": "In my experiments , 1280*1280 got better LB score(I forked Peter's code with little modification)", "217902": "In my experience, going rectangle vs square does not change much (0.0001 in some cases, 0 in others).\nSticking to rectangles though, to feel better with myself! Feels more correct!", "219298": "I get lb score of 0.9966 for 1024 but only 0.9962 for 1280", "217703": "I've been using full resolution (padding to 1920 though), haven't really tried any other sizes really, although the boost in batch size would be nice. \n\nOut of curiosity, I guess you are resizing to 1024 not cropping? ", "218202": "" } }