![\"enter](\"https://image.prntscr.com/image/R2F_OexrROmTMgUloNmeGQ.png\" "\\"\\"")
I've been trying to replicate @Heng CherKeng's results in Keras. I've implemented 128x128, 256x256 and 512x512 U-nets. With U-net_128 I get 0.988 LB score, which seems to be in line with Heng's results.
\n\nCode: https://github.com/petrosgk/Kaggle-Carvana-Image-Masking-Challenge
\n\nHope it'll be of use and thanks again to @Heng CherKeng for sharing his work!
\n\nUpdate 28.8.2017
\n\nUpdate 15.8.2017
\n\nUpdate 10.8.2017:
\n\nUpdate 9.8.2017:
\n\nI was able to hit 0.995 with 512x512 U-net now (batch size = 4). Scores for the smaller U-nets improved as well.
", "messages": [ { "id": 209932, "postDate": "2017-08-03T20:05:14.730Z", "content": "I've been trying to replicate @Heng CherKeng's results in Keras. I've implemented 128x128, 256x256 and 512x512 U-nets. With U-net_128 I get 0.988 LB score, which seems to be in line with Heng's results.
\n\nCode: https://github.com/petrosgk/Kaggle-Carvana-Image-Masking-Challenge
\n\nHope it'll be of use and thanks again to @Heng CherKeng for sharing his work!
\n\nUpdate 28.8.2017
\n\nUpdate 15.8.2017
\n\nUpdate 10.8.2017:
\n\nUpdate 9.8.2017:
\n\nI was able to hit 0.995 with 512x512 U-net now (batch size = 4). Scores for the smaller U-nets improved as well.
", "rawMarkdown": "I've been trying to replicate @Heng CherKeng's results in Keras. I've implemented 128x128, 256x256 and 512x512 U-nets. With U-net_128 I get 0.988 LB score, which seems to be in line with Heng's results.\n\nCode: https://github.com/petrosgk/Kaggle-Carvana-Image-Masking-Challenge\n\nHope it'll be of use and thanks again to @Heng CherKeng for sharing his work!\n\n----------------------------------------------------------------------------------------------------------------------\n\nUpdate 28.8.2017\n\n- Added loss with weighted boundary.\n\nUpdate 15.8.2017\n\n- Added Hue/Saturation/Value augmentation.\n- Switched to RMSprop optimizer as default.\n- Added multithreaded inference with inference and data loading done on separate threads. This reduced inference time by 40% in my tests. You can run 'test_submit_multithreaded.py' to try it.\n\nUpdate 10.8.2017: \n\n - Added 1024x1024 U-net (LB 0.996)\n - Not using predict_generator anymore due to memory constraints with large input\n\nUpdate 9.8.2017: \n\n - Now using Binary Crossentropy Dice Loss in place of Binary Crossentropy\n - Callbacks now use val_dice_loss as a metric in place of val_loss\n\nI was able to hit 0.995 with 512x512 U-net now (batch size = 4). Scores for the smaller U-nets improved as well.", "votes": 101 }, { "id": 210299, "postDate": "2017-08-05T02:17:57.363Z", "content": "Just a quick data point: The 512x512 U-net (with batch size = 8) from Peter's code gave a LB score of 0.993. Training took 27 epochs @ 5.4 min per epoch on a 1080 Ti GPU.
", "rawMarkdown": "Just a quick data point: The 512x512 U-net (with batch size = 8) from Peter's code gave a LB score of 0.993. Training took 27 epochs @ 5.4 min per epoch on a 1080 Ti GPU.", "votes": 7, "replies": [ { "id": 210343, "postDate": "2017-08-05T08:05:24.820Z", "content": "Thanks for the info. I get LB 0.992 with 256x256 U-net. Expected 512x512 U-net to be in the ~0.995 range but I myself can't run it as I get a segmentation fault after the 1st epoch and I don't know what the cause is :(
", "rawMarkdown": "Thanks for the info. I get LB 0.992 with 256x256 U-net. Expected 512x512 U-net to be in the ~0.995 range but I myself can't run it as I get a segmentation fault after the 1st epoch and I don't know what the cause is :(", "votes": 1 }, { "id": 210395, "postDate": "2017-08-05T12:47:38.990Z", "content": "I also got 0.992 @256x256 and thought 512x512 would yield a bit more. My hunch is it has something to do with the small batch size, I have plenty of memory for a larger batch size but I'm having trouble convincing tensorflow otherwise.
", "rawMarkdown": "I also got 0.992 @256x256 and thought 512x512 would yield a bit more. My hunch is it has something to do with the small batch size, I have plenty of memory for a larger batch size but I'm having trouble convincing tensorflow otherwise. ", "votes": 1 }, { "id": 210740, "postDate": "2017-08-06T21:29:18.207Z", "content": "I was able to get a validation dice loss of 0.9919 using the 512 U-net and a batch size of 4. Did you train with the entire train set in the end, or leave some samples out for validation?
", "rawMarkdown": "I was able to get a validation dice loss of 0.9919 using the 512 U-net and a batch size of 4. Did you train with the entire train set in the end, or leave some samples out for validation?", "votes": 1 }, { "id": 210830, "postDate": "2017-08-07T08:11:53.917Z", "content": "I confirm that I got 0.995 using 512x512 using your code. Made few changes in learning rate. Batch size was 12.
", "rawMarkdown": "I confirm that I got 0.995 using 512x512 using your code. Made few changes in learning rate. Batch size was 12.", "votes": 2 }, { "id": 210841, "postDate": "2017-08-07T09:29:41.850Z", "content": "Mh, that's a rather good score I could not reproduce. Did you get the same result on the validation score? What kind of GPU are you using?
", "rawMarkdown": "Mh, that's a rather good score I could not reproduce. Did you get the same result on the validation score? What kind of GPU are you using?" }, { "id": 210846, "postDate": "2017-08-07T09:41:13.520Z", "content": "LB and CV are close. I am using 1080Ti.
", "rawMarkdown": "LB and CV are close. I am using 1080Ti." }, { "id": 218384, "postDate": "2017-09-04T02:33:32.820Z", "content": "Your best net is 1024x1024 u-net with batch_size=8?
", "rawMarkdown": "Your best net is 1024x1024 u-net with batch_size=8?" } ] }, { "id": 210500, "postDate": "2017-08-06T00:29:10.327Z", "content": "Hi Peter. Thanks again for your sharing. A question about the \"steps\" when making prediction on the test set. Is there a reason you added one in the steps? After adding one, the \"steps\" is not a factor of len(ids_test_split). I cited the code here. \n preds = model.predict_generator(generator=test_generator(),\n steps=(len(ids_test_split) // batch_size) + 1)
", "rawMarkdown": "Hi Peter. Thanks again for your sharing. A question about the \"steps\" when making prediction on the test set. Is there a reason you added one in the steps? After adding one, the \"steps\" is not a factor of len(ids_test_split). I cited the code here. \n preds = model.predict_generator(generator=test_generator(),\n steps=(len(ids_test_split) // batch_size) + 1)", "votes": 3, "replies": [ { "id": 210510, "postDate": "2017-08-06T02:02:20.887Z", "content": "The thinking was that if 'len(ids_test_split)' is not exactly divisible by 'batch_size' then 'steps' will be 1 less than needed and the 'test_generator' will leave out some samples at the end.
\n\nEg. if 'test_splits = 8' then 'len(ids_tests_split) = 100064 / 8 = 12508'. With 'batch_size = 8' we get 'steps = len(ids_test_split) // batch_size = 1563'. This would generate 1563 * batch_size = 1563 * 8 = 12504 samples instead of expected 12508. With +1 step we can get the last batch of 4 samples.
\n\nHowever if 'len(ids_test_split)' is exactly divisible by 'batch_size' then +1 is not needed and will produce an error. I think a better way to calculate steps is 'steps = np.ceil(float(len(ids_test_split)) / float(batch_size))'. This should work in all cases. Thanks for spotting the bug!
", "rawMarkdown": "The thinking was that if 'len(ids_test_split)' is not exactly divisible by 'batch_size' then 'steps' will be 1 less than needed and the 'test_generator' will leave out some samples at the end.\n\nEg. if 'test_splits = 8' then 'len(ids_tests_split) = 100064 / 8 = 12508'. With 'batch_size = 8' we get 'steps = len(ids_test_split) // batch_size = 1563'. This would generate 1563 * batch_size = 1563 * 8 = 12504 samples instead of expected 12508. With +1 step we can get the last batch of 4 samples.\n\nHowever if 'len(ids_test_split)' is exactly divisible by 'batch_size' then +1 is not needed and will produce an error. I think a better way to calculate steps is 'steps = np.ceil(float(len(ids_test_split)) / float(batch_size))'. This should work in all cases. Thanks for spotting the bug!", "votes": 5 } ] }, { "id": 215652, "postDate": "2017-08-22T14:40:59.330Z", "content": "Thank you for sharing!
\n\n'dice_loss' function in model/losses.py is actually 'dice_coef'.
", "rawMarkdown": "Thank you for sharing!\n\n'dice_loss' function in model/losses.py is actually 'dice_coef'.", "votes": 1 }, { "id": 210820, "postDate": "2017-08-07T07:22:00.340Z", "content": "Why do you use crossentropy instead of minimizing dice score directly in training?
", "rawMarkdown": "Why do you use crossentropy instead of minimizing dice score directly in training?", "votes": 1, "replies": [ { "id": 211527, "postDate": "2017-08-09T09:41:51.357Z", "content": "I observed slightly better results minimizing bce. Now I'm minimizing (bce + dice loss) which seems to perform best.
", "rawMarkdown": "I observed slightly better results minimizing bce. Now I'm minimizing (bce + dice loss) which seems to perform best.", "votes": 1 }, { "id": 213491, "postDate": "2017-08-14T22:56:47.737Z", "content": "How much better in LB score with minimizing BCE+Dice vs plain Dice, if I may ask?
", "rawMarkdown": "How much better in LB score with minimizing BCE+Dice vs plain Dice, if I may ask?" }, { "id": 216424, "postDate": "2017-08-25T18:41:20.763Z", "content": "I was doing testing of DICE versus (BCE+DICE)/2 using a little bit different implementation of UNet, though it may apply to Keras implementation too:
\n\nDICE\n full size validation: 0.993052\n LB score: 0.993\n(BCE+DICE) / 2\n full size validation: 0.993567\n LB score: 0.994\nThe division by 2 is to keep DICE and BCE+DICE in the same value range, otherwise learning rate needs to be adjusted to match.
", "rawMarkdown": "I was doing testing of DICE versus (BCE+DICE)/2 using a little bit different implementation of UNet, though it may apply to Keras implementation too:\n\n DICE\n full size validation: 0.993052\n LB score: 0.993\n (BCE+DICE) / 2\n full size validation: 0.993567\n LB score: 0.994\n\nThe division by 2 is to keep DICE and BCE+DICE in the same value range, otherwise learning rate needs to be adjusted to match.", "votes": 1 }, { "id": 216429, "postDate": "2017-08-25T18:59:24.370Z", "content": "Thanks! Did you try with BCE only?
", "rawMarkdown": "Thanks! Did you try with BCE only? " }, { "id": 216471, "postDate": "2017-08-25T21:51:29.833Z", "content": "I did try BCE only. Did not write it down for the same model configuration, but it was less than (BCE+DICE) / 2.
", "rawMarkdown": "I did try BCE only. Did not write it down for the same model configuration, but it was less than (BCE+DICE) / 2.", "votes": 1 } ] }, { "id": 224910, "postDate": "2017-09-27T22:44:08.130Z", "content": "Thanks a lot for this well-documented code! I learned a lot.
", "rawMarkdown": "Thanks a lot for this well-documented code! I learned a lot. ", "votes": 2 }, { "id": 216494, "postDate": "2017-08-26T01:43:24.610Z", "content": "Weighing pixels near boundary is suggested in another thread (https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38125)\nI am thinking of doing this by making a custom loss function and encode closeness to the border somehow in y_true, but I thought I might ask here first if anyone has implemented it.
", "rawMarkdown": "Weighing pixels near boundary is suggested in another thread (https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38125)\nI am thinking of doing this by making a custom loss function and encode closeness to the border somehow in y_true, but I thought I might ask here first if anyone has implemented it.", "votes": 2, "replies": [ { "id": 216637, "postDate": "2017-08-27T04:36:42.727Z", "content": "Keras implementation is here. \nhttps://www.kaggle.com/lyakaap/weighing-boundary-pixels-loss-script-by-keras2
", "rawMarkdown": "Keras implementation is here. \nhttps://www.kaggle.com/lyakaap/weighing-boundary-pixels-loss-script-by-keras2", "votes": 5 }, { "id": 216849, "postDate": "2017-08-28T11:41:22.590Z", "content": "I've added it to the repo. Thanks.
", "rawMarkdown": "I've added it to the repo. Thanks." }, { "id": 216947, "postDate": "2017-08-28T19:00:58.050Z", "content": "@lyakaap, thanks for the implementation and sharing. I tried out this loss function (weighted BCE + weighted dice) and it seemed to be quite unstable - after about 6 epochs, the validation Dice score jumped back to 0.68 after reaching 0.99 in one epoch. The (BCE + Dice) function tracked the dice score pretty nicely. I tried coding a function myself and it was also instable - did anyone else have luck with this ? Is the instability to be expected ?
", "rawMarkdown": "@lyakaap, thanks for the implementation and sharing. I tried out this loss function (weighted BCE + weighted dice) and it seemed to be quite unstable - after about 6 epochs, the validation Dice score jumped back to 0.68 after reaching 0.99 in one epoch. The (BCE + Dice) function tracked the dice score pretty nicely. I tried coding a function myself and it was also instable - did anyone else have luck with this ? Is the instability to be expected ?" }, { "id": 216954, "postDate": "2017-08-28T19:22:32.427Z", "content": "I've tried weighted dice loss (without BCE) and while CV was higher than plain dice loss (0.9947 vs 0.9940), LB score was lower (0.990 vs 0.991). That's with 128x128 u-net.
", "rawMarkdown": "I've tried weighted dice loss (without BCE) and while CV was higher than plain dice loss (0.9947 vs 0.9940), LB score was lower (0.990 vs 0.991). That's with 128x128 u-net.", "votes": 1 }, { "id": 216975, "postDate": "2017-08-28T20:28:03.790Z", "content": "Is the average mask the best for weighting the loss?
", "rawMarkdown": "Is the average mask the best for weighting the loss?" }, { "id": 216989, "postDate": "2017-08-28T21:56:39.220Z", "content": "I thought instability of weighted dice loss might be caused from intersection processing. It's not consider weighing when label=0, but actually I don't come up with the way to weighing both label=0 and 1.
", "rawMarkdown": "I thought instability of weighted dice loss might be caused from intersection processing. It's not consider weighing when label=0, but actually I don't come up with the way to weighing both label=0 and 1." }, { "id": 217085, "postDate": "2017-08-29T10:37:13.320Z", "content": "With input images of 512x512 and a kernel size for the weights of 31, it should be 21, else I get the following error:
\n\nValueError: Shape must be rank 4 but is rank 3 for 'AvgPool' (op: 'AvgPool') with input shapes: [?,?,?].
", "rawMarkdown": "With input images of 512x512 and a kernel size for the weights of 31, it should be 21, else I get the following error:\n\nValueError: Shape must be rank 4 but is rank 3 for 'AvgPool' (op: 'AvgPool') with input shapes: [?,?,?]." }, { "id": 217086, "postDate": "2017-08-29T10:43:42.993Z", "content": "Sorry, kernel size for 512 was supposed to be 21. I've fixed the typo.
", "rawMarkdown": "Sorry, kernel size for 512 was supposed to be 21. I've fixed the typo.", "votes": 1 }, { "id": 217363, "postDate": "2017-08-30T10:14:47.490Z", "content": "How about for 1280x1280 input?
", "rawMarkdown": "How about for 1280x1280 input?", "votes": 1 }, { "id": 217376, "postDate": "2017-08-30T11:18:35.953Z", "content": "41 kernel size (same as for 1024 input) should work.
", "rawMarkdown": "41 kernel size (same as for 1024 input) should work." }, { "id": 218283, "postDate": "2017-09-03T10:42:08.020Z", "content": "Thanks for your sharing. I have tried weighted bce dice loss and while CV was higher than plain dice loss , LB score was lower (same score as Peter). That's with 1024x1024 u-net.And my batch size is 8.
", "rawMarkdown": "Thanks for your sharing. I have tried weighted bce dice loss and while CV was higher than plain dice loss , LB score was lower (same score as Peter). That's with 1024x1024 u-net.And my batch size is 8." }, { "id": 218285, "postDate": "2017-09-03T10:47:51.050Z", "content": "Have you tried 1280x1280? My input is 1024x1024.
", "rawMarkdown": "Have you tried 1280x1280? My input is 1024x1024." }, { "id": 218291, "postDate": "2017-09-03T12:45:48.783Z", "content": "I'll get back to you tomorrow, I do have a 1024x1024 weighted loss version running.
", "rawMarkdown": "I'll get back to you tomorrow, I do have a 1024x1024 weighted loss version running." } ] }, { "id": 213718, "postDate": "2017-08-15T05:32:05.653Z", "content": "华人Kaggle交流群, 请加我WX: dragen1860, 备注:kaggle, 目前已有30+华人入群，欢迎大家交流Main idea & tricks.
", "rawMarkdown": "华人Kaggle交流群, 请加我WX: dragen1860, 备注:kaggle, 目前已有30+华人入群，欢迎大家交流Main idea & tricks.", "votes": -16 }, { "id": 213309, "postDate": "2017-08-14T12:34:36.620Z", "content": "Did anyone had success running this code on CPU? I just wonder how long each epoch would take...
", "rawMarkdown": "Did anyone had success running this code on CPU? I just wonder how long each epoch would take...", "votes": -1, "replies": [ { "id": 216379, "postDate": "2017-08-25T13:48:39.647Z", "content": "I was interested in this, so I tested it. 1024x1024 U-net takes 13900 seconds an epoch on average. Model converges around 25 epochs, so it takes 4-5 days to train.
\n\nOn a GPU (1080Ti) it takes 8 hours to train.
", "rawMarkdown": "I was interested in this, so I tested it. 1024x1024 U-net takes 13900 seconds an epoch on average. Model converges around 25 epochs, so it takes 4-5 days to train.\n\nOn a GPU (1080Ti) it takes 8 hours to train." } ] }, { "id": 779888, "postDate": "2020-03-19T19:25:20.783Z", "content": "Thanks a lot for sharing your work, but I experienced something I can't understand.\nIn the training process, I got pretty good training dice coef loss ~= .80, in the same time validation loss ~= .20 just after 1 epoch, which kinda weird.\nI removed all batch normalization layers from uNet architecture, and boom all things go as expected.
\n\nMy settings are:\nuNet version: 128\nbatch size = 32\noptimizer= RMSprop(.0001)\nloss: bce_dice_loss\n metrics: [dice_coef]
", "rawMarkdown": "Thanks a lot for sharing your work, but I experienced something I can't understand.\nIn the training process, I got pretty good training dice coef loss ~= .80, in the same time validation loss ~= .20 just after 1 epoch, which kinda weird.\nI removed all batch normalization layers from uNet architecture, and boom all things go as expected. \n\nMy settings are:\nuNet version: 128\nbatch size = 32\noptimizer= RMSprop(.0001)\nloss: bce_dice_loss\n metrics: [dice_coef]" }, { "id": 308905, "postDate": "2018-04-04T10:00:47.263Z", "content": "Thanks a lot for the code, I'm trying to learn deep learning and your code help me to understand it very well. Nice work! love your code very much
", "rawMarkdown": "Thanks a lot for the code, I'm trying to learn deep learning and your code help me to understand it very well. Nice work! love your code very much" }, { "id": 272516, "postDate": "2018-01-23T07:15:00.113Z", "content": "Hi Peter. Thanks for your sharing. could you tell me how to get \"train_bounds.csv\" in your project
", "rawMarkdown": "Hi Peter. Thanks for your sharing. could you tell me how to get \"train_bounds.csv\" in your project" }, { "id": 224515, "postDate": "2017-09-26T16:51:25.813Z", "content": "This is fantastic work, thanks Peter.
\n\nOne question: why don't you use the use_multiprocessing=True and/or workers=X arguments in \"train.py\" last cell ?\nhttps://keras.io/models/sequential/#fit_generator
Is there any AWS AMI with Keras2.0? It seems that they're all Keras1.
", "rawMarkdown": "Is there any AWS AMI with Keras2.0? It seems that they're all Keras1.", "replies": [ { "id": 223811, "postDate": "2017-09-23T17:43:23.247Z", "content": "Can't say that I've come across any. You might try using one of the newer AMIs in the Ireland region and upgrading Keras with pip or conda.
", "rawMarkdown": "Can't say that I've come across any. You might try using one of the newer AMIs in the Ireland region and upgrading Keras with pip or conda. " }, { "id": 224256, "postDate": "2017-09-25T17:33:05.883Z", "content": "Maybe this tweet by François can help ?
\n\n\"These are the steps you now have to take if you use the AWS Deep Learning AMI and you need to use Keras.
\n\nhttps://twitter.com/fchollet/status/912127671823253504
", "rawMarkdown": "Maybe this tweet by François can help ?\n\n\"These are the steps you now have to take if you use the AWS Deep Learning AMI and you need to use Keras.\n\nhttps://twitter.com/fchollet/status/912127671823253504" } ] }, { "id": 218547, "postDate": "2017-09-04T18:58:14.290Z", "content": "ran into the following error while executing the train.py\nValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None
can anyone help?
", "rawMarkdown": "ran into the following error while executing the train.py\nValueError: output of generator should be a tuple `(x, y, sample_weight)` or `(x, y)`. Found: None\n\ncan anyone help?", "replies": [ { "id": 218548, "postDate": "2017-09-04T19:07:50.007Z", "content": "Check that the file exists and is being parsed properly
", "rawMarkdown": "Check that the file exists and is being parsed properly" } ] }, { "id": 218435, "postDate": "2017-09-04T07:34:52.300Z", "content": "Hi, Peter, thank you for your sharing. Why do you use the function randomHueSaturationValue? It seemed that it didn't make any change to our images.
This augmentation is meant to randomize the color of a car and the overall brightness of the image:
\n\n
You can also see my post in this thread. Note that in images where the car is black-white-gray, the hue changes won't have much of an effect.
", "rawMarkdown": "This augmentation is meant to randomize the color of a car and the overall brightness of the image:\n\n![enter image description here][1]\n\n\nYou can also see my post in [this thread][2]. Note that in images where the car is black-white-gray, the hue changes won't have much of an effect.\n\n\n [1]: https://image.prntscr.com/image/R2F_OexrROmTMgUloNmeGQ.png\n [2]: https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37208#212703", "votes": 3 } ] }, { "id": 217028, "postDate": "2017-08-29T04:47:11.557Z", "content": "Nice work.\nWhen I run this code,i did not get the h5py file in the weights folder.What should I do?
", "rawMarkdown": "Nice work.\nWhen I run this code,i did not get the h5py file in the weights folder.What should I do?", "replies": [ { "id": 217048, "postDate": "2017-08-29T06:48:13.433Z", "content": "Install h5py package:
\n\npip install h5py --upgrade
", "rawMarkdown": "Install *h5py* package:\n\n*pip install h5py --upgrade*" } ] }, { "id": 216286, "postDate": "2017-08-25T05:10:34.320Z", "content": "Thank you for this great starter. I added multi GPU support for submission generation. Should scale almost linear. https://github.com/fks/Kaggle-Carvana-Image-Masking-Challenge
", "rawMarkdown": "Thank you for this great starter. I added multi GPU support for submission generation. Should scale almost linear. [https://github.com/fks/Kaggle-Carvana-Image-Masking-Challenge][1]\n\n\n [1]: https://github.com/fks/Kaggle-Carvana-Image-Masking-Challenge", "replies": [ { "id": 216299, "postDate": "2017-08-25T06:23:44.007Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" }, { "id": 216460, "postDate": "2017-08-25T21:01:40.667Z", "content": "How much was the time gain using multi gpu script?
", "rawMarkdown": "How much was the time gain using multi gpu script?" }, { "id": 216521, "postDate": "2017-08-26T08:04:34.100Z", "content": "For a larger model I went down from 5h per submission with a single GTX1080ti down to 3h with GTX1080ti + GTX1070
", "rawMarkdown": "For a larger model I went down from 5h per submission with a single GTX1080ti down to 3h with GTX1080ti + GTX1070" } ] }, { "id": 216124, "postDate": "2017-08-24T11:53:59.983Z", "content": "How much time does this model take to training one epoch? I find it takes about 20 mins for one epoch on k80 GPU.
", "rawMarkdown": "How much time does this model take to training one epoch? I find it takes about 20 mins for one epoch on k80 GPU.", "replies": [ { "id": 216125, "postDate": "2017-08-24T11:58:57.453Z", "content": "It also takes ~20min per epoch on my GTX 1070 for the 1024x1024 Unet.
", "rawMarkdown": "It also takes ~20min per epoch on my GTX 1070 for the 1024x1024 Unet." }, { "id": 222236, "postDate": "2017-09-18T06:32:39.533Z", "content": "What is your batch size malcolm?\nIt takes around 40 mins for 1 epochs on K80 for me with 256x256 images...with batch of 4.
", "rawMarkdown": "What is your batch size malcolm?\nIt takes around 40 mins for 1 epochs on K80 for me with 256x256 images...with batch of 4." } ] }, { "id": 216120, "postDate": "2017-08-24T11:39:05.043Z", "content": "Hi Peter, thanks for your share. One question, do you use boundary weighting in your model?
", "rawMarkdown": "Hi Peter, thanks for your share. One question, do you use boundary weighting in your model?", "replies": [ { "id": 216122, "postDate": "2017-08-24T11:47:37.863Z", "content": "I haven't implemented weighted dice loss yet.
", "rawMarkdown": "I haven't implemented weighted dice loss yet." } ] }, { "id": 214890, "postDate": "2017-08-18T17:25:54.707Z", "content": "Thank you very much \nthis starter helped me a lot
", "rawMarkdown": "Thank you very much \nthis starter helped me a lot" }, { "id": 214545, "postDate": "2017-08-17T09:53:36.847Z", "content": "Hi Peter,
\n\nThanks for your code, it's really a nice work. But I have a question, obviously binary_crossentropy loss is supposed to be used in this problem and I found you add the sigmoid activation in the last layer. While I found that binary_crossentropy function in Keras invokes sigmoid_cross_entropy_with_logits in Tensorflow (if you are using tf as backend). But as I found in the documentation and source code:
\n\nhttps://www.tensorflow.org/versions/master/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits
\n\nThe sigmoid_cross_entropy_with_logits calculates sigmoid(y_pred) again inside the function. So I don't think it makes sense that add a sigmoid in the last layer. Could you help me have a check? Thanks.
", "rawMarkdown": "Hi Peter,\n\nThanks for your code, it's really a nice work. But I have a question, obviously **binary_crossentropy** loss is supposed to be used in this problem and I found you add the sigmoid activation in the last layer. While I found that **binary_crossentropy** function in Keras invokes **sigmoid_cross_entropy_with_logits** in Tensorflow (if you are using tf as backend). But as I found in the documentation and source code:\n\nhttps://www.tensorflow.org/versions/master/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits\n\nThe **sigmoid_cross_entropy_with_logits** calculates sigmoid(y_pred) again inside the function. So I don't think it makes sense that add a sigmoid in the last layer. Could you help me have a check? Thanks.", "replies": [ { "id": 214549, "postDate": "2017-08-17T10:33:09.663Z", "content": "I looked at Keras backend binary_crossentropy function and it looks like this:
\n\ndef binary_crossentropy(output, target, from_logits=False):\n \"\"\"Binary crossentropy between an output tensor and a target tensor.\n\n # Arguments\n output: A tensor.\n target: A tensor with the same shape as `output`.\n from_logits: Whether `output` is expected to be a logits tensor.\n By default, we consider that `output`\n encodes a probability distribution.\n\n # Returns\n A tensor.\n \"\"\"\n # Note: tf.nn.sigmoid_cross_entropy_with_logits\n # expects logits, Keras expects probabilities.\n if not from_logits:\n # transform back to logits\n epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)\n output = tf.clip_by_value(output, epsilon, 1 - epsilon)\n output = tf.log(output / (1 - output))\n\n return tf.nn.sigmoid_cross_entropy_with_logits(labels=target,\n logits=output)\nKeras expects probabilities while TF's sigmoid_cross_entropy_with_logits function expects logits, so Keras converts the probabilities passed to its binary_crossentropy to logits before passing them to sigmoid_cross_entropy_with_logits.
\n\nSo the conversion path is like: sigmoid network output -> logits (Keras backend) -> sigmoid (TF)
\n\nThis is a design choice of Keras. Also, in the training example in Keras documentation (https://keras.io/getting-started/sequential-model-guide/#training) binary_crossentropy is coupled with sigmoid activation as the network's last layer.
", "rawMarkdown": "I looked at Keras backend **binary_crossentropy** function and it looks like this:\n\n def binary_crossentropy(output, target, from_logits=False):\n \"\"\"Binary crossentropy between an output tensor and a target tensor.\n\n # Arguments\n output: A tensor.\n target: A tensor with the same shape as `output`.\n from_logits: Whether `output` is expected to be a logits tensor.\n By default, we consider that `output`\n encodes a probability distribution.\n\n # Returns\n A tensor.\n \"\"\"\n # Note: tf.nn.sigmoid_cross_entropy_with_logits\n # expects logits, Keras expects probabilities.\n if not from_logits:\n # transform back to logits\n epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)\n output = tf.clip_by_value(output, epsilon, 1 - epsilon)\n output = tf.log(output / (1 - output))\n\n return tf.nn.sigmoid_cross_entropy_with_logits(labels=target,\n logits=output)\n\nKeras expects probabilities while TF's **sigmoid_cross_entropy_with_logits** function expects logits, so Keras converts the probabilities passed to its **binary_crossentropy** to logits before passing them to **sigmoid_cross_entropy_with_logits**.\n\nSo the conversion path is like: **sigmoid network output -> logits (Keras backend) -> sigmoid (TF)**\n\nThis is a design choice of Keras. Also, in the training example in Keras documentation (https://keras.io/getting-started/sequential-model-guide/#training) binary_crossentropy is coupled with sigmoid activation as the network's last layer." }, { "id": 214644, "postDate": "2017-08-17T17:52:27.737Z", "content": "Sorry, I still don't understand why Keras expects probabilities.
\n\n# Note: nn.softmax_cross_entropy_with_logits\n# expects logits, Keras expects probabilities.\nDoesn't it use the nn.softmax_cross_entropy_with_logits function? I don't think it is valid that couple with sigmoid.
", "rawMarkdown": "Sorry, I still don't understand why Keras expects probabilities.\n\n # Note: nn.softmax_cross_entropy_with_logits\n # expects logits, Keras expects probabilities.\n\nDoesn't it use the **nn.softmax_cross_entropy_with_logits** function? I don't think it is valid that couple with sigmoid." }, { "id": 214653, "postDate": "2017-08-17T18:20:15.607Z", "content": "I mean that the Keras backend binary_crossentropy function expects probabilities by default which then converts to logits before passing them to TF's sigmoid_cross_entropy_with_logits.
\n\nIf you want you can directly call keras.backend.binary_crossentropy (the function in my previous post) with the from_logits=True option and then you can pass it the logits directly instead of probabilities. The higher level keras.losses.binary_crossentropy function (below) does just that but doesn't use the from_logits option, hence it always expects probabilities and expects the backend function it calls to do the conversion:
\n\nimport keras.backend as K\n\ndef binary_crossentropy(y_true, y_pred):\n return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)\nThank you Peter. I understand now.
", "rawMarkdown": "Thank you Peter. I understand now." }, { "id": 214665, "postDate": "2017-08-17T19:03:37.323Z", "content": "No problem. I think it'd be better if they'd exposed the from_logits option of keras.backend.binary_crossentropy through keras.losses.binary_crossentropy but seems they decided towards more abstraction.
", "rawMarkdown": "No problem. I think it'd be better if they'd exposed the **from_logits** option of **keras.backend.binary_crossentropy** through **keras.losses.binary_crossentropy** but seems they decided towards more abstraction.", "votes": 1 } ] }, { "id": 214130, "postDate": "2017-08-16T03:00:46.083Z", "content": "Hi, Peter, you have changed RMSprop as your new optimizer. Could you please tell us if you still use ReduceLROnPlateau ? As we know, the new optimizer is adaptive.
It's not technically needed but it can speed up training. RMSprop will eventually un-stuck itself from a local minima without manually reducing the learning rate (unlike SGD) but it can take longer than manually reducing LR when loss plateaus for a few epochs.
\n\nSince reducing LR is not needed and doesn't have as much of an effect as it does with SGD you could try removing it. I got 0.997 by using RMSProp + ReduceLROnPlateau but it may well be doable without reducing LR. I haven't tested.
", "rawMarkdown": "It's not technically needed but it can speed up training. RMSprop will eventually un-stuck itself from a local minima without manually reducing the learning rate (unlike SGD) but it can take longer than manually reducing LR when loss plateaus for a few epochs. \n\nSince reducing LR is not needed and doesn't have as much of an effect as it does with SGD you could try removing it. I got 0.997 by using RMSProp + ReduceLROnPlateau but it may well be doable without reducing LR. I haven't tested. " } ] }, { "id": 214037, "postDate": "2017-08-15T19:51:19.573Z", "content": "Thanks Peter so much, really nice code.
\n\nHave you seen this kernel https://www.kaggle.com/alekseit/simple-bounding-boxes ?\nIf you can add it to your code, I am interested to know how to implement the crop on the fly during training.
", "rawMarkdown": "Thanks Peter so much, really nice code.\n\nHave you seen this kernel https://www.kaggle.com/alekseit/simple-bounding-boxes ?\nIf you can add it to your code, I am interested to know how to implement the crop on the fly during training.", "replies": [ { "id": 214060, "postDate": "2017-08-15T21:12:56.460Z", "content": "Looks interesting. Seems like I've hit the wall at 0.997 so I will probably try it if it improves results. Thanks.
", "rawMarkdown": "Looks interesting. Seems like I've hit the wall at 0.997 so I will probably try it if it improves results. Thanks." }, { "id": 215026, "postDate": "2017-08-19T08:48:08.773Z", "content": "I tried the code, but it seemed to behave worse. I cropped the original image in that way, and then resized it into 512×512, but the dice_loss I got is lower.
Hi, Peter, thank you for your code very much. I ran your code just now, but there is an error saying\n```\n File \"E:/kaggle/Kaggle-Carvana-Image-Masking-Challenge-master/Kaggle-Carvana-Image-Masking-Challenge-master/train.py\", line 140, in
", "rawMarkdown": "Hi, Peter, thank you for your code very much. I ran your code just now, but there is an error saying\n```\n File \"E:/kaggle/Kaggle-Carvana-Image-Masking-Challenge-master/Kaggle-Carvana-Image-Masking-Challenge-master/train.py\", line 140, in ", "replies": [ { "id": 213806, "postDate": "2017-08-15T09:19:10.777Z", "content": "Thank you for your code again, I found the reason why that happened by myself. It is the input size, which should be (128,128).
", "rawMarkdown": "Thank you for your code again, I found the reason why that happened by myself. It is the input size, which should be (128,128)." } ] }, { "id": 213453, "postDate": "2017-08-14T20:32:55.333Z", "content": "@Peter Giannakopoulos I have used a part of your script to achieve my score of 0.995! Thank you very much. If you would like we can team up together.
", "rawMarkdown": "@Peter Giannakopoulos I have used a part of your script to achieve my score of 0.995! Thank you very much. If you would like we can team up together." }, { "id": 212646, "postDate": "2017-08-12T07:14:59.187Z", "content": "A 512 x 512 U_net with batch_size 12, I got 0.994 on LB. Thanks for Peter's sharing.
", "rawMarkdown": "A 512 x 512 U_net with batch_size 12, I got 0.994 on LB. Thanks for Peter's sharing. " }, { "id": 210437, "postDate": "2017-08-05T18:26:52.947Z", "content": "Thanks a lot Peter for sharing your code and sweet document for running it.
", "rawMarkdown": "Thanks a lot Peter for sharing your code and sweet document for running it. " }, { "id": 210365, "postDate": "2017-08-05T10:42:48.617Z", "content": "Hi Peter, many thanks for sharing this code. I am using it to learn keras and how to apply it to image dataset. For now I am using pre-trained weights, but I would like to experiment with training script. As a newbie in deep learning, I have following two questions:
\n\nThe reason I am asking these questions is that, if it takes longer or I can't train with on complete data on my machine, I will try to reduce the size of data for training and testing.
\n\nThanks in advance!!!
", "rawMarkdown": "Hi Peter, many thanks for sharing this code. I am using it to learn keras and how to apply it to image dataset. For now I am using pre-trained weights, but I would like to experiment with training script. As a newbie in deep learning, I have following two questions:\n\n1. Do you think I will be able to train with complete data on a 16GB machine without GPU?\n2. Also, if possible, can you provide some estimate on how long does it take for training and prediction without GPU on 16GB machine.\n\nThe reason I am asking these questions is that, if it takes longer or I can't train with on complete data on my machine, I will try to reduce the size of data for training and testing.\n\nThanks in advance!!!", "replies": [ { "id": 210366, "postDate": "2017-08-05T10:53:35.633Z", "content": "I think without GPU it is possible, but no practical.
\n\n\n\n\nGPUs are critical: The Pascal Titan X with cuDNN is 49x to 74x faster than dual Xeon E5-2630 v3 CPUs.
\n
https://github.com/jcjohnson/cnn-benchmarks
", "rawMarkdown": "I think without GPU it is possible, but no practical.\n\n> GPUs are critical: The Pascal Titan X with cuDNN is 49x to 74x faster than dual Xeon E5-2630 v3 CPUs.\n\nhttps://github.com/jcjohnson/cnn-benchmarks\n", "votes": 1 }, { "id": 210443, "postDate": "2017-08-05T19:13:59.433Z", "content": "So, you're saying I shouldn't try to train the model on full data myself?
\n\nIf that is the case, any suggestions on how should I approach the problem?
", "rawMarkdown": "So, you're saying I shouldn't try to train the model on full data myself?\n\nIf that is the case, any suggestions on how should I approach the problem?" }, { "id": 210473, "postDate": "2017-08-05T21:48:43.443Z", "content": "@AnubhavGupta, Try it on cloud services (AWS, Google Cloud Platform, etc.)
", "rawMarkdown": "@AnubhavGupta, Try it on cloud services (AWS, Google Cloud Platform, etc.)" }, { "id": 211148, "postDate": "2017-08-08T07:11:50.010Z", "content": "@ AnubhavGupta, try looking for papers with \"salient image segmentation\". The old approaches use graph cut to solve the segmentation problem + post processing step. You could get a saliency segmentation result then train a small network to do the refinement :)
", "rawMarkdown": "@ AnubhavGupta, try looking for papers with \"salient image segmentation\". The old approaches use graph cut to solve the segmentation problem + post processing step. You could get a saliency segmentation result then train a small network to do the refinement :)" } ] }, { "id": 210318, "postDate": "2017-08-05T05:23:08.333Z", "content": "What we are trying to achieve from randomshiftscalerotate function
", "rawMarkdown": "What we are trying to achieve from randomshiftscalerotate function", "replies": [ { "id": 210344, "postDate": "2017-08-05T08:19:49.247Z", "content": "This is a form of data augmentation. This way will help your model to better generalize and probably achieve better score in the test dataset.
", "rawMarkdown": "This is a form of data augmentation. This way will help your model to better generalize and probably achieve better score in the test dataset." }, { "id": 210358, "postDate": "2017-08-05T09:44:01.313Z", "content": "so you mean we are shifting , rotating and scaling an single image and creating 3 more images with this function and training it against its corresponding validation image
", "rawMarkdown": "so you mean we are shifting , rotating and scaling an single image and creating 3 more images with this function and training it against its corresponding validation image" }, { "id": 218443, "postDate": "2017-09-04T08:29:52.040Z", "content": "Still a single image.
", "rawMarkdown": "Still a single image. " } ] }, { "id": 210271, "postDate": "2017-08-04T22:42:45.713Z", "content": "Hi Peter. Good job! I was trying to run Heng pytorch script but I'm out of vram (gtx 660 2gb). Now I'm trying to run your script with keras using CPU but it takes like entity on my i7 3820. So my question is - how long does it take for you to train it ? (Are you using CPU or GPU?)
", "rawMarkdown": "Hi Peter. Good job! I was trying to run Heng pytorch script but I'm out of vram (gtx 660 2gb). Now I'm trying to run your script with keras using CPU but it takes like entity on my i7 3820. So my question is - how long does it take for you to train it ? (Are you using CPU or GPU?)", "replies": [ { "id": 210337, "postDate": "2017-08-05T07:53:03.033Z", "content": "Use GPU with a small batch size (like 4) since you only have 2gb vram. Training on CPU is very slow.
", "rawMarkdown": "Use GPU with a small batch size (like 4) since you only have 2gb vram. Training on CPU is very slow.", "votes": 1 }, { "id": 210825, "postDate": "2017-08-07T07:41:20.537Z", "content": "This might be helpful:\nhttps://github.com/fchollet/keras/issues/3556
", "rawMarkdown": "This might be helpful:\nhttps://github.com/fchollet/keras/issues/3556", "votes": 1 } ] }, { "id": 210238, "postDate": "2017-08-04T19:24:01.347Z", "content": "Why is the weights file not working:
\n\n$ python test_submit.py\nUsing TensorFlow backend.\nTraceback (most recent call last):\n File \"test_submit.py\", line 21, in \n model.load_weights(filepath='weights/best_weights.hdf5')\n File \"/usr/lib64/python2.7/site-packages/keras/engine/topology.py\", line 2572, in load_weights\n load_weights_from_hdf5_group(f, self.layers)\n File \"/usr/lib64/python2.7/site-packages/keras/engine/topology.py\", line 3021, in load_weights_from_hdf5_group\n str(len(filtered_layers)) + ' layers.')\nValueError: You are trying to load a weight file containing 45 layers into a model with 55 layers.
", "rawMarkdown": "Why is the weights file not working: \n\n$ python test_submit.py\nUsing TensorFlow backend.\nTraceback (most recent call last):\n File \"test_submit.py\", line 21, in ", "replies": [ { "id": 210239, "postDate": "2017-08-04T19:35:40.450Z", "content": "The pre-trained weights are for the 128x128 U-net but I had set the 256x256 U-net as default in \"test_submit.py\". Should be fixed now.
", "rawMarkdown": "The pre-trained weights are for the 128x128 U-net but I had set the 256x256 U-net as default in \"test_submit.py\". Should be fixed now." }, { "id": 210254, "postDate": "2017-08-04T21:18:26.950Z", "content": "Your latest results use 256x256? I plan to try larger resolutions and let you know after I submit 128x128
", "rawMarkdown": "Your latest results use 256x256? I plan to try larger resolutions and let you know after I submit 128x128" } ] }, { "id": 210085, "postDate": "2017-08-04T09:29:55.670Z", "content": "Hi Peter\nThanks for sharing
\n\nHow much RAM do you have? I noticed that you store all of prediction in variable preds\nWhat is the apprx. size?
", "rawMarkdown": "Hi Peter\nThanks for sharing\n\nHow much RAM do you have? I noticed that you store all of prediction in variable preds\nWhat is the apprx. size?\n", "replies": [ { "id": 210100, "postDate": "2017-08-04T10:31:17.367Z", "content": "For 128x128 input size, predictions need 128x128x8x100064 = 13 GB RAM. 256x256 input would need 56 GB. This obviously isn't ideal and I'm working on a solution for streaming predictions to disk.
", "rawMarkdown": "For 128x128 input size, predictions need 128x128x8x100064 = 13 GB RAM. 256x256 input would need 56 GB. This obviously isn't ideal and I'm working on a solution for streaming predictions to disk. ", "votes": 1 }, { "id": 210139, "postDate": "2017-08-04T12:36:00.347Z", "content": "I've updated the code. Test set is now split in parts, predictions done on each split.
\n\nExample with 8 splits:
\n\nPredicting on 12508 samples (split 1/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.22it/s]\nPredicting on 12508 samples (split 2/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.18it/s]\nPredicting on 12508 samples (split 3/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.44it/s]\nPredicting on 12508 samples (split 4/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:36<00:00, 129.23it/s]\nPredicting on 12508 samples (split 5/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:35<00:00, 130.54it/s]\nPredicting on 12508 samples (split 6/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:35<00:00, 131.18it/s]\nPredicting on 12508 samples (split 7/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.56it/s]\nPredicting on 12508 samples (split 8/8)\nGenerating masks...\n100%|████████████████████████████████████| 12508/12508 [01:36<00:00, 129.84it/s]\nGenerating submission file...\nYou can alter the number of splits depending on available RAM and input size.
", "rawMarkdown": "I've updated the code. Test set is now split in parts, predictions done on each split. \n\nExample with 8 splits:\n\n Predicting on 12508 samples (split 1/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.22it/s]\n Predicting on 12508 samples (split 2/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.18it/s]\n Predicting on 12508 samples (split 3/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.44it/s]\n Predicting on 12508 samples (split 4/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:36<00:00, 129.23it/s]\n Predicting on 12508 samples (split 5/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:35<00:00, 130.54it/s]\n Predicting on 12508 samples (split 6/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:35<00:00, 131.18it/s]\n Predicting on 12508 samples (split 7/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:34<00:00, 132.56it/s]\n Predicting on 12508 samples (split 8/8)\n Generating masks...\n 100%|████████████████████████████████████| 12508/12508 [01:36<00:00, 129.84it/s]\n Generating submission file...\n\nYou can alter the number of splits depending on available RAM and input size.", "votes": 6 }, { "id": 210504, "postDate": "2017-08-06T00:59:23.947Z", "content": "Hi @Peter, would results change based on the number of splits you make (16 vs 32 makes a difference?). Also would you like to team up with me?
", "rawMarkdown": "Hi @Peter, would results change based on the number of splits you make (16 vs 32 makes a difference?). Also would you like to team up with me?" }, { "id": 210513, "postDate": "2017-08-06T02:17:01.837Z", "content": "Number of splits shouldn't produce different results. There was a bug however in test_generator that could produce an error with certain test_splits / batch_size combinations. Should be fixed now.
\n\nAnd we sure can team up :)
", "rawMarkdown": "Number of splits shouldn't produce different results. There was a bug however in test_generator that could produce an error with certain test_splits / batch_size combinations. Should be fixed now.\n\nAnd we sure can team up :)" }, { "id": 210516, "postDate": "2017-08-06T02:27:29.077Z", "content": "OK because I ran with 32 splits and got 99.4 with 1024x1024 (batch of 4) when I thought it should be ~99.6 so maybe 16 splits could have given me better results.
\n\nI will see if I can figure out how to run 1200x1200 images when I have more time.
\n\nFor some reason I see the \"team\" tab but can not click on it to team with other people. How can I team with you then, I think me being leader will save my score so that why I ask.
", "rawMarkdown": "OK because I ran with 32 splits and got 99.4 with 1024x1024 (batch of 4) when I thought it should be ~99.6 so maybe 16 splits could have given me better results.\n\nI will see if I can figure out how to run 1200x1200 images when I have more time.\n\nFor some reason I see the \"team\" tab but can not click on it to team with other people. How can I team with you then, I think me being leader will save my score so that why I ask." }, { "id": 210589, "postDate": "2017-08-06T07:21:26.983Z", "content": "Maybe the small minibatch size impacts accuracy. You could try replacing Batch Normalization with Batch Renormalization (https://arxiv.org/abs/1702.03275) which appears to perform better with small minibatches. BatchRenorm layer implementation for Keras: https://github.com/titu1994/BatchRenormalization
\n\nI have the same problem with the 'team' tab not working. Some temporary issue with Kaggle probably.
", "rawMarkdown": "Maybe the small minibatch size impacts accuracy. You could try replacing Batch Normalization with Batch Renormalization (https://arxiv.org/abs/1702.03275) which appears to perform better with small minibatches. BatchRenorm layer implementation for Keras: https://github.com/titu1994/BatchRenormalization\n\nI have the same problem with the 'team' tab not working. Some temporary issue with Kaggle probably.\n", "votes": 2 }, { "id": 216489, "postDate": "2017-08-26T01:13:27.827Z", "rawMarkdown": "" }, { "id": 216508, "postDate": "2017-08-26T05:38:42.810Z", "content": "@Peter Hi， Peter. I am confused with 'split'. Does 'split' mean that I need to split all test images into several folders?
", "rawMarkdown": "@Peter Hi， Peter. I am confused with 'split'. Does 'split' mean that I need to split all test images into several folders?" }, { "id": 216514, "postDate": "2017-08-26T06:23:56.900Z", "content": "Not using this anymore, get the latest version of the code from github.
", "rawMarkdown": "Not using this anymore, get the latest version of the code from github." } ] }, { "id": 210023, "postDate": "2017-08-04T03:39:37.197Z", "content": "Thank you very much to this. \nI getting when run train.py:\n\"libpng warning: iCCP: profile 'icc': 'RGB ': RGB color space not permitted on grayscale PNG\"\nHow are I fix this?
\n\nAlso why is the loss and val_loss values negatives?
", "rawMarkdown": "Thank you very much to this. \nI getting when run train.py:\n\"libpng warning: iCCP: profile 'icc': 'RGB ': RGB color space not permitted on grayscale PNG\"\nHow are I fix this?\n\nAlso why is the loss and val_loss values negatives?\n", "replies": [ { "id": 210077, "postDate": "2017-08-04T08:43:08.007Z", "content": "Also why is the loss and val_loss values negatives?\nprobably because callback has a min mode
", "rawMarkdown": " Also why is the loss and val_loss values negatives?\n\n\nprobably because callback has a min mode", "votes": -2 }, { "id": 210528, "postDate": "2017-08-06T04:13:57.720Z", "content": "libpng complains about the profile on the image (embedded as metadata on image files) I used this command to strip the metadata of the transformed images:
\n\napt-get install exiftool\nexiftool -overwrite_original -all= *\nnice work ~~~
", "rawMarkdown": "nice work ~~~" }, { "id": 216410, "postDate": "2017-08-25T17:26:53.227Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 216456, "postDate": "2017-08-25T20:57:58.190Z", "content": "Keras handles it for you, Heng calculated all the losses manually, that is why he has to normalize it. Keras just needs the loss definition from y_true and y_preds, it handles batches by itself.
", "rawMarkdown": "Keras handles it for you, Heng calculated all the losses manually, that is why he has to normalize it. Keras just needs the loss definition from y_true and y_preds, it handles batches by itself." }, { "id": 216510, "postDate": "2017-08-26T05:44:51.073Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 215298, "postDate": "2017-08-21T02:30:09.733Z", "content": "Hi @Peter, thanks for this.
", "rawMarkdown": "Hi @Peter, thanks for this." } ], "comments": [ { "id": 210299, "author_name": "David Austin", "author_url": "", "post_date": "2017-08-05T02:17:57.363000", "content": "Just a quick data point: The 512x512 U-net (with batch size = 8) from Peter's code gave a LB score of 0.993. Training took 27 epochs @ 5.4 min per epoch on a 1080 Ti GPU.
", "votes": 7, "replies": [ { "id": 210343, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-05T08:05:24.820000", "content": "Thanks for the info. I get LB 0.992 with 256x256 U-net. Expected 512x512 U-net to be in the ~0.995 range but I myself can't run it as I get a segmentation fault after the 1st epoch and I don't know what the cause is :(
", "votes": 1, "replies": [] }, { "id": 210395, "author_name": "David Austin", "author_url": "", "post_date": "2017-08-05T12:47:38.990000", "content": "I also got 0.992 @256x256 and thought 512x512 would yield a bit more. My hunch is it has something to do with the small batch size, I have plenty of memory for a larger batch size but I'm having trouble convincing tensorflow otherwise.
", "votes": 1, "replies": [] }, { "id": 210740, "author_name": "Alexander Guth", "author_url": "", "post_date": "2017-08-06T21:29:18.207000", "content": "I was able to get a validation dice loss of 0.9919 using the 512 U-net and a batch size of 4. Did you train with the entire train set in the end, or leave some samples out for validation?
", "votes": 1, "replies": [] }, { "id": 210830, "author_name": "DataGeek", "author_url": "", "post_date": "2017-08-07T08:11:53.917000", "content": "I confirm that I got 0.995 using 512x512 using your code. Made few changes in learning rate. Batch size was 12.
", "votes": 2, "replies": [] }, { "id": 210841, "author_name": "Alexander Guth", "author_url": "", "post_date": "2017-08-07T09:29:41.850000", "content": "Mh, that's a rather good score I could not reproduce. Did you get the same result on the validation score? What kind of GPU are you using?
", "votes": 0, "replies": [] }, { "id": 210846, "author_name": "DataGeek", "author_url": "", "post_date": "2017-08-07T09:41:13.520000", "content": "LB and CV are close. I am using 1080Ti.
", "votes": 0, "replies": [] }, { "id": 218384, "author_name": "Will", "author_url": "", "post_date": "2017-09-04T02:33:32.820000", "content": "Your best net is 1024x1024 u-net with batch_size=8?
", "votes": 0, "replies": [] } ] }, { "id": 210500, "author_name": "XiaokangWang", "author_url": "", "post_date": "2017-08-06T00:29:10.327000", "content": "Hi Peter. Thanks again for your sharing. A question about the \"steps\" when making prediction on the test set. Is there a reason you added one in the steps? After adding one, the \"steps\" is not a factor of len(ids_test_split). I cited the code here. \n preds = model.predict_generator(generator=test_generator(),\n steps=(len(ids_test_split) // batch_size) + 1)
", "votes": 3, "replies": [ { "id": 210510, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-06T02:02:20.887000", "content": "The thinking was that if 'len(ids_test_split)' is not exactly divisible by 'batch_size' then 'steps' will be 1 less than needed and the 'test_generator' will leave out some samples at the end.
\n\nEg. if 'test_splits = 8' then 'len(ids_tests_split) = 100064 / 8 = 12508'. With 'batch_size = 8' we get 'steps = len(ids_test_split) // batch_size = 1563'. This would generate 1563 * batch_size = 1563 * 8 = 12504 samples instead of expected 12508. With +1 step we can get the last batch of 4 samples.
\n\nHowever if 'len(ids_test_split)' is exactly divisible by 'batch_size' then +1 is not needed and will produce an error. I think a better way to calculate steps is 'steps = np.ceil(float(len(ids_test_split)) / float(batch_size))'. This should work in all cases. Thanks for spotting the bug!
", "votes": 5, "replies": [] } ] }, { "id": 215652, "author_name": "lyakaap", "author_url": "", "post_date": "2017-08-22T14:40:59.330000", "content": "Thank you for sharing!
\n\n'dice_loss' function in model/losses.py is actually 'dice_coef'.
", "votes": 1, "replies": [] }, { "id": 210820, "author_name": "amitani", "author_url": "", "post_date": "2017-08-07T07:22:00.340000", "content": "Why do you use crossentropy instead of minimizing dice score directly in training?
", "votes": 1, "replies": [ { "id": 211527, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-09T09:41:51.357000", "content": "I observed slightly better results minimizing bce. Now I'm minimizing (bce + dice loss) which seems to perform best.
", "votes": 1, "replies": [] }, { "id": 213491, "author_name": "jackkwok", "author_url": "", "post_date": "2017-08-14T22:56:47.737000", "content": "How much better in LB score with minimizing BCE+Dice vs plain Dice, if I may ask?
", "votes": 0, "replies": [] }, { "id": 216424, "author_name": "Jurand", "author_url": "", "post_date": "2017-08-25T18:41:20.763000", "content": "I was doing testing of DICE versus (BCE+DICE)/2 using a little bit different implementation of UNet, though it may apply to Keras implementation too:
\n\nDICE\n full size validation: 0.993052\n LB score: 0.993\n(BCE+DICE) / 2\n full size validation: 0.993567\n LB score: 0.994\nThe division by 2 is to keep DICE and BCE+DICE in the same value range, otherwise learning rate needs to be adjusted to match.
", "votes": 1, "replies": [] }, { "id": 216429, "author_name": "Charles Jansen", "author_url": "", "post_date": "2017-08-25T18:59:24.370000", "content": "Thanks! Did you try with BCE only?
", "votes": 0, "replies": [] }, { "id": 216471, "author_name": "Jurand", "author_url": "", "post_date": "2017-08-25T21:51:29.833000", "content": "I did try BCE only. Did not write it down for the same model configuration, but it was less than (BCE+DICE) / 2.
", "votes": 1, "replies": [] } ] }, { "id": 224910, "author_name": "hinagi", "author_url": "", "post_date": "2017-09-27T22:44:08.130000", "content": "Thanks a lot for this well-documented code! I learned a lot.
", "votes": 2, "replies": [] }, { "id": 216494, "author_name": "amitani", "author_url": "", "post_date": "2017-08-26T01:43:24.610000", "content": "Weighing pixels near boundary is suggested in another thread (https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38125)\nI am thinking of doing this by making a custom loss function and encode closeness to the border somehow in y_true, but I thought I might ask here first if anyone has implemented it.
", "votes": 2, "replies": [ { "id": 216637, "author_name": "lyakaap", "author_url": "", "post_date": "2017-08-27T04:36:42.727000", "content": "Keras implementation is here. \nhttps://www.kaggle.com/lyakaap/weighing-boundary-pixels-loss-script-by-keras2
", "votes": 5, "replies": [] }, { "id": 216849, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-28T11:41:22.590000", "content": "I've added it to the repo. Thanks.
", "votes": 0, "replies": [] }, { "id": 216947, "author_name": "Darragh", "author_url": "", "post_date": "2017-08-28T19:00:58.050000", "content": "@lyakaap, thanks for the implementation and sharing. I tried out this loss function (weighted BCE + weighted dice) and it seemed to be quite unstable - after about 6 epochs, the validation Dice score jumped back to 0.68 after reaching 0.99 in one epoch. The (BCE + Dice) function tracked the dice score pretty nicely. I tried coding a function myself and it was also instable - did anyone else have luck with this ? Is the instability to be expected ?
", "votes": 0, "replies": [] }, { "id": 216954, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-28T19:22:32.427000", "content": "I've tried weighted dice loss (without BCE) and while CV was higher than plain dice loss (0.9947 vs 0.9940), LB score was lower (0.990 vs 0.991). That's with 128x128 u-net.
", "votes": 1, "replies": [] }, { "id": 216975, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-08-28T20:28:03.790000", "content": "Is the average mask the best for weighting the loss?
", "votes": 0, "replies": [] }, { "id": 216989, "author_name": "lyakaap", "author_url": "", "post_date": "2017-08-28T21:56:39.220000", "content": "I thought instability of weighted dice loss might be caused from intersection processing. It's not consider weighing when label=0, but actually I don't come up with the way to weighing both label=0 and 1.
", "votes": 0, "replies": [] }, { "id": 217085, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-08-29T10:37:13.320000", "content": "With input images of 512x512 and a kernel size for the weights of 31, it should be 21, else I get the following error:
\n\nValueError: Shape must be rank 4 but is rank 3 for 'AvgPool' (op: 'AvgPool') with input shapes: [?,?,?].
", "votes": 0, "replies": [] }, { "id": 217086, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-29T10:43:42.993000", "content": "Sorry, kernel size for 512 was supposed to be 21. I've fixed the typo.
", "votes": 1, "replies": [] }, { "id": 217363, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-08-30T10:14:47.490000", "content": "How about for 1280x1280 input?
", "votes": 1, "replies": [] }, { "id": 217376, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-30T11:18:35.953000", "content": "41 kernel size (same as for 1024 input) should work.
", "votes": 0, "replies": [] }, { "id": 218283, "author_name": "Will", "author_url": "", "post_date": "2017-09-03T10:42:08.020000", "content": "Thanks for your sharing. I have tried weighted bce dice loss and while CV was higher than plain dice loss , LB score was lower (same score as Peter). That's with 1024x1024 u-net.And my batch size is 8.
", "votes": 0, "replies": [] }, { "id": 218285, "author_name": "Will", "author_url": "", "post_date": "2017-09-03T10:47:51.050000", "content": "Have you tried 1280x1280? My input is 1024x1024.
", "votes": 0, "replies": [] }, { "id": 218291, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-09-03T12:45:48.783000", "content": "I'll get back to you tomorrow, I do have a 1024x1024 weighted loss version running.
", "votes": 0, "replies": [] } ] }, { "id": 213718, "author_name": "ZJU-PANDA", "author_url": "", "post_date": "2017-08-15T05:32:05.653000", "content": "华人Kaggle交流群, 请加我WX: dragen1860, 备注:kaggle, 目前已有30+华人入群，欢迎大家交流Main idea & tricks.
", "votes": -16, "replies": [] }, { "id": 213309, "author_name": "Ragnar", "author_url": "", "post_date": "2017-08-14T12:34:36.620000", "content": "Did anyone had success running this code on CPU? I just wonder how long each epoch would take...
", "votes": -1, "replies": [ { "id": 216379, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-08-25T13:48:39.647000", "content": "I was interested in this, so I tested it. 1024x1024 U-net takes 13900 seconds an epoch on average. Model converges around 25 epochs, so it takes 4-5 days to train.
\n\nOn a GPU (1080Ti) it takes 8 hours to train.
", "votes": 0, "replies": [] } ] }, { "id": 779888, "author_name": "Ahmed Saied", "author_url": "", "post_date": "2020-03-19T19:25:20.783000", "content": "Thanks a lot for sharing your work, but I experienced something I can't understand.\nIn the training process, I got pretty good training dice coef loss ~= .80, in the same time validation loss ~= .20 just after 1 epoch, which kinda weird.\nI removed all batch normalization layers from uNet architecture, and boom all things go as expected.
\n\nMy settings are:\nuNet version: 128\nbatch size = 32\noptimizer= RMSprop(.0001)\nloss: bce_dice_loss\n metrics: [dice_coef]
", "votes": 0, "replies": [] }, { "id": 308905, "author_name": "Ilham Kusuma", "author_url": "", "post_date": "2018-04-04T10:00:47.263000", "content": "Thanks a lot for the code, I'm trying to learn deep learning and your code help me to understand it very well. Nice work! love your code very much
", "votes": 0, "replies": [] }, { "id": 272516, "author_name": "lcw666", "author_url": "", "post_date": "2018-01-23T07:15:00.113000", "content": "Hi Peter. Thanks for your sharing. could you tell me how to get \"train_bounds.csv\" in your project
", "votes": 0, "replies": [] }, { "id": 224515, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2017-09-26T16:51:25.813000", "content": "This is fantastic work, thanks Peter.
\n\nOne question: why don't you use the use_multiprocessing=True and/or workers=X arguments in \"train.py\" last cell ?\nhttps://keras.io/models/sequential/#fit_generator
Is there any AWS AMI with Keras2.0? It seems that they're all Keras1.
", "votes": 0, "replies": [ { "id": 223811, "author_name": "JohnM", "author_url": "", "post_date": "2017-09-23T17:43:23.247000", "content": "Can't say that I've come across any. You might try using one of the newer AMIs in the Ireland region and upgrading Keras with pip or conda.
", "votes": 0, "replies": [] }, { "id": 224256, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2017-09-25T17:33:05.883000", "content": "Maybe this tweet by François can help ?
\n\n\"These are the steps you now have to take if you use the AWS Deep Learning AMI and you need to use Keras.
\n\nhttps://twitter.com/fchollet/status/912127671823253504
", "votes": 0, "replies": [] } ] }, { "id": 218547, "author_name": "Chandan Verma", "author_url": "", "post_date": "2017-09-04T18:58:14.290000", "content": "ran into the following error while executing the train.py\nValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None
can anyone help?
", "votes": 0, "replies": [ { "id": 218548, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-04T19:07:50.007000", "content": "Check that the file exists and is being parsed properly
", "votes": 0, "replies": [] } ] }, { "id": 218435, "author_name": "Will", "author_url": "", "post_date": "2017-09-04T07:34:52.300000", "content": "Hi, Peter, thank you for your sharing. Why do you use the function randomHueSaturationValue? It seemed that it didn't make any change to our images.
This augmentation is meant to randomize the color of a car and the overall brightness of the image:
\n\n
You can also see my post in this thread. Note that in images where the car is black-white-gray, the hue changes won't have much of an effect.
", "votes": 3, "replies": [] } ] }, { "id": 217028, "author_name": "Currylee", "author_url": "", "post_date": "2017-08-29T04:47:11.557000", "content": "Nice work.\nWhen I run this code,i did not get the h5py file in the weights folder.What should I do?
", "votes": 0, "replies": [ { "id": 217048, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-29T06:48:13.433000", "content": "Install h5py package:
\n\npip install h5py --upgrade
", "votes": 0, "replies": [] } ] }, { "id": 216286, "author_name": "friedeks", "author_url": "", "post_date": "2017-08-25T05:10:34.320000", "content": "Thank you for this great starter. I added multi GPU support for submission generation. Should scale almost linear. https://github.com/fks/Kaggle-Carvana-Image-Masking-Challenge
", "votes": 0, "replies": [ { "id": 216299, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-25T06:23:44.007000", "content": "Thanks!
", "votes": 0, "replies": [] }, { "id": 216460, "author_name": "Rohit Mehra", "author_url": "", "post_date": "2017-08-25T21:01:40.667000", "content": "How much was the time gain using multi gpu script?
", "votes": 0, "replies": [] }, { "id": 216521, "author_name": "friedeks", "author_url": "", "post_date": "2017-08-26T08:04:34.100000", "content": "For a larger model I went down from 5h per submission with a single GTX1080ti down to 3h with GTX1080ti + GTX1070
", "votes": 0, "replies": [] } ] }, { "id": 216124, "author_name": "malcolm", "author_url": "", "post_date": "2017-08-24T11:53:59.983000", "content": "How much time does this model take to training one epoch? I find it takes about 20 mins for one epoch on k80 GPU.
", "votes": 0, "replies": [ { "id": 216125, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-24T11:58:57.453000", "content": "It also takes ~20min per epoch on my GTX 1070 for the 1024x1024 Unet.
", "votes": 0, "replies": [] }, { "id": 222236, "author_name": "Sankirna", "author_url": "", "post_date": "2017-09-18T06:32:39.533000", "content": "What is your batch size malcolm?\nIt takes around 40 mins for 1 epochs on K80 for me with 256x256 images...with batch of 4.
", "votes": 0, "replies": [] } ] }, { "id": 216120, "author_name": "malcolm", "author_url": "", "post_date": "2017-08-24T11:39:05.043000", "content": "Hi Peter, thanks for your share. One question, do you use boundary weighting in your model?
", "votes": 0, "replies": [ { "id": 216122, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-24T11:47:37.863000", "content": "I haven't implemented weighted dice loss yet.
", "votes": 0, "replies": [] } ] }, { "id": 214890, "author_name": "ahmed anis", "author_url": "", "post_date": "2017-08-18T17:25:54.707000", "content": "Thank you very much \nthis starter helped me a lot
", "votes": 0, "replies": [] }, { "id": 214545, "author_name": "Ming", "author_url": "", "post_date": "2017-08-17T09:53:36.847000", "content": "Hi Peter,
\n\nThanks for your code, it's really a nice work. But I have a question, obviously binary_crossentropy loss is supposed to be used in this problem and I found you add the sigmoid activation in the last layer. While I found that binary_crossentropy function in Keras invokes sigmoid_cross_entropy_with_logits in Tensorflow (if you are using tf as backend). But as I found in the documentation and source code:
\n\nhttps://www.tensorflow.org/versions/master/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits
\n\nThe sigmoid_cross_entropy_with_logits calculates sigmoid(y_pred) again inside the function. So I don't think it makes sense that add a sigmoid in the last layer. Could you help me have a check? Thanks.
", "votes": 0, "replies": [ { "id": 214549, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-17T10:33:09.663000", "content": "I looked at Keras backend binary_crossentropy function and it looks like this:
\n\ndef binary_crossentropy(output, target, from_logits=False):\n \"\"\"Binary crossentropy between an output tensor and a target tensor.\n\n # Arguments\n output: A tensor.\n target: A tensor with the same shape as `output`.\n from_logits: Whether `output` is expected to be a logits tensor.\n By default, we consider that `output`\n encodes a probability distribution.\n\n # Returns\n A tensor.\n \"\"\"\n # Note: tf.nn.sigmoid_cross_entropy_with_logits\n # expects logits, Keras expects probabilities.\n if not from_logits:\n # transform back to logits\n epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)\n output = tf.clip_by_value(output, epsilon, 1 - epsilon)\n output = tf.log(output / (1 - output))\n\n return tf.nn.sigmoid_cross_entropy_with_logits(labels=target,\n logits=output)\nKeras expects probabilities while TF's sigmoid_cross_entropy_with_logits function expects logits, so Keras converts the probabilities passed to its binary_crossentropy to logits before passing them to sigmoid_cross_entropy_with_logits.
\n\nSo the conversion path is like: sigmoid network output -> logits (Keras backend) -> sigmoid (TF)
\n\nThis is a design choice of Keras. Also, in the training example in Keras documentation (https://keras.io/getting-started/sequential-model-guide/#training) binary_crossentropy is coupled with sigmoid activation as the network's last layer.
", "votes": 0, "replies": [] }, { "id": 214644, "author_name": "Ming", "author_url": "", "post_date": "2017-08-17T17:52:27.737000", "content": "Sorry, I still don't understand why Keras expects probabilities.
\n\n# Note: nn.softmax_cross_entropy_with_logits\n# expects logits, Keras expects probabilities.\nDoesn't it use the nn.softmax_cross_entropy_with_logits function? I don't think it is valid that couple with sigmoid.
", "votes": 0, "replies": [] }, { "id": 214653, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-17T18:20:15.607000", "content": "I mean that the Keras backend binary_crossentropy function expects probabilities by default which then converts to logits before passing them to TF's sigmoid_cross_entropy_with_logits.
\n\nIf you want you can directly call keras.backend.binary_crossentropy (the function in my previous post) with the from_logits=True option and then you can pass it the logits directly instead of probabilities. The higher level keras.losses.binary_crossentropy function (below) does just that but doesn't use the from_logits option, hence it always expects probabilities and expects the backend function it calls to do the conversion:
\n\nimport keras.backend as K\n\ndef binary_crossentropy(y_true, y_pred):\n return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)\nThank you Peter. I understand now.
", "votes": 0, "replies": [] }, { "id": 214665, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-17T19:03:37.323000", "content": "No problem. I think it'd be better if they'd exposed the from_logits option of keras.backend.binary_crossentropy through keras.losses.binary_crossentropy but seems they decided towards more abstraction.
", "votes": 1, "replies": [] } ] }, { "id": 214130, "author_name": "Will", "author_url": "", "post_date": "2017-08-16T03:00:46.083000", "content": "Hi, Peter, you have changed RMSprop as your new optimizer. Could you please tell us if you still use ReduceLROnPlateau ? As we know, the new optimizer is adaptive.
It's not technically needed but it can speed up training. RMSprop will eventually un-stuck itself from a local minima without manually reducing the learning rate (unlike SGD) but it can take longer than manually reducing LR when loss plateaus for a few epochs.
\n\nSince reducing LR is not needed and doesn't have as much of an effect as it does with SGD you could try removing it. I got 0.997 by using RMSProp + ReduceLROnPlateau but it may well be doable without reducing LR. I haven't tested.
", "votes": 0, "replies": [] } ] }, { "id": 214037, "author_name": "", "author_url": "", "post_date": "2017-08-15T19:51:19.573000", "content": "Thanks Peter so much, really nice code.
\n\nHave you seen this kernel https://www.kaggle.com/alekseit/simple-bounding-boxes ?\nIf you can add it to your code, I am interested to know how to implement the crop on the fly during training.
", "votes": 0, "replies": [ { "id": 214060, "author_name": "Petros Giannakopoulos", "author_url": "", "post_date": "2017-08-15T21:12:56.460000", "content": "Looks interesting. Seems like I've hit the wall at 0.997 so I will probably try it if it improves results. Thanks.
", "votes": 0, "replies": [] }, { "id": 215026, "author_name": "Will", "author_url": "", "post_date": "2017-08-19T08:48:08.773000", "content": "I tried the code, but it seemed to behave worse. I cropped the original image in that way, and then resized it into 512×512, but the dice_loss I got is lower.
Hi, Peter, thank you for your code very much. I ran your code just now, but there is an error saying\n```\n File \"E:/kaggle/Kaggle-Carvana-Image-Masking-Challenge-master/Kaggle-Carvana-Image-Masking-Challenge-master/train.py\", line 140, in
", "votes": 0, "replies": [ { "id": 213806, "author_name": "Will", "author_url": "", "post_date": "2017-08-15T09:19:10.777000", "content": "Thank you for your code again, I found the reason why that happened by myself. It is the input size, which should be (128,128).
", "votes": 0, "replies": [] } ] }, { "id": 213453, "author_name": "Lbragile", "author_url": "", "post_date": "2017-08-14T20:32:55.333000", "content": "@Peter Giannakopoulos I have used a part of your script to achieve my score of 0.995! Thank you very much. If you would like we can team up together.
", "votes": 0, "replies": [] }, { "id": 212646, "author_name": "timemate", "author_url": "", "post_date": "2017-08-12T07:14:59.187000", "content": "A 512 x 512 U_net with batch_size 12, I got 0.994 on LB. Thanks for Peter's sharing.
", "votes": 0, "replies": [] }, { "id": 210437, "author_name": "XiaokangWang", "author_url": "", "post_date": "2017-08-05T18:26:52.947000", "content": "Thanks a lot Peter for sharing your code and sweet document for running it.
", "votes": 0, "replies": [] }, { "id": 210365, "author_name": "AnubhavGupta", "author_url": "", "post_date": "2017-08-05T10:42:48.617000", "content": "Hi Peter, many thanks for sharing this code. I am using it to learn keras and how to apply it to image dataset. For now I am using pre-trained weights, but I would like to experiment with training script. As a newbie in deep learning, I have following two questions:
\n\nThe reason I am asking these questions is that, if it takes longer or I can't train with on complete data on my machine, I will try to reduce the size of data for training and testing.
\n\nThanks in advance!!!
", "votes": 0, "replies": [ { "id": 210366, "author_name": "Tim Joseph", "author_url": "", "post_date": "2017-08-05T10:53:35.633000", "content": "I think without GPU it is possible, but no practical.
\n\n\n\n\nGPUs are critical: The Pascal Titan X with cuDNN is 49x to 74x faster than dual Xeon E5-2630 v3 CPUs.
\n
https://github.com/jcjohnson/cnn-benchmarks
", "votes": 1, "replies": [] }, { "id": 210443, "author_name": "AnubhavGupta", "author_url": "", "post_date": "2017-08-05T19:13:59.433000", "content": "So, you're saying I shouldn't try to train the model on full data myself?
\n\nIf that is the case, any suggestions on how should I approach the problem?
", "votes": 0, "replies": [] }, { "id": 210473, "author_name": "MykolaSharhan", "author_url": "", "post_date": "2017-08-05T21:48:43.443000", "content": "@AnubhavGupta, Try it on cloud services (AWS, Google Cloud Platform, etc.)
", "votes": 0, "replies": [] }, { "id": 211148, "author_name": "DatTran", "author_url": "", "post_date": "2017-08-08T07:11:50.010000", "content": "@ AnubhavGupta, try looking for papers with \"salient image segmentation\". The old approaches use graph cut to solve the segmentation problem + post processing step. You could get a saliency segmentation result then train a small network to do the refinement :)
", "votes": 0, "replies": [] } ] }, { "id": 210318, "author_name": "mohit narang", "author_url": "", "post_date": "2017-08-05T05:23:08.333000", "content": "What we are trying to achieve from randomshiftscalerotate function
", "votes": 0, "replies": [ { "id": 210344, "author_name": "alup", "author_url": "", "post_date": "2017-08-05T08:19:49.247000", "content": "This is a form of data augmentation. This way will help your model to better generalize and probably achieve better score in the test dataset.
", "votes": 0, "replies": [] }, { "id": 210358, "author_name": "mohit narang", "author_url": "", "post_date": "2017-08-05T09:44:01.313000", "content": "so you mean we are shifting , rotating and scaling an single image and creating 3 more images with this function and training it against its corresponding validation image
", "votes": 0, "replies": [] }, { "id": 218443, "author_name": "Will", "author_url": "", "post_date": "2017-09-04T08:29:52.040000", "content": "Still a single image.
", "votes": 0, "replies": [] } ] }, { "id": 210271, "author_name": "", "author_url": "", "post_date": "2017-08-04T22:42:45.713000", "content": "", "votes": 0, "replies": [ { "id": 210337, "author_name": "", "author_url": "", "post_date": "2017-08-05T07:53:03.033000", "content": "", "votes": 1, "replies": [] }, { "id": 210825, "author_name": "", "author_url": "", "post_date": "2017-08-07T07:41:20.537000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 210238, "author_name": "", "author_url": "", "post_date": "2017-08-04T19:24:01.347000", "content": "", "votes": 0, "replies": [ { "id": 210239, "author_name": "", "author_url": "", "post_date": "2017-08-04T19:35:40.450000", "content": "", "votes": 0, "replies": [] }, { "id": 210254, "author_name": "", "author_url": "", "post_date": "2017-08-04T21:18:26.950000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 210085, "author_name": "", "author_url": "", "post_date": "2017-08-04T09:29:55.670000", "content": "", "votes": 0, "replies": [ { "id": 210100, "author_name": "", "author_url": "", "post_date": "2017-08-04T10:31:17.367000", "content": "", "votes": 1, "replies": [] }, { "id": 210139, "author_name": "", "author_url": "", "post_date": "2017-08-04T12:36:00.347000", "content": "", "votes": 6, "replies": [] }, { "id": 210504, "author_name": "", "author_url": "", "post_date": "2017-08-06T00:59:23.947000", "content": "", "votes": 0, "replies": [] }, { "id": 210513, "author_name": "", "author_url": "", "post_date": "2017-08-06T02:17:01.837000", "content": "", "votes": 0, "replies": [] }, { "id": 210516, "author_name": "", "author_url": "", "post_date": "2017-08-06T02:27:29.077000", "content": "", "votes": 0, "replies": [] }, { "id": 210589, "author_name": "", "author_url": "", "post_date": "2017-08-06T07:21:26.983000", "content": "", "votes": 2, "replies": [] }, { "id": 216489, "author_name": "", "author_url": "", "post_date": "2017-08-26T01:13:27.827000", "content": "", "votes": 0, "replies": [] }, { "id": 216508, "author_name": "", "author_url": "", "post_date": "2017-08-26T05:38:42.810000", "content": "", "votes": 0, "replies": [] }, { "id": 216514, "author_name": "", "author_url": "", "post_date": "2017-08-26T06:23:56.900000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 210023, "author_name": "", "author_url": "", "post_date": "2017-08-04T03:39:37.197000", "content": "", "votes": 0, "replies": [ { "id": 210077, "author_name": "", "author_url": "", "post_date": "2017-08-04T08:43:08.007000", "content": "", "votes": -2, "replies": [] }, { "id": 210528, "author_name": "", "author_url": "", "post_date": "2017-08-06T04:13:57.720000", "content": "", "votes": 8, "replies": [] } ] }, { "id": 209992, "author_name": "", "author_url": "", "post_date": "2017-08-04T00:45:27.760000", "content": "", "votes": 0, "replies": [] }, { "id": 216410, "author_name": "", "author_url": "", "post_date": "2017-08-25T17:26:53.227000", "content": "", "votes": 0, "replies": [ { "id": 216456, "author_name": "", "author_url": "", "post_date": "2017-08-25T20:57:58.190000", "content": "", "votes": 0, "replies": [] }, { "id": 216510, "author_name": "", "author_url": "", "post_date": "2017-08-26T05:44:51.073000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 215298, "author_name": "", "author_url": "", "post_date": "2017-08-21T02:30:09.733000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "209932": "I've been trying to replicate @Heng CherKeng's results in Keras. I've implemented 128x128, 256x256 and 512x512 U-nets. With U-net_128 I get 0.988 LB score, which seems to be in line with Heng's results.\n\nCode: https://github.com/petrosgk/Kaggle-Carvana-Image-Masking-Challenge\n\nHope it'll be of use and thanks again to @Heng CherKeng for sharing his work!\n\n----------------------------------------------------------------------------------------------------------------------\n\nUpdate 28.8.2017\n\n- Added loss with weighted boundary.\n\nUpdate 15.8.2017\n\n- Added Hue/Saturation/Value augmentation.\n- Switched to RMSprop optimizer as default.\n- Added multithreaded inference with inference and data loading done on separate threads. This reduced inference time by 40% in my tests. You can run 'test_submit_multithreaded.py' to try it.\n\nUpdate 10.8.2017: \n\n - Added 1024x1024 U-net (LB 0.996)\n - Not using predict_generator anymore due to memory constraints with large input\n\nUpdate 9.8.2017: \n\n - Now using Binary Crossentropy Dice Loss in place of Binary Crossentropy\n - Callbacks now use val_dice_loss as a metric in place of val_loss\n\nI was able to hit 0.995 with 512x512 U-net now (batch size = 4). Scores for the smaller U-nets improved as well.", "210299": "Just a quick data point: The 512x512 U-net (with batch size = 8) from Peter's code gave a LB score of 0.993. Training took 27 epochs @ 5.4 min per epoch on a 1080 Ti GPU.", "210500": "Hi Peter. Thanks again for your sharing. A question about the \"steps\" when making prediction on the test set. Is there a reason you added one in the steps? After adding one, the \"steps\" is not a factor of len(ids_test_split). I cited the code here. \n preds = model.predict_generator(generator=test_generator(),\n steps=(len(ids_test_split) // batch_size) + 1)", "215652": "Thank you for sharing!\n\n'dice_loss' function in model/losses.py is actually 'dice_coef'.", "210820": "Why do you use crossentropy instead of minimizing dice score directly in training?", "224910": "Thanks a lot for this well-documented code! I learned a lot. ", "216494": "Weighing pixels near boundary is suggested in another thread (https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/38125)\nI am thinking of doing this by making a custom loss function and encode closeness to the border somehow in y_true, but I thought I might ask here first if anyone has implemented it.", "213718": "华人Kaggle交流群, 请加我WX: dragen1860, 备注:kaggle, 目前已有30+华人入群，欢迎大家交流Main idea & tricks.", "213309": "Did anyone had success running this code on CPU? I just wonder how long each epoch would take...", "779888": "Thanks a lot for sharing your work, but I experienced something I can't understand.\nIn the training process, I got pretty good training dice coef loss ~= .80, in the same time validation loss ~= .20 just after 1 epoch, which kinda weird.\nI removed all batch normalization layers from uNet architecture, and boom all things go as expected. \n\nMy settings are:\nuNet version: 128\nbatch size = 32\noptimizer= RMSprop(.0001)\nloss: bce_dice_loss\n metrics: [dice_coef]", "308905": "Thanks a lot for the code, I'm trying to learn deep learning and your code help me to understand it very well. Nice work! love your code very much", "272516": "Hi Peter. Thanks for your sharing. could you tell me how to get \"train_bounds.csv\" in your project", "224515": "This is fantastic work, thanks Peter.\n\nOne question: why don't you use the `use_multiprocessing=True` and/or `workers=X` arguments in \"train.py\" last cell ?\nhttps://keras.io/models/sequential/#fit_generator", "222463": "Is there any AWS AMI with Keras2.0? It seems that they're all Keras1.", "218547": "ran into the following error while executing the train.py\nValueError: output of generator should be a tuple `(x, y, sample_weight)` or `(x, y)`. Found: None\n\ncan anyone help?", "218435": "Hi, Peter, thank you for your sharing. Why do you use the function ```randomHueSaturationValue```? It seemed that it didn't make any change to our images.", "217028": "Nice work.\nWhen I run this code,i did not get the h5py file in the weights folder.What should I do?", "216286": "Thank you for this great starter. I added multi GPU support for submission generation. Should scale almost linear. [https://github.com/fks/Kaggle-Carvana-Image-Masking-Challenge][1]\n\n\n [1]: https://github.com/fks/Kaggle-Carvana-Image-Masking-Challenge", "216124": "How much time does this model take to training one epoch? I find it takes about 20 mins for one epoch on k80 GPU.", "216120": "Hi Peter, thanks for your share. One question, do you use boundary weighting in your model?", "214890": "Thank you very much \nthis starter helped me a lot", "214545": "Hi Peter,\n\nThanks for your code, it's really a nice work. But I have a question, obviously **binary_crossentropy** loss is supposed to be used in this problem and I found you add the sigmoid activation in the last layer. While I found that **binary_crossentropy** function in Keras invokes **sigmoid_cross_entropy_with_logits** in Tensorflow (if you are using tf as backend). But as I found in the documentation and source code:\n\nhttps://www.tensorflow.org/versions/master/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits\n\nThe **sigmoid_cross_entropy_with_logits** calculates sigmoid(y_pred) again inside the function. So I don't think it makes sense that add a sigmoid in the last layer. Could you help me have a check? Thanks.", "214130": "Hi, Peter, you have changed ```RMSprop``` as your new optimizer. Could you please tell us if you still use ```ReduceLROnPlateau``` ? As we know, the new optimizer is adaptive.", "214037": "Thanks Peter so much, really nice code.\n\nHave you seen this kernel https://www.kaggle.com/alekseit/simple-bounding-boxes ?\nIf you can add it to your code, I am interested to know how to implement the crop on the fly during training.", "213784": "Hi, Peter, thank you for your code very much. I ran your code just now, but there is an error saying\n```\n File \"E:/kaggle/Kaggle-Carvana-Image-Masking-Challenge-master/Kaggle-Carvana-Image-Masking-Challenge-master/train.py\", line 140, in ", "213453": "@Peter Giannakopoulos I have used a part of your script to achieve my score of 0.995! Thank you very much. If you would like we can team up together.", "212646": "A 512 x 512 U_net with batch_size 12, I got 0.994 on LB. Thanks for Peter's sharing. ", "210437": "Thanks a lot Peter for sharing your code and sweet document for running it. ", "210365": "Hi Peter, many thanks for sharing this code. I am using it to learn keras and how to apply it to image dataset. For now I am using pre-trained weights, but I would like to experiment with training script. As a newbie in deep learning, I have following two questions:\n\n1. Do you think I will be able to train with complete data on a 16GB machine without GPU?\n2. Also, if possible, can you provide some estimate on how long does it take for training and prediction without GPU on 16GB machine.\n\nThe reason I am asking these questions is that, if it takes longer or I can't train with on complete data on my machine, I will try to reduce the size of data for training and testing.\n\nThanks in advance!!!", "210318": "What we are trying to achieve from randomshiftscalerotate function", "210271": "Hi Peter. Good job! I was trying to run Heng pytorch script but I'm out of vram (gtx 660 2gb). Now I'm trying to run your script with keras using CPU but it takes like entity on my i7 3820. So my question is - how long does it take for you to train it ? (Are you using CPU or GPU?)", "210238": "Why is the weights file not working: \n\n$ python test_submit.py\nUsing TensorFlow backend.\nTraceback (most recent call last):\n File \"test_submit.py\", line 21, in ", "210085": "Hi Peter\nThanks for sharing\n\nHow much RAM do you have? I noticed that you store all of prediction in variable preds\nWhat is the apprx. size?\n", "210023": "Thank you very much to this. \nI getting when run train.py:\n\"libpng warning: iCCP: profile 'icc': 'RGB ': RGB color space not permitted on grayscale PNG\"\nHow are I fix this?\n\nAlso why is the loss and val_loss values negatives?\n", "209992": "nice work ~~~", "216410": "", "215298": "Hi @Peter, thanks for this." } }