Just wanted to check if anyone has had success with a Mask_RCNN model so far ?\nMy best score with Mask_RCNN is 0.055 :( ; 50 epochs or 75 epochs; all scores are in range 0.055. I get much better scores with some modifications to the Jonne kernel.
\n\nI tried various experiments with Mask_RCNN. I reviewed the config file and tried to change various parameters: https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/config.py
\n\nI tried to change the loss_weights. (made mrcnn_mask_loss ==0)
\n\nLOSS_WEIGHTS = {\n \"rpn_class_loss\": 1.,\n \"rpn_bbox_loss\": 1.,\n \"mrcnn_class_loss\": 1.,\n \"mrcnn_bbox_loss\": 1.,\n \"mrcnn_mask_loss\": 0.\n }
\n\nI played with the learning rate and set the following schedule:
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=25, \n layers='heads',\n augmentation=augmentation)
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE/10, \n epochs=50, \n layers='all',\n augmentation=augmentation)
\n\nmodel.train(dataset_train, dataset_val,\n learning_rate=config.LEARNING_RATE/100,\n epochs=75,\n layers='all',\n augmentation=augmentation)
\n\nI am thinking it might be better to give up on Mask-RCNN and move to Faster-RCNN.
", "messages": [ { "id": "388471", "postDate": "09/17/2018 04:22:02", "content": "Just wanted to check if anyone has had success with a Mask_RCNN model so far ?\nMy best score with Mask_RCNN is 0.055 :( ; 50 epochs or 75 epochs; all scores are in range 0.055. I get much better scores with some modifications to the Jonne kernel.
\n\nI tried various experiments with Mask_RCNN. I reviewed the config file and tried to change various parameters: https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/config.py
\n\nI tried to change the loss_weights. (made mrcnn_mask_loss ==0)
\n\nLOSS_WEIGHTS = {\n \"rpn_class_loss\": 1.,\n \"rpn_bbox_loss\": 1.,\n \"mrcnn_class_loss\": 1.,\n \"mrcnn_bbox_loss\": 1.,\n \"mrcnn_mask_loss\": 0.\n }
\n\nI played with the learning rate and set the following schedule:
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=25, \n layers='heads',\n augmentation=augmentation)
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE/10, \n epochs=50, \n layers='all',\n augmentation=augmentation)
\n\nmodel.train(dataset_train, dataset_val,\n learning_rate=config.LEARNING_RATE/100,\n epochs=75,\n layers='all',\n augmentation=augmentation)
\n\nI am thinking it might be better to give up on Mask-RCNN and move to Faster-RCNN.
", "rawMarkdown": "Just wanted to check if anyone has had success with a Mask_RCNN model so far ?\nMy best score with Mask_RCNN is 0.055 :( ; 50 epochs or 75 epochs; all scores are in range 0.055. I get much better scores with some modifications to the Jonne kernel. \n\nI tried various experiments with Mask_RCNN. I reviewed the config file and tried to change various parameters: https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/config.py\n\nI tried to change the loss_weights. (made mrcnn_mask_loss ==0)\n\n\nLOSS_WEIGHTS = {\n \"rpn_class_loss\": 1.,\n \"rpn_bbox_loss\": 1.,\n \"mrcnn_class_loss\": 1.,\n \"mrcnn_bbox_loss\": 1.,\n \"mrcnn_mask_loss\": 0.\n }\n\nI played with the learning rate and set the following schedule:\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=25, \n layers='heads',\n augmentation=augmentation)\n\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE/10, \n epochs=50, \n layers='all',\n augmentation=augmentation)\n\n\nmodel.train(dataset_train, dataset_val,\n learning_rate=config.LEARNING_RATE/100,\n epochs=75,\n layers='all',\n augmentation=augmentation)\n\nI am thinking it might be better to give up on Mask-RCNN and move to Faster-RCNN.", "votes": null }, { "id": "388473", "postDate": "09/17/2018 04:26:29", "content": "I started with this kernel and made modifications to it: \nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission
", "rawMarkdown": "I started with this kernel and made modifications to it: \nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission", "votes": null }, { "id": "388846", "postDate": "09/17/2018 17:26:56", "content": "I did not do much experiments but the starter kernel with input image size of 512x512 gave me 0.14. I did not used any extra augmentation, did reduce the learning rate like you are doing and trained for 75 epochs as well (1 epoch = 2400 images) (1 day training!). I think mask-rcnn has a better architecture than simple faster rcnn, and there are several papers that mention if you do multi-task learning (e.g. doing both bounding box and mask), often the results are better.
", "rawMarkdown": "I did not do much experiments but the starter kernel with input image size of 512x512 gave me 0.14. I did not used any extra augmentation, did reduce the learning rate like you are doing and trained for 75 epochs as well (1 epoch = 2400 images) (1 day training!). I think mask-rcnn has a better architecture than simple faster rcnn, and there are several papers that mention if you do multi-task learning (e.g. doing both bounding box and mask), often the results are better.", "votes": null }, { "id": "388885", "postDate": "09/17/2018 19:07:27", "content": "I tried the Mask_RCNN - starting from starter kernel. What I observed is that it gives me predictions for all test images with confidence > 0.75, which is almost impossible to be true. I combined it with a classifier upfront to weight up bbox confidence and find a decent threshold to cut. Adding a classifier doubled the metric on the validation DS, although on the LB it is much lower.
", "rawMarkdown": "I tried the Mask_RCNN - starting from starter kernel. What I observed is that it gives me predictions for all test images with confidence > 0.75, which is almost impossible to be true. I combined it with a classifier upfront to weight up bbox confidence and find a decent threshold to cut. Adding a classifier doubled the metric on the validation DS, although on the LB it is much lower.", "votes": null }, { "id": "388888", "postDate": "09/17/2018 19:12:34", "content": "Yes, but here our masks are simply fully-painted bboxes. maybe that can still help, but it's hard to tell upfront.
", "rawMarkdown": "Yes, but here our masks are simply fully-painted bboxes. maybe that can still help, but it's hard to tell upfront.", "votes": null }, { "id": "388990", "postDate": "09/18/2018 00:19:48", "content": "That is true. Here masks doesn't have any edge/boundary in the image. So Mask-RCNN may not be perfect choice but still closely serves the purpose. I guess a classification algorithm before doing any RCNN will help a lot.
", "rawMarkdown": "That is true. Here masks doesn't have any edge/boundary in the image. So Mask-RCNN may not be perfect choice but still closely serves the purpose. I guess a classification algorithm before doing any RCNN will help a lot.", "votes": null }, { "id": "389062", "postDate": "09/18/2018 04:47:56", "content": "yes I was thinking along these lines. So i tried to give lower \"importance\" to the loss component related to the mask_loss. i did it this way in the config: LOSS_WEIGHTS = { \"rpn_class_loss\": 1., \"rpn_bbox_loss\": 1., \"mrcnn_class_loss\": 1., \"mrcnn_bbox_loss\": 1., \"mrcnn_mask_loss\": 0. } ; that is set \"mrcnn_mask_loss\"==0; \n Not sure this is correct yet.
\n\nShai you (1 epoch = 2400 images); does this mean you set STEPS_PER_EPOCH = 1000
", "rawMarkdown": "yes I was thinking along these lines. So i tried to give lower \"importance\" to the loss component related to the mask_loss. i did it this way in the config: LOSS_WEIGHTS = { \"rpn_class_loss\": 1., \"rpn_bbox_loss\": 1., \"mrcnn_class_loss\": 1., \"mrcnn_bbox_loss\": 1., \"mrcnn_mask_loss\": 0. } ; that is set \"mrcnn_mask_loss\"==0; \n Not sure this is correct yet. \n\nShai you (1 epoch = 2400 images); does this mean you set STEPS_PER_EPOCH = 1000", "votes": null }, { "id": "389371", "postDate": "09/18/2018 15:51:07", "content": "It is okay to set the loss of a target = 0, simply the target will not be evaluated and will not give feedback to the training process. And here in this pneumonia problem, mask loss should not matter much. I think it's somewhere else your training is going bad. And yes my steps per epoch was 1200 and 2 images per batch.
", "rawMarkdown": "It is okay to set the loss of a target = 0, simply the target will not be evaluated and will not give feedback to the training process. And here in this pneumonia problem, mask loss should not matter much. I think it's somewhere else your training is going bad. And yes my steps per epoch was 1200 and 2 images per batch.", "votes": null }, { "id": "389595", "postDate": "09/19/2018 00:53:35", "content": "Thanks Shai! I only tried steps per epoch = 100 and 320 with 2 images per batch. let me try a higher number and see if it improves. Will write back here if I get some improvements.
", "rawMarkdown": "Thanks Shai! I only tried steps per epoch = 100 and 320 with 2 images per batch. let me try a higher number and see if it improves. Will write back here if I get some improvements.", "votes": null }, { "id": "389730", "postDate": "09/19/2018 06:17:51", "content": "Hi Shai! Sorry one more question: could you post what your mrcnn_class_loss and mrcnn_bbox_loss value was after 75 epochs. I did 40 epochs with 500 steps each; and i am at \nmrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917\nAll the numbers are below: Note: my mrcnn_mask_loss is =0 since i set the weight to zero.
\n\nEpoch 40/40\n500/500 [==============================] - 148s 295ms/step - loss: 0.9549 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3109 - mrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917 - mrcnn_mask_loss: 0.0000e+00 - val_loss: 1.0511 - val_rpn_class_loss: 0.0144 - val_rpn_bbox_loss: 0.3865 - val_mrcnn_class_loss: 0.2253 - val_mrcnn_bbox_loss: 0.4250 - val_mrcnn_mask_loss: 0.0000e+00
", "rawMarkdown": "Hi Shai! Sorry one more question: could you post what your mrcnn_class_loss and mrcnn_bbox_loss value was after 75 epochs. I did 40 epochs with 500 steps each; and i am at \nmrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917\nAll the numbers are below: Note: my mrcnn_mask_loss is =0 since i set the weight to zero. \n\nEpoch 40/40\n500/500 [==============================] - 148s 295ms/step - loss: 0.9549 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3109 - mrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917 - mrcnn_mask_loss: 0.0000e+00 - val_loss: 1.0511 - val_rpn_class_loss: 0.0144 - val_rpn_bbox_loss: 0.3865 - val_mrcnn_class_loss: 0.2253 - val_mrcnn_bbox_loss: 0.4250 - val_mrcnn_mask_loss: 0.0000e+00", "votes": null }, { "id": "389732", "postDate": "09/19/2018 06:21:49", "content": "Also if you are continuing with Mask-RCNN this page might be helpful: \nhttps://github.com/mpsampat/DSB_2018_Parameter_Selection_For_Matterport_Mask_RCNN/blob/master/comparison_of_Matterport_Mask_RCNN_parameters.md
\n\nThis was a comparison of Mask-RCNN parameters from various competitors; I made this comparison after the DSB-2018 challenge.
", "rawMarkdown": "Also if you are continuing with Mask-RCNN this page might be helpful: \nhttps://github.com/mpsampat/DSB_2018_Parameter_Selection_For_Matterport_Mask_RCNN/blob/master/comparison_of_Matterport_Mask_RCNN_parameters.md\n\n This was a comparison of Mask-RCNN parameters from various competitors; I made this comparison after the DSB-2018 challenge.", "votes": null }, { "id": "389855", "postDate": "09/19/2018 10:14:32", "content": "sometimes i find this place abit intimidating :/
", "rawMarkdown": "sometimes i find this place abit intimidating :/", "votes": null }, { "id": "389986", "postDate": "09/19/2018 14:56:32", "content": "In my case after 65 epoch: val_mrcnn_class_loss: 0.11 - val_mrcnn_bbox_loss: 0.18. Try training for longer epochs. Patience is an important parameter in deep learning!
", "rawMarkdown": "In my case after 65 epoch: val_mrcnn_class_loss: 0.11 - val_mrcnn_bbox_loss: 0.18. Try training for longer epochs. Patience is an important parameter in deep learning!", "votes": null }, { "id": "390053", "postDate": "09/19/2018 16:25:53", "content": "Dun be afraid birb. You got this
", "rawMarkdown": "Dun be afraid birb. You got this", "votes": null }, { "id": "390087", "postDate": "09/19/2018 17:25:45", "content": "Okay, Okay :-) I should change my approach from “I want patience now...” ;-)
", "rawMarkdown": "Okay, Okay :-) I should change my approach from “I want patience now...” ;-)", "votes": null }, { "id": "390100", "postDate": "09/19/2018 17:58:16", "content": "Which kind of modifications have you done?\nI've also made several changes on that kernel since then, which improved the result quite considerably. You might want to check it out now...\nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission
", "rawMarkdown": "Which kind of modifications have you done?\nI've also made several changes on that kernel since then, which improved the result quite considerably. You might want to check it out now...\nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission", "votes": null }, { "id": "390298", "postDate": "09/20/2018 02:13:48", "content": "Hi Henrique! \nThanks for updating and sharing it. I see you got 0.075 with the latest version. I just forked it and also ran it and i got 0.051. Clearly there is a lot of variability in the training result. i am trying to understand the source of the variability and the importance of the various parameters.
\n\nAs Shai mentioned, i think one of the key parameters is STEPS_PER_EPOCH; he used 1200 and that gave a decent baseline result of 0.14 for him (i used 100/320/500; but none of these give good results; all in the range 0.0055). I will try a larger value for Steps_Per_Epoch and report back if i get some improvement.
", "rawMarkdown": "Hi Henrique! \nThanks for updating and sharing it. I see you got **0.075** with the latest version. I just forked it and also ran it and i got **0.051**. Clearly there is a lot of variability in the training result. i am trying to understand the source of the variability and the importance of the various parameters. \n\nAs Shai mentioned, i think one of the key parameters is STEPS_PER_EPOCH; he used 1200 and that gave a decent baseline result of 0.14 for him (i used 100/320/500; but none of these give good results; all in the range 0.0055). I will try a larger value for Steps_Per_Epoch and report back if i get some improvement.", "votes": null }, { "id": "390382", "postDate": "09/20/2018 05:39:10", "content": "Yup I agree; dont be afraid. this is a great place to learn new stuff :) . here is a nice blog to get started: \nhttps://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46
\n\nfun fact: my friend Waleed who is author of the matterport mask-rcnn repo wrote this blog :)
", "rawMarkdown": "Yup I agree; dont be afraid. this is a great place to learn new stuff :) . here is a nice blog to get started: \nhttps://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46\n\nfun fact: my friend Waleed who is author of the matterport mask-rcnn repo wrote this blog :)", "votes": null }, { "id": "390823", "postDate": "09/20/2018 22:16:45", "content": "well, neural nets have random initialization... even fixing all random seeds I've personally never seen a really deterministic network :/ (the tip is generally to store every model you have ever trained and use the best ones, you can also give up early if it doesn't look promising on the first epochs...)
\n\nSTEPS_PER_EPOCH is basically just telling you how many images you're using, which is basically STEPS_PER_EPOCH * BATCH_SIZE * NUM_EPOCHS
\n\nthat kernel is limited to the 6hrs kaggle gives us, with 15 epochs it basically just went through all the images once or twice. You can download the weights and train it further locally ;)
", "rawMarkdown": "well, neural nets have random initialization... even fixing all random seeds I've personally never seen a really deterministic network :/ (the tip is generally to store every model you have ever trained and use the best ones, you can also give up early if it doesn't look promising on the first epochs...)\n\nSTEPS_PER_EPOCH is basically just telling you how many images you're using, which is basically STEPS_PER_EPOCH * BATCH_SIZE * NUM_EPOCHS\n\nthat kernel is limited to the 6hrs kaggle gives us, with 15 epochs it basically just went through all the images once or twice. You can download the weights and train it further locally ;)", "votes": null }, { "id": "390981", "postDate": "09/21/2018 05:35:35", "content": "Thanks Henrique. So suppose i have Steps_per_epoch == 1024; Batch_size = 10;\nI agree each epoch we go through 10240 images; \nbut across epochs there maybe some overall in images right ? suppose i have 10 epochs, is there a way to estimate how many times the network saw the same image?
\n\nMaybe here heavy data augmentation might help? so each of the 10 epochs, we can get the network to see a new batch of 10240 images ? what do you think ?
", "rawMarkdown": "Thanks Henrique. So suppose i have Steps_per_epoch == 1024; Batch_size = 10;\nI agree each epoch we go through 10240 images; \nbut across epochs there maybe some overall in images right ? suppose i have 10 epochs, is there a way to estimate how many times the network saw the same image?\n\n Maybe here heavy data augmentation might help? so each of the 10 epochs, we can get the network to see a new batch of 10240 images ? what do you think ?", "votes": null }, { "id": "391310", "postDate": "09/21/2018 15:23:30", "content": "@Shai @Mehul @Thomas Can you guide me in choosing RPN_NMS_THRESHOLD, DETECTION_MIN_CONFIDENCE and DETECTION_NMS_THRESHOLD. Given the fact the threshold choosen for final matrix is from 0.4 to 0.75 Any help regarding choosing these parameters is really appreciated. Thanks a lot
", "rawMarkdown": "Shai @Mehul @Thomas Can you guide me in choosing RPN_NMS_THRESHOLD, DETECTION_MIN_CONFIDENCE and DETECTION_NMS_THRESHOLD. Given the fact the threshold choosen for final matrix is from 0.4 to 0.75 Any help regarding choosing these parameters is really appreciated. Thanks a lot", "votes": null }, { "id": "391388", "postDate": "09/21/2018 18:15:23", "content": "What you already finished looks good. I wish there will be more talking about this.
", "rawMarkdown": "What you already finished looks good. I wish there will be more talking about this.", "votes": null }, { "id": "391406", "postDate": "09/21/2018 18:38:09", "content": "@shai on what platform and machine are you training?
", "rawMarkdown": "shai on what platform and machine are you training?", "votes": null }, { "id": "391465", "postDate": "09/21/2018 21:55:09", "content": "@Nitish I am using a single Titan X GPU. I didn't understand what you meant by platform. Its a Linux machine of 32GB RAM.
", "rawMarkdown": "Nitish I am using a single Titan X GPU. I didn't understand what you meant by platform. Its a Linux machine of 32GB RAM.", "votes": null }, { "id": "391469", "postDate": "09/21/2018 22:00:29", "content": "@Pradeep I am afraid that there is no golden rule to choose these parameters. Usually I start with the defaults and experiment with some changes after visualising the results. Try to look at the results visually (not only loss or accuracy, but also the actual images) and change the parameters what you feel will help.
", "rawMarkdown": "Pradeep I am afraid that there is no golden rule to choose these parameters. Usually I start with the defaults and experiment with some changes after visualising the results. Try to look at the results visually (not only loss or accuracy, but also the actual images) and change the parameters what you feel will help.", "votes": null }, { "id": "391624", "postDate": "09/22/2018 05:51:22", "content": "@shai so you are training your model at one go?
", "rawMarkdown": "shai so you are training your model at one go?", "votes": null }, { "id": "396521", "postDate": "10/01/2018 00:00:10", "content": "We got way better results using pre-trained weights (transfer learning) from the COCO dataset: https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155
\n\ncheck it out ans save a lot of time and energy on training!\nHappy kaggling :)
", "rawMarkdown": "We got way better results using pre-trained weights (transfer learning) from the COCO dataset: https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155\n\ncheck it out ans save a lot of time and energy on training!\nHappy kaggling :)", "votes": null }, { "id": "397787", "postDate": "10/03/2018 05:44:58", "content": "Hi @Shai, Do u use modified mask-rcnn to get 5th?
", "rawMarkdown": "Hi @Shai, Do u use modified mask-rcnn to get 5th?", "votes": null }, { "id": "398273", "postDate": "10/03/2018 19:35:09", "content": "@Nitish I train model in several stages usually. Sometimes it's not time efficient to train everything at one go. I use previously trained weights to start new training to reduce total training time.
", "rawMarkdown": "Nitish I train model in several stages usually. Sometimes it's not time efficient to train everything at one go. I use previously trained weights to start new training to reduce total training time.", "votes": null }, { "id": "398277", "postDate": "10/03/2018 19:42:35", "content": "@Bohdan my mask-rcnn model is still the basic one with some minor modifications which didn't add much value. Single mask-rcnn alone may not be sufficient to score over 0.20 in public lb. We (me and Branden) have ensemble methods and classification models to get current score.
", "rawMarkdown": "Bohdan my mask-rcnn model is still the basic one with some minor modifications which didn't add much value. Single mask-rcnn alone may not be sufficient to score over 0.20 in public lb. We (me and Branden) have ensemble methods and classification models to get current score.", "votes": null }, { "id": "398295", "postDate": "10/03/2018 20:54:52", "content": "@Shai do you have recommendations of models that my perform well besides YOLO and Mask RCNN?
", "rawMarkdown": "Shai do you have recommendations of models that my perform well besides YOLO and Mask RCNN?", "votes": null }, { "id": "398304", "postDate": "10/03/2018 21:30:15", "content": "There are a lot of algorithms you can use. You can treat this problem as semantic segmentation problem and extract bounding box from there. A good U-Net like algorithm might perform better than detection algorithms (who knows!). You can try Faster-RCNN as well. Pytorch implementation of Faster RCNN performed well for us.
", "rawMarkdown": "There are a lot of algorithms you can use. You can treat this problem as semantic segmentation problem and extract bounding box from there. A good U-Net like algorithm might perform better than detection algorithms (who knows!). You can try Faster-RCNN as well. Pytorch implementation of Faster RCNN performed well for us.", "votes": null }, { "id": "398339", "postDate": "10/04/2018 00:34:11", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null } ], "comments": [ { "id": 388473, "author_name": "mpsampat", "author_url": "", "post_date": "09/17/2018 04:26:29", "content": "I started with this kernel and made modifications to it: \nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission
", "votes": null, "replies": [ { "id": 390100, "author_name": "hmendonca", "author_url": "", "post_date": "09/19/2018 17:58:16", "content": "Which kind of modifications have you done?\nI've also made several changes on that kernel since then, which improved the result quite considerably. You might want to check it out now...\nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission
", "votes": null, "replies": [] }, { "id": 390298, "author_name": "mpsampat", "author_url": "", "post_date": "09/20/2018 02:13:48", "content": "Hi Henrique! \nThanks for updating and sharing it. I see you got 0.075 with the latest version. I just forked it and also ran it and i got 0.051. Clearly there is a lot of variability in the training result. i am trying to understand the source of the variability and the importance of the various parameters.
\n\nAs Shai mentioned, i think one of the key parameters is STEPS_PER_EPOCH; he used 1200 and that gave a decent baseline result of 0.14 for him (i used 100/320/500; but none of these give good results; all in the range 0.0055). I will try a larger value for Steps_Per_Epoch and report back if i get some improvement.
", "votes": null, "replies": [] }, { "id": 390823, "author_name": "hmendonca", "author_url": "", "post_date": "09/20/2018 22:16:45", "content": "well, neural nets have random initialization... even fixing all random seeds I've personally never seen a really deterministic network :/ (the tip is generally to store every model you have ever trained and use the best ones, you can also give up early if it doesn't look promising on the first epochs...)
\n\nSTEPS_PER_EPOCH is basically just telling you how many images you're using, which is basically STEPS_PER_EPOCH * BATCH_SIZE * NUM_EPOCHS
\n\nthat kernel is limited to the 6hrs kaggle gives us, with 15 epochs it basically just went through all the images once or twice. You can download the weights and train it further locally ;)
", "votes": null, "replies": [] }, { "id": 390981, "author_name": "mpsampat", "author_url": "", "post_date": "09/21/2018 05:35:35", "content": "Thanks Henrique. So suppose i have Steps_per_epoch == 1024; Batch_size = 10;\nI agree each epoch we go through 10240 images; \nbut across epochs there maybe some overall in images right ? suppose i have 10 epochs, is there a way to estimate how many times the network saw the same image?
\n\nMaybe here heavy data augmentation might help? so each of the 10 epochs, we can get the network to see a new batch of 10240 images ? what do you think ?
", "votes": null, "replies": [] }, { "id": 396521, "author_name": "hmendonca", "author_url": "", "post_date": "10/01/2018 00:00:10", "content": "We got way better results using pre-trained weights (transfer learning) from the COCO dataset: https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155
\n\ncheck it out ans save a lot of time and energy on training!\nHappy kaggling :)
", "votes": null, "replies": [] } ] }, { "id": 388846, "author_name": "sgalib", "author_url": "", "post_date": "09/17/2018 17:26:56", "content": "I did not do much experiments but the starter kernel with input image size of 512x512 gave me 0.14. I did not used any extra augmentation, did reduce the learning rate like you are doing and trained for 75 epochs as well (1 epoch = 2400 images) (1 day training!). I think mask-rcnn has a better architecture than simple faster rcnn, and there are several papers that mention if you do multi-task learning (e.g. doing both bounding box and mask), often the results are better.
", "votes": null, "replies": [ { "id": 388888, "author_name": "kretes", "author_url": "", "post_date": "09/17/2018 19:12:34", "content": "Yes, but here our masks are simply fully-painted bboxes. maybe that can still help, but it's hard to tell upfront.
", "votes": null, "replies": [] }, { "id": 388990, "author_name": "sgalib", "author_url": "", "post_date": "09/18/2018 00:19:48", "content": "That is true. Here masks doesn't have any edge/boundary in the image. So Mask-RCNN may not be perfect choice but still closely serves the purpose. I guess a classification algorithm before doing any RCNN will help a lot.
", "votes": null, "replies": [] }, { "id": 389062, "author_name": "mpsampat", "author_url": "", "post_date": "09/18/2018 04:47:56", "content": "yes I was thinking along these lines. So i tried to give lower \"importance\" to the loss component related to the mask_loss. i did it this way in the config: LOSS_WEIGHTS = { \"rpn_class_loss\": 1., \"rpn_bbox_loss\": 1., \"mrcnn_class_loss\": 1., \"mrcnn_bbox_loss\": 1., \"mrcnn_mask_loss\": 0. } ; that is set \"mrcnn_mask_loss\"==0; \n Not sure this is correct yet.
\n\nShai you (1 epoch = 2400 images); does this mean you set STEPS_PER_EPOCH = 1000
", "votes": null, "replies": [] }, { "id": 389371, "author_name": "sgalib", "author_url": "", "post_date": "09/18/2018 15:51:07", "content": "It is okay to set the loss of a target = 0, simply the target will not be evaluated and will not give feedback to the training process. And here in this pneumonia problem, mask loss should not matter much. I think it's somewhere else your training is going bad. And yes my steps per epoch was 1200 and 2 images per batch.
", "votes": null, "replies": [] }, { "id": 389595, "author_name": "mpsampat", "author_url": "", "post_date": "09/19/2018 00:53:35", "content": "Thanks Shai! I only tried steps per epoch = 100 and 320 with 2 images per batch. let me try a higher number and see if it improves. Will write back here if I get some improvements.
", "votes": null, "replies": [] }, { "id": 389730, "author_name": "mpsampat", "author_url": "", "post_date": "09/19/2018 06:17:51", "content": "Hi Shai! Sorry one more question: could you post what your mrcnn_class_loss and mrcnn_bbox_loss value was after 75 epochs. I did 40 epochs with 500 steps each; and i am at \nmrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917\nAll the numbers are below: Note: my mrcnn_mask_loss is =0 since i set the weight to zero.
\n\nEpoch 40/40\n500/500 [==============================] - 148s 295ms/step - loss: 0.9549 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3109 - mrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917 - mrcnn_mask_loss: 0.0000e+00 - val_loss: 1.0511 - val_rpn_class_loss: 0.0144 - val_rpn_bbox_loss: 0.3865 - val_mrcnn_class_loss: 0.2253 - val_mrcnn_bbox_loss: 0.4250 - val_mrcnn_mask_loss: 0.0000e+00
", "votes": null, "replies": [] }, { "id": 389732, "author_name": "mpsampat", "author_url": "", "post_date": "09/19/2018 06:21:49", "content": "Also if you are continuing with Mask-RCNN this page might be helpful: \nhttps://github.com/mpsampat/DSB_2018_Parameter_Selection_For_Matterport_Mask_RCNN/blob/master/comparison_of_Matterport_Mask_RCNN_parameters.md
\n\nThis was a comparison of Mask-RCNN parameters from various competitors; I made this comparison after the DSB-2018 challenge.
", "votes": null, "replies": [] }, { "id": 389986, "author_name": "sgalib", "author_url": "", "post_date": "09/19/2018 14:56:32", "content": "In my case after 65 epoch: val_mrcnn_class_loss: 0.11 - val_mrcnn_bbox_loss: 0.18. Try training for longer epochs. Patience is an important parameter in deep learning!
", "votes": null, "replies": [] }, { "id": 390087, "author_name": "mpsampat", "author_url": "", "post_date": "09/19/2018 17:25:45", "content": "Okay, Okay :-) I should change my approach from “I want patience now...” ;-)
", "votes": null, "replies": [] }, { "id": 391310, "author_name": "pradeeprathore04", "author_url": "", "post_date": "09/21/2018 15:23:30", "content": "@Shai @Mehul @Thomas Can you guide me in choosing RPN_NMS_THRESHOLD, DETECTION_MIN_CONFIDENCE and DETECTION_NMS_THRESHOLD. Given the fact the threshold choosen for final matrix is from 0.4 to 0.75 Any help regarding choosing these parameters is really appreciated. Thanks a lot
", "votes": null, "replies": [] }, { "id": 391406, "author_name": "nitishsingh41", "author_url": "", "post_date": "09/21/2018 18:38:09", "content": "@shai on what platform and machine are you training?
", "votes": null, "replies": [] }, { "id": 391465, "author_name": "sgalib", "author_url": "", "post_date": "09/21/2018 21:55:09", "content": "@Nitish I am using a single Titan X GPU. I didn't understand what you meant by platform. Its a Linux machine of 32GB RAM.
", "votes": null, "replies": [] }, { "id": 391469, "author_name": "sgalib", "author_url": "", "post_date": "09/21/2018 22:00:29", "content": "@Pradeep I am afraid that there is no golden rule to choose these parameters. Usually I start with the defaults and experiment with some changes after visualising the results. Try to look at the results visually (not only loss or accuracy, but also the actual images) and change the parameters what you feel will help.
", "votes": null, "replies": [] }, { "id": 391624, "author_name": "nitishsingh41", "author_url": "", "post_date": "09/22/2018 05:51:22", "content": "@shai so you are training your model at one go?
", "votes": null, "replies": [] }, { "id": 397787, "author_name": "bonsen", "author_url": "", "post_date": "10/03/2018 05:44:58", "content": "Hi @Shai, Do u use modified mask-rcnn to get 5th?
", "votes": null, "replies": [] }, { "id": 398273, "author_name": "sgalib", "author_url": "", "post_date": "10/03/2018 19:35:09", "content": "@Nitish I train model in several stages usually. Sometimes it's not time efficient to train everything at one go. I use previously trained weights to start new training to reduce total training time.
", "votes": null, "replies": [] }, { "id": 398277, "author_name": "sgalib", "author_url": "", "post_date": "10/03/2018 19:42:35", "content": "@Bohdan my mask-rcnn model is still the basic one with some minor modifications which didn't add much value. Single mask-rcnn alone may not be sufficient to score over 0.20 in public lb. We (me and Branden) have ensemble methods and classification models to get current score.
", "votes": null, "replies": [] }, { "id": 398295, "author_name": "dskswu", "author_url": "", "post_date": "10/03/2018 20:54:52", "content": "@Shai do you have recommendations of models that my perform well besides YOLO and Mask RCNN?
", "votes": null, "replies": [] }, { "id": 398304, "author_name": "sgalib", "author_url": "", "post_date": "10/03/2018 21:30:15", "content": "There are a lot of algorithms you can use. You can treat this problem as semantic segmentation problem and extract bounding box from there. A good U-Net like algorithm might perform better than detection algorithms (who knows!). You can try Faster-RCNN as well. Pytorch implementation of Faster RCNN performed well for us.
", "votes": null, "replies": [] }, { "id": 398339, "author_name": "bonsen", "author_url": "", "post_date": "10/04/2018 00:34:11", "content": "Thanks!
", "votes": null, "replies": [] } ] }, { "id": 388885, "author_name": "kretes", "author_url": "", "post_date": "09/17/2018 19:07:27", "content": "I tried the Mask_RCNN - starting from starter kernel. What I observed is that it gives me predictions for all test images with confidence > 0.75, which is almost impossible to be true. I combined it with a classifier upfront to weight up bbox confidence and find a decent threshold to cut. Adding a classifier doubled the metric on the validation DS, although on the LB it is much lower.
", "votes": null, "replies": [] }, { "id": 389855, "author_name": "aneccentricscientist", "author_url": "", "post_date": "09/19/2018 10:14:32", "content": "sometimes i find this place abit intimidating :/
", "votes": null, "replies": [ { "id": 390053, "author_name": "tigrex161", "author_url": "", "post_date": "09/19/2018 16:25:53", "content": "Dun be afraid birb. You got this
", "votes": null, "replies": [] }, { "id": 390382, "author_name": "mpsampat", "author_url": "", "post_date": "09/20/2018 05:39:10", "content": "Yup I agree; dont be afraid. this is a great place to learn new stuff :) . here is a nice blog to get started: \nhttps://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46
\n\nfun fact: my friend Waleed who is author of the matterport mask-rcnn repo wrote this blog :)
", "votes": null, "replies": [] } ] }, { "id": 391388, "author_name": "marvyu", "author_url": "", "post_date": "09/21/2018 18:15:23", "content": "What you already finished looks good. I wish there will be more talking about this.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "388471": "Just wanted to check if anyone has had success with a Mask_RCNN model so far ?\nMy best score with Mask_RCNN is 0.055 :( ; 50 epochs or 75 epochs; all scores are in range 0.055. I get much better scores with some modifications to the Jonne kernel. \n\nI tried various experiments with Mask_RCNN. I reviewed the config file and tried to change various parameters: https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/config.py\n\nI tried to change the loss_weights. (made mrcnn_mask_loss ==0)\n\n\nLOSS_WEIGHTS = {\n \"rpn_class_loss\": 1.,\n \"rpn_bbox_loss\": 1.,\n \"mrcnn_class_loss\": 1.,\n \"mrcnn_bbox_loss\": 1.,\n \"mrcnn_mask_loss\": 0.\n }\n\nI played with the learning rate and set the following schedule:\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=25, \n layers='heads',\n augmentation=augmentation)\n\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE/10, \n epochs=50, \n layers='all',\n augmentation=augmentation)\n\n\nmodel.train(dataset_train, dataset_val,\n learning_rate=config.LEARNING_RATE/100,\n epochs=75,\n layers='all',\n augmentation=augmentation)\n\nI am thinking it might be better to give up on Mask-RCNN and move to Faster-RCNN.", "388473": "I started with this kernel and made modifications to it: \nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission", "388846": "I did not do much experiments but the starter kernel with input image size of 512x512 gave me 0.14. I did not used any extra augmentation, did reduce the learning rate like you are doing and trained for 75 epochs as well (1 epoch = 2400 images) (1 day training!). I think mask-rcnn has a better architecture than simple faster rcnn, and there are several papers that mention if you do multi-task learning (e.g. doing both bounding box and mask), often the results are better.", "388885": "I tried the Mask_RCNN - starting from starter kernel. What I observed is that it gives me predictions for all test images with confidence > 0.75, which is almost impossible to be true. I combined it with a classifier upfront to weight up bbox confidence and find a decent threshold to cut. Adding a classifier doubled the metric on the validation DS, although on the LB it is much lower.", "388888": "Yes, but here our masks are simply fully-painted bboxes. maybe that can still help, but it's hard to tell upfront.", "388990": "That is true. Here masks doesn't have any edge/boundary in the image. So Mask-RCNN may not be perfect choice but still closely serves the purpose. I guess a classification algorithm before doing any RCNN will help a lot.", "389062": "yes I was thinking along these lines. So i tried to give lower \"importance\" to the loss component related to the mask_loss. i did it this way in the config: LOSS_WEIGHTS = { \"rpn_class_loss\": 1., \"rpn_bbox_loss\": 1., \"mrcnn_class_loss\": 1., \"mrcnn_bbox_loss\": 1., \"mrcnn_mask_loss\": 0. } ; that is set \"mrcnn_mask_loss\"==0; \n Not sure this is correct yet. \n\nShai you (1 epoch = 2400 images); does this mean you set STEPS_PER_EPOCH = 1000", "389371": "It is okay to set the loss of a target = 0, simply the target will not be evaluated and will not give feedback to the training process. And here in this pneumonia problem, mask loss should not matter much. I think it's somewhere else your training is going bad. And yes my steps per epoch was 1200 and 2 images per batch.", "389595": "Thanks Shai! I only tried steps per epoch = 100 and 320 with 2 images per batch. let me try a higher number and see if it improves. Will write back here if I get some improvements.", "389730": "Hi Shai! Sorry one more question: could you post what your mrcnn_class_loss and mrcnn_bbox_loss value was after 75 epochs. I did 40 epochs with 500 steps each; and i am at \nmrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917\nAll the numbers are below: Note: my mrcnn_mask_loss is =0 since i set the weight to zero. \n\nEpoch 40/40\n500/500 [==============================] - 148s 295ms/step - loss: 0.9549 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3109 - mrcnn_class_loss: 0.2381 - mrcnn_bbox_loss: 0.3917 - mrcnn_mask_loss: 0.0000e+00 - val_loss: 1.0511 - val_rpn_class_loss: 0.0144 - val_rpn_bbox_loss: 0.3865 - val_mrcnn_class_loss: 0.2253 - val_mrcnn_bbox_loss: 0.4250 - val_mrcnn_mask_loss: 0.0000e+00", "389732": "Also if you are continuing with Mask-RCNN this page might be helpful: \nhttps://github.com/mpsampat/DSB_2018_Parameter_Selection_For_Matterport_Mask_RCNN/blob/master/comparison_of_Matterport_Mask_RCNN_parameters.md\n\n This was a comparison of Mask-RCNN parameters from various competitors; I made this comparison after the DSB-2018 challenge.", "389855": "sometimes i find this place abit intimidating :/", "389986": "In my case after 65 epoch: val_mrcnn_class_loss: 0.11 - val_mrcnn_bbox_loss: 0.18. Try training for longer epochs. Patience is an important parameter in deep learning!", "390053": "Dun be afraid birb. You got this", "390087": "Okay, Okay :-) I should change my approach from “I want patience now...” ;-)", "390100": "Which kind of modifications have you done?\nI've also made several changes on that kernel since then, which improved the result quite considerably. You might want to check it out now...\nhttps://www.kaggle.com/hmendonca/mask-rcnn-with-submission", "390298": "Hi Henrique! \nThanks for updating and sharing it. I see you got **0.075** with the latest version. I just forked it and also ran it and i got **0.051**. Clearly there is a lot of variability in the training result. i am trying to understand the source of the variability and the importance of the various parameters. \n\nAs Shai mentioned, i think one of the key parameters is STEPS_PER_EPOCH; he used 1200 and that gave a decent baseline result of 0.14 for him (i used 100/320/500; but none of these give good results; all in the range 0.0055). I will try a larger value for Steps_Per_Epoch and report back if i get some improvement.", "390382": "Yup I agree; dont be afraid. this is a great place to learn new stuff :) . here is a nice blog to get started: \nhttps://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46\n\nfun fact: my friend Waleed who is author of the matterport mask-rcnn repo wrote this blog :)", "390823": "well, neural nets have random initialization... even fixing all random seeds I've personally never seen a really deterministic network :/ (the tip is generally to store every model you have ever trained and use the best ones, you can also give up early if it doesn't look promising on the first epochs...)\n\nSTEPS_PER_EPOCH is basically just telling you how many images you're using, which is basically STEPS_PER_EPOCH * BATCH_SIZE * NUM_EPOCHS\n\nthat kernel is limited to the 6hrs kaggle gives us, with 15 epochs it basically just went through all the images once or twice. You can download the weights and train it further locally ;)", "390981": "Thanks Henrique. So suppose i have Steps_per_epoch == 1024; Batch_size = 10;\nI agree each epoch we go through 10240 images; \nbut across epochs there maybe some overall in images right ? suppose i have 10 epochs, is there a way to estimate how many times the network saw the same image?\n\n Maybe here heavy data augmentation might help? so each of the 10 epochs, we can get the network to see a new batch of 10240 images ? what do you think ?", "391310": "Shai @Mehul @Thomas Can you guide me in choosing RPN_NMS_THRESHOLD, DETECTION_MIN_CONFIDENCE and DETECTION_NMS_THRESHOLD. Given the fact the threshold choosen for final matrix is from 0.4 to 0.75 Any help regarding choosing these parameters is really appreciated. Thanks a lot", "391388": "What you already finished looks good. I wish there will be more talking about this.", "391406": "shai on what platform and machine are you training?", "391465": "Nitish I am using a single Titan X GPU. I didn't understand what you meant by platform. Its a Linux machine of 32GB RAM.", "391469": "Pradeep I am afraid that there is no golden rule to choose these parameters. Usually I start with the defaults and experiment with some changes after visualising the results. Try to look at the results visually (not only loss or accuracy, but also the actual images) and change the parameters what you feel will help.", "391624": "shai so you are training your model at one go?", "396521": "We got way better results using pre-trained weights (transfer learning) from the COCO dataset: https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155\n\ncheck it out ans save a lot of time and energy on training!\nHappy kaggling :)", "397787": "Hi @Shai, Do u use modified mask-rcnn to get 5th?", "398273": "Nitish I train model in several stages usually. Sometimes it's not time efficient to train everything at one go. I use previously trained weights to start new training to reduce total training time.", "398277": "Bohdan my mask-rcnn model is still the basic one with some minor modifications which didn't add much value. Single mask-rcnn alone may not be sufficient to score over 0.20 in public lb. We (me and Branden) have ensemble methods and classification models to get current score.", "398295": "Shai do you have recommendations of models that my perform well besides YOLO and Mask RCNN?", "398304": "There are a lot of algorithms you can use. You can treat this problem as semantic segmentation problem and extract bounding box from there. A good U-Net like algorithm might perform better than detection algorithms (who knows!). You can try Faster-RCNN as well. Pytorch implementation of Faster RCNN performed well for us.", "398339": "Thanks!" }, "source": "meta" }