![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5225675%2Fcc4115aeb98965f26a8273ba3821a59f%2FScreen%20Shot%202022-09-02%20at%202.00.49%20PM.png?generation=1662141662504707&alt=media\")
I'm training a single-fold U-Net EfficientNetB7 backbone baseline with naive image rescaling to (340, 340)
\nThose two setups are completely the same, but they have drastically different results. I made sure to run seed_everything before the model is trained. I have data augmentation, but I think that should not cause such a difference. Any ideas why those two completely same setups can have such different results?
\nA million thanks!!
", "messages": [ { "id": "1924104", "postDate": "09/02/2022 18:03:33", "content": "
I'm training a single-fold U-Net EfficientNetB7 backbone baseline with naive image rescaling to (340, 340)
\nThose two setups are completely the same, but they have drastically different results. I made sure to run seed_everything before the model is trained. I have data augmentation, but I think that should not cause such a difference. Any ideas why those two completely same setups can have such different results?
\nA million thanks!!
", "rawMarkdown": "\n\nI'm training a single-fold U-Net EfficientNetB7 backbone baseline with naive image rescaling to (340, 340)\n\nThose two setups are completely the same, but they have drastically different results. I made sure to run seed_everything before the model is trained. I have data augmentation, but I think that should not cause such a difference. Any ideas why those two completely same setups can have such different results?\n\nA million thanks!!", "votes": null }, { "id": "1924133", "postDate": "09/02/2022 18:40:54", "content": "Just curious: Why is your val_dice is not in [0; 1] ?
I'm sorry I should've noted that: I didn't divide by the number of batches in my val set. Here there are 18 batches so the final val_dice should be the one shown divided by 18. Thank you for asking!
", "rawMarkdown": "I'm sorry I should've noted that: I didn't divide by the number of batches in my val set. Here there are 18 batches so the final val_dice should be the one shown divided by 18. Thank you for asking!", "votes": null }, { "id": "1924352", "postDate": "09/03/2022 02:02:44", "content": "algorithm debugging is a painful process
\nin summary:
\ncheck initialization before train, input and output at training
but maybe it is not important if the results is \"about the same\".
\n\"I have data augmentation, but I think that should not cause such a difference.\"
\nyes it may
Thank you so much!! It is indeed quite a challenge. Your suggestions really really helped me to learn a lot, and I will definitely try to do these as much as possible!
\nNow I highly doubt I still didn't do the \"seed everything\" operation correctly, and the UNet I'm using is also possible that the decoder part is not pretrained and the last layer is not also pretrained. I'll also check those!
", "rawMarkdown": "Thank you so much!! It is indeed quite a challenge. Your suggestions really really helped me to learn a lot, and I will definitely try to do these as much as possible!\n\nNow I highly doubt I still didn't do the \"seed everything\" operation correctly, and the UNet I'm using is also possible that the decoder part is not pretrained and the last layer is not also pretrained. I'll also check those!", "votes": null } ], "comments": [ { "id": 1924133, "author_name": "vladimirsydor", "author_url": "", "post_date": "09/02/2022 18:40:54", "content": "Just curious: Why is your val_dice is not in [0; 1] ?
I'm sorry I should've noted that: I didn't divide by the number of batches in my val set. Here there are 18 batches so the final val_dice should be the one shown divided by 18. Thank you for asking!
", "votes": null, "replies": [] } ] }, { "id": 1924352, "author_name": "hengck23", "author_url": "", "post_date": "09/03/2022 02:02:44", "content": "algorithm debugging is a painful process
\nin summary:
\ncheck initialization before train, input and output at training
but maybe it is not important if the results is \"about the same\".
\n\"I have data augmentation, but I think that should not cause such a difference.\"
\nyes it may
Thank you so much!! It is indeed quite a challenge. Your suggestions really really helped me to learn a lot, and I will definitely try to do these as much as possible!
\nNow I highly doubt I still didn't do the \"seed everything\" operation correctly, and the UNet I'm using is also possible that the decoder part is not pretrained and the last layer is not also pretrained. I'll also check those!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1924104": "\n\nI'm training a single-fold U-Net EfficientNetB7 backbone baseline with naive image rescaling to (340, 340)\n\nThose two setups are completely the same, but they have drastically different results. I made sure to run seed_everything before the model is trained. I have data augmentation, but I think that should not cause such a difference. Any ideas why those two completely same setups can have such different results?\n\nA million thanks!!", "1924133": "Just curious: Why is your `val_dice` is not in [0; 1] ?", "1924146": "I'm sorry I should've noted that: I didn't divide by the number of batches in my val set. Here there are 18 batches so the final val_dice should be the one shown divided by 18. Thank you for asking!", "1924352": "algorithm debugging is a painful process\n- compare the initialization of the network parameters (especially those newly added layers that are not loaded with pretrain weights)\n- check input are the same in different experiment\n- check output: save some values during training. e.g. per layer feature map (data, grad) of a batch ...\n- compare the values and you will know what's go wrong.\n\nin summary:\ncheck initialization before train, input and output at training\n\n---\n\nbut maybe it is not important if the results is \"about the same\".\n\n---\n\n\"I have data augmentation, but I think that should not cause such a difference.\"\nyes it may", "1924423": "Thank you so much!! It is indeed quite a challenge. Your suggestions really really helped me to learn a lot, and I will definitely try to do these as much as possible!\n\nNow I highly doubt I still didn't do the \"seed everything\" operation correctly, and the UNet I'm using is also possible that the decoder part is not pretrained and the last layer is not also pretrained. I'll also check those!" }, "source": "meta" }