When I train 3D-efnetb0 using all images, all oofs are the same and val_auc is 0.500000.
\nDoes anyone have the same situation?
", "messages": [ { "id": 1472814, "postDate": "2021-08-15T06:20:32.890Z", "content": "When I train 3D-efnetb0 using all images, all oofs are the same and val_auc is 0.500000.
\nDoes anyone have the same situation?
", "rawMarkdown": "When I train 3D-efnetb0 using all images, all oofs are the same and val_auc is 0.500000.\n\nDoes anyone have the same situation?", "votes": 9 }, { "id": 1476350, "postDate": "2021-08-17T05:15:17.560Z", "content": "Me too! I have the same output of oofs.
\nI find the same issues on GitHub.
\nSame outputs after model.eval()
So, I use model.train and with no grad instead of model.eval .
\nThen I solve this problem.
\nBut I don't know this is a good solution.
Thanks for the info!
\nYour method also worked in my env (not 0.5).
\nSo maybe some error in the original code (dropout / batch norm).
Train and no grad is the recommended approach when using pytorch. Also for future models a good indication to check if your model is working is to train a small dataset (say 10) and check if your model overfits.
", "rawMarkdown": "Train and no grad is the recommended approach when using pytorch. Also for future models a good indication to check if your model is working is to train a small dataset (say 10) and check if your model overfits." }, { "id": 1482911, "postDate": "2021-08-20T10:49:37.760Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1484215, "postDate": "2021-08-21T06:13:03.113Z", "content": "\n\nBut we usually pick model in valid_epoch, where model is model.eval(), so do you set model.train() in validation epoch for your method?
\nBut we usually pick model in valid_epoch, where model is model.eval(), so do you set model.train() in validation epoch for your method?
\n
Yes.
\nI use model.train insted of model.eval when validation and prediction step
Dropout randomly drops out layers with whatever probability you give it. When predicting dropout is turned off.
\nIts turned off when you do model.eval
@lai321
\nSorry, I hadn't concern about that.
\nDropout and Batch norm layers are different behavior when train mode or validation mode.
\nSo this solution has not good.
\nBut I haven't any other idea.
\nthanks for your comment.
From what i have read about AUC an auc of 0.50 roughly indicates that your model is making random predictions i.e. it is what you would expect if you evaluate properties randomly
", "rawMarkdown": "From what i have read about AUC an auc of 0.50 roughly indicates that your model is making random predictions i.e. it is what you would expect if you evaluate properties randomly", "votes": 1, "replies": [ { "id": 1476208, "postDate": "2021-08-17T03:20:38.267Z", "content": "That's right, something is strange.
\n2D-efnet works well although it is still tough for a model to be trained.
What I did is training with this amazing dataset (4x128x128x128).
\nhttps://www.kaggle.com/ren4yu/normalized-voxels-align-planes-and-crop
\nSurely the data itself, dataset and dataloader work well.
\nAs you indicate, we need pretrained weight like imagenet for better score.
\nI don't know why public notebooks have some val_score > 0.5.
I see.
\nBTW, there is memory_efficient_swish at the last layer.
\nDid you change it to something or use as it is?
Would pretrained weights really help a lot? See this discussion: https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/262103#1453708. It seems like the score of the best public notebook is almost as good as random. How was your loss while training?
", "rawMarkdown": "Would pretrained weights really help a lot? See this discussion: https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/262103#1453708. It seems like the score of the best public notebook is almost as good as random. How was your loss while training?" }, { "id": 1473324, "postDate": "2021-08-15T14:06:27.410Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1476000, "postDate": "2021-08-16T23:54:44.657Z", "content": "The last layer is not a swish activation. It's a linear layer (model._fc) outing logits. I know it may look like that in a summary, but the summary is wrong.
", "rawMarkdown": "The last layer is not a swish activation. It's a linear layer (model._fc) outing logits. I know it may look like that in a summary, but the summary is wrong." } ] } ], "comments": [ { "id": 1476350, "author_name": "kazu02", "author_url": "", "post_date": "2021-08-17T05:15:17.560000", "content": "Me too! I have the same output of oofs.
\nI find the same issues on GitHub.
\nSame outputs after model.eval()
So, I use model.train and with no grad instead of model.eval .
\nThen I solve this problem.
\nBut I don't know this is a good solution.
Thanks for the info!
\nYour method also worked in my env (not 0.5).
\nSo maybe some error in the original code (dropout / batch norm).
Train and no grad is the recommended approach when using pytorch. Also for future models a good indication to check if your model is working is to train a small dataset (say 10) and check if your model overfits.
", "votes": 0, "replies": [] }, { "id": 1482911, "author_name": "", "author_url": "", "post_date": "2021-08-20T10:49:37.760000", "content": "", "votes": 0, "replies": [ { "id": 1484215, "author_name": "kazu02", "author_url": "", "post_date": "2021-08-21T06:13:03.113000", "content": "\n\nBut we usually pick model in valid_epoch, where model is model.eval(), so do you set model.train() in validation epoch for your method?
\nBut we usually pick model in valid_epoch, where model is model.eval(), so do you set model.train() in validation epoch for your method?
\n
Yes.
\nI use model.train insted of model.eval when validation and prediction step
Dropout randomly drops out layers with whatever probability you give it. When predicting dropout is turned off.
\nIts turned off when you do model.eval
@lai321
\nSorry, I hadn't concern about that.
\nDropout and Batch norm layers are different behavior when train mode or validation mode.
\nSo this solution has not good.
\nBut I haven't any other idea.
\nthanks for your comment.
From what i have read about AUC an auc of 0.50 roughly indicates that your model is making random predictions i.e. it is what you would expect if you evaluate properties randomly
", "votes": 1, "replies": [ { "id": 1476208, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-08-17T03:20:38.267000", "content": "That's right, something is strange.
\n2D-efnet works well although it is still tough for a model to be trained.
What I did is training with this amazing dataset (4x128x128x128).
\nhttps://www.kaggle.com/ren4yu/normalized-voxels-align-planes-and-crop
\nSurely the data itself, dataset and dataloader work well.
\nAs you indicate, we need pretrained weight like imagenet for better score.
\nI don't know why public notebooks have some val_score > 0.5.
I see.
\nBTW, there is memory_efficient_swish at the last layer.
\nDid you change it to something or use as it is?
Would pretrained weights really help a lot? See this discussion: https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/262103#1453708. It seems like the score of the best public notebook is almost as good as random. How was your loss while training?
", "votes": 0, "replies": [] }, { "id": 1473324, "author_name": "", "author_url": "", "post_date": "2021-08-15T14:06:27.410000", "content": "", "votes": 1, "replies": [] }, { "id": 1476000, "author_name": "Pandagg", "author_url": "", "post_date": "2021-08-16T23:54:44.657000", "content": "The last layer is not a swish activation. It's a linear layer (model._fc) outing logits. I know it may look like that in a summary, but the summary is wrong.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1472814": "When I train 3D-efnetb0 using all images, all oofs are the same and val_auc is 0.500000.\n\nDoes anyone have the same situation?", "1476350": "Me too! I have the same output of oofs.\nI find the same issues on GitHub.\n[Same outputs after model.eval() ](https://github.com/shijianjian/EfficientNet-PyTorch-3D/issues/6)\n\nSo, I use model.train and with no grad instead of model.eval .\nThen I solve this problem.\nBut I don't know this is a good solution.\n", "1476040": "From what i have read about AUC an auc of 0.50 roughly indicates that your model is making random predictions i.e. it is what you would expect if you evaluate properties randomly", "1473109": "" } }