{ "id": 316100, "title": "valid acc >> test acc", "url": "/competitions/ultra-mnist/discussion/316100", "author_name": "", "post_date": "2022-03-31T08:51:00.313296400Z", "votes": 3, "comment_count": 6, "views": 0, "content": "

Hi all,

\n

No matter what I get on my validation accuracy, I always get the same 0.03x accuracy on my submissions. I guess I'm not submitting my preds correctly, but I cannot find the source of the error so I though I could ask here to see if anyone can shed a light.

\n

Context:

\n\n

I'm basically predicting this way:

\n
pred, _ = model.get_preds(dl = test_dl)\npred = torch.argmax(pred, axis=1)\ndf_submit = pd.read_csv(input_path/'sample_submission.csv')\ndf_submit[\"digit_sum\"] = pred\n
\n

I understand that torch.argmax is correct as the output of model.dls.vocab is ordered:
\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]

\n

Any idea that might help really appreciated, thnks!

", "messages": [ { "id": "1740763", "postDate": "03/31/2022 08:51:00", "content": "

Hi all,

\n

No matter what I get on my validation accuracy, I always get the same 0.03x accuracy on my submissions. I guess I'm not submitting my preds correctly, but I cannot find the source of the error so I though I could ask here to see if anyone can shed a light.

\n

Context:

\n\n

I'm basically predicting this way:

\n
pred, _ = model.get_preds(dl = test_dl)\npred = torch.argmax(pred, axis=1)\ndf_submit = pd.read_csv(input_path/'sample_submission.csv')\ndf_submit[\"digit_sum\"] = pred\n
\n

I understand that torch.argmax is correct as the output of model.dls.vocab is ordered:
\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]

\n

Any idea that might help really appreciated, thnks!

", "rawMarkdown": "Hi all,\n\nNo matter what I get on my validation accuracy, I always get the same 0.03x accuracy on my submissions. I guess I'm not submitting my preds correctly, but I cannot find the source of the error so I though I could ask here to see if anyone can shed a light.\n\nContext:\n* Using a grayscale size of 512 to fine tune a resnet50 without normalization using fastai.\n* Valid accuracy is ~0.27, however my submission accuracy is 0.03328.\n* Trained for 27 ephochs the head and 11 epochs for the whole architecture.\n* Don't think I'm overfitting as I'm using early stopping, and valid loss and accuracy seem to be generally decrasing until stopping.\n\nI'm basically predicting this way:\n```python\npred, _ = model.get_preds(dl = test_dl)\npred = torch.argmax(pred, axis=1)\ndf_submit = pd.read_csv(input_path/'sample_submission.csv')\ndf_submit[\"digit_sum\"] = pred\n```\n\nI understand that torch.argmax is correct as the output of `model.dls.vocab` is ordered:\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]\n\nAny idea that might help really appreciated, thnks!", "votes": null }, { "id": "1740957", "postDate": "03/31/2022 12:02:16", "content": "

I am experiencing the same, in fact when my valid acc. improved the test acc dropped.

\n

BTW score of 0.035 is around the random guessing mark see here. so check if your predictions are something similar

", "rawMarkdown": "I am experiencing the same, in fact when my valid acc. improved the test acc dropped.\n\nBTW score of 0.035 is around the random guessing mark see [here](https://www.kaggle.com/code/carlmcbrideellis/ultramnist-baseline-all-class-20). so check if your predictions are something similar", "votes": null }, { "id": "1741154", "postDate": "03/31/2022 14:57:12", "content": "

that's correct, as you said my predicitons are no different from random guessing :(

", "rawMarkdown": "that's correct, as you said my predicitons are no different from random guessing :(", "votes": null }, { "id": "1741904", "postDate": "04/01/2022 09:09:43", "content": "

Same is happening with me, please let me know if you find out the reason for it.

", "rawMarkdown": "Same is happening with me, please let me know if you find out the reason for it.", "votes": null }, { "id": "1742554", "postDate": "04/02/2022 03:37:44", "content": "

i am pretty sure you are overfitting the train data, i trained resnet18 with fine tune for 15 epochs, 0.001 lr and with Normalize Transform for images on the 512px data and the valid loss stopped going down after 5th epoch while the train loss kept going down till the last epoch. Final train loss was: 0.071 and valid loss was 4.28 and valid acc was 13.75%. Considering you are using resnet50, i think it will also overfit

", "rawMarkdown": "i am pretty sure you are overfitting the train data, i trained resnet18 with fine tune for 15 epochs, 0.001 lr and with Normalize Transform for images on the 512px data and the valid loss stopped going down after 5th epoch while the train loss kept going down till the last epoch. Final train loss was: 0.071 and valid loss was 4.28 and valid acc was 13.75%. Considering you are using resnet50, i think it will also overfit", "votes": null }, { "id": "1742998", "postDate": "04/02/2022 13:31:23", "content": "

I also found out that in my own model also i was overfitting, and my valid set was leaking into my test set due the way i was using image transforms, so my valid accuracy was improving a lot (upto 90%+). so please check if that is in your case as well

", "rawMarkdown": "I also found out that in my own model also i was overfitting, and my valid set was leaking into my test set due the way i was using image transforms, so my valid accuracy was improving a lot (upto 90%+). so please check if that is in your case as well", "votes": null }, { "id": "1744130", "postDate": "04/03/2022 16:17:57", "content": "

Overfitting could be the reason, but I'm using early stopping to avoid it. I'll try to adjust the stop condition to do less epochs, thanks for your insights @abhishekdasani.

", "rawMarkdown": "Overfitting could be the reason, but I'm using early stopping to avoid it. I'll try to adjust the stop condition to do less epochs, thanks for your insights @abhishekdasani.", "votes": null } ], "comments": [ { "id": 1740957, "author_name": "abhishekdasani", "author_url": "", "post_date": "03/31/2022 12:02:16", "content": "

I am experiencing the same, in fact when my valid acc. improved the test acc dropped.

\n

BTW score of 0.035 is around the random guessing mark see here. so check if your predictions are something similar

", "votes": null, "replies": [ { "id": 1741154, "author_name": "javiercarnero", "author_url": "", "post_date": "03/31/2022 14:57:12", "content": "

that's correct, as you said my predicitons are no different from random guessing :(

", "votes": null, "replies": [] }, { "id": 1742554, "author_name": "abhishekdasani", "author_url": "", "post_date": "04/02/2022 03:37:44", "content": "

i am pretty sure you are overfitting the train data, i trained resnet18 with fine tune for 15 epochs, 0.001 lr and with Normalize Transform for images on the 512px data and the valid loss stopped going down after 5th epoch while the train loss kept going down till the last epoch. Final train loss was: 0.071 and valid loss was 4.28 and valid acc was 13.75%. Considering you are using resnet50, i think it will also overfit

", "votes": null, "replies": [] }, { "id": 1742998, "author_name": "abhishekdasani", "author_url": "", "post_date": "04/02/2022 13:31:23", "content": "

I also found out that in my own model also i was overfitting, and my valid set was leaking into my test set due the way i was using image transforms, so my valid accuracy was improving a lot (upto 90%+). so please check if that is in your case as well

", "votes": null, "replies": [] }, { "id": 1744130, "author_name": "javiercarnero", "author_url": "", "post_date": "04/03/2022 16:17:57", "content": "

Overfitting could be the reason, but I'm using early stopping to avoid it. I'll try to adjust the stop condition to do less epochs, thanks for your insights @abhishekdasani.

", "votes": null, "replies": [] } ] }, { "id": 1741904, "author_name": "sraj07", "author_url": "", "post_date": "04/01/2022 09:09:43", "content": "

Same is happening with me, please let me know if you find out the reason for it.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1740763": "Hi all,\n\nNo matter what I get on my validation accuracy, I always get the same 0.03x accuracy on my submissions. I guess I'm not submitting my preds correctly, but I cannot find the source of the error so I though I could ask here to see if anyone can shed a light.\n\nContext:\n* Using a grayscale size of 512 to fine tune a resnet50 without normalization using fastai.\n* Valid accuracy is ~0.27, however my submission accuracy is 0.03328.\n* Trained for 27 ephochs the head and 11 epochs for the whole architecture.\n* Don't think I'm overfitting as I'm using early stopping, and valid loss and accuracy seem to be generally decrasing until stopping.\n\nI'm basically predicting this way:\n```python\npred, _ = model.get_preds(dl = test_dl)\npred = torch.argmax(pred, axis=1)\ndf_submit = pd.read_csv(input_path/'sample_submission.csv')\ndf_submit[\"digit_sum\"] = pred\n```\n\nI understand that torch.argmax is correct as the output of `model.dls.vocab` is ordered:\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]\n\nAny idea that might help really appreciated, thnks!", "1740957": "I am experiencing the same, in fact when my valid acc. improved the test acc dropped.\n\nBTW score of 0.035 is around the random guessing mark see [here](https://www.kaggle.com/code/carlmcbrideellis/ultramnist-baseline-all-class-20). so check if your predictions are something similar", "1741154": "that's correct, as you said my predicitons are no different from random guessing :(", "1741904": "Same is happening with me, please let me know if you find out the reason for it.", "1742554": "i am pretty sure you are overfitting the train data, i trained resnet18 with fine tune for 15 epochs, 0.001 lr and with Normalize Transform for images on the 512px data and the valid loss stopped going down after 5th epoch while the train loss kept going down till the last epoch. Final train loss was: 0.071 and valid loss was 4.28 and valid acc was 13.75%. Considering you are using resnet50, i think it will also overfit", "1742998": "I also found out that in my own model also i was overfitting, and my valid set was leaking into my test set due the way i was using image transforms, so my valid accuracy was improving a lot (upto 90%+). so please check if that is in your case as well", "1744130": "Overfitting could be the reason, but I'm using early stopping to avoid it. I'll try to adjust the stop condition to do less epochs, thanks for your insights @abhishekdasani." }, "source": "meta" }