{ "id": 311582, "title": "single model submission train/val/LB question", "url": "/competitions/happy-whale-and-dolphin/discussion/311582", "author_name": "", "post_date": "2022-03-07T16:41:42.612661Z", "votes": 4, "comment_count": 7, "views": 0, "content": "

Due TPU time, I tuned parameters and got a new model weight with better train/val loss/acc.

\n

however, the result submission is worse than before.

\n

what could be the problem? overfitting?

", "messages": [ { "id": "1715113", "postDate": "03/07/2022 16:41:42", "content": "

Due TPU time, I tuned parameters and got a new model weight with better train/val loss/acc.

\n

however, the result submission is worse than before.

\n

what could be the problem? overfitting?

", "rawMarkdown": "Due TPU time, I tuned parameters and got a new model weight with better train/val loss/acc.\n\nhowever, the result submission is worse than before. \n\nwhat could be the problem? overfitting?", "votes": null }, { "id": "1715134", "postDate": "03/07/2022 16:54:28", "content": "

@dragonzhang You may not have actually/effectively improved your model using TPU only acclerated it.

", "rawMarkdown": "dragonzhang You may not have actually/effectively improved your model using TPU only acclerated it.", "votes": null }, { "id": "1715170", "postDate": "03/07/2022 17:26:38", "content": "

both model trained on TPU. Do you think that the val/acc with several percents means nothing?

", "rawMarkdown": "both model trained on TPU. Do you think that the val/acc with several percents means nothing?", "votes": null }, { "id": "1715200", "postDate": "03/07/2022 17:51:06", "content": "

another dataset has improvement compared with previous LB.

\n

I need to check the two notebook codes for difference.

", "rawMarkdown": "another dataset has improvement compared with previous LB.\n\nI need to check the two notebook codes for difference.", "votes": null }, { "id": "1715362", "postDate": "03/07/2022 23:52:57", "content": "

@dragonzhang if you are using xentropy, visualize it and you will see a curve, in that curve, a few percent can contain in a range of error and a percent can also mean incredible improvement.
\nEx-
\nprediction going from 0.1 - 0.2, large increase or decrease in loss
\nprediction going from 0.95 - 0.96, small increase or decrease in loss

\n

Its likely just a margin-of-error problem, lb is currently about 5k images, and your validation is 10k images (if its a 5-fold 20% split), my advice would be to trust CV if its improved (a few percent at your score is small difference in loss).

", "rawMarkdown": "dragonzhang if you are using xentropy, visualize it and you will see a curve, in that curve, a few percent can contain in a range of error and a percent can also mean incredible improvement.\nEx- \nprediction going from 0.1 - 0.2, large increase or decrease in loss\nprediction going from 0.95 - 0.96, small increase or decrease in loss\n\nIts likely just a margin-of-error problem, lb is currently about 5k images, and your validation is 10k images (if its a 5-fold 20% split), my advice would be to trust CV if its improved (a few percent at your score is small difference in loss).", "votes": null }, { "id": "1715507", "postDate": "03/08/2022 04:34:55", "content": "

thanks for your explanation.

", "rawMarkdown": "thanks for your explanation.", "votes": null }, { "id": "1715905", "postDate": "03/08/2022 13:44:24", "content": "

Trust your CV.

", "rawMarkdown": "Trust your CV.", "votes": null }, { "id": "1716415", "postDate": "03/09/2022 02:02:19", "content": "

I saw Accuracy not always correlated with map@5. May be we produce prediction by KNN search not output logit

", "rawMarkdown": "I saw Accuracy not always correlated with map@5. May be we produce prediction by KNN search not output logit", "votes": null } ], "comments": [ { "id": 1715134, "author_name": "gianetan", "author_url": "", "post_date": "03/07/2022 16:54:28", "content": "

@dragonzhang You may not have actually/effectively improved your model using TPU only acclerated it.

", "votes": null, "replies": [ { "id": 1715170, "author_name": "dragonzhang", "author_url": "", "post_date": "03/07/2022 17:26:38", "content": "

both model trained on TPU. Do you think that the val/acc with several percents means nothing?

", "votes": null, "replies": [] }, { "id": 1715362, "author_name": "harshitsheoran", "author_url": "", "post_date": "03/07/2022 23:52:57", "content": "

@dragonzhang if you are using xentropy, visualize it and you will see a curve, in that curve, a few percent can contain in a range of error and a percent can also mean incredible improvement.
\nEx-
\nprediction going from 0.1 - 0.2, large increase or decrease in loss
\nprediction going from 0.95 - 0.96, small increase or decrease in loss

\n

Its likely just a margin-of-error problem, lb is currently about 5k images, and your validation is 10k images (if its a 5-fold 20% split), my advice would be to trust CV if its improved (a few percent at your score is small difference in loss).

", "votes": null, "replies": [] }, { "id": 1715507, "author_name": "dragonzhang", "author_url": "", "post_date": "03/08/2022 04:34:55", "content": "

thanks for your explanation.

", "votes": null, "replies": [] } ] }, { "id": 1715200, "author_name": "dragonzhang", "author_url": "", "post_date": "03/07/2022 17:51:06", "content": "

another dataset has improvement compared with previous LB.

\n

I need to check the two notebook codes for difference.

", "votes": null, "replies": [] }, { "id": 1715905, "author_name": "returnofsputnik", "author_url": "", "post_date": "03/08/2022 13:44:24", "content": "

Trust your CV.

", "votes": null, "replies": [] }, { "id": 1716415, "author_name": "ptran1203", "author_url": "", "post_date": "03/09/2022 02:02:19", "content": "

I saw Accuracy not always correlated with map@5. May be we produce prediction by KNN search not output logit

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1715113": "Due TPU time, I tuned parameters and got a new model weight with better train/val loss/acc.\n\nhowever, the result submission is worse than before. \n\nwhat could be the problem? overfitting?", "1715134": "dragonzhang You may not have actually/effectively improved your model using TPU only acclerated it.", "1715170": "both model trained on TPU. Do you think that the val/acc with several percents means nothing?", "1715200": "another dataset has improvement compared with previous LB.\n\nI need to check the two notebook codes for difference.", "1715362": "dragonzhang if you are using xentropy, visualize it and you will see a curve, in that curve, a few percent can contain in a range of error and a percent can also mean incredible improvement.\nEx- \nprediction going from 0.1 - 0.2, large increase or decrease in loss\nprediction going from 0.95 - 0.96, small increase or decrease in loss\n\nIts likely just a margin-of-error problem, lb is currently about 5k images, and your validation is 10k images (if its a 5-fold 20% split), my advice would be to trust CV if its improved (a few percent at your score is small difference in loss).", "1715507": "thanks for your explanation.", "1715905": "Trust your CV.", "1716415": "I saw Accuracy not always correlated with map@5. May be we produce prediction by KNN search not output logit" }, "source": "meta" }