{ "id": 199468, "title": "Large difference between CV and LB (closed)", "url": "/competitions/cassava-leaf-disease-classification/discussion/199468", "author_name": "", "post_date": "2020-11-25T20:34:32.185304100Z", "votes": 1, "comment_count": 7, "views": 0, "content": "

Hello! Can anybody helps with my problem?

\n

I trained my simple model in this training notebook EfficientNetB0 training and take 0.800+ accuracy with hold-out CV. Results of the last epoch below (and you can see it in notebook):

\n
Epoch 10/10\n1204/1204 - 236s - loss: 0.4054 - accuracy: 0.8602 - val_loss: 0.4383 - val_accuracy: 0.8570\n
\n

Then I saved model in .h5 file and created public dataset with notebook output (and saved model also).

\n

For submission I created separate notebook EfficientNetB0 submission only for prediction using saved model from my freshly created dataset. But after submission… Boom, I have only 0.400+ accuracy on public leaderboard.

\n
Succeeded\n0.417\n
\n

Seems my public score divided by two(
\nWhere should I search my dumb error?

\n

Will be happy to listen any suggestions, thanks.

\n

UPDATE:
\nAs @richardepstein said, there are different order between image_id and label columns in my submission. Thanks to all.

", "messages": [ { "id": "1091177", "postDate": "11/25/2020 20:34:32", "content": "

Hello! Can anybody helps with my problem?

\n

I trained my simple model in this training notebook EfficientNetB0 training and take 0.800+ accuracy with hold-out CV. Results of the last epoch below (and you can see it in notebook):

\n
Epoch 10/10\n1204/1204 - 236s - loss: 0.4054 - accuracy: 0.8602 - val_loss: 0.4383 - val_accuracy: 0.8570\n
\n

Then I saved model in .h5 file and created public dataset with notebook output (and saved model also).

\n

For submission I created separate notebook EfficientNetB0 submission only for prediction using saved model from my freshly created dataset. But after submission… Boom, I have only 0.400+ accuracy on public leaderboard.

\n
Succeeded\n0.417\n
\n

Seems my public score divided by two(
\nWhere should I search my dumb error?

\n

Will be happy to listen any suggestions, thanks.

\n

UPDATE:
\nAs @richardepstein said, there are different order between image_id and label columns in my submission. Thanks to all.

", "rawMarkdown": "Hello! Can anybody helps with my problem?\n\nI trained my simple model in this training notebook [EfficientNetB0 training](https://www.kaggle.com/khlevnov/efficientnetb0-training) and take 0.800+ accuracy with hold-out CV. Results of the last epoch below (and you can see it in notebook):\n```\nEpoch 10/10\n1204/1204 - 236s - loss: 0.4054 - accuracy: 0.8602 - val_loss: 0.4383 - val_accuracy: 0.8570\n```\nThen I saved model in .h5 file and created [public dataset](https://www.kaggle.com/khlevnov/cassava-preptrained-model-2) with notebook output (and saved model also).\n\nFor submission I created separate notebook [EfficientNetB0 submission](https://www.kaggle.com/khlevnov/efficientnetb0-submission) only for prediction using saved model from my freshly created dataset. But after submission... Boom, I have only 0.400+ accuracy on public leaderboard.\n```\nSucceeded\n0.417\n```\nSeems my public score divided by two(\nWhere should I search my dumb error?\n\nWill be happy to listen any suggestions, thanks.\n\nUPDATE:\nAs @richardepstein said, there are different order between `image_id` and `label` columns in my submission. Thanks to all.", "votes": null }, { "id": "1091182", "postDate": "11/25/2020 20:41:33", "content": "

The data in tfrec format is a label error

", "rawMarkdown": "The data in tfrec format is a label error", "votes": null }, { "id": "1091201", "postDate": "11/25/2020 21:08:45", "content": "

But I use images from train_images😦

", "rawMarkdown": "But I use images from `train_images`😦", "votes": null }, { "id": "1091205", "postDate": "11/25/2020 21:18:39", "content": "

Do you have a local implementation of inference with model loading from a file and making predictions on hold out data?

", "rawMarkdown": "Do you have a local implementation of inference with model loading from a file and making predictions on hold out data?", "votes": null }, { "id": "1091208", "postDate": "11/25/2020 21:25:55", "content": "

Is the order of images in sample submission.csv and the order returned by the directory the same?

", "rawMarkdown": "Is the order of images in sample submission.csv and the order returned by the directory the same?", "votes": null }, { "id": "1091224", "postDate": "11/25/2020 21:50:58", "content": "

Thank you! My bad, I forgot to add shuffle arg

\n
.flow_from_dataframe(\n    shuffle=False,\n    ...\n)\n
\n

Now LB score updated to 0.852 for this baseline 😊

", "rawMarkdown": "Thank you! My bad, I forgot to add shuffle arg\n```\n.flow_from_dataframe(\n shuffle=False,\n ...\n)\n```\nNow LB score updated to 0.852 for this baseline 😊", "votes": null }, { "id": "1091234", "postDate": "11/25/2020 22:06:34", "content": "

I've also tested tfrec format data, about the same as you.

", "rawMarkdown": "I've also tested tfrec format data, about the same as you.", "votes": null }, { "id": "1091280", "postDate": "11/25/2020 22:59:51", "content": "

Thanks for advice! I hope it can helps me to prevent same issues in future.
\nI implemented simple solution and saw a difference on my local machine.

\n
score = accuracy_score(hold_out_df[['label']], submission_df[['label']])\nprint(f'Score {score}')\n
\n

Problem was when I shuffled labels in .flow_from_dataframe for prediction.

", "rawMarkdown": "Thanks for advice! I hope it can helps me to prevent same issues in future.\nI implemented simple solution and saw a difference on my local machine.\n```\nscore = accuracy_score(hold_out_df[['label']], submission_df[['label']])\nprint(f'Score {score}')\n```\nProblem was when I shuffled labels in `.flow_from_dataframe` for prediction.", "votes": null } ], "comments": [ { "id": 1091182, "author_name": "zhangeng", "author_url": "", "post_date": "11/25/2020 20:41:33", "content": "

The data in tfrec format is a label error

", "votes": null, "replies": [ { "id": 1091201, "author_name": "khlevnov", "author_url": "", "post_date": "11/25/2020 21:08:45", "content": "

But I use images from train_images😦

", "votes": null, "replies": [] }, { "id": 1091234, "author_name": "zhangeng", "author_url": "", "post_date": "11/25/2020 22:06:34", "content": "

I've also tested tfrec format data, about the same as you.

", "votes": null, "replies": [] } ] }, { "id": 1091205, "author_name": "opanichev", "author_url": "", "post_date": "11/25/2020 21:18:39", "content": "

Do you have a local implementation of inference with model loading from a file and making predictions on hold out data?

", "votes": null, "replies": [ { "id": 1091280, "author_name": "khlevnov", "author_url": "", "post_date": "11/25/2020 22:59:51", "content": "

Thanks for advice! I hope it can helps me to prevent same issues in future.
\nI implemented simple solution and saw a difference on my local machine.

\n
score = accuracy_score(hold_out_df[['label']], submission_df[['label']])\nprint(f'Score {score}')\n
\n

Problem was when I shuffled labels in .flow_from_dataframe for prediction.

", "votes": null, "replies": [] } ] }, { "id": 1091208, "author_name": "richardepstein", "author_url": "", "post_date": "11/25/2020 21:25:55", "content": "

Is the order of images in sample submission.csv and the order returned by the directory the same?

", "votes": null, "replies": [ { "id": 1091224, "author_name": "khlevnov", "author_url": "", "post_date": "11/25/2020 21:50:58", "content": "

Thank you! My bad, I forgot to add shuffle arg

\n
.flow_from_dataframe(\n    shuffle=False,\n    ...\n)\n
\n

Now LB score updated to 0.852 for this baseline 😊

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1091177": "Hello! Can anybody helps with my problem?\n\nI trained my simple model in this training notebook [EfficientNetB0 training](https://www.kaggle.com/khlevnov/efficientnetb0-training) and take 0.800+ accuracy with hold-out CV. Results of the last epoch below (and you can see it in notebook):\n```\nEpoch 10/10\n1204/1204 - 236s - loss: 0.4054 - accuracy: 0.8602 - val_loss: 0.4383 - val_accuracy: 0.8570\n```\nThen I saved model in .h5 file and created [public dataset](https://www.kaggle.com/khlevnov/cassava-preptrained-model-2) with notebook output (and saved model also).\n\nFor submission I created separate notebook [EfficientNetB0 submission](https://www.kaggle.com/khlevnov/efficientnetb0-submission) only for prediction using saved model from my freshly created dataset. But after submission... Boom, I have only 0.400+ accuracy on public leaderboard.\n```\nSucceeded\n0.417\n```\nSeems my public score divided by two(\nWhere should I search my dumb error?\n\nWill be happy to listen any suggestions, thanks.\n\nUPDATE:\nAs @richardepstein said, there are different order between `image_id` and `label` columns in my submission. Thanks to all.", "1091182": "The data in tfrec format is a label error", "1091201": "But I use images from `train_images`😦", "1091205": "Do you have a local implementation of inference with model loading from a file and making predictions on hold out data?", "1091208": "Is the order of images in sample submission.csv and the order returned by the directory the same?", "1091224": "Thank you! My bad, I forgot to add shuffle arg\n```\n.flow_from_dataframe(\n shuffle=False,\n ...\n)\n```\nNow LB score updated to 0.852 for this baseline 😊", "1091234": "I've also tested tfrec format data, about the same as you.", "1091280": "Thanks for advice! I hope it can helps me to prevent same issues in future.\nI implemented simple solution and saw a difference on my local machine.\n```\nscore = accuracy_score(hold_out_df[['label']], submission_df[['label']])\nprint(f'Score {score}')\n```\nProblem was when I shuffled labels in `.flow_from_dataframe` for prediction." }, "source": "meta" }