{ "id": 161439, "title": "Overfitting ?", "url": "/competitions/alaska2-image-steganalysis/discussion/161439", "author_name": "Johnny Lee", "post_date": "2020-06-24T22:21:25.809000", "votes": 8, "comment_count": 16, "views": 0, "content": "

It's too long to train all folds. So, how to check if it is overfitting to the public LB ? Is it impossible?

", "messages": [ { "id": 900695, "postDate": "2020-06-25T02:43:40.413Z", "content": "

After splitting k-fold, undersampling your training set (e.g: keep only 60k images), train your models, and monitor weighted AUC on full validation data. When you find the best parameters, rerun your model with full data.

\n\n

In my experiments, the local CV and public LB are quite consistent (0.888 vs. 0.890 with undersampling, 0.926 vs.0.928 with full data)

", "rawMarkdown": "After splitting k-fold, undersampling your training set (e.g: keep only 60k images), train your models, and monitor weighted AUC on full validation data. When you find the best parameters, rerun your model with full data.\n\nIn my experiments, the local CV and public LB are quite consistent (0.888 vs. 0.890 with undersampling, 0.926 vs.0.928 with full data)", "votes": 16, "replies": [ { "id": 900785, "postDate": "2020-06-25T04:10:38.300Z", "content": "

Nice idea! Thank you. This can save us a lot of time.

", "rawMarkdown": "Nice idea! Thank you. This can save us a lot of time.", "votes": 2 }, { "id": 901454, "postDate": "2020-06-25T13:48:00.013Z", "content": "

What is your local CV score score for your .941 LB score?

", "rawMarkdown": "What is your local CV score score for your .941 LB score? " }, { "id": 901466, "postDate": "2020-06-25T13:58:22.273Z", "content": "

The local CV is 0.944 and LB is 0.942. But it is just one fold, so I doubt it is overfitting to the public LB.

", "rawMarkdown": "The local CV is 0.944 and LB is 0.942. But it is just one fold, so I doubt it is overfitting to the public LB.", "votes": 3 }, { "id": 901470, "postDate": "2020-06-25T14:01:22.353Z", "content": "

:) i think we are all looking forward to see what are you doing there ..... congrats

", "rawMarkdown": ":) i think we are all looking forward to see what are you doing there ..... congrats", "votes": 2 }, { "id": 901564, "postDate": "2020-06-25T15:09:47.603Z", "content": "

0.944 with single model?

", "rawMarkdown": "0.944 with single model?" }, { "id": 902093, "postDate": "2020-06-25T23:42:43.443Z", "content": "

There is a month to go. Maybe it's a little early to stack or ensemble.

", "rawMarkdown": "There is a month to go. Maybe it's a little early to stack or ensemble." } ] }, { "id": 900566, "postDate": "2020-06-24T22:21:25.810Z", "content": "

It's too long to train all folds. So, how to check if it is overfitting to the public LB ? Is it impossible?

", "rawMarkdown": "It's too long to train all folds. So, how to check if it is overfitting to the public LB ? Is it impossible?", "votes": 8 }, { "id": 901427, "postDate": "2020-06-25T13:22:42.960Z", "content": "

@wuliaokaola hey, from our last conversation you told you were using TPU? Are you still using it or shifted to GPU? And may I know how much time each epoch needs in your cases?

", "rawMarkdown": "@wuliaokaola hey, from our last conversation you told you were using TPU? Are you still using it or shifted to GPU? And may I know how much time each epoch needs in your cases?", "votes": 1, "replies": [ { "id": 901446, "postDate": "2020-06-25T13:44:56.703Z", "content": "

Yes, I'm still using TPU, because large batch size give better result. It takes 20~30 mins per epoch. \nI'm planning to use @mathormad 's method to save time.

", "rawMarkdown": "Yes, I'm still using TPU, because large batch size give better result. It takes 20~30 mins per epoch. \nI'm planning to use @mathormad 's method to save time.", "votes": 1 }, { "id": 901464, "postDate": "2020-06-25T13:55:27.033Z", "content": "

using colab ?

", "rawMarkdown": "using colab ?" }, { "id": 901482, "postDate": "2020-06-25T14:16:53.707Z", "content": "

@wuliaokaola thanks, it's great. \n@valanm colab pro maybe worthy, in our region it's not available though :(

", "rawMarkdown": "@wuliaokaola thanks, it's great. \n@valanm colab pro maybe worthy, in our region it's not available though :( ", "votes": 1 }, { "id": 904014, "postDate": "2020-06-27T09:22:10.880Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 910176, "postDate": "2020-07-01T03:12:21.947Z", "content": "

Singel model for the current LB?

", "rawMarkdown": "Singel model for the current LB?", "replies": [ { "id": 912125, "postDate": "2020-07-02T08:44:28.027Z", "content": "

how about you?

", "rawMarkdown": "how about you?" }, { "id": 912597, "postDate": "2020-07-02T15:39:12.063Z", "content": "

I ensmable with three model

", "rawMarkdown": "I ensmable with three model ", "votes": 2 } ] }, { "id": 903664, "postDate": "2020-06-27T03:20:27.933Z", "content": "

different fold shifts a lot in my experiment. 😟

", "rawMarkdown": " different fold shifts a lot in my experiment. 😟 " } ], "comments": [ { "id": 900695, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2020-06-25T02:43:40.413000", "content": "

After splitting k-fold, undersampling your training set (e.g: keep only 60k images), train your models, and monitor weighted AUC on full validation data. When you find the best parameters, rerun your model with full data.

\n\n

In my experiments, the local CV and public LB are quite consistent (0.888 vs. 0.890 with undersampling, 0.926 vs.0.928 with full data)

", "votes": 16, "replies": [ { "id": 900785, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-06-25T04:10:38.300000", "content": "

Nice idea! Thank you. This can save us a lot of time.

", "votes": 2, "replies": [] }, { "id": 901454, "author_name": "robga", "author_url": "", "post_date": "2020-06-25T13:48:00.013000", "content": "

What is your local CV score score for your .941 LB score?

", "votes": 0, "replies": [] }, { "id": 901466, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-06-25T13:58:22.273000", "content": "

The local CV is 0.944 and LB is 0.942. But it is just one fold, so I doubt it is overfitting to the public LB.

", "votes": 3, "replies": [] }, { "id": 901470, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-25T14:01:22.353000", "content": "

:) i think we are all looking forward to see what are you doing there ..... congrats

", "votes": 2, "replies": [] }, { "id": 901564, "author_name": "[ods.ai]Uladzimir Tumanau", "author_url": "", "post_date": "2020-06-25T15:09:47.603000", "content": "

0.944 with single model?

", "votes": 0, "replies": [] }, { "id": 902093, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-06-25T23:42:43.443000", "content": "

There is a month to go. Maybe it's a little early to stack or ensemble.

", "votes": 0, "replies": [] } ] }, { "id": 901427, "author_name": "Innat", "author_url": "", "post_date": "2020-06-25T13:22:42.960000", "content": "

@wuliaokaola hey, from our last conversation you told you were using TPU? Are you still using it or shifted to GPU? And may I know how much time each epoch needs in your cases?

", "votes": 1, "replies": [ { "id": 901446, "author_name": "Johnny Lee", "author_url": "", "post_date": "2020-06-25T13:44:56.703000", "content": "

Yes, I'm still using TPU, because large batch size give better result. It takes 20~30 mins per epoch. \nI'm planning to use @mathormad 's method to save time.

", "votes": 1, "replies": [] }, { "id": 901464, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-25T13:55:27.033000", "content": "

using colab ?

", "votes": 0, "replies": [] }, { "id": 901482, "author_name": "Innat", "author_url": "", "post_date": "2020-06-25T14:16:53.707000", "content": "

@wuliaokaola thanks, it's great. \n@valanm colab pro maybe worthy, in our region it's not available though :(

", "votes": 1, "replies": [] }, { "id": 904014, "author_name": "", "author_url": "", "post_date": "2020-06-27T09:22:10.880000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 910176, "author_name": "BokingChen", "author_url": "", "post_date": "2020-07-01T03:12:21.947000", "content": "

Singel model for the current LB?

", "votes": 0, "replies": [ { "id": 912125, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2020-07-02T08:44:28.027000", "content": "

how about you?

", "votes": 0, "replies": [] }, { "id": 912597, "author_name": "BokingChen", "author_url": "", "post_date": "2020-07-02T15:39:12.063000", "content": "

I ensmable with three model

", "votes": 2, "replies": [] } ] }, { "id": 903664, "author_name": "musicbeer", "author_url": "", "post_date": "2020-06-27T03:20:27.933000", "content": "

different fold shifts a lot in my experiment. 😟

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "900695": "After splitting k-fold, undersampling your training set (e.g: keep only 60k images), train your models, and monitor weighted AUC on full validation data. When you find the best parameters, rerun your model with full data.\n\nIn my experiments, the local CV and public LB are quite consistent (0.888 vs. 0.890 with undersampling, 0.926 vs.0.928 with full data)", "900566": "It's too long to train all folds. So, how to check if it is overfitting to the public LB ? Is it impossible?", "901427": "@wuliaokaola hey, from our last conversation you told you were using TPU? Are you still using it or shifted to GPU? And may I know how much time each epoch needs in your cases?", "910176": "Singel model for the current LB?", "903664": " different fold shifts a lot in my experiment. 😟 " } }