{ "id": 170113, "title": "I am kaggle noobs, My CV and LB are so much inconsistent", "url": "/competitions/siim-isic-melanoma-classification/discussion/170113", "author_name": "", "post_date": "2020-07-26T13:27:43.387826700Z", "votes": 1, "comment_count": 12, "views": 0, "content": "

My CV vs LB\nAll 5 Fold CV

\n\n
    \n
  1. CV 0.9196 -> LB : 0.944
    2. \n
  2. CV 0.9351-> LB : 0.9511
    3. \n
  3. CV 0.9485 -> LB : 0.9434
    4. \n
\n\n

What should I trust?

\n\n

Do I have to concentrate on CV or LB?\nI guess my current LB is over-fitted\nSomebody help me!

", "messages": [ { "id": "946265", "postDate": "07/26/2020 13:27:43", "content": "

My CV vs LB\nAll 5 Fold CV

\n\n
    \n
  1. CV 0.9196 -> LB : 0.944
    2. \n
  2. CV 0.9351-> LB : 0.9511
    3. \n
  3. CV 0.9485 -> LB : 0.9434
    4. \n
\n\n

What should I trust?

\n\n

Do I have to concentrate on CV or LB?\nI guess my current LB is over-fitted\nSomebody help me!

", "rawMarkdown": "**My CV vs LB**\nAll 5 Fold CV\n\n1. CV 0.9196 -> LB : 0.944\n2. CV 0.9351-> LB : 0.9511\n3. CV 0.9485 -> LB : 0.9434\n\nWhat should I trust?\n\nDo I have to concentrate on CV or LB?\nI guess my current LB is over-fitted\nSomebody help me!", "votes": null }, { "id": "946544", "postDate": "07/26/2020 16:32:41", "content": "

The answer is very obvious: if you have a good validation strategy, then your second model is the best.

", "rawMarkdown": "The answer is very obvious: if you have a good validation strategy, then your second model is the best.", "votes": null }, { "id": "946591", "postDate": "07/26/2020 16:59:47", "content": "

A higher CV in your case gives a higher LB. What's the problem?

", "rawMarkdown": "A higher CV in your case gives a higher LB. What's the problem?", "votes": null }, { "id": "947106", "postDate": "07/27/2020 04:41:27", "content": "

=) The bigger the better ?\nThere is a correlation between CV and LB.\nKeep doing what you doing, don't overfit otherwise the shakedown is going to be painful.

", "rawMarkdown": ") The bigger the better ?\nThere is a correlation between CV and LB.\nKeep doing what you doing, don't overfit otherwise the shakedown is going to be painful.", "votes": null }, { "id": "947835", "postDate": "07/27/2020 14:05:39", "content": "

Thank you so much\nI'm gonna keep trying!

", "rawMarkdown": "Thank you so much\nI'm gonna keep trying!", "votes": null }, { "id": "947836", "postDate": "07/27/2020 14:06:05", "content": "

Oh i missed putting This\nCV 0.9485 -> LB : 0.9434

", "rawMarkdown": "Oh i missed putting This\nCV 0.9485 -> LB : 0.9434", "votes": null }, { "id": "948917", "postDate": "07/28/2020 10:18:22", "content": "

@deepkim is it a single fold CV or 5 fold CV?

", "rawMarkdown": "deepkim is it a single fold CV or 5 fold CV?", "votes": null }, { "id": "949428", "postDate": "07/28/2020 16:21:41", "content": "

Fortunately we can choose up to 3 submissions as final score. But I'd prioritize your 3rd model CV 0.9485 -> LB : 0.9434

", "rawMarkdown": "Fortunately we can choose up to 3 submissions as final score. But I'd prioritize your 3rd model CV 0.9485 -> LB : 0.9434", "votes": null }, { "id": "949435", "postDate": "07/28/2020 16:27:46", "content": "

Can you describe your cross validation. What data is in your validation folds (external or original), have you removed duplicates between folds, have you stratified patients and malignant between folds, etc etc. It is hard to comment on your situation without knowing what you are doing.

", "rawMarkdown": "Can you describe your cross validation. What data is in your validation folds (external or original), have you removed duplicates between folds, have you stratified patients and malignant between folds, etc etc. It is hard to comment on your situation without knowing what you are doing.", "votes": null }, { "id": "949454", "postDate": "07/28/2020 16:36:51", "content": "

I'm playing around with Chris' triple stratified notebook and I'm seeing this difference between basically the same model with 2 changes in model. Hard to gauge my experiments.\n\"\"

", "rawMarkdown": "I'm playing around with Chris' triple stratified notebook and I'm seeing this difference between basically the same model with 2 changes in model. Hard to gauge my experiments.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1894505%2F0388770f6ff7f1f26a24ceaa5f0b564e%2FMelanoma_CV_diff.PNG?generation=1595954027134992&alt=media)", "votes": null }, { "id": "953257", "postDate": "07/31/2020 16:25:02", "content": "

5 fold cv!

", "rawMarkdown": "5 fold cv!", "votes": null }, { "id": "953370", "postDate": "07/31/2020 18:21:25", "content": "

Using chris CV strategy: Ensemble (B0-B7, Img Size 384) : CV = 0.934 LB=0.950\nUsing chris CV strategy with some changes to above model, Effnet B5 384*384: CV=0.942, LB=0.945 (5 fold)\nEnsemble (B0-B7, Img Size 384): CV=0.944 LB=0.944\nSo, yes, The inconsistency is there for me!!

", "rawMarkdown": "Using chris CV strategy: Ensemble (B0-B7, Img Size 384) : CV = 0.934 LB=0.950\nUsing chris CV strategy with some changes to above model, Effnet B5 384*384: CV=0.942, LB=0.945 (5 fold)\nEnsemble (B0-B7, Img Size 384): CV=0.944 LB=0.944\nSo, yes, The inconsistency is there for me!!", "votes": null }, { "id": "953731", "postDate": "08/01/2020 03:55:07", "content": "

@deepkim any suggestions on improving CV?

", "rawMarkdown": "deepkim any suggestions on improving CV?", "votes": null } ], "comments": [ { "id": 946544, "author_name": "seif95", "author_url": "", "post_date": "07/26/2020 16:32:41", "content": "

The answer is very obvious: if you have a good validation strategy, then your second model is the best.

", "votes": null, "replies": [] }, { "id": 946591, "author_name": "pasqualed", "author_url": "", "post_date": "07/26/2020 16:59:47", "content": "

A higher CV in your case gives a higher LB. What's the problem?

", "votes": null, "replies": [ { "id": 947836, "author_name": "deepkim", "author_url": "", "post_date": "07/27/2020 14:06:05", "content": "

Oh i missed putting This\nCV 0.9485 -> LB : 0.9434

", "votes": null, "replies": [] } ] }, { "id": 947106, "author_name": "alincijov", "author_url": "", "post_date": "07/27/2020 04:41:27", "content": "

=) The bigger the better ?\nThere is a correlation between CV and LB.\nKeep doing what you doing, don't overfit otherwise the shakedown is going to be painful.

", "votes": null, "replies": [ { "id": 947835, "author_name": "deepkim", "author_url": "", "post_date": "07/27/2020 14:05:39", "content": "

Thank you so much\nI'm gonna keep trying!

", "votes": null, "replies": [] } ] }, { "id": 948917, "author_name": "virajbagal", "author_url": "", "post_date": "07/28/2020 10:18:22", "content": "

@deepkim is it a single fold CV or 5 fold CV?

", "votes": null, "replies": [ { "id": 953257, "author_name": "deepkim", "author_url": "", "post_date": "07/31/2020 16:25:02", "content": "

5 fold cv!

", "votes": null, "replies": [] } ] }, { "id": 949428, "author_name": "santiviquez", "author_url": "", "post_date": "07/28/2020 16:21:41", "content": "

Fortunately we can choose up to 3 submissions as final score. But I'd prioritize your 3rd model CV 0.9485 -> LB : 0.9434

", "votes": null, "replies": [] }, { "id": 949435, "author_name": "cdeotte", "author_url": "", "post_date": "07/28/2020 16:27:46", "content": "

Can you describe your cross validation. What data is in your validation folds (external or original), have you removed duplicates between folds, have you stratified patients and malignant between folds, etc etc. It is hard to comment on your situation without knowing what you are doing.

", "votes": null, "replies": [] }, { "id": 949454, "author_name": "teeyee314", "author_url": "", "post_date": "07/28/2020 16:36:51", "content": "

I'm playing around with Chris' triple stratified notebook and I'm seeing this difference between basically the same model with 2 changes in model. Hard to gauge my experiments.\n\"\"

", "votes": null, "replies": [] }, { "id": 953370, "author_name": "ks2019", "author_url": "", "post_date": "07/31/2020 18:21:25", "content": "

Using chris CV strategy: Ensemble (B0-B7, Img Size 384) : CV = 0.934 LB=0.950\nUsing chris CV strategy with some changes to above model, Effnet B5 384*384: CV=0.942, LB=0.945 (5 fold)\nEnsemble (B0-B7, Img Size 384): CV=0.944 LB=0.944\nSo, yes, The inconsistency is there for me!!

", "votes": null, "replies": [] }, { "id": 953731, "author_name": "virajbagal", "author_url": "", "post_date": "08/01/2020 03:55:07", "content": "

@deepkim any suggestions on improving CV?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "946265": "**My CV vs LB**\nAll 5 Fold CV\n\n1. CV 0.9196 -> LB : 0.944\n2. CV 0.9351-> LB : 0.9511\n3. CV 0.9485 -> LB : 0.9434\n\nWhat should I trust?\n\nDo I have to concentrate on CV or LB?\nI guess my current LB is over-fitted\nSomebody help me!", "946544": "The answer is very obvious: if you have a good validation strategy, then your second model is the best.", "946591": "A higher CV in your case gives a higher LB. What's the problem?", "947106": ") The bigger the better ?\nThere is a correlation between CV and LB.\nKeep doing what you doing, don't overfit otherwise the shakedown is going to be painful.", "947835": "Thank you so much\nI'm gonna keep trying!", "947836": "Oh i missed putting This\nCV 0.9485 -> LB : 0.9434", "948917": "deepkim is it a single fold CV or 5 fold CV?", "949428": "Fortunately we can choose up to 3 submissions as final score. But I'd prioritize your 3rd model CV 0.9485 -> LB : 0.9434", "949435": "Can you describe your cross validation. What data is in your validation folds (external or original), have you removed duplicates between folds, have you stratified patients and malignant between folds, etc etc. It is hard to comment on your situation without knowing what you are doing.", "949454": "I'm playing around with Chris' triple stratified notebook and I'm seeing this difference between basically the same model with 2 changes in model. Hard to gauge my experiments.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1894505%2F0388770f6ff7f1f26a24ceaa5f0b564e%2FMelanoma_CV_diff.PNG?generation=1595954027134992&alt=media)", "953257": "5 fold cv!", "953370": "Using chris CV strategy: Ensemble (B0-B7, Img Size 384) : CV = 0.934 LB=0.950\nUsing chris CV strategy with some changes to above model, Effnet B5 384*384: CV=0.942, LB=0.945 (5 fold)\nEnsemble (B0-B7, Img Size 384): CV=0.944 LB=0.944\nSo, yes, The inconsistency is there for me!!", "953731": "deepkim any suggestions on improving CV?" }, "source": "meta" }