{
  "id": 160149,
  "title": "How to create a valid dataset and what metric to use for EarlyStopping?",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/160149",
  "author_name": "Aman Arora",
  "post_date": "2020-06-20T04:15:59.253000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi there! </p>\n\n<p>Creating this topic for discussion around creating a 'valid' valid dataset. In this competition, as in most others, I believe creating a robust CV scheme is highly important. So far we have seen ideas like using KFold, GroupKFold and even StratifiedGroupKFold. </p>\n\n<p>But from experience and training multiple models for this competition so far, I realized that my CV scores do not match test leaderboard having tried many different schemes. </p>\n\n<p>I have been using 'auc' for Early Stopping and Binary Cross Entropy loss for backward propagation. I have also added TTA using FiveCrop scheme from PyTorch but I still see uncertainty in auc between my CV and test leaderboard. Sometimes they match, sometimes they dont!</p>\n\n<p>Wondering if there are any ideas anyone would like to share on how to create a good validation set and CV scheme for this competition?</p>",
  "messages": [
    {
      "id": 893892,
      "postDate": "2020-06-20T04:15:59.253Z",
      "content": "<p>Hi there! </p>\n\n<p>Creating this topic for discussion around creating a 'valid' valid dataset. In this competition, as in most others, I believe creating a robust CV scheme is highly important. So far we have seen ideas like using KFold, GroupKFold and even StratifiedGroupKFold. </p>\n\n<p>But from experience and training multiple models for this competition so far, I realized that my CV scores do not match test leaderboard having tried many different schemes. </p>\n\n<p>I have been using 'auc' for Early Stopping and Binary Cross Entropy loss for backward propagation. I have also added TTA using FiveCrop scheme from PyTorch but I still see uncertainty in auc between my CV and test leaderboard. Sometimes they match, sometimes they dont!</p>\n\n<p>Wondering if there are any ideas anyone would like to share on how to create a good validation set and CV scheme for this competition?</p>",
      "rawMarkdown": "Hi there! \n\nCreating this topic for discussion around creating a 'valid' valid dataset. In this competition, as in most others, I believe creating a robust CV scheme is highly important. So far we have seen ideas like using KFold, GroupKFold and even StratifiedGroupKFold. \n\nBut from experience and training multiple models for this competition so far, I realized that my CV scores do not match test leaderboard having tried many different schemes. \n\nI have been using 'auc' for Early Stopping and Binary Cross Entropy loss for backward propagation. I have also added TTA using FiveCrop scheme from PyTorch but I still see uncertainty in auc between my CV and test leaderboard. Sometimes they match, sometimes they dont!\n\nWondering if there are any ideas anyone would like to share on how to create a good validation set and CV scheme for this competition?",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "893892": "Hi there! \n\nCreating this topic for discussion around creating a 'valid' valid dataset. In this competition, as in most others, I believe creating a robust CV scheme is highly important. So far we have seen ideas like using KFold, GroupKFold and even StratifiedGroupKFold. \n\nBut from experience and training multiple models for this competition so far, I realized that my CV scores do not match test leaderboard having tried many different schemes. \n\nI have been using 'auc' for Early Stopping and Binary Cross Entropy loss for backward propagation. I have also added TTA using FiveCrop scheme from PyTorch but I still see uncertainty in auc between my CV and test leaderboard. Sometimes they match, sometimes they dont!\n\nWondering if there are any ideas anyone would like to share on how to create a good validation set and CV scheme for this competition?"
  }
}