{
  "id": 21914,
  "title": "Decorrelate final selections",
  "url": "/competitions/draper-satellite-image-chronology/discussion/21914",
  "author_name": "Mike Kim",
  "post_date": "2016-06-28T03:05:50.657000",
  "votes": 0,
  "comment_count": 0,
  "views": 383,
  "content": "<p>I made a big mistake (which was entirely my fault) of selecting two final submissions that were highly correlated to each other. I had a learning to rank model scoring: 0.32806, but doing fairly poorly on the public (0.6X local repeated CV). This was uncorrelated to my best public model which was basically last on private. I could not get stratified cross validation to work even with 300 or so repeats, so I gave up on it. However, even if your CV does not work you can always select two uncorrelated final submissions. </p>\n\n<p>Hopefully this can spare other people the pain of 99% leaderboard drops given a diverse set of models.</p>",
  "messages": [
    {
      "id": 125245,
      "postDate": "2016-06-28T03:05:50.657Z",
      "content": "<p>I made a big mistake (which was entirely my fault) of selecting two final submissions that were highly correlated to each other. I had a learning to rank model scoring: 0.32806, but doing fairly poorly on the public (0.6X local repeated CV). This was uncorrelated to my best public model which was basically last on private. I could not get stratified cross validation to work even with 300 or so repeats, so I gave up on it. However, even if your CV does not work you can always select two uncorrelated final submissions. </p>\n\n<p>Hopefully this can spare other people the pain of 99% leaderboard drops given a diverse set of models.</p>",
      "rawMarkdown": "I made a big mistake (which was entirely my fault) of selecting two final submissions that were highly correlated to each other. I had a learning to rank model scoring: 0.32806, but doing fairly poorly on the public (0.6X local repeated CV). This was uncorrelated to my best public model which was basically last on private. I could not get stratified cross validation to work even with 300 or so repeats, so I gave up on it. However, even if your CV does not work you can always select two uncorrelated final submissions. \r\n\r\nHopefully this can spare other people the pain of 99% leaderboard drops given a diverse set of models."
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "125245": "I made a big mistake (which was entirely my fault) of selecting two final submissions that were highly correlated to each other. I had a learning to rank model scoring: 0.32806, but doing fairly poorly on the public (0.6X local repeated CV). This was uncorrelated to my best public model which was basically last on private. I could not get stratified cross validation to work even with 300 or so repeats, so I gave up on it. However, even if your CV does not work you can always select two uncorrelated final submissions. \r\n\r\nHopefully this can spare other people the pain of 99% leaderboard drops given a diverse set of models."
  }
}