{
  "id": 20983,
  "title": "No change in submission score despite changes",
  "url": "/competitions/draper-satellite-image-chronology/discussion/20983",
  "author_name": "",
  "post_date": "2016-05-15T21:57:44.813Z",
  "votes": 1,
  "comment_count": 4,
  "views": 624,
  "content": "<p>Looking for help from either Admin or another Kaggler,</p>\n\n<p>[I've done a couple of contests, so I know the basics].</p>\n\n<p>I used the &quot;beat the benchmark&quot; entry (4 5 3 2 1) which gets 0.14286 (tried by about 100 people).</p>\n\n<p>I changed 26 entries, leaving the rest the same. I haven't applied my process to the rest of the test set yet, so I left them as &quot;4 5 3 2 1&quot;.</p>\n\n<p>There is no change in my score.</p>\n\n<p>I know the scoring is on 17%, but I would still expect a change at the 4th decimal place at least. </p>\n\n<p>I guess it's possible that I happened to pick 26 test cases that were not part of the 17%, but that seems unlikely. </p>\n\n<p>Obviously, my algorithm could be totally wrong, and no better or worse than random guessing, but I would still expect some change.</p>\n\n<p>I'm trying not to waste submissions troubleshooting.</p>\n\n<p>The format of the submission file (if I understand it correctly), is to list the position of each image, as follows:</p>\n\n<p>If the correct images are in order image 2, image 3, image 4, image 5, image 1, then you would list:</p>\n\n<p>setid, 5 1 2 3 4</p>\n\n<p>since Image 1 belongs in the 5th place, image 2 in the 1st place, etc.</p>\n\n<p>(this seems slightly unintuitive, where I would have listed the images in the expected order, i.e. 2, 3, 4, 5, 1, but that doesn't seem to be what the instructions require. The method the rules describes is probably a little easier to program however.)</p>\n\n<p>Any ideas?</p>\n\n<p>-Rich</p>",
  "messages": [
    {
      "id": "120156",
      "postDate": "05/15/2016 21:57:44",
      "content": "<p>Looking for help from either Admin or another Kaggler,</p>\n\n<p>[I've done a couple of contests, so I know the basics].</p>\n\n<p>I used the &quot;beat the benchmark&quot; entry (4 5 3 2 1) which gets 0.14286 (tried by about 100 people).</p>\n\n<p>I changed 26 entries, leaving the rest the same. I haven't applied my process to the rest of the test set yet, so I left them as &quot;4 5 3 2 1&quot;.</p>\n\n<p>There is no change in my score.</p>\n\n<p>I know the scoring is on 17%, but I would still expect a change at the 4th decimal place at least. </p>\n\n<p>I guess it's possible that I happened to pick 26 test cases that were not part of the 17%, but that seems unlikely. </p>\n\n<p>Obviously, my algorithm could be totally wrong, and no better or worse than random guessing, but I would still expect some change.</p>\n\n<p>I'm trying not to waste submissions troubleshooting.</p>\n\n<p>The format of the submission file (if I understand it correctly), is to list the position of each image, as follows:</p>\n\n<p>If the correct images are in order image 2, image 3, image 4, image 5, image 1, then you would list:</p>\n\n<p>setid, 5 1 2 3 4</p>\n\n<p>since Image 1 belongs in the 5th place, image 2 in the 1st place, etc.</p>\n\n<p>(this seems slightly unintuitive, where I would have listed the images in the expected order, i.e. 2, 3, 4, 5, 1, but that doesn't seem to be what the instructions require. The method the rules describes is probably a little easier to program however.)</p>\n\n<p>Any ideas?</p>\n\n<p>-Rich</p>",
      "rawMarkdown": "Looking for help from either Admin or another Kaggler,\r\n\r\n[I've done a couple of contests, so I know the basics].\r\n\r\nI used the \"beat the benchmark\" entry (4 5 3 2 1) which gets 0.14286 (tried by about 100 people).\r\n\r\nI changed 26 entries, leaving the rest the same. I haven't applied my process to the rest of the test set yet, so I left them as \"4 5 3 2 1\".\r\n\r\nThere is no change in my score.\r\n\r\nI know the scoring is on 17%, but I would still expect a change at the 4th decimal place at least. \r\n\r\nI guess it's possible that I happened to pick 26 test cases that were not part of the 17%, but that seems unlikely. \r\n\r\nObviously, my algorithm could be totally wrong, and no better or worse than random guessing, but I would still expect some change.\r\n\r\nI'm trying not to waste submissions troubleshooting.\r\n\r\nThe format of the submission file (if I understand it correctly), is to list the position of each image, as follows:\r\n\r\nIf the correct images are in order image 2, image 3, image 4, image 5, image 1, then you would list:\r\n\r\nsetid, 5 1 2 3 4\r\n\r\nsince Image 1 belongs in the 5th place, image 2 in the 1st place, etc.\r\n\r\n(this seems slightly unintuitive, where I would have listed the images in the expected order, i.e. 2, 3, 4, 5, 1, but that doesn't seem to be what the instructions require. The method the rules describes is probably a little easier to program however.)\r\n\r\nAny ideas?\r\n\r\n-Rich",
      "votes": null
    },
    {
      "id": "120159",
      "postDate": "05/15/2016 22:17:02",
      "content": "<p>Here's some of the possibilities why your score didn't change</p>\n\n<ul>\n<li>None were in the leaderboard (this can happen).</li>\n<li>Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.</li>\n<li>Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.</li>\n</ul>\n\n<p>If some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...  </p>",
      "rawMarkdown": "Here's some of the possibilities why your score didn't change\r\n\r\n - None were in the leaderboard (this can happen).\r\n - Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.\r\n - Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.\r\n\r\nIf some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...",
      "votes": null
    },
    {
      "id": "120162",
      "postDate": "05/15/2016 23:23:29",
      "content": "<p>[quote=Chris H.;120159]</p>\n\n<p>Here's some of the possibilities why your score didn't change</p>\n\n<ul>\n<li>None were in the leaderboard (this can happen).</li>\n<li>Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.</li>\n<li>Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.</li>\n</ul>\n\n<p>If some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...  </p>\n\n<p>[/quote]</p>\n\n<p>1 &amp; 2 were my thoughts too, but @quadcore makes a good point that out of 26 changes the probability of undergoing zero score change is infinitesimally small.  Those would technically explain the lack of score change, but if I were quad I would still post to ensure that was the issue.</p>",
      "rawMarkdown": "[quote=Chris H.;120159]\r\n\r\nHere's some of the possibilities why your score didn't change\r\n\r\n - None were in the leaderboard (this can happen).\r\n - Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.\r\n - Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.\r\n\r\nIf some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...  \r\n\r\n[/quote]\r\n\r\n1 & 2 were my thoughts too, but @quadcore makes a good point that out of 26 changes the probability of undergoing zero score change is infinitesimally small.  Those would technically explain the lack of score change, but if I were quad I would still post to ensure that was the issue.",
      "votes": null
    },
    {
      "id": "120166",
      "postDate": "05/16/2016 00:36:17",
      "content": "<p>We have also included a small number of ignored image sets in the test set (which increases the odds of no change in score happening by chance).</p>",
      "rawMarkdown": "We have also included a small number of ignored image sets in the test set (which increases the odds of no change in score happening by chance).",
      "votes": null
    },
    {
      "id": "120175",
      "postDate": "05/16/2016 04:05:46",
      "content": "<p>Thanks.</p>\n\n<p>I switched to the original benchmark &quot;1 2 3 4 5&quot;, which scores lower than the &quot;4 5 3 2 1&quot; benchmark (figured I'd make sure my changes were not obscured by the overfitted &quot;4 5 3 2 1&quot; benchmark). Then I applied my guesses on the original 26 (see above) and a few more. I finally saw a small (but positive) change in my score.</p>\n\n<p>So, I probably have things working now. </p>\n\n<p>Again, Thanks.</p>\n\n<p>-Rich</p>",
      "rawMarkdown": "Thanks.\r\n\r\nI switched to the original benchmark \"1 2 3 4 5\", which scores lower than the \"4 5 3 2 1\" benchmark (figured I'd make sure my changes were not obscured by the overfitted \"4 5 3 2 1\" benchmark). Then I applied my guesses on the original 26 (see above) and a few more. I finally saw a small (but positive) change in my score.\r\n\r\nSo, I probably have things working now. \r\n\r\nAgain, Thanks.\r\n\r\n-Rich",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 120159,
      "author_name": "hurlburt",
      "author_url": "",
      "post_date": "05/15/2016 22:17:02",
      "content": "<p>Here's some of the possibilities why your score didn't change</p>\n\n<ul>\n<li>None were in the leaderboard (this can happen).</li>\n<li>Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.</li>\n<li>Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.</li>\n</ul>\n\n<p>If some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120162,
      "author_name": "tylervigen",
      "author_url": "",
      "post_date": "05/15/2016 23:23:29",
      "content": "<p>[quote=Chris H.;120159]</p>\n\n<p>Here's some of the possibilities why your score didn't change</p>\n\n<ul>\n<li>None were in the leaderboard (this can happen).</li>\n<li>Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.</li>\n<li>Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.</li>\n</ul>\n\n<p>If some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...  </p>\n\n<p>[/quote]</p>\n\n<p>1 &amp; 2 were my thoughts too, but @quadcore makes a good point that out of 26 changes the probability of undergoing zero score change is infinitesimally small.  Those would technically explain the lack of score change, but if I were quad I would still post to ensure that was the issue.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120166,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "05/16/2016 00:36:17",
      "content": "<p>We have also included a small number of ignored image sets in the test set (which increases the odds of no change in score happening by chance).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 120175,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "05/16/2016 04:05:46",
      "content": "<p>Thanks.</p>\n\n<p>I switched to the original benchmark &quot;1 2 3 4 5&quot;, which scores lower than the &quot;4 5 3 2 1&quot; benchmark (figured I'd make sure my changes were not obscured by the overfitted &quot;4 5 3 2 1&quot; benchmark). Then I applied my guesses on the original 26 (see above) and a few more. I finally saw a small (but positive) change in my score.</p>\n\n<p>So, I probably have things working now. </p>\n\n<p>Again, Thanks.</p>\n\n<p>-Rich</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "120156": "Looking for help from either Admin or another Kaggler,\r\n\r\n[I've done a couple of contests, so I know the basics].\r\n\r\nI used the \"beat the benchmark\" entry (4 5 3 2 1) which gets 0.14286 (tried by about 100 people).\r\n\r\nI changed 26 entries, leaving the rest the same. I haven't applied my process to the rest of the test set yet, so I left them as \"4 5 3 2 1\".\r\n\r\nThere is no change in my score.\r\n\r\nI know the scoring is on 17%, but I would still expect a change at the 4th decimal place at least. \r\n\r\nI guess it's possible that I happened to pick 26 test cases that were not part of the 17%, but that seems unlikely. \r\n\r\nObviously, my algorithm could be totally wrong, and no better or worse than random guessing, but I would still expect some change.\r\n\r\nI'm trying not to waste submissions troubleshooting.\r\n\r\nThe format of the submission file (if I understand it correctly), is to list the position of each image, as follows:\r\n\r\nIf the correct images are in order image 2, image 3, image 4, image 5, image 1, then you would list:\r\n\r\nsetid, 5 1 2 3 4\r\n\r\nsince Image 1 belongs in the 5th place, image 2 in the 1st place, etc.\r\n\r\n(this seems slightly unintuitive, where I would have listed the images in the expected order, i.e. 2, 3, 4, 5, 1, but that doesn't seem to be what the instructions require. The method the rules describes is probably a little easier to program however.)\r\n\r\nAny ideas?\r\n\r\n-Rich",
    "120159": "Here's some of the possibilities why your score didn't change\r\n\r\n - None were in the leaderboard (this can happen).\r\n - Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.\r\n - Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.\r\n\r\nIf some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...",
    "120162": "[quote=Chris H.;120159]\r\n\r\nHere's some of the possibilities why your score didn't change\r\n\r\n - None were in the leaderboard (this can happen).\r\n - Spearman rank scoring is both positive and negative.  So it's entirely possible that you had 2-5 sets in the public leaderboard where the sum of the score changes from each of them totaled to zero.\r\n - Your orderings are zero correlated with the true ordering.  For example the Spearman rank score of '1 5 4 3 2' and '5 4 3 2 1' is zero.\r\n\r\nIf some of the 26 sets you modified were in the public leaderboard, it's also possible to have combinations of the last two items above, i.e. 3 sets in the public leaderboard where one creates a score change of + 0.00714, one creates a score change of - 0.00714 and one creates a score change of 0...  \r\n\r\n[/quote]\r\n\r\n1 & 2 were my thoughts too, but @quadcore makes a good point that out of 26 changes the probability of undergoing zero score change is infinitesimally small.  Those would technically explain the lack of score change, but if I were quad I would still post to ensure that was the issue.",
    "120166": "We have also included a small number of ignored image sets in the test set (which increases the odds of no change in score happening by chance).",
    "120175": "Thanks.\r\n\r\nI switched to the original benchmark \"1 2 3 4 5\", which scores lower than the \"4 5 3 2 1\" benchmark (figured I'd make sure my changes were not obscured by the overfitted \"4 5 3 2 1\" benchmark). Then I applied my guesses on the original 26 (see above) and a few more. I finally saw a small (but positive) change in my score.\r\n\r\nSo, I probably have things working now. \r\n\r\nAgain, Thanks.\r\n\r\n-Rich"
  },
  "source": "meta"
}