{
  "id": 73145,
  "title": "Validation error as function of redshift",
  "url": "/competitions/PLAsTiCC-2018/discussion/73145",
  "author_name": "",
  "post_date": "2018-11-30T08:06:22.255003100Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>My experience so far has been that my validation error for Milky Way objects is relatively low while the extragalactic objects have been high.  There are a lot of factors behind that, but one relationship I expected to see was validation error on closer objects to be lower on nearer objects and higher on farther objects (measurement errors go up, more extinction, more dilation).  </p>\n\n<p>What I noticed though was more like a bubble - errors on objects between 0 and 0.2 photoz are much higher than on objects farther out (like 0.5 log-loss difference).  Has anyone else seen something like this or is it just my busted models?</p>",
  "messages": [
    {
      "id": "430329",
      "postDate": "11/30/2018 08:06:22",
      "content": "<p>My experience so far has been that my validation error for Milky Way objects is relatively low while the extragalactic objects have been high.  There are a lot of factors behind that, but one relationship I expected to see was validation error on closer objects to be lower on nearer objects and higher on farther objects (measurement errors go up, more extinction, more dilation).  </p>\n\n<p>What I noticed though was more like a bubble - errors on objects between 0 and 0.2 photoz are much higher than on objects farther out (like 0.5 log-loss difference).  Has anyone else seen something like this or is it just my busted models?</p>",
      "rawMarkdown": "My experience so far has been that my validation error for Milky Way objects is relatively low while the extragalactic objects have been high.  There are a lot of factors behind that, but one relationship I expected to see was validation error on closer objects to be lower on nearer objects and higher on farther objects (measurement errors go up, more extinction, more dilation).  \n\nWhat I noticed though was more like a bubble - errors on objects between 0 and 0.2 photoz are much higher than on objects farther out (like 0.5 log-loss difference).  Has anyone else seen something like this or is it just my busted models?",
      "votes": null
    },
    {
      "id": "430341",
      "postDate": "11/30/2018 08:33:32",
      "content": "<p>One observation is that there are very few class 88 or 95's in that band of photoz (0, 0.2] and those classes appear to be relatively easy to classify.  Their absence in that band just highlights how bad my model is on the other classes, I guess.</p>",
      "rawMarkdown": "One observation is that there are very few class 88 or 95's in that band of photoz (0, 0.2] and those classes appear to be relatively easy to classify.  Their absence in that band just highlights how bad my model is on the other classes, I guess.",
      "votes": null
    },
    {
      "id": "430429",
      "postDate": "11/30/2018 11:20:28",
      "content": "<p>Do you use the competition metric?  Frequency of each class should not be relevant.  </p>",
      "rawMarkdown": "Do you use the competition metric?  Frequency of each class should not be relevant.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 430341,
      "author_name": "mikeholcomb",
      "author_url": "",
      "post_date": "11/30/2018 08:33:32",
      "content": "<p>One observation is that there are very few class 88 or 95's in that band of photoz (0, 0.2] and those classes appear to be relatively easy to classify.  Their absence in that band just highlights how bad my model is on the other classes, I guess.</p>",
      "votes": null,
      "replies": [
        {
          "id": 430429,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "11/30/2018 11:20:28",
          "content": "<p>Do you use the competition metric?  Frequency of each class should not be relevant.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "430329": "My experience so far has been that my validation error for Milky Way objects is relatively low while the extragalactic objects have been high.  There are a lot of factors behind that, but one relationship I expected to see was validation error on closer objects to be lower on nearer objects and higher on farther objects (measurement errors go up, more extinction, more dilation).  \n\nWhat I noticed though was more like a bubble - errors on objects between 0 and 0.2 photoz are much higher than on objects farther out (like 0.5 log-loss difference).  Has anyone else seen something like this or is it just my busted models?",
    "430341": "One observation is that there are very few class 88 or 95's in that band of photoz (0, 0.2] and those classes appear to be relatively easy to classify.  Their absence in that band just highlights how bad my model is on the other classes, I guess.",
    "430429": "Do you use the competition metric?  Frequency of each class should not be relevant."
  },
  "source": "meta"
}