{
  "id": 491927,
  "title": "Income data discrepancies",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/491927",
  "author_name": "",
  "post_date": "2024-04-07T22:51:26.815275300Z",
  "votes": 19,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I have merged income data from two sources: \"static0\" (column maininc_215A) and \"person1\" (where num_group1=0, column mainoccupationinc_384A). I expected the income values from both sources to be approximately the same. However, the correlation between them is only 0.35. Examining the distribution of differences between the income data from these sources reveals substantial discrepancies, with differences ranging from -200,000 to +200,000 (which is significant given that the income data is capped at 200,000, and the average value is around 50,000). How could this be possible, given that both sources are internal? Which source is more reliable? </p>\n<p>Here are the quantiles for the differences:</p>\n<p>0.0   -193000.0<br>\n0.1    -28600.0<br>\n0.2    -12000.0<br>\n0.3     -2400.0<br>\n0.4         0.0<br>\n0.5      7000.0<br>\n0.6     14000.0<br>\n0.7     20600.0<br>\n0.8     33000.0<br>\n0.9     52000.0<br>\n1.0    199999.8</p>",
  "messages": [
    {
      "id": "2740727",
      "postDate": "04/07/2024 22:51:26",
      "content": "<p>I have merged income data from two sources: \"static0\" (column maininc_215A) and \"person1\" (where num_group1=0, column mainoccupationinc_384A). I expected the income values from both sources to be approximately the same. However, the correlation between them is only 0.35. Examining the distribution of differences between the income data from these sources reveals substantial discrepancies, with differences ranging from -200,000 to +200,000 (which is significant given that the income data is capped at 200,000, and the average value is around 50,000). How could this be possible, given that both sources are internal? Which source is more reliable? </p>\n<p>Here are the quantiles for the differences:</p>\n<p>0.0   -193000.0<br>\n0.1    -28600.0<br>\n0.2    -12000.0<br>\n0.3     -2400.0<br>\n0.4         0.0<br>\n0.5      7000.0<br>\n0.6     14000.0<br>\n0.7     20600.0<br>\n0.8     33000.0<br>\n0.9     52000.0<br>\n1.0    199999.8</p>",
      "rawMarkdown": "I have merged income data from two sources: \"static0\" (column maininc_215A) and \"person1\" (where num_group1=0, column mainoccupationinc_384A). I expected the income values from both sources to be approximately the same. However, the correlation between them is only 0.35. Examining the distribution of differences between the income data from these sources reveals substantial discrepancies, with differences ranging from -200,000 to +200,000 (which is significant given that the income data is capped at 200,000, and the average value is around 50,000). How could this be possible, given that both sources are internal? Which source is more reliable? \n\nHere are the quantiles for the differences:\n\n0.0   -193000.0\n0.1    -28600.0\n0.2    -12000.0\n0.3     -2400.0\n0.4         0.0\n0.5      7000.0\n0.6     14000.0\n0.7     20600.0\n0.8     33000.0\n0.9     52000.0\n1.0    199999.8",
      "votes": null
    },
    {
      "id": "2743789",
      "postDate": "04/09/2024 16:07:37",
      "content": "<p>Do we have any idea about the postfix for each column like _215A &amp; _384A?</p>",
      "rawMarkdown": "Do we have any idea about the postfix for each column like _215A & _384A?",
      "votes": null
    },
    {
      "id": "2744392",
      "postDate": "04/09/2024 21:45:06",
      "content": "<p>The host's response:</p>\n<blockquote>\n  <p>it doesn't have any special meaning, it's just our internal id to identify attributes</p>\n</blockquote>",
      "rawMarkdown": "The host's response:\n>it doesn't have any special meaning, it's just our internal id to identify attributes",
      "votes": null
    },
    {
      "id": "2759086",
      "postDate": "04/18/2024 13:38:20",
      "content": "<p><a href=\"https://www.kaggle.com/tomasjeline2\" target=\"_blank\">@tomasjeline2</a> <a href=\"https://www.kaggle.com/jetakow\" target=\"_blank\">@jetakow</a> could you possibly share any insights on this? Thank you!</p>",
      "rawMarkdown": "tomasjeline2 @jetakow could you possibly share any insights on this? Thank you!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2743789,
      "author_name": "kunduruanil",
      "author_url": "",
      "post_date": "04/09/2024 16:07:37",
      "content": "<p>Do we have any idea about the postfix for each column like _215A &amp; _384A?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2744392,
          "author_name": "eivolkova",
          "author_url": "",
          "post_date": "04/09/2024 21:45:06",
          "content": "<p>The host's response:</p>\n<blockquote>\n  <p>it doesn't have any special meaning, it's just our internal id to identify attributes</p>\n</blockquote>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2759086,
      "author_name": "eivolkova",
      "author_url": "",
      "post_date": "04/18/2024 13:38:20",
      "content": "<p><a href=\"https://www.kaggle.com/tomasjeline2\" target=\"_blank\">@tomasjeline2</a> <a href=\"https://www.kaggle.com/jetakow\" target=\"_blank\">@jetakow</a> could you possibly share any insights on this? Thank you!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2740727": "I have merged income data from two sources: \"static0\" (column maininc_215A) and \"person1\" (where num_group1=0, column mainoccupationinc_384A). I expected the income values from both sources to be approximately the same. However, the correlation between them is only 0.35. Examining the distribution of differences between the income data from these sources reveals substantial discrepancies, with differences ranging from -200,000 to +200,000 (which is significant given that the income data is capped at 200,000, and the average value is around 50,000). How could this be possible, given that both sources are internal? Which source is more reliable? \n\nHere are the quantiles for the differences:\n\n0.0   -193000.0\n0.1    -28600.0\n0.2    -12000.0\n0.3     -2400.0\n0.4         0.0\n0.5      7000.0\n0.6     14000.0\n0.7     20600.0\n0.8     33000.0\n0.9     52000.0\n1.0    199999.8",
    "2743789": "Do we have any idea about the postfix for each column like _215A & _384A?",
    "2744392": "The host's response:\n>it doesn't have any special meaning, it's just our internal id to identify attributes",
    "2759086": "tomasjeline2 @jetakow could you possibly share any insights on this? Thank you!"
  },
  "source": "meta"
}