{
  "id": 327161,
  "title": "Unique number of categorical features",
  "url": "/competitions/amex-default-prediction/discussion/327161",
  "author_name": "",
  "post_date": "2022-05-25T22:12:37.042016800Z",
  "votes": 29,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Categorical features are defined as follows.<br>\n<code>['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']</code></p>\n<p>There seems to be a difference in the number of categories between D_64, D_66 and D_68 in train_data.csv and test_data.csv.</p>\n<pre><code>===========B_30===========\ntrain nunique: 4\ntest nunique: 4\n===========B_38===========\ntrain nunique: 8\ntest nunique: 8\n===========D_114===========\ntrain nunique: 3\ntest nunique: 3\n===========D_116===========\ntrain nunique: 3\ntest nunique: 3\n===========D_117===========\ntrain nunique: 8\ntest nunique: 8\n===========D_120===========\ntrain nunique: 3\ntest nunique: 3\n===========D_126===========\ntrain nunique: 4\ntest nunique: 4\n===========D_63===========\ntrain nunique: 6\ntest nunique: 6\n\n\n===========D_64===========\ntrain nunique: 5\ntest nunique: 4\n===========D_66===========\ntrain nunique: 3\ntest nunique: 2\n===========D_68===========\ntrain nunique: 8\ntest nunique: 7\n</code></pre>\n<p>Check D_64 and I will find the following differences.</p>\n<pre><code>===========D_64===========\ntrain value counts:\nO       2913244\nU       1523448\nR        840112\n&lt;NA&gt;     217442\n-1        37205\n\ntest value counts:\nO       5993483\nU       2952811\nR       2039630\n&lt;NA&gt;     377838\n</code></pre>\n<p>Should \"-1\" and \"Nan\" be treated as separate categories?</p>",
  "messages": [
    {
      "id": "1801568",
      "postDate": "05/25/2022 22:12:37",
      "content": "<p>Categorical features are defined as follows.<br>\n<code>['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']</code></p>\n<p>There seems to be a difference in the number of categories between D_64, D_66 and D_68 in train_data.csv and test_data.csv.</p>\n<pre><code>===========B_30===========\ntrain nunique: 4\ntest nunique: 4\n===========B_38===========\ntrain nunique: 8\ntest nunique: 8\n===========D_114===========\ntrain nunique: 3\ntest nunique: 3\n===========D_116===========\ntrain nunique: 3\ntest nunique: 3\n===========D_117===========\ntrain nunique: 8\ntest nunique: 8\n===========D_120===========\ntrain nunique: 3\ntest nunique: 3\n===========D_126===========\ntrain nunique: 4\ntest nunique: 4\n===========D_63===========\ntrain nunique: 6\ntest nunique: 6\n\n\n===========D_64===========\ntrain nunique: 5\ntest nunique: 4\n===========D_66===========\ntrain nunique: 3\ntest nunique: 2\n===========D_68===========\ntrain nunique: 8\ntest nunique: 7\n</code></pre>\n<p>Check D_64 and I will find the following differences.</p>\n<pre><code>===========D_64===========\ntrain value counts:\nO       2913244\nU       1523448\nR        840112\n&lt;NA&gt;     217442\n-1        37205\n\ntest value counts:\nO       5993483\nU       2952811\nR       2039630\n&lt;NA&gt;     377838\n</code></pre>\n<p>Should \"-1\" and \"Nan\" be treated as separate categories?</p>",
      "rawMarkdown": "Categorical features are defined as follows.\n`['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']`\n\nThere seems to be a difference in the number of categories between D_64, D_66 and D_68 in train_data.csv and test_data.csv.\n\n```\n===========B_30===========\ntrain nunique: 4\ntest nunique: 4\n===========B_38===========\ntrain nunique: 8\ntest nunique: 8\n===========D_114===========\ntrain nunique: 3\ntest nunique: 3\n===========D_116===========\ntrain nunique: 3\ntest nunique: 3\n===========D_117===========\ntrain nunique: 8\ntest nunique: 8\n===========D_120===========\ntrain nunique: 3\ntest nunique: 3\n===========D_126===========\ntrain nunique: 4\ntest nunique: 4\n===========D_63===========\ntrain nunique: 6\ntest nunique: 6\n\n\n===========D_64===========\ntrain nunique: 5\ntest nunique: 4\n===========D_66===========\ntrain nunique: 3\ntest nunique: 2\n===========D_68===========\ntrain nunique: 8\ntest nunique: 7\n```\n\n\nCheck D_64 and I will find the following differences.\n\n```\n===========D_64===========\ntrain value counts:\nO       2913244\nU       1523448\nR        840112\n<NA>     217442\n-1        37205\n\ntest value counts:\nO       5993483\nU       2952811\nR       2039630\n<NA>     377838\n```\n\nShould \"-1\" and \"Nan\" be treated as separate categories?",
      "votes": null
    },
    {
      "id": "1806143",
      "postDate": "05/30/2022 20:21:45",
      "content": "<p>Col: D_64; value train: {nan, '-1', 'U', 'O', 'R'}; test: {nan, 'R', 'O', 'U'}<br>\nCol: D_66; value train: {0, 1, }; test: {1, }<br>\nCol: D_68; value train: {0, 1, 2, 3, 4, 5, 6, }; test: {1, 2, 3, 4, 5, 6, }</p>",
      "rawMarkdown": "Col: D_64; value train: {nan, '-1', 'U', 'O', 'R'}; test: {nan, 'R', 'O', 'U'}\nCol: D_66; value train: {0, 1, <NA>}; test: {1, <NA>}\nCol: D_68; value train: {0, 1, 2, 3, 4, 5, 6, <NA>}; test: {1, 2, 3, 4, 5, 6, <NA>}",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1806143,
      "author_name": "stenford23",
      "author_url": "",
      "post_date": "05/30/2022 20:21:45",
      "content": "<p>Col: D_64; value train: {nan, '-1', 'U', 'O', 'R'}; test: {nan, 'R', 'O', 'U'}<br>\nCol: D_66; value train: {0, 1, }; test: {1, }<br>\nCol: D_68; value train: {0, 1, 2, 3, 4, 5, 6, }; test: {1, 2, 3, 4, 5, 6, }</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1801568": "Categorical features are defined as follows.\n`['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']`\n\nThere seems to be a difference in the number of categories between D_64, D_66 and D_68 in train_data.csv and test_data.csv.\n\n```\n===========B_30===========\ntrain nunique: 4\ntest nunique: 4\n===========B_38===========\ntrain nunique: 8\ntest nunique: 8\n===========D_114===========\ntrain nunique: 3\ntest nunique: 3\n===========D_116===========\ntrain nunique: 3\ntest nunique: 3\n===========D_117===========\ntrain nunique: 8\ntest nunique: 8\n===========D_120===========\ntrain nunique: 3\ntest nunique: 3\n===========D_126===========\ntrain nunique: 4\ntest nunique: 4\n===========D_63===========\ntrain nunique: 6\ntest nunique: 6\n\n\n===========D_64===========\ntrain nunique: 5\ntest nunique: 4\n===========D_66===========\ntrain nunique: 3\ntest nunique: 2\n===========D_68===========\ntrain nunique: 8\ntest nunique: 7\n```\n\n\nCheck D_64 and I will find the following differences.\n\n```\n===========D_64===========\ntrain value counts:\nO       2913244\nU       1523448\nR        840112\n<NA>     217442\n-1        37205\n\ntest value counts:\nO       5993483\nU       2952811\nR       2039630\n<NA>     377838\n```\n\nShould \"-1\" and \"Nan\" be treated as separate categories?",
    "1806143": "Col: D_64; value train: {nan, '-1', 'U', 'O', 'R'}; test: {nan, 'R', 'O', 'U'}\nCol: D_66; value train: {0, 1, <NA>}; test: {1, <NA>}\nCol: D_68; value train: {0, 1, 2, 3, 4, 5, 6, <NA>}; test: {1, 2, 3, 4, 5, 6, <NA>}"
  },
  "source": "meta"
}