{
  "id": 208886,
  "title": "how to select features",
  "url": "/competitions/riiid-test-answer-prediction/discussion/208886",
  "author_name": "qiaqia",
  "post_date": "2021-01-05T12:51:29.913000",
  "votes": 1,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I have about 60 features now,and I try to reduce to about 30 features,however it seems worse  when I drop some features.<br>\nI try to select features by importance gain, null importance and shap .but it didn't work well.<br>\nDo you have any good suggestions for feature selection?<br>\nLooking forward to reply👀👀</p>",
  "messages": [
    {
      "id": 1139513,
      "postDate": "2021-01-05T12:51:29.913Z",
      "content": "<p>I have about 60 features now,and I try to reduce to about 30 features,however it seems worse  when I drop some features.<br>\nI try to select features by importance gain, null importance and shap .but it didn't work well.<br>\nDo you have any good suggestions for feature selection?<br>\nLooking forward to reply👀👀</p>",
      "rawMarkdown": "I have about 60 features now,and I try to reduce to about 30 features,however it seems worse  when I drop some features.\nI try to select features by importance gain, null importance and shap .but it didn't work well.\nDo you have any good suggestions for feature selection?\nLooking forward to reply👀👀",
      "votes": 1
    },
    {
      "id": 1141540,
      "postDate": "2021-01-06T18:47:16.907Z",
      "content": "<p>You have tried PCA?</p>",
      "rawMarkdown": "You have tried PCA?"
    },
    {
      "id": 1139531,
      "postDate": "2021-01-05T13:12:24.343Z",
      "content": "<p>Are you checking the feature importance for your validation set only ?</p>",
      "rawMarkdown": "Are you checking the feature importance for your validation set only ?",
      "replies": [
        {
          "id": 1139535,
          "postDate": "2021-01-05T13:18:06.793Z",
          "content": "<p>yes, Should I use the training set to check feature importance? now I train 26M and valid 2.4M.To save time, I have not calculated the auc for the training set.</p>",
          "rawMarkdown": "yes, Should I use the training set to check feature importance? now I train 26M and valid 2.4M.To save time, I have not calculated the auc for the training set."
        },
        {
          "id": 1139549,
          "postDate": "2021-01-05T13:29:33.427Z",
          "content": "<p>No you shall just use the test set ! <br>\nIn my case I am calculating the SHAP values on a subsamples of 100 000 rows, overwise it takes too long to calculate… </p>\n<p>I then display them using:</p>\n<blockquote>\n  <p>shap.summary_plot(shap_values[0], sub, plot_type=\"bar\", max_display = n_features)</p>\n</blockquote>\n<p>And try to rerun the model without the lowest ones. I probably lost a bit of accuracy in the change, but at one point it was really hard for me to run/train with the number of features I had ! </p>",
          "rawMarkdown": "No you shall just use the test set ! \nIn my case I am calculating the SHAP values on a subsamples of 100 000 rows, overwise it takes too long to calculate... \n\nI then display them using:\n> shap.summary_plot(shap_values[0], sub, plot_type=\"bar\", max_display = n_features)\n\nAnd try to rerun the model without the lowest ones. I probably lost a bit of accuracy in the change, but at one point it was really hard for me to run/train with the number of features I had ! "
        },
        {
          "id": 1139570,
          "postDate": "2021-01-05T13:52:36.797Z",
          "content": "<p>I did the same thing, but when I removed the lowest features, the score of the validation set also decreased,These 3 methods all lowered my score when I removed about 20 features. I am a bit confused</p>",
          "rawMarkdown": "I did the same thing, but when I removed the lowest features, the score of the validation set also decreased,These 3 methods all lowered my score when I removed about 20 features. I am a bit confused"
        },
        {
          "id": 1140196,
          "postDate": "2021-01-05T21:02:12.320Z",
          "content": "<p>Then probably your features have a predictive power</p>",
          "rawMarkdown": "Then probably your features have a predictive power",
          "votes": 1
        },
        {
          "id": 1141219,
          "postDate": "2021-01-06T15:10:29.757Z",
          "rawMarkdown": "",
          "votes": -2,
          "isDeleted": true
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1141540,
      "author_name": "Vadim Irtlach",
      "author_url": "",
      "post_date": "2021-01-06T18:47:16.907000",
      "content": "<p>You have tried PCA?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1139531,
      "author_name": "Jacky",
      "author_url": "",
      "post_date": "2021-01-05T13:12:24.343000",
      "content": "<p>Are you checking the feature importance for your validation set only ?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1139535,
          "author_name": "qiaqia",
          "author_url": "",
          "post_date": "2021-01-05T13:18:06.793000",
          "content": "<p>yes, Should I use the training set to check feature importance? now I train 26M and valid 2.4M.To save time, I have not calculated the auc for the training set.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1139549,
          "author_name": "Jacky",
          "author_url": "",
          "post_date": "2021-01-05T13:29:33.427000",
          "content": "<p>No you shall just use the test set ! <br>\nIn my case I am calculating the SHAP values on a subsamples of 100 000 rows, overwise it takes too long to calculate… </p>\n<p>I then display them using:</p>\n<blockquote>\n  <p>shap.summary_plot(shap_values[0], sub, plot_type=\"bar\", max_display = n_features)</p>\n</blockquote>\n<p>And try to rerun the model without the lowest ones. I probably lost a bit of accuracy in the change, but at one point it was really hard for me to run/train with the number of features I had ! </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1139570,
          "author_name": "qiaqia",
          "author_url": "",
          "post_date": "2021-01-05T13:52:36.797000",
          "content": "<p>I did the same thing, but when I removed the lowest features, the score of the validation set also decreased,These 3 methods all lowered my score when I removed about 20 features. I am a bit confused</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1140196,
          "author_name": "Ioannis M",
          "author_url": "",
          "post_date": "2021-01-05T21:02:12.320000",
          "content": "<p>Then probably your features have a predictive power</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1141219,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-01-06T15:10:29.757000",
          "content": "",
          "votes": -2,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1139513": "I have about 60 features now,and I try to reduce to about 30 features,however it seems worse  when I drop some features.\nI try to select features by importance gain, null importance and shap .but it didn't work well.\nDo you have any good suggestions for feature selection?\nLooking forward to reply👀👀",
    "1141540": "You have tried PCA?",
    "1139531": "Are you checking the feature importance for your validation set only ?"
  }
}