{
  "id": 209576,
  "title": "52nd place brief write up [priv 0.802 / GBDTs]",
  "url": "/competitions/riiid-test-answer-prediction/discussion/209576",
  "author_name": "ML_Bear",
  "post_date": "2021-01-08T00:01:50.417000",
  "votes": 42,
  "comment_count": 9,
  "views": 0,
  "content": "<p>First of all, I would like to thank Riiid Labs, partners and Kaggle for hosting this interesting competition. And congratulates to the all participants and especially the winners!</p>\n<p>This was so interesting and well designed competition and I enjoyed for three months. I share a summary of my solution, hoping that this will be of some help to other participants.</p>\n<h2>Final submission summary</h2>\n<ul>\n<li>Weighted averaging of <code>LightGBMx2 + CatBoostx1</code><ul>\n<li>public: 51st (0.801)</li>\n<li>private: 52nd (0.802)</li></ul></li>\n<li>We use about 130 features</li>\n<li>We were trying to make a model using NN(transfomer) in last few weeks, but we couldn't complete it.</li>\n</ul>\n<h2>Key Ideas</h2>\n<h3>Features of the badness of a user_answer</h3>\n<ul>\n<li>+0.006, by far the brightest of the features we created😇.</li>\n<li>The aim of creating the feature<ul>\n<li>A person who chooses a bad answer that most people don't choose is probably bad</li>\n<li>People who can do well will not choose bad choices even if they are wrong</li>\n<li>This idea is probably correct, and the std aggregation below was very effective.</li></ul></li>\n<li>How to make it<ul>\n<li>Use all of train to calculate how many of each answer are selected for each question</li>\n<li>e.g. for content_id=XXXX<ul>\n<li>Answer 1: 9% of users choice</li>\n<li>Answer 2: 5% of users choice</li>\n<li>Answer 3: 1% of users choice</li>\n<li>Answer 4: 85% of users choice</li></ul></li>\n<li>Calculate the percentile for each choice by accumulating the percentage of choices for each question</li>\n<li>Example: In the example above<ul>\n<li>Answer 1: 15% (=1+5+9)</li>\n<li>Answer 2: 6% (=1+5)</li>\n<li>Answer 3: 1% (=1+5)</li>\n<li>Answer 4: 100% (=1+5+9+85)</li></ul></li>\n<li>Aggregation (std, avg, min, etc.) of the percentiles of the last N answers for each user.</li></ul></li>\n</ul>\n<h3>A little devised Word2Vec (by teammate)</h3>\n<ul>\n<li>This looks a bit leaky, but it worked about +0.004, so it was very effective too.</li>\n<li>The aim of creating the feature<ul>\n<li>Vectorizing users based on their past history of correct and incorrect answers, and calculate their approximation to the next question's correct or incorrect answer.</li></ul></li>\n<li>How to make it<ul>\n<li>For each user, line up <code>Question_(correct answer|incorrect answer)</code> (last N questions)</li>\n<li>Vectorize <code>Question_(correct answer|incorrect answer)</code> by word2vec</li>\n<li>Vectorize user by averaging the vector of <code>Question_(correct answer|incorrect answer)</code> of user's last N questions</li>\n<li>Calculate cosine similarity between user's vector and next question's <code>Question_(correct answer)</code> <code>Question_(incorrect answer)</code>.</li></ul></li>\n</ul>\n<h3>Trueskill (by teammate)</h3>\n<ul>\n<li>importance was always at the top, but +0.001<ul>\n<li>may have been conflicting with other features because we create this feature in the last stage</li></ul></li>\n<li>How to make it<ul>\n<li>Using <a href=\"https://trueskill.org/\" target=\"_blank\">trueskill</a> to score the strength of each question and each user.</li>\n<li>From there, calculate the probability that the user will win (answer correctly) the question.</li></ul></li>\n</ul>\n<h3>Weighted counting of correct answers (by teammate)</h3>\n<ul>\n<li>Weight is the reciprocal of the percentage of correct answers, and correct answers are counted with weight.</li>\n</ul>\n<h2>Other Features</h2>\n<p>The following is an excerpt from what worked for us.</p>\n<ul>\n<li>TargetEncodings<ul>\n<li>percentage of questions answered correctly</li>\n<li>percentage of users who answered the question correctly</li>\n<li>and many more.</li></ul></li>\n<li>User logs<ul>\n<li>percentage of correct answers in the last 400 questions</li>\n<li>percentage of correct answers in the same part of the last 400 questions</li>\n<li>and many more.</li></ul></li>\n<li>Lag features of timestamp<ul>\n<li>The time elapsed since the last time user solved the same question.</li>\n<li>Time elapsed since the previous question.</li></ul></li>\n<li>Features processed from timestamp<ul>\n<li>Things that use elapsed time</li>\n<li>elapsed time since the previous question / (all users') average time taken for that task_container</li>\n<li>elapsed time since the previous question / (all users') average time for that task_container (only correct answers are counted)</li>\n<li>I thought the timestamp was the <code>timestamp of answering the question</code> since these were working.</li>\n<li>the lag time simlar to SAINT's paper.</li>\n<li>time elapsed since previous question - time spent on previous question</li></ul></li>\n<li>Simple Word2Vec<ul>\n<li>Line up questions for each user -&gt; vectorize questions by word2vec</li>\n<li>We've made a lot of these, by part, only correct questions, only wrong questions, various windows, etc.</li>\n<li>The more patterns we added, the higher our score became.</li></ul></li>\n</ul>\n<h2>What did not work</h2>\n<p>There are a lot, but I'm going to write down a few that I wondered why they don't work.</p>\n<ul>\n<li>Tags<ul>\n<li>I was able to categorize almost all tags from the distribution of the parts where they appear.</li>\n<li>I found the grammatical tags, intonation tags, etc.</li>\n<li>I tried various ways to put it into the model, but it didn't work at all.</li>\n<li>I tried target encoding and counting the number of correct answers of the user's past tags.</li></ul></li>\n<li>Lecture<ul>\n<li>I tried simple counting, but it did not work at all.</li>\n<li>I gave up on using it as soon as I saw that no one seemed to be making use of it in discussions.</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 1143516,
      "postDate": "2021-01-08T00:01:50.417Z",
      "content": "<p>First of all, I would like to thank Riiid Labs, partners and Kaggle for hosting this interesting competition. And congratulates to the all participants and especially the winners!</p>\n<p>This was so interesting and well designed competition and I enjoyed for three months. I share a summary of my solution, hoping that this will be of some help to other participants.</p>\n<h2>Final submission summary</h2>\n<ul>\n<li>Weighted averaging of <code>LightGBMx2 + CatBoostx1</code><ul>\n<li>public: 51st (0.801)</li>\n<li>private: 52nd (0.802)</li></ul></li>\n<li>We use about 130 features</li>\n<li>We were trying to make a model using NN(transfomer) in last few weeks, but we couldn't complete it.</li>\n</ul>\n<h2>Key Ideas</h2>\n<h3>Features of the badness of a user_answer</h3>\n<ul>\n<li>+0.006, by far the brightest of the features we created😇.</li>\n<li>The aim of creating the feature<ul>\n<li>A person who chooses a bad answer that most people don't choose is probably bad</li>\n<li>People who can do well will not choose bad choices even if they are wrong</li>\n<li>This idea is probably correct, and the std aggregation below was very effective.</li></ul></li>\n<li>How to make it<ul>\n<li>Use all of train to calculate how many of each answer are selected for each question</li>\n<li>e.g. for content_id=XXXX<ul>\n<li>Answer 1: 9% of users choice</li>\n<li>Answer 2: 5% of users choice</li>\n<li>Answer 3: 1% of users choice</li>\n<li>Answer 4: 85% of users choice</li></ul></li>\n<li>Calculate the percentile for each choice by accumulating the percentage of choices for each question</li>\n<li>Example: In the example above<ul>\n<li>Answer 1: 15% (=1+5+9)</li>\n<li>Answer 2: 6% (=1+5)</li>\n<li>Answer 3: 1% (=1+5)</li>\n<li>Answer 4: 100% (=1+5+9+85)</li></ul></li>\n<li>Aggregation (std, avg, min, etc.) of the percentiles of the last N answers for each user.</li></ul></li>\n</ul>\n<h3>A little devised Word2Vec (by teammate)</h3>\n<ul>\n<li>This looks a bit leaky, but it worked about +0.004, so it was very effective too.</li>\n<li>The aim of creating the feature<ul>\n<li>Vectorizing users based on their past history of correct and incorrect answers, and calculate their approximation to the next question's correct or incorrect answer.</li></ul></li>\n<li>How to make it<ul>\n<li>For each user, line up <code>Question_(correct answer|incorrect answer)</code> (last N questions)</li>\n<li>Vectorize <code>Question_(correct answer|incorrect answer)</code> by word2vec</li>\n<li>Vectorize user by averaging the vector of <code>Question_(correct answer|incorrect answer)</code> of user's last N questions</li>\n<li>Calculate cosine similarity between user's vector and next question's <code>Question_(correct answer)</code> <code>Question_(incorrect answer)</code>.</li></ul></li>\n</ul>\n<h3>Trueskill (by teammate)</h3>\n<ul>\n<li>importance was always at the top, but +0.001<ul>\n<li>may have been conflicting with other features because we create this feature in the last stage</li></ul></li>\n<li>How to make it<ul>\n<li>Using <a href=\"https://trueskill.org/\" target=\"_blank\">trueskill</a> to score the strength of each question and each user.</li>\n<li>From there, calculate the probability that the user will win (answer correctly) the question.</li></ul></li>\n</ul>\n<h3>Weighted counting of correct answers (by teammate)</h3>\n<ul>\n<li>Weight is the reciprocal of the percentage of correct answers, and correct answers are counted with weight.</li>\n</ul>\n<h2>Other Features</h2>\n<p>The following is an excerpt from what worked for us.</p>\n<ul>\n<li>TargetEncodings<ul>\n<li>percentage of questions answered correctly</li>\n<li>percentage of users who answered the question correctly</li>\n<li>and many more.</li></ul></li>\n<li>User logs<ul>\n<li>percentage of correct answers in the last 400 questions</li>\n<li>percentage of correct answers in the same part of the last 400 questions</li>\n<li>and many more.</li></ul></li>\n<li>Lag features of timestamp<ul>\n<li>The time elapsed since the last time user solved the same question.</li>\n<li>Time elapsed since the previous question.</li></ul></li>\n<li>Features processed from timestamp<ul>\n<li>Things that use elapsed time</li>\n<li>elapsed time since the previous question / (all users') average time taken for that task_container</li>\n<li>elapsed time since the previous question / (all users') average time for that task_container (only correct answers are counted)</li>\n<li>I thought the timestamp was the <code>timestamp of answering the question</code> since these were working.</li>\n<li>the lag time simlar to SAINT's paper.</li>\n<li>time elapsed since previous question - time spent on previous question</li></ul></li>\n<li>Simple Word2Vec<ul>\n<li>Line up questions for each user -&gt; vectorize questions by word2vec</li>\n<li>We've made a lot of these, by part, only correct questions, only wrong questions, various windows, etc.</li>\n<li>The more patterns we added, the higher our score became.</li></ul></li>\n</ul>\n<h2>What did not work</h2>\n<p>There are a lot, but I'm going to write down a few that I wondered why they don't work.</p>\n<ul>\n<li>Tags<ul>\n<li>I was able to categorize almost all tags from the distribution of the parts where they appear.</li>\n<li>I found the grammatical tags, intonation tags, etc.</li>\n<li>I tried various ways to put it into the model, but it didn't work at all.</li>\n<li>I tried target encoding and counting the number of correct answers of the user's past tags.</li></ul></li>\n<li>Lecture<ul>\n<li>I tried simple counting, but it did not work at all.</li>\n<li>I gave up on using it as soon as I saw that no one seemed to be making use of it in discussions.</li></ul></li>\n</ul>",
      "rawMarkdown": "First of all, I would like to thank Riiid Labs, partners and Kaggle for hosting this interesting competition. And congratulates to the all participants and especially the winners!\n\nThis was so interesting and well designed competition and I enjoyed for three months. I share a summary of my solution, hoping that this will be of some help to other participants.\n\n## Final submission summary\n* Weighted averaging of `LightGBMx2 + CatBoostx1`\n  * public: 51st (0.801)\n  * private: 52nd (0.802)\n* We use about 130 features\n* We were trying to make a model using NN(transfomer) in last few weeks, but we couldn't complete it.\n\n## Key Ideas\n### Features of the badness of a user_answer\n* +0.006, by far the brightest of the features we created😇.\n* The aim of creating the feature\n  * A person who chooses a bad answer that most people don't choose is probably bad\n  * People who can do well will not choose bad choices even if they are wrong\n  * This idea is probably correct, and the std aggregation below was very effective.\n* How to make it\n  * Use all of train to calculate how many of each answer are selected for each question\n    * e.g. for content_id=XXXX\n      * Answer 1: 9% of users choice\n      * Answer 2: 5% of users choice\n      * Answer 3: 1% of users choice\n      * Answer 4: 85% of users choice\n  * Calculate the percentile for each choice by accumulating the percentage of choices for each question\n    * Example: In the example above\n      * Answer 1: 15% (=1+5+9)\n      * Answer 2: 6% (=1+5)\n      * Answer 3: 1% (=1+5)\n      * Answer 4: 100% (=1+5+9+85)\n  * Aggregation (std, avg, min, etc.) of the percentiles of the last N answers for each user.\n\n### A little devised Word2Vec (by teammate)\n* This looks a bit leaky, but it worked about +0.004, so it was very effective too.\n* The aim of creating the feature\n  * Vectorizing users based on their past history of correct and incorrect answers, and calculate their approximation to the next question's correct or incorrect answer.\n* How to make it\n  * For each user, line up `Question_(correct answer|incorrect answer)` (last N questions)\n  * Vectorize `Question_(correct answer|incorrect answer)` by word2vec\n  * Vectorize user by averaging the vector of `Question_(correct answer|incorrect answer)` of user's last N questions\n  * Calculate cosine similarity between user's vector and next question's `Question_(correct answer)` `Question_(incorrect answer)`.\n\n### Trueskill (by teammate)\n* importance was always at the top, but +0.001\n  * may have been conflicting with other features because we create this feature in the last stage\n* How to make it\n  * Using [trueskill](https://trueskill.org/) to score the strength of each question and each user.\n  * From there, calculate the probability that the user will win (answer correctly) the question.\n\n### Weighted counting of correct answers (by teammate)\n* Weight is the reciprocal of the percentage of correct answers, and correct answers are counted with weight.\n\n## Other Features\nThe following is an excerpt from what worked for us.\n\n* TargetEncodings\n  * percentage of questions answered correctly\n  * percentage of users who answered the question correctly\n  * and many more.\n* User logs\n  * percentage of correct answers in the last 400 questions\n  * percentage of correct answers in the same part of the last 400 questions\n  * and many more.\n* Lag features of timestamp\n  * The time elapsed since the last time user solved the same question.\n  * Time elapsed since the previous question.\n* Features processed from timestamp\n  * Things that use elapsed time\n    * elapsed time since the previous question / (all users') average time taken for that task_container\n    * elapsed time since the previous question / (all users') average time for that task_container (only correct answers are counted)\n    * I thought the timestamp was the `timestamp of answering the question` since these were working.\n  * the lag time simlar to SAINT's paper.\n    * time elapsed since previous question - time spent on previous question\n* Simple Word2Vec\n  * Line up questions for each user -> vectorize questions by word2vec\n  * We've made a lot of these, by part, only correct questions, only wrong questions, various windows, etc.\n    * The more patterns we added, the higher our score became.\n\n## What did not work\nThere are a lot, but I'm going to write down a few that I wondered why they don't work.\n\n* Tags\n  * I was able to categorize almost all tags from the distribution of the parts where they appear.\n    * I found the grammatical tags, intonation tags, etc.\n  * I tried various ways to put it into the model, but it didn't work at all.\n    * I tried target encoding and counting the number of correct answers of the user's past tags.\n* Lecture\n  * I tried simple counting, but it did not work at all.\n  * I gave up on using it as soon as I saw that no one seemed to be making use of it in discussions.\n",
      "votes": 42
    },
    {
      "id": 1143548,
      "postDate": "2021-01-08T00:21:01.147Z",
      "content": "<p>Good score for boosting models, thanks for sharing solution <a href=\"https://www.kaggle.com/naotaka1128\" target=\"_blank\">@naotaka1128</a> </p>",
      "rawMarkdown": "Good score for boosting models, thanks for sharing solution @naotaka1128 ",
      "votes": 1
    },
    {
      "id": 1599786,
      "postDate": "2021-11-29T20:08:39.720Z",
      "content": "<p>Thank you for putting this together and sharing what worked and what not <a href=\"https://www.kaggle.com/naotaka1128\" target=\"_blank\">@naotaka1128</a> </p>",
      "rawMarkdown": "Thank you for putting this together and sharing what worked and what not @naotaka1128 ",
      "votes": -1
    },
    {
      "id": 1144015,
      "postDate": "2021-01-08T07:21:28.053Z",
      "content": "<p>Really like the idea of trueskill feature. That is good.<br>\nHow did you come up with it? <br>\nAnd In the inference, you have created a dict and used the train data to make these features if I understand correctly.</p>",
      "rawMarkdown": "Really like the idea of trueskill feature. That is good.\nHow did you come up with it? \nAnd In the inference, you have created a dict and used the train data to make these features if I understand correctly."
    },
    {
      "id": 1143759,
      "postDate": "2021-01-08T03:43:41.590Z",
      "content": "<p>Thank you for your sharing wonderful ideas.<br>\nOur team also came up for using \"badness of answers\", but failed to implement the feature…</p>",
      "rawMarkdown": "Thank you for your sharing wonderful ideas.\nOur team also came up for using \"badness of answers\", but failed to implement the feature..."
    },
    {
      "id": 1143604,
      "postDate": "2021-01-08T01:02:30.150Z",
      "content": "<p>\"badness of a user_answer\" That feature is mind blowing!</p>",
      "rawMarkdown": "\"badness of a user_answer\" That feature is mind blowing!"
    },
    {
      "id": 1143541,
      "postDate": "2021-01-08T00:17:48.860Z",
      "content": "<p>Thank you for sharing! The idea of rating users and questions by trueskill sounds interesting.</p>",
      "rawMarkdown": "Thank you for sharing! The idea of rating users and questions by trueskill sounds interesting."
    },
    {
      "id": 1143535,
      "postDate": "2021-01-08T00:15:31.047Z",
      "content": "<p>Wow love the creativeness in badness of answers.  My LGBM scored 796 in private so I guess the ~006 improvement by this feature is what my model lacked.   Congrats! </p>",
      "rawMarkdown": "Wow love the creativeness in badness of answers.  My LGBM scored 796 in private so I guess the ~006 improvement by this feature is what my model lacked.   Congrats! "
    },
    {
      "id": 1583340,
      "postDate": "2021-11-15T18:32:03.630Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 1143607,
      "postDate": "2021-01-08T01:04:57.817Z",
      "content": "<p>thanks for sharing,Brilliant ideas</p>",
      "rawMarkdown": "thanks for sharing,Brilliant ideas"
    }
  ],
  "comments": [
    {
      "id": 1143548,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2021-01-08T00:21:01.147000",
      "content": "<p>Good score for boosting models, thanks for sharing solution <a href=\"https://www.kaggle.com/naotaka1128\" target=\"_blank\">@naotaka1128</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1599786,
      "author_name": "George Zoto",
      "author_url": "",
      "post_date": "2021-11-29T20:08:39.720000",
      "content": "<p>Thank you for putting this together and sharing what worked and what not <a href=\"https://www.kaggle.com/naotaka1128\" target=\"_blank\">@naotaka1128</a> </p>",
      "votes": -1,
      "replies": []
    },
    {
      "id": 1144015,
      "author_name": "RoBi",
      "author_url": "",
      "post_date": "2021-01-08T07:21:28.053000",
      "content": "<p>Really like the idea of trueskill feature. That is good.<br>\nHow did you come up with it? <br>\nAnd In the inference, you have created a dict and used the train data to make these features if I understand correctly.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143759,
      "author_name": "naoki56",
      "author_url": "",
      "post_date": "2021-01-08T03:43:41.590000",
      "content": "<p>Thank you for your sharing wonderful ideas.<br>\nOur team also came up for using \"badness of answers\", but failed to implement the feature…</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143604,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2021-01-08T01:02:30.150000",
      "content": "<p>\"badness of a user_answer\" That feature is mind blowing!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143541,
      "author_name": "u++",
      "author_url": "",
      "post_date": "2021-01-08T00:17:48.860000",
      "content": "<p>Thank you for sharing! The idea of rating users and questions by trueskill sounds interesting.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143535,
      "author_name": "Tonghui Li",
      "author_url": "",
      "post_date": "2021-01-08T00:15:31.047000",
      "content": "<p>Wow love the creativeness in badness of answers.  My LGBM scored 796 in private so I guess the ~006 improvement by this feature is what my model lacked.   Congrats! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1583340,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-11-15T18:32:03.630000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143607,
      "author_name": "qiaqia",
      "author_url": "",
      "post_date": "2021-01-08T01:04:57.817000",
      "content": "<p>thanks for sharing,Brilliant ideas</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1143516": "First of all, I would like to thank Riiid Labs, partners and Kaggle for hosting this interesting competition. And congratulates to the all participants and especially the winners!\n\nThis was so interesting and well designed competition and I enjoyed for three months. I share a summary of my solution, hoping that this will be of some help to other participants.\n\n## Final submission summary\n* Weighted averaging of `LightGBMx2 + CatBoostx1`\n  * public: 51st (0.801)\n  * private: 52nd (0.802)\n* We use about 130 features\n* We were trying to make a model using NN(transfomer) in last few weeks, but we couldn't complete it.\n\n## Key Ideas\n### Features of the badness of a user_answer\n* +0.006, by far the brightest of the features we created😇.\n* The aim of creating the feature\n  * A person who chooses a bad answer that most people don't choose is probably bad\n  * People who can do well will not choose bad choices even if they are wrong\n  * This idea is probably correct, and the std aggregation below was very effective.\n* How to make it\n  * Use all of train to calculate how many of each answer are selected for each question\n    * e.g. for content_id=XXXX\n      * Answer 1: 9% of users choice\n      * Answer 2: 5% of users choice\n      * Answer 3: 1% of users choice\n      * Answer 4: 85% of users choice\n  * Calculate the percentile for each choice by accumulating the percentage of choices for each question\n    * Example: In the example above\n      * Answer 1: 15% (=1+5+9)\n      * Answer 2: 6% (=1+5)\n      * Answer 3: 1% (=1+5)\n      * Answer 4: 100% (=1+5+9+85)\n  * Aggregation (std, avg, min, etc.) of the percentiles of the last N answers for each user.\n\n### A little devised Word2Vec (by teammate)\n* This looks a bit leaky, but it worked about +0.004, so it was very effective too.\n* The aim of creating the feature\n  * Vectorizing users based on their past history of correct and incorrect answers, and calculate their approximation to the next question's correct or incorrect answer.\n* How to make it\n  * For each user, line up `Question_(correct answer|incorrect answer)` (last N questions)\n  * Vectorize `Question_(correct answer|incorrect answer)` by word2vec\n  * Vectorize user by averaging the vector of `Question_(correct answer|incorrect answer)` of user's last N questions\n  * Calculate cosine similarity between user's vector and next question's `Question_(correct answer)` `Question_(incorrect answer)`.\n\n### Trueskill (by teammate)\n* importance was always at the top, but +0.001\n  * may have been conflicting with other features because we create this feature in the last stage\n* How to make it\n  * Using [trueskill](https://trueskill.org/) to score the strength of each question and each user.\n  * From there, calculate the probability that the user will win (answer correctly) the question.\n\n### Weighted counting of correct answers (by teammate)\n* Weight is the reciprocal of the percentage of correct answers, and correct answers are counted with weight.\n\n## Other Features\nThe following is an excerpt from what worked for us.\n\n* TargetEncodings\n  * percentage of questions answered correctly\n  * percentage of users who answered the question correctly\n  * and many more.\n* User logs\n  * percentage of correct answers in the last 400 questions\n  * percentage of correct answers in the same part of the last 400 questions\n  * and many more.\n* Lag features of timestamp\n  * The time elapsed since the last time user solved the same question.\n  * Time elapsed since the previous question.\n* Features processed from timestamp\n  * Things that use elapsed time\n    * elapsed time since the previous question / (all users') average time taken for that task_container\n    * elapsed time since the previous question / (all users') average time for that task_container (only correct answers are counted)\n    * I thought the timestamp was the `timestamp of answering the question` since these were working.\n  * the lag time simlar to SAINT's paper.\n    * time elapsed since previous question - time spent on previous question\n* Simple Word2Vec\n  * Line up questions for each user -> vectorize questions by word2vec\n  * We've made a lot of these, by part, only correct questions, only wrong questions, various windows, etc.\n    * The more patterns we added, the higher our score became.\n\n## What did not work\nThere are a lot, but I'm going to write down a few that I wondered why they don't work.\n\n* Tags\n  * I was able to categorize almost all tags from the distribution of the parts where they appear.\n    * I found the grammatical tags, intonation tags, etc.\n  * I tried various ways to put it into the model, but it didn't work at all.\n    * I tried target encoding and counting the number of correct answers of the user's past tags.\n* Lecture\n  * I tried simple counting, but it did not work at all.\n  * I gave up on using it as soon as I saw that no one seemed to be making use of it in discussions.\n",
    "1143548": "Good score for boosting models, thanks for sharing solution @naotaka1128 ",
    "1599786": "Thank you for putting this together and sharing what worked and what not @naotaka1128 ",
    "1144015": "Really like the idea of trueskill feature. That is good.\nHow did you come up with it? \nAnd In the inference, you have created a dict and used the train data to make these features if I understand correctly.",
    "1143759": "Thank you for your sharing wonderful ideas.\nOur team also came up for using \"badness of answers\", but failed to implement the feature...",
    "1143604": "\"badness of a user_answer\" That feature is mind blowing!",
    "1143541": "Thank you for sharing! The idea of rating users and questions by trueskill sounds interesting.",
    "1143535": "Wow love the creativeness in badness of answers.  My LGBM scored 796 in private so I guess the ~006 improvement by this feature is what my model lacked.   Congrats! ",
    "1583340": "",
    "1143607": "thanks for sharing,Brilliant ideas"
  }
}