Also available at https://github.com/yoonseok312/riiid-answer-correctness-prediction with codes.
\nThis is a solution write up for our model, which is an ensemble between a single Light Gradient Boosted Machine model, and a single Encoder-Decoder based Transformer model. It was our first time competing in a Kaggle competition and none of us had previous AI/ML/Stats experience so we learned a lot throughout this competition. Should you have any questions, feel free to contact me at yoonseok@berkeley.edu.
\nts_delta: the gap between timestamp of current content with previous content of the same user.
\ntask_container_id
\nprior_question_elapsed_time: in second, rounded
\nprior_question_had_explanation
\npart
\nnum_tag: number of tags in that question
\nu_chance: the average correctness of the user until current time
\nu_attempts: number of content the user have done
\nu_attempt_c: number of times the user interacted with the specific content in the past (only counting from >1 interactions due to memory)
\nc_chance: the average correctness of the question until current time
\nc_attempts: number of encounter of that question (all user)
\nu_part_chance: the average correctness of the user doing the same part as the question
\nu_part_attempts: number of question of the same part the user have done
\nu_skill_chance: the average correctness of the user doing the same skill as the question (part < 5: listening, part >= 5: reading)
\nu_skill_attempts: number of question of the same skill the user have done
\nt_chance: the average correctness of the user of questions with specific tag until current time
\nt_attempts: user's number of encounter of that tag
\ntotal_explained: number of times explanation was provided to the user until current time (all contents)
\n10_recent_correctness: user correctness of the most recent questions (up to 10)
\n10_recent_mean_gap: mean ts_delta of the most recent questions (up to 10)
\nbundle_elapsed: the mean elapsed time of the bundle, up until the abs time.
\nmean_elapsed: the mean elapsed time of the user until now.
\nprev_t1: tag of the last question
\nprev_cor: correctness of the last question
\ntrueskill_possibility: possibility of the user 'beating' the question (getting the question correct) based on trueskill
\nmu: mu value (mean of trueskill ratings) of user
\nsigma: sigma value (standard deviation of trueskill ratings) of user
\nColumns with NaN value was filled with -1.
We used less than 30 features, but considering that most of the single LGBM models above 0.79 AUC used 40+ features, we did a decent work on focusing on imoportant features.
\nLB score: AUC 0.789
\nNumber of epochs: 6650 (around 15 hours of training in total)
Added below layers with positional encoding.
\n1) Excercise Related
\nmin_delta: minute difference from between this question and the previous. Cap at 1443 (1 day)
\nday_delta: day difference from between this question and the previous. Cap at 30
\nmonth_delta: month difference from between this question and the previous. Cap at 9
\ntid: task container id
\nis_with: if the question is presented with another question. Usually have the same task container
\nc_part: part, one hot encode and denote skill (listening, reading, part1,2,…)
\ntag1…6: tags of question (t1 to t6 are the tag of one question, t1 being the most important tag.) Above embeddings or Dense layer concatenated.
2) Content id (cid)
\nDense layer
Added below layers with positional encoding.
\n1) Response Related
\nprev_answered_correct: correctness of previous answer.
\nprior_elapsed: prior elapsed tiem
\nprior_explained: prior has explanation Above embeddings or Dense layer concatenated.
2) Answered Correctly
\nConcatenate Lecture related Embeddings/Dense
\nnum_lect: number of lecture the user have seen
\nlec_type: 1 hot encode of most recent lecture type, (llecty1, 2…)
\nlec_h_past: time since most recent lecture Above embeddings or Dense layer concatenated.
WINDOW_SIZE: 100
\nEMBED_DIM: 256
\nNUM_HEADS: 16
Use first 80% of data as train set and last 20% as validation set.
\nAUC 0.786
\nSAINT model has plenty of room for improvement, but as 1 epoch took more than 10 hours to train we decided to focus on improving LGBM.
Ensembled a single LGBM model and a Transformer model in 0.55 (LGBM) / 0.45 ratio.
\nAUC: 0.793
Ensembled two LGBM and a Transformer model. 2nd LGBM was same as the first LGBM but except features related to Trueskill. When 2 models out of three models predicted that the user is likely to answer correctly, we used the max value among the 3 predictions. When 2 models out of three models predicted that the user is likely to answer wronly, we used the min value among the 3 predictions. For remaining cases, we mixed three models in 0.4 (Transformer) / 0.45 (First LGBM) / 0.15 (Second LGBM) ratio.
\nAUC: 0.793 (slightly higher than the 1st Inference)
Our biggest mistake was thinking that all the feature engineering, training, and inferencing process must be done in the Kaggle environment. We were only using Kaggle environment until 2 weeks before the competiton ended, and from then we started to use Google Colab with GPU and 25GB of RAM. Still, there were several times when Colab took GPU from us and didn't give it for several hours as we were constanly using their GPU.
\nThanks to my teammates @shhrkre, @ysgong, @sanmaruum, @kuraji.
", "messages": [ { "id": "1156923", "postDate": "01/17/2021 14:21:34", "content": "Also available at https://github.com/yoonseok312/riiid-answer-correctness-prediction with codes.
\nThis is a solution write up for our model, which is an ensemble between a single Light Gradient Boosted Machine model, and a single Encoder-Decoder based Transformer model. It was our first time competing in a Kaggle competition and none of us had previous AI/ML/Stats experience so we learned a lot throughout this competition. Should you have any questions, feel free to contact me at yoonseok@berkeley.edu.
\nts_delta: the gap between timestamp of current content with previous content of the same user.
\ntask_container_id
\nprior_question_elapsed_time: in second, rounded
\nprior_question_had_explanation
\npart
\nnum_tag: number of tags in that question
\nu_chance: the average correctness of the user until current time
\nu_attempts: number of content the user have done
\nu_attempt_c: number of times the user interacted with the specific content in the past (only counting from >1 interactions due to memory)
\nc_chance: the average correctness of the question until current time
\nc_attempts: number of encounter of that question (all user)
\nu_part_chance: the average correctness of the user doing the same part as the question
\nu_part_attempts: number of question of the same part the user have done
\nu_skill_chance: the average correctness of the user doing the same skill as the question (part < 5: listening, part >= 5: reading)
\nu_skill_attempts: number of question of the same skill the user have done
\nt_chance: the average correctness of the user of questions with specific tag until current time
\nt_attempts: user's number of encounter of that tag
\ntotal_explained: number of times explanation was provided to the user until current time (all contents)
\n10_recent_correctness: user correctness of the most recent questions (up to 10)
\n10_recent_mean_gap: mean ts_delta of the most recent questions (up to 10)
\nbundle_elapsed: the mean elapsed time of the bundle, up until the abs time.
\nmean_elapsed: the mean elapsed time of the user until now.
\nprev_t1: tag of the last question
\nprev_cor: correctness of the last question
\ntrueskill_possibility: possibility of the user 'beating' the question (getting the question correct) based on trueskill
\nmu: mu value (mean of trueskill ratings) of user
\nsigma: sigma value (standard deviation of trueskill ratings) of user
\nColumns with NaN value was filled with -1.
We used less than 30 features, but considering that most of the single LGBM models above 0.79 AUC used 40+ features, we did a decent work on focusing on imoportant features.
\nLB score: AUC 0.789
\nNumber of epochs: 6650 (around 15 hours of training in total)
Added below layers with positional encoding.
\n1) Excercise Related
\nmin_delta: minute difference from between this question and the previous. Cap at 1443 (1 day)
\nday_delta: day difference from between this question and the previous. Cap at 30
\nmonth_delta: month difference from between this question and the previous. Cap at 9
\ntid: task container id
\nis_with: if the question is presented with another question. Usually have the same task container
\nc_part: part, one hot encode and denote skill (listening, reading, part1,2,…)
\ntag1…6: tags of question (t1 to t6 are the tag of one question, t1 being the most important tag.) Above embeddings or Dense layer concatenated.
2) Content id (cid)
\nDense layer
Added below layers with positional encoding.
\n1) Response Related
\nprev_answered_correct: correctness of previous answer.
\nprior_elapsed: prior elapsed tiem
\nprior_explained: prior has explanation Above embeddings or Dense layer concatenated.
2) Answered Correctly
\nConcatenate Lecture related Embeddings/Dense
\nnum_lect: number of lecture the user have seen
\nlec_type: 1 hot encode of most recent lecture type, (llecty1, 2…)
\nlec_h_past: time since most recent lecture Above embeddings or Dense layer concatenated.
WINDOW_SIZE: 100
\nEMBED_DIM: 256
\nNUM_HEADS: 16
Use first 80% of data as train set and last 20% as validation set.
\nAUC 0.786
\nSAINT model has plenty of room for improvement, but as 1 epoch took more than 10 hours to train we decided to focus on improving LGBM.
Ensembled a single LGBM model and a Transformer model in 0.55 (LGBM) / 0.45 ratio.
\nAUC: 0.793
Ensembled two LGBM and a Transformer model. 2nd LGBM was same as the first LGBM but except features related to Trueskill. When 2 models out of three models predicted that the user is likely to answer correctly, we used the max value among the 3 predictions. When 2 models out of three models predicted that the user is likely to answer wronly, we used the min value among the 3 predictions. For remaining cases, we mixed three models in 0.4 (Transformer) / 0.45 (First LGBM) / 0.15 (Second LGBM) ratio.
\nAUC: 0.793 (slightly higher than the 1st Inference)
Our biggest mistake was thinking that all the feature engineering, training, and inferencing process must be done in the Kaggle environment. We were only using Kaggle environment until 2 weeks before the competiton ended, and from then we started to use Google Colab with GPU and 25GB of RAM. Still, there were several times when Colab took GPU from us and didn't give it for several hours as we were constanly using their GPU.
\nThanks to my teammates @shhrkre, @ysgong, @sanmaruum, @kuraji.
", "rawMarkdown": "Also available at https://github.com/yoonseok312/riiid-answer-correctness-prediction with codes.\n\nThis is a solution write up for our model, which is an ensemble between a single Light Gradient Boosted Machine model, and a single Encoder-Decoder based Transformer model. It was our first time competing in a Kaggle competition and none of us had previous AI/ML/Stats experience so we learned a lot throughout this competition. Should you have any questions, feel free to contact me at yoonseok@berkeley.edu.\n\n# LightGBM\n## Features\nts_delta: the gap between timestamp of current content with previous content of the same user.\ntask_container_id\nprior_question_elapsed_time: in second, rounded\nprior_question_had_explanation\npart\nnum_tag: number of tags in that question\nu_chance: the average correctness of the user until current time\nu_attempts: number of content the user have done\nu_attempt_c: number of times the user interacted with the specific content in the past (only counting from >1 interactions due to memory)\nc_chance: the average correctness of the question until current time\nc_attempts: number of encounter of that question (all user)\nu_part_chance: the average correctness of the user doing the same part as the question\nu_part_attempts: number of question of the same part the user have done\nu_skill_chance: the average correctness of the user doing the same skill as the question (part < 5: listening, part >= 5: reading)\nu_skill_attempts: number of question of the same skill the user have done\nt_chance: the average correctness of the user of questions with specific tag until current time\nt_attempts: user's number of encounter of that tag\ntotal_explained: number of times explanation was provided to the user until current time (all contents)\n10_recent_correctness: user correctness of the most recent questions (up to 10)\n10_recent_mean_gap: mean ts_delta of the most recent questions (up to 10)\nbundle_elapsed: the mean elapsed time of the bundle, up until the abs time.\nmean_elapsed: the mean elapsed time of the user until now.\nprev_t1: tag of the last question\nprev_cor: correctness of the last question\ntrueskill_possibility: possibility of the user 'beating' the question (getting the question correct) based on trueskill\nmu: mu value (mean of trueskill ratings) of user\nsigma: sigma value (standard deviation of trueskill ratings) of user\nColumns with NaN value was filled with -1.\n\n## Cross Validation and train strategy\n1. Define an absolute time for the whole database (abs_time = user_id//50 + timestamp//1000)\n2. Sort by abs_time. All features mentioned above were engineered such that we will not take data from the future (higher abs_time) into account.\n3. Drop first 25% of the data. This data contains some noise, i.e. when nobody studied plenty of questions yet.\n4. Take last 25% of data as validation set.\n5. Train the model with remaining 50% of the data.\n\n## Single Model AUC\nWe used less than 30 features, but considering that most of the single LGBM models above 0.79 AUC used 40+ features, we did a decent work on focusing on imoportant features.\nLB score: AUC 0.789\nNumber of epochs: 6650 (around 15 hours of training in total)\n\n# Transformer\n## Encoder\nAdded below layers with positional encoding.\n\n1) Excercise Related\nmin_delta: minute difference from between this question and the previous. Cap at 1443 (1 day)\nday_delta: day difference from between this question and the previous. Cap at 30\nmonth_delta: month difference from between this question and the previous. Cap at 9\ntid: task container id\nis_with: if the question is presented with another question. Usually have the same task container\nc_part: part, one hot encode and denote skill (listening, reading, part1,2,...)\ntag1...6: tags of question (t1 to t6 are the tag of one question, t1 being the most important tag.) Above embeddings or Dense layer concatenated.\n\n2) Content id (cid)\nDense layer\n\n## Decoder\nAdded below layers with positional encoding.\n\n1) Response Related\nprev_answered_correct: correctness of previous answer.\nprior_elapsed: prior elapsed tiem\nprior_explained: prior has explanation Above embeddings or Dense layer concatenated.\n\n2) Answered Correctly\nConcatenate Lecture related Embeddings/Dense\nnum_lect: number of lecture the user have seen\nlec_type: 1 hot encode of most recent lecture type, (llecty1, 2...)\nlec_h_past: time since most recent lecture Above embeddings or Dense layer concatenated.\n\n## Parameters\nWINDOW_SIZE: 100\nEMBED_DIM: 256\nNUM_HEADS: 16\n\n## Cross Validation and train strategy\nUse first 80% of data as train set and last 20% as validation set.\n\n## Single Model AUC\nAUC 0.786\nSAINT model has plenty of room for improvement, but as 1 epoch took more than 10 hours to train we decided to focus on improving LGBM.\n\n# Inference\n## 1. Ensembling two models\nEnsembled a single LGBM model and a Transformer model in 0.55 (LGBM) / 0.45 ratio.\nAUC: 0.793\n\n## 2. Ensembling three models\nEnsembled two LGBM and a Transformer model. 2nd LGBM was same as the first LGBM but except features related to Trueskill. When 2 models out of three models predicted that the user is likely to answer correctly, we used the max value among the 3 predictions. When 2 models out of three models predicted that the user is likely to answer wronly, we used the min value among the 3 predictions. For remaining cases, we mixed three models in 0.4 (Transformer) / 0.45 (First LGBM) / 0.15 (Second LGBM) ratio.\nAUC: 0.793 (slightly higher than the 1st Inference)\n\n# Training Environment\nOur biggest mistake was thinking that all the feature engineering, training, and inferencing process must be done in the Kaggle environment. We were only using Kaggle environment until 2 weeks before the competiton ended, and from then we started to use Google Colab with GPU and 25GB of RAM. Still, there were several times when Colab took GPU from us and didn't give it for several hours as we were constanly using their GPU.\n\nThanks to my teammates @shhrkre, @ysgong, @sanmaruum, @kuraji.", "votes": null }, { "id": "1165470", "postDate": "01/23/2021 02:06:18", "content": "good work. THX👍
", "rawMarkdown": "good work. THX👍", "votes": null }, { "id": "1183116", "postDate": "02/02/2021 18:53:58", "content": "nice sharing
", "rawMarkdown": "nice sharing", "votes": null } ], "comments": [ { "id": 1165470, "author_name": "shhrkre", "author_url": "", "post_date": "01/23/2021 02:06:18", "content": "good work. THX👍
", "votes": null, "replies": [] }, { "id": 1183116, "author_name": "henryhzy", "author_url": "", "post_date": "02/02/2021 18:53:58", "content": "nice sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1156923": "Also available at https://github.com/yoonseok312/riiid-answer-correctness-prediction with codes.\n\nThis is a solution write up for our model, which is an ensemble between a single Light Gradient Boosted Machine model, and a single Encoder-Decoder based Transformer model. It was our first time competing in a Kaggle competition and none of us had previous AI/ML/Stats experience so we learned a lot throughout this competition. Should you have any questions, feel free to contact me at yoonseok@berkeley.edu.\n\n# LightGBM\n## Features\nts_delta: the gap between timestamp of current content with previous content of the same user.\ntask_container_id\nprior_question_elapsed_time: in second, rounded\nprior_question_had_explanation\npart\nnum_tag: number of tags in that question\nu_chance: the average correctness of the user until current time\nu_attempts: number of content the user have done\nu_attempt_c: number of times the user interacted with the specific content in the past (only counting from >1 interactions due to memory)\nc_chance: the average correctness of the question until current time\nc_attempts: number of encounter of that question (all user)\nu_part_chance: the average correctness of the user doing the same part as the question\nu_part_attempts: number of question of the same part the user have done\nu_skill_chance: the average correctness of the user doing the same skill as the question (part < 5: listening, part >= 5: reading)\nu_skill_attempts: number of question of the same skill the user have done\nt_chance: the average correctness of the user of questions with specific tag until current time\nt_attempts: user's number of encounter of that tag\ntotal_explained: number of times explanation was provided to the user until current time (all contents)\n10_recent_correctness: user correctness of the most recent questions (up to 10)\n10_recent_mean_gap: mean ts_delta of the most recent questions (up to 10)\nbundle_elapsed: the mean elapsed time of the bundle, up until the abs time.\nmean_elapsed: the mean elapsed time of the user until now.\nprev_t1: tag of the last question\nprev_cor: correctness of the last question\ntrueskill_possibility: possibility of the user 'beating' the question (getting the question correct) based on trueskill\nmu: mu value (mean of trueskill ratings) of user\nsigma: sigma value (standard deviation of trueskill ratings) of user\nColumns with NaN value was filled with -1.\n\n## Cross Validation and train strategy\n1. Define an absolute time for the whole database (abs_time = user_id//50 + timestamp//1000)\n2. Sort by abs_time. All features mentioned above were engineered such that we will not take data from the future (higher abs_time) into account.\n3. Drop first 25% of the data. This data contains some noise, i.e. when nobody studied plenty of questions yet.\n4. Take last 25% of data as validation set.\n5. Train the model with remaining 50% of the data.\n\n## Single Model AUC\nWe used less than 30 features, but considering that most of the single LGBM models above 0.79 AUC used 40+ features, we did a decent work on focusing on imoportant features.\nLB score: AUC 0.789\nNumber of epochs: 6650 (around 15 hours of training in total)\n\n# Transformer\n## Encoder\nAdded below layers with positional encoding.\n\n1) Excercise Related\nmin_delta: minute difference from between this question and the previous. Cap at 1443 (1 day)\nday_delta: day difference from between this question and the previous. Cap at 30\nmonth_delta: month difference from between this question and the previous. Cap at 9\ntid: task container id\nis_with: if the question is presented with another question. Usually have the same task container\nc_part: part, one hot encode and denote skill (listening, reading, part1,2,...)\ntag1...6: tags of question (t1 to t6 are the tag of one question, t1 being the most important tag.) Above embeddings or Dense layer concatenated.\n\n2) Content id (cid)\nDense layer\n\n## Decoder\nAdded below layers with positional encoding.\n\n1) Response Related\nprev_answered_correct: correctness of previous answer.\nprior_elapsed: prior elapsed tiem\nprior_explained: prior has explanation Above embeddings or Dense layer concatenated.\n\n2) Answered Correctly\nConcatenate Lecture related Embeddings/Dense\nnum_lect: number of lecture the user have seen\nlec_type: 1 hot encode of most recent lecture type, (llecty1, 2...)\nlec_h_past: time since most recent lecture Above embeddings or Dense layer concatenated.\n\n## Parameters\nWINDOW_SIZE: 100\nEMBED_DIM: 256\nNUM_HEADS: 16\n\n## Cross Validation and train strategy\nUse first 80% of data as train set and last 20% as validation set.\n\n## Single Model AUC\nAUC 0.786\nSAINT model has plenty of room for improvement, but as 1 epoch took more than 10 hours to train we decided to focus on improving LGBM.\n\n# Inference\n## 1. Ensembling two models\nEnsembled a single LGBM model and a Transformer model in 0.55 (LGBM) / 0.45 ratio.\nAUC: 0.793\n\n## 2. Ensembling three models\nEnsembled two LGBM and a Transformer model. 2nd LGBM was same as the first LGBM but except features related to Trueskill. When 2 models out of three models predicted that the user is likely to answer correctly, we used the max value among the 3 predictions. When 2 models out of three models predicted that the user is likely to answer wronly, we used the min value among the 3 predictions. For remaining cases, we mixed three models in 0.4 (Transformer) / 0.45 (First LGBM) / 0.15 (Second LGBM) ratio.\nAUC: 0.793 (slightly higher than the 1st Inference)\n\n# Training Environment\nOur biggest mistake was thinking that all the feature engineering, training, and inferencing process must be done in the Kaggle environment. We were only using Kaggle environment until 2 weeks before the competiton ended, and from then we started to use Google Colab with GPU and 25GB of RAM. Still, there were several times when Colab took GPU from us and didn't give it for several hours as we were constanly using their GPU.\n\nThanks to my teammates @shhrkre, @ysgong, @sanmaruum, @kuraji.", "1165470": "good work. THX👍", "1183116": "nice sharing" }, "source": "meta" }