{ "id": 209580, "title": "403 place solution", "url": "/competitions/riiid-test-answer-prediction/discussion/209580", "author_name": "Takamichi Toda", "post_date": "2021-01-08T00:12:49.155000", "votes": 9, "comment_count": 4, "views": 0, "content": "

403 place solution

\n

My main strategy is training my model by using BigQuery ML because I wanted to learn it.
\nI think it was a good way for me. And one other good thing about BigQuery ML is I can use all data to train.

\n

Model

\n

XGBoost

\n

It looks that most people use LGBM but I use XGBoost.
\nI'm not familiar GBDT algorithm, I hear that LGBM is better than XGBoost.
\nUnfortunately, BigQuery ML only has XGBoost. So, I use XGBoost.

\n

BigQuery ML has other ML models. For example, Logistic Regression, ARIMA, and Neural Network.
\nI tried them but XGBoost was best.

\n

Parameter

\n

This is my BigQuery code to train my XGB model.
\nThe parameter can be seen here.

\n
CREATE MODEL `<My model PATH>`\nOPTIONS(MODEL_TYPE='BOOSTED_TREE_CLASSIFIER',\n        BOOSTER_TYPE = 'GBTREE',\n        NUM_PARALLEL_TREE = 1,\n        MAX_ITERATIONS = 300,\n        TREE_METHOD = 'HIST',\n        EARLY_STOP = False,\n        MIN_REL_PROGRESS=0.0001,\n        LEARN_RATE =0.3,\n        MAX_TREE_DEPTH=11,\n        COLSAMPLE_BYTREE=1.0,\n        COLSAMPLE_BYLEVEL=0.4,\n        SUBSAMPLE = 0.9,\n        MIN_TREE_CHILD_WEIGHT=2,\n        L1_REG=0,\n        L2_REG=1.0,\n        INPUT_LABEL_COLS = ['answered_correctly'],\n        DATA_SPLIT_METHOD='CUSTOM',\n        DATA_SPLIT_COL='is_test')    \nAS \nSELECT `<My Feature Tables>`\n
\n

CV strategy

\n

leave-one-out

\n

I use this strategy by leave-one-out. It has looked good to me.

\n

Ensemble

\n

I use the best XGBoost model and two SAKT models.
\nSAKT models are made by forking this notebook.
\nI little updated three in this notebook that:

\n
    \n
  1. Randomly cut to a length of 200 in train data. (original code had used last 200 data)
    2. \n
  2. Save best score epoch model.
    3. \n
  3. Parameter tuning.
    4. \n
\n

Each score is here.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
modelCVLB
XGB0.77840.781
SAKT10.7720.774
SAKT20.7720.773
\n

I weighted averaging 3:1:1 this model and I got 0.787 in the public leaderboard.

\n

I tried iteration ensemble and CV fold ensemble in XGBoost, however, both got the same score as my single model.

\n

Final my submission notebook is here.

\n

Feature

\n

My features is:

\n\n

Reference

\n

Notebook

\n\n

EDIT:

\n", "messages": [ { "id": 1143533, "postDate": "2021-01-08T00:12:49.157Z", "content": "

403 place solution

\n

My main strategy is training my model by using BigQuery ML because I wanted to learn it.
\nI think it was a good way for me. And one other good thing about BigQuery ML is I can use all data to train.

\n

Model

\n

XGBoost

\n

It looks that most people use LGBM but I use XGBoost.
\nI'm not familiar GBDT algorithm, I hear that LGBM is better than XGBoost.
\nUnfortunately, BigQuery ML only has XGBoost. So, I use XGBoost.

\n

BigQuery ML has other ML models. For example, Logistic Regression, ARIMA, and Neural Network.
\nI tried them but XGBoost was best.

\n

Parameter

\n

This is my BigQuery code to train my XGB model.
\nThe parameter can be seen here.

\n
CREATE MODEL `<My model PATH>`\nOPTIONS(MODEL_TYPE='BOOSTED_TREE_CLASSIFIER',\n        BOOSTER_TYPE = 'GBTREE',\n        NUM_PARALLEL_TREE = 1,\n        MAX_ITERATIONS = 300,\n        TREE_METHOD = 'HIST',\n        EARLY_STOP = False,\n        MIN_REL_PROGRESS=0.0001,\n        LEARN_RATE =0.3,\n        MAX_TREE_DEPTH=11,\n        COLSAMPLE_BYTREE=1.0,\n        COLSAMPLE_BYLEVEL=0.4,\n        SUBSAMPLE = 0.9,\n        MIN_TREE_CHILD_WEIGHT=2,\n        L1_REG=0,\n        L2_REG=1.0,\n        INPUT_LABEL_COLS = ['answered_correctly'],\n        DATA_SPLIT_METHOD='CUSTOM',\n        DATA_SPLIT_COL='is_test')    \nAS \nSELECT `<My Feature Tables>`\n
\n

CV strategy

\n

leave-one-out

\n

I use this strategy by leave-one-out. It has looked good to me.

\n

Ensemble

\n

I use the best XGBoost model and two SAKT models.
\nSAKT models are made by forking this notebook.
\nI little updated three in this notebook that:

\n
    \n
  1. Randomly cut to a length of 200 in train data. (original code had used last 200 data)
    2. \n
  2. Save best score epoch model.
    3. \n
  3. Parameter tuning.
    4. \n
\n

Each score is here.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
modelCVLB
XGB0.77840.781
SAKT10.7720.774
SAKT20.7720.773
\n

I weighted averaging 3:1:1 this model and I got 0.787 in the public leaderboard.

\n

I tried iteration ensemble and CV fold ensemble in XGBoost, however, both got the same score as my single model.

\n

Final my submission notebook is here.

\n

Feature

\n

My features is:

\n\n

Reference

\n

Notebook

\n\n

EDIT:

\n", "rawMarkdown": "# 403 place solution\n\nMy main strategy is training my model by using BigQuery ML because I wanted to learn it.\nI think it was a good way for me. And one other good thing about BigQuery ML is I can use all data to train.\n\n## Model\n### XGBoost\nIt looks that most people use LGBM but I use XGBoost.\nI'm not familiar GBDT algorithm, I hear that LGBM is better than XGBoost.\nUnfortunately, BigQuery ML only has XGBoost. So, I use XGBoost.\n\nBigQuery ML has other ML models. For example, Logistic Regression, ARIMA, and Neural Network.\nI tried them but XGBoost was best.\n\n### Parameter\n\nThis is my BigQuery code to train my XGB model.\nThe parameter can be seen here.\n\n```sql\nCREATE MODEL ``\nOPTIONS(MODEL_TYPE='BOOSTED_TREE_CLASSIFIER',\n BOOSTER_TYPE = 'GBTREE',\n NUM_PARALLEL_TREE = 1,\n MAX_ITERATIONS = 300,\n TREE_METHOD = 'HIST',\n EARLY_STOP = False,\n MIN_REL_PROGRESS=0.0001,\n LEARN_RATE =0.3,\n MAX_TREE_DEPTH=11,\n COLSAMPLE_BYTREE=1.0,\n COLSAMPLE_BYLEVEL=0.4,\n SUBSAMPLE = 0.9,\n MIN_TREE_CHILD_WEIGHT=2,\n L1_REG=0,\n L2_REG=1.0,\n INPUT_LABEL_COLS = ['answered_correctly'],\n DATA_SPLIT_METHOD='CUSTOM',\n DATA_SPLIT_COL='is_test') \nAS \nSELECT ``\n```\n\n## CV strategy\n### leave-one-out\nI use [this strategy](https://www.kaggle.com/its7171/cv-strategy) by leave-one-out. It has looked good to me.\n\n## Ensemble\n\nI use the best XGBoost model and two SAKT models.\nSAKT models are made by forking [this notebook](https://www.kaggle.com/tarique7/v4-fork-of-riiid-sakt-model-full).\nI little updated three in this notebook that:\n1. Randomly cut to a length of 200 in train data. (original code had used last 200 data)\n2. Save best score epoch model.\n3. Parameter tuning.\n\nEach score is here.\n\n|model|CV|LB|\n| -- | -- | -- |\n|XGB|0.7784|0.781|\n|SAKT1|0.772|0.774|\n|SAKT2|0.772|0.773|\n\nI weighted averaging 3:1:1 this model and I got 0.787 in the public leaderboard.\n\nI tried iteration ensemble and CV fold ensemble in XGBoost, however, both got the same score as my single model.\n\nFinal my submission notebook is [here](https://www.kaggle.com/takamichitoda/riiid-infer-v8-xgb-sakt?scriptVersionId=51291197).\n\n## Feature\nMy features is:\n - aggregation by contents\n - answered_correctl: count/sum/avg/std\n - timestamp: max/min/avg/std\n - user_id: count/unique_count\n - task_container_id: max/min/avg/std\n - user_answer_0~3: sum/avg/std\n - question_elapsed_time: max/min/avg/std\n - question_had_explanation: sum/avg/std\n - each question tags cumulative sum\n - number of try\n - number of correct answer\n - correct answer rate\n - each lecture tag cumulative sum\n - number of try\n - time window features that size is 200 \n - answer correctry: sum/avg/std\n - question_elapsed_time: avg/max\n - question_had_explanation: sum/avg/std\n - timestamp lag: avg/max\n - number of content_id=0\n - number of content_id=1\n - timestamp lag\n\n## Reference\n### Notebook\n- Create SQL: https://www.kaggle.com/takamichitoda/riiid-create-sql\n- Create Tags Onehot Encode Table: https://www.kaggle.com/takamichitoda/riiid-create-feature-table-csv\n- Create CV index: https://www.kaggle.com/takamichitoda/riiid-make-cv-index\n\nEDIT:\n- [my blog](https://www.ai-shift.co.jp/techblog/1505)(Japanese)\n- [SQL to generate Window Features.](https://github.com/trtd56/Riiid/tree/master/work/sql)", "votes": 9 }, { "id": 1143565, "postDate": "2021-01-08T00:29:54.020Z", "content": "

Thanks for sharing. I'm a bit surprised to see you did almost everything in BigQuery.

", "rawMarkdown": "Thanks for sharing. I'm a bit surprised to see you did almost everything in BigQuery.", "votes": 1, "replies": [ { "id": 1143675, "postDate": "2021-01-08T02:02:47.413Z", "content": "

Thank you comment!

\n

I have tried this idea by seen tkm2261's discussion thread.

\n

BigQuery ML merit is here:

\n\n

BigQuery ML demerit is here:

\n\n

I'm satisfied to learn many things.
\nBut it’s still regrettable and frustrating because I lost the medal since I shakedown.
\nI'm studying top solutions now for the next competition.

", "rawMarkdown": "Thank you comment!\n\nI have tried this idea by seen [tkm2261's discussion thread](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190270).\n\nBigQuery ML merit is here:\n- You can treat very large data easily.\n- You can manage feature values with BigQuery\n\nBigQuery ML demerit is here:\n- If you make a sloppy query, you will have to spend a lot of money.\n- In the GBDT algorithm, it can only use XGB (looking at some solutions, LGBM is better.)\n\nI'm satisfied to learn many things.\nBut it’s still regrettable and frustrating because I lost the medal since I shakedown.\nI'm studying top solutions now for the next competition.\n\n", "votes": 1 }, { "id": 1143682, "postDate": "2021-01-08T02:07:41.043Z", "content": "

I feel that BigQuery ML can be one of the powerful options (not only for competitions but particularly in the business use). Thank you again for sharing your experience on this competition.

", "rawMarkdown": "I feel that BigQuery ML can be one of the powerful options (not only for competitions but particularly in the business use). Thank you again for sharing your experience on this competition.", "votes": 1 } ] }, { "id": 1585636, "postDate": "2021-11-17T11:59:38.453Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1143565, "author_name": "u++", "author_url": "", "post_date": "2021-01-08T00:29:54.020000", "content": "

Thanks for sharing. I'm a bit surprised to see you did almost everything in BigQuery.

", "votes": 1, "replies": [ { "id": 1143675, "author_name": "Takamichi Toda", "author_url": "", "post_date": "2021-01-08T02:02:47.413000", "content": "

Thank you comment!

\n

I have tried this idea by seen tkm2261's discussion thread.

\n

BigQuery ML merit is here:

\n\n

BigQuery ML demerit is here:

\n\n

I'm satisfied to learn many things.
\nBut it’s still regrettable and frustrating because I lost the medal since I shakedown.
\nI'm studying top solutions now for the next competition.

", "votes": 1, "replies": [] }, { "id": 1143682, "author_name": "u++", "author_url": "", "post_date": "2021-01-08T02:07:41.043000", "content": "

I feel that BigQuery ML can be one of the powerful options (not only for competitions but particularly in the business use). Thank you again for sharing your experience on this competition.

", "votes": 1, "replies": [] } ] }, { "id": 1585636, "author_name": "", "author_url": "", "post_date": "2021-11-17T11:59:38.453000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1143533": "# 403 place solution\n\nMy main strategy is training my model by using BigQuery ML because I wanted to learn it.\nI think it was a good way for me. And one other good thing about BigQuery ML is I can use all data to train.\n\n## Model\n### XGBoost\nIt looks that most people use LGBM but I use XGBoost.\nI'm not familiar GBDT algorithm, I hear that LGBM is better than XGBoost.\nUnfortunately, BigQuery ML only has XGBoost. So, I use XGBoost.\n\nBigQuery ML has other ML models. For example, Logistic Regression, ARIMA, and Neural Network.\nI tried them but XGBoost was best.\n\n### Parameter\n\nThis is my BigQuery code to train my XGB model.\nThe parameter can be seen here.\n\n```sql\nCREATE MODEL ``\nOPTIONS(MODEL_TYPE='BOOSTED_TREE_CLASSIFIER',\n BOOSTER_TYPE = 'GBTREE',\n NUM_PARALLEL_TREE = 1,\n MAX_ITERATIONS = 300,\n TREE_METHOD = 'HIST',\n EARLY_STOP = False,\n MIN_REL_PROGRESS=0.0001,\n LEARN_RATE =0.3,\n MAX_TREE_DEPTH=11,\n COLSAMPLE_BYTREE=1.0,\n COLSAMPLE_BYLEVEL=0.4,\n SUBSAMPLE = 0.9,\n MIN_TREE_CHILD_WEIGHT=2,\n L1_REG=0,\n L2_REG=1.0,\n INPUT_LABEL_COLS = ['answered_correctly'],\n DATA_SPLIT_METHOD='CUSTOM',\n DATA_SPLIT_COL='is_test') \nAS \nSELECT ``\n```\n\n## CV strategy\n### leave-one-out\nI use [this strategy](https://www.kaggle.com/its7171/cv-strategy) by leave-one-out. It has looked good to me.\n\n## Ensemble\n\nI use the best XGBoost model and two SAKT models.\nSAKT models are made by forking [this notebook](https://www.kaggle.com/tarique7/v4-fork-of-riiid-sakt-model-full).\nI little updated three in this notebook that:\n1. Randomly cut to a length of 200 in train data. (original code had used last 200 data)\n2. Save best score epoch model.\n3. Parameter tuning.\n\nEach score is here.\n\n|model|CV|LB|\n| -- | -- | -- |\n|XGB|0.7784|0.781|\n|SAKT1|0.772|0.774|\n|SAKT2|0.772|0.773|\n\nI weighted averaging 3:1:1 this model and I got 0.787 in the public leaderboard.\n\nI tried iteration ensemble and CV fold ensemble in XGBoost, however, both got the same score as my single model.\n\nFinal my submission notebook is [here](https://www.kaggle.com/takamichitoda/riiid-infer-v8-xgb-sakt?scriptVersionId=51291197).\n\n## Feature\nMy features is:\n - aggregation by contents\n - answered_correctl: count/sum/avg/std\n - timestamp: max/min/avg/std\n - user_id: count/unique_count\n - task_container_id: max/min/avg/std\n - user_answer_0~3: sum/avg/std\n - question_elapsed_time: max/min/avg/std\n - question_had_explanation: sum/avg/std\n - each question tags cumulative sum\n - number of try\n - number of correct answer\n - correct answer rate\n - each lecture tag cumulative sum\n - number of try\n - time window features that size is 200 \n - answer correctry: sum/avg/std\n - question_elapsed_time: avg/max\n - question_had_explanation: sum/avg/std\n - timestamp lag: avg/max\n - number of content_id=0\n - number of content_id=1\n - timestamp lag\n\n## Reference\n### Notebook\n- Create SQL: https://www.kaggle.com/takamichitoda/riiid-create-sql\n- Create Tags Onehot Encode Table: https://www.kaggle.com/takamichitoda/riiid-create-feature-table-csv\n- Create CV index: https://www.kaggle.com/takamichitoda/riiid-make-cv-index\n\nEDIT:\n- [my blog](https://www.ai-shift.co.jp/techblog/1505)(Japanese)\n- [SQL to generate Window Features.](https://github.com/trtd56/Riiid/tree/master/work/sql)", "1143565": "Thanks for sharing. I'm a bit surprised to see you did almost everything in BigQuery.", "1585636": "" } }