{
  "id": 209596,
  "title": "[0.805 Private, 42nd place]  LGB + 5 RAINT+ Ensemble",
  "url": "/competitions/riiid-test-answer-prediction/discussion/209596",
  "author_name": "william.wu",
  "post_date": "2021-01-08T01:12:44.635000",
  "votes": 19,
  "comment_count": 9,
  "views": 0,
  "content": "<p>This is the 2nd competition I have attended and the 1st competition I got top 50, share the final Model I used. Please free feel to comment.<br>\n<strong>Ensemble</strong><br>\n<strong>AUC</strong> Private: 0.805, Public: 0.803</p>\n<pre><code>saint_models = [saint1, saint2, saint3, saint4, saint5]\nsaint_weights = [0.09, 0.091, 0.116, 0.168, 0.368]\nlgb_w = 0.167\n</code></pre>\n<p>With <strong>fully optimized implementation</strong>, the online inference running time with these <strong>6</strong> models is about <strong>3</strong> hours(<strong>GPU</strong> on)</p>\n<h2>LGBM</h2>\n<p><strong>73 features</strong> AUC 0.787 in both public and private</p>\n<pre><code> 0:{'name': 'prior_question_elapsed_time'}\n 1:{'name': 'prior_question_had_explanation', 'dim_type': 'categorical'}\n 2:{'name': 'ques_part', 'dim_type': 'categorical'}\n 3:{'name': 'user_ans_acc', 'depends': ['user_ans_corr_cnt', 'user_ans_cnt'], 'cal_func': 'div', 'desc': 'user questions answering accuracy'}\n 4:{'name': 'user_ans_cnt', 'desc': 'number of questions the user answered'}\n 5:{'name': 'user_ans_corr_cnt', 'desc': 'number of correct answers the user gave'}\n 6:{'name': 'user_ques_part_1_cnt', 'desc': 'number of answers the user answered that are part 1'}\n 7:{'name': 'user_ques_part_1_corr_cnt', 'desc': 'number of correct answers the user gave to the questions that are part 1'}\n 8:{'name': 'user_ques_part_1_acc', 'depends': ['user_ques_part_1_corr_cnt', 'user_ques_part_1_cnt'], 'cal_func': 'div'}\n 9:{'name': 'user_ques_part_2_cnt'}\n10:{'name': 'user_ques_part_2_corr_cnt'}\n11:{'name': 'user_ques_part_2_acc', 'depends': ['user_ques_part_2_corr_cnt', 'user_ques_part_2_cnt'], 'cal_func': 'div'}\n12:{'name': 'user_ques_part_3_cnt'}\n13:{'name': 'user_ques_part_3_corr_cnt'}\n14:{'name': 'user_ques_part_3_acc', 'depends': ['user_ques_part_3_corr_cnt', 'user_ques_part_3_cnt'], 'cal_func': 'div'}\n15:{'name': 'user_ques_part_4_cnt'}\n16:{'name': 'user_ques_part_4_corr_cnt'}\n17:{'name': 'user_ques_part_4_acc', 'depends': ['user_ques_part_4_corr_cnt', 'user_ques_part_4_cnt'], 'cal_func': 'div'}\n18:{'name': 'user_ques_part_5_cnt'}\n19:{'name': 'user_ques_part_5_corr_cnt'}\n20:{'name': 'user_ques_part_5_acc', 'depends': ['user_ques_part_5_corr_cnt', 'user_ques_part_5_cnt'], 'cal_func': 'div'}\n21:{'name': 'user_ques_part_6_cnt'}\n22:{'name': 'user_ques_part_6_corr_cnt'}\n23:{'name': 'user_ques_part_6_acc', 'depends': ['user_ques_part_6_corr_cnt', 'user_ques_part_6_cnt'], 'cal_func': 'div'}\n24:{'name': 'user_ques_part_7_cnt'}\n25:{'name': 'user_ques_part_7_corr_cnt'}\n26:{'name': 'user_ques_part_7_acc', 'depends': ['user_ques_part_7_corr_cnt', 'user_ques_part_7_cnt'], 'cal_func': 'div'}\n27:{'name': 'ques_acc'}\n28:{'name': 'ques_cnt'}\n29:{'name': 'ques_corr_cnt'}\n30:{'name': 'ques_tags_len', 'dim_type': 'categorical'}\n31:{'name': 'user_acc_ques_acc_hmean', 'depends': ['user_ans_acc', 'ques_acc'], 'cal_func': 'harmonic_mean'}\n32:{'name': 'ques_part_acc'}\n33:{'name': 'user_ques_part_1_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_1_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n34:{'name': 'user_ques_part_2_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_2_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n35:{'name': 'user_ques_part_3_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_3_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n36:{'name': 'user_ques_part_4_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_4_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n37:{'name': 'user_ques_part_5_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_5_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n38:{'name': 'user_ques_part_6_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_6_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n39:{'name': 'user_ques_part_7_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_7_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n40:{'name': 'user_last_incorr_till_now'}\n41:{'name': 'user_last_corr_till_now'}\n42:{'name': 'user_ts_diff_lag_1'}\n43:{'name': 'user_ts_diff_lag_2'}\n44:{'name': 'user_ts_diff_lag_3'}\n45:{'name': 'user_last_10_corr_cnt'}\n46:{'name': 'user_acc_mov_avg'}\n47:{'name': 'user_elapsed_time_sum'}\n48:{'name': 'user_viewed_explanations_sum'}\n49:{'name': 'user_elapsed_time_avg', 'depends': ['user_elapsed_time_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n50:{'name': 'user_viewed_explanations_avg', 'depends': ['user_viewed_explanations_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n51:{'name': 'ques_tags_id', 'dim_type': 'categorical'}\n52:{'name': 'ques_tags_acc'}\n53:{'name': 'user_ques_tags_cnt'}\n54:{'name': 'user_ques_tags_corr_cnt'}\n55:{'name': 'user_ques_tags_acc', 'depends': ['user_ques_tags_corr_cnt', 'user_ques_tags_cnt'], 'cal_func': 'div'}\n56:{'name': 'user_ques_tags_max_cnt'}\n57:{'name': 'user_ques_tags_max_corr_cnt'}\n58:{'name': 'user_ques_tags_max_acc', 'depends': ['user_ques_tags_max_corr_cnt', 'user_ques_tags_max_cnt'], 'cal_func': 'div'}\n59:{'name': 'user_ques_tags_min_cnt'}\n60:{'name': 'user_ques_tags_min_corr_cnt'}\n61:{'name': 'user_ques_tags_min_acc', 'depends': ['user_ques_tags_min_corr_cnt', 'user_ques_tags_min_cnt'], 'cal_func': 'div'}\n62:{'name': 'ques_prior_elapsed_time_avg'}\n63:{'name': 'ques_prior_had_explan_avg'}\n64:{'name': 'ques_prior_had_explan_acc'}\n65:{'name': 'ques_prior_hadnot_explan_acc'}\n66:{'name': 'ques_elapsed_time_avg'}\n67:{'name': 'ques_elapsed_time_std'}\n68:{'name': 'ques_explan_avg'}\n69:{'name': 'ques_u_ans_std'}\n70:{'name': 'ques_n_answers'}\n71:{'name': 'user_lect_viewed'}\n72:{'name': 'user_lect_viewed_over_ques_answered', 'depends': ['user_lect_viewed', 'user_ans_cnt'], 'cal_func': 'div'}\n</code></pre>\n<h2>RAINT+</h2>\n<p><strong>1</strong> RAINT+ with 4 encoders/decoders<br>\nencoder side inputs: questions, question parts, timestamp lags<br>\ndecoder side inputs: response, timestamp lags<br>\nPrivate: 0.799, Public: 0.797</p>\n<pre><code>saint1_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n)\n</code></pre>\n<p><strong>2</strong> RAINT+ with 6 encoders/decoders<br>\nencoder/decoder inputs are the same as the 1st one<br>\nConcat the embedding of the inputs and * W (shape: N x 128)<br>\nPrivate: 0.798, Public: 0.796</p>\n<pre><code>saint2_cfg = SaintModelConfig(\n    n_enc=6,\n    n_dec=6,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=tuple(),\n    emb_with_bias=True\n)\n</code></pre>\n<p><strong>3</strong> RAINT+ with 4 encoders/decoders<br>\nAdding question frequency/correct frequency  to encoder side, other parts the same as the 1st one</p>\n<pre><code>saint3_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\")\n)\n</code></pre>\n<p><strong>4</strong> RAINT+ with 4 encoders/decoders<br>\nAdding tags_id to the encoder side, other parts the same as the 3rd one<br>\nPrivate: 0.802, Public: 0.800</p>\n<pre><code>saint4_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\")\n)\n</code></pre>\n<p><strong>5</strong> RAINT+ with 4 encoders/decoders<br>\nConcatenating embedding outputs * W (shape: N x 128, bias=False), other parts are the same as the 4th one.<br>\nDidn't submit alone.</p>\n<pre><code>saint5_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    n_tag_ids=1520,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\"),\n    emb_with_bias=False\n)\n</code></pre>",
  "messages": [
    {
      "id": 1143616,
      "postDate": "2021-01-08T01:12:44.637Z",
      "content": "<p>This is the 2nd competition I have attended and the 1st competition I got top 50, share the final Model I used. Please free feel to comment.<br>\n<strong>Ensemble</strong><br>\n<strong>AUC</strong> Private: 0.805, Public: 0.803</p>\n<pre><code>saint_models = [saint1, saint2, saint3, saint4, saint5]\nsaint_weights = [0.09, 0.091, 0.116, 0.168, 0.368]\nlgb_w = 0.167\n</code></pre>\n<p>With <strong>fully optimized implementation</strong>, the online inference running time with these <strong>6</strong> models is about <strong>3</strong> hours(<strong>GPU</strong> on)</p>\n<h2>LGBM</h2>\n<p><strong>73 features</strong> AUC 0.787 in both public and private</p>\n<pre><code> 0:{'name': 'prior_question_elapsed_time'}\n 1:{'name': 'prior_question_had_explanation', 'dim_type': 'categorical'}\n 2:{'name': 'ques_part', 'dim_type': 'categorical'}\n 3:{'name': 'user_ans_acc', 'depends': ['user_ans_corr_cnt', 'user_ans_cnt'], 'cal_func': 'div', 'desc': 'user questions answering accuracy'}\n 4:{'name': 'user_ans_cnt', 'desc': 'number of questions the user answered'}\n 5:{'name': 'user_ans_corr_cnt', 'desc': 'number of correct answers the user gave'}\n 6:{'name': 'user_ques_part_1_cnt', 'desc': 'number of answers the user answered that are part 1'}\n 7:{'name': 'user_ques_part_1_corr_cnt', 'desc': 'number of correct answers the user gave to the questions that are part 1'}\n 8:{'name': 'user_ques_part_1_acc', 'depends': ['user_ques_part_1_corr_cnt', 'user_ques_part_1_cnt'], 'cal_func': 'div'}\n 9:{'name': 'user_ques_part_2_cnt'}\n10:{'name': 'user_ques_part_2_corr_cnt'}\n11:{'name': 'user_ques_part_2_acc', 'depends': ['user_ques_part_2_corr_cnt', 'user_ques_part_2_cnt'], 'cal_func': 'div'}\n12:{'name': 'user_ques_part_3_cnt'}\n13:{'name': 'user_ques_part_3_corr_cnt'}\n14:{'name': 'user_ques_part_3_acc', 'depends': ['user_ques_part_3_corr_cnt', 'user_ques_part_3_cnt'], 'cal_func': 'div'}\n15:{'name': 'user_ques_part_4_cnt'}\n16:{'name': 'user_ques_part_4_corr_cnt'}\n17:{'name': 'user_ques_part_4_acc', 'depends': ['user_ques_part_4_corr_cnt', 'user_ques_part_4_cnt'], 'cal_func': 'div'}\n18:{'name': 'user_ques_part_5_cnt'}\n19:{'name': 'user_ques_part_5_corr_cnt'}\n20:{'name': 'user_ques_part_5_acc', 'depends': ['user_ques_part_5_corr_cnt', 'user_ques_part_5_cnt'], 'cal_func': 'div'}\n21:{'name': 'user_ques_part_6_cnt'}\n22:{'name': 'user_ques_part_6_corr_cnt'}\n23:{'name': 'user_ques_part_6_acc', 'depends': ['user_ques_part_6_corr_cnt', 'user_ques_part_6_cnt'], 'cal_func': 'div'}\n24:{'name': 'user_ques_part_7_cnt'}\n25:{'name': 'user_ques_part_7_corr_cnt'}\n26:{'name': 'user_ques_part_7_acc', 'depends': ['user_ques_part_7_corr_cnt', 'user_ques_part_7_cnt'], 'cal_func': 'div'}\n27:{'name': 'ques_acc'}\n28:{'name': 'ques_cnt'}\n29:{'name': 'ques_corr_cnt'}\n30:{'name': 'ques_tags_len', 'dim_type': 'categorical'}\n31:{'name': 'user_acc_ques_acc_hmean', 'depends': ['user_ans_acc', 'ques_acc'], 'cal_func': 'harmonic_mean'}\n32:{'name': 'ques_part_acc'}\n33:{'name': 'user_ques_part_1_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_1_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n34:{'name': 'user_ques_part_2_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_2_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n35:{'name': 'user_ques_part_3_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_3_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n36:{'name': 'user_ques_part_4_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_4_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n37:{'name': 'user_ques_part_5_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_5_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n38:{'name': 'user_ques_part_6_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_6_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n39:{'name': 'user_ques_part_7_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_7_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n40:{'name': 'user_last_incorr_till_now'}\n41:{'name': 'user_last_corr_till_now'}\n42:{'name': 'user_ts_diff_lag_1'}\n43:{'name': 'user_ts_diff_lag_2'}\n44:{'name': 'user_ts_diff_lag_3'}\n45:{'name': 'user_last_10_corr_cnt'}\n46:{'name': 'user_acc_mov_avg'}\n47:{'name': 'user_elapsed_time_sum'}\n48:{'name': 'user_viewed_explanations_sum'}\n49:{'name': 'user_elapsed_time_avg', 'depends': ['user_elapsed_time_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n50:{'name': 'user_viewed_explanations_avg', 'depends': ['user_viewed_explanations_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n51:{'name': 'ques_tags_id', 'dim_type': 'categorical'}\n52:{'name': 'ques_tags_acc'}\n53:{'name': 'user_ques_tags_cnt'}\n54:{'name': 'user_ques_tags_corr_cnt'}\n55:{'name': 'user_ques_tags_acc', 'depends': ['user_ques_tags_corr_cnt', 'user_ques_tags_cnt'], 'cal_func': 'div'}\n56:{'name': 'user_ques_tags_max_cnt'}\n57:{'name': 'user_ques_tags_max_corr_cnt'}\n58:{'name': 'user_ques_tags_max_acc', 'depends': ['user_ques_tags_max_corr_cnt', 'user_ques_tags_max_cnt'], 'cal_func': 'div'}\n59:{'name': 'user_ques_tags_min_cnt'}\n60:{'name': 'user_ques_tags_min_corr_cnt'}\n61:{'name': 'user_ques_tags_min_acc', 'depends': ['user_ques_tags_min_corr_cnt', 'user_ques_tags_min_cnt'], 'cal_func': 'div'}\n62:{'name': 'ques_prior_elapsed_time_avg'}\n63:{'name': 'ques_prior_had_explan_avg'}\n64:{'name': 'ques_prior_had_explan_acc'}\n65:{'name': 'ques_prior_hadnot_explan_acc'}\n66:{'name': 'ques_elapsed_time_avg'}\n67:{'name': 'ques_elapsed_time_std'}\n68:{'name': 'ques_explan_avg'}\n69:{'name': 'ques_u_ans_std'}\n70:{'name': 'ques_n_answers'}\n71:{'name': 'user_lect_viewed'}\n72:{'name': 'user_lect_viewed_over_ques_answered', 'depends': ['user_lect_viewed', 'user_ans_cnt'], 'cal_func': 'div'}\n</code></pre>\n<h2>RAINT+</h2>\n<p><strong>1</strong> RAINT+ with 4 encoders/decoders<br>\nencoder side inputs: questions, question parts, timestamp lags<br>\ndecoder side inputs: response, timestamp lags<br>\nPrivate: 0.799, Public: 0.797</p>\n<pre><code>saint1_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n)\n</code></pre>\n<p><strong>2</strong> RAINT+ with 6 encoders/decoders<br>\nencoder/decoder inputs are the same as the 1st one<br>\nConcat the embedding of the inputs and * W (shape: N x 128)<br>\nPrivate: 0.798, Public: 0.796</p>\n<pre><code>saint2_cfg = SaintModelConfig(\n    n_enc=6,\n    n_dec=6,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=tuple(),\n    emb_with_bias=True\n)\n</code></pre>\n<p><strong>3</strong> RAINT+ with 4 encoders/decoders<br>\nAdding question frequency/correct frequency  to encoder side, other parts the same as the 1st one</p>\n<pre><code>saint3_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\")\n)\n</code></pre>\n<p><strong>4</strong> RAINT+ with 4 encoders/decoders<br>\nAdding tags_id to the encoder side, other parts the same as the 3rd one<br>\nPrivate: 0.802, Public: 0.800</p>\n<pre><code>saint4_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\")\n)\n</code></pre>\n<p><strong>5</strong> RAINT+ with 4 encoders/decoders<br>\nConcatenating embedding outputs * W (shape: N x 128, bias=False), other parts are the same as the 4th one.<br>\nDidn't submit alone.</p>\n<pre><code>saint5_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    n_tag_ids=1520,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\"),\n    emb_with_bias=False\n)\n</code></pre>",
      "rawMarkdown": "This is the 2nd competition I have attended and the 1st competition I got top 50, share the final Model I used. Please free feel to comment.\n**Ensemble**\n**AUC** Private: 0.805, Public: 0.803\n```\nsaint_models = [saint1, saint2, saint3, saint4, saint5]\nsaint_weights = [0.09, 0.091, 0.116, 0.168, 0.368]\nlgb_w = 0.167\n```\nWith **fully optimized implementation**, the online inference running time with these **6** models is about **3** hours(**GPU** on)\n\n## LGBM\n**73 features** AUC 0.787 in both public and private\n```\n 0:{'name': 'prior_question_elapsed_time'}\n 1:{'name': 'prior_question_had_explanation', 'dim_type': 'categorical'}\n 2:{'name': 'ques_part', 'dim_type': 'categorical'}\n 3:{'name': 'user_ans_acc', 'depends': ['user_ans_corr_cnt', 'user_ans_cnt'], 'cal_func': 'div', 'desc': 'user questions answering accuracy'}\n 4:{'name': 'user_ans_cnt', 'desc': 'number of questions the user answered'}\n 5:{'name': 'user_ans_corr_cnt', 'desc': 'number of correct answers the user gave'}\n 6:{'name': 'user_ques_part_1_cnt', 'desc': 'number of answers the user answered that are part 1'}\n 7:{'name': 'user_ques_part_1_corr_cnt', 'desc': 'number of correct answers the user gave to the questions that are part 1'}\n 8:{'name': 'user_ques_part_1_acc', 'depends': ['user_ques_part_1_corr_cnt', 'user_ques_part_1_cnt'], 'cal_func': 'div'}\n 9:{'name': 'user_ques_part_2_cnt'}\n10:{'name': 'user_ques_part_2_corr_cnt'}\n11:{'name': 'user_ques_part_2_acc', 'depends': ['user_ques_part_2_corr_cnt', 'user_ques_part_2_cnt'], 'cal_func': 'div'}\n12:{'name': 'user_ques_part_3_cnt'}\n13:{'name': 'user_ques_part_3_corr_cnt'}\n14:{'name': 'user_ques_part_3_acc', 'depends': ['user_ques_part_3_corr_cnt', 'user_ques_part_3_cnt'], 'cal_func': 'div'}\n15:{'name': 'user_ques_part_4_cnt'}\n16:{'name': 'user_ques_part_4_corr_cnt'}\n17:{'name': 'user_ques_part_4_acc', 'depends': ['user_ques_part_4_corr_cnt', 'user_ques_part_4_cnt'], 'cal_func': 'div'}\n18:{'name': 'user_ques_part_5_cnt'}\n19:{'name': 'user_ques_part_5_corr_cnt'}\n20:{'name': 'user_ques_part_5_acc', 'depends': ['user_ques_part_5_corr_cnt', 'user_ques_part_5_cnt'], 'cal_func': 'div'}\n21:{'name': 'user_ques_part_6_cnt'}\n22:{'name': 'user_ques_part_6_corr_cnt'}\n23:{'name': 'user_ques_part_6_acc', 'depends': ['user_ques_part_6_corr_cnt', 'user_ques_part_6_cnt'], 'cal_func': 'div'}\n24:{'name': 'user_ques_part_7_cnt'}\n25:{'name': 'user_ques_part_7_corr_cnt'}\n26:{'name': 'user_ques_part_7_acc', 'depends': ['user_ques_part_7_corr_cnt', 'user_ques_part_7_cnt'], 'cal_func': 'div'}\n27:{'name': 'ques_acc'}\n28:{'name': 'ques_cnt'}\n29:{'name': 'ques_corr_cnt'}\n30:{'name': 'ques_tags_len', 'dim_type': 'categorical'}\n31:{'name': 'user_acc_ques_acc_hmean', 'depends': ['user_ans_acc', 'ques_acc'], 'cal_func': 'harmonic_mean'}\n32:{'name': 'ques_part_acc'}\n33:{'name': 'user_ques_part_1_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_1_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n34:{'name': 'user_ques_part_2_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_2_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n35:{'name': 'user_ques_part_3_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_3_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n36:{'name': 'user_ques_part_4_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_4_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n37:{'name': 'user_ques_part_5_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_5_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n38:{'name': 'user_ques_part_6_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_6_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n39:{'name': 'user_ques_part_7_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_7_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n40:{'name': 'user_last_incorr_till_now'}\n41:{'name': 'user_last_corr_till_now'}\n42:{'name': 'user_ts_diff_lag_1'}\n43:{'name': 'user_ts_diff_lag_2'}\n44:{'name': 'user_ts_diff_lag_3'}\n45:{'name': 'user_last_10_corr_cnt'}\n46:{'name': 'user_acc_mov_avg'}\n47:{'name': 'user_elapsed_time_sum'}\n48:{'name': 'user_viewed_explanations_sum'}\n49:{'name': 'user_elapsed_time_avg', 'depends': ['user_elapsed_time_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n50:{'name': 'user_viewed_explanations_avg', 'depends': ['user_viewed_explanations_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n51:{'name': 'ques_tags_id', 'dim_type': 'categorical'}\n52:{'name': 'ques_tags_acc'}\n53:{'name': 'user_ques_tags_cnt'}\n54:{'name': 'user_ques_tags_corr_cnt'}\n55:{'name': 'user_ques_tags_acc', 'depends': ['user_ques_tags_corr_cnt', 'user_ques_tags_cnt'], 'cal_func': 'div'}\n56:{'name': 'user_ques_tags_max_cnt'}\n57:{'name': 'user_ques_tags_max_corr_cnt'}\n58:{'name': 'user_ques_tags_max_acc', 'depends': ['user_ques_tags_max_corr_cnt', 'user_ques_tags_max_cnt'], 'cal_func': 'div'}\n59:{'name': 'user_ques_tags_min_cnt'}\n60:{'name': 'user_ques_tags_min_corr_cnt'}\n61:{'name': 'user_ques_tags_min_acc', 'depends': ['user_ques_tags_min_corr_cnt', 'user_ques_tags_min_cnt'], 'cal_func': 'div'}\n62:{'name': 'ques_prior_elapsed_time_avg'}\n63:{'name': 'ques_prior_had_explan_avg'}\n64:{'name': 'ques_prior_had_explan_acc'}\n65:{'name': 'ques_prior_hadnot_explan_acc'}\n66:{'name': 'ques_elapsed_time_avg'}\n67:{'name': 'ques_elapsed_time_std'}\n68:{'name': 'ques_explan_avg'}\n69:{'name': 'ques_u_ans_std'}\n70:{'name': 'ques_n_answers'}\n71:{'name': 'user_lect_viewed'}\n72:{'name': 'user_lect_viewed_over_ques_answered', 'depends': ['user_lect_viewed', 'user_ans_cnt'], 'cal_func': 'div'}\n```\n## RAINT+\n**1** RAINT+ with 4 encoders/decoders\nencoder side inputs: questions, question parts, timestamp lags\ndecoder side inputs: response, timestamp lags\nPrivate: 0.799, Public: 0.797\n```\nsaint1_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n)\n```\n\n**2** RAINT+ with 6 encoders/decoders\nencoder/decoder inputs are the same as the 1st one\nConcat the embedding of the inputs and * W (shape: N x 128)\nPrivate: 0.798, Public: 0.796\n```\nsaint2_cfg = SaintModelConfig(\n    n_enc=6,\n    n_dec=6,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=tuple(),\n    emb_with_bias=True\n)\n```\n\n**3** RAINT+ with 4 encoders/decoders\nAdding question frequency/correct frequency  to encoder side, other parts the same as the 1st one\n```\nsaint3_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\")\n)\n```\n\n**4** RAINT+ with 4 encoders/decoders\nAdding tags_id to the encoder side, other parts the same as the 3rd one\nPrivate: 0.802, Public: 0.800\n```\nsaint4_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\")\n)\n```\n\n**5** RAINT+ with 4 encoders/decoders\nConcatenating embedding outputs * W (shape: N x 128, bias=False), other parts are the same as the 4th one.\nDidn't submit alone.\n```\nsaint5_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    n_tag_ids=1520,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\"),\n    emb_with_bias=False\n)\n```",
      "votes": 19
    },
    {
      "id": 1145657,
      "postDate": "2021-01-09T09:30:26.953Z",
      "content": "<p>nice! a great job!</p>",
      "rawMarkdown": "nice! a great job!"
    },
    {
      "id": 1143964,
      "postDate": "2021-01-08T06:43:29.540Z",
      "content": "<p>太强了！膜拜大佬！！</p>",
      "rawMarkdown": "太强了！膜拜大佬！！"
    },
    {
      "id": 1143681,
      "postDate": "2021-01-08T02:07:21.807Z",
      "content": "<p>🙈Congratulations</p>",
      "rawMarkdown": "🙈Congratulations",
      "replies": [
        {
          "id": 1143719,
          "postDate": "2021-01-08T02:59:33.363Z",
          "content": "<p>Thank you :)</p>",
          "rawMarkdown": "Thank you :)"
        }
      ]
    },
    {
      "id": 1143626,
      "postDate": "2021-01-08T01:18:38.250Z",
      "content": "<p>Thank you for sharing. How did you decide the weight of the models?</p>",
      "rawMarkdown": "Thank you for sharing. How did you decide the weight of the models?",
      "replies": [
        {
          "id": 1143643,
          "postDate": "2021-01-08T01:33:26.763Z",
          "content": "<p>Split the validation set into train/test sets, and use hyperparameters tuning to get the weights. I also tried adding another logistic regression layer and takes the predictions of the 6 models as inputs, the performance is not good.</p>",
          "rawMarkdown": "Split the validation set into train/test sets, and use hyperparameters tuning to get the weights. I also tried adding another logistic regression layer and takes the predictions of the 6 models as inputs, the performance is not good.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1143625,
      "postDate": "2021-01-08T01:18:34.673Z",
      "content": "<p>I am impressed you could ensemble as many models.</p>\n<p>GPU or TPU?</p>",
      "rawMarkdown": "I am impressed you could ensemble as many models.\n\nGPU or TPU?",
      "replies": [
        {
          "id": 1143640,
          "postDate": "2021-01-08T01:28:43.393Z",
          "content": "<p>Using GPU, the bottleneck is not the calculation on GPU, but the feature engineering part to generate the inputs of SAINT+ models. This part should be fully optimized.</p>",
          "rawMarkdown": "Using GPU, the bottleneck is not the calculation on GPU, but the feature engineering part to generate the inputs of SAINT+ models. This part should be fully optimized."
        }
      ]
    },
    {
      "id": 1583348,
      "postDate": "2021-11-15T18:38:43.963Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1145657,
      "author_name": "2981",
      "author_url": "",
      "post_date": "2021-01-09T09:30:26.953000",
      "content": "<p>nice! a great job!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143964,
      "author_name": "朴大福",
      "author_url": "",
      "post_date": "2021-01-08T06:43:29.540000",
      "content": "<p>太强了！膜拜大佬！！</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1143681,
      "author_name": "Mingjie Wang",
      "author_url": "",
      "post_date": "2021-01-08T02:07:21.807000",
      "content": "<p>🙈Congratulations</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1143719,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2021-01-08T02:59:33.363000",
          "content": "<p>Thank you :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1143626,
      "author_name": "u++",
      "author_url": "",
      "post_date": "2021-01-08T01:18:38.250000",
      "content": "<p>Thank you for sharing. How did you decide the weight of the models?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1143643,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2021-01-08T01:33:26.763000",
          "content": "<p>Split the validation set into train/test sets, and use hyperparameters tuning to get the weights. I also tried adding another logistic regression layer and takes the predictions of the 6 models as inputs, the performance is not good.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1143625,
      "author_name": "Andrés Miguel Torrubia Sáez",
      "author_url": "",
      "post_date": "2021-01-08T01:18:34.673000",
      "content": "<p>I am impressed you could ensemble as many models.</p>\n<p>GPU or TPU?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1143640,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2021-01-08T01:28:43.393000",
          "content": "<p>Using GPU, the bottleneck is not the calculation on GPU, but the feature engineering part to generate the inputs of SAINT+ models. This part should be fully optimized.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1583348,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-11-15T18:38:43.963000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1143616": "This is the 2nd competition I have attended and the 1st competition I got top 50, share the final Model I used. Please free feel to comment.\n**Ensemble**\n**AUC** Private: 0.805, Public: 0.803\n```\nsaint_models = [saint1, saint2, saint3, saint4, saint5]\nsaint_weights = [0.09, 0.091, 0.116, 0.168, 0.368]\nlgb_w = 0.167\n```\nWith **fully optimized implementation**, the online inference running time with these **6** models is about **3** hours(**GPU** on)\n\n## LGBM\n**73 features** AUC 0.787 in both public and private\n```\n 0:{'name': 'prior_question_elapsed_time'}\n 1:{'name': 'prior_question_had_explanation', 'dim_type': 'categorical'}\n 2:{'name': 'ques_part', 'dim_type': 'categorical'}\n 3:{'name': 'user_ans_acc', 'depends': ['user_ans_corr_cnt', 'user_ans_cnt'], 'cal_func': 'div', 'desc': 'user questions answering accuracy'}\n 4:{'name': 'user_ans_cnt', 'desc': 'number of questions the user answered'}\n 5:{'name': 'user_ans_corr_cnt', 'desc': 'number of correct answers the user gave'}\n 6:{'name': 'user_ques_part_1_cnt', 'desc': 'number of answers the user answered that are part 1'}\n 7:{'name': 'user_ques_part_1_corr_cnt', 'desc': 'number of correct answers the user gave to the questions that are part 1'}\n 8:{'name': 'user_ques_part_1_acc', 'depends': ['user_ques_part_1_corr_cnt', 'user_ques_part_1_cnt'], 'cal_func': 'div'}\n 9:{'name': 'user_ques_part_2_cnt'}\n10:{'name': 'user_ques_part_2_corr_cnt'}\n11:{'name': 'user_ques_part_2_acc', 'depends': ['user_ques_part_2_corr_cnt', 'user_ques_part_2_cnt'], 'cal_func': 'div'}\n12:{'name': 'user_ques_part_3_cnt'}\n13:{'name': 'user_ques_part_3_corr_cnt'}\n14:{'name': 'user_ques_part_3_acc', 'depends': ['user_ques_part_3_corr_cnt', 'user_ques_part_3_cnt'], 'cal_func': 'div'}\n15:{'name': 'user_ques_part_4_cnt'}\n16:{'name': 'user_ques_part_4_corr_cnt'}\n17:{'name': 'user_ques_part_4_acc', 'depends': ['user_ques_part_4_corr_cnt', 'user_ques_part_4_cnt'], 'cal_func': 'div'}\n18:{'name': 'user_ques_part_5_cnt'}\n19:{'name': 'user_ques_part_5_corr_cnt'}\n20:{'name': 'user_ques_part_5_acc', 'depends': ['user_ques_part_5_corr_cnt', 'user_ques_part_5_cnt'], 'cal_func': 'div'}\n21:{'name': 'user_ques_part_6_cnt'}\n22:{'name': 'user_ques_part_6_corr_cnt'}\n23:{'name': 'user_ques_part_6_acc', 'depends': ['user_ques_part_6_corr_cnt', 'user_ques_part_6_cnt'], 'cal_func': 'div'}\n24:{'name': 'user_ques_part_7_cnt'}\n25:{'name': 'user_ques_part_7_corr_cnt'}\n26:{'name': 'user_ques_part_7_acc', 'depends': ['user_ques_part_7_corr_cnt', 'user_ques_part_7_cnt'], 'cal_func': 'div'}\n27:{'name': 'ques_acc'}\n28:{'name': 'ques_cnt'}\n29:{'name': 'ques_corr_cnt'}\n30:{'name': 'ques_tags_len', 'dim_type': 'categorical'}\n31:{'name': 'user_acc_ques_acc_hmean', 'depends': ['user_ans_acc', 'ques_acc'], 'cal_func': 'harmonic_mean'}\n32:{'name': 'ques_part_acc'}\n33:{'name': 'user_ques_part_1_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_1_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n34:{'name': 'user_ques_part_2_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_2_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n35:{'name': 'user_ques_part_3_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_3_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n36:{'name': 'user_ques_part_4_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_4_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n37:{'name': 'user_ques_part_5_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_5_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n38:{'name': 'user_ques_part_6_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_6_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n39:{'name': 'user_ques_part_7_acc_ques_part_acc_hmean', 'depends': ['user_ques_part_7_acc', 'ques_part_acc'], 'cal_func': 'harmonic_mean'}\n40:{'name': 'user_last_incorr_till_now'}\n41:{'name': 'user_last_corr_till_now'}\n42:{'name': 'user_ts_diff_lag_1'}\n43:{'name': 'user_ts_diff_lag_2'}\n44:{'name': 'user_ts_diff_lag_3'}\n45:{'name': 'user_last_10_corr_cnt'}\n46:{'name': 'user_acc_mov_avg'}\n47:{'name': 'user_elapsed_time_sum'}\n48:{'name': 'user_viewed_explanations_sum'}\n49:{'name': 'user_elapsed_time_avg', 'depends': ['user_elapsed_time_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n50:{'name': 'user_viewed_explanations_avg', 'depends': ['user_viewed_explanations_sum', 'user_ans_cnt'], 'cal_func': 'div'}\n51:{'name': 'ques_tags_id', 'dim_type': 'categorical'}\n52:{'name': 'ques_tags_acc'}\n53:{'name': 'user_ques_tags_cnt'}\n54:{'name': 'user_ques_tags_corr_cnt'}\n55:{'name': 'user_ques_tags_acc', 'depends': ['user_ques_tags_corr_cnt', 'user_ques_tags_cnt'], 'cal_func': 'div'}\n56:{'name': 'user_ques_tags_max_cnt'}\n57:{'name': 'user_ques_tags_max_corr_cnt'}\n58:{'name': 'user_ques_tags_max_acc', 'depends': ['user_ques_tags_max_corr_cnt', 'user_ques_tags_max_cnt'], 'cal_func': 'div'}\n59:{'name': 'user_ques_tags_min_cnt'}\n60:{'name': 'user_ques_tags_min_corr_cnt'}\n61:{'name': 'user_ques_tags_min_acc', 'depends': ['user_ques_tags_min_corr_cnt', 'user_ques_tags_min_cnt'], 'cal_func': 'div'}\n62:{'name': 'ques_prior_elapsed_time_avg'}\n63:{'name': 'ques_prior_had_explan_avg'}\n64:{'name': 'ques_prior_had_explan_acc'}\n65:{'name': 'ques_prior_hadnot_explan_acc'}\n66:{'name': 'ques_elapsed_time_avg'}\n67:{'name': 'ques_elapsed_time_std'}\n68:{'name': 'ques_explan_avg'}\n69:{'name': 'ques_u_ans_std'}\n70:{'name': 'ques_n_answers'}\n71:{'name': 'user_lect_viewed'}\n72:{'name': 'user_lect_viewed_over_ques_answered', 'depends': ['user_lect_viewed', 'user_ans_cnt'], 'cal_func': 'div'}\n```\n## RAINT+\n**1** RAINT+ with 4 encoders/decoders\nencoder side inputs: questions, question parts, timestamp lags\ndecoder side inputs: response, timestamp lags\nPrivate: 0.799, Public: 0.797\n```\nsaint1_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n)\n```\n\n**2** RAINT+ with 6 encoders/decoders\nencoder/decoder inputs are the same as the 1st one\nConcat the embedding of the inputs and * W (shape: N x 128)\nPrivate: 0.798, Public: 0.796\n```\nsaint2_cfg = SaintModelConfig(\n    n_enc=6,\n    n_dec=6,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=tuple(),\n    emb_with_bias=True\n)\n```\n\n**3** RAINT+ with 4 encoders/decoders\nAdding question frequency/correct frequency  to encoder side, other parts the same as the 1st one\n```\nsaint3_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\")\n)\n```\n\n**4** RAINT+ with 4 encoders/decoders\nAdding tags_id to the encoder side, other parts the same as the 3rd one\nPrivate: 0.802, Public: 0.800\n```\nsaint4_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    emb_add_fn=\"add\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\")\n)\n```\n\n**5** RAINT+ with 4 encoders/decoders\nConcatenating embedding outputs * W (shape: N x 128, bias=False), other parts are the same as the 4th one.\nDidn't submit alone.\n```\nsaint5_cfg = SaintModelConfig(\n    n_enc=4,\n    n_dec=4,\n    enc_heads=8,\n    dec_heads=8,\n    seq_len=100,\n    n_dims=128,\n    n_ques=13523,\n    n_parts=7,\n    n_ets=301,\n    n_lts=154,\n    n_resp=2,\n    n_explains=2,\n    n_freqs=111,\n    n_corr_freqs=94,\n    n_tag_ids=1520,\n    emb_add_fn=\"cat\",\n    dropout=0.2,\n    enc_extra_inputs=(\"freqs\", \"corr_freqs\", \"tag_ids\"),\n    emb_with_bias=False\n)\n```",
    "1145657": "nice! a great job!",
    "1143964": "太强了！膜拜大佬！！",
    "1143681": "🙈Congratulations",
    "1143626": "Thank you for sharing. How did you decide the weight of the models?",
    "1143625": "I am impressed you could ensemble as many models.\n\nGPU or TPU?",
    "1583348": ""
  }
}