{ "id": 200851, "title": "Post your non LGBM Score", "url": "/competitions/riiid-test-answer-prediction/discussion/200851", "author_name": "Jaideep", "post_date": "2020-12-02T05:46:56.495000", "votes": 18, "comment_count": 28, "views": 0, "content": "

Was curious to know how far SAINT/SAKT model scoring.
\nSo far i think for quite a many LGBM is one scoring above 76+ .

\n

Please update your model score if you are using one other than LGBM also
\n.I am experimenting SAKT currently.
\n1) Model
\n2) CV..

", "messages": [ { "id": 1099143, "postDate": "2020-12-02T05:46:56.497Z", "content": "

Was curious to know how far SAINT/SAKT model scoring.
\nSo far i think for quite a many LGBM is one scoring above 76+ .

\n

Please update your model score if you are using one other than LGBM also
\n.I am experimenting SAKT currently.
\n1) Model
\n2) CV..

", "rawMarkdown": "Was curious to know how far SAINT/SAKT model scoring. \nSo far i think for quite a many LGBM is one scoring above 76+ . \n\nPlease update your model score if you are using one other than LGBM also\n.I am experimenting SAKT currently. \n1) Model\n2) CV..\n", "votes": 18 }, { "id": 1109423, "postDate": "2020-12-11T16:55:00.373Z", "content": "

My two last versions of SAINT+:

\n

AUC val: 0.7768; AUC LB: 0.781
\nAUC val: 0.7801; AUC LB: 0.784

", "rawMarkdown": "My two last versions of SAINT+:\n\nAUC val: 0.7768; AUC LB: 0.781\nAUC val: 0.7801; AUC LB: 0.784", "votes": 5, "replies": [ { "id": 1109425, "postDate": "2020-12-11T16:57:28.820Z", "content": "

you are close or same as bench mark of SAINT+
\ngreat going..

", "rawMarkdown": "you are close or same as bench mark of SAINT+ \ngreat going..", "votes": 1 } ] }, { "id": 1102228, "postDate": "2020-12-04T17:57:34.180Z", "content": "

CNN with feature engineering, local score: 0.770

", "rawMarkdown": "CNN with feature engineering, local score: 0.770 ", "votes": 3, "replies": [ { "id": 1105746, "postDate": "2020-12-08T06:49:42.663Z", "content": "

Did u use all data?

", "rawMarkdown": "Did u use all data?" }, { "id": 1110068, "postDate": "2020-12-12T11:54:14.383Z", "content": "

@zekun98 No, only 18M rows

", "rawMarkdown": "@zekun98 No, only 18M rows" } ] }, { "id": 1127332, "postDate": "2020-12-26T12:58:13.377Z", "content": "

Update: 78.6 so far with SAINT..

", "rawMarkdown": "Update: 78.6 so far with SAINT.. ", "votes": 1, "replies": [ { "id": 1128195, "postDate": "2020-12-27T08:48:37.687Z", "content": "

Glad to hear that your SAINT model improvement.

", "rawMarkdown": "Glad to hear that your SAINT model improvement." }, { "id": 1128431, "postDate": "2020-12-27T12:40:48.073Z", "content": "

still long way to go :)… not sure how far we can reach.

", "rawMarkdown": "still long way to go :)... not sure how far we can reach.", "votes": 1 } ] }, { "id": 1106188, "postDate": "2020-12-08T15:24:17.003Z", "content": "

Validation 0.758 , public score of 0.765. This was achieved using a standard NN with feature engineering

", "rawMarkdown": "Validation 0.758 , public score of 0.765. This was achieved using a standard NN with feature engineering\n", "votes": 1 }, { "id": 1102119, "postDate": "2020-12-04T15:58:09.360Z", "content": "

Simple matrix factorization model, LB:757

", "rawMarkdown": "Simple matrix factorization model, LB:757", "votes": 1, "replies": [ { "id": 1110423, "postDate": "2020-12-12T18:14:26.173Z", "content": "

Great ,what is matrix factorization model ,any overview @phucdkbk

", "rawMarkdown": "Great ,what is matrix factorization model ,any overview @phucdkbk " } ] }, { "id": 1104160, "postDate": "2020-12-06T17:06:03.457Z", "content": "

my SAINT Base version with not all features included gets me 74.71 compared to 74.56 for SAKT. i think this should get LB of 75.2+

", "rawMarkdown": "my SAINT Base version with not all features included gets me 74.71 compared to 74.56 for SAKT. i think this should get LB of 75.2+", "votes": 2, "replies": [ { "id": 1104191, "postDate": "2020-12-06T17:35:10.157Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 1104374, "postDate": "2020-12-06T21:41:20.533Z", "content": "

For my SAINT, I got even 76.xx in the CV (and in the training AUC too), but only 74.7 in the LB… I don't see any signal of overfitting with my model, but still, got a too lousy score on the LB compared to my expectation…

", "rawMarkdown": "For my SAINT, I got even 76.xx in the CV (and in the training AUC too), but only 74.7 in the LB... I don't see any signal of overfitting with my model, but still, got a too lousy score on the LB compared to my expectation..." }, { "id": 1108180, "postDate": "2020-12-10T11:21:21.647Z", "content": "

@shinomoriaoshi it should not have been the case could be some issue with inputs during inference. i presume you might have used same strategy of cv as SAKT..
\nwere u able to narrow down the gap ?

", "rawMarkdown": "@shinomoriaoshi it should not have been the case could be some issue with inputs during inference. i presume you might have used same strategy of cv as SAKT..\nwere u able to narrow down the gap ?", "votes": 1 }, { "id": 1109034, "postDate": "2020-12-11T09:12:53.660Z", "content": "

@jaideepvalani, I found the problem. It actually comes from my feature engineering steps, there were some sources of data leakage there. Now things make sense.

", "rawMarkdown": "@jaideepvalani, I found the problem. It actually comes from my feature engineering steps, there were some sources of data leakage there. Now things make sense." }, { "id": 1109059, "postDate": "2020-12-11T09:42:16.600Z", "content": "

good to hear..
\nAre there any modifications needed in the loss , i simply pass outputs for q[1:] with labels[1:],i see some posts people using masked BCE. not sure what is to mask here out.

", "rawMarkdown": "good to hear.. \nAre there any modifications needed in the loss , i simply pass outputs for q[1:] with labels[1:],i see some posts people using masked BCE. not sure what is to mask here out.\n" }, { "id": 1109092, "postDate": "2020-12-11T10:21:08.693Z", "content": "

Actually I haven't tried masked BCELoss yet, but my gut feeling is, because the labels are a sequence of answers which different length, so some will be padded. Thee masked BCELoss is just one way to ignore the padded values. But it is just my feeling, not sure it will serve any other purposes

", "rawMarkdown": "Actually I haven't tried masked BCELoss yet, but my gut feeling is, because the labels are a sequence of answers which different length, so some will be padded. Thee masked BCELoss is just one way to ignore the padded values. But it is just my feeling, not sure it will serve any other purposes" }, { "id": 1109154, "postDate": "2020-12-11T11:23:58.023Z", "content": "

i get exactly same score as yours in first attempt.. may be i too could be making silly mistake.. if you could suggest me that will save some time of mine.. :)
\nI suspect i may be passing wrong attn mask to decoder
\npossible Wrong input
\n ```
\ntarget_id=q[1:].copy()
\n label=qa[1:].copy()

\n
    rt = np.zeros(self.max_seq-1, dtype=int)\n    rt=qa[:-1]\n
\n

```

", "rawMarkdown": "i get exactly same score as yours in first attempt.. may be i too could be making silly mistake.. if you could suggest me that will save some time of mine.. :) \nI suspect i may be passing wrong attn mask to decoder\npossible Wrong input\n ```\ntarget_id=q[1:].copy()\n label=qa[1:].copy()\n \n rt = np.zeros(self.max_seq-1, dtype=int)\n rt=qa[:-1]\n```" }, { "id": 1109166, "postDate": "2020-12-11T11:32:56.707Z", "content": "

Just by look at this, I think it is correct, what makes you think it could be wrong?

", "rawMarkdown": "Just by look at this, I think it is correct, what makes you think it could be wrong?" }, { "id": 1109473, "postDate": "2020-12-11T18:16:52.087Z", "content": "

because m unable to match the cv score to lb score. few possible mistakes i could find was incorrect embedding sizes as per indices of features which i corrected it now, will see.
\nSecondly, is the score you got including status update for test df . I have been so far testing only without test df status update.

", "rawMarkdown": " because m unable to match the cv score to lb score. few possible mistakes i could find was incorrect embedding sizes as per indices of features which i corrected it now, will see. \n\nSecondly, is the score you got including status update for test df . I have been so far testing only without test df status update." }, { "id": 1109483, "postDate": "2020-12-11T18:25:15.357Z", "content": "

For the basic SAINT model, we got 0.754 without state updates, for the SAINT+ model, we got 0.770 with state updates. Funny thing is, the CV of both models are 0.7392 and 0.7475 respectively. Are you training and making an inference in one notebook or in two separate notebooks. We did them in 2 notebooks, and our first attempt was horrible because the feature engineering in these two notebooks was different. So, I suggest you check the test iterator, first, then check your feature engineering steps before training, there could be some leakage there (use some \"sklearn\" package for them)

", "rawMarkdown": "For the basic SAINT model, we got 0.754 without state updates, for the SAINT+ model, we got 0.770 with state updates. Funny thing is, the CV of both models are 0.7392 and 0.7475 respectively. Are you training and making an inference in one notebook or in two separate notebooks. We did them in 2 notebooks, and our first attempt was horrible because the feature engineering in these two notebooks was different. So, I suggest you check the test iterator, first, then check your feature engineering steps before training, there could be some leakage there (use some \"sklearn\" package for them)" }, { "id": 1110419, "postDate": "2020-12-12T18:13:18.090Z", "content": "

Finally after 5 says of unrelentless work ,77.1 with basic saint

", "rawMarkdown": "Finally after 5 says of unrelentless work ,77.1 with basic saint", "votes": 1 }, { "id": 1110444, "postDate": "2020-12-12T18:31:57.993Z", "content": "

congrats, an impressive score

", "rawMarkdown": "congrats, an impressive score" }, { "id": 1110447, "postDate": "2020-12-12T18:32:45.790Z", "content": "

can I ask how did you get on top of that, I mean how you can overcome all problems before

", "rawMarkdown": "can I ask how did you get on top of that, I mean how you can overcome all problems before" }, { "id": 1110750, "postDate": "2020-12-13T03:19:24.367Z", "content": "

M yet to determine exact problem i just thought I would start with basics wont add any features ,other than what we have in sakt. Rectified some embedding issues .but don't yet wat was causing issue earlier . Input wise I was good you said above .
\nBut I know some one would post baseline of 78 soon ,so this will be broken :)

", "rawMarkdown": "M yet to determine exact problem i just thought I would start with basics wont add any features ,other than what we have in sakt. Rectified some embedding issues .but don't yet wat was causing issue earlier . Input wise I was good you said above .\nBut I know some one would post baseline of 78 soon ,so this will be broken :)" } ] }, { "id": 1127347, "postDate": "2020-12-26T13:08:55.303Z", "content": "

0.789 SAINT + like model

", "rawMarkdown": "0.789 SAINT + like model" }, { "id": 1105250, "postDate": "2020-12-07T17:06:50.040Z", "content": "

looks like LGB is all you need.

", "rawMarkdown": "looks like LGB is all you need." } ], "comments": [ { "id": 1109423, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-12-11T16:55:00.373000", "content": "

My two last versions of SAINT+:

\n

AUC val: 0.7768; AUC LB: 0.781
\nAUC val: 0.7801; AUC LB: 0.784

", "votes": 5, "replies": [ { "id": 1109425, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-11T16:57:28.820000", "content": "

you are close or same as bench mark of SAINT+
\ngreat going..

", "votes": 1, "replies": [] } ] }, { "id": 1102228, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-04T17:57:34.180000", "content": "

CNN with feature engineering, local score: 0.770

", "votes": 3, "replies": [ { "id": 1105746, "author_name": "HAaHAa", "author_url": "", "post_date": "2020-12-08T06:49:42.663000", "content": "

Did u use all data?

", "votes": 0, "replies": [] }, { "id": 1110068, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-12T11:54:14.383000", "content": "

@zekun98 No, only 18M rows

", "votes": 0, "replies": [] } ] }, { "id": 1127332, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-26T12:58:13.377000", "content": "

Update: 78.6 so far with SAINT..

", "votes": 1, "replies": [ { "id": 1128195, "author_name": "Dean", "author_url": "", "post_date": "2020-12-27T08:48:37.687000", "content": "

Glad to hear that your SAINT model improvement.

", "votes": 0, "replies": [] }, { "id": 1128431, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-27T12:40:48.073000", "content": "

still long way to go :)… not sure how far we can reach.

", "votes": 1, "replies": [] } ] }, { "id": 1106188, "author_name": "Darren Lahr", "author_url": "", "post_date": "2020-12-08T15:24:17.003000", "content": "

Validation 0.758 , public score of 0.765. This was achieved using a standard NN with feature engineering

", "votes": 1, "replies": [] }, { "id": 1102119, "author_name": "Do Phuc", "author_url": "", "post_date": "2020-12-04T15:58:09.360000", "content": "

Simple matrix factorization model, LB:757

", "votes": 1, "replies": [ { "id": 1110423, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-12T18:14:26.173000", "content": "

Great ,what is matrix factorization model ,any overview @phucdkbk

", "votes": 0, "replies": [] } ] }, { "id": 1104160, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-06T17:06:03.457000", "content": "

my SAINT Base version with not all features included gets me 74.71 compared to 74.56 for SAKT. i think this should get LB of 75.2+

", "votes": 2, "replies": [ { "id": 1104191, "author_name": "", "author_url": "", "post_date": "2020-12-06T17:35:10.157000", "content": "", "votes": -1, "replies": [] }, { "id": 1104374, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-06T21:41:20.533000", "content": "

For my SAINT, I got even 76.xx in the CV (and in the training AUC too), but only 74.7 in the LB… I don't see any signal of overfitting with my model, but still, got a too lousy score on the LB compared to my expectation…

", "votes": 0, "replies": [] }, { "id": 1108180, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-10T11:21:21.647000", "content": "

@shinomoriaoshi it should not have been the case could be some issue with inputs during inference. i presume you might have used same strategy of cv as SAKT..
\nwere u able to narrow down the gap ?

", "votes": 1, "replies": [] }, { "id": 1109034, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-11T09:12:53.660000", "content": "

@jaideepvalani, I found the problem. It actually comes from my feature engineering steps, there were some sources of data leakage there. Now things make sense.

", "votes": 0, "replies": [] }, { "id": 1109059, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-11T09:42:16.600000", "content": "

good to hear..
\nAre there any modifications needed in the loss , i simply pass outputs for q[1:] with labels[1:],i see some posts people using masked BCE. not sure what is to mask here out.

", "votes": 0, "replies": [] }, { "id": 1109092, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-11T10:21:08.693000", "content": "

Actually I haven't tried masked BCELoss yet, but my gut feeling is, because the labels are a sequence of answers which different length, so some will be padded. Thee masked BCELoss is just one way to ignore the padded values. But it is just my feeling, not sure it will serve any other purposes

", "votes": 0, "replies": [] }, { "id": 1109154, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-11T11:23:58.023000", "content": "

i get exactly same score as yours in first attempt.. may be i too could be making silly mistake.. if you could suggest me that will save some time of mine.. :)
\nI suspect i may be passing wrong attn mask to decoder
\npossible Wrong input
\n ```
\ntarget_id=q[1:].copy()
\n label=qa[1:].copy()

\n
    rt = np.zeros(self.max_seq-1, dtype=int)\n    rt=qa[:-1]\n
\n

```

", "votes": 0, "replies": [] }, { "id": 1109166, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-11T11:32:56.707000", "content": "

Just by look at this, I think it is correct, what makes you think it could be wrong?

", "votes": 0, "replies": [] }, { "id": 1109473, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-11T18:16:52.087000", "content": "

because m unable to match the cv score to lb score. few possible mistakes i could find was incorrect embedding sizes as per indices of features which i corrected it now, will see.
\nSecondly, is the score you got including status update for test df . I have been so far testing only without test df status update.

", "votes": 0, "replies": [] }, { "id": 1109483, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-11T18:25:15.357000", "content": "

For the basic SAINT model, we got 0.754 without state updates, for the SAINT+ model, we got 0.770 with state updates. Funny thing is, the CV of both models are 0.7392 and 0.7475 respectively. Are you training and making an inference in one notebook or in two separate notebooks. We did them in 2 notebooks, and our first attempt was horrible because the feature engineering in these two notebooks was different. So, I suggest you check the test iterator, first, then check your feature engineering steps before training, there could be some leakage there (use some \"sklearn\" package for them)

", "votes": 0, "replies": [] }, { "id": 1110419, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-12T18:13:18.090000", "content": "

Finally after 5 says of unrelentless work ,77.1 with basic saint

", "votes": 1, "replies": [] }, { "id": 1110444, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-12T18:31:57.993000", "content": "

congrats, an impressive score

", "votes": 0, "replies": [] }, { "id": 1110447, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2020-12-12T18:32:45.790000", "content": "

can I ask how did you get on top of that, I mean how you can overcome all problems before

", "votes": 0, "replies": [] }, { "id": 1110750, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-13T03:19:24.367000", "content": "

M yet to determine exact problem i just thought I would start with basics wont add any features ,other than what we have in sakt. Rectified some embedding issues .but don't yet wat was causing issue earlier . Input wise I was good you said above .
\nBut I know some one would post baseline of 78 soon ,so this will be broken :)

", "votes": 0, "replies": [] } ] }, { "id": 1127347, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-26T13:08:55.303000", "content": "

0.789 SAINT + like model

", "votes": 0, "replies": [] }, { "id": 1105250, "author_name": "biubiuG", "author_url": "", "post_date": "2020-12-07T17:06:50.040000", "content": "

looks like LGB is all you need.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1099143": "Was curious to know how far SAINT/SAKT model scoring. \nSo far i think for quite a many LGBM is one scoring above 76+ . \n\nPlease update your model score if you are using one other than LGBM also\n.I am experimenting SAKT currently. \n1) Model\n2) CV..\n", "1109423": "My two last versions of SAINT+:\n\nAUC val: 0.7768; AUC LB: 0.781\nAUC val: 0.7801; AUC LB: 0.784", "1102228": "CNN with feature engineering, local score: 0.770 ", "1127332": "Update: 78.6 so far with SAINT.. ", "1106188": "Validation 0.758 , public score of 0.765. This was achieved using a standard NN with feature engineering\n", "1102119": "Simple matrix factorization model, LB:757", "1104160": "my SAINT Base version with not all features included gets me 74.71 compared to 74.56 for SAKT. i think this should get LB of 75.2+", "1127347": "0.789 SAINT + like model", "1105250": "looks like LGB is all you need." } }