in Saint Paper there is mention of Category Embeddings as one of the input.
\n does this category refers to Part or Tag of the question?
in Saint Paper there is mention of Category Embeddings as one of the input.
\n does this category refers to Part or Tag of the question?
I believe for SAINT/SAINT+ the embeddings used are
\nIn Saint plus they say part but in Saint its exercise category, while part is section of assessment comprising of various question
", "rawMarkdown": "In Saint plus they say part but in Saint its exercise category, while part is section of assessment comprising of various question", "votes": null }, { "id": "1105688", "postDate": "12/08/2020 05:05:28", "content": "Oh, I might have mixed them then. Thanks for clarifying. I don't see a reason for not using both at the same time. They both incorporate different information to the model.
", "rawMarkdown": "Oh, I might have mixed them then. Thanks for clarifying. I don't see a reason for not using both at the same time. They both incorporate different information to the model.", "votes": null }, { "id": "1105696", "postDate": "12/08/2020 05:16:25", "content": "@abdurrafae wcm , I do see good improvement upon adding on top of Excercise and Output seq , further embeddings
\nis your current LB score based on SAINT or LGB..
\nIf yes how much CV you get with SAINT Base version.
\nThey also mention about Start token Embeddings i dint quite get that ..
", "rawMarkdown": "abdurrafae wcm , I do see good improvement upon adding on top of Excercise and Output seq , further embeddings\n\n\n is your current LB score based on SAINT or LGB.. \n\nIf yes how much CV you get with SAINT Base version. \n\nThey also mention about Start token Embeddings i dint quite get that ..", "votes": null }, { "id": "1105821", "postDate": "12/08/2020 08:27:12", "content": "I got upto 0.775 with SAINT+ on validation data. my validation is closely co-related to LB +/- 0.001.
\nI only trained it for one session of Kaggle environment so not sure if it would have improved further or not, but the gain in each epoch was too low for me to invest in it further.
\nThe issue I was having was that the model diverged when I increase number of layers or model dimensions. So I built it from scratch again and made some modifications along the way. Now it doesn't match the exact SAINT/SAINT+ but it's quite similar.
\nFor start token, I was just padding the sequence with zeros on left (with length 1) and cropping (with length 1) from right
", "rawMarkdown": "I got upto 0.775 with SAINT+ on validation data. my validation is closely co-related to LB +/- 0.001.\n\nI only trained it for one session of Kaggle environment so not sure if it would have improved further or not, but the gain in each epoch was too low for me to invest in it further. \n\nThe issue I was having was that the model diverged when I increase number of layers or model dimensions. So I built it from scratch again and made some modifications along the way. Now it doesn't match the exact SAINT/SAINT+ but it's quite similar.\n\nFor start token, I was just padding the sequence with zeros on left (with length 1) and cropping (with length 1) from right", "votes": null }, { "id": "1105889", "postDate": "12/08/2020 09:48:41", "content": "ok thanks , in Saint i get 75.75 as cv yet to test cv .
\nChallenge m finding is how do I handle user ID not in test as i wont get the same in responses that iterations .
In Saint + did you add lag time also?
While addong prior elapsed time were u able to get right chronology of question .there is some discussion on going about the caveat involved if rely just on default sequence .
What is purpose of start token ,I dint gave any.
1) I use dummy responses for new users till I get them in next iteration.
\n2) I used a feature based on lag time, so not lag time exactly.
\n3) I used lag time as absolute difference between bundles (i.e. disregarding the time elapsed field to calculate this)
\n4) Start token is to mis-align your questions and response streams since at test time you'll have access to question (1:t) and responses (1:t-1).
@abdurrafae how did perform merge operation between train csv and any other df during inference
", "rawMarkdown": "abdurrafae how did perform merge operation between train csv and any other df during inference", "votes": null }, { "id": "1107510", "postDate": "12/09/2020 18:39:57", "content": "I use pandas for that. I only have 1 dataframe to join to, so it doesn't take that long
", "rawMarkdown": "I use pandas for that. I only have 1 dataframe to join to, so it doesn't take that long", "votes": null }, { "id": "1107512", "postDate": "12/09/2020 18:40:49", "content": "I don't have the train_df at inference, I've a customized object that stores data for each user.
", "rawMarkdown": "I don't have the train_df at inference, I've a customized object that stores data for each user.", "votes": null }, { "id": "1108158", "postDate": "12/10/2020 10:49:27", "content": "Thanks
\n1) https://www.kaggle.com/mpware/sakt-fork/
\nin this kernel what is significance of starting question from 2: rather than from 0 or 1:
questions = np.append(q[2:], [target_id])
2) what was your first sub score with SAINT .. i get good cv with base SAINT but LB not aligned with CV. 76.42/ 74.7 following the same input indexing as above and not accumulating the Response history ,to start with.
\n not sure if am making any mistake in inference
No idea about the SAKT kernel.
\nI had a validation score of 0.775 with SAINT, didn't get an LB on that.
\nMy Validation score is closely related to LB with +/- 0.001
from paper\nThe encoder takes the sequence\nEe = [Ee1,··· ,Eek] of exercise embeddings and feeds the processed output O = [O1,··· ,Ok\n] to the decoder. The decoder\ntakes O and another sequential input R\ne = [S,Re1,··· ,Rek−1] of\nresponse embeddings with the start token embedding S, and\nproduces the predicted responses ˆr = [rˆ1,··· ,rˆk\n].\nI had one more doubt wrt to SAINT.. for any user id if i understand well for any current Exercise E1 model has got previous response as correct or not but in the case of test , when we add in Test questions we dont have Test users exercise history for test question ..
\nAt any given user iteration
\neg: User 1- Test question T1,T2,T3.
\nExercise seq could become :
\nE1(from Train or val) -T1,T2,T3
\nResponse sequence
\nE1,T1,T2,T3
\nR0 R1,???????
We do get the correct answers in test as well, but it's lagging by one bundle. You'll have to figure out how to manage that according to your pipeline. I use dummy values to keep the seq_len maintained and update them when I get the actual values.
", "rawMarkdown": "We do get the correct answers in test as well, but it's lagging by one bundle. You'll have to figure out how to manage that according to your pipeline. I use dummy values to keep the seq_len maintained and update them when I get the actual values.", "votes": null }, { "id": "1108829", "postDate": "12/11/2020 03:45:41", "content": "@abdurrafae thanks for responses. I am getting to depth of it now.
\nBetween would you like to team up . I have access to multiple V16 gpus (2 to 4) which is helping me to do lot of experiments.
\nIn addition to GPUs ,below are other values i can add for sure.
\n1) Hard work..
\nhere is related post that supports my words
\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/189251
2) Continuous ideas about features that can strengthen model. Once base gets going, additional temporal features
\n3) Came across a New Paper released few months ago with potential of overshooting the SAINT/+ benchmark. In discussion with author for its implementation.
", "rawMarkdown": "abdurrafae thanks for responses. I am getting to depth of it now.\nBetween would you like to team up . I have access to multiple V16 gpus (2 to 4) which is helping me to do lot of experiments. \nIn addition to GPUs ,below are other values i can add for sure.\n1) Hard work..\nhere is related post that supports my words\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/189251\n\n2) Continuous ideas about features that can strengthen model. Once base gets going, additional temporal features\n\n3) Came across a New Paper released few months ago with potential of overshooting the SAINT/+ benchmark. In discussion with author for its implementation.", "votes": null }, { "id": "1108866", "postDate": "12/11/2020 05:01:23", "content": "Thanks for the kind offer, your profile looks really good. However I'll be really busy and probably would not have time to be doing anything other than making minor changes and checking model results for next 2 weeks.
\nLet's connect again around 23rd and see what we can do then.
", "rawMarkdown": "Thanks for the kind offer, your profile looks really good. However I'll be really busy and probably would not have time to be doing anything other than making minor changes and checking model results for next 2 weeks.\n\nLet's connect again around 23rd and see what we can do then.", "votes": null }, { "id": "1108878", "postDate": "12/11/2020 05:25:43", "content": "Thanks Sure I have good bandwidth to work on i think till end ,in osic just iafoss stood by till end rest frustratingly gave up ,
\nMy intent to team up was to try n do some thing on top of what some thing like adding more temporal feature on top of some thing similar and successfully already done .
I'll also be free from 23rd onwards till end of competition so that's at least one good thing going for me.
", "rawMarkdown": "I'll also be free from 23rd onwards till end of competition so that's at least one good thing going for me.", "votes": null }, { "id": "1108951", "postDate": "12/11/2020 07:16:22", "content": "good to hear…
\ntell me one thing
\n1) I see the usage of Masked BCE loss in saint topic discussion, is that useful
\n2 ) I dont do shifting of embeddings as mpware is showing through
response_size = 2\nself.response_embedding = nn.Embedding(response_size + 1, embedding_dim) # +1 to include start token\n...\nx_correctness = self.response_embedding(response_sequence)\n...\n# Add start token to correctness\nx_correctness = torch.roll(x_correctness, shifts=(0, 1, 0), dims=(0, 1, 0)) # Shift right the sequence\nx_correctness[:,0,:] = self.response_size # Start token\nrather i just do shift of inputs and assume embeddings will be passed accordingly.
\nlabel = qa[1:]\nrt=qa[1:-1].copy()\n rt = np.append(np.zeros((1,)),rt)\nwill it make a difference ?
", "rawMarkdown": "good to hear...\ntell me one thing\n1) I see the usage of Masked BCE loss in saint topic discussion, is that useful \n2 ) I dont do shifting of embeddings as mpware is showing through \n```\nresponse_size = 2\nself.response_embedding = nn.Embedding(response_size + 1, embedding_dim) # +1 to include start token\n...\nx_correctness = self.response_embedding(response_sequence)\n...\n# Add start token to correctness\nx_correctness = torch.roll(x_correctness, shifts=(0, 1, 0), dims=(0, 1, 0)) # Shift right the sequence\nx_correctness[:,0,:] = self.response_size # Start token\n```\nrather i just do shift of inputs and assume embeddings will be passed accordingly.\n```\nlabel = qa[1:]\nrt=qa[1:-1].copy()\n rt = np.append(np.zeros((1,)),rt)\n```\nwill it make a difference ?", "votes": null }, { "id": "1108974", "postDate": "12/11/2020 07:59:43", "content": "For the shifting. I believe doing it either way is okay, I had implemented it by shifting the embeddings earlier but then did it by shifting the inputs in a later re-factoring of my code.
\nFor loss, I initially didn't use any mask (as no lectures were included). I have added a mask for loss for the lecture interactions now.
", "rawMarkdown": "For the shifting. I believe doing it either way is okay, I had implemented it by shifting the embeddings earlier but then did it by shifting the inputs in a later re-factoring of my code.\n\nFor loss, I initially didn't use any mask (as no lectures were included). I have added a mask for loss for the lecture interactions now.", "votes": null }, { "id": "1117917", "postDate": "12/18/2020 15:28:33", "content": "@abdurrafae my saint baseline stands at 77.3 ,trying few more things .
\nSo just thought of checking are you open for teaming up next week .
@abdurrafae you are doing good alone . But still just last ask.
\nWould you like to team up. If solo gold/ silver medal standing is of less importance to you or you think reaching winning range/gold zone could be difficult for you.
\nour SAINT pipeline got us 78.8
\nwe are fixing small bugs bugs in saint plus (current is 79.3) and we have good lgb 79.2.
\nwe may reach to winning range of 81+ after mixing your model with our lgb
Thanks man, I'll be going solo for this one I think.
\nBest of luck mate
\n\nThanks man, I'll be going solo for this one I think.
\n
\n Best of luck mate
No issues i appreciate your hard work till date..
\nmay u reach winning alone in last few days.
I believe for SAINT/SAINT+ the embeddings used are
\nIn Saint plus they say part but in Saint its exercise category, while part is section of assessment comprising of various question
", "votes": null, "replies": [] }, { "id": 1105688, "author_name": "abdurrafae", "author_url": "", "post_date": "12/08/2020 05:05:28", "content": "Oh, I might have mixed them then. Thanks for clarifying. I don't see a reason for not using both at the same time. They both incorporate different information to the model.
", "votes": null, "replies": [] }, { "id": 1105696, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/08/2020 05:16:25", "content": "@abdurrafae wcm , I do see good improvement upon adding on top of Excercise and Output seq , further embeddings
\nis your current LB score based on SAINT or LGB..
\nIf yes how much CV you get with SAINT Base version.
\nThey also mention about Start token Embeddings i dint quite get that ..
", "votes": null, "replies": [] }, { "id": 1105821, "author_name": "abdurrafae", "author_url": "", "post_date": "12/08/2020 08:27:12", "content": "I got upto 0.775 with SAINT+ on validation data. my validation is closely co-related to LB +/- 0.001.
\nI only trained it for one session of Kaggle environment so not sure if it would have improved further or not, but the gain in each epoch was too low for me to invest in it further.
\nThe issue I was having was that the model diverged when I increase number of layers or model dimensions. So I built it from scratch again and made some modifications along the way. Now it doesn't match the exact SAINT/SAINT+ but it's quite similar.
\nFor start token, I was just padding the sequence with zeros on left (with length 1) and cropping (with length 1) from right
", "votes": null, "replies": [] }, { "id": 1105889, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/08/2020 09:48:41", "content": "ok thanks , in Saint i get 75.75 as cv yet to test cv .
\nChallenge m finding is how do I handle user ID not in test as i wont get the same in responses that iterations .
In Saint + did you add lag time also?
While addong prior elapsed time were u able to get right chronology of question .there is some discussion on going about the caveat involved if rely just on default sequence .
What is purpose of start token ,I dint gave any.
1) I use dummy responses for new users till I get them in next iteration.
\n2) I used a feature based on lag time, so not lag time exactly.
\n3) I used lag time as absolute difference between bundles (i.e. disregarding the time elapsed field to calculate this)
\n4) Start token is to mis-align your questions and response streams since at test time you'll have access to question (1:t) and responses (1:t-1).
@abdurrafae how did perform merge operation between train csv and any other df during inference
", "votes": null, "replies": [] }, { "id": 1107510, "author_name": "abdurrafae", "author_url": "", "post_date": "12/09/2020 18:39:57", "content": "I use pandas for that. I only have 1 dataframe to join to, so it doesn't take that long
", "votes": null, "replies": [] }, { "id": 1107512, "author_name": "abdurrafae", "author_url": "", "post_date": "12/09/2020 18:40:49", "content": "I don't have the train_df at inference, I've a customized object that stores data for each user.
", "votes": null, "replies": [] }, { "id": 1108158, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/10/2020 10:49:27", "content": "Thanks
\n1) https://www.kaggle.com/mpware/sakt-fork/
\nin this kernel what is significance of starting question from 2: rather than from 0 or 1:
questions = np.append(q[2:], [target_id])
2) what was your first sub score with SAINT .. i get good cv with base SAINT but LB not aligned with CV. 76.42/ 74.7 following the same input indexing as above and not accumulating the Response history ,to start with.
\n not sure if am making any mistake in inference
No idea about the SAKT kernel.
\nI had a validation score of 0.775 with SAINT, didn't get an LB on that.
\nMy Validation score is closely related to LB with +/- 0.001
from paper\nThe encoder takes the sequence\nEe = [Ee1,··· ,Eek] of exercise embeddings and feeds the processed output O = [O1,··· ,Ok\n] to the decoder. The decoder\ntakes O and another sequential input R\ne = [S,Re1,··· ,Rek−1] of\nresponse embeddings with the start token embedding S, and\nproduces the predicted responses ˆr = [rˆ1,··· ,rˆk\n].\nI had one more doubt wrt to SAINT.. for any user id if i understand well for any current Exercise E1 model has got previous response as correct or not but in the case of test , when we add in Test questions we dont have Test users exercise history for test question ..
\nAt any given user iteration
\neg: User 1- Test question T1,T2,T3.
\nExercise seq could become :
\nE1(from Train or val) -T1,T2,T3
\nResponse sequence
\nE1,T1,T2,T3
\nR0 R1,???????
We do get the correct answers in test as well, but it's lagging by one bundle. You'll have to figure out how to manage that according to your pipeline. I use dummy values to keep the seq_len maintained and update them when I get the actual values.
", "votes": null, "replies": [] }, { "id": 1108829, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/11/2020 03:45:41", "content": "@abdurrafae thanks for responses. I am getting to depth of it now.
\nBetween would you like to team up . I have access to multiple V16 gpus (2 to 4) which is helping me to do lot of experiments.
\nIn addition to GPUs ,below are other values i can add for sure.
\n1) Hard work..
\nhere is related post that supports my words
\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/189251
2) Continuous ideas about features that can strengthen model. Once base gets going, additional temporal features
\n3) Came across a New Paper released few months ago with potential of overshooting the SAINT/+ benchmark. In discussion with author for its implementation.
", "votes": null, "replies": [] }, { "id": 1108866, "author_name": "abdurrafae", "author_url": "", "post_date": "12/11/2020 05:01:23", "content": "Thanks for the kind offer, your profile looks really good. However I'll be really busy and probably would not have time to be doing anything other than making minor changes and checking model results for next 2 weeks.
\nLet's connect again around 23rd and see what we can do then.
", "votes": null, "replies": [] }, { "id": 1108878, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/11/2020 05:25:43", "content": "Thanks Sure I have good bandwidth to work on i think till end ,in osic just iafoss stood by till end rest frustratingly gave up ,
\nMy intent to team up was to try n do some thing on top of what some thing like adding more temporal feature on top of some thing similar and successfully already done .
I'll also be free from 23rd onwards till end of competition so that's at least one good thing going for me.
", "votes": null, "replies": [] }, { "id": 1108951, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/11/2020 07:16:22", "content": "good to hear…
\ntell me one thing
\n1) I see the usage of Masked BCE loss in saint topic discussion, is that useful
\n2 ) I dont do shifting of embeddings as mpware is showing through
response_size = 2\nself.response_embedding = nn.Embedding(response_size + 1, embedding_dim) # +1 to include start token\n...\nx_correctness = self.response_embedding(response_sequence)\n...\n# Add start token to correctness\nx_correctness = torch.roll(x_correctness, shifts=(0, 1, 0), dims=(0, 1, 0)) # Shift right the sequence\nx_correctness[:,0,:] = self.response_size # Start token\nrather i just do shift of inputs and assume embeddings will be passed accordingly.
\nlabel = qa[1:]\nrt=qa[1:-1].copy()\n rt = np.append(np.zeros((1,)),rt)\nwill it make a difference ?
", "votes": null, "replies": [] }, { "id": 1108974, "author_name": "abdurrafae", "author_url": "", "post_date": "12/11/2020 07:59:43", "content": "For the shifting. I believe doing it either way is okay, I had implemented it by shifting the embeddings earlier but then did it by shifting the inputs in a later re-factoring of my code.
\nFor loss, I initially didn't use any mask (as no lectures were included). I have added a mask for loss for the lecture interactions now.
", "votes": null, "replies": [] }, { "id": 1117917, "author_name": "jaideepvalani", "author_url": "", "post_date": "12/18/2020 15:28:33", "content": "@abdurrafae my saint baseline stands at 77.3 ,trying few more things .
\nSo just thought of checking are you open for teaming up next week .
@abdurrafae you are doing good alone . But still just last ask.
\nWould you like to team up. If solo gold/ silver medal standing is of less importance to you or you think reaching winning range/gold zone could be difficult for you.
\nour SAINT pipeline got us 78.8
\nwe are fixing small bugs bugs in saint plus (current is 79.3) and we have good lgb 79.2.
\nwe may reach to winning range of 81+ after mixing your model with our lgb
Thanks man, I'll be going solo for this one I think.
\nBest of luck mate
\n\nThanks man, I'll be going solo for this one I think.
\n
\n Best of luck mate
No issues i appreciate your hard work till date..
\nmay u reach winning alone in last few days.