![\"in](\"https://i.ibb.co/DtzQ7h5/Capture3.png\")
In the paper SAINT+: A Transformer-based model for correctness prediction, the authors use a elapsed time between a response and its corresponding questions. Like
And they claim
\n\n\nIf the student does not have enough knowledge and skills for the exercise, it would be hard to respond correctly within the recommended time limit. Hence, elapsed time provides strong evidence for a student’s proficiency in knowledge and skills, and student’s understanding of concepts associated with the exercise.
\n
However, in reality, when we are at the moment having a question in hand and we want to predict if a student will be able to answer correctly, this elapsed time information shouldn't exist at that time.
For example, if the real life application of the model is to accurately predict how students will perform on future interactions and make students enjoy the benefits of a personalized learning experience, we must predict the probability of answer correction among a set of questions, before the students answering it (them).
\nIn this competition, we have prior_question_elapsed_time, but this is the (averaged) time used to answer each question in the PREVIOUS bundle.
So I am confused if it is a reasonable solution proposed in SAINT+: A Transformer-based model for correctness prediction. I don't mean to criticize, but I am wondering other people's thought on this. Thank you.
", "messages": [ { "id": "1081198", "postDate": "11/16/2020 22:56:17", "content": "In the paper SAINT+: A Transformer-based model for correctness prediction, the authors use a elapsed time between a response and its corresponding questions. Like
And they claim
\n\n\nIf the student does not have enough knowledge and skills for the exercise, it would be hard to respond correctly within the recommended time limit. Hence, elapsed time provides strong evidence for a student’s proficiency in knowledge and skills, and student’s understanding of concepts associated with the exercise.
\n
However, in reality, when we are at the moment having a question in hand and we want to predict if a student will be able to answer correctly, this elapsed time information shouldn't exist at that time.
For example, if the real life application of the model is to accurately predict how students will perform on future interactions and make students enjoy the benefits of a personalized learning experience, we must predict the probability of answer correction among a set of questions, before the students answering it (them).
\nIn this competition, we have prior_question_elapsed_time, but this is the (averaged) time used to answer each question in the PREVIOUS bundle.
So I am confused if it is a reasonable solution proposed in SAINT+: A Transformer-based model for correctness prediction. I don't mean to criticize, but I am wondering other people's thought on this. Thank you.
", "rawMarkdown": "In the paper [SAINT+: A Transformer-based model for correctness prediction](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193250), the authors use a `elapsed time` between a response and its corresponding questions. Like\n\n\n\nAnd they claim \n\n> If the student does not have enough knowledge and skills for the exercise, it would be hard to respond correctly within the recommended time limit. Hence, elapsed time provides strong evidence for a student’s proficiency in knowledge and skills, and student’s understanding of concepts associated with the exercise.\n\nHowever, in reality, when we are at the moment having a question in hand and we want to predict if a student will be able to answer correctly, this `elapsed time` information shouldn't exist at that time.\n\nFor example, if the real life application of the model is to accurately predict how students will perform on future interactions and make students enjoy the benefits of a personalized learning experience, we must predict the probability of answer correction among a set of questions, before the students answering it (them).\n\nIn this competition, we have `prior_question_elapsed_time`, but this is the (averaged) time used to answer each question in the PREVIOUS bundle.\n\nSo I am confused if it is a reasonable solution proposed in [SAINT+: A Transformer-based model for correctness prediction](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193250). I don't mean to criticize, but I am wondering other people's thought on this. Thank you.", "votes": null }, { "id": "1081849", "postDate": "11/17/2020 10:42:23", "content": "This was exactly my thought while reading the paper :) We should only use prior_question_elapsed_time, while they used present_question_elapsed_time.
If I understood correctly: in the paper they add a start token to the elapsed time like they did with the response variable. So for example the current exercise id e2 corresponds to et1 i.e. the prior question elapsed time ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F474194%2F3b93deaf5be6f03c0bb85e672d6ccdd3%2FScreenshot%202020-11-17%20at%2012.15.18.png?generation=1605611931952141&alt=media\")
You are right, thank you for helping
", "rawMarkdown": "You are right, thank you for helping", "votes": null }, { "id": "1126872", "postDate": "12/26/2020 04:08:55", "content": "I have one question, in saint++ paper, it use nn.embedding on elapsed time, but the elapsed time is consequent not dispersed value, how does it embedded the elapsed time ? can you explain it plz? @yihdarshieh @rafiko1
", "rawMarkdown": "I have one question, in saint++ paper, it use nn.embedding on elapsed time, but the elapsed time is consequent not dispersed value, how does it embedded the elapsed time ? can you explain it plz? @yihdarshieh @rafiko1", "votes": null }, { "id": "1128603", "postDate": "12/27/2020 15:37:47", "content": "@bacicnikola
\nwhat is right scale value one should use..
\n/100060 or 3001000
I see in some notebooks it is also 300*1000
\nsecondly..
\nin paper they say discretized into distinct integer minutes.. does it means creating categories of continuous minute values first using any of bin method ?
This was exactly my thought while reading the paper :) We should only use prior_question_elapsed_time, while they used present_question_elapsed_time.
@bacicnikola
\nwhat is right scale value one should use..
\n/100060 or 3001000
I see in some notebooks it is also 300*1000
\nsecondly..
\nin paper they say discretized into distinct integer minutes.. does it means creating categories of continuous minute values first using any of bin method ?
If I understood correctly: in the paper they add a start token to the elapsed time like they did with the response variable. So for example the current exercise id e2 corresponds to et1 i.e. the prior question elapsed time
You are right, thank you for helping
", "votes": null, "replies": [] }, { "id": 1126872, "author_name": "cswwp347724", "author_url": "", "post_date": "12/26/2020 04:08:55", "content": "I have one question, in saint++ paper, it use nn.embedding on elapsed time, but the elapsed time is consequent not dispersed value, how does it embedded the elapsed time ? can you explain it plz? @yihdarshieh @rafiko1
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1081198": "In the paper [SAINT+: A Transformer-based model for correctness prediction](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193250), the authors use a `elapsed time` between a response and its corresponding questions. Like\n\n\n\nAnd they claim \n\n> If the student does not have enough knowledge and skills for the exercise, it would be hard to respond correctly within the recommended time limit. Hence, elapsed time provides strong evidence for a student’s proficiency in knowledge and skills, and student’s understanding of concepts associated with the exercise.\n\nHowever, in reality, when we are at the moment having a question in hand and we want to predict if a student will be able to answer correctly, this `elapsed time` information shouldn't exist at that time.\n\nFor example, if the real life application of the model is to accurately predict how students will perform on future interactions and make students enjoy the benefits of a personalized learning experience, we must predict the probability of answer correction among a set of questions, before the students answering it (them).\n\nIn this competition, we have `prior_question_elapsed_time`, but this is the (averaged) time used to answer each question in the PREVIOUS bundle.\n\nSo I am confused if it is a reasonable solution proposed in [SAINT+: A Transformer-based model for correctness prediction](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193250). I don't mean to criticize, but I am wondering other people's thought on this. Thank you.", "1081849": "This was exactly my thought while reading the paper :) We should only use `prior_question_elapsed_time`, while they used `present_question_elapsed_time`.", "1081891": "If I understood correctly: in the paper they add a start token to the elapsed time like they did with the response variable. So for example the current exercise id `e2` corresponds to `et1` i.e. the prior question elapsed time ", "1083390": "You are right, thank you for helping", "1126872": "I have one question, in saint++ paper, it use nn.embedding on elapsed time, but the elapsed time is consequent not dispersed value, how does it embedded the elapsed time ? can you explain it plz? @yihdarshieh @rafiko1", "1128603": "bacicnikola \nwhat is right scale value one should use..\n/1000*60 or 300*1000\n\nI see in some notebooks it is also 300*1000\n\nsecondly..\nin paper they say discretized into distinct integer minutes.. does it means creating categories of continuous minute values first using any of bin method ?" }, "source": "meta" }