According to the explanation of App of Santa-toeic, questions and lectures are personalized.
\nIt says the AI analyzes the weaknesses of users and gives priority to the questions and lectures from these areas. With MYNOTE function, the user can focus on the \"problems you made a mistake\" and \"problems of the part you are not good at\".
@aravindpadman has found that the mean accuracy of first attempt of repeated questions is 0.23. I think it is too low (same as random choice 1/4!) compared to the over all mean around 0.6.
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194266
\nI don't have a good idea for utilizing the personalization as we don't know whether the questions will be repeated for test set. Any comments are welcome.
", "messages": [ { "id": "1092631", "postDate": "11/27/2020 03:58:24", "content": "According to the explanation of App of Santa-toeic, questions and lectures are personalized.
\nIt says the AI analyzes the weaknesses of users and gives priority to the questions and lectures from these areas. With MYNOTE function, the user can focus on the \"problems you made a mistake\" and \"problems of the part you are not good at\".
@aravindpadman has found that the mean accuracy of first attempt of repeated questions is 0.23. I think it is too low (same as random choice 1/4!) compared to the over all mean around 0.6.
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194266
\nI don't have a good idea for utilizing the personalization as we don't know whether the questions will be repeated for test set. Any comments are welcome.
", "rawMarkdown": "According to the explanation of App of Santa-toeic, questions and lectures are personalized.\nIt says the AI analyzes the weaknesses of users and gives priority to the questions and lectures from these areas. With MYNOTE function, the user can focus on the \"problems you made a mistake\" and \"problems of the part you are not good at\".\n\nhttps://apps.apple.com/jp/app/santa-toeic-ai%E3%82%92%E6%B4%BB%E7%94%A8%E3%81%97%E3%81%9Ftoeic%E5%AD%A6%E7%BF%92%E3%82%A2%E3%83%97%E3%83%AA/id1456346606\n(In Japanese)\n\n@aravindpadman has found that the mean accuracy of first attempt of repeated questions is 0.23. I think it is too low (same as random choice 1/4!) compared to the over all mean around 0.6.\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194266\n\nI don't have a good idea for utilizing the personalization as we don't know whether the questions will be repeated for test set. Any comments are welcome.", "votes": null }, { "id": "1093406", "postDate": "11/27/2020 17:27:39", "content": "Agree that it is personalized - my reading of the Riiid website and related links indicates they have made a serious attempt to 'personalize' the training flow. If you examine their report for 2019 on the \"test\" they have identified some very strong relationships between the test score and background data supplied by each user. For example, users with little use for English in their daily working life will score much worse than those who need to use English daily. I would assume that Riiid uses the background info from the very early training questions.
\nRepeating questions that users have failed to answer correctly would seem a very early and basic step for any attempt to personalize training.
\nEvaluating how well users have 'learned' the right answer based on short term memory would be my next way to personalize a training flow (repeat question within a short time span of the first failed attempt).
\nEvaluating how well users have 'learned' the right answer based on long term memory would than be my next way to personalize a training flow (repeat question within a long time span after they got it right).
\nBased on how hard the question is (data for all users) I would probably establish different metrics for how often to 'repeat' questions. I would likely want to repeat a very hard question at least one time to make sure the user did not just get in a good guess.
\nDoes their process have at least these basic thoughts - I would bet that they have those basics (much sooner than I will bet that the Pittsburgh Steelers go 16-0) .
\nSo my belief - the data set is the result of a reasonably good AI managed training process that is personalized from the very beginning by user background data.
\n\n\n", "rawMarkdown": "Agree that it is personalized - my reading of the Riiid website and related links indicates they have made a serious attempt to 'personalize' the training flow. If you examine their report for 2019 on the \"test\" they have identified some very strong relationships between the test score and background data supplied by each user. For example, users with little use for English in their daily working life will score much worse than those who need to use English daily. I would assume that Riiid uses the background info from the very early training questions.\n\nRepeating questions that users have failed to answer correctly would seem a very early and basic step for any attempt to personalize training.\n\nEvaluating how well users have 'learned' the right answer based on short term memory would be my next way to personalize a training flow (repeat question within a short time span of the first failed attempt).\n\nEvaluating how well users have 'learned' the right answer based on long term memory would than be my next way to personalize a training flow (repeat question within a long time span after they got it right).\n\nBased on how hard the question is (data for all users) I would probably establish different metrics for how often to 'repeat' questions. I would likely want to repeat a very hard question at least one time to make sure the user did not just get in a good guess.\n\nDoes their process have at least these basic thoughts - I would bet that they have those basics (much sooner than I will bet that the Pittsburgh Steelers go 16-0) .\n\nSo my belief - the data set is the result of a reasonably good AI managed training process that is personalized from the very beginning by user background data.\n\n[Paper](https://arxiv.org/abs/1912.03072)\n\n> With advances in Artificial Intelligence in Education (AIEd) and the ever-growing scale of Interactive Educational Systems (IESs), data-driven approach has become a common recipe for various tasks such as knowledge tracing and learning path recommendation. Unfortunately, collecting real students' interaction data is often challenging, which results in the lack of public large-scale benchmark dataset reflecting a wide variety of student behaviors in modern IESs. Although several datasets, such as ASSISTments, Junyi Academy, Synthetic and STATICS, are publicly available and widely used, they are not large enough to leverage the full potential of state-of-the-art data-driven models and limits the recorded behaviors to question-solving activities. To this end, we introduce EdNet, a large-scale hierarchical dataset of diverse student activities collected by Santa, a multi-platform self-study solution equipped with artificial intelligence tutoring system. EdNet contains 131,441,538 interactions from 784,309 students collected over more than 2 years, which is the largest among the ITS datasets released to the public so far. Unlike existing datasets, EdNet provides a wide variety of student actions ranging from question-solving to lecture consumption and item purchasing. Also, EdNet has a hierarchical structure where the student actions are divided into 4 different levels of abstractions. The features of EdNet are domain-agnostic, allowing EdNet to be extended to different domains easily. The dataset is publicly released under Creative Commons Attribution-NonCommercial 4.0 International license for research purposes. We plan to host challenges in multiple AIEd tasks with EdNet to provide a common ground for the fair comparison between different state of the art models and encourage the development of practical and effective methods.", "votes": null } ], "comments": [ { "id": 1093406, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/27/2020 17:27:39", "content": "With advances in Artificial Intelligence in Education (AIEd) and the ever-growing scale of Interactive Educational Systems (IESs), data-driven approach has become a common recipe for various tasks such as knowledge tracing and learning path recommendation. Unfortunately, collecting real students' interaction data is often challenging, which results in the lack of public large-scale benchmark dataset reflecting a wide variety of student behaviors in modern IESs. Although several datasets, such as ASSISTments, Junyi Academy, Synthetic and STATICS, are publicly available and widely used, they are not large enough to leverage the full potential of state-of-the-art data-driven models and limits the recorded behaviors to question-solving activities. To this end, we introduce EdNet, a large-scale hierarchical dataset of diverse student activities collected by Santa, a multi-platform self-study solution equipped with artificial intelligence tutoring system. EdNet contains 131,441,538 interactions from 784,309 students collected over more than 2 years, which is the largest among the ITS datasets released to the public so far. Unlike existing datasets, EdNet provides a wide variety of student actions ranging from question-solving to lecture consumption and item purchasing. Also, EdNet has a hierarchical structure where the student actions are divided into 4 different levels of abstractions. The features of EdNet are domain-agnostic, allowing EdNet to be extended to different domains easily. The dataset is publicly released under Creative Commons Attribution-NonCommercial 4.0 International license for research purposes. We plan to host challenges in multiple AIEd tasks with EdNet to provide a common ground for the fair comparison between different state of the art models and encourage the development of practical and effective methods.
\n
Agree that it is personalized - my reading of the Riiid website and related links indicates they have made a serious attempt to 'personalize' the training flow. If you examine their report for 2019 on the \"test\" they have identified some very strong relationships between the test score and background data supplied by each user. For example, users with little use for English in their daily working life will score much worse than those who need to use English daily. I would assume that Riiid uses the background info from the very early training questions.
\nRepeating questions that users have failed to answer correctly would seem a very early and basic step for any attempt to personalize training.
\nEvaluating how well users have 'learned' the right answer based on short term memory would be my next way to personalize a training flow (repeat question within a short time span of the first failed attempt).
\nEvaluating how well users have 'learned' the right answer based on long term memory would than be my next way to personalize a training flow (repeat question within a long time span after they got it right).
\nBased on how hard the question is (data for all users) I would probably establish different metrics for how often to 'repeat' questions. I would likely want to repeat a very hard question at least one time to make sure the user did not just get in a good guess.
\nDoes their process have at least these basic thoughts - I would bet that they have those basics (much sooner than I will bet that the Pittsburgh Steelers go 16-0) .
\nSo my belief - the data set is the result of a reasonably good AI managed training process that is personalized from the very beginning by user background data.
\n\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1092631": "According to the explanation of App of Santa-toeic, questions and lectures are personalized.\nIt says the AI analyzes the weaknesses of users and gives priority to the questions and lectures from these areas. With MYNOTE function, the user can focus on the \"problems you made a mistake\" and \"problems of the part you are not good at\".\n\nhttps://apps.apple.com/jp/app/santa-toeic-ai%E3%82%92%E6%B4%BB%E7%94%A8%E3%81%97%E3%81%9Ftoeic%E5%AD%A6%E7%BF%92%E3%82%A2%E3%83%97%E3%83%AA/id1456346606\n(In Japanese)\n\n@aravindpadman has found that the mean accuracy of first attempt of repeated questions is 0.23. I think it is too low (same as random choice 1/4!) compared to the over all mean around 0.6.\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194266\n\nI don't have a good idea for utilizing the personalization as we don't know whether the questions will be repeated for test set. Any comments are welcome.", "1093406": "Agree that it is personalized - my reading of the Riiid website and related links indicates they have made a serious attempt to 'personalize' the training flow. If you examine their report for 2019 on the \"test\" they have identified some very strong relationships between the test score and background data supplied by each user. For example, users with little use for English in their daily working life will score much worse than those who need to use English daily. I would assume that Riiid uses the background info from the very early training questions.\n\nRepeating questions that users have failed to answer correctly would seem a very early and basic step for any attempt to personalize training.\n\nEvaluating how well users have 'learned' the right answer based on short term memory would be my next way to personalize a training flow (repeat question within a short time span of the first failed attempt).\n\nEvaluating how well users have 'learned' the right answer based on long term memory would than be my next way to personalize a training flow (repeat question within a long time span after they got it right).\n\nBased on how hard the question is (data for all users) I would probably establish different metrics for how often to 'repeat' questions. I would likely want to repeat a very hard question at least one time to make sure the user did not just get in a good guess.\n\nDoes their process have at least these basic thoughts - I would bet that they have those basics (much sooner than I will bet that the Pittsburgh Steelers go 16-0) .\n\nSo my belief - the data set is the result of a reasonably good AI managed training process that is personalized from the very beginning by user background data.\n\n[Paper](https://arxiv.org/abs/1912.03072)\n\n> With advances in Artificial Intelligence in Education (AIEd) and the ever-growing scale of Interactive Educational Systems (IESs), data-driven approach has become a common recipe for various tasks such as knowledge tracing and learning path recommendation. Unfortunately, collecting real students' interaction data is often challenging, which results in the lack of public large-scale benchmark dataset reflecting a wide variety of student behaviors in modern IESs. Although several datasets, such as ASSISTments, Junyi Academy, Synthetic and STATICS, are publicly available and widely used, they are not large enough to leverage the full potential of state-of-the-art data-driven models and limits the recorded behaviors to question-solving activities. To this end, we introduce EdNet, a large-scale hierarchical dataset of diverse student activities collected by Santa, a multi-platform self-study solution equipped with artificial intelligence tutoring system. EdNet contains 131,441,538 interactions from 784,309 students collected over more than 2 years, which is the largest among the ITS datasets released to the public so far. Unlike existing datasets, EdNet provides a wide variety of student actions ranging from question-solving to lecture consumption and item purchasing. Also, EdNet has a hierarchical structure where the student actions are divided into 4 different levels of abstractions. The features of EdNet are domain-agnostic, allowing EdNet to be extended to different domains easily. The dataset is publicly released under Creative Commons Attribution-NonCommercial 4.0 International license for research purposes. We plan to host challenges in multiple AIEd tasks with EdNet to provide a common ground for the fair comparison between different state of the art models and encourage the development of practical and effective methods." }, "source": "meta" }With advances in Artificial Intelligence in Education (AIEd) and the ever-growing scale of Interactive Educational Systems (IESs), data-driven approach has become a common recipe for various tasks such as knowledge tracing and learning path recommendation. Unfortunately, collecting real students' interaction data is often challenging, which results in the lack of public large-scale benchmark dataset reflecting a wide variety of student behaviors in modern IESs. Although several datasets, such as ASSISTments, Junyi Academy, Synthetic and STATICS, are publicly available and widely used, they are not large enough to leverage the full potential of state-of-the-art data-driven models and limits the recorded behaviors to question-solving activities. To this end, we introduce EdNet, a large-scale hierarchical dataset of diverse student activities collected by Santa, a multi-platform self-study solution equipped with artificial intelligence tutoring system. EdNet contains 131,441,538 interactions from 784,309 students collected over more than 2 years, which is the largest among the ITS datasets released to the public so far. Unlike existing datasets, EdNet provides a wide variety of student actions ranging from question-solving to lecture consumption and item purchasing. Also, EdNet has a hierarchical structure where the student actions are divided into 4 different levels of abstractions. The features of EdNet are domain-agnostic, allowing EdNet to be extended to different domains easily. The dataset is publicly released under Creative Commons Attribution-NonCommercial 4.0 International license for research purposes. We plan to host challenges in multiple AIEd tasks with EdNet to provide a common ground for the fair comparison between different state of the art models and encourage the development of practical and effective methods.
\n