![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F781502%2F6eaa6ef40ef72d203ffec8e97e0b05c9%2Fnewplot%20(4).png?generation=1604089560900300&alt=media\")
I was creating some features that calculated the cumulative and rolling totals of questions answered correctly and put the charts below together to visually inspect whether the calculations were being performed correctly. They turned out to be a useful way to gain some insight into users' learning progress.
\nThere first chart shows the cumulative number of task containers and correclty and incorrectly answered questions over time. You can tell from the task_container_id line that this user answered approximately 120 questions over a total period of almost 600 hours. It looks like they were more active in the first 120 hours and that their performance improved between 80 and 120 hours. Overall, by looking at the difference between answered_correctly_cumusum and answered_incorrecltly_cumsum at the end of the time period, it looks like this user answered about twice as many questions correctly as incorrectly.
The second chart shows the rolling totals of correctly and incorrectly answered questions over the preceding 10 task containers along with whether questions were answered correctly or incorrectly on an individual basis. It looks like this user had several episodes of strong performance starting around 40 task containers. It would be interesting to investigate the causes of the changes in performance to distinquish between learning, i.e., better performance on questions with similar characteristics, and changes in question characteristics (perhaps adding companion charts for question part and difficulty, for starters). It seems likely that questions are delivered in sequences with similar characteristics and that difficulty varies depending on user performance. Once a user's performance improves to the point at which they are determined to have achieved mastery of the characteristics of the sequence, perhaps they would begin another sequence of questions with different characteristics. Alternatively, if the user's performance did not improve, perhaps they would be delivered less difficult questions to allow them to build their knowledge base before progressing to more difficult questions.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F781502%2Fb078941b8532d8fc19f091cfa65a626a%2Fnewplot%20(6).png?generation=1604089664139102&alt=media\")
I was creating some features that calculated the cumulative and rolling totals of questions answered correctly and put the charts below together to visually inspect whether the calculations were being performed correctly. They turned out to be a useful way to gain some insight into users' learning progress.
\nThere first chart shows the cumulative number of task containers and correclty and incorrectly answered questions over time. You can tell from the task_container_id line that this user answered approximately 120 questions over a total period of almost 600 hours. It looks like they were more active in the first 120 hours and that their performance improved between 80 and 120 hours. Overall, by looking at the difference between answered_correctly_cumusum and answered_incorrecltly_cumsum at the end of the time period, it looks like this user answered about twice as many questions correctly as incorrectly.
The second chart shows the rolling totals of correctly and incorrectly answered questions over the preceding 10 task containers along with whether questions were answered correctly or incorrectly on an individual basis. It looks like this user had several episodes of strong performance starting around 40 task containers. It would be interesting to investigate the causes of the changes in performance to distinquish between learning, i.e., better performance on questions with similar characteristics, and changes in question characteristics (perhaps adding companion charts for question part and difficulty, for starters). It seems likely that questions are delivered in sequences with similar characteristics and that difficulty varies depending on user performance. Once a user's performance improves to the point at which they are determined to have achieved mastery of the characteristics of the sequence, perhaps they would begin another sequence of questions with different characteristics. Alternatively, if the user's performance did not improve, perhaps they would be delivered less difficult questions to allow them to build their knowledge base before progressing to more difficult questions.
\n
@calebeverett Good stats for this user: 5382
\nI think the inferences make sense but it will be interesting to know how these stats come out for a wider group of users, not just one. This will be different for a different user.
", "rawMarkdown": "calebeverett Good stats for this user: 5382\n\nI think the inferences make sense but it will be interesting to know how these stats come out for a wider group of users, not just one. This will be different for a different user.", "votes": null }, { "id": "1065095", "postDate": "10/30/2020 21:03:45", "content": "Yes, perhaps some common patterns would emerge that could be developed in to features. I was finding it interesting to parse through a bunch of graphs like this for individual users to gain a better understanding of the data, similar to how you might parse though a bunch of images in a classification project.
", "rawMarkdown": "Yes, perhaps some common patterns would emerge that could be developed in to features. I was finding it interesting to parse through a bunch of graphs like this for individual users to gain a better understanding of the data, similar to how you might parse though a bunch of images in a classification project.", "votes": null }, { "id": "1065205", "postDate": "10/31/2020 02:59:42", "content": "Plots and the idea is beautiful ! But i am not really sure that this user has 500 hours + elapsed time. Maybe you have a logic to determine elapsed hours
", "rawMarkdown": "Plots and the idea is beautiful ! But i am not really sure that this user has 500 hours + elapsed time. Maybe you have a logic to determine elapsed hours", "votes": null }, { "id": "1065214", "postDate": "10/31/2020 03:13:46", "content": "That's probably faulty terminology on my part. That is just the timestamp, so what it shows in the first graph is that they used the system over a long period of time, not necessarily continuously, answering questions at a consistent rate. The flat lines are essentially the gaps between sessions, when answer count isn't increasing. Maybe there is a feature there - could be a difference in performance after long gaps?
", "rawMarkdown": "That's probably faulty terminology on my part. That is just the timestamp, so what it shows in the first graph is that they used the system **over** a long period of time, not necessarily continuously, answering questions at a consistent rate. The flat lines are essentially the gaps between sessions, when answer count isn't increasing. Maybe there is a feature there - could be a difference in performance after long gaps?", "votes": null }, { "id": "1065247", "postDate": "10/31/2020 04:36:18", "content": "\n\nit looks like this user answered about twice as many questions correctly as incorrectly
\n
That's what you would expect if any random user is chosen from the data since the overall correct rate of the dataset is ~ 65% (which is same as getting twice as many questions correctly as incorrectly)
", "rawMarkdown": "> it looks like this user answered about twice as many questions correctly as incorrectly\n\nThat's what you would expect if any random user is chosen from the data since the overall correct rate of the dataset is ~ 65% (which is same as getting twice as many questions correctly as incorrectly)", "votes": null }, { "id": "1065255", "postDate": "10/31/2020 04:52:38", "content": "Plus, Another observation in case you didn't notice is that A total of 30 questions is given to test user skills when they start using the app as a starter test for using Riiidi platform AFAIU.
\n![\"image_from_santa\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F835774%2F42f22abccf1a0450443564a3aac831fa%2FScreenshot%202020-10-31%20at%2010.21.27%20AM.png?generation=1604119908011993&alt=media\")
I noticed the ramp at the beginning of the sequence as well. The initial diagnostic questions make sense and maybe there is some evidence of that in the cumulative chart as indicated by this users improved performance starting around 60 elapsed hours. But, I think most of the reason for the ramps at the very beginning is because of the way the rolling totals are calculated. They start out at zero since there were no prior questions and don't stabilize until questions from ten containers have been answered. I was thinking about whether the rolling totals should start out at five for both, or maybe 6.5 correct and 3.5 incorrect to reflect the average across all users for all questions. But maybe the interaction with task_container_id makes that unnecessary.
Could you share the website where you found this information? I tried myself the diagnosis test on the Santa page and, based on the background questions, I only had to solve 9 questions. Do you know if these 30 questions are presented after you purchase the service?
", "rawMarkdown": "Could you share the website where you found this information? I tried myself the diagnosis test on the [Santa page](https://aitutorsanta.com/intro) and, based on the background questions, I only had to solve 9 questions. Do you know if these 30 questions are presented after you purchase the service?", "votes": null }, { "id": "1067782", "postDate": "11/02/2020 17:53:17", "content": "Well I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.
\nRef (Go to the section of \"With the most accurate AI tutor\" and click \"In Detail\") .
", "rawMarkdown": "Well I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.\n\n[Ref](https://santatoeic.jp/intro) (Go to the section of \"With the most accurate AI tutor\" and click \"In Detail\") .", "votes": null }, { "id": "1067792", "postDate": "11/02/2020 18:02:01", "content": "\n\nWell I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.
\n
Sharing right information has nothing to do with LB position.
", "rawMarkdown": "> Well I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.\n\nSharing right information has nothing to do with LB position.", "votes": null } ], "comments": [ { "id": 1065090, "author_name": "lplenka", "author_url": "", "post_date": "10/30/2020 20:40:29", "content": "@calebeverett Good stats for this user: 5382
\nI think the inferences make sense but it will be interesting to know how these stats come out for a wider group of users, not just one. This will be different for a different user.
", "votes": null, "replies": [] }, { "id": 1065095, "author_name": "calebeverett", "author_url": "", "post_date": "10/30/2020 21:03:45", "content": "Yes, perhaps some common patterns would emerge that could be developed in to features. I was finding it interesting to parse through a bunch of graphs like this for individual users to gain a better understanding of the data, similar to how you might parse though a bunch of images in a classification project.
", "votes": null, "replies": [] }, { "id": 1065205, "author_name": "adityaecdrid", "author_url": "", "post_date": "10/31/2020 02:59:42", "content": "Plots and the idea is beautiful ! But i am not really sure that this user has 500 hours + elapsed time. Maybe you have a logic to determine elapsed hours
", "votes": null, "replies": [] }, { "id": 1065214, "author_name": "calebeverett", "author_url": "", "post_date": "10/31/2020 03:13:46", "content": "That's probably faulty terminology on my part. That is just the timestamp, so what it shows in the first graph is that they used the system over a long period of time, not necessarily continuously, answering questions at a consistent rate. The flat lines are essentially the gaps between sessions, when answer count isn't increasing. Maybe there is a feature there - could be a difference in performance after long gaps?
", "votes": null, "replies": [] }, { "id": 1065247, "author_name": "rohanrao", "author_url": "", "post_date": "10/31/2020 04:36:18", "content": "\n\nit looks like this user answered about twice as many questions correctly as incorrectly
\n
That's what you would expect if any random user is chosen from the data since the overall correct rate of the dataset is ~ 65% (which is same as getting twice as many questions correctly as incorrectly)
", "votes": null, "replies": [] }, { "id": 1065255, "author_name": "adityaecdrid", "author_url": "", "post_date": "10/31/2020 04:52:38", "content": "Plus, Another observation in case you didn't notice is that A total of 30 questions is given to test user skills when they start using the app as a starter test for using Riiidi platform AFAIU.
\n
I noticed the ramp at the beginning of the sequence as well. The initial diagnostic questions make sense and maybe there is some evidence of that in the cumulative chart as indicated by this users improved performance starting around 60 elapsed hours. But, I think most of the reason for the ramps at the very beginning is because of the way the rolling totals are calculated. They start out at zero since there were no prior questions and don't stabilize until questions from ten containers have been answered. I was thinking about whether the rolling totals should start out at five for both, or maybe 6.5 correct and 3.5 incorrect to reflect the average across all users for all questions. But maybe the interaction with task_container_id makes that unnecessary.
Could you share the website where you found this information? I tried myself the diagnosis test on the Santa page and, based on the background questions, I only had to solve 9 questions. Do you know if these 30 questions are presented after you purchase the service?
", "votes": null, "replies": [] }, { "id": 1067782, "author_name": "adityaecdrid", "author_url": "", "post_date": "11/02/2020 17:53:17", "content": "Well I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.
\nRef (Go to the section of \"With the most accurate AI tutor\" and click \"In Detail\") .
", "votes": null, "replies": [] }, { "id": 1067792, "author_name": "rohanrao", "author_url": "", "post_date": "11/02/2020 18:02:01", "content": "\n\nWell I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.
\n
Sharing right information has nothing to do with LB position.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1065085": "I was creating some features that calculated the cumulative and rolling totals of questions answered correctly and put the charts below together to visually inspect whether the calculations were being performed correctly. They turned out to be a useful way to gain some insight into users' learning progress.\n\nThere first chart shows the cumulative number of task containers and correclty and incorrectly answered questions over time. You can tell from the task_container_id line that this user answered approximately 120 questions over a total period of almost 600 hours. It looks like they were more active in the first 120 hours and that their performance improved between 80 and 120 hours. Overall, by looking at the difference between `answered_correctly_cumusum` and `answered_incorrecltly_cumsum` at the end of the time period, it looks like this user answered about twice as many questions correctly as incorrectly.\n\nThe second chart shows the rolling totals of correctly and incorrectly answered questions over the preceding 10 task containers along with whether questions were answered correctly or incorrectly on an individual basis. It looks like this user had several episodes of strong performance starting around 40 task containers. It would be interesting to investigate the causes of the changes in performance to distinquish between learning, i.e., better performance on questions with similar characteristics, and changes in question characteristics (perhaps adding companion charts for question part and difficulty, for starters). It seems likely that questions are delivered in sequences with similar characteristics and that difficulty varies depending on user performance. Once a user's performance improves to the point at which they are determined to have achieved mastery of the characteristics of the sequence, perhaps they would begin another sequence of questions with different characteristics. Alternatively, if the user's performance did not improve, perhaps they would be delivered less difficult questions to allow them to build their knowledge base before progressing to more difficult questions.\n\n.png?generation=1604089560900300&alt=media)\n\n.png?generation=1604089664139102&alt=media)", "1065090": "calebeverett Good stats for this user: 5382\n\nI think the inferences make sense but it will be interesting to know how these stats come out for a wider group of users, not just one. This will be different for a different user.", "1065095": "Yes, perhaps some common patterns would emerge that could be developed in to features. I was finding it interesting to parse through a bunch of graphs like this for individual users to gain a better understanding of the data, similar to how you might parse though a bunch of images in a classification project.", "1065205": "Plots and the idea is beautiful ! But i am not really sure that this user has 500 hours + elapsed time. Maybe you have a logic to determine elapsed hours", "1065214": "That's probably faulty terminology on my part. That is just the timestamp, so what it shows in the first graph is that they used the system **over** a long period of time, not necessarily continuously, answering questions at a consistent rate. The flat lines are essentially the gaps between sessions, when answer count isn't increasing. Maybe there is a feature there - could be a difference in performance after long gaps?", "1065247": "> it looks like this user answered about twice as many questions correctly as incorrectly\n\nThat's what you would expect if any random user is chosen from the data since the overall correct rate of the dataset is ~ 65% (which is same as getting twice as many questions correctly as incorrectly)", "1065255": "Plus, Another observation in case you didn't notice is that A total of 30 questions is given to test user skills when they start using the app as a starter test for using Riiidi platform AFAIU.\n\n", "1065326": "I noticed the ramp at the beginning of the sequence as well. The initial diagnostic questions make sense and maybe there is some evidence of that in the cumulative chart as indicated by this users improved performance starting around 60 elapsed hours. But, I think most of the reason for the ramps at the very beginning is because of the way the rolling totals are calculated. They start out at zero since there were no prior questions and don't stabilize until questions from ten containers have been answered. I was thinking about whether the rolling totals should start out at five for both, or maybe 6.5 correct and 3.5 incorrect to reflect the average across all users for all questions. But maybe the interaction with `task_container_id` makes that unnecessary.", "1067468": "Could you share the website where you found this information? I tried myself the diagnosis test on the [Santa page](https://aitutorsanta.com/intro) and, based on the background questions, I only had to solve 9 questions. Do you know if these 30 questions are presented after you purchase the service?", "1067782": "Well I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.\n\n[Ref](https://santatoeic.jp/intro) (Go to the section of \"With the most accurate AI tutor\" and click \"In Detail\") .", "1067792": "> Well I can share but i don't think you should be taking my comments seriously as i am no where on the LB despite my attempts.\n\nSharing right information has nothing to do with LB position." }, "source": "meta" }