{ "id": 194032, "title": "The next question: Is it model determined?", "url": "/competitions/riiid-test-answer-prediction/discussion/194032", "author_name": "", "post_date": "2020-10-30T09:53:45.342903400Z", "votes": 8, "comment_count": 9, "views": 0, "content": "

Is the next question in sequence (or next bundle of questions) to the user emitted by clever Riiid models (on their part likely also digesting user and question mileage somewhat like in this challenge)?

", "messages": [ { "id": "1064600", "postDate": "10/30/2020 09:53:45", "content": "

Is the next question in sequence (or next bundle of questions) to the user emitted by clever Riiid models (on their part likely also digesting user and question mileage somewhat like in this challenge)?

", "rawMarkdown": "Is the next question in sequence (or next bundle of questions) to the user emitted by clever Riiid models (on their part likely also digesting user and question mileage somewhat like in this challenge)?", "votes": null }, { "id": "1065256", "postDate": "10/31/2020 04:53:47", "content": "

It's an interesting question which only the hosts would be able to answer. My hunch is it's model determined.
\nI'm curious to know if getting the answer would help improve models for this competition?

", "rawMarkdown": "It's an interesting question which only the hosts would be able to answer. My hunch is it's model determined.\nI'm curious to know if getting the answer would help improve models for this competition?", "votes": null }, { "id": "1065261", "postDate": "10/31/2020 05:04:04", "content": "

It's highly likely that it's model determined as they have Self-Personalization and Self-Engagement (pre-determined based on user's metrics etc), Check this paper by Riiid!

", "rawMarkdown": "It's highly likely that it's model determined as they have Self-Personalization and Self-Engagement (pre-determined based on user's metrics etc), Check this [paper](https://arxiv.org/pdf/2005.03818.pdf) by Riiid!", "votes": null }, { "id": "1065263", "postDate": "10/31/2020 05:10:02", "content": "

The only little doubt I had is if the dataset was compiled 'before' the Riiid models were put into production.

\n

I couldn't find a statement that clarifies this. If you noticed it somewhere, let me know!

", "rawMarkdown": "The only little doubt I had is if the dataset was compiled 'before' the Riiid models were put into production.\n\nI couldn't find a statement that clarifies this. If you noticed it somewhere, let me know!", "votes": null }, { "id": "1065539", "postDate": "10/31/2020 12:16:41", "content": "

EdNet description says that this is a model (one of the sources!).

\n
\n

For each day, Santa recommends questions and lectures based on each student's current knowledge status, i.e. correctness probabilities predicted by the Collaborative Filtering model. Such source is called Today's Recommendation.

\n
\n

another important point

\n
\n

Once the number of incorrect answers to questions with particular tags exceeds certain threshold, Santa suggests lectures and questions with corresponding tags. Such suggestion is recorded as adaptive_offer. It also offers lectures and questions if the average correctness rate of questions with particular tags decreased by more than a certain threshold.

\n
", "rawMarkdown": "EdNet [description](https://github.com/riiid/ednet) says that this is a model (one of the sources!). \n> For each day, Santa recommends questions and lectures based on each student's current knowledge status, i.e. correctness probabilities predicted by the Collaborative Filtering model. Such source is called Today's Recommendation.\n\nanother important point\n> Once the number of incorrect answers to questions with particular tags exceeds certain threshold, Santa suggests lectures and questions with corresponding tags. Such suggestion is recorded as adaptive_offer. It also offers lectures and questions if the average correctness rate of questions with particular tags decreased by more than a certain threshold.", "votes": null }, { "id": "1066080", "postDate": "11/01/2020 09:36:10", "content": "

How could knowledge of such a running question producing model help progress in this competition? I am unsure if at all, but have been thinking along these lines:

\n

Assumptions:
\n*This question model is indeed running emitting targeted questions, to which we, immediately after, are figuring out answer correctness.

\n

*The question model is purposed to pose questions at moderate difficulty level - questions that are hard to predict answer correctness to. I would reckon that both silly easy and stupid hard questions would be wastes of time and counter-examples of a tailored user journey.

\n

With this given, we might try to seperately setup and train a next-question model towards the set of questions, multiclass. The idea is then to bring this model's question predictions as data into the competion's answer model: Not only its best question guess, but say the best five, with measurements of prediction uncertainty et.c.,
\nand/or a trained layer out of a neural net, if the model is of such a kind.

\n

The idea then, or notion or hope, is that our question model is mimicking the real but unseen riiiid! next question model so well that it is picking up on complex interactions originally trained on data that we are not given in this competition, which can be data recorded during other time periods, or different feature sets from what we are given here.

\n

That's a whole lot of if's and assumptions, and an implementation can amount to a time costly detour of exactly zero return. It also appears I have shuffled the sequence of:

\n

1.Read what's actually documented and 2.Start assuming and rambling. I am referring to Pavel Orlov's findings (and thanks!)

\n

I would in any case be interested to learn about logic errors in my reasoning above, of which there surely may be one or two. Thanks.

", "rawMarkdown": "How could knowledge of such a running question producing model help progress in this competition? I am unsure if at all, but have been thinking along these lines:\n\nAssumptions:\n*This question model is indeed running emitting targeted questions, to which we, immediately after, are figuring out answer correctness.\n\n*The question model is purposed to pose questions at moderate difficulty level - questions that are hard to predict answer correctness to. I would reckon that both silly easy and stupid hard questions would be wastes of time and counter-examples of a tailored user journey.\n \nWith this given, we might try to seperately setup and train a next-question model towards the set of questions, multiclass. The idea is then to bring this model's question predictions as data into the competion's answer model: Not only its best question guess, but say the best five, with measurements of prediction uncertainty et.c., \nand/or a trained layer out of a neural net, if the model is of such a kind.\n\nThe idea then, or notion or hope, is that our question model is mimicking the real but unseen riiiid! next question model so well that it is picking up on complex interactions originally trained on data that we are not given in this competition, which can be data recorded during other time periods, or different feature sets from what we are given here.\n\nThat's a whole lot of if's and assumptions, and an implementation can amount to a time costly detour of exactly zero return. It also appears I have shuffled the sequence of:\n\n 1.Read what's actually documented and 2.Start assuming and rambling. I am referring to Pavel Orlov's findings (and thanks!)\n\nI would in any case be interested to learn about logic errors in my reasoning above, of which there surely may be one or two. Thanks.", "votes": null }, { "id": "1066083", "postDate": "11/01/2020 09:45:30", "content": "

I understand the thought process because I had thought about it as well. The place where I was not convinced of it's use in this competition is:

\n

1 - We actually do know the next question that is asked to user because it is part of the test data (before scoring). So is there still value in predicting or knowing why a certain question was asked? Well, I don't know 🤔

\n

2 - Hypothetically even if we did end up building the perfect model to predict the next question, how/why would that help in predicting the answer correctness? Maybe if we knew that for a certain user the intention is to ask a question that the user is unlikely to solve, then yes maybe it is useful.

\n

Thoughts galore but unconvincing to me yet 😬

", "rawMarkdown": "I understand the thought process because I had thought about it as well. The place where I was not convinced of it's use in this competition is:\n\n1 - We actually do know the next question that is asked to user because it is part of the test data (before scoring). So is there still value in predicting or knowing why a certain question was asked? Well, I don't know 🤔\n\n2 - Hypothetically even if we did end up building the perfect model to predict the next question, how/why would that help in predicting the answer correctness? Maybe if we knew that for a certain user the intention is to ask a question that the user is unlikely to solve, then yes maybe it is useful.\n\nThoughts galore but unconvincing to me yet 😬", "votes": null }, { "id": "1066111", "postDate": "11/01/2020 10:40:50", "content": "

For me, the program's tactics for changing the complexity of the proposed questions are more important

\n
\n

Once the number of incorrect answers to questions with particular tags exceeds certain threshold, then …

\n
\n

and

\n
\n

… if the average correctness rate of questions with particular tags decreased by more than a certain threshold …

\n
\n

So I see the benefit of predicting: a) is the question suggested by the program or selected by the user? b) the complexity of questions for the user : \"did not change\", \"increased\", \"decreased\".
\nIf you know this, you can somehow use it for prediction.

\n

If I understand correctly, these are the 'jumps' we see on the chart

", "rawMarkdown": "For me, the program's tactics for changing the complexity of the proposed questions are more important\n> Once the number of incorrect answers to questions with particular tags exceeds certain threshold, then ...\n\nand\n\n> ... if the average correctness rate of questions with particular tags decreased by more than a certain threshold ...\n\nSo I see the benefit of predicting: a) is the question suggested by the program or selected by the user? b) the complexity of questions for the user : \"did not change\", \"increased\", \"decreased\".\nIf you know this, you can somehow use it for prediction.\n\nIf I understand correctly, these are the 'jumps' we see on the [chart](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194141)", "votes": null }, { "id": "1068218", "postDate": "11/03/2020 07:25:46", "content": "

The idea is that alongside the next question itself, which we already know, the model would also provide: Runner up best questions, certainty scores and more, and that these new features might contribute to a better understanding of the actually issued questions's fit to the situation.

\n

But that said, I am unconvinced, too, and may leave efforts in this direction for now. Thanks again for coming back.

", "rawMarkdown": "The idea is that alongside the next question itself, which we already know, the model would also provide: Runner up best questions, certainty scores and more, and that these new features might contribute to a better understanding of the actually issued questions's fit to the situation.\n\nBut that said, I am unconvinced, too, and may leave efforts in this direction for now. Thanks again for coming back.", "votes": null }, { "id": "1068242", "postDate": "11/03/2020 07:48:45", "content": "

Thanks for bringing the existence of this rule-based engine into my knowledge, Pavel Orlov. I'll browse up on its likely actions with respect to various jumps and more. My own assumptions and attempted approach start to seem a poor fit to the reality of this competiion - and quite 'academic'.

", "rawMarkdown": "Thanks for bringing the existence of this rule-based engine into my knowledge, Pavel Orlov. I'll browse up on its likely actions with respect to various jumps and more. My own assumptions and attempted approach start to seem a poor fit to the reality of this competiion - and quite 'academic'.", "votes": null } ], "comments": [ { "id": 1065256, "author_name": "rohanrao", "author_url": "", "post_date": "10/31/2020 04:53:47", "content": "

It's an interesting question which only the hosts would be able to answer. My hunch is it's model determined.
\nI'm curious to know if getting the answer would help improve models for this competition?

", "votes": null, "replies": [ { "id": 1065261, "author_name": "adityaecdrid", "author_url": "", "post_date": "10/31/2020 05:04:04", "content": "

It's highly likely that it's model determined as they have Self-Personalization and Self-Engagement (pre-determined based on user's metrics etc), Check this paper by Riiid!

", "votes": null, "replies": [] }, { "id": 1065263, "author_name": "rohanrao", "author_url": "", "post_date": "10/31/2020 05:10:02", "content": "

The only little doubt I had is if the dataset was compiled 'before' the Riiid models were put into production.

\n

I couldn't find a statement that clarifies this. If you noticed it somewhere, let me know!

", "votes": null, "replies": [] } ] }, { "id": 1065539, "author_name": "sapr3s", "author_url": "", "post_date": "10/31/2020 12:16:41", "content": "

EdNet description says that this is a model (one of the sources!).

\n
\n

For each day, Santa recommends questions and lectures based on each student's current knowledge status, i.e. correctness probabilities predicted by the Collaborative Filtering model. Such source is called Today's Recommendation.

\n
\n

another important point

\n
\n

Once the number of incorrect answers to questions with particular tags exceeds certain threshold, Santa suggests lectures and questions with corresponding tags. Such suggestion is recorded as adaptive_offer. It also offers lectures and questions if the average correctness rate of questions with particular tags decreased by more than a certain threshold.

\n
", "votes": null, "replies": [] }, { "id": 1066080, "author_name": "nohitwonder", "author_url": "", "post_date": "11/01/2020 09:36:10", "content": "

How could knowledge of such a running question producing model help progress in this competition? I am unsure if at all, but have been thinking along these lines:

\n

Assumptions:
\n*This question model is indeed running emitting targeted questions, to which we, immediately after, are figuring out answer correctness.

\n

*The question model is purposed to pose questions at moderate difficulty level - questions that are hard to predict answer correctness to. I would reckon that both silly easy and stupid hard questions would be wastes of time and counter-examples of a tailored user journey.

\n

With this given, we might try to seperately setup and train a next-question model towards the set of questions, multiclass. The idea is then to bring this model's question predictions as data into the competion's answer model: Not only its best question guess, but say the best five, with measurements of prediction uncertainty et.c.,
\nand/or a trained layer out of a neural net, if the model is of such a kind.

\n

The idea then, or notion or hope, is that our question model is mimicking the real but unseen riiiid! next question model so well that it is picking up on complex interactions originally trained on data that we are not given in this competition, which can be data recorded during other time periods, or different feature sets from what we are given here.

\n

That's a whole lot of if's and assumptions, and an implementation can amount to a time costly detour of exactly zero return. It also appears I have shuffled the sequence of:

\n

1.Read what's actually documented and 2.Start assuming and rambling. I am referring to Pavel Orlov's findings (and thanks!)

\n

I would in any case be interested to learn about logic errors in my reasoning above, of which there surely may be one or two. Thanks.

", "votes": null, "replies": [ { "id": 1066083, "author_name": "rohanrao", "author_url": "", "post_date": "11/01/2020 09:45:30", "content": "

I understand the thought process because I had thought about it as well. The place where I was not convinced of it's use in this competition is:

\n

1 - We actually do know the next question that is asked to user because it is part of the test data (before scoring). So is there still value in predicting or knowing why a certain question was asked? Well, I don't know 🤔

\n

2 - Hypothetically even if we did end up building the perfect model to predict the next question, how/why would that help in predicting the answer correctness? Maybe if we knew that for a certain user the intention is to ask a question that the user is unlikely to solve, then yes maybe it is useful.

\n

Thoughts galore but unconvincing to me yet 😬

", "votes": null, "replies": [] }, { "id": 1066111, "author_name": "sapr3s", "author_url": "", "post_date": "11/01/2020 10:40:50", "content": "

For me, the program's tactics for changing the complexity of the proposed questions are more important

\n
\n

Once the number of incorrect answers to questions with particular tags exceeds certain threshold, then …

\n
\n

and

\n
\n

… if the average correctness rate of questions with particular tags decreased by more than a certain threshold …

\n
\n

So I see the benefit of predicting: a) is the question suggested by the program or selected by the user? b) the complexity of questions for the user : \"did not change\", \"increased\", \"decreased\".
\nIf you know this, you can somehow use it for prediction.

\n

If I understand correctly, these are the 'jumps' we see on the chart

", "votes": null, "replies": [] }, { "id": 1068218, "author_name": "nohitwonder", "author_url": "", "post_date": "11/03/2020 07:25:46", "content": "

The idea is that alongside the next question itself, which we already know, the model would also provide: Runner up best questions, certainty scores and more, and that these new features might contribute to a better understanding of the actually issued questions's fit to the situation.

\n

But that said, I am unconvinced, too, and may leave efforts in this direction for now. Thanks again for coming back.

", "votes": null, "replies": [] }, { "id": 1068242, "author_name": "nohitwonder", "author_url": "", "post_date": "11/03/2020 07:48:45", "content": "

Thanks for bringing the existence of this rule-based engine into my knowledge, Pavel Orlov. I'll browse up on its likely actions with respect to various jumps and more. My own assumptions and attempted approach start to seem a poor fit to the reality of this competiion - and quite 'academic'.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1064600": "Is the next question in sequence (or next bundle of questions) to the user emitted by clever Riiid models (on their part likely also digesting user and question mileage somewhat like in this challenge)?", "1065256": "It's an interesting question which only the hosts would be able to answer. My hunch is it's model determined.\nI'm curious to know if getting the answer would help improve models for this competition?", "1065261": "It's highly likely that it's model determined as they have Self-Personalization and Self-Engagement (pre-determined based on user's metrics etc), Check this [paper](https://arxiv.org/pdf/2005.03818.pdf) by Riiid!", "1065263": "The only little doubt I had is if the dataset was compiled 'before' the Riiid models were put into production.\n\nI couldn't find a statement that clarifies this. If you noticed it somewhere, let me know!", "1065539": "EdNet [description](https://github.com/riiid/ednet) says that this is a model (one of the sources!). \n> For each day, Santa recommends questions and lectures based on each student's current knowledge status, i.e. correctness probabilities predicted by the Collaborative Filtering model. Such source is called Today's Recommendation.\n\nanother important point\n> Once the number of incorrect answers to questions with particular tags exceeds certain threshold, Santa suggests lectures and questions with corresponding tags. Such suggestion is recorded as adaptive_offer. It also offers lectures and questions if the average correctness rate of questions with particular tags decreased by more than a certain threshold.", "1066080": "How could knowledge of such a running question producing model help progress in this competition? I am unsure if at all, but have been thinking along these lines:\n\nAssumptions:\n*This question model is indeed running emitting targeted questions, to which we, immediately after, are figuring out answer correctness.\n\n*The question model is purposed to pose questions at moderate difficulty level - questions that are hard to predict answer correctness to. I would reckon that both silly easy and stupid hard questions would be wastes of time and counter-examples of a tailored user journey.\n \nWith this given, we might try to seperately setup and train a next-question model towards the set of questions, multiclass. The idea is then to bring this model's question predictions as data into the competion's answer model: Not only its best question guess, but say the best five, with measurements of prediction uncertainty et.c., \nand/or a trained layer out of a neural net, if the model is of such a kind.\n\nThe idea then, or notion or hope, is that our question model is mimicking the real but unseen riiiid! next question model so well that it is picking up on complex interactions originally trained on data that we are not given in this competition, which can be data recorded during other time periods, or different feature sets from what we are given here.\n\nThat's a whole lot of if's and assumptions, and an implementation can amount to a time costly detour of exactly zero return. It also appears I have shuffled the sequence of:\n\n 1.Read what's actually documented and 2.Start assuming and rambling. I am referring to Pavel Orlov's findings (and thanks!)\n\nI would in any case be interested to learn about logic errors in my reasoning above, of which there surely may be one or two. Thanks.", "1066083": "I understand the thought process because I had thought about it as well. The place where I was not convinced of it's use in this competition is:\n\n1 - We actually do know the next question that is asked to user because it is part of the test data (before scoring). So is there still value in predicting or knowing why a certain question was asked? Well, I don't know 🤔\n\n2 - Hypothetically even if we did end up building the perfect model to predict the next question, how/why would that help in predicting the answer correctness? Maybe if we knew that for a certain user the intention is to ask a question that the user is unlikely to solve, then yes maybe it is useful.\n\nThoughts galore but unconvincing to me yet 😬", "1066111": "For me, the program's tactics for changing the complexity of the proposed questions are more important\n> Once the number of incorrect answers to questions with particular tags exceeds certain threshold, then ...\n\nand\n\n> ... if the average correctness rate of questions with particular tags decreased by more than a certain threshold ...\n\nSo I see the benefit of predicting: a) is the question suggested by the program or selected by the user? b) the complexity of questions for the user : \"did not change\", \"increased\", \"decreased\".\nIf you know this, you can somehow use it for prediction.\n\nIf I understand correctly, these are the 'jumps' we see on the [chart](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194141)", "1068218": "The idea is that alongside the next question itself, which we already know, the model would also provide: Runner up best questions, certainty scores and more, and that these new features might contribute to a better understanding of the actually issued questions's fit to the situation.\n\nBut that said, I am unconvinced, too, and may leave efforts in this direction for now. Thanks again for coming back.", "1068242": "Thanks for bringing the existence of this rule-based engine into my knowledge, Pavel Orlov. I'll browse up on its likely actions with respect to various jumps and more. My own assumptions and attempted approach start to seem a poor fit to the reality of this competiion - and quite 'academic'." }, "source": "meta" }