Note: Part 1 is here: From Bayesian to Transformers - Tracing the 'Knowledge Tracing' models over time: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/201481
\nPart 3: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206185 - Some additional clarifications on SAKT/SAINT
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206584 - A small discussion on position embeddings for those interested.
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206719 - On lectures, the art of forgetting and why I retired hurt
\nFor many reasons, this is an interesting competition. The sheer volume of data, the process for submission, the scope for online learning and the imposed constraints - all mimic real-world situations. This is a gold-mine for aspiring data scientists (newbies) because there is so much to learn. Rarely does someone give this kind of rich data. Such learning experiences are unique in case you are not already on this ship - do hop on for a fascinating ride.
\nLast weekend, I had shared my interpretation of the general domain and some of the interesting developments in this area recent times. I had also shared my interpretations of the top 5 SOTA papers in this area. Let us now quickly look at the actual data shared by RiiiD and focus on the interesting columns. We will jot down some intuitions along the way and then test those against actual data (I am afraid the 'testing' part will have to wait till next weekend)..
\n
time in milliseconds between this user interaction and the first event completion from that user
\nIt took me a while to understand this. The exercises are structured in such a way that sometimes there could be an image or a video displayed and then a set of exercises that follow - all of which are related to the same. This combined set of related exercises is called a bundle… all share the same bundle id. Because exercises can be repeated multiple times over the course of a students interactions, they are identified each time by a unique task_container_id, which reflects the order in which the user starts the tasks. So it is an important positional information and definitely a strong feature. One of the notebooks already proved the intuition that 'when questions are repeated rate, the success rate is higher'. That example showed many cases where questions were repeated even if user got them right the last time around. This could be because the user may be learning a new concept and that new concept is based on 2-3 old concepts. Probably if the user gets the new concept wrong, the AI in the system (yes! I am sure AI would already be built in the system…so in a way, we are modelling an AI model :) Now u see why this competition is so fascinating) would probably display the exercises of the associated concepts (the old ones) to refresh the student memory. Why am I mentioning this? This 'might' be useful in clustering exercises. Clustering exercises is going to play an imp role and while one way to do it would be using tags, another would be using the temporal information (if user answers Q4 and Q7 correctly, he is very likely to get Q17 also correct) and the third way would be the above
\n
Average time taken to answer each question in the previous exercise bundle. If the exercise was not a part of a bundle but an individual entity, then it is just the time taken to answer the prev question.
\nThis is defined as \"Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle\".
\nThe above seems to indicate that after answering a question, the student sees the correct response and an explanation for the same. The user can choose to read this explanation. Possibly this could be a hyperlink given below the correct answer. If the user clicks the hyperlink, then probably this boolean flag is recorded. This immediately strikes me as a very good feature. If the user is correct, then he already knows about the concept and is probably further strengthening his memory. Sincere students who were wrong, would definitely want to see where they are going wrong and correct themselves. IF an user gives a wrong answer AND this flag is FALSE is almost certainly going to get subsequent exercises (related to this topic) as incorrect. Firstly, I hope my understanding of this feature is correct. Please correct me if I am wrong. The next interesting question is how do we vectorize this interesting feature. We can retain it as it is and let the model automatically discover its relation with answer correctness in predicting the labels or we can experiment if adding this type of OHE helps:
\n1- Answer correct and flag is false: 1 0 0
\n2- Answer correct and flag is true: 0 1 0
\n3- Answer wrong and flag is false: 0 0 1
\n4- Answer wrong and flag is true: 1 0 0
\nUPDATE: Yana has shown that field is mainly for diagnostic questions. If so then it is better to discard this field for predictions
Now let us look at the interesting fields in the other 2 datasets:
\nThen we have lectures - each lecture is mapped to 1 tag and most likely strengthens the students knowledge for that tag.
\nLastly we have Parts - This may be too generic but could still be useful. We could take the student average correctness for each Part rather than their performance as a whole. This is because students may be stronger in certain 'Part's of the exam compared to other parts. Kkiller took the harmonic mean of student correctness and exercise correctness and gets around 74%. I am sure this can be improved by taking student correctness for each of the 6-7 parts of the exam.
\nLet us see what else is important:
\nWe have covered all possible features that we could intuitively think of. For more we need to do an elaborate EDA. We also need to explore the relationship between exercise-lecture-tags in a deeper way.
\nLastly, let us come to the time-series aspect of the data and this could guide us regarding the possible architectures. It was a bit hard for me to intuitively grasp the architectures in the SOTA papers, so I hope this below discussion helps newbies like me who are the intended audience. Surprisingly I couldn't find too many explanations of using transformer like architectures in Time-series situations..beyond the actual papers themselves. My focus is on the intuitive aspects of the architecture.
\nIt is possible to fit a Time-series model into a encoder, decoder type of architecture & by that extension into a transformer architecture. In case of a simple sequence classification (like the covid mRNA Kaggle challenge), we needed only the encoder part of the transformer. But in case we want to predict a series (say the next 1 month of the financial index), then this can be a seq-seq exercise. In case we want to predict the entire 1 month at one shot, we can still restrict ourselves to the encoder architecture but in case we want to predict something and then use the predicted value to predict the next day's index we will need a decoder as well. The decoder uses the 'just decoded value' from the previous step and the prev hidden state (in case of LSTM) to predict the next word. In case we add attention, it will also look at the encoder outputs to determine which of the outputs could have a strong 'say' in predicting the label and give weightage to it accordingly. Discussing attention or transformer architecture itself is not the scope here, but those interested for a background can refer to: https://towardsdatascience.com/create-your-own-custom-attention-layer-understand-all-flavours-2201b5e8be9e.
\nThere are minor differences between NLP and Time-series Seq to Seq and it would be good to analyse those because most of us are familiar with NLP Seq-Seq architectures and we just need to 'adapt' that model to other time-series situations. In NLP there is a limit to which the sentence or the para or even the document can stretch backwards. In Time-series, we can have financial indices that are a century old. So determining a 'window' where we are going to focus our 'attention' is going to be the key. Local and soft attention techniques are better here. In summary, to produce a labeled dataset, we need to use a fixed-length sliding time window approach to construct X, Y pairs for model training and evaluation. Having such a window means we may need to pad up inputs with dummy data. Could the diagnostic set of 30 odd questions act like some sort of a padding? This is just a thought. Obviously we have to do look-ahead masking and one-position offset between the decoder input and target output in the decoder to ensure that prediction of a time series data point will only depend on previous data points. Most importantly, for a time-series model, the input is a vector of continuous numbers already. There is no need for an NLP like embedding layer. HOWEVER, we need to ensure that the data is richly represented. We could make use of a (learnable) weight matrix to transform the one-dimension vector into a certain dimension vector. Alternatively we could use self-attention to embed the input. Self-attention with all the fancy multiheads, residuals and all the standard tricks result in capturing the richness of the data well. Using self-attention will require positional encodings as the embedding has no knowledge of sequence order. There are some standard well-defined hacks to incorporate this and we will not go into the details. This is a quick generic summary of how a time-series data can leverage the transformer architecture. But I felt there are two unanswered questions regarding this. We will come to that later but first let us look at the specificities of this competition.
\nLet us assume there is only one student. We have a sequence of all her interactions 'I' from I1 to I1000. She viewed 100 lectures and solved 900 exercises. She has reached the fag end of the course now and another 100 exercises are left. We have to predict how she performs in these 100 exercises. I assume there could be a couple of lectures embedded here also, let us just ignore those for now as we dont have to predict those. The way we could train now is to break the data into time chunks. We could have Q1 to Q1000 fed to the encoder. The decoder will consist of responses to Q2 to Q1000. Let us now predict Q2 using just the questions Q1, Q2 and Q1's response by the student. Now you may see the justification for masks on both the encoder and the decoder. When predicting Q2, we just know the question Q1, the question Q2 and we just know the student's response to Q1 (and of course all the associated meta data). Possibly this sentence makes better sense now - \"The key is to do look-ahead masking and one-position offset between the decoder input and target output in the decoder to ensure that prediction of a time series data point will only depend on previous data points\".
\nNow repeat the above, this time the input on encoder side will be the questions Q1, Q2, Q3 and the decoder side will be the response to Q1, Q2 and our goal is to predict the response for Q3. Similarly assume we have reached Q100. We notice a pattern now. The predictions of certain question responses depend on the response to some of the previous questions. Maybe it is because some questions are repeated or maybe it is because questions with same tags are somewhat similar and if a student has answered one she will almost certainly answer the other. So your model determines (after going thru' many such student interactions) that Q100's response depends on the student's past response to Q78, Q99 and Q46. This is attention logic at work. The decoder is paying attention to all the states of the encoder output (Q1 to Q100) while predicting Q100th response (in addition to considering the decoder inputs. There will also be student specific temporal information which will be captured by the model and thus the model is trained. Every additional student contributes to strengthening the models weights and biases and now the model is ready to predict. Notice that for the 1000th question, it will be computationally expensive to go all the way back to Q1, so we take a sliding window and carve out time slices of student interaction data. If the window is 100, we will have Q1-Q100 questions and responses as 1 sliced row input and Q101 to be predicted. Then we will have Q2-Q101 and Q102 to be predicted…and all the way till Q901-Q1000 questions and responses as inputs and Q 1001 to be predicted.
\nSo far so good. But where do we feed what? If we take the case of a language translator for whom the original encoder-decoder arch was conceived, the source language inputs are fed to the encoder and the destination language to the decoder. The situation in time-series is such that we can feed the inputs to the encoder and the outputs to the decoder. So Q1-Q100 goes to encoder and also Q101. The responses of Q1 to Q100 goes to decoder and now we have to predict the response for Q101. We can assume that each response is like the translated sentence of the concerned question.
\nSo all questions and question related information could be fed to the encoder. This includes the exercises, the tags, the task_container_id's and other derived features related to QUESTIONS etc etc. The response related meta data along with the response themselves go into the decoder. This includes data like prev question explanation, elapsed time, lag time, saw lecture and all derived features related to response….basically everything other than what is put in the encoder. I like this style of segregation between encoder and decoder and I believe this is what is done in the SAINT paper. HOWEVER this need not be the only solution. Some papers have fed the coupled interactions (exercise+response) on decoder end. Others have tried many other things. This could be an area that will need a bit of experimentation. I feel it might not altogether be a bad idea to add tags meta-data in the responses?
\nThe rest of the paraphernalia is the same as in any standard transformer architecture and there are source codes in every framework for the same which you can re-use.
\nOther possible considerations
\nLastly, coming to the two things which I couldn't understand intuitively. I can see the following issues when using transformers (or encoder-decoder) on time-series data and I hope someone can guide me on where my understanding is incorrect:
\nIssue 1: In NLP every translation has one and only one correct answer. While the inputs may change to form different sequences, the output will always be the same for a given sequence of words. The model converges towards that output…it cannot deviate astray with more data. In a Time-series data, there could be different outputs for a given input sequence. In this case it is based on the student profile. The two problem statements are similar but not the same and using the same architecture for both may not be optimal
\nIssue 2: Somewhat linked to Issue 1. Let us think of it in terms of financial indices. We want to determine the NYSE index future. Let us say NYSE index is represented by a proportion of every stock traded on NYSE. One option is to take the NYSE index itself over time and only use that to model. The other option is to look at all the constituents. However modelling based on the constituents could be tricky. Each constituent follows a different path. Every stock listed in the NYSE stock exchange could behave differently. Using their daily data to predict the NYSE index futures is at best a tricky job. Considering the problem at hand, we see each different student as a individual stock being traded on NYSE. There is no equivalent of NYSE index here. Instead we want to predict how these stocks continue to move and also we want to predict how new stocks added to the stock exchange would move. We do have lots and lots of data but doesn't this introduce tons of noise also? One obvious solution is to cluster the constituents. So in case of the NYSE scenario, we could have a bunch of stocks getting represented by a single index - the tech index. We could have brick n mortar companies as another cluster, retail, housing and form dozen odd clusters. Calculate the movements for each cluster separately…as they could have different weights and balances. Now we are left with time-series data of a dozen odd indices - all of which will help determine the NYSE index futures more predictably. If we apply this sort of a scenario to RiiiD, this means that clusters of students need to be created and then it makes sense to have separate sets of weights and biases for each category? Let existing student continue in the cluster they already are in and the new students are moved to one of the clusters. But I haven't seen this kind of approach in any time-series model leveraging encpder/decoders.
\nWe still haven't talked of validation strategies, thought about what could be the best combination of keys queries and values or the importance of online learning in this competition or usage of reductions and many others. But we have covered a fair bit of theory otherwise. While kernels are shared abundantly, I felt some theoretical discussion on the features and architectures could help newcomers like me and hence quickly jotted a few points that stuck me as I went thru' a few of the beautiful kernels.
\nIn Silogram's golden words, this competition is not just a test of AI skills, but is also a test of our engineering skills. How creatively we make use of the imposed constraints will decide the winner. With unlimited processing power and memory, the NN models would have won hands down but with the imposed constraints a blend of NN and non-NN models may hold the key.
\nAs with all features and architectures, there are risks/rewards.
\nChoose wisely. All the very best to all!
", "messages": [ { "id": "1111913", "postDate": "12/14/2020 06:18:54", "content": "Note: Part 1 is here: From Bayesian to Transformers - Tracing the 'Knowledge Tracing' models over time: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/201481
\nPart 3: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206185 - Some additional clarifications on SAKT/SAINT
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206584 - A small discussion on position embeddings for those interested.
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206719 - On lectures, the art of forgetting and why I retired hurt
\nFor many reasons, this is an interesting competition. The sheer volume of data, the process for submission, the scope for online learning and the imposed constraints - all mimic real-world situations. This is a gold-mine for aspiring data scientists (newbies) because there is so much to learn. Rarely does someone give this kind of rich data. Such learning experiences are unique in case you are not already on this ship - do hop on for a fascinating ride.
\nLast weekend, I had shared my interpretation of the general domain and some of the interesting developments in this area recent times. I had also shared my interpretations of the top 5 SOTA papers in this area. Let us now quickly look at the actual data shared by RiiiD and focus on the interesting columns. We will jot down some intuitions along the way and then test those against actual data (I am afraid the 'testing' part will have to wait till next weekend)..
\n
time in milliseconds between this user interaction and the first event completion from that user
\nIt took me a while to understand this. The exercises are structured in such a way that sometimes there could be an image or a video displayed and then a set of exercises that follow - all of which are related to the same. This combined set of related exercises is called a bundle… all share the same bundle id. Because exercises can be repeated multiple times over the course of a students interactions, they are identified each time by a unique task_container_id, which reflects the order in which the user starts the tasks. So it is an important positional information and definitely a strong feature. One of the notebooks already proved the intuition that 'when questions are repeated rate, the success rate is higher'. That example showed many cases where questions were repeated even if user got them right the last time around. This could be because the user may be learning a new concept and that new concept is based on 2-3 old concepts. Probably if the user gets the new concept wrong, the AI in the system (yes! I am sure AI would already be built in the system…so in a way, we are modelling an AI model :) Now u see why this competition is so fascinating) would probably display the exercises of the associated concepts (the old ones) to refresh the student memory. Why am I mentioning this? This 'might' be useful in clustering exercises. Clustering exercises is going to play an imp role and while one way to do it would be using tags, another would be using the temporal information (if user answers Q4 and Q7 correctly, he is very likely to get Q17 also correct) and the third way would be the above
\n
Average time taken to answer each question in the previous exercise bundle. If the exercise was not a part of a bundle but an individual entity, then it is just the time taken to answer the prev question.
\nThis is defined as \"Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle\".
\nThe above seems to indicate that after answering a question, the student sees the correct response and an explanation for the same. The user can choose to read this explanation. Possibly this could be a hyperlink given below the correct answer. If the user clicks the hyperlink, then probably this boolean flag is recorded. This immediately strikes me as a very good feature. If the user is correct, then he already knows about the concept and is probably further strengthening his memory. Sincere students who were wrong, would definitely want to see where they are going wrong and correct themselves. IF an user gives a wrong answer AND this flag is FALSE is almost certainly going to get subsequent exercises (related to this topic) as incorrect. Firstly, I hope my understanding of this feature is correct. Please correct me if I am wrong. The next interesting question is how do we vectorize this interesting feature. We can retain it as it is and let the model automatically discover its relation with answer correctness in predicting the labels or we can experiment if adding this type of OHE helps:
\n1- Answer correct and flag is false: 1 0 0
\n2- Answer correct and flag is true: 0 1 0
\n3- Answer wrong and flag is false: 0 0 1
\n4- Answer wrong and flag is true: 1 0 0
\nUPDATE: Yana has shown that field is mainly for diagnostic questions. If so then it is better to discard this field for predictions
Now let us look at the interesting fields in the other 2 datasets:
\nThen we have lectures - each lecture is mapped to 1 tag and most likely strengthens the students knowledge for that tag.
\nLastly we have Parts - This may be too generic but could still be useful. We could take the student average correctness for each Part rather than their performance as a whole. This is because students may be stronger in certain 'Part's of the exam compared to other parts. Kkiller took the harmonic mean of student correctness and exercise correctness and gets around 74%. I am sure this can be improved by taking student correctness for each of the 6-7 parts of the exam.
\nLet us see what else is important:
\nWe have covered all possible features that we could intuitively think of. For more we need to do an elaborate EDA. We also need to explore the relationship between exercise-lecture-tags in a deeper way.
\nLastly, let us come to the time-series aspect of the data and this could guide us regarding the possible architectures. It was a bit hard for me to intuitively grasp the architectures in the SOTA papers, so I hope this below discussion helps newbies like me who are the intended audience. Surprisingly I couldn't find too many explanations of using transformer like architectures in Time-series situations..beyond the actual papers themselves. My focus is on the intuitive aspects of the architecture.
\nIt is possible to fit a Time-series model into a encoder, decoder type of architecture & by that extension into a transformer architecture. In case of a simple sequence classification (like the covid mRNA Kaggle challenge), we needed only the encoder part of the transformer. But in case we want to predict a series (say the next 1 month of the financial index), then this can be a seq-seq exercise. In case we want to predict the entire 1 month at one shot, we can still restrict ourselves to the encoder architecture but in case we want to predict something and then use the predicted value to predict the next day's index we will need a decoder as well. The decoder uses the 'just decoded value' from the previous step and the prev hidden state (in case of LSTM) to predict the next word. In case we add attention, it will also look at the encoder outputs to determine which of the outputs could have a strong 'say' in predicting the label and give weightage to it accordingly. Discussing attention or transformer architecture itself is not the scope here, but those interested for a background can refer to: https://towardsdatascience.com/create-your-own-custom-attention-layer-understand-all-flavours-2201b5e8be9e.
\nThere are minor differences between NLP and Time-series Seq to Seq and it would be good to analyse those because most of us are familiar with NLP Seq-Seq architectures and we just need to 'adapt' that model to other time-series situations. In NLP there is a limit to which the sentence or the para or even the document can stretch backwards. In Time-series, we can have financial indices that are a century old. So determining a 'window' where we are going to focus our 'attention' is going to be the key. Local and soft attention techniques are better here. In summary, to produce a labeled dataset, we need to use a fixed-length sliding time window approach to construct X, Y pairs for model training and evaluation. Having such a window means we may need to pad up inputs with dummy data. Could the diagnostic set of 30 odd questions act like some sort of a padding? This is just a thought. Obviously we have to do look-ahead masking and one-position offset between the decoder input and target output in the decoder to ensure that prediction of a time series data point will only depend on previous data points. Most importantly, for a time-series model, the input is a vector of continuous numbers already. There is no need for an NLP like embedding layer. HOWEVER, we need to ensure that the data is richly represented. We could make use of a (learnable) weight matrix to transform the one-dimension vector into a certain dimension vector. Alternatively we could use self-attention to embed the input. Self-attention with all the fancy multiheads, residuals and all the standard tricks result in capturing the richness of the data well. Using self-attention will require positional encodings as the embedding has no knowledge of sequence order. There are some standard well-defined hacks to incorporate this and we will not go into the details. This is a quick generic summary of how a time-series data can leverage the transformer architecture. But I felt there are two unanswered questions regarding this. We will come to that later but first let us look at the specificities of this competition.
\nLet us assume there is only one student. We have a sequence of all her interactions 'I' from I1 to I1000. She viewed 100 lectures and solved 900 exercises. She has reached the fag end of the course now and another 100 exercises are left. We have to predict how she performs in these 100 exercises. I assume there could be a couple of lectures embedded here also, let us just ignore those for now as we dont have to predict those. The way we could train now is to break the data into time chunks. We could have Q1 to Q1000 fed to the encoder. The decoder will consist of responses to Q2 to Q1000. Let us now predict Q2 using just the questions Q1, Q2 and Q1's response by the student. Now you may see the justification for masks on both the encoder and the decoder. When predicting Q2, we just know the question Q1, the question Q2 and we just know the student's response to Q1 (and of course all the associated meta data). Possibly this sentence makes better sense now - \"The key is to do look-ahead masking and one-position offset between the decoder input and target output in the decoder to ensure that prediction of a time series data point will only depend on previous data points\".
\nNow repeat the above, this time the input on encoder side will be the questions Q1, Q2, Q3 and the decoder side will be the response to Q1, Q2 and our goal is to predict the response for Q3. Similarly assume we have reached Q100. We notice a pattern now. The predictions of certain question responses depend on the response to some of the previous questions. Maybe it is because some questions are repeated or maybe it is because questions with same tags are somewhat similar and if a student has answered one she will almost certainly answer the other. So your model determines (after going thru' many such student interactions) that Q100's response depends on the student's past response to Q78, Q99 and Q46. This is attention logic at work. The decoder is paying attention to all the states of the encoder output (Q1 to Q100) while predicting Q100th response (in addition to considering the decoder inputs. There will also be student specific temporal information which will be captured by the model and thus the model is trained. Every additional student contributes to strengthening the models weights and biases and now the model is ready to predict. Notice that for the 1000th question, it will be computationally expensive to go all the way back to Q1, so we take a sliding window and carve out time slices of student interaction data. If the window is 100, we will have Q1-Q100 questions and responses as 1 sliced row input and Q101 to be predicted. Then we will have Q2-Q101 and Q102 to be predicted…and all the way till Q901-Q1000 questions and responses as inputs and Q 1001 to be predicted.
\nSo far so good. But where do we feed what? If we take the case of a language translator for whom the original encoder-decoder arch was conceived, the source language inputs are fed to the encoder and the destination language to the decoder. The situation in time-series is such that we can feed the inputs to the encoder and the outputs to the decoder. So Q1-Q100 goes to encoder and also Q101. The responses of Q1 to Q100 goes to decoder and now we have to predict the response for Q101. We can assume that each response is like the translated sentence of the concerned question.
\nSo all questions and question related information could be fed to the encoder. This includes the exercises, the tags, the task_container_id's and other derived features related to QUESTIONS etc etc. The response related meta data along with the response themselves go into the decoder. This includes data like prev question explanation, elapsed time, lag time, saw lecture and all derived features related to response….basically everything other than what is put in the encoder. I like this style of segregation between encoder and decoder and I believe this is what is done in the SAINT paper. HOWEVER this need not be the only solution. Some papers have fed the coupled interactions (exercise+response) on decoder end. Others have tried many other things. This could be an area that will need a bit of experimentation. I feel it might not altogether be a bad idea to add tags meta-data in the responses?
\nThe rest of the paraphernalia is the same as in any standard transformer architecture and there are source codes in every framework for the same which you can re-use.
\nOther possible considerations
\nLastly, coming to the two things which I couldn't understand intuitively. I can see the following issues when using transformers (or encoder-decoder) on time-series data and I hope someone can guide me on where my understanding is incorrect:
\nIssue 1: In NLP every translation has one and only one correct answer. While the inputs may change to form different sequences, the output will always be the same for a given sequence of words. The model converges towards that output…it cannot deviate astray with more data. In a Time-series data, there could be different outputs for a given input sequence. In this case it is based on the student profile. The two problem statements are similar but not the same and using the same architecture for both may not be optimal
\nIssue 2: Somewhat linked to Issue 1. Let us think of it in terms of financial indices. We want to determine the NYSE index future. Let us say NYSE index is represented by a proportion of every stock traded on NYSE. One option is to take the NYSE index itself over time and only use that to model. The other option is to look at all the constituents. However modelling based on the constituents could be tricky. Each constituent follows a different path. Every stock listed in the NYSE stock exchange could behave differently. Using their daily data to predict the NYSE index futures is at best a tricky job. Considering the problem at hand, we see each different student as a individual stock being traded on NYSE. There is no equivalent of NYSE index here. Instead we want to predict how these stocks continue to move and also we want to predict how new stocks added to the stock exchange would move. We do have lots and lots of data but doesn't this introduce tons of noise also? One obvious solution is to cluster the constituents. So in case of the NYSE scenario, we could have a bunch of stocks getting represented by a single index - the tech index. We could have brick n mortar companies as another cluster, retail, housing and form dozen odd clusters. Calculate the movements for each cluster separately…as they could have different weights and balances. Now we are left with time-series data of a dozen odd indices - all of which will help determine the NYSE index futures more predictably. If we apply this sort of a scenario to RiiiD, this means that clusters of students need to be created and then it makes sense to have separate sets of weights and biases for each category? Let existing student continue in the cluster they already are in and the new students are moved to one of the clusters. But I haven't seen this kind of approach in any time-series model leveraging encpder/decoders.
\nWe still haven't talked of validation strategies, thought about what could be the best combination of keys queries and values or the importance of online learning in this competition or usage of reductions and many others. But we have covered a fair bit of theory otherwise. While kernels are shared abundantly, I felt some theoretical discussion on the features and architectures could help newcomers like me and hence quickly jotted a few points that stuck me as I went thru' a few of the beautiful kernels.
\nIn Silogram's golden words, this competition is not just a test of AI skills, but is also a test of our engineering skills. How creatively we make use of the imposed constraints will decide the winner. With unlimited processing power and memory, the NN models would have won hands down but with the imposed constraints a blend of NN and non-NN models may hold the key.
\nAs with all features and architectures, there are risks/rewards.
\nChoose wisely. All the very best to all!
", "rawMarkdown": "Note: Part 1 is here: From Bayesian to Transformers - Tracing the 'Knowledge Tracing' models over time: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/201481\n\nPart 3: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206185 - Some additional clarifications on SAKT/SAINT\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206584 - A small discussion on position embeddings for those interested.\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206719 - On lectures, the art of forgetting and why I retired hurt\n\nFor many reasons, this is an interesting competition. The sheer volume of data, the process for submission, the scope for online learning and the imposed constraints - all mimic real-world situations. This is a gold-mine for aspiring data scientists (newbies) because there is so much to learn. Rarely does someone give this kind of rich data. Such learning experiences are unique in case you are not already on this ship - do hop on for a fascinating ride. \n\nLast weekend, I had shared my interpretation of the general domain and some of the interesting developments in this area recent times. I had also shared my interpretations of the top 5 SOTA papers in this area. Let us now quickly look at the actual data shared by RiiiD and focus on the interesting columns. We will jot down some intuitions along the way and then test those against actual data (I am afraid the 'testing' part will have to wait till next weekend)..\nI have benefited from every discussion you have had and look forward to next week's discussion.I will check my feature engineering
", "rawMarkdown": "I have benefited from every discussion you have had and look forward to next week's discussion.I will check my feature engineering", "votes": null }, { "id": "1112108", "postDate": "12/14/2020 09:30:54", "content": "Hey !
\nThanks for the messages.
\nSome thoughts from me:
I prefere to use Timestamp rather than prior_question_elapsed_time to define if a user is new or not (it is precised that timestamp is set to 0 for every first interaction of a user.
Something very interesting but that is hard to capture is the lag_time between two exercices. It can gives indication of a student taking notes of the last question for example.
Still in the timestamp aspect, I think it can be interesting to check last time a user had an interaction with a section of the toeic. If a user start to improve on a topic, but then don't go to it for days, he might have some troubles redoing the exercices of that section.
Thanks Qiaqia. All the very best to you for the competition!!!
\nAs only couple of weekends left now, I may not be able to post anything tangible…but all the best once again!
Point 1 - Yes
\nPoint 2 - The difference between the two consecutive timestamps for a student should give an indication of lag time? So like in the Saint+ paper, we do have both the elapsed time and the lag time here.. Of course the elapsed time is for the prev question…Still it does help apparently…
just to be clear, their definition is as follows:
\nelapsed time - the time taken for a student to answer (the previous question)
\nlag time - the time interval between adjacent learning activities.
From this, I guess we can calculate the difference between times to see if the student was taking some notes after exercises if that is what is needed. I dont know how strong that feature would be though. However this would indeed be a very strong feature for lectures - amount of time student watched the lecture versus average time lecture is typically watched but as far as I can see they dont give elapsed time for lectures. Still we can try to guess try to play around with timestamp differences and make some guesses based on lag times (of course we have to assume that student is doing all this in one continuous session and hasn't opened a new window to play their favorite video game :) could be a noisy feature but worth a try
", "rawMarkdown": "Point 1 - Yes\nPoint 2 - The difference between the two consecutive timestamps for a student should give an indication of lag time? So like in the Saint+ paper, we do have both the elapsed time and the lag time here.. Of course the elapsed time is for the prev question...Still it does help apparently...\n\njust to be clear, their definition is as follows:\nelapsed time - the time taken for a student to answer (the previous question)\nlag time - the time interval between adjacent learning activities.\n\nFrom this, I guess we can calculate the difference between times to see if the student was taking some notes after exercises if that is what is needed. I dont know how strong that feature would be though. However this would indeed be a very strong feature for lectures - amount of time student watched the lecture versus average time lecture is typically watched but as far as I can see they dont give elapsed time for lectures. Still we can try to guess try to play around with timestamp differences and make some guesses based on lag times (of course we have to assume that student is doing all this in one continuous session and hasn't opened a new window to play their favorite video game :) could be a noisy feature but worth a try", "votes": null }, { "id": "1139368", "postDate": "01/05/2021 10:52:14", "content": "@allohvk thank you for compiling such a helpful post!! actually all of your discussions are worth a lot -:) I'll have to read many times to digest.. regarding the following point:
\n\n\nHOWEVER, we need to ensure that the data is richly represented. We could make use of a (learnable) weight matrix to transform the one-dimension vector into a certain dimension vector. Alternatively we could use self-attention to embed the input. Self-attention with all the fancy multiheads, residuals and all the standard tricks result in capturing the richness of the data well. Using self-attention will require positional encodings as the embedding has no knowledge of sequence order. There are some standard well-defined hacks to incorporate this and we will not go into the details.
\n
could you please point me to any example code/paper that uses self-att to capture the input? I don't remember to have come across smth like that - I'll give a try anyway but any source with info will help. Thank you and good luck to the comp
", "rawMarkdown": "allohvk thank you for compiling such a helpful post!! actually all of your discussions are worth a lot -:) I'll have to read many times to digest.. regarding the following point:\n> HOWEVER, we need to ensure that the data is richly represented. We could make use of a (learnable) weight matrix to transform the one-dimension vector into a certain dimension vector. Alternatively we could use self-attention to embed the input. Self-attention with all the fancy multiheads, residuals and all the standard tricks result in capturing the richness of the data well. Using self-attention will require positional encodings as the embedding has no knowledge of sequence order. There are some standard well-defined hacks to incorporate this and we will not go into the details. \n\n could you please point me to any example code/paper that uses self-att to capture the input? I don't remember to have come across smth like that - I'll give a try anyway but any source with info will help. Thank you and good luck to the comp", "votes": null }, { "id": "1139426", "postDate": "01/05/2021 11:39:06", "content": "Thanks @imeintanis :)
\nThis was written at the very beginning of the competition. So there could be slight changes here and there. Definitely self-attention should help. There are couple of Saint implementations and I believe they use self-attention because that is part of Saint architecture by default. Have you checked those. Unfortnuately my Kaggle ID had an issue till last week and only last weekend I took the SAKT kernel from Wang and trying to make changes to make it work. I am still stuck at basic changes..
Thank you I got it (actually I had misunderstood your point). Happy that you resolved your issues & you 're back again
\nps: I started from the same SAKT kernel but cannot surpass the 0780 stage.. we'll see
prior_question_elapsed_time is <NA> for lectures, regrettably. The full dataset has it; but not the version they prepared for our Kaggle.
@authman did u find a way to create a moving mask for the question bundles so that they dont attend to answers in same bundle? I can share my thought if u r looking for a solution
", "rawMarkdown": "authman did u find a way to create a moving mask for the question bundles so that they dont attend to answers in same bundle? I can share my thought if u r looking for a solution", "votes": null }, { "id": "1139647", "postDate": "01/05/2021 14:42:49", "content": "Not satisfactorily =\\
", "rawMarkdown": "Not satisfactorily =\\", "votes": null }, { "id": "1139745", "postDate": "01/05/2021 15:39:29", "content": "see if this works..
\nThe trick here is to create separate masks for interaction sequence and question sequence in each row (if u r using SAKT) and create multiple rows
\nQ1 Q2 Q3 Q4 Q5 Q6, Q7
\nI0 I1 I2 I3 I4 I5 I6
Let us say Q3, Q4, Q5 are a bundle, then:
\nI0 I1 I2 0 0 0 0
\nQ1 Q2 Q3 Q4 Q5 0 0
and
\nI0 I1 I2 I3 I4 I5 I6
\n0 0 0 0 0 Q6 Q7
Repeat for all bundles in row
", "rawMarkdown": "see if this works..\n\nThe trick here is to create separate masks for interaction sequence and question sequence in each row (if u r using SAKT) and create multiple rows\nQ1 Q2 Q3 Q4 Q5 Q6, Q7\nI0 I1 I2 I3 I4 I5 I6\n\nLet us say Q3, Q4, Q5 are a bundle, then:\nI0 I1 I2 0 0 0 0 \nQ1 Q2 Q3 Q4 Q5 0 0\n\nand\nI0 I1 I2 I3 I4 I5 I6\n0 0 0 0 0 Q6 Q7\n\nRepeat for all bundles in row", "votes": null } ], "comments": [ { "id": 1112034, "author_name": "yangxiaoshuai", "author_url": "", "post_date": "12/14/2020 08:18:47", "content": "I have benefited from every discussion you have had and look forward to next week's discussion.I will check my feature engineering
", "votes": null, "replies": [ { "id": 1112189, "author_name": "allohvk", "author_url": "", "post_date": "12/14/2020 11:16:41", "content": "Thanks Qiaqia. All the very best to you for the competition!!!
\nAs only couple of weekends left now, I may not be able to post anything tangible…but all the best once again!
Hey !
\nThanks for the messages.
\nSome thoughts from me:
I prefere to use Timestamp rather than prior_question_elapsed_time to define if a user is new or not (it is precised that timestamp is set to 0 for every first interaction of a user.
Something very interesting but that is hard to capture is the lag_time between two exercices. It can gives indication of a student taking notes of the last question for example.
Still in the timestamp aspect, I think it can be interesting to check last time a user had an interaction with a section of the toeic. If a user start to improve on a topic, but then don't go to it for days, he might have some troubles redoing the exercices of that section.
Point 1 - Yes
\nPoint 2 - The difference between the two consecutive timestamps for a student should give an indication of lag time? So like in the Saint+ paper, we do have both the elapsed time and the lag time here.. Of course the elapsed time is for the prev question…Still it does help apparently…
just to be clear, their definition is as follows:
\nelapsed time - the time taken for a student to answer (the previous question)
\nlag time - the time interval between adjacent learning activities.
From this, I guess we can calculate the difference between times to see if the student was taking some notes after exercises if that is what is needed. I dont know how strong that feature would be though. However this would indeed be a very strong feature for lectures - amount of time student watched the lecture versus average time lecture is typically watched but as far as I can see they dont give elapsed time for lectures. Still we can try to guess try to play around with timestamp differences and make some guesses based on lag times (of course we have to assume that student is doing all this in one continuous session and hasn't opened a new window to play their favorite video game :) could be a noisy feature but worth a try
", "votes": null, "replies": [] }, { "id": 1139568, "author_name": "authman", "author_url": "", "post_date": "01/05/2021 13:52:21", "content": "prior_question_elapsed_time is <NA> for lectures, regrettably. The full dataset has it; but not the version they prepared for our Kaggle.
@authman did u find a way to create a moving mask for the question bundles so that they dont attend to answers in same bundle? I can share my thought if u r looking for a solution
", "votes": null, "replies": [] }, { "id": 1139647, "author_name": "authman", "author_url": "", "post_date": "01/05/2021 14:42:49", "content": "Not satisfactorily =\\
", "votes": null, "replies": [] }, { "id": 1139745, "author_name": "allohvk", "author_url": "", "post_date": "01/05/2021 15:39:29", "content": "see if this works..
\nThe trick here is to create separate masks for interaction sequence and question sequence in each row (if u r using SAKT) and create multiple rows
\nQ1 Q2 Q3 Q4 Q5 Q6, Q7
\nI0 I1 I2 I3 I4 I5 I6
Let us say Q3, Q4, Q5 are a bundle, then:
\nI0 I1 I2 0 0 0 0
\nQ1 Q2 Q3 Q4 Q5 0 0
and
\nI0 I1 I2 I3 I4 I5 I6
\n0 0 0 0 0 Q6 Q7
Repeat for all bundles in row
", "votes": null, "replies": [] } ] }, { "id": 1139368, "author_name": "imeintanis", "author_url": "", "post_date": "01/05/2021 10:52:14", "content": "@allohvk thank you for compiling such a helpful post!! actually all of your discussions are worth a lot -:) I'll have to read many times to digest.. regarding the following point:
\n\n\nHOWEVER, we need to ensure that the data is richly represented. We could make use of a (learnable) weight matrix to transform the one-dimension vector into a certain dimension vector. Alternatively we could use self-attention to embed the input. Self-attention with all the fancy multiheads, residuals and all the standard tricks result in capturing the richness of the data well. Using self-attention will require positional encodings as the embedding has no knowledge of sequence order. There are some standard well-defined hacks to incorporate this and we will not go into the details.
\n
could you please point me to any example code/paper that uses self-att to capture the input? I don't remember to have come across smth like that - I'll give a try anyway but any source with info will help. Thank you and good luck to the comp
", "votes": null, "replies": [ { "id": 1139426, "author_name": "allohvk", "author_url": "", "post_date": "01/05/2021 11:39:06", "content": "Thanks @imeintanis :)
\nThis was written at the very beginning of the competition. So there could be slight changes here and there. Definitely self-attention should help. There are couple of Saint implementations and I believe they use self-attention because that is part of Saint architecture by default. Have you checked those. Unfortnuately my Kaggle ID had an issue till last week and only last weekend I took the SAKT kernel from Wang and trying to make changes to make it work. I am still stuck at basic changes..
Thank you I got it (actually I had misunderstood your point). Happy that you resolved your issues & you 're back again
\nps: I started from the same SAKT kernel but cannot surpass the 0780 stage.. we'll see