\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5845653%2F3d0b08620b0f8a9f3d8329cfb7d553cb%2FScreen%20Shot%202021-01-02%20at%2021.23.42.png?generation=1609595335852893&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5845653%2Ffb5ff5a592c5d1fd89bc279e388cf59f%2FScreen%20Shot%202021-01-02%20at%2021.22.02.png?generation=1609595402043305&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5845653%2F454ac35106fc09b72a4cefa956dff2f8%2FScreen%20Shot%202021-01-02%20at%2021.17.23.png?generation=1609595420452670&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5845653%2F59d4e988997b7255e0b766368cd48475%2FScreen%20Shot%202021-01-02%20at%2021.16.29.png?generation=1609595439205662&alt=media\")
Hey there all~
\nI've only just started working on my EDA (way too late actually XD) and just when I was thinking about things I could do I remembered the times I took Multiple Choice Question exams. I remembered that students get a bit uneasy when they choose the same LETTER answer (in our case [ 3, 2, 1, 0 ] numbers) multiple times in a row. And because of that we tend to guess the answer based on that trend when we are not sure or mark the incorrect the answer because it feels like a trap set up by the exam makers. So, I tested if my above statement was correct by following steps:
\nAs you can see above some users had excellent streak of correct answers while some had failed in between. In this 10M rows of data about 20 users had consecutive questions so I think in the full training set there will be a lot more. And as there will not be new question in the testing set I think there will also be consecutive questions with same number correct answer. As I'm beginner I'm not sure how to proceed with this finding. Perhaps this finding is not useful at all and may only add more noise to our prediction. What do you think?
\nP.S As this is my first discussion I must have worded what I wanted to inform slightly off.
\nCheers…
Hey there all~
\nI've only just started working on my EDA (way too late actually XD) and just when I was thinking about things I could do I remembered the times I took Multiple Choice Question exams. I remembered that students get a bit uneasy when they choose the same LETTER answer (in our case [ 3, 2, 1, 0 ] numbers) multiple times in a row. And because of that we tend to guess the answer based on that trend when we are not sure or mark the incorrect the answer because it feels like a trap set up by the exam makers. So, I tested if my above statement was correct by following steps:
\nAs you can see above some users had excellent streak of correct answers while some had failed in between. In this 10M rows of data about 20 users had consecutive questions so I think in the full training set there will be a lot more. And as there will not be new question in the testing set I think there will also be consecutive questions with same number correct answer. As I'm beginner I'm not sure how to proceed with this finding. Perhaps this finding is not useful at all and may only add more noise to our prediction. What do you think?
\nP.S As this is my first discussion I must have worded what I wanted to inform slightly off.
\nCheers…
Hi @temuujinerdene,
\nWelcome to the competition and perhaps more importantly, welcome to the participating side of the discussions :-).
\nThe points you bring up are very valid. However, although it hasn't been demonstratively proven and the host hasn't released any information confirming this, I think it's safe to say that the order the questions were presented to the users (A-B-C-D) vs the order we have in the dataset are not one and the same.
\nEach question in the dataset we're given has only a single correct_response for all users. Whereas the ITS (Intelligent Training System) app this data was collected within is called SANTA, and it'd be absolutely wild to assume they didn't do either a random shuffling of the response placement, or even intelligently permuting them to address the very concern you raise here:
\n\n\nstudents get a bit uneasy when they choose the same LETTER answer (in our case [ 3, 2, 1, 0 ] numbers) multiple times in a row
\n
If we were given truly raw dhat, this sort of EDA would surely provide some signal. But in our case I personally do not believe there is anything here.
", "rawMarkdown": "Hi @temuujinerdene,\n\nWelcome to the competition and perhaps more importantly, welcome to the participating side of the discussions :-).\n\nThe points you bring up are very valid. However, although it hasn't been demonstratively proven and the host hasn't released any information confirming this, I think it's safe to say that the order the questions were presented to the users (A-B-C-D) vs the order we have in the dataset are not one and the same.\n\nEach question in the dataset we're given has only a single correct_response for all users. Whereas the ITS (Intelligent Training System) app this data was collected within is called [SANTA](https://www.riiid.co/en/product), and it'd be absolutely **wild** to assume they didn't do either a random shuffling of the response placement, or even intelligently permuting them to address the very concern you raise here:\n\n> students get a bit uneasy when they choose the same LETTER answer (in our case [ 3, 2, 1, 0 ] numbers) multiple times in a row\n\nIf we were given truly raw dhat, this sort of EDA would surely provide some signal. But in our case I personally do not believe there is anything here.", "votes": null }, { "id": "1135892", "postDate": "01/02/2021 15:22:20", "content": "Your idea is nice and i did check on this around 3 weeks back, A plot below for the user_id \"1660941992\", the person as correctness streak as high as 35-40.
\n\n\nwelcome to the participating side of the discussions :-).
\n
Happy to see this line @authman!
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F835774%2F6c5d7ca0c33eda48dac9b8374a8dbf8f%2FScreenshot%202021-01-02%20at%208.51.03%20PM.png?generation=1609600885181634&alt=media\")
I also checked this and confirmed that it can increase auc when working with the training set. But I am struggling to understand how to implement this with the actual test set…
", "rawMarkdown": "I also checked this and confirmed that it can increase auc when working with the training set. But I am struggling to understand how to implement this with the actual test set...", "votes": null }, { "id": "1136210", "postDate": "01/02/2021 20:38:20", "content": "Hi @michaelj8
\n\n\nconfirmed that it can increase auc when working with the training set
\n
How did you confirm it increased auc in training set without that transferring into test set? By that I mean, the only way I am aware of to confirm would be to split train set and test it. But if such a test were possible, the same test should be replicateable against our submission set.
", "rawMarkdown": "Hi @michaelj8\n\n> confirmed that it can increase auc when working with the training set\n\nHow did you confirm it increased auc in training set without that transferring into test set? By that I mean, the only way I am aware of to confirm would be to split train set and test it. But if such a test were possible, the same test should be replicateable against our submission set.", "votes": null }, { "id": "1136330", "postDate": "01/03/2021 00:53:12", "content": "I see, thank you very much for your response.
", "rawMarkdown": "I see, thank you very much for your response.", "votes": null }, { "id": "1140183", "postDate": "01/05/2021 20:50:04", "content": "In my desperation for FE, I've revisited every single idea I've had along with everything mentioned by others—even things my gut disagreed with. In looking at this feature particularly, it does indeed seem as though there is some signal here. I don't know yet if its generalizeable or not. Completed some initial EDA on it and now attempting to featurize + test it.
", "rawMarkdown": "In my desperation for FE, I've revisited every single idea I've had along with everything mentioned by others—even things my gut disagreed with. In looking at this feature particularly, it does indeed seem as though there is some signal here. I don't know yet if its generalizeable or not. Completed some initial EDA on it and now attempting to featurize + test it.", "votes": null }, { "id": "1145200", "postDate": "01/09/2021 01:13:56", "content": "I used something similar but in two different ways
\nIn some of my models it help (as determined by feature importance in a lightgbm model which is what I was building everything off of). But as I started to add different variables this \"user answer bias\" became much less important and I decided to leave it out as it wasn't adding AUC improvement to my model.
\nI used MY test set from the training set if that makes sense. I have another reply above, but basically MY test set and the competition test set are different. I didn't really leave enough time to code for the formatting difference in the competition test set… there were some nuisances and I just didn't leave the time to figure it out.. (two kids, full time job… basically I wasn't worried about winning this thing[nor do I think I have the skills yet to do so)
", "rawMarkdown": "I used MY test set from the training set if that makes sense. I have another reply above, but basically MY test set and the competition test set are different. I didn't really leave enough time to code for the formatting difference in the competition test set... there were some nuisances and I just didn't leave the time to figure it out.. (two kids, full time job... basically I wasn't worried about winning this thing[nor do I think I have the skills yet to do so)", "votes": null } ], "comments": [ { "id": 1135878, "author_name": "authman", "author_url": "", "post_date": "01/02/2021 15:13:26", "content": "Hi @temuujinerdene,
\nWelcome to the competition and perhaps more importantly, welcome to the participating side of the discussions :-).
\nThe points you bring up are very valid. However, although it hasn't been demonstratively proven and the host hasn't released any information confirming this, I think it's safe to say that the order the questions were presented to the users (A-B-C-D) vs the order we have in the dataset are not one and the same.
\nEach question in the dataset we're given has only a single correct_response for all users. Whereas the ITS (Intelligent Training System) app this data was collected within is called SANTA, and it'd be absolutely wild to assume they didn't do either a random shuffling of the response placement, or even intelligently permuting them to address the very concern you raise here:
\n\n\nstudents get a bit uneasy when they choose the same LETTER answer (in our case [ 3, 2, 1, 0 ] numbers) multiple times in a row
\n
If we were given truly raw dhat, this sort of EDA would surely provide some signal. But in our case I personally do not believe there is anything here.
", "votes": null, "replies": [ { "id": 1136330, "author_name": "temuujinerdene", "author_url": "", "post_date": "01/03/2021 00:53:12", "content": "I see, thank you very much for your response.
", "votes": null, "replies": [] }, { "id": 1140183, "author_name": "authman", "author_url": "", "post_date": "01/05/2021 20:50:04", "content": "In my desperation for FE, I've revisited every single idea I've had along with everything mentioned by others—even things my gut disagreed with. In looking at this feature particularly, it does indeed seem as though there is some signal here. I don't know yet if its generalizeable or not. Completed some initial EDA on it and now attempting to featurize + test it.
", "votes": null, "replies": [] } ] }, { "id": 1135892, "author_name": "adityaecdrid", "author_url": "", "post_date": "01/02/2021 15:22:20", "content": "Your idea is nice and i did check on this around 3 weeks back, A plot below for the user_id \"1660941992\", the person as correctness streak as high as 35-40.
\n\n\nwelcome to the participating side of the discussions :-).
\n
Happy to see this line @authman!
\n
I also checked this and confirmed that it can increase auc when working with the training set. But I am struggling to understand how to implement this with the actual test set…
", "votes": null, "replies": [ { "id": 1136210, "author_name": "authman", "author_url": "", "post_date": "01/02/2021 20:38:20", "content": "Hi @michaelj8
\n\n\nconfirmed that it can increase auc when working with the training set
\n
How did you confirm it increased auc in training set without that transferring into test set? By that I mean, the only way I am aware of to confirm would be to split train set and test it. But if such a test were possible, the same test should be replicateable against our submission set.
", "votes": null, "replies": [] }, { "id": 1145204, "author_name": "michaelj8", "author_url": "", "post_date": "01/09/2021 01:17:42", "content": "I used MY test set from the training set if that makes sense. I have another reply above, but basically MY test set and the competition test set are different. I didn't really leave enough time to code for the formatting difference in the competition test set… there were some nuisances and I just didn't leave the time to figure it out.. (two kids, full time job… basically I wasn't worried about winning this thing[nor do I think I have the skills yet to do so)
", "votes": null, "replies": [] } ] }, { "id": 1145200, "author_name": "michaelj8", "author_url": "", "post_date": "01/09/2021 01:13:56", "content": "I used something similar but in two different ways
\nIn some of my models it help (as determined by feature importance in a lightgbm model which is what I was building everything off of). But as I started to add different variables this \"user answer bias\" became much less important and I decided to leave it out as it wasn't adding AUC improvement to my model.
\n