I wanted to enter this competition and have worked on it for a few weeks. However, my software engineering isn't good enough and I will not be able to process the rows in sufficient time. I'm therefore giving up and haven't made a submission.
\nHere are some titbits:
\nMy features :
\npd.DataFrame(columns=['index', 'row_id', 'timestamp', 'user_id', 'answered_correctly',
\n 'number_lectures_attended', 'question_id', 'bundle_id', 'part', 'tags',
\n 'no_of_tags_in_question', 'no_questions_in_bundle',
\n 'no_answers_in_question', 'no_of_tags_with_lectures',
\n 'percent_of_tags_with_lectures', 'cluster', 'average_bundle_time',
\n 'average_question_time', 'question_id_mean', 'bundle_id_mean',
\n 'part_mean', 'tags_mean', 'cluster_mean', 'average_bundle_time_mean',
\n 'average_question_time_mean', 'question_rank',
\n 'user_count', 'user_mean', 'user_correct', 'user_part_count',
\n 'user_part_mean', 'user_part_correct', 'user_cluster_count', 'user_cluster_mean', 'user_cluster_correct',
\n 'first_bundle', 'first_bundle_user_count', 'first_cluster', 'first_cluster_user_count',
\n 'last_timestamp', 'last_timestamp_incorrect'])
I preprocessed the questions and got mean scores, number of answers in question (using training data) etc. I then bucketed them. When there's 100 million rows in training and 2 million in test, I doubt the average for bundles, time, questions would change by updating them. I dealt with them as if knowing a question was hard or not as a priori. I just had this as a csv to be used by the kernel. These values are not updated by the model - they are fixed.
I used the hashing trick for ftrl for user_id, question_id, bundle id etc
I binarised the tags and clustered them into 20 clusters. I also ftrl hashed the tags
For training, update after a user bundle not a user question. A lot of the scripts update after each question but in the test data , you will only have the correct answer after the whole bundle (so can't use pandas shift etc)
Look at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.
Number 6 is important and should boost your score
\nMerry Christmas
\nBluefool
I wanted to enter this competition and have worked on it for a few weeks. However, my software engineering isn't good enough and I will not be able to process the rows in sufficient time. I'm therefore giving up and haven't made a submission.
\nHere are some titbits:
\nMy features :
\npd.DataFrame(columns=['index', 'row_id', 'timestamp', 'user_id', 'answered_correctly',
\n 'number_lectures_attended', 'question_id', 'bundle_id', 'part', 'tags',
\n 'no_of_tags_in_question', 'no_questions_in_bundle',
\n 'no_answers_in_question', 'no_of_tags_with_lectures',
\n 'percent_of_tags_with_lectures', 'cluster', 'average_bundle_time',
\n 'average_question_time', 'question_id_mean', 'bundle_id_mean',
\n 'part_mean', 'tags_mean', 'cluster_mean', 'average_bundle_time_mean',
\n 'average_question_time_mean', 'question_rank',
\n 'user_count', 'user_mean', 'user_correct', 'user_part_count',
\n 'user_part_mean', 'user_part_correct', 'user_cluster_count', 'user_cluster_mean', 'user_cluster_correct',
\n 'first_bundle', 'first_bundle_user_count', 'first_cluster', 'first_cluster_user_count',
\n 'last_timestamp', 'last_timestamp_incorrect'])
I preprocessed the questions and got mean scores, number of answers in question (using training data) etc. I then bucketed them. When there's 100 million rows in training and 2 million in test, I doubt the average for bundles, time, questions would change by updating them. I dealt with them as if knowing a question was hard or not as a priori. I just had this as a csv to be used by the kernel. These values are not updated by the model - they are fixed.
I used the hashing trick for ftrl for user_id, question_id, bundle id etc
I binarised the tags and clustered them into 20 clusters. I also ftrl hashed the tags
For training, update after a user bundle not a user question. A lot of the scripts update after each question but in the test data , you will only have the correct answer after the whole bundle (so can't use pandas shift etc)
Look at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.
Number 6 is important and should boost your score
\nMerry Christmas
\nBluefool
\n\nLook at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.
\n
I am very suspicious about this. 50% of the students in train set have 7900 as the first bundle_id. While for our purposes with this dataset they have anonymized the absolute datetime from us, on the Kaggle / Riiid AIEd side, I am almost all but certain that they will be evaluating on a time-based split. If that is the case, it's possible that early on in the program they had a smaller subset of questions, or less 'AI' implemented, resulting in everyone having the same initial question/bundle.
\nWhile it's possible that someone on the Kaggle or host side thought, \"hey, now that we've masked seasonal effects they can focus just on per user time series\" and then subsequently also prepared the public / private lbs as a random split… it'd strike me as very odd. Even in the saint/saint+ papers, eval was always the most recent data.
\nI guess best way to test is to prepare a sub against public lb, see how that handles compared to cv boost if any, and then pray that translates into private lb.
", "rawMarkdown": "> Look at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.\n\nI am very suspicious about this. 50% of the students in train set have 7900 as the first bundle_id. While for our purposes with this dataset they have anonymized the absolute datetime from us, on the Kaggle / Riiid AIEd side, I am almost all but certain that they will be evaluating on a time-based split. If that is the case, it's possible that early on in the program they had a smaller subset of questions, or less 'AI' implemented, resulting in everyone having the same initial question/bundle. \n\nWhile it's possible that someone on the Kaggle or host side thought, \"hey, now that we've masked seasonal effects they can focus just on per user time series\" and then subsequently also prepared the public / private lbs as a random split... it'd strike me as very odd. Even in the saint/saint+ papers, eval was always the most recent data.\n\nI guess best way to test is to prepare a sub against public lb, see how that handles compared to cv boost if any, and then pray that translates into private lb.", "votes": 4, "replies": [ { "id": 1125757, "postDate": "2020-12-25T03:17:31.403Z", "content": "Has Point#6 above helped anybody? I tried a single first_bundle==7900 feature, a generic first_bundle embedding, and first_bundle embedding where bundles < 40 users all get bucketed into one bin. None of these schemes had a positive impact on my local validation.
\nAnyone?
\nI haven't attempted first tag clustering, though I have made use of tag clustering features per the public kernels.
", "rawMarkdown": "Has Point#6 above helped anybody? I tried a single first_bundle==7900 feature, a generic first_bundle embedding, and first_bundle embedding where bundles < 40 users all get bucketed into one bin. None of these schemes had a positive impact on my local validation.\n\nAnyone?\n\nI haven't attempted first tag clustering, though I have made use of tag clustering features per the public kernels." }, { "id": 1125924, "postDate": "2020-12-25T07:19:43.373Z", "content": "Did tag clusters help you?
", "rawMarkdown": "Did tag clusters help you?" }, { "id": 1126261, "postDate": "2020-12-25T13:22:16.690Z", "content": "@nicohrubec please see my comment on @spacelx's kernel here.
", "rawMarkdown": "@nicohrubec please see my comment on @spacelx's kernel [here](https://www.kaggle.com/spacelx/2020-r3id-clustering-question-tags/comments#1124399).", "votes": 1 }, { "id": 1126269, "postDate": "2020-12-25T13:29:02.100Z", "content": "@authman Interesting. Thank you!
", "rawMarkdown": "@authman Interesting. Thank you!" } ] }, { "id": 1125462, "postDate": "2020-12-24T18:18:44.810Z", "content": "Currently, I use 69 features, and the submission running time is < 40 mins which means my codes are deeply optimized with many tricks I have never seen in the public notebooks. Thanks for your ideas, I will try your ideas this Sunday after finishing one certification exam. If it boosts my score, I will team up with you.
", "rawMarkdown": "Currently, I use 69 features, and the submission running time is < 40 mins which means my codes are deeply optimized with many tricks I have never seen in the public notebooks. Thanks for your ideas, I will try your ideas this Sunday after finishing one certification exam. If it boosts my score, I will team up with you.", "votes": 2 }, { "id": 1131847, "postDate": "2020-12-30T03:07:21.883Z", "content": "Thanks for the tips! I am just wondering what does hashing trick means?
", "rawMarkdown": "Thanks for the tips! I am just wondering what does hashing trick means?", "votes": 1, "replies": [ { "id": 1133211, "postDate": "2020-12-31T02:57:24.613Z", "content": "I wonder about that too
", "rawMarkdown": "I wonder about that too\n" }, { "id": 1133572, "postDate": "2020-12-31T10:32:36.367Z", "content": "Here is the sklearn implementation and description:
\nhttps://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html
Thanks @bowaka for the link. One follow up question: How can we use hashing here? (For ftrl)
", "rawMarkdown": "Thanks @bowaka for the link. One follow up question: How can we use hashing here? (For ftrl) " } ] }, { "id": 1128999, "postDate": "2020-12-28T01:30:31.850Z", "content": "I added starter bundle the AUC on the training set increased from 0.810418 to 0.818764 but the AUC on the validation set decreased from 0.787183 to 0.783996
Thanks for reporting and corroborating my assumption. If 1/2 the questions share the same starter bundle (and we already know that SANTA app starts with a series of diagnostic questions) then it doesn't make sense to assume there is signal in the first bundle a user is exposed to. It'd be a different story if there was a pre-pre-test that then bucketed people and had them start with questions based on accessed skill level. But we aren't told when the diagnostic questions end and when the real questions begin to make that calculation.
", "rawMarkdown": "Thanks for reporting and corroborating my assumption. If 1/2 the questions share the same starter bundle (and we already know that SANTA app starts with a series of diagnostic questions) then it doesn't make sense to assume there is signal in the first bundle a user is exposed to. It'd be a different story if there was a pre-pre-test that then bucketed people and had them start with questions based on accessed skill level. But we aren't told when the diagnostic questions end and when the real questions begin to make that calculation." }, { "id": 1129763, "postDate": "2020-12-28T14:32:18.790Z", "content": "The diagnostic questions dont have explanation..So maybe we keep rolling till we get questions with explanation and then we know that diagnostic questions have ended. But this is tricky biz for sure..
\nBut Isn't 5 a brilliant insight? wonder why no-one is talking about it.
", "rawMarkdown": "The diagnostic questions dont have explanation..So maybe we keep rolling till we get questions with explanation and then we know that diagnostic questions have ended. But this is tricky biz for sure..\n\nBut Isn't 5 a brilliant insight? wonder why no-one is talking about it. " }, { "id": 1129771, "postDate": "2020-12-28T14:39:32.137Z", "content": "It is but it's a pain to implement. Maybe for top lb scorers it'll help curb overfitting a bit. But if anything, it's a regularizer not a feature to boost score. @rodolphelampe mentioned doing something similar by means of using the attention mask itself, e.g. doing the update per-question, but only allowing the transformer to attend to out-of-bundle questions. I thought that was a more elegant implementation to accomplish more or less the same purpose. Coincidentally, his post also didn't get much 'attention' :P.
", "rawMarkdown": "It is but it's a pain to implement. Maybe for top lb scorers it'll help curb overfitting a bit. But if anything, it's a regularizer not a feature to boost score. @rodolphelampe mentioned doing something similar by means of using the [attention mask itself](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206620), e.g. doing the update per-question, but only allowing the transformer to attend to out-of-bundle questions. I thought that was a more elegant implementation to accomplish more or less the same purpose. Coincidentally, his post also didn't get much 'attention' :P." }, { "id": 1130146, "postDate": "2020-12-28T19:05:27.150Z", "content": "Not sure if its the best approach, but for item 5, after all your feature extraction and updates, you can do something like:
\ntrain.groupby([\"user_id\", \"task_container_id\"])[\"your_feature\"].transform(\"first\")
I personnaly create a temporary list, in which I put the rows from the same bundle_id. I then loop over that list to create the features, and once all features are created, I reloop on the list to update the cache files. It allows me as well to consider bundle_size as a feature.
", "rawMarkdown": "I personnaly create a temporary list, in which I put the rows from the same bundle_id. I then loop over that list to create the features, and once all features are created, I reloop on the list to update the cache files. It allows me as well to consider bundle_size as a feature.", "votes": 1 }, { "id": 1130247, "postDate": "2020-12-28T21:47:52.487Z", "content": "I take my comment back. I highly recommend everyone try out @jsaguiar's great tip above:
\n\n", "rawMarkdown": "I take my comment back. I highly recommend everyone try out @jsaguiar's great tip above:\n\n> `train.groupby([\"user_id\", \"task_container_id\"])[\"your_feature\"].transform(\"first\")`\n", "votes": 1 }, { "id": 1130430, "postDate": "2020-12-29T03:34:29.247Z", "content": "\n
train.groupby([\"user_id\", \"task_container_id\"])[\"your_feature\"].transform(\"first\")
elegant!
\nalso intuitively this should help for seq models surely
@wuwenmin how you calculate the cv? all sub item(not include padding value) in sequence AUC? or just last sub item in each sequence AUC?
", "rawMarkdown": "@wuwenmin how you calculate the cv? all sub item(not include padding value) in sequence AUC? or just last sub item in each sequence AUC?\n" }, { "id": 1130685, "postDate": "2020-12-29T08:30:02.737Z", "content": "I use timestamp check if the question in the same bundle,update dict while different bundle.it's useful for my LGBM
", "rawMarkdown": "I use timestamp check if the question in the same bundle,update dict while different bundle.it's useful for my LGBM" } ] }, { "id": 1124719, "postDate": "2020-12-24T06:54:53.563Z", "content": "Hi Bluefool,
\nThanks for sharing. I have one question on this part:
\nWhat do you mean when you say extract all the users with the same start?
\nMerry Christmas,
\nCakey
possibly he means that is one way of clustering students?
", "rawMarkdown": "possibly he means that is one way of clustering students?", "votes": 1 } ] }, { "id": 1124377, "postDate": "2020-12-23T21:46:00.943Z", "content": "Thank you, that's very kind to share so much !
\nI feel bad for you as you must have spend a lot of time on this competition. I spent a lot of time improving the computation time and I think we could help you. Do you have any script to share that shows where it is slow ?
\nI used viztracer which is an awesome library for profiling your code (using the chrome tracing tool), you could share that or I don't know what but feel free to share your difficulties and there might be solutions.
Thank you for your sharing,
\nCould you explain what does it mean
I've found that users whose first bundle_id is same have same subsequent content_ids (thanks to you)
\nsame amount of questions answered, same scores mean this?
Wish you Merry Christmas
\nBlueBird
Anybody running into this error: https://github.com/pytorch/pytorch/pull/24888
\nThis is during applying padding mask. Any suggestions or reference on padding masks welcome
\nNote: This is not peek-ahead mask. That is well implemented in the public kernels
Thanks for sharing and merry Christmas @domcastro
", "rawMarkdown": "Thanks for sharing and merry Christmas @domcastro " }, { "id": 1125578, "postDate": "2020-12-24T20:36:03.473Z", "content": "sir, are there any good links for ftrl trick and I want to learn about it.
", "rawMarkdown": "sir, are there any good links for ftrl trick and I want to learn about it.", "replies": [ { "id": 1125600, "postDate": "2020-12-24T21:07:12.947Z", "content": "https://research.google.com/pubs/archive/41159.pdf
\nIt is implemented as an optimizer in Keras.
", "rawMarkdown": "https://research.google.com/pubs/archive/41159.pdf\n\nIt is implemented as an optimizer in Keras.", "votes": 2 } ] }, { "id": 1125406, "postDate": "2020-12-24T17:30:48.073Z", "content": "Also, how many data fields per user do you need to maintain for your model? If it is a reasonable number (<100 or so), then it should not be difficult to make it run in 9 hours. But if it is a large number (like 10,000 or more), then it may not be possible to make it work at all.
", "rawMarkdown": "Also, how many data fields per user do you need to maintain for your model? If it is a reasonable number (<100 or so), then it should not be difficult to make it run in 9 hours. But if it is a large number (like 10,000 or more), then it may not be possible to make it work at all." }, { "id": 1125399, "postDate": "2020-12-24T17:20:32.863Z", "content": "What is your CV? If it is high enough (>0.800) you can team up with somebody who can help you optimize the submission code.
", "rawMarkdown": "What is your CV? If it is high enough (>0.800) you can team up with somebody who can help you optimize the submission code.", "replies": [ { "id": 1130569, "postDate": "2020-12-29T06:04:22.007Z", "content": "@ymatioun would u like to team up. our standing is just contributed by Saint model of 78.6 & basic lgb 77.2, we have built draft saint plus with 78.8 lb . we are making it better like saint. We target for gold let me know if you want to team up ..With our pipeline scores can be boosted further. we will have an lgb model with score of 79.2 . Towards the end we can get more boost and we will lend up in gold zone.
", "rawMarkdown": "@ymatioun would u like to team up. our standing is just contributed by Saint model of 78.6 & basic lgb 77.2, we have built draft saint plus with 78.8 lb . we are making it better like saint. We target for gold let me know if you want to team up ..With our pipeline scores can be boosted further. we will have an lgb model with score of 79.2 . Towards the end we can get more boost and we will lend up in gold zone.", "votes": -1 }, { "id": 1130774, "postDate": "2020-12-29T09:52:20.720Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ] }, { "id": 1124508, "postDate": "2020-12-24T02:14:13.123Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1128988, "postDate": "2020-12-28T00:53:00.843Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": 1 }, { "id": 1126057, "postDate": "2020-12-25T09:55:12.647Z", "content": "Thank you for your selfless sharing！
", "rawMarkdown": "Thank you for your selfless sharing！", "votes": 1 } ], "comments": [ { "id": 1125033, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-24T10:59:42.093000", "content": "\n\nLook at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.
\n
I am very suspicious about this. 50% of the students in train set have 7900 as the first bundle_id. While for our purposes with this dataset they have anonymized the absolute datetime from us, on the Kaggle / Riiid AIEd side, I am almost all but certain that they will be evaluating on a time-based split. If that is the case, it's possible that early on in the program they had a smaller subset of questions, or less 'AI' implemented, resulting in everyone having the same initial question/bundle.
\nWhile it's possible that someone on the Kaggle or host side thought, \"hey, now that we've masked seasonal effects they can focus just on per user time series\" and then subsequently also prepared the public / private lbs as a random split… it'd strike me as very odd. Even in the saint/saint+ papers, eval was always the most recent data.
\nI guess best way to test is to prepare a sub against public lb, see how that handles compared to cv boost if any, and then pray that translates into private lb.
", "votes": 4, "replies": [ { "id": 1125757, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-25T03:17:31.403000", "content": "Has Point#6 above helped anybody? I tried a single first_bundle==7900 feature, a generic first_bundle embedding, and first_bundle embedding where bundles < 40 users all get bucketed into one bin. None of these schemes had a positive impact on my local validation.
\nAnyone?
\nI haven't attempted first tag clustering, though I have made use of tag clustering features per the public kernels.
", "votes": 0, "replies": [] }, { "id": 1125924, "author_name": "hrunic", "author_url": "", "post_date": "2020-12-25T07:19:43.373000", "content": "Did tag clusters help you?
", "votes": 0, "replies": [] }, { "id": 1126261, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-25T13:22:16.690000", "content": "@nicohrubec please see my comment on @spacelx's kernel here.
", "votes": 1, "replies": [] }, { "id": 1126269, "author_name": "hrunic", "author_url": "", "post_date": "2020-12-25T13:29:02.100000", "content": "@authman Interesting. Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1125462, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-24T18:18:44.810000", "content": "Currently, I use 69 features, and the submission running time is < 40 mins which means my codes are deeply optimized with many tricks I have never seen in the public notebooks. Thanks for your ideas, I will try your ideas this Sunday after finishing one certification exam. If it boosts my score, I will team up with you.
", "votes": 2, "replies": [] }, { "id": 1131847, "author_name": "SHYJohn", "author_url": "", "post_date": "2020-12-30T03:07:21.883000", "content": "Thanks for the tips! I am just wondering what does hashing trick means?
", "votes": 1, "replies": [ { "id": 1133211, "author_name": "jwc", "author_url": "", "post_date": "2020-12-31T02:57:24.613000", "content": "I wonder about that too
", "votes": 0, "replies": [] }, { "id": 1133572, "author_name": "Jacky", "author_url": "", "post_date": "2020-12-31T10:32:36.367000", "content": "Here is the sklearn implementation and description:
\nhttps://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html
Thanks @bowaka for the link. One follow up question: How can we use hashing here? (For ftrl)
", "votes": 0, "replies": [] } ] }, { "id": 1128999, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-28T01:30:31.850000", "content": "I added starter bundle the AUC on the training set increased from 0.810418 to 0.818764 but the AUC on the validation set decreased from 0.787183 to 0.783996
Thanks for reporting and corroborating my assumption. If 1/2 the questions share the same starter bundle (and we already know that SANTA app starts with a series of diagnostic questions) then it doesn't make sense to assume there is signal in the first bundle a user is exposed to. It'd be a different story if there was a pre-pre-test that then bucketed people and had them start with questions based on accessed skill level. But we aren't told when the diagnostic questions end and when the real questions begin to make that calculation.
", "votes": 0, "replies": [] }, { "id": 1129763, "author_name": "Allohvk", "author_url": "", "post_date": "2020-12-28T14:32:18.790000", "content": "The diagnostic questions dont have explanation..So maybe we keep rolling till we get questions with explanation and then we know that diagnostic questions have ended. But this is tricky biz for sure..
\nBut Isn't 5 a brilliant insight? wonder why no-one is talking about it.
", "votes": 0, "replies": [] }, { "id": 1129771, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-28T14:39:32.137000", "content": "It is but it's a pain to implement. Maybe for top lb scorers it'll help curb overfitting a bit. But if anything, it's a regularizer not a feature to boost score. @rodolphelampe mentioned doing something similar by means of using the attention mask itself, e.g. doing the update per-question, but only allowing the transformer to attend to out-of-bundle questions. I thought that was a more elegant implementation to accomplish more or less the same purpose. Coincidentally, his post also didn't get much 'attention' :P.
", "votes": 0, "replies": [] }, { "id": 1130146, "author_name": "Aguiar", "author_url": "", "post_date": "2020-12-28T19:05:27.150000", "content": "Not sure if its the best approach, but for item 5, after all your feature extraction and updates, you can do something like:
\ntrain.groupby([\"user_id\", \"task_container_id\"])[\"your_feature\"].transform(\"first\")
I personnaly create a temporary list, in which I put the rows from the same bundle_id. I then loop over that list to create the features, and once all features are created, I reloop on the list to update the cache files. It allows me as well to consider bundle_size as a feature.
", "votes": 1, "replies": [] }, { "id": 1130247, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-28T21:47:52.487000", "content": "I take my comment back. I highly recommend everyone try out @jsaguiar's great tip above:
\n\n", "votes": 1, "replies": [] }, { "id": 1130430, "author_name": "Allohvk", "author_url": "", "post_date": "2020-12-29T03:34:29.247000", "content": "\n
train.groupby([\"user_id\", \"task_container_id\"])[\"your_feature\"].transform(\"first\")
elegant!
\nalso intuitively this should help for seq models surely
@wuwenmin how you calculate the cv? all sub item(not include padding value) in sequence AUC? or just last sub item in each sequence AUC?
", "votes": 0, "replies": [] }, { "id": 1130685, "author_name": "qiaqia", "author_url": "", "post_date": "2020-12-29T08:30:02.737000", "content": "I use timestamp check if the question in the same bundle,update dict while different bundle.it's useful for my LGBM
", "votes": 0, "replies": [] } ] }, { "id": 1124719, "author_name": "Cakey", "author_url": "", "post_date": "2020-12-24T06:54:53.563000", "content": "Hi Bluefool,
\nThanks for sharing. I have one question on this part:
\nWhat do you mean when you say extract all the users with the same start?
\nMerry Christmas,
\nCakey
possibly he means that is one way of clustering students?
", "votes": 1, "replies": [] } ] }, { "id": 1124377, "author_name": "Rodolphe Lampe", "author_url": "", "post_date": "2020-12-23T21:46:00.943000", "content": "Thank you, that's very kind to share so much !
\nI feel bad for you as you must have spend a lot of time on this competition. I spent a lot of time improving the computation time and I think we could help you. Do you have any script to share that shows where it is slow ?
\nI used viztracer which is an awesome library for profiling your code (using the chrome tracing tool), you could share that or I don't know what but feel free to share your difficulties and there might be solutions.
Thank you for your sharing,
\nCould you explain what does it mean
I've found that users whose first bundle_id is same have same subsequent content_ids (thanks to you)
\nsame amount of questions answered, same scores mean this?
Wish you Merry Christmas
\nBlueBird
Anybody running into this error: https://github.com/pytorch/pytorch/pull/24888
\nThis is during applying padding mask. Any suggestions or reference on padding masks welcome
\nNote: This is not peek-ahead mask. That is well implemented in the public kernels
Thanks for sharing and merry Christmas @domcastro
", "votes": 0, "replies": [] }, { "id": 1125578, "author_name": "Zhenghan Chen", "author_url": "", "post_date": "2020-12-24T20:36:03.473000", "content": "sir, are there any good links for ftrl trick and I want to learn about it.
", "votes": 0, "replies": [ { "id": 1125600, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-24T21:07:12.947000", "content": "https://research.google.com/pubs/archive/41159.pdf
\nIt is implemented as an optimizer in Keras.
", "votes": 2, "replies": [] } ] }, { "id": 1125406, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2020-12-24T17:30:48.073000", "content": "Also, how many data fields per user do you need to maintain for your model? If it is a reasonable number (<100 or so), then it should not be difficult to make it run in 9 hours. But if it is a large number (like 10,000 or more), then it may not be possible to make it work at all.
", "votes": 0, "replies": [] }, { "id": 1125399, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2020-12-24T17:20:32.863000", "content": "What is your CV? If it is high enough (>0.800) you can team up with somebody who can help you optimize the submission code.
", "votes": 0, "replies": [ { "id": 1130569, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-29T06:04:22.007000", "content": "@ymatioun would u like to team up. our standing is just contributed by Saint model of 78.6 & basic lgb 77.2, we have built draft saint plus with 78.8 lb . we are making it better like saint. We target for gold let me know if you want to team up ..With our pipeline scores can be boosted further. we will have an lgb model with score of 79.2 . Towards the end we can get more boost and we will lend up in gold zone.
", "votes": -1, "replies": [] }, { "id": 1130774, "author_name": "", "author_url": "", "post_date": "2020-12-29T09:52:20.720000", "content": "", "votes": -2, "replies": [] } ] }, { "id": 1124508, "author_name": "", "author_url": "", "post_date": "2020-12-24T02:14:13.123000", "content": "", "votes": 0, "replies": [] }, { "id": 1128988, "author_name": "Lokesh", "author_url": "", "post_date": "2020-12-28T00:53:00.843000", "content": "Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 1126057, "author_name": "Yi Song", "author_url": "", "post_date": "2020-12-25T09:55:12.647000", "content": "Thank you for your selfless sharing！
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1124357": "I wanted to enter this competition and have worked on it for a few weeks. However, my software engineering isn't good enough and I will not be able to process the rows in sufficient time. I'm therefore giving up and haven't made a submission.\n\nHere are some titbits:\n\n1. My features :\npd.DataFrame(columns=['index', 'row_id', 'timestamp', 'user_id', 'answered_correctly',\n 'number_lectures_attended', 'question_id', 'bundle_id', 'part', 'tags',\n 'no_of_tags_in_question', 'no_questions_in_bundle',\n 'no_answers_in_question', 'no_of_tags_with_lectures',\n 'percent_of_tags_with_lectures', 'cluster', 'average_bundle_time',\n 'average_question_time', 'question_id_mean', 'bundle_id_mean',\n 'part_mean', 'tags_mean', 'cluster_mean', 'average_bundle_time_mean',\n 'average_question_time_mean', 'question_rank', \n 'user_count', 'user_mean', 'user_correct', 'user_part_count', \n 'user_part_mean', 'user_part_correct', 'user_cluster_count', 'user_cluster_mean', 'user_cluster_correct',\n 'first_bundle', 'first_bundle_user_count', 'first_cluster', 'first_cluster_user_count',\n 'last_timestamp', 'last_timestamp_incorrect'])\n\n2. I preprocessed the questions and got mean scores, number of answers in question (using training data) etc. I then bucketed them. When there's 100 million rows in training and 2 million in test, I doubt the average for bundles, time, questions would change by updating them. I dealt with them as if knowing a question was hard or not as a priori. I just had this as a csv to be used by the kernel. These values are not updated by the model - they are fixed.\n\n3. I used the hashing trick for ftrl for user_id, question_id, bundle id etc\n\n4. I binarised the tags and clustered them into 20 clusters. I also ftrl hashed the tags\n\n5. For training, update after a user bundle not a user question. A lot of the scripts update after each question but in the test data , you will only have the correct answer after the whole bundle (so can't use pandas shift etc)\n\n6. Look at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.\n\nNumber 6 is important and should boost your score\n\nMerry Christmas\nBluefool\n\n\n\n", "1125033": "> Look at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and extract all the users with the same start.\n\nI am very suspicious about this. 50% of the students in train set have 7900 as the first bundle_id. While for our purposes with this dataset they have anonymized the absolute datetime from us, on the Kaggle / Riiid AIEd side, I am almost all but certain that they will be evaluating on a time-based split. If that is the case, it's possible that early on in the program they had a smaller subset of questions, or less 'AI' implemented, resulting in everyone having the same initial question/bundle. \n\nWhile it's possible that someone on the Kaggle or host side thought, \"hey, now that we've masked seasonal effects they can focus just on per user time series\" and then subsequently also prepared the public / private lbs as a random split... it'd strike me as very odd. Even in the saint/saint+ papers, eval was always the most recent data.\n\nI guess best way to test is to prepare a sub against public lb, see how that handles compared to cv boost if any, and then pray that translates into private lb.", "1125462": "Currently, I use 69 features, and the submission running time is < 40 mins which means my codes are deeply optimized with many tricks I have never seen in the public notebooks. Thanks for your ideas, I will try your ideas this Sunday after finishing one certification exam. If it boosts my score, I will team up with you.", "1131847": "Thanks for the tips! I am just wondering what does hashing trick means?", "1128999": "I added `starter bundle` the AUC on the training set increased from 0.810418 to 0.818764 but the AUC on the validation set decreased from 0.787183 to 0.783996", "1124719": "Hi Bluefool, \n\nThanks for sharing. I have one question on this part: \n\n6. Look at the starter bundle of each user. 7900 is very common. You will see that the bundle that the user starts has a subsequent pattern - same amount of questions answered, same scores. Therefore I capture the users very first bundle and first cluster (of tags) and **extract all the users with the same start**.\n\nWhat do you mean when you say extract all the users with the same start? \n\nMerry Christmas, \nCakey\n", "1124377": "Thank you, that's very kind to share so much !\nI feel bad for you as you must have spend a lot of time on this competition. I spent a lot of time improving the computation time and I think we could help you. Do you have any script to share that shows where it is slow ?\nI used viztracer which is an awesome library for profiling your code (using the chrome tracing tool), you could share that or I don't know what but feel free to share your difficulties and there might be solutions.", "1124645": "Thank you for your sharing,\nCould you explain what does it mean\n6. 'same amount of questions answered, same scores.'\n\nI've found that users whose first bundle_id is same have same subsequent content_ids (thanks to you)\nsame amount of questions answered, same scores mean this?\n\nWish you Merry Christmas\nBlueBird", "1130989": "Anybody running into this error: https://github.com/pytorch/pytorch/pull/24888 \nThis is during applying padding mask. Any suggestions or reference on padding masks welcome\nNote: This is not peek-ahead mask. That is well implemented in the public kernels", "1127018": "Thanks for sharing and merry Christmas @domcastro ", "1125578": "sir, are there any good links for ftrl trick and I want to learn about it.", "1125406": "Also, how many data fields per user do you need to maintain for your model? If it is a reasonable number (<100 or so), then it should not be difficult to make it run in 9 hours. But if it is a large number (like 10,000 or more), then it may not be possible to make it work at all.", "1125399": "What is your CV? If it is high enough (>0.800) you can team up with somebody who can help you optimize the submission code.", "1124508": "", "1128988": "Thanks for sharing.", "1126057": "Thank you for your selfless sharing！" } }