\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F291298%2Feb4ec9b30ff5ceae322d677dba13aff4%2FScreen%20Shot%202020-10-26%20at%2011.24.00%20AM.png?generation=1603736688404605&alt=media\")
I'm updating the dataset now so lecture tags will match the tags in questions.csv. The updated lectures.csv should be live within the next hour. Apologies for the inconvenience.
Correction- the hidden test set contains new users but not new questions.
The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data. It remains possible that some questions weren't logged due to other issues that all datasets of mobile users are susceptible to,
\nsuch as if a user lost their connection mid-question.
The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.
I'm updating the dataset now so lecture tags will match the tags in questions.csv. The updated lectures.csv should be live within the next hour. Apologies for the inconvenience.
Correction- the hidden test set contains new users but not new questions.
The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data. It remains possible that some questions weren't logged due to other issues that all datasets of mobile users are susceptible to,
\nsuch as if a user lost their connection mid-question.
The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.
@sohier Thanks for the clarification! Now we can start to build the sequential model. Would you mind to answer the following questions?
\nQ1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).
\nQ2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?
\nQ3. If the submission will be re-run after the deadline, would questions.csv, lectures.csv and the private test dataset remain the same as when we submit for the public LB score before the deadline?
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190791
", "rawMarkdown": "@sohier Thanks for the clarification! Now we can start to build the sequential model. Would you mind to answer the following questions?\n\n\nQ1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\nQ2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?\n\nQ3. If the submission will be re-run after the deadline, would questions.csv, lectures.csv and the private test dataset remain the same as when we submit for the public LB score before the deadline?\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190791", "votes": 5 }, { "id": 1055372, "postDate": "2020-10-20T17:50:47.367Z", "content": "Roughly what percentage of users in the hidden test set will be new users?
", "rawMarkdown": "Roughly what percentage of users in the hidden test set will be new users?", "votes": 4 }, { "id": 1065394, "postDate": "2020-10-31T08:36:39.717Z", "content": "@sohier When I try to use TPUs to make a submission it errs with \"Your Notebook cannot use TPUs in this competition.\" But as per the code requirements we can use TPUs, so is this a glitch?
", "rawMarkdown": "@sohier When I try to use TPUs to make a submission it errs with \"Your Notebook cannot use TPUs in this competition.\" But as per the code requirements we can use TPUs, so is this a glitch?", "votes": 3, "replies": [ { "id": 1065401, "postDate": "2020-10-31T08:51:48.433Z", "content": "The Code Requirements have been updated / changed since the start of the competition.
\nInitially internet access was not allowed either but that condition isn't mentioned anymore.
@sohier
\nWould be good to get clarification / confirmation on this since there has been no official post about the updates.
@rohanrao I just tried to submit a notebook with Internet access turned on, and I got
\n\n", "rawMarkdown": "@rohanrao I just tried to submit a notebook with Internet access turned on, and I got \n> Your Notebook cannot use internet access in this competition. Please disable internet in the Notebook editor and save a new version.", "votes": 1 }, { "id": 1069885, "postDate": "2020-11-05T03:40:58.783Z", "content": "Your Notebook cannot use internet access in this competition. Please disable internet in the Notebook editor and save a new version.
\n
@sohier needs to confirm regarding the mismatch between Code Requirements and actual backend since something changed.
", "rawMarkdown": "@sohier needs to confirm regarding the mismatch between Code Requirements and actual backend since something changed.", "votes": 4 } ] }, { "id": 1050278, "postDate": "2020-10-15T08:08:37.387Z", "content": "Thanks for the clarifications. I have a request that could you please show more details of the \"Submission Scoring Error\" when we make submission. Each submission spends really long time and finnally \"Submission Scoring Error\" without any addition details. It's better to add some logs of the error, so we will not waste our time in debug and forcus more on the feature engineering and modeling.
", "rawMarkdown": "Thanks for the clarifications. I have a request that could you please show more details of the \"Submission Scoring Error\" when we make submission. Each submission spends really long time and finnally \"Submission Scoring Error\" without any addition details. It's better to add some logs of the error, so we will not waste our time in debug and forcus more on the feature engineering and modeling.", "votes": 3, "replies": [ { "id": 1050342, "postDate": "2020-10-15T09:49:25.227Z", "content": "They cannot add it else people will dump the whole data into logs, you can always raise an error at will.
", "rawMarkdown": "They cannot add it else people will dump the whole data into logs, you can always raise an error at will.", "votes": 5 } ] }, { "id": 1049944, "postDate": "2020-10-14T22:51:46.527Z", "content": "Thanks for the clarifications. Will the hidden set not contain new lectures as well?
", "rawMarkdown": "Thanks for the clarifications. Will the hidden set not contain new lectures as well?\n ", "votes": 3, "replies": [ { "id": 1060721, "postDate": "2020-10-26T14:00:22.507Z", "content": "well, it's obvious that if there is no new question that implies there no new lectures.
", "rawMarkdown": "well, it's obvious that if there is no new question that implies there no new lectures.\n" }, { "id": 1060731, "postDate": "2020-10-26T14:09:16.593Z", "content": "questions and lectures are different datasets so it's not completely obvious. There can be new lectures.
", "rawMarkdown": "questions and lectures are different datasets so it's not completely obvious. There can be new lectures.", "votes": 4 } ] }, { "id": 1060803, "postDate": "2020-10-26T14:50:11.390Z", "content": "Roughly what percentage of users in the hidden test set will be new users?
", "rawMarkdown": "Roughly what percentage of users in the hidden test set will be new users?", "votes": 2, "replies": [ { "id": 1060827, "postDate": "2020-10-26T14:57:50.530Z", "content": "Just like in real-world scenarios, we look at historical (train) data, observe timestamps and frequency of users in order to estimate future (test) data distribution.
Since the data is time-series, you could also take 1st 80% data as train and last 20% data as test, and then perform this calculation viz., how many users are \"new\" i.e. present only in the last 20% of the train data.
Hey @sohier and community! I have one last question about the code competition set up. It has been a while since I have participated in code competition but it looks like we can train models offline these days and load them to the kernel environment.
\n\n\nPlease note that for this competition training is not required in Notebooks.
\n
My question is if we have to make it available to the community as well? Does it count as a pre-trained model?
\n\n\nFreely & publicly available external data is allowed, including pre-trained models
\n
I am sure this has been answered multiple times since the code competitions have been updated but I wasn't sure where to look.
", "rawMarkdown": "Hey @sohier and community! I have one last question about the code competition set up. It has been a while since I have participated in code competition but it looks like we can train models offline these days and load them to the kernel environment.\n\n> Please note that for this competition training is not required in Notebooks.\n\nMy question is if we have to make it available to the community as well? Does it count as a pre-trained model?\n\n> Freely & publicly available external data is allowed, including pre-trained models\n\nI am sure this has been answered multiple times since the code competitions have been updated but I wasn't sure where to look.", "votes": 1, "replies": [ { "id": 1134308, "postDate": "2021-01-01T06:03:08.023Z", "content": "Hi @rdizzl3, did you find an answer for these questions? I am also interested. Thanks and good luck in this competition!
", "rawMarkdown": "Hi @rdizzl3, did you find an answer for these questions? I am also interested. Thanks and good luck in this competition!" } ] }, { "id": 1066177, "postDate": "2020-11-01T12:22:47.757Z", "content": "Thank you @sohier for the clarification.
", "rawMarkdown": "Thank you @sohier for the clarification.", "votes": 1 }, { "id": 1061416, "postDate": "2020-10-27T02:16:20.263Z", "content": "content_type_id: Please include a lecture in the test data (Example.csv) so that we can verify that lectures are handled correctly before submitting to the hidden data.
\nThanks.
", "rawMarkdown": "content_type_id: Please include a lecture in the test data (Example.csv) so that we can verify that lectures are handled correctly before submitting to the hidden data.\n\nThanks.", "votes": 1, "replies": [ { "id": 1061521, "postDate": "2020-10-27T04:01:20.410Z", "content": "Mike, you can take a look at this!
", "rawMarkdown": "Mike, you can take a look at [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191856)!" }, { "id": 1061536, "postDate": "2020-10-27T04:28:47.707Z", "content": "Thanks. Now have a solution to my problem. Here is how to insert lectures into the Example dataframe test_df. This works for me 😊
\nImprovements welcome!
\nBorrowing from: https://www.geeksforgeeks.org/insert-row-at-given-position-in-pandas-dataframe/
\ndef Insert_row(row_number, df, row_value):
\n # Starting value of upper half
\n start_upper = 0
# End value of upper half \nend_upper = row_number \n\n# Start value of lower half \nstart_lower = row_number \n\n# End value of lower half \nend_lower = df.shape[0] \n\n# Create a list of upper_half index \nupper_half = [*range(start_upper, end_upper, 1)] \n\n# Create a list of lower_half index \nlower_half = [*range(start_lower, end_lower, 1)] \n\n# Increment the value of lower half by 1 \nlower_half = [x.__add__(1) for x in lower_half] \n\n# Combine the two lists \nindex_ = upper_half + lower_half \n\n# Update the index of the dataframe \ndf.index = index_ \n\n# Insert a row at the end \ndf.loc[row_number] = row_value \n\n# Sort the index labels \ndf = df.sort_index() \n\n# return the dataframe \nreturn df \niter_test = env.iter_test()
\nfor (test_df, sample_prediction_df) in iter_test:
# Let's create a row which we want to insert \n row_number = 2\n row_value = [89,653762,2746,6808,1,14,-1,-1,0,False] \n\n if row_number > test_df.index.max()+1: \n print(\"Invalid row_number\", row_number, test_df.index.max()) \n else: \n\n# Let's call the function and insert the row \n test_df = Insert_row(row_number, test_df, row_value) \n\n# Print the updated dataframe \n print(test_df) \n….
", "rawMarkdown": "Thanks. Now have a solution to my problem. Here is how to insert lectures into the Example dataframe test_df. This works for me 😊\n\nImprovements welcome!\n\nBorrowing from: https://www.geeksforgeeks.org/insert-row-at-given-position-in-pandas-dataframe/\n\n# Function to insert row in the dataframe \ndef Insert_row(row_number, df, row_value): \n # Starting value of upper half \n start_upper = 0\n \n # End value of upper half \n end_upper = row_number \n \n # Start value of lower half \n start_lower = row_number \n \n # End value of lower half \n end_lower = df.shape[0] \n \n # Create a list of upper_half index \n upper_half = [*range(start_upper, end_upper, 1)] \n \n # Create a list of lower_half index \n lower_half = [*range(start_lower, end_lower, 1)] \n \n # Increment the value of lower half by 1 \n lower_half = [x.__add__(1) for x in lower_half] \n \n # Combine the two lists \n index_ = upper_half + lower_half \n \n # Update the index of the dataframe \n df.index = index_ \n \n # Insert a row at the end \n df.loc[row_number] = row_value \n \n # Sort the index labels \n df = df.sort_index() \n \n # return the dataframe \n return df \n\niter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n\n # Let's create a row which we want to insert \n row_number = 2\n row_value = [89,653762,2746,6808,1,14,-1,-1,0,False] \n\n if row_number > test_df.index.max()+1: \n print(\"Invalid row_number\", row_number, test_df.index.max()) \n else: \n \n # Let's call the function and insert the row \n test_df = Insert_row(row_number, test_df, row_value) \n \n # Print the updated dataframe \n print(test_df) \n\n....", "votes": 1 }, { "id": 1067188, "postDate": "2020-11-02T11:48:52.100Z", "content": "@mikel1
\n1) could u help understand the need for insert of new lectures ?
2) Can test set contain only question ids that are present in questions.csv ?
", "rawMarkdown": "@mikel1 \n1) could u help understand the need for insert of new lectures ?\n\n2) Can test set contain only question ids that are present in questions.csv ?" } ] }, { "id": 1054875, "postDate": "2020-10-20T08:47:22.993Z", "content": "Sohier, in your very instructive \"Competition API Detailed Introduction\", you stop a little too soon.
\nHere are the next steps:
\n1) Have a Notebook that successfully predicts the Example test data when \"Run All\" is clicked
\n Be sure that Setting \"Internet\" is off.
\n2) Save to share.
\n3) Share with the public
\n4) On the Notebooks page, click on it
\n5) Execution info
\n6) Click on \"Submit\"
\nYou will be told that your Notebook is running.
Thanks for the clarification! The new question_id in the hidden test set and the lack of sequence of test data have bothered me for a long time.
", "rawMarkdown": "Thanks for the clarification! The new **question_id** in the hidden test set and the lack of sequence of test data have bothered me for a long time.", "votes": 1 }, { "id": 1123192, "postDate": "2020-12-23T02:58:12.217Z", "content": "@sohier apologies if this has been covered elsewhere - I read through the updates and searched the other discussions, but didn't see anything.
\nI ran this book to see whether there were many instances of gaps of more than one in the sequence of task_container_ids. I set it to error out if the count got over 100,000 and it errored out soon after it started. I understand there may be gaps due to the timing of user interactions on multiple devices and as a result task_container_ids are not sequential when records are ordered for each user by timestamp. I calculated that there are 2.5 million of such instances, in the training set, or approximately 2.5% of all of the observations. If the proportion was similar in the test set that would equate to approximately 62,500 instances, so erroring out after 100,000 would seem to indicate another source of them not being sequential.
", "rawMarkdown": "@sohier apologies if this has been covered elsewhere - I read through the updates and searched the other discussions, but didn't see anything.\n\nI ran [this book](https://www.kaggle.com/calebeverett/test-tid-deltas) to see whether there were many instances of gaps of more than one in the sequence of task_container_ids. I set it to error out if the count got over 100,000 and it errored out soon after it started. I understand there may be gaps due to the timing of user interactions on multiple devices and as a result task_container_ids are not sequential when records are ordered for each user by timestamp. I calculated that there are 2.5 million of such instances, in the training set, or approximately 2.5% of all of the observations. If the proportion was similar in the test set that would equate to approximately 62,500 instances, so erroring out after 100,000 would seem to indicate another source of them not being sequential.", "votes": 2 }, { "id": 1061101, "postDate": "2020-10-26T18:35:28.533Z", "content": "Hey @sohier thank you for the corrections and clarifications. I am about to enter the competition but I feel like I don't quite understand how the test API works for the new test users. I am looking at the example_test.csv and there is a user present in the test data and not in the training data.
Here is what that user looks like:
\n
My first question is making sure I understand the groups correctly -
\nThe lists from group 1 are the correct answers, etc. from questions in group 0?
If the list from group 1 is their data from group zero, shouldn't there be different content ids (possibly) for the different questions? There is only one content id in group 0.
Sorry if this has been explained somewhere else. I read quite a few topics in the forum but couldn't figure out how to correctly map and use this info.
", "rawMarkdown": "Hey @sohier thank you for the corrections and clarifications. I am about to enter the competition but I feel like I don't quite understand how the test API works for the new test users. I am looking at the `example_test.csv` and there is a user present in the test data and not in the training data.\n\nHere is what that user looks like:\n\n\nMy first question is making sure I understand the groups correctly - \n1. The lists from group 1 are the correct answers, etc. from questions in group 0?\n\n2. If the list from group 1 is their data from group zero, shouldn't there be different content ids (possibly) for the different questions? There is only one content id in group 0.\n\nSorry if this has been explained somewhere else. I read quite a few topics in the forum but couldn't figure out how to correctly map and use this info.", "votes": 2, "replies": [ { "id": 1061108, "postDate": "2020-10-26T18:45:51.650Z", "content": "\n\nThe lists from group 1 are the correct answers, etc. from questions in group 0?
\n
Yes.
\n\n\nThere is only one content id in group 0.
\n
Don't filter on a user (it is irrelevant / incorrect). The length of the answer list in group n will exactly match the number of rows in group (n-1) (considering all users together).
Highly recommend to go through this.
", "rawMarkdown": "> The lists from group 1 are the correct answers, etc. from questions in group 0?\n\nYes.\n\n> There is only one content id in group 0.\n\nDon't filter on a user (it is irrelevant / incorrect). The length of the answer list in `group n` will exactly match the number of rows in `group (n-1)` (considering all users together).\n\nHighly recommend to go through [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192124).", "votes": 3 }, { "id": 1061115, "postDate": "2020-10-26T18:53:54.480Z", "content": "Plus there's video's labels in that list as well! So be careful with that as we don't have to make preds on video_rows so to say (content_type_id == 1).
", "rawMarkdown": "Plus there's video's labels in that list as well! So be careful with that as we don't have to make preds on video_rows so to say (content_type_id == 1).", "votes": 2 }, { "id": 1061122, "postDate": "2020-10-26T19:01:19.230Z", "content": "thank you @vopani! I did not realize that topic had all this info! But I was able to verify everything I was questioning. Appreciate the comment!
", "rawMarkdown": "thank you @vopani! I did not realize that topic had all this info! But I was able to verify everything I was questioning. Appreciate the comment!", "votes": 1 } ] }, { "id": 1050025, "postDate": "2020-10-15T02:14:32.560Z", "content": "Thanks for the clarifications. Will it be possible to simply return the data from questions and lectures pre-appended to the test-df as well? Because it will be joined by us either ways, so IMO it's a good thing to return it when we are iterating over the test_df as it's columns?
\nAnd the only file updated is lectures.csv, right?
", "rawMarkdown": "Thanks for the clarifications. Will it be possible to simply return the data from questions and lectures pre-appended to the test-df as well? Because it will be joined by us either ways, so IMO it's a good thing to return it when we are iterating over the test_df as it's columns?\n\nAnd the only file updated is lectures.csv, right?", "votes": 2 }, { "id": 1050004, "postDate": "2020-10-15T01:17:38.407Z", "content": "Thanks for the clarification about private set. Now it help and have better clarification about private set.
\n@sohier Do you mean.. train/test set is complete having all user activities logged except few missing values are negligible ratio ?
", "rawMarkdown": "Thanks for the clarification about private set. Now it help and have better clarification about private set.\n\n@sohier Do you mean.. **train/test set is complete** having all user activities logged except **few missing** values are **negligible ratio** ?", "votes": 2 }, { "id": 1134231, "postDate": "2021-01-01T03:14:54.877Z", "content": "@sohier Thanks for the clarification about private set. Now it help and have better clarification about private set.
\n\n\nThe test data follows chronologically after the train data. The test iterations give interactions of users chronologically.
\n
Regarding this matter, does chronologically mean that in order of timestamp, rather than task_container_id?
", "rawMarkdown": "@sohier Thanks for the clarification about private set. Now it help and have better clarification about private set.\n\n> The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.\n\nRegarding this matter, does chronologically mean that in order of **timestamp**, rather than task_container_id?" }, { "id": 1121782, "postDate": "2020-12-21T22:44:02.397Z", "content": "I was curious if the task_container_ids continue in sequence from the training data to the test data. It looks like there are gaps between the training data and sample test data.
\nBelow are the first ten sample test records.
\n| \n | user_id | \nrow_id | \ntask_container_id | \nmax_train_tid | \ntid_delta | \nts_delta_sec | \n
|---|---|---|---|---|---|---|
| 1 | \n554169193 | \n1 | \n4427 | \n4421.0 | \n6.0 | \n184.627 | \n
| 2 | \n1720860329 | \n2 | \n240 | \n235.0 | \n5.0 | \n299.297 | \n
| 3 | \n288641214 | \n3 | \n266 | \n262.0 | \n4.0 | \n72252.433 | \n
| 4 | \n1728340777 | \n4 | \n162 | \n161.0 | \n1.0 | \n67009.619 | \n
| 5 | \n1364159702 | \n5 | \n4424 | \n4421.0 | \n3.0 | \n173642.470 | \n
| 6 | \n1521618396 | \n6 | \n1367 | \n1361.0 | \n6.0 | \n202.134 | \n
| 7 | \n1317245193 | \n7 | \n5314 | \n5307.0 | \n7.0 | \n384674.701 | \n
| 8 | \n1700555100 | \n8 | \n532 | \n530.0 | \n2.0 | \n246.422 | \n
| 9 | \n998511398 | \n9 | \n393 | \n389.0 | \n4.0 | \n37118.438 | \n
| 10 | \n1422853669 | \n10 | \n85 | \n79.0 | \n6.0 | \n194.973 | \n
Are those gaps related to the preparation of the sample test data or are there similar gaps in the actual test data as well?
\nEDIT: updated to include timetamp delta as well, which look like they might span gaps in task_container_ids. I had a median gap of approx. 42 seconds on the last set of user training records.
", "rawMarkdown": "I was curious if the task_container_ids continue in sequence from the training data to the test data. It looks like there are gaps between the training data and sample test data.\n\nBelow are the first ten sample test records.\n\n| \n | user_id | \nrow_id | \ntask_container_id | \nmax_train_tid | \ntid_delta | \nts_delta_sec | \n
|---|---|---|---|---|---|---|
| 1 | \n554169193 | \n1 | \n4427 | \n4421.0 | \n6.0 | \n184.627 | \n
| 2 | \n1720860329 | \n2 | \n240 | \n235.0 | \n5.0 | \n299.297 | \n
| 3 | \n288641214 | \n3 | \n266 | \n262.0 | \n4.0 | \n72252.433 | \n
| 4 | \n1728340777 | \n4 | \n162 | \n161.0 | \n1.0 | \n67009.619 | \n
| 5 | \n1364159702 | \n5 | \n4424 | \n4421.0 | \n3.0 | \n173642.470 | \n
| 6 | \n1521618396 | \n6 | \n1367 | \n1361.0 | \n6.0 | \n202.134 | \n
| 7 | \n1317245193 | \n7 | \n5314 | \n5307.0 | \n7.0 | \n384674.701 | \n
| 8 | \n1700555100 | \n8 | \n532 | \n530.0 | \n2.0 | \n246.422 | \n
| 9 | \n998511398 | \n9 | \n393 | \n389.0 | \n4.0 | \n37118.438 | \n
| 10 | \n1422853669 | \n10 | \n85 | \n79.0 | \n6.0 | \n194.973 | \n
Is the part in lectures same as part in questions?
", "rawMarkdown": "Is the part in lectures same as part in questions?\n" }, { "id": 1095031, "postDate": "2020-11-29T07:45:25.093Z", "content": "That makes it clearer
", "rawMarkdown": "That makes it clearer" }, { "id": 1085024, "postDate": "2020-11-20T15:59:35.760Z", "content": "@sohier I had a query that, In the column \"prior_question_had_explanation\" the student can explain the question at any time or just after the question is answered?
\nplease look at this query
\n thanks
Hi @sohier and community, I'm still confused about the tags/tag columns in questions.csv and lectures.csv. In this post you write, that these would \"match\" now, but there are 37 tags that occur in questions.csv that have no matching entries in lectures.csv. Am I missing something obvious here?
", "rawMarkdown": "Hi @sohier and community, I'm still confused about the tags/tag columns in **questions.csv** and **lectures.csv**. In this post you write, that these would \"match\" now, but there are 37 tags that occur in questions.csv that have no matching entries in lectures.csv. Am I missing something obvious here?", "replies": [ { "id": 1079199, "postDate": "2020-11-15T18:32:48.267Z", "content": "I think of the same but in the above author discussion it's written as follows,
\n\n\na)I'm updating the dataset now so lecture tags will match the tags in questions.csv.
\n
\n b)Correction- the hidden test set contains new users but not new questions
Coming to point 'B' there are no new questions which means there is a chance of introducing new lectures which then the tag of questions.csv and lecture.csv may match completely.
", "rawMarkdown": "I think of the same but in the above author discussion it's written as follows, \n\n> \na)I'm updating the dataset now so **lecture tags will match the tags in questions.csv**. \nb)Correction- the hidden test set contains **new users but not new questions**\n\nComing to point 'B' there are no new questions which means there is a chance of introducing new lectures which then the tag of questions.csv and lecture.csv may match completely. " } ] }, { "id": 1067278, "postDate": "2020-11-02T12:43:17.600Z", "content": "@sohier
\n1) why have we got multiple lecture tags associated to a single question id ,how do we interpret those
2) Can a test set have different lecture id than what is there in lecture,csv that we are provided with
", "rawMarkdown": "@sohier\n1) why have we got multiple lecture tags associated to a single question id ,how do we interpret those\n\n2) Can a test set have different lecture id than what is there in lecture,csv that we are provided with\n", "replies": [ { "id": 1079217, "postDate": "2020-11-15T18:52:41.153Z", "content": "for the first point,
\nlecture tag and question id are different things actually question_id is a content_id(which is in train.csv when cotent_type_id is zero ). and for the understanding of tag, it just says particular lecture or question belongs to that topic.
\n\nYou can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\"
\n
This looks great!
", "rawMarkdown": "> You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\"\n\nThis looks great!", "replies": [ { "id": 1068727, "postDate": "2020-11-03T17:21:07.413Z", "content": "Please tell me where the quote came from? (not met)
", "rawMarkdown": "Please tell me where the quote came from? (not met)" }, { "id": 1069053, "postDate": "2020-11-04T02:40:58.557Z", "content": "It comes from the pre-typed section on the kernel. Here's the formal announcement now though…
", "rawMarkdown": "It comes from the pre-typed section on the kernel. Here's the formal [announcement](https://www.kaggle.com/product-feedback/195163) now though...", "votes": 1 } ] }, { "id": 1063501, "postDate": "2020-10-29T02:10:52.560Z", "content": "In the 'Time-series API Details' section of Data Description, there is a statement,
\n\n\nExpect to see roughly 2.5 million questions in the hidden test set.
\n
I think the term 'questions' is misleading. Would you change that to 'rows' or 'interactions'?
", "rawMarkdown": "In the 'Time-series API Details' section of [Data Description](https://www.kaggle.com/c/riiid-test-answer-prediction/data), there is a statement, \n> Expect to see roughly 2.5 million questions in the hidden test set. \n\nI think the term 'questions' is misleading. Would you change that to 'rows' or 'interactions'?", "replies": [ { "id": 1065260, "postDate": "2020-10-31T05:03:15.990Z", "content": "Has it been confirmed somewhere that the test set has 2.5M rows (and not questions)?
\nI've been assuming there are 2.5M questions and (based on train data %) an additional few thousand lectures.
", "rawMarkdown": "Has it been confirmed somewhere that the test set has 2.5M rows (and not questions)?\n\nI've been assuming there are 2.5M questions and (based on train data %) an additional few thousand lectures.", "votes": 3 }, { "id": 1067968, "postDate": "2020-11-02T23:24:44.640Z", "content": "Thank you for your reply, Vopani.
\nI just thought that '2.5 million rows' would be more appropriate to explain the volume of test set.
\nI don't know that someone verified the volume or not.
Yes I agree using rows or observations is more appropriate.
", "rawMarkdown": "Yes I agree using rows or observations is more appropriate.", "votes": 1 } ] }, { "id": 1062003, "postDate": "2020-10-27T13:52:39.920Z", "content": "Thank you @sohier for the clarification.
", "rawMarkdown": "Thank you @sohier for the clarification. " }, { "id": 1059534, "postDate": "2020-10-25T08:02:56.573Z", "content": "The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.
\n-- Does it mean all samples in test data happen after train data? And GROUP 0 of test data happened before GROUP 1 of test data?
Yes that is my understanding and I'm assuming the API iterates over the groups chronologically, so group n happens before group n+1
Yes, the User_ID in train set and test set are same that means test set is continuing train set but one more point to notice that there are some new User_ID in test set.
\nAnd the group num proptionals to time series
Maybe I missed something but the defination of timestamp is vague. Can the host clarify what does \"first event completion from that user.\" exactly mean? What does event completion imply? What happens when the (mobile) user never completes that event?
", "rawMarkdown": "Maybe I missed something but the defination of timestamp is vague. Can the host clarify what does \"first event completion from that user.\" exactly mean? What does event completion imply? What happens when the (mobile) user never completes that event? " }, { "id": 1053755, "postDate": "2020-10-19T10:27:31.780Z", "content": "yiiiiihaaa
", "rawMarkdown": "yiiiiihaaa" }, { "id": 1054629, "postDate": "2020-10-20T04:09:04.433Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1053439, "postDate": "2020-10-19T02:25:56.423Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1051706, "postDate": "2020-10-16T18:46:22.690Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1051688, "postDate": "2020-10-16T18:21:16.407Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1050323, "postDate": "2020-10-15T09:22:00.957Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1052395, "postDate": "2020-10-17T17:08:15.867Z", "content": "Thank you! I really appreciate it.
", "rawMarkdown": "Thank you! I really appreciate it. " }, { "id": 1052090, "postDate": "2020-10-17T09:37:15.097Z", "content": "Thanks for the clarifications.
", "rawMarkdown": "Thanks for the clarifications." }, { "id": 1051983, "postDate": "2020-10-17T06:27:56.870Z", "content": "Thank for clearance
", "rawMarkdown": "Thank for clearance" }, { "id": 1050747, "postDate": "2020-10-15T17:23:01.023Z", "content": "thanks for clearance
", "rawMarkdown": "thanks for clearance" } ], "comments": [ { "id": 1049825, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-14T20:04:54.367000", "content": "@sohier Thanks for the clarification! Now we can start to build the sequential model. Would you mind to answer the following questions?
\nQ1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).
\nQ2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?
\nQ3. If the submission will be re-run after the deadline, would questions.csv, lectures.csv and the private test dataset remain the same as when we submit for the public LB score before the deadline?
\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190791
", "votes": 5, "replies": [] }, { "id": 1055372, "author_name": "Alex", "author_url": "", "post_date": "2020-10-20T17:50:47.367000", "content": "Roughly what percentage of users in the hidden test set will be new users?
", "votes": 4, "replies": [] }, { "id": 1065394, "author_name": "Trigram", "author_url": "", "post_date": "2020-10-31T08:36:39.717000", "content": "@sohier When I try to use TPUs to make a submission it errs with \"Your Notebook cannot use TPUs in this competition.\" But as per the code requirements we can use TPUs, so is this a glitch?
", "votes": 3, "replies": [ { "id": 1065401, "author_name": "Vopani", "author_url": "", "post_date": "2020-10-31T08:51:48.433000", "content": "The Code Requirements have been updated / changed since the start of the competition.
\nInitially internet access was not allowed either but that condition isn't mentioned anymore.
@sohier
\nWould be good to get clarification / confirmation on this since there has been no official post about the updates.
@rohanrao I just tried to submit a notebook with Internet access turned on, and I got
\n\n", "votes": 1, "replies": [] }, { "id": 1069885, "author_name": "Vopani", "author_url": "", "post_date": "2020-11-05T03:40:58.783000", "content": "Your Notebook cannot use internet access in this competition. Please disable internet in the Notebook editor and save a new version.
\n
@sohier needs to confirm regarding the mismatch between Code Requirements and actual backend since something changed.
", "votes": 4, "replies": [] } ] }, { "id": 1050278, "author_name": "Ethan", "author_url": "", "post_date": "2020-10-15T08:08:37.387000", "content": "Thanks for the clarifications. I have a request that could you please show more details of the \"Submission Scoring Error\" when we make submission. Each submission spends really long time and finnally \"Submission Scoring Error\" without any addition details. It's better to add some logs of the error, so we will not waste our time in debug and forcus more on the feature engineering and modeling.
", "votes": 3, "replies": [ { "id": 1050342, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-15T09:49:25.227000", "content": "They cannot add it else people will dump the whole data into logs, you can always raise an error at will.
", "votes": 5, "replies": [] } ] }, { "id": 1049944, "author_name": "Elman Mansimov", "author_url": "", "post_date": "2020-10-14T22:51:46.527000", "content": "Thanks for the clarifications. Will the hidden set not contain new lectures as well?
", "votes": 3, "replies": [ { "id": 1060721, "author_name": "sahilabs", "author_url": "", "post_date": "2020-10-26T14:00:22.507000", "content": "well, it's obvious that if there is no new question that implies there no new lectures.
", "votes": 0, "replies": [] }, { "id": 1060731, "author_name": "Vopani", "author_url": "", "post_date": "2020-10-26T14:09:16.593000", "content": "questions and lectures are different datasets so it's not completely obvious. There can be new lectures.
", "votes": 4, "replies": [] } ] }, { "id": 1060803, "author_name": "ManjotSinghDhillon", "author_url": "", "post_date": "2020-10-26T14:50:11.390000", "content": "Roughly what percentage of users in the hidden test set will be new users?
", "votes": 2, "replies": [ { "id": 1060827, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-10-26T14:57:50.530000", "content": "Just like in real-world scenarios, we look at historical (train) data, observe timestamps and frequency of users in order to estimate future (test) data distribution.
Since the data is time-series, you could also take 1st 80% data as train and last 20% data as test, and then perform this calculation viz., how many users are \"new\" i.e. present only in the last 20% of the train data.
Hey @sohier and community! I have one last question about the code competition set up. It has been a while since I have participated in code competition but it looks like we can train models offline these days and load them to the kernel environment.
\n\n\nPlease note that for this competition training is not required in Notebooks.
\n
My question is if we have to make it available to the community as well? Does it count as a pre-trained model?
\n\n\nFreely & publicly available external data is allowed, including pre-trained models
\n
I am sure this has been answered multiple times since the code competitions have been updated but I wasn't sure where to look.
", "votes": 1, "replies": [ { "id": 1134308, "author_name": "Douglas K.G. Araujo", "author_url": "", "post_date": "2021-01-01T06:03:08.023000", "content": "Hi @rdizzl3, did you find an answer for these questions? I am also interested. Thanks and good luck in this competition!
", "votes": 0, "replies": [] } ] }, { "id": 1066177, "author_name": "yash", "author_url": "", "post_date": "2020-11-01T12:22:47.757000", "content": "Thank you @sohier for the clarification.
", "votes": 1, "replies": [] }, { "id": 1061416, "author_name": "Mike L.", "author_url": "", "post_date": "2020-10-27T02:16:20.263000", "content": "content_type_id: Please include a lecture in the test data (Example.csv) so that we can verify that lectures are handled correctly before submitting to the hidden data.
\nThanks.
", "votes": 1, "replies": [ { "id": 1061521, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-27T04:01:20.410000", "content": "Mike, you can take a look at this!
", "votes": 0, "replies": [] }, { "id": 1061536, "author_name": "Mike L.", "author_url": "", "post_date": "2020-10-27T04:28:47.707000", "content": "Thanks. Now have a solution to my problem. Here is how to insert lectures into the Example dataframe test_df. This works for me 😊
\nImprovements welcome!
\nBorrowing from: https://www.geeksforgeeks.org/insert-row-at-given-position-in-pandas-dataframe/
\ndef Insert_row(row_number, df, row_value):
\n # Starting value of upper half
\n start_upper = 0
# End value of upper half \nend_upper = row_number \n\n# Start value of lower half \nstart_lower = row_number \n\n# End value of lower half \nend_lower = df.shape[0] \n\n# Create a list of upper_half index \nupper_half = [*range(start_upper, end_upper, 1)] \n\n# Create a list of lower_half index \nlower_half = [*range(start_lower, end_lower, 1)] \n\n# Increment the value of lower half by 1 \nlower_half = [x.__add__(1) for x in lower_half] \n\n# Combine the two lists \nindex_ = upper_half + lower_half \n\n# Update the index of the dataframe \ndf.index = index_ \n\n# Insert a row at the end \ndf.loc[row_number] = row_value \n\n# Sort the index labels \ndf = df.sort_index() \n\n# return the dataframe \nreturn df \niter_test = env.iter_test()
\nfor (test_df, sample_prediction_df) in iter_test:
# Let's create a row which we want to insert \n row_number = 2\n row_value = [89,653762,2746,6808,1,14,-1,-1,0,False] \n\n if row_number > test_df.index.max()+1: \n print(\"Invalid row_number\", row_number, test_df.index.max()) \n else: \n\n# Let's call the function and insert the row \n test_df = Insert_row(row_number, test_df, row_value) \n\n# Print the updated dataframe \n print(test_df) \n….
", "votes": 1, "replies": [] }, { "id": 1067188, "author_name": "Jaideep", "author_url": "", "post_date": "2020-11-02T11:48:52.100000", "content": "@mikel1
\n1) could u help understand the need for insert of new lectures ?
2) Can test set contain only question ids that are present in questions.csv ?
", "votes": 0, "replies": [] } ] }, { "id": 1054875, "author_name": "Mike L.", "author_url": "", "post_date": "2020-10-20T08:47:22.993000", "content": "Sohier, in your very instructive \"Competition API Detailed Introduction\", you stop a little too soon.
\nHere are the next steps:
\n1) Have a Notebook that successfully predicts the Example test data when \"Run All\" is clicked
\n Be sure that Setting \"Internet\" is off.
\n2) Save to share.
\n3) Share with the public
\n4) On the Notebooks page, click on it
\n5) Execution info
\n6) Click on \"Submit\"
\nYou will be told that your Notebook is running.
Thanks for the clarification! The new question_id in the hidden test set and the lack of sequence of test data have bothered me for a long time.
", "votes": 1, "replies": [] }, { "id": 1123192, "author_name": "Caleb", "author_url": "", "post_date": "2020-12-23T02:58:12.217000", "content": "@sohier apologies if this has been covered elsewhere - I read through the updates and searched the other discussions, but didn't see anything.
\nI ran this book to see whether there were many instances of gaps of more than one in the sequence of task_container_ids. I set it to error out if the count got over 100,000 and it errored out soon after it started. I understand there may be gaps due to the timing of user interactions on multiple devices and as a result task_container_ids are not sequential when records are ordered for each user by timestamp. I calculated that there are 2.5 million of such instances, in the training set, or approximately 2.5% of all of the observations. If the proportion was similar in the test set that would equate to approximately 62,500 instances, so erroring out after 100,000 would seem to indicate another source of them not being sequential.
", "votes": 2, "replies": [] }, { "id": 1061101, "author_name": "RDizzl3", "author_url": "", "post_date": "2020-10-26T18:35:28.533000", "content": "Hey @sohier thank you for the corrections and clarifications. I am about to enter the competition but I feel like I don't quite understand how the test API works for the new test users. I am looking at the example_test.csv and there is a user present in the test data and not in the training data.
Here is what that user looks like:
\n
My first question is making sure I understand the groups correctly -
\nThe lists from group 1 are the correct answers, etc. from questions in group 0?
If the list from group 1 is their data from group zero, shouldn't there be different content ids (possibly) for the different questions? There is only one content id in group 0.
Sorry if this has been explained somewhere else. I read quite a few topics in the forum but couldn't figure out how to correctly map and use this info.
", "votes": 2, "replies": [ { "id": 1061108, "author_name": "Vopani", "author_url": "", "post_date": "2020-10-26T18:45:51.650000", "content": "\n\nThe lists from group 1 are the correct answers, etc. from questions in group 0?
\n
Yes.
\n\n\nThere is only one content id in group 0.
\n
Don't filter on a user (it is irrelevant / incorrect). The length of the answer list in group n will exactly match the number of rows in group (n-1) (considering all users together).
Highly recommend to go through this.
", "votes": 3, "replies": [] }, { "id": 1061115, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-26T18:53:54.480000", "content": "Plus there's video's labels in that list as well! So be careful with that as we don't have to make preds on video_rows so to say (content_type_id == 1).
", "votes": 2, "replies": [] }, { "id": 1061122, "author_name": "RDizzl3", "author_url": "", "post_date": "2020-10-26T19:01:19.230000", "content": "thank you @vopani! I did not realize that topic had all this info! But I was able to verify everything I was questioning. Appreciate the comment!
", "votes": 1, "replies": [] } ] }, { "id": 1050025, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-15T02:14:32.560000", "content": "Thanks for the clarifications. Will it be possible to simply return the data from questions and lectures pre-appended to the test-df as well? Because it will be joined by us either ways, so IMO it's a good thing to return it when we are iterating over the test_df as it's columns?
\nAnd the only file updated is lectures.csv, right?
", "votes": 2, "replies": [] }, { "id": 1050004, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2020-10-15T01:17:38.407000", "content": "Thanks for the clarification about private set. Now it help and have better clarification about private set.
\n@sohier Do you mean.. train/test set is complete having all user activities logged except few missing values are negligible ratio ?
", "votes": 2, "replies": [] }, { "id": 1134231, "author_name": "S.Kodai", "author_url": "", "post_date": "2021-01-01T03:14:54.877000", "content": "@sohier Thanks for the clarification about private set. Now it help and have better clarification about private set.
\n\n\nThe test data follows chronologically after the train data. The test iterations give interactions of users chronologically.
\n
Regarding this matter, does chronologically mean that in order of timestamp, rather than task_container_id?
", "votes": 0, "replies": [] }, { "id": 1121782, "author_name": "Caleb", "author_url": "", "post_date": "2020-12-21T22:44:02.397000", "content": "I was curious if the task_container_ids continue in sequence from the training data to the test data. It looks like there are gaps between the training data and sample test data.
\nBelow are the first ten sample test records.
\n| \n | user_id | \nrow_id | \ntask_container_id | \nmax_train_tid | \ntid_delta | \nts_delta_sec | \n
|---|---|---|---|---|---|---|
| 1 | \n554169193 | \n1 | \n4427 | \n4421.0 | \n6.0 | \n184.627 | \n
| 2 | \n1720860329 | \n2 | \n240 | \n235.0 | \n5.0 | \n299.297 | \n
| 3 | \n288641214 | \n3 | \n266 | \n262.0 | \n4.0 | \n72252.433 | \n
| 4 | \n1728340777 | \n4 | \n162 | \n161.0 | \n1.0 | \n67009.619 | \n
| 5 | \n1364159702 | \n5 | \n4424 | \n4421.0 | \n3.0 | \n173642.470 | \n
| 6 | \n1521618396 | \n6 | \n1367 | \n1361.0 | \n6.0 | \n202.134 | \n
| 7 | \n1317245193 | \n7 | \n5314 | \n5307.0 | \n7.0 | \n384674.701 | \n
| 8 | \n1700555100 | \n8 | \n532 | \n530.0 | \n2.0 | \n246.422 | \n
| 9 | \n998511398 | \n9 | \n393 | \n389.0 | \n4.0 | \n37118.438 | \n
| 10 | \n1422853669 | \n10 | \n85 | \n79.0 | \n6.0 | \n194.973 | \n
Are those gaps related to the preparation of the sample test data or are there similar gaps in the actual test data as well?
\nEDIT: updated to include timetamp delta as well, which look like they might span gaps in task_container_ids. I had a median gap of approx. 42 seconds on the last set of user training records.
", "votes": 0, "replies": [] }, { "id": 1117061, "author_name": "skw1990", "author_url": "", "post_date": "2020-12-17T17:30:35.587000", "content": "Is the part in lectures same as part in questions?
", "votes": 0, "replies": [] }, { "id": 1095031, "author_name": "Yueqi Wang 0001", "author_url": "", "post_date": "2020-11-29T07:45:25.093000", "content": "That makes it clearer
", "votes": 0, "replies": [] }, { "id": 1085024, "author_name": "sahilabs", "author_url": "", "post_date": "2020-11-20T15:59:35.760000", "content": "@sohier I had a query that, In the column \"prior_question_had_explanation\" the student can explain the question at any time or just after the question is answered?
\nplease look at this query
\n thanks
Hi @sohier and community, I'm still confused about the tags/tag columns in questions.csv and lectures.csv. In this post you write, that these would \"match\" now, but there are 37 tags that occur in questions.csv that have no matching entries in lectures.csv. Am I missing something obvious here?
", "votes": 0, "replies": [ { "id": 1079199, "author_name": "sahilabs", "author_url": "", "post_date": "2020-11-15T18:32:48.267000", "content": "I think of the same but in the above author discussion it's written as follows,
\n\n\na)I'm updating the dataset now so lecture tags will match the tags in questions.csv.
\n
\n b)Correction- the hidden test set contains new users but not new questions
Coming to point 'B' there are no new questions which means there is a chance of introducing new lectures which then the tag of questions.csv and lecture.csv may match completely.
", "votes": 0, "replies": [] } ] }, { "id": 1067278, "author_name": "Jaideep", "author_url": "", "post_date": "2020-11-02T12:43:17.600000", "content": "@sohier
\n1) why have we got multiple lecture tags associated to a single question id ,how do we interpret those
2) Can a test set have different lecture id than what is there in lecture,csv that we are provided with
", "votes": 0, "replies": [ { "id": 1079217, "author_name": "sahilabs", "author_url": "", "post_date": "2020-11-15T18:52:41.153000", "content": "for the first point,
\nlecture tag and question id are different things actually question_id is a content_id(which is in train.csv when cotent_type_id is zero ). and for the understanding of tag, it just says particular lecture or question belongs to that topic.
\n\nYou can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\"
\n
This looks great!
", "votes": 0, "replies": [ { "id": 1068727, "author_name": "Pavel Orlov", "author_url": "", "post_date": "2020-11-03T17:21:07.413000", "content": "Please tell me where the quote came from? (not met)
", "votes": 0, "replies": [] }, { "id": 1069053, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-04T02:40:58.557000", "content": "It comes from the pre-typed section on the kernel. Here's the formal announcement now though…
", "votes": 1, "replies": [] } ] }, { "id": 1063501, "author_name": "kuroyuli", "author_url": "", "post_date": "2020-10-29T02:10:52.560000", "content": "In the 'Time-series API Details' section of Data Description, there is a statement,
\n\n\nExpect to see roughly 2.5 million questions in the hidden test set.
\n
I think the term 'questions' is misleading. Would you change that to 'rows' or 'interactions'?
", "votes": 0, "replies": [ { "id": 1065260, "author_name": "Vopani", "author_url": "", "post_date": "2020-10-31T05:03:15.990000", "content": "Has it been confirmed somewhere that the test set has 2.5M rows (and not questions)?
\nI've been assuming there are 2.5M questions and (based on train data %) an additional few thousand lectures.
", "votes": 3, "replies": [] }, { "id": 1067968, "author_name": "kuroyuli", "author_url": "", "post_date": "2020-11-02T23:24:44.640000", "content": "Thank you for your reply, Vopani.
\nI just thought that '2.5 million rows' would be more appropriate to explain the volume of test set.
\nI don't know that someone verified the volume or not.
Yes I agree using rows or observations is more appropriate.
", "votes": 1, "replies": [] } ] }, { "id": 1062003, "author_name": "Md. Abdullah Al Mamun", "author_url": "", "post_date": "2020-10-27T13:52:39.920000", "content": "Thank you @sohier for the clarification.
", "votes": 0, "replies": [] }, { "id": 1059534, "author_name": "Travis", "author_url": "", "post_date": "2020-10-25T08:02:56.573000", "content": "The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.
\n-- Does it mean all samples in test data happen after train data? And GROUP 0 of test data happened before GROUP 1 of test data?
Yes that is my understanding and I'm assuming the API iterates over the groups chronologically, so group n happens before group n+1
Yes, the User_ID in train set and test set are same that means test set is continuing train set but one more point to notice that there are some new User_ID in test set.
\nAnd the group num proptionals to time series
Maybe I missed something but the defination of timestamp is vague. Can the host clarify what does \"first event completion from that user.\" exactly mean? What does event completion imply? What happens when the (mobile) user never completes that event?
", "votes": 0, "replies": [] }, { "id": 1053755, "author_name": "EnricRovira", "author_url": "", "post_date": "2020-10-19T10:27:31.780000", "content": "yiiiiihaaa
", "votes": 0, "replies": [] }, { "id": 1054629, "author_name": "", "author_url": "", "post_date": "2020-10-20T04:09:04.433000", "content": "", "votes": 0, "replies": [] }, { "id": 1053439, "author_name": "", "author_url": "", "post_date": "2020-10-19T02:25:56.423000", "content": "", "votes": 0, "replies": [] }, { "id": 1051706, "author_name": "", "author_url": "", "post_date": "2020-10-16T18:46:22.690000", "content": "", "votes": 0, "replies": [] }, { "id": 1051688, "author_name": "", "author_url": "", "post_date": "2020-10-16T18:21:16.407000", "content": "", "votes": 0, "replies": [] }, { "id": 1050323, "author_name": "", "author_url": "", "post_date": "2020-10-15T09:22:00.957000", "content": "", "votes": 0, "replies": [] }, { "id": 1052395, "author_name": "", "author_url": "", "post_date": "2020-10-17T17:08:15.867000", "content": "", "votes": 0, "replies": [] }, { "id": 1052090, "author_name": "", "author_url": "", "post_date": "2020-10-17T09:37:15.097000", "content": "", "votes": 0, "replies": [] }, { "id": 1051983, "author_name": "", "author_url": "", "post_date": "2020-10-17T06:27:56.870000", "content": "", "votes": 0, "replies": [] }, { "id": 1050747, "author_name": "", "author_url": "", "post_date": "2020-10-15T17:23:01.023000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1049702": "- I'm updating the dataset now so lecture tags will match the tags in **questions.csv**. The updated **lectures.csv** should be live within the next hour. Apologies for the inconvenience.\n\n- Correction- the hidden test set contains new _users_ but not new _questions_.\n\n- The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data. It remains possible that some questions weren't logged due to other issues that all datasets of mobile users are susceptible to,\nsuch as if a user lost their connection mid-question.\n\n- The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.", "1049825": "@sohier Thanks for the clarification! Now we can start to build the sequential model. Would you mind to answer the following questions?\n\n\nQ1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\nQ2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?\n\nQ3. If the submission will be re-run after the deadline, would questions.csv, lectures.csv and the private test dataset remain the same as when we submit for the public LB score before the deadline?\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190791", "1055372": "Roughly what percentage of users in the hidden test set will be new users?", "1065394": "@sohier When I try to use TPUs to make a submission it errs with \"Your Notebook cannot use TPUs in this competition.\" But as per the code requirements we can use TPUs, so is this a glitch?", "1050278": "Thanks for the clarifications. I have a request that could you please show more details of the \"Submission Scoring Error\" when we make submission. Each submission spends really long time and finnally \"Submission Scoring Error\" without any addition details. It's better to add some logs of the error, so we will not waste our time in debug and forcus more on the feature engineering and modeling.", "1049944": "Thanks for the clarifications. Will the hidden set not contain new lectures as well?\n ", "1060803": "Roughly what percentage of users in the hidden test set will be new users?", "1068603": "Hey @sohier and community! I have one last question about the code competition set up. It has been a while since I have participated in code competition but it looks like we can train models offline these days and load them to the kernel environment.\n\n> Please note that for this competition training is not required in Notebooks.\n\nMy question is if we have to make it available to the community as well? Does it count as a pre-trained model?\n\n> Freely & publicly available external data is allowed, including pre-trained models\n\nI am sure this has been answered multiple times since the code competitions have been updated but I wasn't sure where to look.", "1066177": "Thank you @sohier for the clarification.", "1061416": "content_type_id: Please include a lecture in the test data (Example.csv) so that we can verify that lectures are handled correctly before submitting to the hidden data.\n\nThanks.", "1054875": "Sohier, in your very instructive \"Competition API Detailed Introduction\", you stop a little too soon.\n\nHere are the next steps:\n\n1) Have a Notebook that successfully predicts the Example test data when \"Run All\" is clicked\n Be sure that Setting \"Internet\" is off.\n2) Save to share.\n3) Share with the public\n4) On the Notebooks page, click on it \n5) Execution info\n6) Click on \"Submit\"\nYou will be told that your Notebook is running.", "1049921": "Thanks for the clarification! The new **question_id** in the hidden test set and the lack of sequence of test data have bothered me for a long time.", "1123192": "@sohier apologies if this has been covered elsewhere - I read through the updates and searched the other discussions, but didn't see anything.\n\nI ran [this book](https://www.kaggle.com/calebeverett/test-tid-deltas) to see whether there were many instances of gaps of more than one in the sequence of task_container_ids. I set it to error out if the count got over 100,000 and it errored out soon after it started. I understand there may be gaps due to the timing of user interactions on multiple devices and as a result task_container_ids are not sequential when records are ordered for each user by timestamp. I calculated that there are 2.5 million of such instances, in the training set, or approximately 2.5% of all of the observations. If the proportion was similar in the test set that would equate to approximately 62,500 instances, so erroring out after 100,000 would seem to indicate another source of them not being sequential.", "1061101": "Hey @sohier thank you for the corrections and clarifications. I am about to enter the competition but I feel like I don't quite understand how the test API works for the new test users. I am looking at the `example_test.csv` and there is a user present in the test data and not in the training data.\n\nHere is what that user looks like:\n\n\nMy first question is making sure I understand the groups correctly - \n1. The lists from group 1 are the correct answers, etc. from questions in group 0?\n\n2. If the list from group 1 is their data from group zero, shouldn't there be different content ids (possibly) for the different questions? There is only one content id in group 0.\n\nSorry if this has been explained somewhere else. I read quite a few topics in the forum but couldn't figure out how to correctly map and use this info.", "1050025": "Thanks for the clarifications. Will it be possible to simply return the data from questions and lectures pre-appended to the test-df as well? Because it will be joined by us either ways, so IMO it's a good thing to return it when we are iterating over the test_df as it's columns?\n\nAnd the only file updated is lectures.csv, right?", "1050004": "Thanks for the clarification about private set. Now it help and have better clarification about private set.\n\n@sohier Do you mean.. **train/test set is complete** having all user activities logged except **few missing** values are **negligible ratio** ?", "1134231": "@sohier Thanks for the clarification about private set. Now it help and have better clarification about private set.\n\n> The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.\n\nRegarding this matter, does chronologically mean that in order of **timestamp**, rather than task_container_id?", "1121782": "I was curious if the task_container_ids continue in sequence from the training data to the test data. It looks like there are gaps between the training data and sample test data.\n\nBelow are the first ten sample test records.\n\n| \n | user_id | \nrow_id | \ntask_container_id | \nmax_train_tid | \ntid_delta | \nts_delta_sec | \n
|---|---|---|---|---|---|---|
| 1 | \n554169193 | \n1 | \n4427 | \n4421.0 | \n6.0 | \n184.627 | \n
| 2 | \n1720860329 | \n2 | \n240 | \n235.0 | \n5.0 | \n299.297 | \n
| 3 | \n288641214 | \n3 | \n266 | \n262.0 | \n4.0 | \n72252.433 | \n
| 4 | \n1728340777 | \n4 | \n162 | \n161.0 | \n1.0 | \n67009.619 | \n
| 5 | \n1364159702 | \n5 | \n4424 | \n4421.0 | \n3.0 | \n173642.470 | \n
| 6 | \n1521618396 | \n6 | \n1367 | \n1361.0 | \n6.0 | \n202.134 | \n
| 7 | \n1317245193 | \n7 | \n5314 | \n5307.0 | \n7.0 | \n384674.701 | \n
| 8 | \n1700555100 | \n8 | \n532 | \n530.0 | \n2.0 | \n246.422 | \n
| 9 | \n998511398 | \n9 | \n393 | \n389.0 | \n4.0 | \n37118.438 | \n
| 10 | \n1422853669 | \n10 | \n85 | \n79.0 | \n6.0 | \n194.973 | \n