{
  "id": 189351,
  "title": "Questions about timestamp and prior_question_elapsed_time",
  "url": "/competitions/riiid-test-answer-prediction/discussion/189351",
  "author_name": "Gunes Evitan",
  "post_date": "2020-10-07T10:31:36.115000",
  "votes": 40,
  "comment_count": 22,
  "views": 0,
  "content": "<p>On data tab, <code>timestamp</code> is defined as \"the time between this user interaction and the first event from that user\", and <code>prior_question_elapsed_time</code> is defined as \"How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle\".</p>\n<p>I had a hard time understanding the difference between those two columns. If I understood correctly, every row can be a question or another content, but a user can also have other interactions between them that are not listed in this dataset. Am I correct? If that assumption is correct, then why <code>prior_question_elapsed_time</code> is greater than <code>timestamp</code> in some rows?</p>\n<p><img src=\"https://i.ibb.co/S7wYLS0/aaa.jpg\" alt=\"img\"></p>",
  "messages": [
    {
      "id": 1040755,
      "postDate": "2020-10-07T10:31:36.117Z",
      "content": "<p>On data tab, <code>timestamp</code> is defined as \"the time between this user interaction and the first event from that user\", and <code>prior_question_elapsed_time</code> is defined as \"How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle\".</p>\n<p>I had a hard time understanding the difference between those two columns. If I understood correctly, every row can be a question or another content, but a user can also have other interactions between them that are not listed in this dataset. Am I correct? If that assumption is correct, then why <code>prior_question_elapsed_time</code> is greater than <code>timestamp</code> in some rows?</p>\n<p><img src=\"https://i.ibb.co/S7wYLS0/aaa.jpg\" alt=\"img\"></p>",
      "rawMarkdown": "On data tab, `timestamp` is defined as \"the time between this user interaction and the first event from that user\", and `prior_question_elapsed_time` is defined as \"How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle\".\n\nI had a hard time understanding the difference between those two columns. If I understood correctly, every row can be a question or another content, but a user can also have other interactions between them that are not listed in this dataset. Am I correct? If that assumption is correct, then why `prior_question_elapsed_time` is greater than `timestamp` in some rows?\n\n![img](https://i.ibb.co/S7wYLS0/aaa.jpg)\n",
      "votes": 40
    },
    {
      "id": 1041405,
      "postDate": "2020-10-07T18:10:11.583Z",
      "content": "<ul>\n<li>The <code>timestamp</code> column shows when an activity is finished, not when it started. A user could have spent an arbitrary amount of time working on the first problem before the first <code>timestamp</code> value would get logged.</li>\n<li>You seem to be skipping over the \"prior\" part of the column name, which is critical.</li>\n</ul>",
      "rawMarkdown": "- The `timestamp` column shows when an activity is finished, not when it started. A user could have spent an arbitrary amount of time working on the first problem before the first `timestamp` value would get logged.\n- You seem to be skipping over the \"prior\" part of the column name, which is critical.\n",
      "votes": 10,
      "replies": [
        {
          "id": 1049564,
          "postDate": "2020-10-14T14:31:12.387Z",
          "content": "<p>Sir, I think timestamp is the time when current activity started, proved by the laws of the prior_time with current activity with prior activity.</p>",
          "rawMarkdown": "Sir, I think timestamp is the time when current activity started, proved by the laws of the prior_time with current activity with prior activity.",
          "votes": 2
        },
        {
          "id": 1062698,
          "postDate": "2020-10-28T05:35:52.807Z",
          "content": "<p><a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a> Regarding the discussion <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191695\" target=\"_blank\">here</a>, it seems that for a single user in train.csv, same content_id may appear in multiple entries with different timestamp and different task_container_id. I am not quite sure about these cases. Is it that the same question had been completed by the user multiple times or it had been viewed multiple times? </p>",
          "rawMarkdown": "@sohier Regarding the discussion [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191695), it seems that for a single user in train.csv, same content_id may appear in multiple entries with different timestamp and different task_container_id. I am not quite sure about these cases. Is it that the same question had been completed by the user multiple times or it had been viewed multiple times? ",
          "votes": 1
        },
        {
          "id": 1067325,
          "postDate": "2020-11-02T13:27:36.647Z",
          "content": "<p><a href=\"https://www.kaggle.com/cealia\" target=\"_blank\">@cealia</a> 's question should be answered!</p>",
          "rawMarkdown": "@cealia 's question should be answered!"
        }
      ]
    },
    {
      "id": 1041237,
      "postDate": "2020-10-07T16:24:40.793Z",
      "content": "<p><img src=\"https://i.ibb.co/Tk8KrDQ/ad.jpg\" alt=\"img\"></p>\n<p>From my understanding, this user spent 28000 on first question and 21592 passed since the users last interaction. Only logical explanation is <code>timestamp</code> timer starts after first question is answered or lecture is finished. <code>prior_question_elapsed_time</code> timer starts when the user starts doing the previous question and it ends when the user moves to another question.</p>",
      "rawMarkdown": "![img](https://i.ibb.co/Tk8KrDQ/ad.jpg)\n\nFrom my understanding, this user spent 28000 on first question and 21592 passed since the users last interaction. Only logical explanation is `timestamp` timer starts after first question is answered or lecture is finished. `prior_question_elapsed_time` timer starts when the user starts doing the previous question and it ends when the user moves to another question.",
      "votes": 3,
      "replies": [
        {
          "id": 1041361,
          "postDate": "2020-10-07T17:43:04.773Z",
          "content": "<p>Or maybe the units are different after all? That would explain why one is a float and the other an int. Maybe <code>timestamp</code> is seconds, and <code>prior_question_elapsed_time</code> is in milliseconds (but rounded to the next second)? </p>\n<p>It would seem reasonable to spend 20-something seconds on a question and have thousands of seconds (i.e., hours/days) since the first interaction.</p>",
          "rawMarkdown": "Or maybe the units are different after all? That would explain why one is a float and the other an int. Maybe <code>timestamp</code> is seconds, and <code>prior_question_elapsed_time</code> is in milliseconds (but rounded to the next second)? \n\nIt would seem reasonable to spend 20-something seconds on a question and have thousands of seconds (i.e., hours/days) since the first interaction.",
          "votes": 1
        },
        {
          "id": 1041568,
          "postDate": "2020-10-07T20:16:14.303Z",
          "content": "<p><code>timestamp</code> cannot be seconds. The max value of the first 100000 rows is 59495311427.  It only makes sense in milliseconds for such a value. </p>",
          "rawMarkdown": "`timestamp` cannot be seconds. The max value of the first 100000 rows is 59495311427.  It only makes sense in milliseconds for such a value. "
        },
        {
          "id": 1041575,
          "postDate": "2020-10-07T20:26:13.703Z",
          "content": "<blockquote>\n  <p>Only logical explanation is timestamp timer starts after first question is answered or lecture is finished. </p>\n</blockquote>\n<p>I think your explanation is reasonable. If that's true, for each user, <code>timestamp</code>column should add the first valid <code>prior_question_elapsed_time</code> value.  Let's check that. </p>",
          "rawMarkdown": "> Only logical explanation is timestamp timer starts after first question is answered or lecture is finished. \n\nI think your explanation is reasonable. If that's true, for each user, `timestamp `column should add the first valid `prior_question_elapsed_time ` value.  Let's check that. ",
          "votes": 1
        },
        {
          "id": 1041594,
          "postDate": "2020-10-07T20:37:45.887Z",
          "content": "<p><a href=\"https://www.kaggle.com/ningjiaca\" target=\"_blank\">@ningjiaca</a> You can't get any information by adding <code>timestamp</code> and <code>prior_question_elapsed_time</code>. You could subtract <code>prior_question_elapsed_time</code> from the previous question timestamp for that user to get the timestamp for when a user started working on a question, though I don't know why you would want that. </p>\n<p>In rough terms you might want to look at the <code>timestamp</code> column to check whether a user worked on 10 questions a day or 10 questions per month and the <code>prior_question_elapsed_time</code> column to see if they need 1 second to respond or three minutes.</p>",
          "rawMarkdown": "@ningjiaca You can't get any information by adding `timestamp` and `prior_question_elapsed_time`. You could subtract `prior_question_elapsed_time` from the previous question timestamp for that user to get the timestamp for when a user started working on a question, though I don't know why you would want that. \n\nIn rough terms you might want to look at the `timestamp` column to check whether a user worked on 10 questions a day or 10 questions per month and the `prior_question_elapsed_time` column to see if they need 1 second to respond or three minutes.",
          "votes": 12
        },
        {
          "id": 1041968,
          "postDate": "2020-10-08T02:01:15.200Z",
          "content": "<p>Thanks for your reply.</p>\n<p>It helps my feature engineering. </p>",
          "rawMarkdown": "Thanks for your reply.\n\nIt helps my feature engineering. "
        },
        {
          "id": 1046042,
          "postDate": "2020-10-11T09:48:42.360Z",
          "content": "<blockquote>\n  <p>In rough terms you might want to look at the timestamp column to check whether a user worked on 10 questions a day or 10 questions per month and the prior_question_elapsed_time column to see if they need 1 second to respond or three minutes.</p>\n</blockquote>\n<p>Any pointers how can we even calculate this?</p>\n<p>My thought is, assume a time stamp \"k\" and then scan till \"k+86400000msec\", that might give us a single day activity of the user.</p>",
          "rawMarkdown": ">In rough terms you might want to look at the timestamp column to check whether a user worked on 10 questions a day or 10 questions per month and the prior_question_elapsed_time column to see if they need 1 second to respond or three minutes.\n\nAny pointers how can we even calculate this?\n\nMy thought is, assume a time stamp \"k\" and then scan till \"k+86400000msec\", that might give us a single day activity of the user."
        }
      ]
    },
    {
      "id": 1040881,
      "postDate": "2020-10-07T12:21:20.987Z",
      "content": "<p>It seems to me that in the example you have shown, <code>prior_question_elapsed_time</code> = 37000 and <code>timestamp</code> = 59943 and thus <code>prior_question_elapsed_time</code> is smaller than <code>timestamp</code> and not greater.</p>\n<p>However, similar to your question, I have a hard time understanding the relationship between <code>timestamp</code> and <code>task_container_id</code>, the latter being defined as</p>\n<blockquote>\n  <p>Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a <code>task_container_id</code>. Monotonically increasing for each user.</p>\n</blockquote>\n<p>In my understanding, the interaction at <code>timestamp</code> = 0 for any user should be the first interaction he/she has ever had, be it a question or a lecture (or any other type of interaction that exists). However, at <code>row_id</code> = 0, the <code>timestamp</code> is equal to 0 but <code>task_container_id</code> is equal to 1, not 0, which implies that the user has already had another question (<code>content_type</code> = 0) previously… It seems contradictory to me but maybe my understanding of these fields is incorrect.</p>\n<p>I have also checked other cases like this and there seems to be quite a few others.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5589062%2Fb87552ba9c768c016e22eb4e394fe30a%2Fkaggle.png?generation=1602073116276058&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "It seems to me that in the example you have shown, `prior_question_elapsed_time` = 37000 and `timestamp` = 59943 and thus `prior_question_elapsed_time` is smaller than `timestamp` and not greater.\n\nHowever, similar to your question, I have a hard time understanding the relationship between `timestamp` and `task_container_id`, the latter being defined as\n\n> Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a `task_container_id`. Monotonically increasing for each user.\n\nIn my understanding, the interaction at `timestamp` = 0 for any user should be the first interaction he/she has ever had, be it a question or a lecture (or any other type of interaction that exists). However, at `row_id` = 0, the `timestamp` is equal to 0 but `task_container_id` is equal to 1, not 0, which implies that the user has already had another question (`content_type` = 0) previously... It seems contradictory to me but maybe my understanding of these fields is incorrect.\n\nI have also checked other cases like this and there seems to be quite a few others.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5589062%2Fb87552ba9c768c016e22eb4e394fe30a%2Fkaggle.png?generation=1602073116276058&alt=media)",
      "votes": 3,
      "replies": [
        {
          "id": 1040993,
          "postDate": "2020-10-07T13:38:08.080Z",
          "content": "<p>This is an important question. The <code>task_container_id</code> variable is supposed to be monotonically increasing for each user. But that doesn't seem to be the case. For instance, see user 115.</p>\n<p>To fix this, I suggest renumbering the tasks to make sure they're monotonically increasing for each user:</p>\n<pre><code>train['task_container_id'] = (\n    train\n    .groupby('user_id')['task_container_id']\n    .transform(lambda x: pd.factorize(x)[0])\n    .astype('int16')\n)\n</code></pre>",
          "rawMarkdown": "This is an important question. The `task_container_id` variable is supposed to be monotonically increasing for each user. But that doesn't seem to be the case. For instance, see user 115.\n\nTo fix this, I suggest renumbering the tasks to make sure they're monotonically increasing for each user:\n\n```python\ntrain['task_container_id'] = (\n    train\n    .groupby('user_id')['task_container_id']\n    .transform(lambda x: pd.factorize(x)[0])\n    .astype('int16')\n)\n```",
          "votes": 11
        },
        {
          "id": 1041231,
          "postDate": "2020-10-07T16:21:07.107Z",
          "content": "<p>Thank you for your suggestion!</p>\n<p>Now I am just wondering if this is an actual mistake in the data and how much it will affect the training. It might also be better to report it to the hosts of the competition and let other people know about it so they can verify the impact of applying this fix or not. I will create a new discussion around this topic.</p>",
          "rawMarkdown": "Thank you for your suggestion!\n\nNow I am just wondering if this is an actual mistake in the data and how much it will affect the training. It might also be better to report it to the hosts of the competition and let other people know about it so they can verify the impact of applying this fix or not. I will create a new discussion around this topic.",
          "votes": 3
        },
        {
          "id": 1126352,
          "postDate": "2020-12-25T14:38:32.910Z",
          "content": "<p>-- still thinking about this</p>",
          "rawMarkdown": "-- still thinking about this"
        }
      ]
    },
    {
      "id": 1041437,
      "postDate": "2020-10-07T18:33:04.573Z",
      "content": "<p>I think they mean the time elapsed on the <strong>prior</strong> question.</p>\n<p>Its not cummulative it only depends on the interaction between the user and the prior question.</p>\n<p>Hope it helps</p>",
      "rawMarkdown": "I think they mean the time elapsed on the **prior** question.\n\nIts not cummulative it only depends on the interaction between the user and the prior question.\n\nHope it helps",
      "votes": 1,
      "replies": [
        {
          "id": 1041452,
          "postDate": "2020-10-07T18:43:24.403Z",
          "content": "<p>[Correction] on the prior <strong>bundle</strong></p>",
          "rawMarkdown": "[Correction] on the prior **bundle**"
        }
      ]
    },
    {
      "id": 1064682,
      "postDate": "2020-10-30T12:07:43.753Z",
      "content": "<p>Probably something escape me but i don't understand why you put the prior_question_elapsed_time and not the current one ?</p>",
      "rawMarkdown": "Probably something escape me but i don't understand why you put the prior_question_elapsed_time and not the current one ?",
      "replies": [
        {
          "id": 1095257,
          "postDate": "2020-11-29T12:08:26.403Z",
          "content": "<p>It is to reflect what information we have at inference time when we need to predict result for a question the user has not yet seen. In that case we don't know the elapsed time yet.</p>",
          "rawMarkdown": "It is to reflect what information we have at inference time when we need to predict result for a question the user has not yet seen. In that case we don't know the elapsed time yet."
        },
        {
          "id": 1095369,
          "postDate": "2020-11-29T14:31:45.413Z",
          "content": "<p>Yes , thanks ISMAX, i understood it soon after my post.</p>",
          "rawMarkdown": "Yes , thanks ISMAX, i understood it soon after my post."
        }
      ]
    },
    {
      "id": 1044576,
      "postDate": "2020-10-09T23:26:26.280Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 1040790,
      "postDate": "2020-10-07T10:52:02.817Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1041405,
      "author_name": "Sohier Dane",
      "author_url": "",
      "post_date": "2020-10-07T18:10:11.583000",
      "content": "<ul>\n<li>The <code>timestamp</code> column shows when an activity is finished, not when it started. A user could have spent an arbitrary amount of time working on the first problem before the first <code>timestamp</code> value would get logged.</li>\n<li>You seem to be skipping over the \"prior\" part of the column name, which is critical.</li>\n</ul>",
      "votes": 10,
      "replies": [
        {
          "id": 1049564,
          "author_name": "Zhenghan Chen",
          "author_url": "",
          "post_date": "2020-10-14T14:31:12.387000",
          "content": "<p>Sir, I think timestamp is the time when current activity started, proved by the laws of the prior_time with current activity with prior activity.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1062698,
          "author_name": "panda",
          "author_url": "",
          "post_date": "2020-10-28T05:35:52.807000",
          "content": "<p><a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a> Regarding the discussion <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191695\" target=\"_blank\">here</a>, it seems that for a single user in train.csv, same content_id may appear in multiple entries with different timestamp and different task_container_id. I am not quite sure about these cases. Is it that the same question had been completed by the user multiple times or it had been viewed multiple times? </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1067325,
          "author_name": "jwc",
          "author_url": "",
          "post_date": "2020-11-02T13:27:36.647000",
          "content": "<p><a href=\"https://www.kaggle.com/cealia\" target=\"_blank\">@cealia</a> 's question should be answered!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1041237,
      "author_name": "Gunes Evitan",
      "author_url": "",
      "post_date": "2020-10-07T16:24:40.793000",
      "content": "<p><img src=\"https://i.ibb.co/Tk8KrDQ/ad.jpg\" alt=\"img\"></p>\n<p>From my understanding, this user spent 28000 on first question and 21592 passed since the users last interaction. Only logical explanation is <code>timestamp</code> timer starts after first question is answered or lecture is finished. <code>prior_question_elapsed_time</code> timer starts when the user starts doing the previous question and it ends when the user moves to another question.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1041361,
          "author_name": "Alex Bader",
          "author_url": "",
          "post_date": "2020-10-07T17:43:04.773000",
          "content": "<p>Or maybe the units are different after all? That would explain why one is a float and the other an int. Maybe <code>timestamp</code> is seconds, and <code>prior_question_elapsed_time</code> is in milliseconds (but rounded to the next second)? </p>\n<p>It would seem reasonable to spend 20-something seconds on a question and have thousands of seconds (i.e., hours/days) since the first interaction.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1041568,
          "author_name": "Ning Jia",
          "author_url": "",
          "post_date": "2020-10-07T20:16:14.303000",
          "content": "<p><code>timestamp</code> cannot be seconds. The max value of the first 100000 rows is 59495311427.  It only makes sense in milliseconds for such a value. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1041575,
          "author_name": "Ning Jia",
          "author_url": "",
          "post_date": "2020-10-07T20:26:13.703000",
          "content": "<blockquote>\n  <p>Only logical explanation is timestamp timer starts after first question is answered or lecture is finished. </p>\n</blockquote>\n<p>I think your explanation is reasonable. If that's true, for each user, <code>timestamp</code>column should add the first valid <code>prior_question_elapsed_time</code> value.  Let's check that. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1041594,
          "author_name": "Sohier Dane",
          "author_url": "",
          "post_date": "2020-10-07T20:37:45.887000",
          "content": "<p><a href=\"https://www.kaggle.com/ningjiaca\" target=\"_blank\">@ningjiaca</a> You can't get any information by adding <code>timestamp</code> and <code>prior_question_elapsed_time</code>. You could subtract <code>prior_question_elapsed_time</code> from the previous question timestamp for that user to get the timestamp for when a user started working on a question, though I don't know why you would want that. </p>\n<p>In rough terms you might want to look at the <code>timestamp</code> column to check whether a user worked on 10 questions a day or 10 questions per month and the <code>prior_question_elapsed_time</code> column to see if they need 1 second to respond or three minutes.</p>",
          "votes": 12,
          "replies": []
        },
        {
          "id": 1041968,
          "author_name": "Ning Jia",
          "author_url": "",
          "post_date": "2020-10-08T02:01:15.200000",
          "content": "<p>Thanks for your reply.</p>\n<p>It helps my feature engineering. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1046042,
          "author_name": "Aditya Soni",
          "author_url": "",
          "post_date": "2020-10-11T09:48:42.360000",
          "content": "<blockquote>\n  <p>In rough terms you might want to look at the timestamp column to check whether a user worked on 10 questions a day or 10 questions per month and the prior_question_elapsed_time column to see if they need 1 second to respond or three minutes.</p>\n</blockquote>\n<p>Any pointers how can we even calculate this?</p>\n<p>My thought is, assume a time stamp \"k\" and then scan till \"k+86400000msec\", that might give us a single day activity of the user.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1040881,
      "author_name": "Zhao Chow",
      "author_url": "",
      "post_date": "2020-10-07T12:21:20.987000",
      "content": "<p>It seems to me that in the example you have shown, <code>prior_question_elapsed_time</code> = 37000 and <code>timestamp</code> = 59943 and thus <code>prior_question_elapsed_time</code> is smaller than <code>timestamp</code> and not greater.</p>\n<p>However, similar to your question, I have a hard time understanding the relationship between <code>timestamp</code> and <code>task_container_id</code>, the latter being defined as</p>\n<blockquote>\n  <p>Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a <code>task_container_id</code>. Monotonically increasing for each user.</p>\n</blockquote>\n<p>In my understanding, the interaction at <code>timestamp</code> = 0 for any user should be the first interaction he/she has ever had, be it a question or a lecture (or any other type of interaction that exists). However, at <code>row_id</code> = 0, the <code>timestamp</code> is equal to 0 but <code>task_container_id</code> is equal to 1, not 0, which implies that the user has already had another question (<code>content_type</code> = 0) previously… It seems contradictory to me but maybe my understanding of these fields is incorrect.</p>\n<p>I have also checked other cases like this and there seems to be quite a few others.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5589062%2Fb87552ba9c768c016e22eb4e394fe30a%2Fkaggle.png?generation=1602073116276058&amp;alt=media\" alt=\"\"></p>",
      "votes": 3,
      "replies": [
        {
          "id": 1040993,
          "author_name": "Max Halford",
          "author_url": "",
          "post_date": "2020-10-07T13:38:08.080000",
          "content": "<p>This is an important question. The <code>task_container_id</code> variable is supposed to be monotonically increasing for each user. But that doesn't seem to be the case. For instance, see user 115.</p>\n<p>To fix this, I suggest renumbering the tasks to make sure they're monotonically increasing for each user:</p>\n<pre><code>train['task_container_id'] = (\n    train\n    .groupby('user_id')['task_container_id']\n    .transform(lambda x: pd.factorize(x)[0])\n    .astype('int16')\n)\n</code></pre>",
          "votes": 11,
          "replies": []
        },
        {
          "id": 1041231,
          "author_name": "Zhao Chow",
          "author_url": "",
          "post_date": "2020-10-07T16:21:07.107000",
          "content": "<p>Thank you for your suggestion!</p>\n<p>Now I am just wondering if this is an actual mistake in the data and how much it will affect the training. It might also be better to report it to the hosts of the competition and let other people know about it so they can verify the impact of applying this fix or not. I will create a new discussion around this topic.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1126352,
          "author_name": "عثمان",
          "author_url": "",
          "post_date": "2020-12-25T14:38:32.910000",
          "content": "<p>-- still thinking about this</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1041437,
      "author_name": "EnricRovira",
      "author_url": "",
      "post_date": "2020-10-07T18:33:04.573000",
      "content": "<p>I think they mean the time elapsed on the <strong>prior</strong> question.</p>\n<p>Its not cummulative it only depends on the interaction between the user and the prior question.</p>\n<p>Hope it helps</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1041452,
          "author_name": "EnricRovira",
          "author_url": "",
          "post_date": "2020-10-07T18:43:24.403000",
          "content": "<p>[Correction] on the prior <strong>bundle</strong></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1064682,
      "author_name": "TitiTest",
      "author_url": "",
      "post_date": "2020-10-30T12:07:43.753000",
      "content": "<p>Probably something escape me but i don't understand why you put the prior_question_elapsed_time and not the current one ?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1095257,
          "author_name": "ISMAX",
          "author_url": "",
          "post_date": "2020-11-29T12:08:26.403000",
          "content": "<p>It is to reflect what information we have at inference time when we need to predict result for a question the user has not yet seen. In that case we don't know the elapsed time yet.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1095369,
          "author_name": "TitiTest",
          "author_url": "",
          "post_date": "2020-11-29T14:31:45.413000",
          "content": "<p>Yes , thanks ISMAX, i understood it soon after my post.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1044576,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-09T23:26:26.280000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1040790,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-10-07T10:52:02.817000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1040755": "On data tab, `timestamp` is defined as \"the time between this user interaction and the first event from that user\", and `prior_question_elapsed_time` is defined as \"How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle\".\n\nI had a hard time understanding the difference between those two columns. If I understood correctly, every row can be a question or another content, but a user can also have other interactions between them that are not listed in this dataset. Am I correct? If that assumption is correct, then why `prior_question_elapsed_time` is greater than `timestamp` in some rows?\n\n![img](https://i.ibb.co/S7wYLS0/aaa.jpg)\n",
    "1041405": "- The `timestamp` column shows when an activity is finished, not when it started. A user could have spent an arbitrary amount of time working on the first problem before the first `timestamp` value would get logged.\n- You seem to be skipping over the \"prior\" part of the column name, which is critical.\n",
    "1041237": "![img](https://i.ibb.co/Tk8KrDQ/ad.jpg)\n\nFrom my understanding, this user spent 28000 on first question and 21592 passed since the users last interaction. Only logical explanation is `timestamp` timer starts after first question is answered or lecture is finished. `prior_question_elapsed_time` timer starts when the user starts doing the previous question and it ends when the user moves to another question.",
    "1040881": "It seems to me that in the example you have shown, `prior_question_elapsed_time` = 37000 and `timestamp` = 59943 and thus `prior_question_elapsed_time` is smaller than `timestamp` and not greater.\n\nHowever, similar to your question, I have a hard time understanding the relationship between `timestamp` and `task_container_id`, the latter being defined as\n\n> Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a `task_container_id`. Monotonically increasing for each user.\n\nIn my understanding, the interaction at `timestamp` = 0 for any user should be the first interaction he/she has ever had, be it a question or a lecture (or any other type of interaction that exists). However, at `row_id` = 0, the `timestamp` is equal to 0 but `task_container_id` is equal to 1, not 0, which implies that the user has already had another question (`content_type` = 0) previously... It seems contradictory to me but maybe my understanding of these fields is incorrect.\n\nI have also checked other cases like this and there seems to be quite a few others.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5589062%2Fb87552ba9c768c016e22eb4e394fe30a%2Fkaggle.png?generation=1602073116276058&alt=media)",
    "1041437": "I think they mean the time elapsed on the **prior** question.\n\nIts not cummulative it only depends on the interaction between the user and the prior question.\n\nHope it helps",
    "1064682": "Probably something escape me but i don't understand why you put the prior_question_elapsed_time and not the current one ?",
    "1044576": "",
    "1040790": ""
  }
}