{ "id": 199053, "title": "Is there problem with timestamp", "url": "/competitions/riiid-test-answer-prediction/discussion/199053", "author_name": "Do Phuc", "post_date": "2020-11-24T07:04:48.211000", "votes": 7, "comment_count": 15, "views": 0, "content": "

\"\"

\n

I tried to get lag time - time from user submit previous question to time start new question, but some value is less than zero.
\nCan you help me to find out where I am wrong.

\n

Thanks!

", "messages": [ { "id": 1089178, "postDate": "2020-11-24T09:07:46.050Z", "content": "

You need to group by users before shifting and subtracting. Something like that should work:
\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()
\nThen don't forget to handle zeros for same task container ids :)

", "rawMarkdown": "You need to group by users before shifting and subtracting. Something like that should work:\n`train['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()`\nThen don't forget to handle zeros for same task container ids :)", "votes": 7, "replies": [ { "id": 1089179, "postDate": "2020-11-24T09:10:31.967Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1089197, "postDate": "2020-11-24T09:32:39.993Z", "content": "

thanks a lot

", "rawMarkdown": "thanks a lot" }, { "id": 1089263, "postDate": "2020-11-24T10:56:13.433Z", "content": "

At inference time, you need to track the last seen timestamp as well, right?

", "rawMarkdown": "At inference time, you need to track the last seen timestamp as well, right?" }, { "id": 1089265, "postDate": "2020-11-24T10:58:54.157Z", "content": "

Exactly, I've got a dictionary in which is stored the last timestamp for each user

", "rawMarkdown": "Exactly, I've got a dictionary in which is stored the last timestamp for each user" }, { "id": 1090335, "postDate": "2020-11-25T09:08:32.523Z", "content": "

replace 'zero' to 'np.nan' and fillna(method='ffill'), is that right?

", "rawMarkdown": "replace 'zero' to 'np.nan' and fillna(method='ffill'), is that right?" }, { "id": 1095267, "postDate": "2020-11-29T12:26:46.723Z", "content": "

You can use replace(0, method='ffill') instead of fillna()

", "rawMarkdown": "You can use replace(0, method='ffill') instead of fillna()" }, { "id": 1101195, "postDate": "2020-12-03T17:49:49.803Z", "content": "

I want to know that the following formula is enought to compute the lag time?
\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()

", "rawMarkdown": "I want to know that the following formula is enought to compute the lag time?\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()" }, { "id": 1103980, "postDate": "2020-12-06T13:41:45Z", "content": "

Let me know if this is right as per the image. lag time lt2 = curr_ts (t4) - prior_ts (t2) - elapsed time. \"![\"]

", "rawMarkdown": "Let me know if this is right as per the image. lag time lt2 = curr_ts (t4) - prior_ts (t2) - elapsed time. ![![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F601447%2Fadfc21559b29329af2f32429bfc090ce%2Fsaint_lag_time.png?generation=1607262045161464&alt=media)]" } ] }, { "id": 1089054, "postDate": "2020-11-24T07:04:48.210Z", "content": "

\"\"

\n

I tried to get lag time - time from user submit previous question to time start new question, but some value is less than zero.
\nCan you help me to find out where I am wrong.

\n

Thanks!

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1580342%2F03df4be4190007012d3f0f0320b39409%2Ftimstamp.png?generation=1606201415546999&alt=media)\n\nI tried to get lag time - time from user submit previous question to time start new question, but some value is less than zero.\nCan you help me to find out where I am wrong.\n\nThanks!", "votes": 7 }, { "id": 1089119, "postDate": "2020-11-24T08:04:58.143Z", "content": "

timestamp is the time of activity finished,not activity start

", "rawMarkdown": "timestamp is the time of activity finished,not activity start", "votes": 1, "replies": [ { "id": 1089132, "postDate": "2020-11-24T08:16:44.967Z", "content": "

thanks so much.
\nBut if timestamp is the time of activity finished, is It leaked?
\nBecause we know the time that user submit answer of the question

", "rawMarkdown": "thanks so much. \nBut if timestamp is the time of activity finished, is It leaked?\nBecause we know the time that user submit answer of the question\n" }, { "id": 1089152, "postDate": "2020-11-24T08:40:05.637Z", "content": "

the time of user submit answer of the current question is't future ,I also use lag_time,it worked

", "rawMarkdown": "the time of user submit answer of the current question is't future ,I also use lag_time,it worked", "votes": 1 } ] }, { "id": 1101193, "postDate": "2020-12-03T17:48:42.547Z", "content": "

Help Please,
\nI want to know that the following formula is enought to compute the lag time?
\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()

", "rawMarkdown": "Help Please, \nI want to know that the following formula is enought to compute the lag time?\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()" }, { "id": 1094503, "postDate": "2020-11-28T17:07:32.487Z", "content": "

Yes, I think there is. I have done groupby user Id and arrange by row id… but I have noticed something off too. I think there are some data quality issues here.

\n

I have the following example after processing the data bit: row id=2 timestamp = 118363 for user id =115. Then I have row id=3 timestamp =131167 for user id =115, suggesting 12804 for the elapsed time for the prior question. However, at row id =3 for prior_question_elapsed_time =19000. This means the prior question (row id =2) took longer than the time stamp would suggest…

\n

which one is correct? did the user start the next question prior to finishing the previous question?
\nPerhaps I did something wrong here with my code? Could someone confirm this for user id =115?
\nOr is this not the right way to use the variables? Am I misunderstanding the definitions of the variables?

", "rawMarkdown": "Yes, I think there is. I have done groupby user Id and arrange by row id... but I have noticed something off too. I think there are some data quality issues here.\n\nI have the following example after processing the data bit: row id=2 timestamp = 118363 for user id =115. Then I have row id=3 timestamp =131167 for user id =115, suggesting 12804 for the elapsed time for the prior question. However, at row id =3 for prior_question_elapsed_time =19000. This means the prior question (row id =2) took longer than the time stamp would suggest... \n\nwhich one is correct? did the user start the next question prior to finishing the previous question?\nPerhaps I did something wrong here with my code? Could someone confirm this for user id =115?\nOr is this not the right way to use the variables? Am I misunderstanding the definitions of the variables?\n" }, { "id": 1103972, "postDate": "2020-12-06T13:34:28.657Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1089178, "author_name": "Alex", "author_url": "", "post_date": "2020-11-24T09:07:46.050000", "content": "

You need to group by users before shifting and subtracting. Something like that should work:
\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()
\nThen don't forget to handle zeros for same task container ids :)

", "votes": 7, "replies": [ { "id": 1089179, "author_name": "", "author_url": "", "post_date": "2020-11-24T09:10:31.967000", "content": "", "votes": 0, "replies": [] }, { "id": 1089197, "author_name": "Do Phuc", "author_url": "", "post_date": "2020-11-24T09:32:39.993000", "content": "

thanks a lot

", "votes": 0, "replies": [] }, { "id": 1089263, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-24T10:56:13.433000", "content": "

At inference time, you need to track the last seen timestamp as well, right?

", "votes": 0, "replies": [] }, { "id": 1089265, "author_name": "Alex", "author_url": "", "post_date": "2020-11-24T10:58:54.157000", "content": "

Exactly, I've got a dictionary in which is stored the last timestamp for each user

", "votes": 0, "replies": [] }, { "id": 1090335, "author_name": "Joseph Zhou", "author_url": "", "post_date": "2020-11-25T09:08:32.523000", "content": "

replace 'zero' to 'np.nan' and fillna(method='ffill'), is that right?

", "votes": 0, "replies": [] }, { "id": 1095267, "author_name": "Dean", "author_url": "", "post_date": "2020-11-29T12:26:46.723000", "content": "

You can use replace(0, method='ffill') instead of fillna()

", "votes": 0, "replies": [] }, { "id": 1101195, "author_name": "Sahin Isik", "author_url": "", "post_date": "2020-12-03T17:49:49.803000", "content": "

I want to know that the following formula is enought to compute the lag time?
\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()

", "votes": 0, "replies": [] }, { "id": 1103980, "author_name": "Ashish Lal", "author_url": "", "post_date": "2020-12-06T13:41:45", "content": "

Let me know if this is right as per the image. lag time lt2 = curr_ts (t4) - prior_ts (t2) - elapsed time. \"![\"]

", "votes": 0, "replies": [] } ] }, { "id": 1089119, "author_name": "qiaqia", "author_url": "", "post_date": "2020-11-24T08:04:58.143000", "content": "

timestamp is the time of activity finished,not activity start

", "votes": 1, "replies": [ { "id": 1089132, "author_name": "Do Phuc", "author_url": "", "post_date": "2020-11-24T08:16:44.967000", "content": "

thanks so much.
\nBut if timestamp is the time of activity finished, is It leaked?
\nBecause we know the time that user submit answer of the question

", "votes": 0, "replies": [] }, { "id": 1089152, "author_name": "qiaqia", "author_url": "", "post_date": "2020-11-24T08:40:05.637000", "content": "

the time of user submit answer of the current question is't future ,I also use lag_time,it worked

", "votes": 1, "replies": [] } ] }, { "id": 1101193, "author_name": "Sahin Isik", "author_url": "", "post_date": "2020-12-03T17:48:42.547000", "content": "

Help Please,
\nI want to know that the following formula is enought to compute the lag time?
\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()

", "votes": 0, "replies": [] }, { "id": 1094503, "author_name": "Michael Bryant", "author_url": "", "post_date": "2020-11-28T17:07:32.487000", "content": "

Yes, I think there is. I have done groupby user Id and arrange by row id… but I have noticed something off too. I think there are some data quality issues here.

\n

I have the following example after processing the data bit: row id=2 timestamp = 118363 for user id =115. Then I have row id=3 timestamp =131167 for user id =115, suggesting 12804 for the elapsed time for the prior question. However, at row id =3 for prior_question_elapsed_time =19000. This means the prior question (row id =2) took longer than the time stamp would suggest…

\n

which one is correct? did the user start the next question prior to finishing the previous question?
\nPerhaps I did something wrong here with my code? Could someone confirm this for user id =115?
\nOr is this not the right way to use the variables? Am I misunderstanding the definitions of the variables?

", "votes": 0, "replies": [] }, { "id": 1103972, "author_name": "", "author_url": "", "post_date": "2020-12-06T13:34:28.657000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1089178": "You need to group by users before shifting and subtracting. Something like that should work:\n`train['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()`\nThen don't forget to handle zeros for same task container ids :)", "1089054": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1580342%2F03df4be4190007012d3f0f0320b39409%2Ftimstamp.png?generation=1606201415546999&alt=media)\n\nI tried to get lag time - time from user submit previous question to time start new question, but some value is less than zero.\nCan you help me to find out where I am wrong.\n\nThanks!", "1089119": "timestamp is the time of activity finished,not activity start", "1101193": "Help Please, \nI want to know that the following formula is enought to compute the lag time?\ntrain['lag'] = train[['user_id', 'timestamp']].groupby(['user_id'])['timestamp'].diff()", "1094503": "Yes, I think there is. I have done groupby user Id and arrange by row id... but I have noticed something off too. I think there are some data quality issues here.\n\nI have the following example after processing the data bit: row id=2 timestamp = 118363 for user id =115. Then I have row id=3 timestamp =131167 for user id =115, suggesting 12804 for the elapsed time for the prior question. However, at row id =3 for prior_question_elapsed_time =19000. This means the prior question (row id =2) took longer than the time stamp would suggest... \n\nwhich one is correct? did the user start the next question prior to finishing the previous question?\nPerhaps I did something wrong here with my code? Could someone confirm this for user id =115?\nOr is this not the right way to use the variables? Am I misunderstanding the definitions of the variables?\n", "1103972": "" } }