{ "id": 6254, "title": "Data issues", "url": "/competitions/yandex-personalized-web-search-challenge/discussion/6254", "author_name": "", "post_date": "2013-11-07T15:27:39.280Z", "votes": null, "comment_count": 19, "views": 3643, "content": "

While checking the data consistency, I've found some uncertainty.

\n

Here is the beginning of "train":

\n

0    M    4    0 

\n

0    0    Q    0    10047345    ...    68893581,5149883    ...

\n

0    108    C    0    50628761

\n

0    1080    C    0    50628761

\n

We see two different click on the same url.

\n

Is this the last click?

\n

Suppose we have one more click in this session:

\n

0    2000   C    0    68893581

\n

What is the dwell time: 1080-108, or 2000-1080, or 2000-108 ?

\n

We need to reproduce exactly your algorithm of relevance calculation, so how should we operate with repeated clicking?

", "messages": [ { "id": "33375", "postDate": "11/07/2013 15:27:39", "content": "

While checking the data consistency, I've found some uncertainty.

\n

Here is the beginning of "train":

\n

0    M    4    0 

\n

0    0    Q    0    10047345    ...    68893581,5149883    ...

\n

0    108    C    0    50628761

\n

0    1080    C    0    50628761

\n

We see two different click on the same url.

\n

Is this the last click?

\n

Suppose we have one more click in this session:

\n

0    2000   C    0    68893581

\n

What is the dwell time: 1080-108, or 2000-1080, or 2000-108 ?

\n

We need to reproduce exactly your algorithm of relevance calculation, so how should we operate with repeated clicking?

", "rawMarkdown": "", "votes": null }, { "id": "33379", "postDate": "11/07/2013 16:07:18", "content": "

The last click is the click with the latest timestamp in the session, so the second  click on 50628761 in train.gz is the latest in the session with its SessionId equal to 0.

\n

 

\n

In your example, two clicks have the following dwell times: 1080 - 108 and 2000 - 1080. When labelling a URLID's relevance, we consider all the clicks associated with the URLID and use the maximum of the dwell times (1080 - 108 > 2000 - 1080 in your example).

\n

You can find an additional discussion in another topic .

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "33437", "postDate": "11/08/2013 18:31:18", "content": "

file

", "rawMarkdown": "", "votes": null }, { "id": "33439", "postDate": "11/08/2013 18:42:34", "content": "

The file above contains the relevances for the first 1000 queries.

\n

Calculation of relevances for the training sample is purely technical task, but one may easily make mistake on it. So I think it is a good idea to share such results for comparison and checking.

\n

It would be nice to have such file in competition data...

", "rawMarkdown": "", "votes": null }, { "id": "33505", "postDate": "11/10/2013 13:16:22", "content": "

my first diff:

\n

6th line (session 4, serp 0), your result 0000000002

\n

data: 1st dwell time 86-27>=50, 3rd dwell time 216-121>=50, so i guess it should be 0001010002

", "rawMarkdown": "", "votes": null }, { "id": "33507", "postDate": "11/10/2013 14:52:23", "content": "

Thank you very much. I've found a mistake in my program.

\n

Here is the corrected file.

\n

I could post the full file, but I am not sure that this is allowed by the rules.

", "rawMarkdown": "", "votes": null }, { "id": "34867", "postDate": "11/18/2013 22:24:24", "content": "

I don't think you're interpreting the last click in a SERP properly, you're treating it as "highly relevant," even when there's another SERP in the same session.

\n

For example, session 13 SERP 2 has a query then a single click at time 5658.  In the same session, the user re-submits their search at time 5753, so the dwell time on that click should be 5753 - 5658 = 95, a "1" grade.  But you have it marked as a "2" grade.

", "rawMarkdown": "", "votes": null }, { "id": "34883", "postDate": "11/19/2013 08:16:13", "content": "

[quote=Martin C. Martin;34867]

\n

I don't think you're interpreting the last click in a SERP properly, you're treating it as "highly relevant," even when there's another SERP in the same session.

\n

[/quote]

\n

Thank you for the comment.

\n

But here is citation from "Evaluation": "2 (highly relevant) grade corresponds to documents with clicks and dwell time not shorter than 400 time units and to the
document with the last click in the search session".

\n

So, the click in session 13 SERP 2 should be highly relevant because it is really the last click in the session, since there were only queries but no clicks later!

", "rawMarkdown": "", "votes": null }, { "id": "34893", "postDate": "11/19/2013 11:42:29", "content": "

Thanks.  Eugene, can you confirm that the last click counts as "highly relevant," even when the user comes back and does another search but doesn't click on anything?  Doing another search suggests they didn't find what they wanted with their click, and are looking for something else.

", "rawMarkdown": "", "votes": null }, { "id": "34915", "postDate": "11/19/2013 17:05:29", "content": "

Sorry for the ambiguity in the description. We'll fix it asap.

\n

The relevance level of 2 is assigned to the clicks which have dwell time not smaller than 400 time units and to the clicks that are the last actions in the corresponding sessions.

\n

In other words, if the dwell time can be calculated then the relevance is assigned according to it. If the click is the last action in the session (in other words, the next action is performed after a long time window separating two sessions) then the relevance is set to 2.

\n

We believe this definition is more consistent.

", "rawMarkdown": "", "votes": null }, { "id": "34940", "postDate": "11/19/2013 22:44:17", "content": "

Victor, any chance you can update your file?

", "rawMarkdown": "", "votes": null }, { "id": "34973", "postDate": "11/20/2013 19:38:05", "content": "

Hi

\n

I download the Data, unziped it , but I can not open Them. which software should I use to open them. files do not have  any extension !!!

\n

Please help me.

\n

 

\n

TNX

", "rawMarkdown": "", "votes": null }, { "id": "34974", "postDate": "11/20/2013 21:15:59", "content": "

You can open such files with good Text Editors like Notepad++. (except the train file ;))

\n

But for further editings you need to get line for line I think. Your RAM is not that big the store the data.

", "rawMarkdown": "", "votes": null }, { "id": "34984", "postDate": "11/21/2013 05:09:48", "content": "

[quote=Martin C. Martin;34940]

\n

Victor, any chance you can update your file?

\n

[/quote]

\n

Here it is.

", "rawMarkdown": "", "votes": null }, { "id": "34985", "postDate": "11/21/2013 05:14:05", "content": "

[quote=Ali Reza Honarvar;34973]

\n

I download the Data, unziped it , but I can not open Them. which software should I use to open them. files do not have  any extension !!!

\n

[/quote]

\n

I use the F3-viewer in FAR-manager.

", "rawMarkdown": "", "votes": null }, { "id": "35093", "postDate": "11/23/2013 15:03:02", "content": "

[quote=Eugene;34915]

\n

Sorry for the ambiguity in the description. We'll fix it asap.

\n

[/quote]

\n

May be is it possible to provide a file with calculated relevances?

\n

There is still some uncertainty with the cases when some clicks occurs later than the next query. And with simultaneous events too.

\n

Here is my implementation of relevance (grade) calculation:    

", "rawMarkdown": "", "votes": null }, { "id": "35150", "postDate": "11/25/2013 08:55:34", "content": "

[quote=Ali Reza Honarvar;34973]

\n

Hi

\n

I download the Data, unziped it , but I can not open Them. which software should I use to open them. files do not have  any extension !!!

\n

Please help me.

\n

[/quote]

\n

I don't think you would want to open the complete train file at once. Under Linux, you can use 'more' to view the files. I advice you to develop your code on a small sample. You can create a sample without unzipping using the following command: 'gzip -cd train.gz | head -1000 > train.sample'. Or on the unzipped file: 'cat train |  head -1000 > train.sample'

\n

If you are using java, then you can read the gz-files without unzipping them using the class GZIPInputStream.

\n

Good luck!

", "rawMarkdown": "", "votes": null }, { "id": "35204", "postDate": "11/25/2013 18:12:04", "content": "

Hi

\n

in windows you can use 'Large Texte File Viewer' to open the file.

\n

also you can use 'HJSplit' to split train file in many parts.

", "rawMarkdown": "", "votes": null }, { "id": "35483", "postDate": "11/29/2013 08:42:05", "content": "

I found this strange piece of data:

\n

22      0       Q       0       16185818        4078517,3143668,1988399 35767635,3364600        57421304,4536568        35296159,3323845        49235419,4140722        68222659,5116467        55142634,4429180        36577088,3407786        61943983,4767047        65157339,4954478        42867421,3782572

\n

22      25      Q       2       16185820        4078517,3143668,1988399,2133749 30325446,2938326        28846733,2833948        35769151,3364600        57008295,4511361        28175892,2768660        28864909,2833991        60703868,4697323        22790363,2294429        57692687,4551821        6473264,800608

\n

22      25      Q       1       16185818        4078517,3143668,1988399 35767635,3364600        57421304,4536568        35296159,3323845        49235419,4140722        68222659,5116467        55142634,4429180        36577088,3407786        61943983,4767047        65157339,4954478        42867421,3782572

\n

22      31      C       2       30325446

\n

22      112     C       2       28175892

\n

Notice two queries at the same time! 

\n

This corresponds to strings #51-53 in your file :

\n

0000000000  (serpid = 0)

\n

1000100000  (serpid = 2)

\n

0000000000  (serpid = 1)

\n

But don't you think we should sort them by SERPID?

\n

In the first 1000 queries this trouble (with two queries in one unit of time written in reverse order of SERPIDs) appears one another time in session #458.

", "rawMarkdown": "", "votes": null }, { "id": "35486", "postDate": "11/29/2013 10:04:54", "content": "

...and also I found mistake in your file:

\n

309     0       Q       0       11466723        3332767,4291604 29370045,2863645        55303432,4433251        3276363,443265  29322278,2863497        29371141,2863645        36829352,3418807        48935893,4127107        27402385,2655130        28137920,2761328        62315277,4791822

\n

309     13      Q       1       11466723        3332767,4291604 29370045,2863645        55303432,4433251        3276363,443265  29322278,2863497        29371141,2863645        36829352,3418807        48935893,4127107        27402385,2655130        28137920,2761328        62315277,4791822

\n

309     13      C       0       29370045

\n

310     M       2       52      

\n

Here should be "2000000000" instead of "0000000000" in your file (line 557) because the click is the last in this session. I suppose, this may be associated with mixed SERPIDs.

\n

Excluding this two things I have the same results for relevances.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 33379, "author_name": "eugene1751", "author_url": "", "post_date": "11/07/2013 16:07:18", "content": "

The last click is the click with the latest timestamp in the session, so the second  click on 50628761 in train.gz is the latest in the session with its SessionId equal to 0.

\n

 

\n

In your example, two clicks have the following dwell times: 1080 - 108 and 2000 - 1080. When labelling a URLID's relevance, we consider all the clicks associated with the URLID and use the maximum of the dwell times (1080 - 108 > 2000 - 1080 in your example).

\n

You can find an additional discussion in another topic .

\n

 

", "votes": null, "replies": [] }, { "id": 33437, "author_name": "nedelko", "author_url": "", "post_date": "11/08/2013 18:31:18", "content": "

file

", "votes": null, "replies": [] }, { "id": 33439, "author_name": "nedelko", "author_url": "", "post_date": "11/08/2013 18:42:34", "content": "

The file above contains the relevances for the first 1000 queries.

\n

Calculation of relevances for the training sample is purely technical task, but one may easily make mistake on it. So I think it is a good idea to share such results for comparison and checking.

\n

It would be nice to have such file in competition data...

", "votes": null, "replies": [] }, { "id": 33505, "author_name": "corrision", "author_url": "", "post_date": "11/10/2013 13:16:22", "content": "

my first diff:

\n

6th line (session 4, serp 0), your result 0000000002

\n

data: 1st dwell time 86-27>=50, 3rd dwell time 216-121>=50, so i guess it should be 0001010002

", "votes": null, "replies": [] }, { "id": 33507, "author_name": "nedelko", "author_url": "", "post_date": "11/10/2013 14:52:23", "content": "

Thank you very much. I've found a mistake in my program.

\n

Here is the corrected file.

\n

I could post the full file, but I am not sure that this is allowed by the rules.

", "votes": null, "replies": [] }, { "id": 34867, "author_name": "martincmartin", "author_url": "", "post_date": "11/18/2013 22:24:24", "content": "

I don't think you're interpreting the last click in a SERP properly, you're treating it as "highly relevant," even when there's another SERP in the same session.

\n

For example, session 13 SERP 2 has a query then a single click at time 5658.  In the same session, the user re-submits their search at time 5753, so the dwell time on that click should be 5753 - 5658 = 95, a "1" grade.  But you have it marked as a "2" grade.

", "votes": null, "replies": [] }, { "id": 34883, "author_name": "nedelko", "author_url": "", "post_date": "11/19/2013 08:16:13", "content": "

[quote=Martin C. Martin;34867]

\n

I don't think you're interpreting the last click in a SERP properly, you're treating it as "highly relevant," even when there's another SERP in the same session.

\n

[/quote]

\n

Thank you for the comment.

\n

But here is citation from "Evaluation": "2 (highly relevant) grade corresponds to documents with clicks and dwell time not shorter than 400 time units and to the
document with the last click in the search session".

\n

So, the click in session 13 SERP 2 should be highly relevant because it is really the last click in the session, since there were only queries but no clicks later!

", "votes": null, "replies": [] }, { "id": 34893, "author_name": "martincmartin", "author_url": "", "post_date": "11/19/2013 11:42:29", "content": "

Thanks.  Eugene, can you confirm that the last click counts as "highly relevant," even when the user comes back and does another search but doesn't click on anything?  Doing another search suggests they didn't find what they wanted with their click, and are looking for something else.

", "votes": null, "replies": [] }, { "id": 34915, "author_name": "eugene1751", "author_url": "", "post_date": "11/19/2013 17:05:29", "content": "

Sorry for the ambiguity in the description. We'll fix it asap.

\n

The relevance level of 2 is assigned to the clicks which have dwell time not smaller than 400 time units and to the clicks that are the last actions in the corresponding sessions.

\n

In other words, if the dwell time can be calculated then the relevance is assigned according to it. If the click is the last action in the session (in other words, the next action is performed after a long time window separating two sessions) then the relevance is set to 2.

\n

We believe this definition is more consistent.

", "votes": null, "replies": [] }, { "id": 34940, "author_name": "martincmartin", "author_url": "", "post_date": "11/19/2013 22:44:17", "content": "

Victor, any chance you can update your file?

", "votes": null, "replies": [] }, { "id": 34973, "author_name": "alirezahonarvar", "author_url": "", "post_date": "11/20/2013 19:38:05", "content": "

Hi

\n

I download the Data, unziped it , but I can not open Them. which software should I use to open them. files do not have  any extension !!!

\n

Please help me.

\n

 

\n

TNX

", "votes": null, "replies": [] }, { "id": 34974, "author_name": "curtis0", "author_url": "", "post_date": "11/20/2013 21:15:59", "content": "

You can open such files with good Text Editors like Notepad++. (except the train file ;))

\n

But for further editings you need to get line for line I think. Your RAM is not that big the store the data.

", "votes": null, "replies": [] }, { "id": 34984, "author_name": "nedelko", "author_url": "", "post_date": "11/21/2013 05:09:48", "content": "

[quote=Martin C. Martin;34940]

\n

Victor, any chance you can update your file?

\n

[/quote]

\n

Here it is.

", "votes": null, "replies": [] }, { "id": 34985, "author_name": "nedelko", "author_url": "", "post_date": "11/21/2013 05:14:05", "content": "

[quote=Ali Reza Honarvar;34973]

\n

I download the Data, unziped it , but I can not open Them. which software should I use to open them. files do not have  any extension !!!

\n

[/quote]

\n

I use the F3-viewer in FAR-manager.

", "votes": null, "replies": [] }, { "id": 35093, "author_name": "nedelko", "author_url": "", "post_date": "11/23/2013 15:03:02", "content": "

[quote=Eugene;34915]

\n

Sorry for the ambiguity in the description. We'll fix it asap.

\n

[/quote]

\n

May be is it possible to provide a file with calculated relevances?

\n

There is still some uncertainty with the cases when some clicks occurs later than the next query. And with simultaneous events too.

\n

Here is my implementation of relevance (grade) calculation:    

", "votes": null, "replies": [] }, { "id": 35150, "author_name": "suzanv", "author_url": "", "post_date": "11/25/2013 08:55:34", "content": "

[quote=Ali Reza Honarvar;34973]

\n

Hi

\n

I download the Data, unziped it , but I can not open Them. which software should I use to open them. files do not have  any extension !!!

\n

Please help me.

\n

[/quote]

\n

I don't think you would want to open the complete train file at once. Under Linux, you can use 'more' to view the files. I advice you to develop your code on a small sample. You can create a sample without unzipping using the following command: 'gzip -cd train.gz | head -1000 > train.sample'. Or on the unzipped file: 'cat train |  head -1000 > train.sample'

\n

If you are using java, then you can read the gz-files without unzipping them using the class GZIPInputStream.

\n

Good luck!

", "votes": null, "replies": [] }, { "id": 35204, "author_name": "habib128987", "author_url": "", "post_date": "11/25/2013 18:12:04", "content": "

Hi

\n

in windows you can use 'Large Texte File Viewer' to open the file.

\n

also you can use 'HJSplit' to split train file in many parts.

", "votes": null, "replies": [] }, { "id": 35483, "author_name": "chervinskii", "author_url": "", "post_date": "11/29/2013 08:42:05", "content": "

I found this strange piece of data:

\n

22      0       Q       0       16185818        4078517,3143668,1988399 35767635,3364600        57421304,4536568        35296159,3323845        49235419,4140722        68222659,5116467        55142634,4429180        36577088,3407786        61943983,4767047        65157339,4954478        42867421,3782572

\n

22      25      Q       2       16185820        4078517,3143668,1988399,2133749 30325446,2938326        28846733,2833948        35769151,3364600        57008295,4511361        28175892,2768660        28864909,2833991        60703868,4697323        22790363,2294429        57692687,4551821        6473264,800608

\n

22      25      Q       1       16185818        4078517,3143668,1988399 35767635,3364600        57421304,4536568        35296159,3323845        49235419,4140722        68222659,5116467        55142634,4429180        36577088,3407786        61943983,4767047        65157339,4954478        42867421,3782572

\n

22      31      C       2       30325446

\n

22      112     C       2       28175892

\n

Notice two queries at the same time! 

\n

This corresponds to strings #51-53 in your file :

\n

0000000000  (serpid = 0)

\n

1000100000  (serpid = 2)

\n

0000000000  (serpid = 1)

\n

But don't you think we should sort them by SERPID?

\n

In the first 1000 queries this trouble (with two queries in one unit of time written in reverse order of SERPIDs) appears one another time in session #458.

", "votes": null, "replies": [] }, { "id": 35486, "author_name": "chervinskii", "author_url": "", "post_date": "11/29/2013 10:04:54", "content": "

...and also I found mistake in your file:

\n

309     0       Q       0       11466723        3332767,4291604 29370045,2863645        55303432,4433251        3276363,443265  29322278,2863497        29371141,2863645        36829352,3418807        48935893,4127107        27402385,2655130        28137920,2761328        62315277,4791822

\n

309     13      Q       1       11466723        3332767,4291604 29370045,2863645        55303432,4433251        3276363,443265  29322278,2863497        29371141,2863645        36829352,3418807        48935893,4127107        27402385,2655130        28137920,2761328        62315277,4791822

\n

309     13      C       0       29370045

\n

310     M       2       52      

\n

Here should be "2000000000" instead of "0000000000" in your file (line 557) because the click is the last in this session. I suppose, this may be associated with mixed SERPIDs.

\n

Excluding this two things I have the same results for relevances.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "33375": "", "33379": "", "33437": "", "33439": "", "33505": "", "33507": "", "34867": "", "34883": "", "34893": "", "34915": "", "34940": "", "34973": "", "34974": "", "34984": "", "34985": "", "35093": "", "35150": "", "35204": "", "35483": "", "35486": "" }, "source": "meta" }