{
  "id": 6229,
  "title": "Sorting based on Global clicks performs worse than Default Baseline",
  "url": "/competitions/yandex-personalized-web-search-challenge/discussion/6229",
  "author_name": "",
  "post_date": "2013-11-05T01:41:22.907Z",
  "votes": null,
  "comment_count": 10,
  "views": 2435,
  "content": "<p>In order to validate that my submission generation was working as planned, I generated a file with the URLs reordered simply based on total clicks.&nbsp;</p>\n<p>The Score was 0.70939.&nbsp; This is less than not doing anything to the order, which is the &quot;Default Ranking Baseline&quot;&nbsp; (which scores 0.79056).&nbsp;</p>\n<p>It seems that Yandex already does more than just sort by clicks.&nbsp; I'm guessing they already have a good amount of personalization built in and they are looking for small incremental improvements.&nbsp; This is the real challenge of this contest.&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
  "messages": [
    {
      "id": "33226",
      "postDate": "11/05/2013 01:41:22",
      "content": "<p>In order to validate that my submission generation was working as planned, I generated a file with the URLs reordered simply based on total clicks.&nbsp;</p>\n<p>The Score was 0.70939.&nbsp; This is less than not doing anything to the order, which is the &quot;Default Ranking Baseline&quot;&nbsp; (which scores 0.79056).&nbsp;</p>\n<p>It seems that Yandex already does more than just sort by clicks.&nbsp; I'm guessing they already have a good amount of personalization built in and they are looking for small incremental improvements.&nbsp; This is the real challenge of this contest.&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33277",
      "postDate": "11/05/2013 16:21:56",
      "content": "<p>Maybe it is too small difference considering that we only see the public score. I have not noticed anything in the data which can support the fact the searches are different for different users.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33364",
      "postDate": "11/07/2013 12:35:24",
      "content": "<p>Yandex is a web search engine. It is most probably using a variant of pagerank within its scoring.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33479",
      "postDate": "11/09/2013 19:22:44",
      "content": "<p>Did anybody calculate Default Ranking Baseline on the training sample? i've got 0.769674 that is very far from 0.79056 on the test sample.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33488",
      "postDate": "11/10/2013 02:00:01",
      "content": "<p>Did you calculate it on all&nbsp;queries in training set?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33494",
      "postDate": "11/10/2013 06:17:24",
      "content": "<p>[quote=DuckTile;33488]</p>\n<p>Did you calculate it on all&nbsp;queries in training set?</p>\n<p>[/quote]</p>\n<p>Yes, but now I've calculated the gain with weights to hold the condition of single test query from each user. The score is 0.762542.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33495",
      "postDate": "11/10/2013 06:24:38",
      "content": "<p>Victor, are you following the procedure of sampling test queries which is described <a href=\"https://www.kaggle.com/c/yandex-personalized-web-search-challenge/data\">here</a>? In particular, in this procedure the queries without any positive labels are not included in the test set.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33498",
      "postDate": "11/10/2013 06:45:47",
      "content": "<p>[quote=Eugene;33495]</p>\n<p>Victor, are you following the procedure of sampling test queries which is described <a href=\"https://www.kaggle.com/c/yandex-personalized-web-search-challenge/data\">here</a>?</p>\n<p>[/quote]</p>\n<p>Yes, but with the following differencies.</p>\n<p>1) I took the all 30 days instead of 3.</p>\n<p>2) I didn't perform random sampling, but I multiplied gains by the weights those are the probabilities of selection the query in test: 1/number_of_relevant_queries_for_the_user.</p>\n<p>3) I didn't implement: &quot;From this set of queries we filter out all queries with clicks performed at the same unit of time&quot;. I don't understand what is &quot;same&quot;: the same with the query time, or the same with any click (or with any event) in the session.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33508",
      "postDate": "11/10/2013 14:55:31",
      "content": "<p>I made a mistake in the program calculating a relevance, the valid score on training sample is 0.79581.</p>\n<p>After one more correction of evaluation procedure, the score is 0.796746.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "34991",
      "postDate": "11/21/2013 10:40:41",
      "content": "<p>seems you did not submit this one.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "34992",
      "postDate": "11/21/2013 10:53:44",
      "content": "<p>@Victor<br>I get 0.79838393939157726 on the last week of the training set :S<br>I used to be much closer when working with the last three days.</p>\n<p>@vbs 0711<br>Victor is trying to compute a &nbsp;score on the training set, not the test set.<br><br></p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 33277,
      "author_name": "emiliogortizg",
      "author_url": "",
      "post_date": "11/05/2013 16:21:56",
      "content": "<p>Maybe it is too small difference considering that we only see the public score. I have not noticed anything in the data which can support the fact the searches are different for different users.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33364,
      "author_name": "",
      "author_url": "",
      "post_date": "11/07/2013 12:35:24",
      "content": "<p>Yandex is a web search engine. It is most probably using a variant of pagerank within its scoring.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33479,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "11/09/2013 19:22:44",
      "content": "<p>Did anybody calculate Default Ranking Baseline on the training sample? i've got 0.769674 that is very far from 0.79056 on the test sample.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33488,
      "author_name": "ducktile",
      "author_url": "",
      "post_date": "11/10/2013 02:00:01",
      "content": "<p>Did you calculate it on all&nbsp;queries in training set?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33494,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "11/10/2013 06:17:24",
      "content": "<p>[quote=DuckTile;33488]</p>\n<p>Did you calculate it on all&nbsp;queries in training set?</p>\n<p>[/quote]</p>\n<p>Yes, but now I've calculated the gain with weights to hold the condition of single test query from each user. The score is 0.762542.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33495,
      "author_name": "eugene1751",
      "author_url": "",
      "post_date": "11/10/2013 06:24:38",
      "content": "<p>Victor, are you following the procedure of sampling test queries which is described <a href=\"https://www.kaggle.com/c/yandex-personalized-web-search-challenge/data\">here</a>? In particular, in this procedure the queries without any positive labels are not included in the test set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33498,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "11/10/2013 06:45:47",
      "content": "<p>[quote=Eugene;33495]</p>\n<p>Victor, are you following the procedure of sampling test queries which is described <a href=\"https://www.kaggle.com/c/yandex-personalized-web-search-challenge/data\">here</a>?</p>\n<p>[/quote]</p>\n<p>Yes, but with the following differencies.</p>\n<p>1) I took the all 30 days instead of 3.</p>\n<p>2) I didn't perform random sampling, but I multiplied gains by the weights those are the probabilities of selection the query in test: 1/number_of_relevant_queries_for_the_user.</p>\n<p>3) I didn't implement: &quot;From this set of queries we filter out all queries with clicks performed at the same unit of time&quot;. I don't understand what is &quot;same&quot;: the same with the query time, or the same with any click (or with any event) in the session.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33508,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "11/10/2013 14:55:31",
      "content": "<p>I made a mistake in the program calculating a relevance, the valid score on training sample is 0.79581.</p>\n<p>After one more correction of evaluation procedure, the score is 0.796746.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 34991,
      "author_name": "vbs0711",
      "author_url": "",
      "post_date": "11/21/2013 10:40:41",
      "content": "<p>seems you did not submit this one.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 34992,
      "author_name": "",
      "author_url": "",
      "post_date": "11/21/2013 10:53:44",
      "content": "<p>@Victor<br>I get 0.79838393939157726 on the last week of the training set :S<br>I used to be much closer when working with the last three days.</p>\n<p>@vbs 0711<br>Victor is trying to compute a &nbsp;score on the training set, not the test set.<br><br></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "33226": "",
    "33277": "",
    "33364": "",
    "33479": "",
    "33488": "",
    "33494": "",
    "33495": "",
    "33498": "",
    "33508": "",
    "34991": "",
    "34992": ""
  },
  "source": "meta"
}