{
  "id": 21233,
  "title": "To Admin - what all duplicates in generationMethod 2",
  "url": "/competitions/avito-duplicate-ads-detection/discussion/21233",
  "author_name": "",
  "post_date": "2016-05-26T06:15:19.147Z",
  "votes": null,
  "comment_count": 6,
  "views": 894,
  "content": "<p>Hi</p>\n\n<p>I posted this query in the Welcome thread. Did not get a reply hence creating a new thread.\nCan you please explain why all are duplicates for generationMethod 2 and are all the test pairs with generationMethod 2 (I know that is not provided but still) duplicate too ?</p>\n\n<p>Regds</p>\n\n<p>And there was a typo on the subject - cannot change it. Can some one with privilege please change the what to why \n&quot;why all duplicates in generationMethod 2&quot;</p>",
  "messages": [
    {
      "id": "121409",
      "postDate": "05/26/2016 06:15:19",
      "content": "<p>Hi</p>\n\n<p>I posted this query in the Welcome thread. Did not get a reply hence creating a new thread.\nCan you please explain why all are duplicates for generationMethod 2 and are all the test pairs with generationMethod 2 (I know that is not provided but still) duplicate too ?</p>\n\n<p>Regds</p>\n\n<p>And there was a typo on the subject - cannot change it. Can some one with privilege please change the what to why \n&quot;why all duplicates in generationMethod 2&quot;</p>",
      "rawMarkdown": "Hi\r\n\r\nI posted this query in the Welcome thread. Did not get a reply hence creating a new thread.\r\nCan you please explain why all are duplicates for generationMethod 2 and are all the test pairs with generationMethod 2 (I know that is not provided but still) duplicate too ?\r\n\r\nRegds\r\n\r\nAnd there was a typo on the subject - cannot change it. Can some one with privilege please change the what to why \r\n\"why all duplicates in generationMethod 2\"",
      "votes": null
    },
    {
      "id": "121411",
      "postDate": "05/26/2016 06:54:45",
      "content": "<p>That's just the property of generation method 2. It selects sure duplicates. Don't have anything else to add, sorry.</p>",
      "rawMarkdown": "That's just the property of generation method 2. It selects sure duplicates. Don't have anything else to add, sorry.",
      "votes": null
    },
    {
      "id": "121412",
      "postDate": "05/26/2016 06:57:33",
      "content": "<p>Ok - thanks. That's all right. I mentioned because you said somewhere we might want to use/weigh training per generationMethod. At least that will not work for generationMethod 2. </p>\n\n<p>Regds</p>",
      "rawMarkdown": "Ok - thanks. That's all right. I mentioned because you said somewhere we might want to use/weigh training per generationMethod. At least that will not work for generationMethod 2. \r\n\r\nRegds",
      "votes": null
    },
    {
      "id": "121851",
      "postDate": "05/30/2016 10:51:21",
      "content": "<p>More interestingly, can you tell us whether P(generationMethod | train) == P(generationMethod | test)? And P(duplicate | train) == P(duplicate | test)?</p>\n\n<p>In other words, whether train and evaluation are <em>randomly</em> sampled from the same distribution?</p>",
      "rawMarkdown": "More interestingly, can you tell us whether P(generationMethod | train) == P(generationMethod | test)? And P(duplicate | train) == P(duplicate | test)?\r\n\r\nIn other words, whether train and evaluation are *randomly* sampled from the same distribution?",
      "votes": null
    },
    {
      "id": "121853",
      "postDate": "05/30/2016 11:23:33",
      "content": "<p>[quote=Ricardo Cruz;121851]</p>\n\n<p>In other words, whether train and evaluation are <em>randomly</em> sampled from the same distribution?</p>\n\n<p>[/quote]</p>\n\n<p>I can't answer about probabilities, but the training and evaluation data are not randomly split, rather there they are sampled at different points in time. So you will find that the items in the evaluation set are different from those in training.</p>",
      "rawMarkdown": "[quote=Ricardo Cruz;121851]\r\n\r\nIn other words, whether train and evaluation are *randomly* sampled from the same distribution?\r\n\r\n[/quote]\r\n\r\nI can't answer about probabilities, but the training and evaluation data are not randomly split, rather there they are sampled at different points in time. So you will find that the items in the evaluation set are different from those in training.",
      "votes": null
    },
    {
      "id": "121858",
      "postDate": "05/30/2016 12:07:14",
      "content": "<p>I understand the items are different, but do you know if the duplication &quot;generation methods&quot; were employed in the same proportions in both training and evaluation?</p>",
      "rawMarkdown": "I understand the items are different, but do you know if the duplication \"generation methods\" were employed in the same proportions in both training and evaluation?",
      "votes": null
    },
    {
      "id": "122030",
      "postDate": "05/31/2016 20:25:06",
      "content": "<p>Yes, proportions are the same.</p>",
      "rawMarkdown": "Yes, proportions are the same.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 121411,
      "author_name": "ivanguz",
      "author_url": "",
      "post_date": "05/26/2016 06:54:45",
      "content": "<p>That's just the property of generation method 2. It selects sure duplicates. Don't have anything else to add, sorry.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121412,
      "author_name": "rightfit",
      "author_url": "",
      "post_date": "05/26/2016 06:57:33",
      "content": "<p>Ok - thanks. That's all right. I mentioned because you said somewhere we might want to use/weigh training per generationMethod. At least that will not work for generationMethod 2. </p>\n\n<p>Regds</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121851,
      "author_name": "rpmcruz",
      "author_url": "",
      "post_date": "05/30/2016 10:51:21",
      "content": "<p>More interestingly, can you tell us whether P(generationMethod | train) == P(generationMethod | test)? And P(duplicate | train) == P(duplicate | test)?</p>\n\n<p>In other words, whether train and evaluation are <em>randomly</em> sampled from the same distribution?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121853,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "05/30/2016 11:23:33",
      "content": "<p>[quote=Ricardo Cruz;121851]</p>\n\n<p>In other words, whether train and evaluation are <em>randomly</em> sampled from the same distribution?</p>\n\n<p>[/quote]</p>\n\n<p>I can't answer about probabilities, but the training and evaluation data are not randomly split, rather there they are sampled at different points in time. So you will find that the items in the evaluation set are different from those in training.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121858,
      "author_name": "rpmcruz",
      "author_url": "",
      "post_date": "05/30/2016 12:07:14",
      "content": "<p>I understand the items are different, but do you know if the duplication &quot;generation methods&quot; were employed in the same proportions in both training and evaluation?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 122030,
      "author_name": "ivanguz",
      "author_url": "",
      "post_date": "05/31/2016 20:25:06",
      "content": "<p>Yes, proportions are the same.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "121409": "Hi\r\n\r\nI posted this query in the Welcome thread. Did not get a reply hence creating a new thread.\r\nCan you please explain why all are duplicates for generationMethod 2 and are all the test pairs with generationMethod 2 (I know that is not provided but still) duplicate too ?\r\n\r\nRegds\r\n\r\nAnd there was a typo on the subject - cannot change it. Can some one with privilege please change the what to why \r\n\"why all duplicates in generationMethod 2\"",
    "121411": "That's just the property of generation method 2. It selects sure duplicates. Don't have anything else to add, sorry.",
    "121412": "Ok - thanks. That's all right. I mentioned because you said somewhere we might want to use/weigh training per generationMethod. At least that will not work for generationMethod 2. \r\n\r\nRegds",
    "121851": "More interestingly, can you tell us whether P(generationMethod | train) == P(generationMethod | test)? And P(duplicate | train) == P(duplicate | test)?\r\n\r\nIn other words, whether train and evaluation are *randomly* sampled from the same distribution?",
    "121853": "[quote=Ricardo Cruz;121851]\r\n\r\nIn other words, whether train and evaluation are *randomly* sampled from the same distribution?\r\n\r\n[/quote]\r\n\r\nI can't answer about probabilities, but the training and evaluation data are not randomly split, rather there they are sampled at different points in time. So you will find that the items in the evaluation set are different from those in training.",
    "121858": "I understand the items are different, but do you know if the duplication \"generation methods\" were employed in the same proportions in both training and evaluation?",
    "122030": "Yes, proportions are the same."
  },
  "source": "meta"
}