{
  "id": 14514,
  "title": "Slow querying",
  "url": "/competitions/avito-context-ad-clicks/discussion/14514",
  "author_name": "",
  "post_date": "2015-06-03T16:11:44.543Z",
  "votes": null,
  "comment_count": 6,
  "views": 1582,
  "content": "<p>This competition involves working with LOTS of data. The problem is joining or merging tables together. In case anyone's wondering...</p>\n\n<p><code>sqlite&gt; SELECT COUNT(*) FROM trainsearchstream;<br>392356948</code></p>\n\n<p>(That took several minutes...)</p>\n<p>Yes, that's 392 million rows on a table. You can imagine how hard it is joining tables without indexes... or creating them. Dumping the entire database to a more powerful one, like MySQL might as well take a long time too, though it seems the best option.</p>\n<p>Well I might not use all the data so... why not sample?</p>\n<p><code>&nbsp;sqlite&gt; SELECT * FROM trainsearchstream ORDER BY RANDOM() LIMIT 1000;<br><br></code></p>\n<p>I don't know how much time this can take. I think I'll end up working only with a subset of the data! Good luck guys.</p>",
  "messages": [
    {
      "id": "80781",
      "postDate": "06/03/2015 16:11:44",
      "content": "<p>This competition involves working with LOTS of data. The problem is joining or merging tables together. In case anyone's wondering...</p>\n\n<p><code>sqlite&gt; SELECT COUNT(*) FROM trainsearchstream;<br>392356948</code></p>\n\n<p>(That took several minutes...)</p>\n<p>Yes, that's 392 million rows on a table. You can imagine how hard it is joining tables without indexes... or creating them. Dumping the entire database to a more powerful one, like MySQL might as well take a long time too, though it seems the best option.</p>\n<p>Well I might not use all the data so... why not sample?</p>\n<p><code>&nbsp;sqlite&gt; SELECT * FROM trainsearchstream ORDER BY RANDOM() LIMIT 1000;<br><br></code></p>\n<p>I don't know how much time this can take. I think I'll end up working only with a subset of the data! Good luck guys.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80783",
      "postDate": "06/03/2015 16:15:21",
      "content": "<p>I wonder if it will end up an online competition with Tinrtgu's code?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80791",
      "postDate": "06/03/2015 16:41:17",
      "content": "<p>yes:&nbsp;https://www.kaggle.com/abhishek/avito-context-ad-clicks/beating-the-benchmark</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80800",
      "postDate": "06/03/2015 17:01:59",
      "content": "<p>I don't think use SQL is a good way to join those data. You may try to sort both SearchStream and other table, say,&nbsp;AdsInfo on AdID. And then merge them row by row if they got same AdID, turn to next row otherwise.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80801",
      "postDate": "06/03/2015 17:04:20",
      "content": "<p>[quote=Abhishek;80791]</p>\n<p>yes:&nbsp;https://www.kaggle.com/abhishek/avito-context-ad-clicks/beating-the-benchmark</p>\n<p>[/quote]</p>\n<p>ha I'm psychic and you're quick</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80802",
      "postDate": "06/03/2015 17:05:12",
      "content": "<p>[quote=Jose Jimenez;80781]</p>\n<p>This competition involves working with LOTS of data. The problem is joining or merging tables together. In case anyone's wondering...</p>\n<p><code>sqlite&gt; SELECT COUNT(*) FROM trainsearchstream;<br>392356948</code></p>\n<p>[/quote]</p>\n<p>For the sake of simplicity you can omit those where ObjectType &lt;&gt; 3.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "89804",
      "postDate": "08/19/2015 15:18:18",
      "content": "<p>I am having trouble importing from a single sqlite file into mysql, the import wizard unable to parse the default column type it seems. Anyone else had this issue?</p>\n\n<p>Boris</p>",
      "rawMarkdown": "I am having trouble importing from a single sqlite file into mysql, the import wizard unable to parse the default column type it seems. Anyone else had this issue?\r\n\r\nBoris",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 80783,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "06/03/2015 16:15:21",
      "content": "<p>I wonder if it will end up an online competition with Tinrtgu's code?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80791,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "06/03/2015 16:41:17",
      "content": "<p>yes:&nbsp;https://www.kaggle.com/abhishek/avito-context-ad-clicks/beating-the-benchmark</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80800,
      "author_name": "yejiming",
      "author_url": "",
      "post_date": "06/03/2015 17:01:59",
      "content": "<p>I don't think use SQL is a good way to join those data. You may try to sort both SearchStream and other table, say,&nbsp;AdsInfo on AdID. And then merge them row by row if they got same AdID, turn to next row otherwise.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80801,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "06/03/2015 17:04:20",
      "content": "<p>[quote=Abhishek;80791]</p>\n<p>yes:&nbsp;https://www.kaggle.com/abhishek/avito-context-ad-clicks/beating-the-benchmark</p>\n<p>[/quote]</p>\n<p>ha I'm psychic and you're quick</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80802,
      "author_name": "nightbat",
      "author_url": "",
      "post_date": "06/03/2015 17:05:12",
      "content": "<p>[quote=Jose Jimenez;80781]</p>\n<p>This competition involves working with LOTS of data. The problem is joining or merging tables together. In case anyone's wondering...</p>\n<p><code>sqlite&gt; SELECT COUNT(*) FROM trainsearchstream;<br>392356948</code></p>\n<p>[/quote]</p>\n<p>For the sake of simplicity you can omit those where ObjectType &lt;&gt; 3.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89804,
      "author_name": "bmizhen",
      "author_url": "",
      "post_date": "08/19/2015 15:18:18",
      "content": "<p>I am having trouble importing from a single sqlite file into mysql, the import wizard unable to parse the default column type it seems. Anyone else had this issue?</p>\n\n<p>Boris</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "80781": "",
    "80783": "",
    "80791": "",
    "80800": "",
    "80801": "",
    "80802": "",
    "89804": "I am having trouble importing from a single sqlite file into mysql, the import wizard unable to parse the default column type it seems. Anyone else had this issue?\r\n\r\nBoris"
  },
  "source": "meta"
}