{
  "id": 14980,
  "title": "PERL Script: Add Ads to SearchStream with low memory",
  "url": "/competitions/avito-context-ad-clicks/discussion/14980",
  "author_name": "Bluefool",
  "post_date": "2015-07-01T17:44:03.597000",
  "votes": 1,
  "comment_count": 0,
  "views": 460,
  "content": "<p>Takes 30 minutes to run on a 16 GB Mac.</p>\n<p>I had trouble merging and joining in R because of memory constraints. I was even having trouble with a 25% sample.</p>\n<p>This script stores category and ad into hashtables, then reads the searchstream files line by line and joins the relevant fields. As I'm only using Contextual Ads, Location is ignored and only Contextual Ads are written out. This can be changed by removing thisline[3]==3 condition.</p>\n<p>I have pretty much used the same code to also add in the user and search info - just edit the column numbers (Perl indexes from 0)</p>\n<p>I have included the add_user file too but it takes hours. It can only be run AFTER add_ads.pl as the output of one is the input to the other</p>",
  "messages": [
    {
      "id": 83111,
      "postDate": "2015-07-01T17:44:03.597Z",
      "content": "<p>Takes 30 minutes to run on a 16 GB Mac.</p>\n<p>I had trouble merging and joining in R because of memory constraints. I was even having trouble with a 25% sample.</p>\n<p>This script stores category and ad into hashtables, then reads the searchstream files line by line and joins the relevant fields. As I'm only using Contextual Ads, Location is ignored and only Contextual Ads are written out. This can be changed by removing thisline[3]==3 condition.</p>\n<p>I have pretty much used the same code to also add in the user and search info - just edit the column numbers (Perl indexes from 0)</p>\n<p>I have included the add_user file too but it takes hours. It can only be run AFTER add_ads.pl as the output of one is the input to the other</p>",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "83111": ""
  }
}