{
  "id": 51347,
  "title": "Shrink the data from command line",
  "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/51347",
  "author_name": "",
  "post_date": "2018-03-07T22:24:23.343514700Z",
  "votes": 76,
  "comment_count": 14,
  "views": 0,
  "content": "<p>Since all the 'action' we are analysing happens within three days , the information on year and month is not important =&gt; we can safely get rid of it to shrink the files. Below a fairly compact way to do it from command line </p>\n\n<pre><code>sed  s/2017-11-//g &lt; train.csv &gt; train_reduced.csv\n</code></pre>\n\n<p>It does take a while to execute, but the size should drop by ~ 15-20 pct (based on preliminary result for train_sample.csv)</p>",
  "messages": [
    {
      "id": "292400",
      "postDate": "03/07/2018 22:24:23",
      "content": "<p>Since all the 'action' we are analysing happens within three days , the information on year and month is not important =&gt; we can safely get rid of it to shrink the files. Below a fairly compact way to do it from command line </p>\n\n<pre><code>sed  s/2017-11-//g &lt; train.csv &gt; train_reduced.csv\n</code></pre>\n\n<p>It does take a while to execute, but the size should drop by ~ 15-20 pct (based on preliminary result for train_sample.csv)</p>",
      "rawMarkdown": "Since all the 'action' we are analysing happens within three days , the information on year and month is not important =&gt; we can safely get rid of it to shrink the files. Below a fairly compact way to do it from command line \n\n    sed  s/2017-11-//g &lt; train.csv &gt; train_reduced.csv\n\n\nIt does take a while to execute, but the size should drop by ~ 15-20 pct (based on preliminary result for train_sample.csv)",
      "votes": null
    },
    {
      "id": "292421",
      "postDate": "03/07/2018 23:48:02",
      "content": "<p>Good catch, thanks!</p>",
      "rawMarkdown": "Good catch, thanks!",
      "votes": null
    },
    {
      "id": "292762",
      "postDate": "03/08/2018 15:25:24",
      "content": "<p>Great idea! I would suggest that one goes even further and removes everything except the hour. You can still spot multiple clicks from one IP in \"unrealistic\" time (say, more than 5 within an hour). Given that the training data is from only two days and that the test is from a different, single day, it should have a very minor effect on the prediction performance</p>",
      "rawMarkdown": "Great idea! I would suggest that one goes even further and removes everything except the hour. You can still spot multiple clicks from one IP in \"unrealistic\" time (say, more than 5 within an hour). Given that the training data is from only two days and that the test is from a different, single day, it should have a very minor effect on the prediction performance",
      "votes": null
    },
    {
      "id": "292764",
      "postDate": "03/08/2018 15:27:41",
      "content": "<p>Indeed - I was rather happy with the quick way to shrink by 20pct and did not follow through on the idea. Well spotted.</p>",
      "rawMarkdown": "Indeed - I was rather happy with the quick way to shrink by 20pct and did not follow through on the idea. Well spotted.",
      "votes": null
    },
    {
      "id": "293518",
      "postDate": "03/10/2018 03:13:33",
      "content": "<p>great idea, thanks</p>",
      "rawMarkdown": "great idea, thanks",
      "votes": null
    },
    {
      "id": "294682",
      "postDate": "03/12/2018 12:20:13",
      "content": "<p>Great!</p>",
      "rawMarkdown": "Great!",
      "votes": null
    },
    {
      "id": "294863",
      "postDate": "03/12/2018 18:30:44",
      "content": "<p>@Konrad</p>\n\n<p>And what about test data ? </p>\n\n<p>Are you sure that we get the same distribution ? Allways doubtful about loosing data...</p>\n\n<p>I may be wrong.</p>",
      "rawMarkdown": "Konrad\n\nAnd what about test data ? \n\nAre you sure that we get the same distribution ? Allways doubtful about loosing data...\n\nI may be wrong.",
      "votes": null
    },
    {
      "id": "294954",
      "postDate": "03/12/2018 21:56:19",
      "content": "<p>Pretty sure - I checked manually in R and the range is  2017-11-10 04:00:00 -&gt; 2017-11-10 15:00:00, so we can kill year and month as well.</p>",
      "rawMarkdown": "Pretty sure - I checked manually in R and the range is  2017-11-10 04:00:00 -&gt; 2017-11-10 15:00:00, so we can kill year and month as well.",
      "votes": null
    },
    {
      "id": "295150",
      "postDate": "03/13/2018 06:03:53",
      "content": "<p>sed is a Unix utility.. any way to do it in windows OS ?</p>",
      "rawMarkdown": "sed is a Unix utility.. any way to do it in windows OS ?",
      "votes": null
    },
    {
      "id": "295154",
      "postDate": "03/13/2018 06:11:21",
      "content": "<p><a href=\"https://www.datascienceatthecommandline.com/\">https://www.datascienceatthecommandline.com</a></p>",
      "rawMarkdown": "https://www.datascienceatthecommandline.com",
      "votes": null
    },
    {
      "id": "295155",
      "postDate": "03/13/2018 06:12:55",
      "content": "<p>thanks..i will check </p>",
      "rawMarkdown": "thanks..i will check",
      "votes": null
    },
    {
      "id": "295204",
      "postDate": "03/13/2018 08:51:15",
      "content": "<p>You could also download <strong>Git Bash</strong> for Windows and <code>sed</code> would work just fine.</p>",
      "rawMarkdown": "You could also download **Git Bash** for Windows and `sed` would work just fine.",
      "votes": null
    },
    {
      "id": "301726",
      "postDate": "03/23/2018 06:52:34",
      "content": "<p>BusyBox is a superb stripped-down free distribution of 300 Linux command-line tools, including sed and many Gnu tools. There are binary downloads available for Windows.</p>",
      "rawMarkdown": "BusyBox is a superb stripped-down free distribution of 300 Linux command-line tools, including sed and many Gnu tools. There are binary downloads available for Windows.",
      "votes": null
    },
    {
      "id": "321443",
      "postDate": "05/01/2018 09:53:43",
      "content": "<p>Hi Prince, did you have any luck with editing the data as stated above with windows? </p>",
      "rawMarkdown": "Hi Prince, did you have any luck with editing the data as stated above with windows?",
      "votes": null
    },
    {
      "id": "321450",
      "postDate": "05/01/2018 10:15:59",
      "content": "<p>Hi Karthik, I am so new to git and sed and editing files via cmd. I tried to download GitBash and put the train_sample.csv in the C:\\Temp folder and ran the following command:\nsed  's/2017-11-//g' C:\\Temp\\train_sample.csv train_reduced.csv</p>\n\n<p>The output I got is:\nsed: can't read C:Temptrain_sample.csv: No such file or directory\nsed: can't read train_reduced.csv: No such file or directory</p>\n\n<p>Could you please help me here on what am I doing wrong?</p>",
      "rawMarkdown": "Hi Karthik, I am so new to git and sed and editing files via cmd. I tried to download GitBash and put the train_sample.csv in the C:\\Temp folder and ran the following command:\nsed  's/2017-11-//g' C:\\Temp\\train_sample.csv train_reduced.csv\n\nThe output I got is:\nsed: can't read C:Temptrain_sample.csv: No such file or directory\nsed: can't read train_reduced.csv: No such file or directory\n\nCould you please help me here on what am I doing wrong?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 292421,
      "author_name": "tobycheese",
      "author_url": "",
      "post_date": "03/07/2018 23:48:02",
      "content": "<p>Good catch, thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 292762,
      "author_name": "asparuhhristov",
      "author_url": "",
      "post_date": "03/08/2018 15:25:24",
      "content": "<p>Great idea! I would suggest that one goes even further and removes everything except the hour. You can still spot multiple clicks from one IP in \"unrealistic\" time (say, more than 5 within an hour). Given that the training data is from only two days and that the test is from a different, single day, it should have a very minor effect on the prediction performance</p>",
      "votes": null,
      "replies": [
        {
          "id": 292764,
          "author_name": "konradb",
          "author_url": "",
          "post_date": "03/08/2018 15:27:41",
          "content": "<p>Indeed - I was rather happy with the quick way to shrink by 20pct and did not follow through on the idea. Well spotted.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 293518,
      "author_name": "masaladata",
      "author_url": "",
      "post_date": "03/10/2018 03:13:33",
      "content": "<p>great idea, thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 294682,
      "author_name": "wyllian",
      "author_url": "",
      "post_date": "03/12/2018 12:20:13",
      "content": "<p>Great!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 294863,
      "author_name": "pfichou",
      "author_url": "",
      "post_date": "03/12/2018 18:30:44",
      "content": "<p>@Konrad</p>\n\n<p>And what about test data ? </p>\n\n<p>Are you sure that we get the same distribution ? Allways doubtful about loosing data...</p>\n\n<p>I may be wrong.</p>",
      "votes": null,
      "replies": [
        {
          "id": 294954,
          "author_name": "konradb",
          "author_url": "",
          "post_date": "03/12/2018 21:56:19",
          "content": "<p>Pretty sure - I checked manually in R and the range is  2017-11-10 04:00:00 -&gt; 2017-11-10 15:00:00, so we can kill year and month as well.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 295150,
      "author_name": "princeatul",
      "author_url": "",
      "post_date": "03/13/2018 06:03:53",
      "content": "<p>sed is a Unix utility.. any way to do it in windows OS ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 295154,
          "author_name": "konradb",
          "author_url": "",
          "post_date": "03/13/2018 06:11:21",
          "content": "<p><a href=\"https://www.datascienceatthecommandline.com/\">https://www.datascienceatthecommandline.com</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 295155,
          "author_name": "princeatul",
          "author_url": "",
          "post_date": "03/13/2018 06:12:55",
          "content": "<p>thanks..i will check </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 295204,
          "author_name": "kartikkannapur",
          "author_url": "",
          "post_date": "03/13/2018 08:51:15",
          "content": "<p>You could also download <strong>Git Bash</strong> for Windows and <code>sed</code> would work just fine.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 301726,
          "author_name": "smcinerney",
          "author_url": "",
          "post_date": "03/23/2018 06:52:34",
          "content": "<p>BusyBox is a superb stripped-down free distribution of 300 Linux command-line tools, including sed and many Gnu tools. There are binary downloads available for Windows.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 321443,
          "author_name": "vanaml",
          "author_url": "",
          "post_date": "05/01/2018 09:53:43",
          "content": "<p>Hi Prince, did you have any luck with editing the data as stated above with windows? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 321450,
          "author_name": "vanaml",
          "author_url": "",
          "post_date": "05/01/2018 10:15:59",
          "content": "<p>Hi Karthik, I am so new to git and sed and editing files via cmd. I tried to download GitBash and put the train_sample.csv in the C:\\Temp folder and ran the following command:\nsed  's/2017-11-//g' C:\\Temp\\train_sample.csv train_reduced.csv</p>\n\n<p>The output I got is:\nsed: can't read C:Temptrain_sample.csv: No such file or directory\nsed: can't read train_reduced.csv: No such file or directory</p>\n\n<p>Could you please help me here on what am I doing wrong?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "292400": "Since all the 'action' we are analysing happens within three days , the information on year and month is not important =&gt; we can safely get rid of it to shrink the files. Below a fairly compact way to do it from command line \n\n    sed  s/2017-11-//g &lt; train.csv &gt; train_reduced.csv\n\n\nIt does take a while to execute, but the size should drop by ~ 15-20 pct (based on preliminary result for train_sample.csv)",
    "292421": "Good catch, thanks!",
    "292762": "Great idea! I would suggest that one goes even further and removes everything except the hour. You can still spot multiple clicks from one IP in \"unrealistic\" time (say, more than 5 within an hour). Given that the training data is from only two days and that the test is from a different, single day, it should have a very minor effect on the prediction performance",
    "292764": "Indeed - I was rather happy with the quick way to shrink by 20pct and did not follow through on the idea. Well spotted.",
    "293518": "great idea, thanks",
    "294682": "Great!",
    "294863": "Konrad\n\nAnd what about test data ? \n\nAre you sure that we get the same distribution ? Allways doubtful about loosing data...\n\nI may be wrong.",
    "294954": "Pretty sure - I checked manually in R and the range is  2017-11-10 04:00:00 -&gt; 2017-11-10 15:00:00, so we can kill year and month as well.",
    "295150": "sed is a Unix utility.. any way to do it in windows OS ?",
    "295154": "https://www.datascienceatthecommandline.com",
    "295155": "thanks..i will check",
    "295204": "You could also download **Git Bash** for Windows and `sed` would work just fine.",
    "301726": "BusyBox is a superb stripped-down free distribution of 300 Linux command-line tools, including sed and many Gnu tools. There are binary downloads available for Windows.",
    "321443": "Hi Prince, did you have any luck with editing the data as stated above with windows?",
    "321450": "Hi Karthik, I am so new to git and sed and editing files via cmd. I tried to download GitBash and put the train_sample.csv in the C:\\Temp folder and ran the following command:\nsed  's/2017-11-//g' C:\\Temp\\train_sample.csv train_reduced.csv\n\nThe output I got is:\nsed: can't read C:Temptrain_sample.csv: No such file or directory\nsed: can't read train_reduced.csv: No such file or directory\n\nCould you please help me here on what am I doing wrong?"
  },
  "source": "meta"
}