{
  "id": 52602,
  "title": "How to sort from command line?",
  "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/52602",
  "author_name": "",
  "post_date": "2018-03-21T12:18:13.993844500Z",
  "votes": 2,
  "comment_count": 12,
  "views": 0,
  "content": "<p>Question for someone good at command line:  what would be the command to sort train.csv file by ip, then by date and output to new cvs?  I tried some combos I found online, but time stamps aren't sorting properly.  </p>\n\n<p>(Mac OS)</p>\n\n<p>EDITED TO ADD:  I tried this one: <code>sort -k6 -n -t, train.csv  &gt;&gt; sorted.csv</code> but the dates didn't sort properly.  So I'm guessing I did something wrong in here.  I think it must be the 'n' argument, cause it's just saying sort numerically, but not that it's a date...</p>",
  "messages": [
    {
      "id": "300206",
      "postDate": "03/21/2018 12:18:13",
      "content": "<p>Question for someone good at command line:  what would be the command to sort train.csv file by ip, then by date and output to new cvs?  I tried some combos I found online, but time stamps aren't sorting properly.  </p>\n\n<p>(Mac OS)</p>\n\n<p>EDITED TO ADD:  I tried this one: <code>sort -k6 -n -t, train.csv  &gt;&gt; sorted.csv</code> but the dates didn't sort properly.  So I'm guessing I did something wrong in here.  I think it must be the 'n' argument, cause it's just saying sort numerically, but not that it's a date...</p>",
      "rawMarkdown": "Question for someone good at command line:  what would be the command to sort train.csv file by ip, then by date and output to new cvs?  I tried some combos I found online, but time stamps aren't sorting properly.  \n\n(Mac OS)\n\nEDITED TO ADD:  I tried this one: `sort -k6 -n -t, train.csv  &gt;&gt; sorted.csv  ` but the dates didn't sort properly.  So I'm guessing I did something wrong in here.  I think it must be the 'n' argument, cause it's just saying sort numerically, but not that it's a date...",
      "votes": null
    },
    {
      "id": "300225",
      "postDate": "03/21/2018 12:41:02",
      "content": "<p>Why do you need to sort from <code>command line</code>? \nTasks like sorting is too difficult for command line. Maybe anyone can implement it, please let me know.</p>\n\n<p>It's quiet easier to sort from python. </p>",
      "rawMarkdown": "Why do you need to sort from `command line`? \nTasks like sorting is too difficult for command line. Maybe anyone can implement it, please let me know.\n\nIt's quiet easier to sort from python.",
      "votes": null
    },
    {
      "id": "300282",
      "postDate": "03/21/2018 13:35:57",
      "content": "<p>If you have 16gb ram, I wouldn't bother with command line - data.table in R handles this problem quite well.</p>",
      "rawMarkdown": "If you have 16gb ram, I wouldn't bother with command line - data.table in R handles this problem quite well.",
      "votes": null
    },
    {
      "id": "300299",
      "postDate": "03/21/2018 13:52:47",
      "content": "<p>if there is issue with memory you can try to use this package specifically designed to work with large pandas DataFrame and bumpy array </p>\n\n<p><a href=\"http://blaze.readthedocs.io/en/latest/ooc.html\">http://blaze.readthedocs.io/en/latest/ooc.html</a></p>",
      "rawMarkdown": "if there is issue with memory you can try to use this package specifically designed to work with large pandas DataFrame and bumpy array \n\nhttp://blaze.readthedocs.io/en/latest/ooc.html",
      "votes": null
    },
    {
      "id": "300472",
      "postDate": "03/21/2018 16:14:48",
      "content": "<p>I don't have R.  It's all python here</p>",
      "rawMarkdown": "I don't have R.  It's all python here",
      "votes": null
    },
    {
      "id": "300476",
      "postDate": "03/21/2018 16:16:43",
      "content": "<p>Python struggling to load and sort same thing.  Too big.  I managed to sort (inaccurately) from command line.  So I thought it could work better, but don't know how to sort dates</p>",
      "rawMarkdown": "Python struggling to load and sort same thing.  Too big.  I managed to sort (inaccurately) from command line.  So I thought it could work better, but don't know how to sort dates",
      "votes": null
    },
    {
      "id": "300481",
      "postDate": "03/21/2018 16:21:52",
      "content": "<p>Thank you, I'm going to try that.</p>",
      "rawMarkdown": "Thank you, I'm going to try that.",
      "votes": null
    },
    {
      "id": "300488",
      "postDate": "03/21/2018 16:29:21",
      "content": "<p>if this will fail you might want to save file in hdf5 format and work with them. <a href=\"https://www.uetke.com/blog/python/how-to-use-hdf5-files-in-python/\"> Here is very nice guide </a></p>\n\n<p>Good luck =)</p>",
      "rawMarkdown": "if this will fail you might want to save file in hdf5 format and work with them. [ Here is very nice guide ][1]\n\n\n  [1]: https://www.uetke.com/blog/python/how-to-use-hdf5-files-in-python/. \n\nGood luck =)",
      "votes": null
    },
    {
      "id": "300519",
      "postDate": "03/21/2018 17:07:22",
      "content": "<p>Did you try <a href=\"https://stackoverflow.com/questions/28161356/sort-pandas-dataframe-by-date\">this</a>?</p>",
      "rawMarkdown": "Did you try [this][1]?\n\n\n  [1]: https://stackoverflow.com/questions/28161356/sort-pandas-dataframe-by-date",
      "votes": null
    },
    {
      "id": "300534",
      "postDate": "03/21/2018 17:26:18",
      "content": "<p>Have you tried via pandas in Python? Sorting on data frames is quite effective in my opinion.</p>",
      "rawMarkdown": "Have you tried via pandas in Python? Sorting on data frames is quite effective in my opinion.",
      "votes": null
    },
    {
      "id": "300627",
      "postDate": "03/21/2018 19:24:22",
      "content": "<p>For pure pandas it's too big.  Trying with dask.  Wanted to try command line.  It's been very good for file slicing.  </p>",
      "rawMarkdown": "For pure pandas it's too big.  Trying with dask.  Wanted to try command line.  It's been very good for file slicing.",
      "votes": null
    },
    {
      "id": "300642",
      "postDate": "03/21/2018 19:42:23",
      "content": "<p>Thanks.  I'll read it.   I think in this competition I'm spending more time figuring out memory management than actual analysis or modelling...</p>",
      "rawMarkdown": "Thanks.  I'll read it.   I think in this competition I'm spending more time figuring out memory management than actual analysis or modelling...",
      "votes": null
    },
    {
      "id": "301189",
      "postDate": "03/22/2018 13:09:35",
      "content": "<p>@yulia if you set up swap space on your computer it might help you to do it using pandas as its like boosting the RAM on your machine. </p>",
      "rawMarkdown": "yulia if you set up swap space on your computer it might help you to do it using pandas as its like boosting the RAM on your machine.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 300225,
      "author_name": "liujilong",
      "author_url": "",
      "post_date": "03/21/2018 12:41:02",
      "content": "<p>Why do you need to sort from <code>command line</code>? \nTasks like sorting is too difficult for command line. Maybe anyone can implement it, please let me know.</p>\n\n<p>It's quiet easier to sort from python. </p>",
      "votes": null,
      "replies": [
        {
          "id": 300476,
          "author_name": "yuliagm",
          "author_url": "",
          "post_date": "03/21/2018 16:16:43",
          "content": "<p>Python struggling to load and sort same thing.  Too big.  I managed to sort (inaccurately) from command line.  So I thought it could work better, but don't know how to sort dates</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 300519,
          "author_name": "sohaibomar",
          "author_url": "",
          "post_date": "03/21/2018 17:07:22",
          "content": "<p>Did you try <a href=\"https://stackoverflow.com/questions/28161356/sort-pandas-dataframe-by-date\">this</a>?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 300282,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "03/21/2018 13:35:57",
      "content": "<p>If you have 16gb ram, I wouldn't bother with command line - data.table in R handles this problem quite well.</p>",
      "votes": null,
      "replies": [
        {
          "id": 300472,
          "author_name": "yuliagm",
          "author_url": "",
          "post_date": "03/21/2018 16:14:48",
          "content": "<p>I don't have R.  It's all python here</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 300534,
          "author_name": "konradb",
          "author_url": "",
          "post_date": "03/21/2018 17:26:18",
          "content": "<p>Have you tried via pandas in Python? Sorting on data frames is quite effective in my opinion.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 300627,
          "author_name": "yuliagm",
          "author_url": "",
          "post_date": "03/21/2018 19:24:22",
          "content": "<p>For pure pandas it's too big.  Trying with dask.  Wanted to try command line.  It's been very good for file slicing.  </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 301189,
          "author_name": "garlsham",
          "author_url": "",
          "post_date": "03/22/2018 13:09:35",
          "content": "<p>@yulia if you set up swap space on your computer it might help you to do it using pandas as its like boosting the RAM on your machine. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 300299,
      "author_name": "drhabib",
      "author_url": "",
      "post_date": "03/21/2018 13:52:47",
      "content": "<p>if there is issue with memory you can try to use this package specifically designed to work with large pandas DataFrame and bumpy array </p>\n\n<p><a href=\"http://blaze.readthedocs.io/en/latest/ooc.html\">http://blaze.readthedocs.io/en/latest/ooc.html</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 300481,
          "author_name": "yuliagm",
          "author_url": "",
          "post_date": "03/21/2018 16:21:52",
          "content": "<p>Thank you, I'm going to try that.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 300488,
          "author_name": "drhabib",
          "author_url": "",
          "post_date": "03/21/2018 16:29:21",
          "content": "<p>if this will fail you might want to save file in hdf5 format and work with them. <a href=\"https://www.uetke.com/blog/python/how-to-use-hdf5-files-in-python/\"> Here is very nice guide </a></p>\n\n<p>Good luck =)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 300642,
          "author_name": "yuliagm",
          "author_url": "",
          "post_date": "03/21/2018 19:42:23",
          "content": "<p>Thanks.  I'll read it.   I think in this competition I'm spending more time figuring out memory management than actual analysis or modelling...</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "300206": "Question for someone good at command line:  what would be the command to sort train.csv file by ip, then by date and output to new cvs?  I tried some combos I found online, but time stamps aren't sorting properly.  \n\n(Mac OS)\n\nEDITED TO ADD:  I tried this one: `sort -k6 -n -t, train.csv  &gt;&gt; sorted.csv  ` but the dates didn't sort properly.  So I'm guessing I did something wrong in here.  I think it must be the 'n' argument, cause it's just saying sort numerically, but not that it's a date...",
    "300225": "Why do you need to sort from `command line`? \nTasks like sorting is too difficult for command line. Maybe anyone can implement it, please let me know.\n\nIt's quiet easier to sort from python.",
    "300282": "If you have 16gb ram, I wouldn't bother with command line - data.table in R handles this problem quite well.",
    "300299": "if there is issue with memory you can try to use this package specifically designed to work with large pandas DataFrame and bumpy array \n\nhttp://blaze.readthedocs.io/en/latest/ooc.html",
    "300472": "I don't have R.  It's all python here",
    "300476": "Python struggling to load and sort same thing.  Too big.  I managed to sort (inaccurately) from command line.  So I thought it could work better, but don't know how to sort dates",
    "300481": "Thank you, I'm going to try that.",
    "300488": "if this will fail you might want to save file in hdf5 format and work with them. [ Here is very nice guide ][1]\n\n\n  [1]: https://www.uetke.com/blog/python/how-to-use-hdf5-files-in-python/. \n\nGood luck =)",
    "300519": "Did you try [this][1]?\n\n\n  [1]: https://stackoverflow.com/questions/28161356/sort-pandas-dataframe-by-date",
    "300534": "Have you tried via pandas in Python? Sorting on data frames is quite effective in my opinion.",
    "300627": "For pure pandas it's too big.  Trying with dask.  Wanted to try command line.  It's been very good for file slicing.",
    "300642": "Thanks.  I'll read it.   I think in this competition I'm spending more time figuring out memory management than actual analysis or modelling...",
    "301189": "yulia if you set up swap space on your computer it might help you to do it using pandas as its like boosting the RAM on your machine."
  },
  "source": "meta"
}