\n
I had previously posted some of this information in another thread, but I thought it deserved its own thread so anyone can find and also contribute to this information.
For those of you still struggling to open and process the files for this competition, here are some ideas that I used to successfully process the files:
1. You can view the first 1000 lines of the file by using the linux command:
head -n 1000 train > trainhead.txt
2. There is a lot of data in those files. Figure out which pieces of data you are going to work with (at least for your first iteration). Then write a separate parser to read the file, strip out the info that you need and write that to a separate file(s). In my first pass, I wanted global clicks per URL and personalization data based on dwell time. You can always go back, re-parse and get more data as needed, such as Domain vs URL
3. Your parser should be written in a very fast compiled language such as C, C++ OR make sure you use a language with fast JIT compiler such as Java, Pypy, Julia, etc.. (I don't know if R does JIT) My parser runs in about 10 minutes, my submission process takes about 2-3 hours (which I want to rewrite to get down to about 10 minutes).
4. In your parser, you can read the train data in session chunks. Extract what you need, save that data away, delete that session's memory and then read the next session. This cuts down tremendously on the RAM required to parse the file. You should be able to parse the file with a 4GB machine.
5. When writing out the intermediate file, you can save a lot of RAM if instead of building up the entire output file in storage, just write it out to the outFile in chunks. More specifically, write your output lines to an array, when the array length hits 1000 rows of output, write it to the file, clear the array and continue with parsing.
6. If you really want to use a database, make sure it is up to the task. The transactions need to be be completed asynchronously, you don't want your write transactions to slow down your parser.
7. You can speed things up if you have two separate (physical) hard drives. Reading and writing to the same hard drive will really slow things down. Read from one drive, write to the other if you can.
8. If your database supports bulk load try to use that. In this case, you would parse the file, generate the bulk load format and write that out to a flat file. Afterward, run the the bulk loader with this file.
9. If you plan to use SQL, make sure you are using static not dynamic SQL. (BTW its been a while so the terminology may have changed). The "access plan" for the SQL statements needs to be bound to the database before your start, at compile time. If you write dynamic SQL, then each and every Insert statement needs to go through the access planning process during runtime, therefore much slower.
10. If you can afford it an SSD will really speed things up. I initially considered upgrading my RAM, and do the whole thing in core, but I was out of memory slots which meant I needed to buy ALL new RAM. Not worth it, so I just got smarter in my parser.
Hopefully these ideas should get you started. On the other hand if what I am saying is totally foreign to you, you may not have sufficient computer science background to compete in this one.
Good Luck !
\n
I had previously posted some of this information in another thread, but I thought it deserved its own thread so anyone can find and also contribute to this information.
For those of you still struggling to open and process the files for this competition, here are some ideas that I used to successfully process the files:
1. You can view the first 1000 lines of the file by using the linux command:
head -n 1000 train > trainhead.txt
2. There is a lot of data in those files. Figure out which pieces of data you are going to work with (at least for your first iteration). Then write a separate parser to read the file, strip out the info that you need and write that to a separate file(s). In my first pass, I wanted global clicks per URL and personalization data based on dwell time. You can always go back, re-parse and get more data as needed, such as Domain vs URL
3. Your parser should be written in a very fast compiled language such as C, C++ OR make sure you use a language with fast JIT compiler such as Java, Pypy, Julia, etc.. (I don't know if R does JIT) My parser runs in about 10 minutes, my submission process takes about 2-3 hours (which I want to rewrite to get down to about 10 minutes).
4. In your parser, you can read the train data in session chunks. Extract what you need, save that data away, delete that session's memory and then read the next session. This cuts down tremendously on the RAM required to parse the file. You should be able to parse the file with a 4GB machine.
5. When writing out the intermediate file, you can save a lot of RAM if instead of building up the entire output file in storage, just write it out to the outFile in chunks. More specifically, write your output lines to an array, when the array length hits 1000 rows of output, write it to the file, clear the array and continue with parsing.
6. If you really want to use a database, make sure it is up to the task. The transactions need to be be completed asynchronously, you don't want your write transactions to slow down your parser.
7. You can speed things up if you have two separate (physical) hard drives. Reading and writing to the same hard drive will really slow things down. Read from one drive, write to the other if you can.
8. If your database supports bulk load try to use that. In this case, you would parse the file, generate the bulk load format and write that out to a flat file. Afterward, run the the bulk loader with this file.
9. If you plan to use SQL, make sure you are using static not dynamic SQL. (BTW its been a while so the terminology may have changed). The "access plan" for the SQL statements needs to be bound to the database before your start, at compile time. If you write dynamic SQL, then each and every Insert statement needs to go through the access planning process during runtime, therefore much slower.
10. If you can afford it an SSD will really speed things up. I initially considered upgrading my RAM, and do the whole thing in core, but I was out of memory slots which meant I needed to buy ALL new RAM. Not worth it, so I just got smarter in my parser.
Hopefully these ideas should get you started. On the other hand if what I am saying is totally foreign to you, you may not have sufficient computer science background to compete in this one.
Good Luck !
Here is a blog post I found about speeding up R code. It may be as simple putting all your important code in a function and compiling that function:
\nhttp://lookingatdata.blogspot.com/2012/04/speeding-up-r-computations-pt-ii.html
\n\n
Enjoy!
\n", "rawMarkdown": "", "votes": null }, { "id": "35231", "postDate": "11/25/2013 21:27:39", "content": "
11. You can spend 1-2 evenings for implement binary serialization/deserialization. I use this approach and load train file for 3 minutes instead of 18 minutes (speedup 6 times). Additionally you deal with smaller file (7.52Gb vs 15.3Gb).
\n12. Use a data properties. Note that datasets are sorted by USERID and Day fields - it would help.
\n13. If you use C# or Python look toward "yield" keyword. You can read train and test files interlaced (remember tip 12?) and save a lot of memory.
", "rawMarkdown": "", "votes": null }, { "id": "35441", "postDate": "11/28/2013 13:16:25", "content": "To read large files try reading them in chunks. Pandas.read_table has native support for chunking (setting chunk_size) which is useful if you want to chunk multi-line CSV's. If all data is structured to be on a single line (like in this Kaggle Challenge), you can also chunk with Python native code like so:
\ni = 0
with open(input_file) as f:
for next_n_lines in izip_longest(*[f] * 1000): #1k chunk size
for line in list(next_n_lines):
i += 1
If you insert data into a database like sqlite3 you can use transactions or batch commits.
\nif i % 5000000 == 0:
conn.commit()
This commits the inserts at 5 million line count intervals. You can get 50k+ linereads, line manipulation and inserts per second this way.
\nOnce you have the data in a database look at optimizing your database with indexed columns so a select query will take a few milliseconds.
\nThis will allow you to make a submission in a reasonable time frame (around an hour) on a budget laptop with limited memory.
\nI can second the OP's post that using SSD will speed-up this process a lot (by a factor 3-4) as write read is the bottleneck here.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 35209, "author_name": "geringer", "author_url": "", "post_date": "11/25/2013 18:35:25", "content": "Here is a blog post I found about speeding up R code. It may be as simple putting all your important code in a function and compiling that function:
\nhttp://lookingatdata.blogspot.com/2012/04/speeding-up-r-computations-pt-ii.html
\n\n
Enjoy!
\n", "votes": null, "replies": [] }, { "id": 35231, "author_name": "radium1", "author_url": "", "post_date": "11/25/2013 21:27:39", "content": "
11. You can spend 1-2 evenings for implement binary serialization/deserialization. I use this approach and load train file for 3 minutes instead of 18 minutes (speedup 6 times). Additionally you deal with smaller file (7.52Gb vs 15.3Gb).
\n12. Use a data properties. Note that datasets are sorted by USERID and Day fields - it would help.
\n13. If you use C# or Python look toward "yield" keyword. You can read train and test files interlaced (remember tip 12?) and save a lot of memory.
", "votes": null, "replies": [] }, { "id": 35441, "author_name": "triskelion", "author_url": "", "post_date": "11/28/2013 13:16:25", "content": "To read large files try reading them in chunks. Pandas.read_table has native support for chunking (setting chunk_size) which is useful if you want to chunk multi-line CSV's. If all data is structured to be on a single line (like in this Kaggle Challenge), you can also chunk with Python native code like so:
\ni = 0
with open(input_file) as f:
for next_n_lines in izip_longest(*[f] * 1000): #1k chunk size
for line in list(next_n_lines):
i += 1
If you insert data into a database like sqlite3 you can use transactions or batch commits.
\nif i % 5000000 == 0:
conn.commit()
This commits the inserts at 5 million line count intervals. You can get 50k+ linereads, line manipulation and inserts per second this way.
\nOnce you have the data in a database look at optimizing your database with indexed columns so a select query will take a few milliseconds.
\nThis will allow you to make a submission in a reasonable time frame (around an hour) on a budget laptop with limited memory.
\nI can second the OP's post that using SSD will speed-up this process a lot (by a factor 3-4) as write read is the bottleneck here.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "35200": "", "35209": "", "35231": "", "35441": "" }, "source": "meta" }