{ "id": 12644, "title": "Code for parallel feature extraction without unzipping the data", "url": "/competitions/malware-classification/discussion/12644", "author_name": "", "post_date": "2015-03-01T08:46:29.827Z", "votes": 13, "comment_count": 9, "views": 3631, "content": "

Hello, I've written a sample code in Python how to traverse .7z archive in parallel and extract byte counts.

\n

You will need libarchive, compile it from source, do not use the version from ubuntu repo. And then install python wrapper for libarchive and you are done if you use Anaconda. 

\n

Hope it will help you to get to the top :)

", "messages": [ { "id": "65172", "postDate": "03/01/2015 08:46:29", "content": "

Hello, I've written a sample code in Python how to traverse .7z archive in parallel and extract byte counts.

\n

You will need libarchive, compile it from source, do not use the version from ubuntu repo. And then install python wrapper for libarchive and you are done if you use Anaconda. 

\n

Hope it will help you to get to the top :)

", "rawMarkdown": "", "votes": null }, { "id": "65598", "postDate": "03/06/2015 12:23:35", "content": "

I wonder why there is no reply here .... I am anyways going to test your code, thanks.

", "rawMarkdown": "", "votes": null }, { "id": "65600", "postDate": "03/06/2015 13:22:54", "content": "

Just one question, can you give me a clue of how long it did take you to run the code with only byte count as feature.

", "rawMarkdown": "", "votes": null }, { "id": "65671", "postDate": "03/07/2015 10:15:01", "content": "

[quote=AnotherFloyd;65600]

\n

Just one question, can you give me a clue of how long it did take you to run the code with only byte count as feature.

\n

[/quote]

\n

Have no idea to be true, I just wanted to show where you can insert your feature extractor. I have not run exactly this one for testing the performance and as long as you extract something simple unzipping the data is the bottleneck, so this code will not be 6 times faster for pool of 6 processes than without multiprocessing. But when doing regex multiprocessing helps a lot. 

", "rawMarkdown": "", "votes": null }, { "id": "68787", "postDate": "03/29/2015 00:29:46", "content": "

Hi Dmitry,

\n

Could you please let me know how to properly install lib-archive on Windows. When ever i run your code I always get this error in python shell.

\n

Traceback (most recent call last):
File "E:\\MMCL TT\\feat_extract.py", line 1, in <module>
import libarchive.public
File "C:\\Python27\\lib\\site-packages\\libarchive\\public.py", line 1, in <module>
from libarchive.adapters.archive_read import \\
File "C:\\Python27\\lib\\site-packages\\libarchive\\adapters\\archive_read.py", line 7, in <module>
import libarchive.calls.archive_read
File "C:\\Python27\\lib\\site-packages\\libarchive\\calls\\archive_read.py", line 3, in <module>
from libarchive.library import libarchive
File "C:\\Python27\\lib\\site-packages\\libarchive\\library.py", line 16, in <module>
libarchive = ctypes.cdll.LoadLibrary(_FILEPATH)
File "C:\\Python27\\lib\\ctypes\\__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "C:\\Python27\\lib\\ctypes\\__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
WindowsError: [Error 126] The specified module could not be found

\n\n

It would of great help if you can provide me a solution for this.

\n

Thanks for your code :)

\n

Kaushik

", "rawMarkdown": "", "votes": null }, { "id": "68788", "postDate": "03/29/2015 00:31:43", "content": "

The error code is truncated.. I'm attaching it in a text file

", "rawMarkdown": "", "votes": null }, { "id": "68820", "postDate": "03/29/2015 06:29:28", "content": "

[quote=Kaushik;68788]

\n

The error code is truncated.. I'm attaching it in a text file

\n

[/quote]

\n

I have no idea how to install libraries in Windows. It is much easier in linux that's why I use Ubuntu. 

\n

I assume pylibarchive cannot find libarchive dll, there's probably a path variable in python for dll search dirs, you need to append your libarchive build dir there (only a guess).

", "rawMarkdown": "", "votes": null }, { "id": "68990", "postDate": "03/30/2015 09:21:19", "content": "

Hi Dmitry, 

\n

Thank you for sharing the code. I tried to run it for train.7z but the code returns following exception: 
"ValueError: Archive iteration (read_next_header) returned error: (-30) [Damaged 7-Zip archive] "

I am pretty sure that archive is fine and is not damaged (check sum is fine).  BTW the code works fine for the sampleData.7z . Could you please give any hints on it?

", "rawMarkdown": "", "votes": null }, { "id": "69003", "postDate": "03/30/2015 12:42:59", "content": "

Hi,

\n

I faced with the same error. To be able to work with the archive without extracting all the data I developed an utility that converts 7z to ordinary zip archive with which it is possible to work with Python's zipfile module.

\n

This utility unlike other tools extracts and zips files one by one. Thus, you do not need ~200 Gb of free space on your hard drive. If you have this amount of space, it is better (faster) to use tools like "atool" to convert. 

\n

The code of the utility is here:

\n

https://github.com/zyrikby/7z2zip_converter/

", "rawMarkdown": "", "votes": null }, { "id": "69097", "postDate": "03/30/2015 22:13:53", "content": "

[quote=ITankoyeu;68990]

\n

Hi Dmitry, 

\n

Thank you for sharing the code. I tried to run it for train.7z but the code returns following exception: 
"ValueError: Archive iteration (read_next_header) returned error: (-30) [Damaged 7-Zip archive] "

I am pretty sure that archive is fine and is not damaged (check sum is fine).  BTW the code works fine for the sampleData.7z . Could you please give any hints on it?

\n

[/quote]

\n

Yep, I faced a lot of strange libarchve exceptions, and this one too. Have no solution for this one, for me restarting the terminal with ipython notebook helped, but that is just magic.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 65598, "author_name": "shubhabrata", "author_url": "", "post_date": "03/06/2015 12:23:35", "content": "

I wonder why there is no reply here .... I am anyways going to test your code, thanks.

", "votes": null, "replies": [] }, { "id": 65600, "author_name": "shubhabrata", "author_url": "", "post_date": "03/06/2015 13:22:54", "content": "

Just one question, can you give me a clue of how long it did take you to run the code with only byte count as feature.

", "votes": null, "replies": [] }, { "id": 65671, "author_name": "dulyanov", "author_url": "", "post_date": "03/07/2015 10:15:01", "content": "

[quote=AnotherFloyd;65600]

\n

Just one question, can you give me a clue of how long it did take you to run the code with only byte count as feature.

\n

[/quote]

\n

Have no idea to be true, I just wanted to show where you can insert your feature extractor. I have not run exactly this one for testing the performance and as long as you extract something simple unzipping the data is the bottleneck, so this code will not be 6 times faster for pool of 6 processes than without multiprocessing. But when doing regex multiprocessing helps a lot. 

", "votes": null, "replies": [] }, { "id": 68787, "author_name": "kaushik256147", "author_url": "", "post_date": "03/29/2015 00:29:46", "content": "

Hi Dmitry,

\n

Could you please let me know how to properly install lib-archive on Windows. When ever i run your code I always get this error in python shell.

\n

Traceback (most recent call last):
File "E:\\MMCL TT\\feat_extract.py", line 1, in <module>
import libarchive.public
File "C:\\Python27\\lib\\site-packages\\libarchive\\public.py", line 1, in <module>
from libarchive.adapters.archive_read import \\
File "C:\\Python27\\lib\\site-packages\\libarchive\\adapters\\archive_read.py", line 7, in <module>
import libarchive.calls.archive_read
File "C:\\Python27\\lib\\site-packages\\libarchive\\calls\\archive_read.py", line 3, in <module>
from libarchive.library import libarchive
File "C:\\Python27\\lib\\site-packages\\libarchive\\library.py", line 16, in <module>
libarchive = ctypes.cdll.LoadLibrary(_FILEPATH)
File "C:\\Python27\\lib\\ctypes\\__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "C:\\Python27\\lib\\ctypes\\__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
WindowsError: [Error 126] The specified module could not be found

\n\n

It would of great help if you can provide me a solution for this.

\n

Thanks for your code :)

\n

Kaushik

", "votes": null, "replies": [] }, { "id": 68788, "author_name": "kaushik256147", "author_url": "", "post_date": "03/29/2015 00:31:43", "content": "

The error code is truncated.. I'm attaching it in a text file

", "votes": null, "replies": [] }, { "id": 68820, "author_name": "dulyanov", "author_url": "", "post_date": "03/29/2015 06:29:28", "content": "

[quote=Kaushik;68788]

\n

The error code is truncated.. I'm attaching it in a text file

\n

[/quote]

\n

I have no idea how to install libraries in Windows. It is much easier in linux that's why I use Ubuntu. 

\n

I assume pylibarchive cannot find libarchive dll, there's probably a path variable in python for dll search dirs, you need to append your libarchive build dir there (only a guess).

", "votes": null, "replies": [] }, { "id": 68990, "author_name": "itankoyeu", "author_url": "", "post_date": "03/30/2015 09:21:19", "content": "

Hi Dmitry, 

\n

Thank you for sharing the code. I tried to run it for train.7z but the code returns following exception: 
"ValueError: Archive iteration (read_next_header) returned error: (-30) [Damaged 7-Zip archive] "

I am pretty sure that archive is fine and is not damaged (check sum is fine).  BTW the code works fine for the sampleData.7z . Could you please give any hints on it?

", "votes": null, "replies": [] }, { "id": 69003, "author_name": "zyrikby", "author_url": "", "post_date": "03/30/2015 12:42:59", "content": "

Hi,

\n

I faced with the same error. To be able to work with the archive without extracting all the data I developed an utility that converts 7z to ordinary zip archive with which it is possible to work with Python's zipfile module.

\n

This utility unlike other tools extracts and zips files one by one. Thus, you do not need ~200 Gb of free space on your hard drive. If you have this amount of space, it is better (faster) to use tools like "atool" to convert. 

\n

The code of the utility is here:

\n

https://github.com/zyrikby/7z2zip_converter/

", "votes": null, "replies": [] }, { "id": 69097, "author_name": "dulyanov", "author_url": "", "post_date": "03/30/2015 22:13:53", "content": "

[quote=ITankoyeu;68990]

\n

Hi Dmitry, 

\n

Thank you for sharing the code. I tried to run it for train.7z but the code returns following exception: 
"ValueError: Archive iteration (read_next_header) returned error: (-30) [Damaged 7-Zip archive] "

I am pretty sure that archive is fine and is not damaged (check sum is fine).  BTW the code works fine for the sampleData.7z . Could you please give any hints on it?

\n

[/quote]

\n

Yep, I faced a lot of strange libarchve exceptions, and this one too. Have no solution for this one, for me restarting the terminal with ipython notebook helped, but that is just magic.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "65172": "", "65598": "", "65600": "", "65671": "", "68787": "", "68788": "", "68820": "", "68990": "", "69003": "", "69097": "" }, "source": "meta" }