{ "id": 13511, "title": "Brief Description of 3rd Solution", "url": "/competitions/malware-classification/writeups/mikhail-dmitry-stanislav-brief-description-of-3rd-", "author_name": "", "post_date": "2015-04-21T14:19:33.373Z", "votes": 11, "comment_count": 5, "views": 2495, "content": "

Hi everyone, thanks for this competition! Special thanks to my teammates - you are amazing!

\n

In this post I will briefly describe my features and semi-supervised trick.

\n

The features we used consists of ~200 features from me and 800 features from Dmitry.
My features are:

\n
    \n
  1. count of lines for every sections ('.data', '.rsrc', '.text' and so on) in *.asm and entropy of this distribution. I found this kind of features are very useful!
    2. \n
  2. asm instruction count, like ‘jmp’, 'mov', … I use 30 most frequent instructions.
    3. \n
  3. sizes of *.asm and *.bytes and it’s ratio.
    4. \n
  4. sys calls (grepped by “__stdcall” from *.asm) + TF-IDF + NMF(n_components=10). I found it that TF-IDF works well and NMF performs better than PCA for dimentionallity reduction (at least for this task)
    5. \n
  5. like previous, but for functions (grepped by “FUNCTION”)
    6. \n
  6. 4-gramms. We perform feature selection by freqeuncy analisys + LinearSVC(penalty=’l1’).transform() + RF.feature_importance(). It’s allow as to select ~130 4-gramms which solely gave 0.01193/0.00958 on public/private LB using single model!
    7. \n
  7. 10-gramms. The process was similar to 4-gramms extraction and selecting, but we use hashing and some magic(see below) on last stage. Finally we use only 14 10-gramms -- and they improved our scores!
    8. \n
\n

Using this features (without 10-gramms), xgboost and some kind of bagging-magic (thanks Stanislav for this!) we got score public/private: 0.00847 / 0.00559

\n

Similarly to LittleBoat&Co's approach, we’ve done semi-supervised trick, but instead choosing label by the max probability, we sampled labels based on test predictions, and averaged several runs. This trick improves our score from 0.0055 to 0.0045 on private LB (single model)

\n

We were able to go down from 0.0045 to 0.0041 on private LB by adding 14 10-gram feats. We selected them as follows: we generated out of fold predictions for train and sorted objects by true class probability, labeled as ‘1’ top 100 worse, and run RF on 500 preselected 10-grams. We selected the most important features, and it turned out we can have 1.0 accuracy for this binary task just with 14 feats. Since the feats were naturally 10 gramms, the objects of class ‘1’ were rather different, the number of feats is 10 times less than number of objects, we wanted to think that we will not overfit using these feats (but you will in general).

\n

Dmitry will describe his approach and some mixing stuff soon.

", "messages": [ { "id": "72467", "postDate": "04/19/2015 16:48:54", "content": "

Hi everyone, thanks for this competition! Special thanks to my teammates - you are amazing!

\n

In this post I will briefly describe my features and semi-supervised trick.

\n

The features we used consists of ~200 features from me and 800 features from Dmitry.
My features are:

\n
    \n
  1. count of lines for every sections ('.data', '.rsrc', '.text' and so on) in *.asm and entropy of this distribution. I found this kind of features are very useful!
    2. \n
  2. asm instruction count, like ‘jmp’, 'mov', … I use 30 most frequent instructions.
    3. \n
  3. sizes of *.asm and *.bytes and it’s ratio.
    4. \n
  4. sys calls (grepped by “__stdcall” from *.asm) + TF-IDF + NMF(n_components=10). I found it that TF-IDF works well and NMF performs better than PCA for dimentionallity reduction (at least for this task)
    5. \n
  5. like previous, but for functions (grepped by “FUNCTION”)
    6. \n
  6. 4-gramms. We perform feature selection by freqeuncy analisys + LinearSVC(penalty=’l1’).transform() + RF.feature_importance(). It’s allow as to select ~130 4-gramms which solely gave 0.01193/0.00958 on public/private LB using single model!
    7. \n
  7. 10-gramms. The process was similar to 4-gramms extraction and selecting, but we use hashing and some magic(see below) on last stage. Finally we use only 14 10-gramms -- and they improved our scores!
    8. \n
\n

Using this features (without 10-gramms), xgboost and some kind of bagging-magic (thanks Stanislav for this!) we got score public/private: 0.00847 / 0.00559

\n

Similarly to LittleBoat&Co's approach, we’ve done semi-supervised trick, but instead choosing label by the max probability, we sampled labels based on test predictions, and averaged several runs. This trick improves our score from 0.0055 to 0.0045 on private LB (single model)

\n

We were able to go down from 0.0045 to 0.0041 on private LB by adding 14 10-gram feats. We selected them as follows: we generated out of fold predictions for train and sorted objects by true class probability, labeled as ‘1’ top 100 worse, and run RF on 500 preselected 10-grams. We selected the most important features, and it turned out we can have 1.0 accuracy for this binary task just with 14 feats. Since the feats were naturally 10 gramms, the objects of class ‘1’ were rather different, the number of feats is 10 times less than number of objects, we wanted to think that we will not overfit using these feats (but you will in general).

\n

Dmitry will describe his approach and some mixing stuff soon.

", "rawMarkdown": "", "votes": null }, { "id": "72468", "postDate": "04/19/2015 16:51:36", "content": "

Nice

", "rawMarkdown": "", "votes": null }, { "id": "72470", "postDate": "04/19/2015 16:56:52", "content": "

Hello, it was amazing competition. Thanks our tuning master Stanislav for introducing incredible training techuniques for our classifiers.

\n
    \n
  1. I extracted byte counts, bigrams, i divided byte file into 4 parts and extracted byte counts from parts. 
    2. \n
  2. From .asm file I extracted counts for asm commands (about 100), registers, section names, segments, ‘_stdcall *’ names, out types of stcalls ‘* __ stdcall’, ‘Import from *’, ‘extern *’, ‘loc_*’, ‘*.dll’ (a lot of noise). Some keywords, that I saw in files like ‘microsoft’, ‘installdir’, ‘cookie’, ‘HKEY_LOCAL_MACHINE’, ‘sp-analysis failed’ and so on. I aslo extracted reglike strings (e.g {0231-.. -1401}).
    3. \n
  3. I extracted strings, but end up not using them directly. I just computed some statistics of strings length distribution, and it helped a lot. 
    4. \n
  4. Another good features I had was entropy features -- I computed it for sliding windows over the .bytes sequence and used some statistics of its distibution (quantiles, percentiles, mean, max, std; all of these with diff(entropy)), ended up with more than 200 entropy feats. 
    5. \n
  5. Line counts, size, etc.
    6. \n
\n

I did not want to use counts from 1) and any features Mikhail used directly, so I used NMF(X), NMF (log (X+1)) (they differ a lot!). All the counts from 2) were fed into linearSVC with L1 penalty in order to select features.

\n

All of the resulting features were used in one XGBoost model, averaged over several runs.

\n

Mixing:
We found, that our classifiers were making mistakes at different classes differently, so we mixed them for every class separately. Our overall mixing scheme was extremely sophisticated and we had mix of all our temporary solutions throughout the competition in the final submission. But all we did is per-class mixing and simple linear mixing with using 2d and 3rd order interaction between predictions.

\n

We were bold enough to trim the predictions to 0 and 1 for 1,3 and 7th classes. We got lucky and got 0.0041 -> 0.0039 private score(but we had some thoughts why it is ok to do it with these classes). The second submission had no trimming, so we were expecting not to fall down too much.

", "rawMarkdown": "", "votes": null }, { "id": "72480", "postDate": "04/19/2015 17:07:40", "content": "

cool!

", "rawMarkdown": "", "votes": null }, { "id": "73054", "postDate": "04/22/2015 00:35:05", "content": "", "rawMarkdown": "", "votes": null }, { "id": "74130", "postDate": "04/22/2015 23:19:20", "content": "

We are going to share our code later

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 72468, "author_name": "kazanova", "author_url": "", "post_date": "04/19/2015 16:51:36", "content": "

Nice

", "votes": null, "replies": [] }, { "id": 72470, "author_name": "dulyanov", "author_url": "", "post_date": "04/19/2015 16:56:52", "content": "

Hello, it was amazing competition. Thanks our tuning master Stanislav for introducing incredible training techuniques for our classifiers.

\n
    \n
  1. I extracted byte counts, bigrams, i divided byte file into 4 parts and extracted byte counts from parts. 
    2. \n
  2. From .asm file I extracted counts for asm commands (about 100), registers, section names, segments, ‘_stdcall *’ names, out types of stcalls ‘* __ stdcall’, ‘Import from *’, ‘extern *’, ‘loc_*’, ‘*.dll’ (a lot of noise). Some keywords, that I saw in files like ‘microsoft’, ‘installdir’, ‘cookie’, ‘HKEY_LOCAL_MACHINE’, ‘sp-analysis failed’ and so on. I aslo extracted reglike strings (e.g {0231-.. -1401}).
    3. \n
  3. I extracted strings, but end up not using them directly. I just computed some statistics of strings length distribution, and it helped a lot. 
    4. \n
  4. Another good features I had was entropy features -- I computed it for sliding windows over the .bytes sequence and used some statistics of its distibution (quantiles, percentiles, mean, max, std; all of these with diff(entropy)), ended up with more than 200 entropy feats. 
    5. \n
  5. Line counts, size, etc.
    6. \n
\n

I did not want to use counts from 1) and any features Mikhail used directly, so I used NMF(X), NMF (log (X+1)) (they differ a lot!). All the counts from 2) were fed into linearSVC with L1 penalty in order to select features.

\n

All of the resulting features were used in one XGBoost model, averaged over several runs.

\n

Mixing:
We found, that our classifiers were making mistakes at different classes differently, so we mixed them for every class separately. Our overall mixing scheme was extremely sophisticated and we had mix of all our temporary solutions throughout the competition in the final submission. But all we did is per-class mixing and simple linear mixing with using 2d and 3rd order interaction between predictions.

\n

We were bold enough to trim the predictions to 0 and 1 for 1,3 and 7th classes. We got lucky and got 0.0041 -> 0.0039 private score(but we had some thoughts why it is ok to do it with these classes). The second submission had no trimming, so we were expecting not to fall down too much.

", "votes": null, "replies": [] }, { "id": 72480, "author_name": "xiaozhouwang", "author_url": "", "post_date": "04/19/2015 17:07:40", "content": "

cool!

", "votes": null, "replies": [] }, { "id": 73054, "author_name": "soumyasmruti", "author_url": "", "post_date": "04/22/2015 00:35:05", "content": "", "votes": null, "replies": [] }, { "id": 74130, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "04/22/2015 23:19:20", "content": "

We are going to share our code later

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "72467": "", "72468": "", "72470": "", "72480": "", "73054": "", "74130": "" }, "source": "meta" }