{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Microsoft Malware Classification Challenge\n- Hola amigos, this notebook covers my code for the **Microsoft Malware Classification** challenge, which can be found [here](https://www.kaggle.com/c/malware-classification).","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# 1.Business/Real-world Problem\n## 1.1. What is Malware?\n\nThe term malware is a contraction of malicious software. Put simply, malware is any piece of software that was written with the intent of doing harm to data, devices or to people. <br> \nSource: https://www.avg.com/en/signal/what-is-malware\n    \n## 1.2. Problem Statement\nIn the past few years, the malware industry has grown very rapidly that, the syndicates invest heavily in technologies to evade traditional protection, forcing the anti-malware groups/communities to build more robust softwares to detect and terminate these attacks. The major part of protecting a computer system from a malware attack is to **identify whether a given piece of file/software** is a malware. \n\n## 1.3 Source/Useful Links\nMicrosoft has been very active in building anti-malware products over the years  and it runs it’s anti-malware utilities over **150 million computers** around the world. This generates tens of millions of daily data points to be analyzed as potential malware. In order to be effective in analyzing and classifying such large amounts of data, we need to be able to group them into groups and identify their respective families. \n<br>\n<br>\nThis dataset provided by Microsoft contains about 9 classes of malware.\n<br>\n**Source:** https://www.kaggle.com/c/malware-classification\n\n## 1.4. Real-world/Business objectives and constraints.\n1. Minimize multi-class error.\n2. Multi-class probability estimates.\n3. Malware detection should not take hours and block the user's computer. It should fininsh in a few seconds or a minute.\n\n# 2. Machine Learning Problem\n## 2.1. Data\n### 2.1.1. Data Overview\n\n<li> Source : https://www.kaggle.com/c/malware-classification/data </li>\n<li> For every malware, we have two files <ol> <li> .asm file (read more: https://www.reviversoft.com/file-extensions/asm) </li><li>.bytes file (the raw data contains the hexadecimal representation of the file's binary content, without the PE header)</li></ol></li> \n    \n<li>Total train dataset consist of 200GB data out of which 50Gb of data is .bytes files and 150GB of data is .asm files:  </li>\n<li><b>Lots of Data for a single-box/computer.</b> </li>\n\n<li>There are total 10,868 .bytes files and 10,868 asm files total 21,736 files </li>\n\n<li>There are 9 types of malwares (9 classes) in our give data</li>\n<li> Types of Malware:\n    <ol>\n        <li> Ramnit </li>\n        <li> Lollipop </li>\n        <li> Kelihos_ver3 </li>\n        <li> Vundo </li>\n        <li> Simda </li>\n        <li> Tracur </li>\n        <li> Kelihos_ver1 </li>\n        <li> Obfuscator.ACY </li>\n        <li> Gatak </li>\n    </ol>\n</li>\n\n### 2.1.2. Example Data Point\n<p style = \"font-size:18px\"><b> .asm file</b></p>\n<pre>\n.text:00401000\t\t\t\t\t\t\t\t       assume es:nothing, ss:nothing, ds:_data,\tfs:nothing, gs:nothing\n.text:00401000 56\t\t\t\t\t\t\t       push    esi\n.text:00401001 8D 44 24\t08\t\t\t\t\t\t       lea     eax, [esp+8]\n.text:00401005 50\t\t\t\t\t\t\t       push    eax\n.text:00401006 8B F1\t\t\t\t\t\t\t       mov     esi, ecx\n.text:00401008 E8 1C 1B\t00 00\t\t\t\t\t\t       call    ??0exception@std@@QAE@ABQBD@Z ; std::exception::exception(char const * const &)\n.text:0040100D C7 06 08\tBB 42 00\t\t\t\t\t       mov     dword ptr [esi],\toffset off_42BB08\n.text:00401013 8B C6\t\t\t\t\t\t\t       mov     eax, esi\n.text:00401015 5E\t\t\t\t\t\t\t       pop     esi\n.text:00401016 C2 04 00\t\t\t\t\t\t\t       retn    4\n.text:00401016\t\t\t\t\t\t       ; ---------------------------------------------------------------------------\n.text:00401019 CC CC CC\tCC CC CC CC\t\t\t\t\t       align 10h\n.text:00401020 C7 01 08\tBB 42 00\t\t\t\t\t       mov     dword ptr [ecx],\toffset off_42BB08\n.text:00401026 E9 26 1C\t00 00\t\t\t\t\t\t       jmp     sub_402C51\n.text:00401026\t\t\t\t\t\t       ; ---------------------------------------------------------------------------\n.text:0040102B CC CC CC\tCC CC\t\t\t\t\t\t       align 10h\n.text:00401030 56\t\t\t\t\t\t\t       push    esi\n.text:00401031 8B F1\t\t\t\t\t\t\t       mov     esi, ecx\n.text:00401033 C7 06 08\tBB 42 00\t\t\t\t\t       mov     dword ptr [esi],\toffset off_42BB08\n.text:00401039 E8 13 1C\t00 00\t\t\t\t\t\t       call    sub_402C51\n.text:0040103E F6 44 24\t08 01\t\t\t\t\t\t       test    byte ptr\t[esp+8], 1\n.text:00401043 74 09\t\t\t\t\t\t\t       jz      short loc_40104E\n.text:00401045 56\t\t\t\t\t\t\t       push    esi\n.text:00401046 E8 6C 1E\t00 00\t\t\t\t\t\t       call    ??3@YAXPAX@Z    ; operator delete(void *)\n.text:0040104B 83 C4 04\t\t\t\t\t\t\t       add     esp, 4\n.text:0040104E\n.text:0040104E\t\t\t\t\t\t       loc_40104E:\t\t\t       ; CODE XREF: .text:00401043\u0018j\n.text:0040104E 8B C6\t\t\t\t\t\t\t       mov     eax, esi\n.text:00401050 5E\t\t\t\t\t\t\t       pop     esi\n.text:00401051 C2 04 00\t\t\t\t\t\t\t       retn    4\n.text:00401051\t\t\t\t\t\t       ; ---------------------------------------------------------------------------\n</pre>\n<p style = \"font-size:18px\"><b> .bytes file</b></p>\n<pre>\n00401000 00 00 80 40 40 28 00 1C 02 42 00 C4 00 20 04 20\n00401010 00 00 20 09 2A 02 00 00 00 00 8E 10 41 0A 21 01\n00401020 40 00 02 01 00 90 21 00 32 40 00 1C 01 40 C8 18\n00401030 40 82 02 63 20 00 00 09 10 01 02 21 00 82 00 04\n00401040 82 20 08 83 00 08 00 00 00 00 02 00 60 80 10 80\n00401050 18 00 00 20 A9 00 00 00 00 04 04 78 01 02 70 90\n00401060 00 02 00 08 20 12 00 00 00 40 10 00 80 00 40 19\n00401070 00 00 00 00 11 20 80 04 80 10 00 20 00 00 25 00\n00401080 00 00 01 00 00 04 00 10 02 C1 80 80 00 20 20 00\n00401090 08 A0 01 01 44 28 00 00 08 10 20 00 02 08 00 00\n004010A0 00 40 00 00 00 34 40 40 00 04 00 08 80 08 00 08\n004010B0 10 00 40 00 68 02 40 04 E1 00 28 14 00 08 20 0A\n004010C0 06 01 02 00 40 00 00 00 00 00 00 20 00 02 00 04\n004010D0 80 18 90 00 00 10 A0 00 45 09 00 10 04 40 44 82\n004010E0 90 00 26 10 00 00 04 00 82 00 00 00 20 40 00 00\n004010F0 B4 00 00 40 00 02 20 25 08 00 00 00 00 00 00 00\n00401100 08 00 00 50 00 08 40 50 00 02 06 22 08 85 30 00\n00401110 00 80 00 80 60 00 09 00 04 20 00 00 00 00 00 00\n00401120 00 82 40 02 00 11 46 01 4A 01 8C 01 E6 00 86 10\n00401130 4C 01 22 00 64 00 AE 01 EA 01 2A 11 E8 10 26 11\n00401140 4E 11 8E 11 C2 00 6C 00 0C 11 60 01 CA 00 62 10\n00401150 6C 01 A0 11 CE 10 2C 11 4E 10 8C 00 CE 01 AE 01\n00401160 6C 10 6C 11 A2 01 AE 00 46 11 EE 10 22 00 A8 00\n00401170 EC 01 08 11 A2 01 AE 10 6C 00 6E 00 AC 11 8C 00\n00401180 EC 01 2A 10 2A 01 AE 00 40 00 C8 10 48 01 4E 11\n00401190 0E 00 EC 11 24 10 4A 10 04 01 C8 11 E6 01 C2 00\n</pre>\n\n## 2.2. Mapping the real-world problem to an ML problem\n### 2.2.1. Type of Machine Learning Problem\nThere are nine different classes of malware that we need to classify a given a data point => Multi class classification problem    \n\n### 2.2.2. Performance Metric\nSource: https://www.kaggle.com/c/malware-classification#evaluation\n\nMetric(s): \n* Multi class log-loss \n* Confusion matrix \n\n### 2.2.3. Machine Learing Objectives and Constraints\n**Objective**: Predict the probability of each data-point belonging to each of the nine classes.\n<br>\n<br>\n**Constraints**:\n* Class probabilities are needed.\n* Penalize the errors in class probabilites => Metric is Log-loss.\n* Some Latency constraints.\n\n## 2.3. Train and Test Dataset\nSplit the dataset randomly into three parts train, cross validation and test with 64%,16%, 20% of data respectively\n\n## 2.4. Useful blogs, videos and reference papers\nhttp://blog.kaggle.com/2015/05/26/microsoft-malware-winners-interview-1st-place-no-to-overfitting/ <br>\nhttps://arxiv.org/pdf/1511.04317.pdf <br>\nFirst place solution in Kaggle competition: https://www.youtube.com/watch?v=VLQTRlLGz5Y <br>\nhttps://github.com/dchad/malware-detection <br>\nhttp://vizsec.org/files/2011/Nataraj.pdf <br>\nhttps://www.dropbox.com/sh/gfqzv0ckgs4l1bf/AAB6EelnEjvvuQg2nu_pIB6ua?dl=0 <br>\n\" Cross validation is more trustworthy than domain knowledge.\" \n","metadata":{}},{"cell_type":"markdown","source":"# 3. Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"import os\nimport pickle\nimport shutil\nimport warnings\nimport random as r\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\n# Matplotlib Imports\nimport matplotlib\nimport matplotlib.pyplot as plt\nmatplotlib.use(u'nbAgg')\n\n# This is used for multithreading\nimport multiprocessing\nfrom multiprocessing import Process\n\n# This is used for file operations\nimport codecs  \n\n# Sklearn Imports\nfrom sklearn import preprocessing\nfrom sklearn.manifold import TSNE\nfrom sklearn.model_selection import RandomizedSearchCV, train_test_split\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.calibration import CalibratedClassifierCV\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.metrics import log_loss, confusion_matrix\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier\n\nfrom xgboost import XGBClassifier\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:50:48.760119Z","iopub.execute_input":"2022-05-28T11:50:48.760403Z","iopub.status.idle":"2022-05-28T11:50:48.76845Z","shell.execute_reply.started":"2022-05-28T11:50:48.760375Z","shell.execute_reply":"2022-05-28T11:50:48.767804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Separating byte files and asm files \nsource = 'train'\ndestination = 'byteFiles'\n\n# We will check if the folder 'byteFiles' exists if it it is not there,\n# then we will create a folder with the same name\nif not os.path.isdir(destination):\n    os.makedirs(destination)\n\n# If we have folder called 'train' (train folder containsboth .asm files and .bytes files), we will rename it to 'asmFiles'\n# For every file that we have in our 'asmFiles' directory we check if it is ending with .bytes, \n# if yes we will move it to 'byteFiles' folder\n\n# So ,by the end of this snippet we will separate all the .byte files and .asm files\nif os.path.isdir(source):\n    os.rename(source,'asmFiles')\n    source = 'asmFiles'\n    data_files = os.listdir(source)\n    for file in asm_files:\n        if (file.endswith(\"bytes\")):\n            shutil.move(source + file, destination)\n# print(source)\n# print('This cell is done.')","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:50:53.291751Z","iopub.execute_input":"2022-05-28T11:50:53.292014Z","iopub.status.idle":"2022-05-28T11:50:53.300064Z","shell.execute_reply.started":"2022-05-28T11:50:53.291979Z","shell.execute_reply":"2022-05-28T11:50:53.298128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.1. Distribution of malware classes in whole data set","metadata":{}},{"cell_type":"code","source":"# !pip install ipympl\n!pip install nodejs\n!jupyter labextension install @jupyter-widgets/jupyterlab-manager\n!jupyter labextension install jupyter-matplotlib\n\nY = pd.read_csv(\"../input/malware-classification/trainLabels.csv\")\ntotal = len(Y) * 1.\nax = sns.countplot(x = \"Class\", data = Y)\nfor p in ax.patches:\n        ax.annotate('{:.1f}%'.format(100*p.get_height()/total), (p.get_x()+0.1, p.get_height()+5))\n\n# Put 11 ticks (therefore 10 steps), from 0 to the total number of rows in the dataframe\nax.yaxis.set_ticks(np.linspace(0, total, 11))\n\n# Adjust the ticklabel to the desired format, without changing the position of the ticks. \nax.set_yticklabels(map('{:.1f}%'.format, 100*ax.yaxis.get_majorticklocs()/total))\n# %matplotlib widget\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:59:04.761837Z","iopub.execute_input":"2022-05-28T11:59:04.762294Z","iopub.status.idle":"2022-05-28T12:07:47.210003Z","shell.execute_reply.started":"2022-05-28T11:59:04.762258Z","shell.execute_reply":"2022-05-28T12:07:47.209246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2. Feature extraction\n### 3.2.1 File size  of byte files as a feature","metadata":{}},{"cell_type":"code","source":"# File sizes of byte files\nfiles = os.listdir('byteFiles')\nfilenames = Y['Id'].tolist()\nclass_y = Y['Class'].tolist()\nclass_bytes = []\nsizebytes = []\nfnames = []\nfor file in files:\n    # print(os.stat('byteFiles/0A32eTdBKayjCWhZqDOQ.txt'))\n    # os.stat_result(st_mode=33206, st_ino=1125899906874507, st_dev=3561571700, st_nlink=1, st_uid=0, st_gid=0, \n    # st_size=3680109, st_atime=1519638522, st_mtime=1519638522, st_ctime=1519638522)\n    # Read more about os.stat: here https://www.tutorialspoint.com/python/os_stat.htm\n    statinfo = os.stat('byteFiles/' + file)\n    # Split the file name at '.' and take the first part of it i.e the file name\n    file = file.split('.')[0]\n    if any(file == filename for filename in filenames):\n        i = filenames.index(file)\n        class_bytes.append(class_y[i])\n        # Converting into Mb's\n        sizebytes.append(statinfo.st_size/(1024.0 * 1024.0))\n        fnames.append(file)\ndata_size_byte = pd.DataFrame({'ID':fnames,'size':sizebytes,'Class':class_bytes})\nprint (data_size_byte.head())","metadata":{"execution":{"iopub.status.busy":"2022-05-28T12:07:47.211763Z","iopub.execute_input":"2022-05-28T12:07:47.212615Z","iopub.status.idle":"2022-05-28T12:07:47.224765Z","shell.execute_reply.started":"2022-05-28T12:07:47.212574Z","shell.execute_reply":"2022-05-28T12:07:47.223995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2.2 Box plots of file size (.byte files) feature","metadata":{}},{"cell_type":"code","source":"# Boxplot of byte files\n# %matplotlib widget\nax = sns.boxplot(x = \"Class\", y = \"size\", data = data_size_byte)\nplt.title(\"Boxplot of .bytes file sizes\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T12:07:47.479626Z","iopub.status.idle":"2022-05-28T12:07:47.480193Z","shell.execute_reply.started":"2022-05-28T12:07:47.479947Z","shell.execute_reply":"2022-05-28T12:07:47.479972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2.3 Feature extraction from byte files","metadata":{}},{"cell_type":"code","source":"# Removal of addres from byte files\n# We remove the starting address 00401000\n\n# Contents of .byte files\n# 00401000 56 8D 44 24 08 50 8B F1 E8 1C 1B 00 00 C7 06 08 \n\nfiles = os.listdir('byteFiles')\nfilenames = []\narray = []\nfor file in files:\n    if(f.endswith(\"bytes\")):\n        file = file.split('.')[0]\n        text_file = open('byteFiles/' + file + \".txt\", 'w+')\n        with open('byteFiles/' + file, \"r\") as fp:\n            lines = \"\"\n            for line in fp:\n                a = line.rstrip().split(\" \")[1:]\n                b = ' '.join(a)\n                b = b + \"\\n\"\n                text_file.write(b)\n            fp.close()\n            os.remove('byteFiles/' + file)\n        text_file.close()\n\nfiles = os.listdir('byteFiles')\nfilenames2 = []\nfeature_matrix = np.zeros((len(files), 257),dtype=int)\nk = 0\n\n# Program to convert into bag of words of bytefiles\n# This is custom-built bag of words this is unigram bag of words\nbyte_feature_file = open('result.csv','w+')\nbyte_feature_file.write(\"ID,0,1,2,3,4,5,6,7,8,9,0a,0b,0c,0d,0e,0f,10,11,12,13,14,15,16,17,18,19,1a,1b,1c,1d,1e,1f,20,21,22,23,24,25,26,27,28,29,2a,2b,2c,2d,2e,2f,30,31,32,33,34,35,36,37,38,39,3a,3b,3c,3d,3e,3f,40,41,42,43,44,45,46,47,48,49,4a,4b,4c,4d,4e,4f,50,51,52,53,54,55,56,57,58,59,5a,5b,5c,5d,5e,5f,60,61,62,63,64,65,66,67,68,69,6a,6b,6c,6d,6e,6f,70,71,72,73,74,75,76,77,78,79,7a,7b,7c,7d,7e,7f,80,81,82,83,84,85,86,87,88,89,8a,8b,8c,8d,8e,8f,90,91,92,93,94,95,96,97,98,99,9a,9b,9c,9d,9e,9f,a0,a1,a2,a3,a4,a5,a6,a7,a8,a9,aa,ab,ac,ad,ae,af,b0,b1,b2,b3,b4,b5,b6,b7,b8,b9,ba,bb,bc,bd,be,bf,c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,ca,cb,cc,cd,ce,cf,d0,d1,d2,d3,d4,d5,d6,d7,d8,d9,da,db,dc,dd,de,df,e0,e1,e2,e3,e4,e5,e6,e7,e8,e9,ea,eb,ec,ed,ee,ef,f0,f1,f2,f3,f4,f5,f6,f7,f8,f9,fa,fb,fc,fd,fe,ff,??\")\n\nfor file in files:\n    filenames2.append(f)\n    byte_feature_file.write(file + \",\")\n    if(file.endswith(\"txt\")):\n        with open('byteFiles/' + file,\"r\") as byte_flie:\n            for lines in byte_flie:\n                line = lines.rstrip().split(\" \")\n                for hex_code in line:\n                    if hex_code == '??': feature_matrix[k][256] += 1\n                    else: feature_matrix[k][int(hex_code,16)] += 1\n        byte_flie.close()\n    for i in feature_matrix[k]:\n        byte_feature_file.write(str(i) + \",\")\n    byte_feature_file.write(\"\\n\")\n    k += 1\n\nbyte_feature_file.close()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:52:28.788252Z","iopub.execute_input":"2022-05-28T11:52:28.7886Z","iopub.status.idle":"2022-05-28T11:52:28.802751Z","shell.execute_reply.started":"2022-05-28T11:52:28.788568Z","shell.execute_reply":"2022-05-28T11:52:28.801929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"byte_features = pd.read_csv(\"result.csv\")\nbyte_features.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:52:34.467701Z","iopub.execute_input":"2022-05-28T11:52:34.467965Z","iopub.status.idle":"2022-05-28T11:52:34.543187Z","shell.execute_reply.started":"2022-05-28T11:52:34.467937Z","shell.execute_reply":"2022-05-28T11:52:34.542407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = pd.merge(byte_features, data_size_byte, on = 'ID', how = 'left')\nresult.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:52:39.970039Z","iopub.execute_input":"2022-05-28T11:52:39.97029Z","iopub.status.idle":"2022-05-28T11:52:39.993876Z","shell.execute_reply.started":"2022-05-28T11:52:39.970263Z","shell.execute_reply":"2022-05-28T11:52:39.992904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://stackoverflow.com/a/29651514\ndef normalize(df):\n    result1 = df.copy()\n    for feature_name in df.columns:\n        if (str(feature_name) != str('ID') and str(feature_name)!=str('Class')):\n            max_value = df[feature_name].max()\n            min_value = df[feature_name].min()\n            result1[feature_name] = (df[feature_name] - min_value) / (max_value - min_value)\n    return result1\nresult = normalize(result)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:52:44.338619Z","iopub.execute_input":"2022-05-28T11:52:44.338879Z","iopub.status.idle":"2022-05-28T11:52:44.358125Z","shell.execute_reply.started":"2022-05-28T11:52:44.338852Z","shell.execute_reply":"2022-05-28T11:52:44.357315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_y = result['Class']\nresult.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:52:47.627032Z","iopub.execute_input":"2022-05-28T11:52:47.627431Z","iopub.status.idle":"2022-05-28T11:52:47.64299Z","shell.execute_reply.started":"2022-05-28T11:52:47.627399Z","shell.execute_reply":"2022-05-28T11:52:47.641963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2.4 Multivariate Analysis","metadata":{}},{"cell_type":"code","source":"# Multivariate analysis on byte files, with perplexity 50\nxtsne = TSNE(perplexity = 50)\nresults = xtsne.fit_transform(result.drop(['ID','Class'], axis = 1))\nvis_x = results[:, 0]\nvis_y = results[:, 1]\nplt.scatter(vis_x, vis_y, c = data_y, cmap = plt.cm.get_cmap(\"jet\", 9))\nplt.colorbar(ticks = range(10))\nplt.clim(0.5, 9)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:52:50.381684Z","iopub.execute_input":"2022-05-28T11:52:50.382295Z","iopub.status.idle":"2022-05-28T11:52:50.401859Z","shell.execute_reply.started":"2022-05-28T11:52:50.382258Z","shell.execute_reply":"2022-05-28T11:52:50.400882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Multivariate analysis on byte files, with perplexity 30\nxtsne = TSNE(perplexity = 30)\nresults = xtsne.fit_transform(result.drop(['ID','Class'], axis = 1))\nvis_x = results[:, 0]\nvis_y = results[:, 1]\nplt.scatter(vis_x, vis_y, c = data_y, cmap = plt.cm.get_cmap(\"jet\", 9))\nplt.colorbar(ticks = range(10))\nplt.clim(0.5, 9)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.147204Z","iopub.status.idle":"2022-05-28T11:32:43.147857Z","shell.execute_reply.started":"2022-05-28T11:32:43.147635Z","shell.execute_reply":"2022-05-28T11:32:43.147656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Test Split\n- Since we have used **random splitting**, hence, the train, the test and the cross-validation sets have more or less, the same distribution among the classes.\n- Had we have used the **time-based splitting**, it would have been of the utmost importance to check the distribution among the classes of the train, test and cross-validation sets, as that would give us some useful insights. ","metadata":{}},{"cell_type":"code","source":"data_y = result['Class']\n\n# Split the data into test and train by maintaining the same distribution of output \n# variable 'y_true' [stratify = y_true]\nX_train, X_test, y_train, y_test = train_test_split(result.drop(['ID','Class'], axis = 1), \n    data_y, stratify = data_y, test_size = 0.20)\n\n# Split the train data into train and cross validation by maintaining the same distribution of \n# output variable 'y_train' [stratify = y_train]\nX_train, X_cv, y_train, y_cv = train_test_split(X_train, y_train, stratify = y_train, test_size = 0.20)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.150938Z","iopub.status.idle":"2022-05-28T11:32:43.151526Z","shell.execute_reply.started":"2022-05-28T11:32:43.151286Z","shell.execute_reply":"2022-05-28T11:32:43.151322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Number of data points in train data:', X_train.shape[0])\nprint('Number of data points in test data:', X_test.shape[0])\nprint('Number of data points in cross validation data:', X_cv.shape[0])","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.152828Z","iopub.status.idle":"2022-05-28T11:32:43.154267Z","shell.execute_reply.started":"2022-05-28T11:32:43.153878Z","shell.execute_reply":"2022-05-28T11:32:43.153903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# It returns a dict, keys as class labels and values as the number of data points in that class\ntrain_class_distribution = y_train.value_counts().sortlevel()\ntest_class_distribution = y_test.value_counts().sortlevel()\ncv_class_distribution = y_cv.value_counts().sortlevel()\n\nmy_colors = 'rgbkymc'\ntrain_class_distribution.plot(kind='bar', color=my_colors)\nplt.xlabel('Class')\nplt.ylabel('Data points per Class')\nplt.title('Distribution of yi in train data')\nplt.grid()\nplt.show()\n\n#  https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html\n# -(train_class_distribution.values): The minus sign will give us in decreasing order\nsorted_yi = np.argsort(-train_class_distribution.values)\nfor i in sorted_yi:\n    print('Number of data points in class', i+1, ':', train_class_distribution.values[i], \n      '(', np.round((train_class_distribution.values[i]/y_train.shape[0]*100), 3), '%)')\n\nprint('-'*80)\nmy_colors = 'rgbkymc'\ntest_class_distribution.plot(kind='bar', color=my_colors)\nplt.xlabel('Class')\nplt.ylabel('Data points per Class')\nplt.title('Distribution of yi in test data')\nplt.grid()\nplt.show()\n\n# https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html\n# -(train_class_distribution.values): The minus sign will give us in decreasing order\nsorted_yi = np.argsort(-test_class_distribution.values)\nfor i in sorted_yi:\n    print('Number of data points in class', i+1, ':',test_class_distribution.values[i], \n      '(', np.round((test_class_distribution.values[i]/y_test.shape[0]*100), 3), '%)')\n\nprint('-'*80)\nmy_colors = 'rgbkymc'\ncv_class_distribution.plot(kind='bar', color=my_colors)\nplt.xlabel('Class')\nplt.ylabel('Data points per Class')\nplt.title('Distribution of yi in cross validation data')\nplt.grid()\nplt.show()\n\n# https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html\n# -(train_class_distribution.values): The minus sign will give us in decreasing order\nsorted_yi = np.argsort(-train_class_distribution.values)\nfor i in sorted_yi:\n    print('Number of data points in class', i+1, ':',cv_class_distribution.values[i], \n      '(', np.round((cv_class_distribution.values[i]/y_cv.shape[0]*100), 3), '%)')","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.156684Z","iopub.status.idle":"2022-05-28T11:32:43.158023Z","shell.execute_reply.started":"2022-05-28T11:32:43.157693Z","shell.execute_reply":"2022-05-28T11:32:43.157718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_confusion_matrix(test_y, predict_y):\n    # C = 9,9 matrix, each cell (i,j) represents number of points of class i are predicted class j\n    C = confusion_matrix(test_y, predict_y)\n    print(\"Number of misclassified points \",(len(test_y)-np.trace(C))/len(test_y)*100)\n    \n    # Divide each element of the confusion matrix with the sum of elements in that column\n    A = (((C.T)/(C.sum(axis=1))).T)\n    \n    # Divide each element of the confusion matrix with the sum of elements in that row\n    B = (C/C.sum(axis=0))\n\n    labels = [1,2,3,4,5,6,7,8,9]\n    cmap = sns.light_palette(\"green\")\n    \n    # Representing C in heatmap format\n    print(\"-\"*50, \"Confusion matrix\", \"-\"*50)\n    plt.figure(figsize=(10,5))\n    sns.heatmap(C, annot=True, cmap=cmap, fmt=\".3f\", xticklabels=labels, yticklabels=labels)\n    plt.xlabel('Predicted Class')\n    plt.ylabel('Original Class')\n    plt.show()\n\n    # Representing A in heatmap format\n    print(\"-\"*50, \"Precision matrix\", \"-\"*50)\n    plt.figure(figsize=(10,5))\n    sns.heatmap(A, annot=True, cmap=cmap, fmt=\".3f\", xticklabels=labels, yticklabels=labels)\n    plt.xlabel('Predicted Class')\n    plt.ylabel('Original Class')\n    plt.show()\n    print(\"Sum of columns in precision matrix\",B.sum(axis=0))\n    \n    # Representing B in heatmap format\n    print(\"-\"*50, \"Recall matrix\"    , \"-\"*50)\n    plt.figure(figsize=(10,5))\n    sns.heatmap(B, annot=True, cmap=cmap, fmt=\".3f\", xticklabels=labels, yticklabels=labels)\n    plt.xlabel('Predicted Class')\n    plt.ylabel('Original Class')\n    plt.show()\n    print(\"Sum of rows in precision matrix\",A.sum(axis=1))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.160539Z","iopub.status.idle":"2022-05-28T11:32:43.161118Z","shell.execute_reply.started":"2022-05-28T11:32:43.1609Z","shell.execute_reply":"2022-05-28T11:32:43.160921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Machine Learning Models\n## 4.1. Machine Leaning Models on bytes files\n### 4.1.1. Random Model\n- We know that the range of MCLL is (0 - Inf). In other words, we can say that we don't know the maximum value of MCLL, or the worst case. \n- So, in order to know the maximum possible (worst case) value of MCLL, we train a random model on our dataset.\n- And since in the output, we need to give 9 probabilities for every data-point that sum upto 1, we simply predict 9 random numbers for every data-point and then normalize them.","metadata":{}},{"cell_type":"code","source":"# We need to generate 9 numbers & the sum of numbers should be 1\n# One solution is to generate 9 numbers & divide each of the numbers by their sum\n# https://stackoverflow.com/a/18662466/4084039\n\ntest_data_len = X_test.shape[0]\ncv_data_len = X_cv.shape[0]\n\n# We create an output array that has exactly same size as the CV data\ncv_predicted_y = np.zeros((cv_data_len,9))\nfor i in range(cv_data_len):\n    rand_probs = np.random.rand(1,9)\n    cv_predicted_y[i] = ((rand_probs/sum(sum(rand_probs)))[0])\nprint(\"Log loss on Cross Validation Data using Random Model\",log_loss(y_cv,cv_predicted_y, eps=1e-15))\n\n# Test-Set error.\n# We create an output array that has exactly same as the test data\ntest_predicted_y = np.zeros((test_data_len,9))\nfor i in range(test_data_len):\n    rand_probs = np.random.rand(1,9)\n    test_predicted_y[i] = ((rand_probs/sum(sum(rand_probs)))[0])\nprint(\"Log loss on Test Data using Random Model\", log_loss(y_test,test_predicted_y, eps = 1e-15))\n\npredicted_y = np.argmax(test_predicted_y, axis = 1)\nplot_confusion_matrix(y_test, predicted_y+1)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.164327Z","iopub.status.idle":"2022-05-28T11:32:43.164903Z","shell.execute_reply.started":"2022-05-28T11:32:43.164685Z","shell.execute_reply":"2022-05-28T11:32:43.164707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1.2. K Nearest Neighbour Classification","metadata":{}},{"cell_type":"code","source":"# http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\n# http://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html\n  \nalpha = [x for x in range(1, 15, 2)]\ncv_log_error_array = []\nfor i in alpha:\n    k_cfl = KNeighborsClassifier(n_neighbors = i)\n    k_cfl.fit(X_train,y_train)\n    sig_clf = CalibratedClassifierCV(k_cfl, method = \"sigmoid\")\n    sig_clf.fit(X_train, y_train)\n    predict_y = sig_clf.predict_proba(X_cv)\n    cv_log_error_array.append(log_loss(y_cv, predict_y, labels = k_cfl.classes_, eps = 1e-15))\n    \nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for k = ', alpha[i], 'is', cv_log_error_array[i])\n\nbest_alpha = np.argmin(cv_log_error_array)\n    \nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nk_cfl = KNeighborsClassifier(n_neighbors = alpha[best_alpha])\nk_cfl.fit(X_train,y_train)\nsig_clf = CalibratedClassifierCV(k_cfl, method = \"sigmoid\")\nsig_clf.fit(X_train, y_train)\n    \npredict_y = sig_clf.predict_proba(X_train)\nprint ('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train, predict_y))\npredict_y = sig_clf.predict_proba(X_cv)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv, predict_y))\npredict_y = sig_clf.predict_proba(X_test)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test, predict_y))\nplot_confusion_matrix(y_test, sig_clf.predict(X_test))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.166148Z","iopub.status.idle":"2022-05-28T11:32:43.167869Z","shell.execute_reply.started":"2022-05-28T11:32:43.167537Z","shell.execute_reply":"2022-05-28T11:32:43.167562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1.3. Logistic Regression","metadata":{}},{"cell_type":"code","source":"# http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html\n\nalpha = [10 ** x for x in range(-5, 4)]\ncv_log_error_array = []\nfor i in alpha:\n    logisticR = LogisticRegression(penalty='l2',C = i,class_weight = 'balanced')\n    logisticR.fit(X_train,y_train)\n    sig_clf = CalibratedClassifierCV(logisticR, method = \"sigmoid\")\n    sig_clf.fit(X_train, y_train)\n    predict_y = sig_clf.predict_proba(X_cv)\n    cv_log_error_array.append(log_loss(y_cv, predict_y, labels = logisticR.classes_, eps = 1e-15))\n    \nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ',alpha[i],'is', cv_log_error_array[i])\n\nbest_alpha = np.argmin(cv_log_error_array)\n    \nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array,c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nlogisticR = LogisticRegression(penalty = 'l2',C = alpha[best_alpha],class_weight = 'balanced')\nlogisticR.fit(X_train,y_train)\nsig_clf = CalibratedClassifierCV(logisticR, method=\"sigmoid\")\nsig_clf.fit(X_train, y_train)\npred_y = sig_clf.predict(X_test)\n\npredict_y = sig_clf.predict_proba(X_train)\nprint ('log loss for train data',log_loss(y_train, predict_y, labels=logisticR.classes_, eps=1e-15))\npredict_y = sig_clf.predict_proba(X_cv)\nprint ('log loss for cv data',log_loss(y_cv, predict_y, labels=logisticR.classes_, eps=1e-15))\npredict_y = sig_clf.predict_proba(X_test)\nprint ('log loss for test data',log_loss(y_test, predict_y, labels=logisticR.classes_, eps=1e-15))\nplot_confusion_matrix(y_test, sig_clf.predict(X_test))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.170334Z","iopub.status.idle":"2022-05-28T11:32:43.170909Z","shell.execute_reply.started":"2022-05-28T11:32:43.170693Z","shell.execute_reply":"2022-05-28T11:32:43.170715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1.4. Random Forest Classifier","metadata":{}},{"cell_type":"code","source":"alpha = [10,50,100,500,1000,2000,3000]\ncv_log_error_array = []\ntrain_log_error_array = []\nfrom sklearn.ensemble import RandomForestClassifier\nfor i in alpha:\n    r_cfl = RandomForestClassifier(n_estimators = i, random_state = 42, n_jobs = -1)\n    r_cfl.fit(X_train,y_train)\n    sig_clf = CalibratedClassifierCV(r_cfl, method = \"sigmoid\")\n    sig_clf.fit(X_train, y_train)\n    predict_y = sig_clf.predict_proba(X_cv)\n    cv_log_error_array.append(log_loss(y_cv, predict_y, labels = r_cfl.classes_, eps = 1e-15))\n\nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\n\nbest_alpha = np.argmin(cv_log_error_array)\n\nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt,3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nr_cfl = RandomForestClassifier(n_estimators = alpha[best_alpha], random_state = 42,n_jobs = -1)\nr_cfl.fit(X_train,y_train)\nsig_clf = CalibratedClassifierCV(r_cfl, method = \"sigmoid\")\nsig_clf.fit(X_train, y_train)\n\npredict_y = sig_clf.predict_proba(X_train)\nprint('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train, predict_y))\npredict_y = sig_clf.predict_proba(X_cv)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv, predict_y))\npredict_y = sig_clf.predict_proba(X_test)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test, predict_y))\nplot_confusion_matrix(y_test, sig_clf.predict(X_test))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.174244Z","iopub.status.idle":"2022-05-28T11:32:43.174851Z","shell.execute_reply.started":"2022-05-28T11:32:43.174635Z","shell.execute_reply":"2022-05-28T11:32:43.174656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1.5. XgBoost Classification","metadata":{}},{"cell_type":"code","source":"# http://xgboost.readthedocs.io/en/latest/python/python_api.html?#xgboost.XGBClassifier\n\nalpha = [10,50,100,500,1000,2000]\ncv_log_error_array = []\nfor i in alpha:\n    x_cfl = XGBClassifier(n_estimators = i, nthread = -1)\n    x_cfl.fit(X_train,y_train)\n    sig_clf = CalibratedClassifierCV(x_cfl, method = \"sigmoid\")\n    sig_clf.fit(X_train, y_train)\n    predict_y = sig_clf.predict_proba(X_cv)\n    cv_log_error_array.append(log_loss(y_cv, predict_y, labels = x_cfl.classes_, eps = 1e-15))\n\nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\n\nbest_alpha = np.argmin(cv_log_error_array)\n\nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nx_cfl = XGBClassifier(n_estimators = alpha[best_alpha], nthread = -1)\nx_cfl.fit(X_train,y_train)\nsig_clf = CalibratedClassifierCV(x_cfl, method = \"sigmoid\")\nsig_clf.fit(X_train, y_train)\n    \npredict_y = sig_clf.predict_proba(X_train)\nprint ('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train, predict_y))\npredict_y = sig_clf.predict_proba(X_cv)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv, predict_y))\npredict_y = sig_clf.predict_proba(X_test)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test, predict_y))\nplot_confusion_matrix(y_test, sig_clf.predict(X_test))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.178001Z","iopub.status.idle":"2022-05-28T11:32:43.178639Z","shell.execute_reply.started":"2022-05-28T11:32:43.178405Z","shell.execute_reply":"2022-05-28T11:32:43.178426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1.5. XgBoost Classification with best hyper parameters using RandomSearch","metadata":{}},{"cell_type":"code","source":"# https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/\nx_cfl = XGBClassifier()\n\nprams={\n    'learning_rate':[0.01,0.03,0.05,0.1,0.15,0.2],\n     'n_estimators':[100,200,500,1000,2000],\n     'max_depth':[3,5,10],\n    'colsample_bytree':[0.1,0.3,0.5,1],\n    'subsample':[0.1,0.3,0.5,1]\n}\nrandom_cfl1 = RandomizedSearchCV(x_cfl, param_distributions = prams, verbose = 10, n_jobs = -1,)\nrandom_cfl1.fit(X_train,y_train)\nprint (random_cfl1.best_params_)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.179744Z","iopub.status.idle":"2022-05-28T11:32:43.180489Z","shell.execute_reply.started":"2022-05-28T11:32:43.180246Z","shell.execute_reply":"2022-05-28T11:32:43.18027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://xgboost.readthedocs.io/en/latest/python/python_api.html?#xgboost.XGBClassifier\nx_cfl = XGBClassifier(n_estimators = 2000, learning_rate = 0.05, colsample_bytree = 1, max_depth = 3)\nx_cfl.fit(X_train,y_train)\nc_cfl = CalibratedClassifierCV(x_cfl, method = 'sigmoid')\nc_cfl.fit(X_train,y_train)\n\npredict_y = c_cfl.predict_proba(X_train)\nprint ('train loss', log_loss(y_train, predict_y))\npredict_y = c_cfl.predict_proba(X_cv)\nprint ('cv loss', log_loss(y_cv, predict_y))\npredict_y = c_cfl.predict_proba(X_test)\nprint ('test loss', log_loss(y_test, predict_y))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.182888Z","iopub.status.idle":"2022-05-28T11:32:43.183486Z","shell.execute_reply.started":"2022-05-28T11:32:43.183252Z","shell.execute_reply":"2022-05-28T11:32:43.183274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Modeling with .asm files\n<pre>\nThere are 10868 files of asm \nAll the files make up about 150 GB\nThe asm files contains :\n1. Address\n2. Segments\n3. Opcodes\n4. Registers\n5. function calls\n6. APIs\nWith the help of parallel processing we extracted all the features.In parallel we can use all the cores that are present in our computer.\n</pre>\n\nHere we extracted 52 features from all the asm files which are important.\n\nWe read the top solutions and handpicked the features from those papers/videos/blogs. <br> Refer: https://www.kaggle.com/c/malware-classification/discussion\n","metadata":{}},{"cell_type":"markdown","source":"### 4.2.1 Feature extraction from asm files\n- To extract the unigram features from the .asm files we need to process ~150GB of data </li>\n- **Note: Below two cells will take lot of time (over 48 hours to complete)**\n- In order to perform this step, we used the concept of **multi-processing**. This is only possible due to the existence of multiple cores in a system.\n- Suppose a system has 4 cores. It means that there are 4 processors in all, all of which have access to RAM, and all of which can execute programs in parallel.\n- So, in the case of this system, we will make 4 folders, and divide our ASM files into these 4 folders. Then, each of these cores will bring a single .asm file from each one of these folders, and will process them in parallel.\n- This will speed-up the feature extraction from asm files, and we can speed it up as much as the number of cores are there in our system.","metadata":{}},{"cell_type":"code","source":"# Intially create five folders\n# first, second, third, fourth, fifth\n# This code tells us about random split of files into five folders\n\nfolder_1 = 'first'\nfolder_2 = 'second'\nfolder_3 = 'third'\nfolder_4 = 'fourth'\nfolder_5 = 'fifth'\nfolder_6 = 'output'\nfor i in [folder_1, folder_2, folder_3, folder_4, folder_5, folder_6]:\n    if not os.path.isdir(i):\n        os.makedirs(i)\n\nsource = 'train/'\nfiles = os.listdir('train')\nID = df['Id'].tolist()\ndata = range(0,10868)\nr.shuffle(data)\ncount = 0\n\nfor i in range(0,10868):\n    if i%5 == 0:\n        shutil.move(source + files[data[i]],'first')\n    elif i%5 == 1:\n        shutil.move(source + files[data[i]],'second')\n    elif i%5 == 2:\n        shutil.move(source + files[data[i]],'third')\n    elif i%5 == 3:\n        shutil.move(source + files[data[i]],'fourth')\n    elif i%5 == 4:\n        shutil.move(source + files[data[i]],'fifth')","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.186761Z","iopub.status.idle":"2022-05-28T11:32:43.187352Z","shell.execute_reply.started":"2022-05-28T11:32:43.187119Z","shell.execute_reply":"2022-05-28T11:32:43.187139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://flint.cs.yale.edu/cs421/papers/x86-asm/asm.html\ndef firstprocess():\n    # https://en.wikipedia.org/wiki/Data_segment\n    # The prefixes tells about the segments that are present in the asm files\n    # There are 450 segments(approx) present in all asm files.\n    # These prefixes are best segments that gives us best values.\n    prefixes = ['HEADER:','.text:','.Pav:','.idata:','.data:','.bss:','.rdata:','.edata:','.rsrc:','.tls:','.reloc:','.BSS:','.CODE']\n    \n    # https://en.wikipedia.org/wiki/X86_instruction_listings\n    # These are opcodes that are used to get best results\n    opcodes = ['jmp', 'mov', 'retf', 'push', 'pop', 'xor', 'retn', 'nop', 'sub', 'inc', 'dec', 'add','imul', 'xchg', 'or', 'shr', 'cmp', 'call', 'shl', 'ror', 'rol', 'jnb','jz','rtn','lea','movzx']\n    \n    # Best keywords that are taken from different blogs\n    keywords = ['.dll','std::',':dword']\n    \n    # Below taken registers are general purpose registers and special registers\n    # All the registers which are taken are best \n    registers = ['edx','esi','eax','ebx','ecx','edi','ebp','esp','eip']\n    \n    file1 = open(\"output\\asmsmallfile.txt\",\"w+\")\n    files = os.listdir('first')\n    for f in files:\n        # Filling the values with zeros into the arrays\n        prefixescount = np.zeros(len(prefixes), dtype = int)\n        opcodescount = np.zeros(len(opcodes),dtype = int)\n        keywordcount = np.zeros(len(keywords),dtype = int)\n        registerscount = np.zeros(len(registers),dtype = int)\n        features = []\n        f2 = f.split('.')[0]\n        file1.write(f2 + \",\")\n        opcodefile.write(f2 + \" \")\n        \n        # https://docs.python.org/3/library/codecs.html#codecs.ignore_errors\n        # https://docs.python.org/3/library/codecs.html#codecs.Codec.encode\n        with codecs.open('first/' + f, encoding='cp1252', errors ='replace') as fli:\n            for lines in fli:\n                # https://www.tutorialspoint.com/python3/string_rstrip.htm\n                line = lines.rstrip().split()\n                l = line[0]\n                # Counting the prefixs in each and every line\n                for i in range(len(prefixes)):\n                    if prefixes[i] in line[0]:\n                        prefixescount[i] += 1\n                line = line[1:]\n                # Counting the opcodes in each and every line\n                for i in range(len(opcodes)):\n                    if any(opcodes[i] == li for li in line):\n                        features.append(opcodes[i])\n                        opcodescount[i] += 1\n                # Counting registers in the line\n                for i in range(len(registers)):\n                    for li in line:\n                        # We will use registers only in 'text' and 'CODE' segments\n                        if registers[i] in li and ('text' in l or 'CODE' in l):\n                            registerscount[i] += 1\n                # Counting keywords in the line\n                for i in range(len(keywords)):\n                    for li in line:\n                        if keywords[i] in li:\n                            keywordcount[i] += 1\n        # Pushing the values into the file after reading whole file\n        for prefix in prefixescount:\n            file1.write(str(prefix) + \",\")\n        for opcode in opcodescount:\n            file1.write(str(opcode) + \",\")\n        for register in registerscount:\n            file1.write(str(register) + \",\")\n        for key in keywordcount:\n            file1.write(str(key) + \",\")\n        file1.write(\"\\n\")\n    file1.close()\n\n# Same as above \ndef secondprocess():\n    prefixes = ['HEADER:','.text:','.Pav:','.idata:','.data:','.bss:','.rdata:','.edata:','.rsrc:','.tls:','.reloc:','.BSS:','.CODE']\n    opcodes = ['jmp', 'mov', 'retf', 'push', 'pop', 'xor', 'retn', 'nop', 'sub', 'inc', 'dec', 'add','imul', 'xchg', 'or', 'shr', 'cmp', 'call', 'shl', 'ror', 'rol', 'jnb','jz','rtn','lea','movzx']\n    keywords = ['.dll','std::',':dword']\n    registers = ['edx','esi','eax','ebx','ecx','edi','ebp','esp','eip']\n    file1 = open(\"output\\mediumasmfile.txt\",\"w+\")\n    files = os.listdir('second')\n    for f in files:\n        prefixescount = np.zeros(len(prefixes), dtype = int)\n        opcodescount = np.zeros(len(opcodes), dtype = int)\n        keywordcount = np.zeros(len(keywords), dtype = int)\n        registerscount = np.zeros(len(registers), dtype = int)\n        features  =[]\n        f2 = f.split('.')[0]\n        file1.write(f2 + \",\")\n        opcodefile.write(f2 + \" \")\n        with codecs.open('second/' + f, encoding = 'cp1252',errors = 'replace') as fli:\n            for lines in fli:\n                line = lines.rstrip().split()\n                l = line[0]\n                for i in range(len(prefixes)):\n                    if prefixes[i] in line[0]:\n                        prefixescount[i] += 1\n                line = line[1:]\n                for i in range(len(opcodes)):\n                    if any(opcodes[i] == li for li in line):\n                        features.append(opcodes[i])\n                        opcodescount[i] += 1\n                for i in range(len(registers)):\n                    for li in line:\n                        if registers[i] in li and ('text' in l or 'CODE' in l):\n                            registerscount[i] += 1\n                for i in range(len(keywords)):\n                    for li in line:\n                        if keywords[i] in li:\n                            keywordcount[i] += 1\n        for prefix in prefixescount:\n            file1.write(str(prefix) + \",\")\n        for opcode in opcodescount:\n            file1.write(str(opcode) + \",\")\n        for register in registerscount:\n            file1.write(str(register) + \",\")\n        for key in keywordcount:\n            file1.write(str(key) + \",\")\n        file1.write(\"\\n\")\n    file1.close()\n\n# Same as secondprocess() function\ndef thirdprocess():\n    prefixes = ['HEADER:','.text:','.Pav:','.idata:','.data:','.bss:','.rdata:','.edata:','.rsrc:','.tls:','.reloc:','.BSS:','.CODE']\n    opcodes = ['jmp', 'mov', 'retf', 'push', 'pop', 'xor', 'retn', 'nop', 'sub', 'inc', 'dec', 'add','imul', 'xchg', 'or', 'shr', 'cmp', 'call', 'shl', 'ror', 'rol', 'jnb','jz','rtn','lea','movzx']\n    keywords = ['.dll','std::',':dword']\n    registers=['edx','esi','eax','ebx','ecx','edi','ebp','esp','eip']\n    file1 = open(\"output\\largeasmfile.txt\",\"w+\")\n    files = os.listdir('third')\n    for f in files:\n        prefixescount = np.zeros(len(prefixes), dtype = int)\n        opcodescount = np.zeros(len(opcodes), dtype = int)\n        keywordcount = np.zeros(len(keywords), dtype = int)\n        registerscount = np.zeros(len(registers), dtype = int)\n        features = []\n        f2 = f.split('.')[0]\n        file1.write(f2 + \",\")\n        opcodefile.write(f2 + \" \")\n        with codecs.open('third/' + f, encoding = 'cp1252', errors ='replace') as fli:\n            for lines in fli:\n                line = lines.rstrip().split()\n                l = line[0]\n                for i in range(len(prefixes)):\n                    if prefixes[i] in line[0]:\n                        prefixescount[i] += 1\n                line = line[1:]\n                for i in range(len(opcodes)):\n                    if any(opcodes[i] == li for li in line):\n                        features.append(opcodes[i])\n                        opcodescount[i] += 1\n                for i in range(len(registers)):\n                    for li in line:\n                        if registers[i] in li and ('text' in l or 'CODE' in l):\n                            registerscount[i] += 1\n                for i in range(len(keywords)):\n                    for li in line:\n                        if keywords[i] in li:\n                            keywordcount[i] += 1\n        for prefix in prefixescount:\n            file1.write(str(prefix) + \",\")\n        for opcode in opcodescount:\n            file1.write(str(opcode) + \",\")\n        for register in registerscount:\n            file1.write(str(register) + \",\")\n        for key in keywordcount:\n            file1.write(str(key) + \",\")\n        file1.write(\"\\n\")\n    file1.close()\n\ndef fourthprocess():\n    prefixes = ['HEADER:','.text:','.Pav:','.idata:','.data:','.bss:','.rdata:','.edata:','.rsrc:','.tls:','.reloc:','.BSS:','.CODE']\n    opcodes = ['jmp', 'mov', 'retf', 'push', 'pop', 'xor', 'retn', 'nop', 'sub', 'inc', 'dec', 'add','imul', 'xchg', 'or', 'shr', 'cmp', 'call', 'shl', 'ror', 'rol', 'jnb','jz','rtn','lea','movzx']\n    keywords = ['.dll','std::',':dword']\n    registers = ['edx','esi','eax','ebx','ecx','edi','ebp','esp','eip']\n    file1 = open(\"output\\hugeasmfile.txt\",\"w+\")\n    files = os.listdir('fourth/')\n    for f in files:\n        prefixescount = np.zeros(len(prefixes), dtype = int)\n        opcodescount = np.zeros(len(opcodes), dtype = int)\n        keywordcount = np.zeros(len(keywords), dtype = int)\n        registerscount = np.zeros(len(registers), dtype = int)\n        features = []\n        f2 = f.split('.')[0]\n        file1.write(f2 + \",\")\n        opcodefile.write(f2 + \" \")\n        with codecs.open('fourth/' + f, encoding = 'cp1252', errors ='replace') as fli:\n            for lines in fli:\n                line = lines.rstrip().split()\n                l = line[0]\n                for i in range(len(prefixes)):\n                    if prefixes[i] in line[0]:\n                        prefixescount[i] += 1\n                line = line[1:]\n                for i in range(len(opcodes)):\n                    if any(opcodes[i] == li for li in line):\n                        features.append(opcodes[i])\n                        opcodescount[i] += 1\n                for i in range(len(registers)):\n                    for li in line:\n                        if registers[i] in li and ('text' in l or 'CODE' in l):\n                            registerscount[i] += 1\n                for i in range(len(keywords)):\n                    for li in line:\n                        if keywords[i] in li:\n                            keywordcount[i] += 1\n        for prefix in prefixescount:\n            file1.write(str(prefix) + \",\")\n        for opcode in opcodescount:\n            file1.write(str(opcode) + \",\")\n        for register in registerscount:\n            file1.write(str(register) + \",\")\n        for key in keywordcount:\n            file1.write(str(key) + \",\")\n        file1.write(\"\\n\")\n    file1.close()\n\ndef fifthprocess():\n    prefixes = ['HEADER:','.text:','.Pav:','.idata:','.data:','.bss:','.rdata:','.edata:','.rsrc:','.tls:','.reloc:','.BSS:','.CODE']\n    opcodes = ['jmp', 'mov', 'retf', 'push', 'pop', 'xor', 'retn', 'nop', 'sub', 'inc', 'dec', 'add','imul', 'xchg', 'or', 'shr', 'cmp', 'call', 'shl', 'ror', 'rol', 'jnb','jz','rtn','lea','movzx']\n    keywords = ['.dll','std::',':dword']\n    registers = ['edx','esi','eax','ebx','ecx','edi','ebp','esp','eip']\n    file1 = open(\"output\\trainasmfile.txt\",\"w+\")\n    files = os.listdir('fifth/')\n    for f in files:\n        prefixescount = np.zeros(len(prefixes), dtype = int)\n        opcodescount = np.zeros(len(opcodes), dtype = int)\n        keywordcount = np.zeros(len(keywords), dtype = int)\n        registerscount = np.zeros(len(registers), dtype = int)\n        features = []\n        f2 = f.split('.')[0]\n        file1.write(f2 + \",\")\n        opcodefile.write(f2 + \" \")\n        with codecs.open('fifth/' + f, encoding = 'cp1252', errors ='replace') as fli:\n            for lines in fli:\n                line = lines.rstrip().split()\n                l = line[0]\n                for i in range(len(prefixes)):\n                    if prefixes[i] in line[0]:\n                        prefixescount[i] += 1\n                line = line[1:]\n                for i in range(len(opcodes)):\n                    if any(opcodes[i] == li for li in line):\n                        features.append(opcodes[i])\n                        opcodescount[i] += 1\n                for i in range(len(registers)):\n                    for li in line:\n                        if registers[i] in li and ('text' in l or 'CODE' in l):\n                            registerscount[i] += 1\n                for i in range(len(keywords)):\n                    for li in line:\n                        if keywords[i] in li:\n                            keywordcount[i] += 1\n        for prefix in prefixescount:\n            file1.write(str(prefix) + \",\")\n        for opcode in opcodescount: \n            file1.write(str(opcode) + \",\")\n        for register in registerscount:\n            file1.write(str(register) + \",\")\n        for key in keywordcount:\n            file1.write(str(key) + \",\")\n        file1.write(\"\\n\")\n    file1.close()\n\ndef main():\n    # The below code is used for multi-processing\n    # The number of processes depends upon the number of cores present in the system\n    # Process is used to call multi-processing\n    manager = multiprocessing.Manager() \t\n    p1 = Process(target = firstprocess)\n    p2 = Process(target = secondprocess)\n    p3 = Process(target = thirdprocess)\n    p4 = Process(target = fourthprocess)\n    p5 = Process(target = fifthprocess)\n    \n    # p1.start() is used to start the thread execution\n    p1.start()\n    p2.start()\n    p3.start()\n    p4.start()\n    p5.start()\n    \n    # After completion all the threads are joined\n    p1.join()\n    p2.join()\n    p3.join()\n    p4.join()\n    p5.join()\n\nif __name__ == \"__main__\":\n    main()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.190965Z","iopub.status.idle":"2022-05-28T11:32:43.191571Z","shell.execute_reply.started":"2022-05-28T11:32:43.191344Z","shell.execute_reply":"2022-05-28T11:32:43.191367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# asmoutputfile.csv (output generated from the above two cells) will contain all the extracted \n# features from .asm files.\ndfasm = pd.read_csv(\"asmoutputfile.csv\")\nY.columns = ['ID', 'Class']\nresult_asm = pd.merge(dfasm, Y, on = 'ID', how = 'left')\nresult_asm.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.192809Z","iopub.status.idle":"2022-05-28T11:32:43.195651Z","shell.execute_reply.started":"2022-05-28T11:32:43.19541Z","shell.execute_reply":"2022-05-28T11:32:43.195434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 4.2.1.1 Files sizes of each .asm file","metadata":{}},{"cell_type":"code","source":"# File sizes of byte files\nfiles = os.listdir('asmFiles')\nfilenames = Y['ID'].tolist()\nclass_y = Y['Class'].tolist()\nclass_bytes = []\nsizebytes = []\nfnames = []\nfor file in files:\n    # print(os.stat('byteFiles/0A32eTdBKayjCWhZqDOQ.txt'))\n    # os.stat_result(st_mode=33206, st_ino=1125899906874507, st_dev=3561571700, st_nlink=1, st_uid=0, st_gid=0, \n    # st_size=3680109, st_atime=1519638522, st_mtime=1519638522, st_ctime=1519638522)\n    # here https://www.tutorialspoint.com/python/os_stat.htm\n    statinfo = os.stat('asmFiles/' + file)\n    # Split the file name at '.' and take the first part of it i.e the file name\n    file = file.split('.')[0]\n    if any(file == filename for filename in filenames):\n        i = filenames.index(file)\n        class_bytes.append(class_y[i])\n        # Converting into Mb's\n        sizebytes.append(statinfo.st_size / (1024.0*1024.0))\n        fnames.append(file)\nasm_size_byte = pd.DataFrame({'ID':fnames, 'size':sizebytes, 'Class':class_bytes})\nprint (asm_size_byte.head())","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.196764Z","iopub.status.idle":"2022-05-28T11:32:43.197447Z","shell.execute_reply.started":"2022-05-28T11:32:43.197132Z","shell.execute_reply":"2022-05-28T11:32:43.197154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 4.2.1.2 Distribution of .asm file sizes","metadata":{}},{"cell_type":"code","source":"# Boxplot of asm files\nax = sns.boxplot(x = \"Class\", y = \"size\", data = asm_size_byte)\nplt.title(\"Boxplot of .asm file sizes\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.198628Z","iopub.status.idle":"2022-05-28T11:32:43.199209Z","shell.execute_reply.started":"2022-05-28T11:32:43.198995Z","shell.execute_reply":"2022-05-28T11:32:43.199016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add the file size feature to previous extracted features\nprint(result_asm.shape)\nprint(asm_size_byte.shape)\nresult_asm = pd.merge(result_asm, asm_size_byte.drop(['Class'], axis=1), on='ID', how='left')\nresult_asm.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.200405Z","iopub.status.idle":"2022-05-28T11:32:43.201056Z","shell.execute_reply.started":"2022-05-28T11:32:43.200778Z","shell.execute_reply":"2022-05-28T11:32:43.2008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We normalize the data of each column \nresult_asm = normalize(result_asm)\nresult_asm.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.20223Z","iopub.status.idle":"2022-05-28T11:32:43.202871Z","shell.execute_reply.started":"2022-05-28T11:32:43.20264Z","shell.execute_reply":"2022-05-28T11:32:43.202662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.2.2 Univariate analysis on  asm file features","metadata":{}},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \".text:\", data = result_asm)\nplt.title(\"Boxplot of .asm text segment\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.204075Z","iopub.status.idle":"2022-05-28T11:32:43.204691Z","shell.execute_reply.started":"2022-05-28T11:32:43.204454Z","shell.execute_reply":"2022-05-28T11:32:43.204476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \".Pav:\", data = result_asm)\nplt.title(\"Boxplot of .asm pav segment\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.205843Z","iopub.status.idle":"2022-05-28T11:32:43.206535Z","shell.execute_reply.started":"2022-05-28T11:32:43.20622Z","shell.execute_reply":"2022-05-28T11:32:43.206242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \".data:\", data = result_asm)\nplt.title(\"Boxplot of .asm data segment\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.207786Z","iopub.status.idle":"2022-05-28T11:32:43.208436Z","shell.execute_reply.started":"2022-05-28T11:32:43.208198Z","shell.execute_reply":"2022-05-28T11:32:43.20822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \".bss:\", data = result_asm)\nplt.title(\"Boxplot of .asm bss segment\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.209733Z","iopub.status.idle":"2022-05-28T11:32:43.210362Z","shell.execute_reply.started":"2022-05-28T11:32:43.210126Z","shell.execute_reply":"2022-05-28T11:32:43.210148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \".rdata:\", data = result_asm)\nplt.title(\"Boxplot of .asm rdata segment\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.211617Z","iopub.status.idle":"2022-05-28T11:32:43.212245Z","shell.execute_reply.started":"2022-05-28T11:32:43.212013Z","shell.execute_reply":"2022-05-28T11:32:43.212037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \"jmp\", data = result_asm)\nplt.title(\"Boxplot of .asm jmp opcode\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.213474Z","iopub.status.idle":"2022-05-28T11:32:43.214248Z","shell.execute_reply.started":"2022-05-28T11:32:43.213952Z","shell.execute_reply":"2022-05-28T11:32:43.213975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \"mov\", data = result_asm)\nplt.title(\"Boxplot of .asm mov opcode\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.21552Z","iopub.status.idle":"2022-05-28T11:32:43.216118Z","shell.execute_reply.started":"2022-05-28T11:32:43.215887Z","shell.execute_reply":"2022-05-28T11:32:43.21591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \"retf\", data = result_asm)\nplt.title(\"Boxplot of .asm retf opcode\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.217279Z","iopub.status.idle":"2022-05-28T11:32:43.217909Z","shell.execute_reply.started":"2022-05-28T11:32:43.217683Z","shell.execute_reply":"2022-05-28T11:32:43.217704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax = sns.boxplot(x = \"Class\", y = \"push\", data = result_asm)\nplt.title(\"Boxplot of .asm push opcode\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.219079Z","iopub.status.idle":"2022-05-28T11:32:43.219706Z","shell.execute_reply.started":"2022-05-28T11:32:43.21947Z","shell.execute_reply":"2022-05-28T11:32:43.219493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.2.2 Multivariate Analysis on .asm file features","metadata":{}},{"cell_type":"code","source":"# Multivariate analysis on byte files\n# This is with perplexity 50\nxtsne = TSNE(perplexity = 50)\nresults = xtsne.fit_transform(result_asm.drop(['ID','Class'], axis = 1).fillna(0))\nvis_x = results[:, 0]\nvis_y = results[:, 1   ]\nplt.scatter(vis_x, vis_y, c = data_y, cmap = plt.cm.get_cmap(\"jet\", 9))\nplt.colorbar(ticks = range(10))\nplt.clim(0.5, 9)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.221074Z","iopub.status.idle":"2022-05-28T11:32:43.221698Z","shell.execute_reply.started":"2022-05-28T11:32:43.221469Z","shell.execute_reply":"2022-05-28T11:32:43.22149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# By Univariate analysis on the .asm file features, we are getting very negligible information from \n# 'rtn', '.BSS:' '.CODE' features, so here, we are trying multivariate analysis after removing \n# those features. The plot looks very messy\n\nxtsne = TSNE(perplexity = 30)\nresults = xtsne.fit_transform(result_asm.drop(['ID','Class','rtn','.BSS:','.CODE','size'], axis = 1))\nvis_x = results[:, 0]\nvis_y = results[:, 1]\nplt.scatter(vis_x, vis_y, c = data_y, cmap = plt.cm.get_cmap(\"jet\", 9))\nplt.colorbar(ticks = range(10))\nplt.clim(0.5, 9)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.223017Z","iopub.status.idle":"2022-05-28T11:32:43.223672Z","shell.execute_reply.started":"2022-05-28T11:32:43.223438Z","shell.execute_reply":"2022-05-28T11:32:43.223461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.2.3 Conclusion on EDA\n- We have taken only **52 features** from asm files (after reading through many blogs and research papers).\n- The univariate analysis was done only on few important features.\n- Take-aways:\n    1. Class 3 can be easily separated because of the frequency of segments, opcodes and keywords being less.\n    2. Each feature has its unique importance in separating the class labels.","metadata":{}},{"cell_type":"markdown","source":"## 4.3 Train and test split","metadata":{}},{"cell_type":"code","source":"asm_y = result_asm['Class']\nasm_x = result_asm.drop(['ID','Class','.BSS:','rtn','.CODE'], axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.224771Z","iopub.status.idle":"2022-05-28T11:32:43.225502Z","shell.execute_reply.started":"2022-05-28T11:32:43.225231Z","shell.execute_reply":"2022-05-28T11:32:43.225255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_asm, X_test_asm, y_train_asm, y_test_asm = train_test_split(asm_x,asm_y ,stratify = asm_y, test_size = 0.20)\nX_train_asm, X_cv_asm, y_train_asm, y_cv_asm = train_test_split(X_train_asm, y_train_asm, stratify = y_train_asm, test_size = 0.20)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.226631Z","iopub.status.idle":"2022-05-28T11:32:43.227212Z","shell.execute_reply.started":"2022-05-28T11:32:43.226994Z","shell.execute_reply":"2022-05-28T11:32:43.227015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print( X_cv_asm.isnull().all())","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.228564Z","iopub.status.idle":"2022-05-28T11:32:43.229155Z","shell.execute_reply.started":"2022-05-28T11:32:43.228933Z","shell.execute_reply":"2022-05-28T11:32:43.228954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.4. Machine Learning models on features of .asm files\n### 4.4.1 K-Nearest Neigbors","metadata":{}},{"cell_type":"code","source":"# http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html\nalpha = [x for x in range(1, 21,2)]\ncv_log_error_array = []\nfor i in alpha:\n    k_cfl = KNeighborsClassifier(n_neighbors = i)\n    k_cfl.fit(X_train_asm,y_train_asm)\n    sig_clf = CalibratedClassifierCV(k_cfl, method = \"sigmoid\")\n    sig_clf.fit(X_train_asm, y_train_asm)\n    predict_y = sig_clf.predict_proba(X_cv_asm)\n    cv_log_error_array.append(log_loss(y_cv_asm, predict_y, labels = k_cfl.classes_, eps = 1e-15))\n    \nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for k = ', alpha[i], 'is', cv_log_error_array[i])\nbest_alpha = np.argmin(cv_log_error_array)\n    \nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nk_cfl = KNeighborsClassifier(n_neighbors = alpha[best_alpha])\nk_cfl.fit(X_train_asm,y_train_asm)\nsig_clf = CalibratedClassifierCV(k_cfl, method = \"sigmoid\")\nsig_clf.fit(X_train_asm, y_train_asm)\npred_y = sig_clf.predict(X_test_asm)\n\npredict_y = sig_clf.predict_proba(X_train_asm)\nprint ('log loss for train data', log_loss(y_train_asm, predict_y))\npredict_y = sig_clf.predict_proba(X_cv_asm)\nprint ('log loss for cv data', log_loss(y_cv_asm, predict_y))\npredict_y = sig_clf.predict_proba(X_test_asm)\nprint ('log loss for test data', log_loss(y_test_asm, predict_y))\nplot_confusion_matrix(y_test_asm, sig_clf.predict(X_test_asm))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.230372Z","iopub.status.idle":"2022-05-28T11:32:43.230971Z","shell.execute_reply.started":"2022-05-28T11:32:43.230744Z","shell.execute_reply":"2022-05-28T11:32:43.230766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.4.2 Logistic Regression","metadata":{}},{"cell_type":"code","source":"# http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html\nalpha = [10 ** x for x in range(-5, 4)]\ncv_log_error_array = []\nfor i in alpha:\n    logisticR = LogisticRegression(penalty = 'l2', C = i, class_weight = 'balanced')\n    logisticR.fit(X_train_asm,y_train_asm)\n    sig_clf = CalibratedClassifierCV(logisticR, method = \"sigmoid\")\n    sig_clf.fit(X_train_asm, y_train_asm)\n    predict_y = sig_clf.predict_proba(X_cv_asm)\n    cv_log_error_array.append(log_loss(y_cv_asm, predict_y, labels = logisticR.classes_, eps = 1e-15))\n    \nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\nbest_alpha = np.argmin(cv_log_error_array)\n    \nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nlogisticR = LogisticRegression(penalty = 'l2',C = alpha[best_alpha], class_weight = 'balanced')\nlogisticR.fit(X_train_asm,y_train_asm)\nsig_clf = CalibratedClassifierCV(logisticR, method = \"sigmoid\")\nsig_clf.fit(X_train_asm, y_train_asm)\n\npredict_y = sig_clf.predict_proba(X_train_asm)\nprint ('log loss for train data', (log_loss(y_train_asm, predict_y, labels = logisticR.classes_, eps = 1e-15)))\npredict_y = sig_clf.predict_proba(X_cv_asm)\nprint ('log loss for cv data', (log_loss(y_cv_asm, predict_y, labels = logisticR.classes_, eps = 1e-15)))\npredict_y = sig_clf.predict_proba(X_test_asm)\nprint ('log loss for test data',(log_loss(y_test_asm, predict_y, labels = logisticR.classes_, eps = 1e-15)))\nplot_confusion_matrix(y_test_asm, sig_clf.predict(X_test_asm))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.232216Z","iopub.status.idle":"2022-05-28T11:32:43.232843Z","shell.execute_reply.started":"2022-05-28T11:32:43.232615Z","shell.execute_reply":"2022-05-28T11:32:43.232637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.4.3 Random Forest Classifier","metadata":{}},{"cell_type":"code","source":"alpha = [10,50,100,500,1000,2000,3000]\ncv_log_error_array = []\nfor i in alpha:\n    r_cfl = RandomForestClassifier(n_estimators = i,random_state = 42,n_jobs = -1)\n    r_cfl.fit(X_train_asm, y_train_asm)\n    sig_clf = CalibratedClassifierCV(r_cfl, method = \"sigmoid\")\n    sig_clf.fit(X_train_asm, y_train_asm)\n    predict_y = sig_clf.predict_proba(X_cv_asm)\n    cv_log_error_array.append(log_loss(y_cv_asm, predict_y, labels = r_cfl.classes_, eps = 1e-15))\n\nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\nbest_alpha = np.argmin(cv_log_error_array)\n\nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nr_cfl = RandomForestClassifier(n_estimators = alpha[best_alpha], random_state = 42, n_jobs = -1)\nr_cfl.fit(X_train_asm, y_train_asm)\nsig_clf = CalibratedClassifierCV(r_cfl, method = \"sigmoid\")\nsig_clf.fit(X_train_asm, y_train_asm)\npredict_y = sig_clf.predict_proba(X_train_asm)\nprint ('log loss for train data', (log_loss(y_train_asm, predict_y, labels = sig_clf.classes_, eps = 1e-15)))\npredict_y = sig_clf.predict_proba(X_cv_asm)\nprint ('log loss for cv data', (log_loss(y_cv_asm, predict_y, labels = sig_clf.classes_, eps = 1e-15)))\npredict_y = sig_clf.predict_proba(X_test_asm)\nprint ('log loss for test data', (log_loss(y_test_asm, predict_y, labels = sig_clf.classes_, eps = 1e-15)))\nplot_confusion_matrix(y_test_asm, sig_clf.predict(X_test_asm))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.234043Z","iopub.status.idle":"2022-05-28T11:32:43.234661Z","shell.execute_reply.started":"2022-05-28T11:32:43.234421Z","shell.execute_reply":"2022-05-28T11:32:43.234443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.4.4 XgBoost Classifier","metadata":{}},{"cell_type":"code","source":"# http://xgboost.readthedocs.io/en/latest/python/python_api.html?#xgboost.XGBClassifier\nalpha = [10,50,100,500,1000,2000,3000]\ncv_log_error_array = []\nfor i in alpha:\n    x_cfl = XGBClassifier(n_estimators = i, nthread = -1)\n    x_cfl.fit(X_train_asm, y_train_asm)\n    sig_clf = CalibratedClassifierCV(x_cfl, method = \"sigmoid\")\n    sig_clf.fit(X_train_asm, y_train_asm)\n    predict_y = sig_clf.predict_proba(X_cv_asm)\n    cv_log_error_array.append(log_loss(y_cv_asm, predict_y, labels = x_cfl.classes_, eps = 1e-15))\n\nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\nbest_alpha = np.argmin(cv_log_error_array)\n\nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nx_cfl = XGBClassifier(n_estimators = alpha[best_alpha], nthread = -1)\nx_cfl.fit(X_train_asm, y_train_asm)\nsig_clf = CalibratedClassifierCV(x_cfl, method = \"sigmoid\")\nsig_clf.fit(X_train_asm, y_train_asm)\npredict_y = sig_clf.predict_proba(X_train_asm)\nprint ('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train_asm, predict_y))\npredict_y = sig_clf.predict_proba(X_cv_asm)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv_asm, predict_y))\npredict_y = sig_clf.predict_proba(X_test_asm)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test_asm, predict_y))\nplot_confusion_matrix(y_test_asm, sig_clf.predict(X_test_asm))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.235928Z","iopub.status.idle":"2022-05-28T11:32:43.236544Z","shell.execute_reply.started":"2022-05-28T11:32:43.236303Z","shell.execute_reply":"2022-05-28T11:32:43.236338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.4.5 Xgboost Classifier with best hyperparameters","metadata":{}},{"cell_type":"code","source":"x_cfl = XGBClassifier()\n\nparams = {\n    'learning_rate': [0.01,0.03,0.05,0.1,0.15,0.2],\n     'n_estimators': [100,200,500,1000,2000],\n     'max_depth': [3,5,10],\n    'colsample_bytree': [0.1,0.3,0.5,1],\n    'subsample': [0.1,0.3,0.5,1]\n}\nrandom_cfl = RandomizedSearchCV(x_cfl, param_distributions = params, verbose = 10, n_jobs = -1,)\nrandom_cfl.fit(X_train_asm,y_train_asm)\nprint(random_cfl.best_params_)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.23769Z","iopub.status.idle":"2022-05-28T11:32:43.238555Z","shell.execute_reply.started":"2022-05-28T11:32:43.238194Z","shell.execute_reply":"2022-05-28T11:32:43.238217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://xgboost.readthedocs.io/en/latest/python/python_api.html?#xgboost.XGBClassifier\nx_cfl = XGBClassifier(n_estimators = 200, subsample = 0.5, learning_rate = 0.15, \n  colsample_bytree = 0.5, max_depth = 3)\nx_cfl.fit(X_train_asm,y_train_asm)\nc_cfl = CalibratedClassifierCV(x_cfl, method = 'sigmoid')\nc_cfl.fit(X_train_asm, y_train_asm)\n\npredict_y = c_cfl.predict_proba(X_train_asm)\nprint ('train loss', log_loss(y_train_asm, predict_y))\npredict_y = c_cfl.predict_proba(X_cv_asm)\nprint ('cv loss', log_loss(y_cv_asm, predict_y))\npredict_y = c_cfl.predict_proba(X_test_asm)\nprint ('test loss', log_loss(y_test_asm, predict_y))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.240196Z","iopub.status.idle":"2022-05-28T11:32:43.241076Z","shell.execute_reply.started":"2022-05-28T11:32:43.240722Z","shell.execute_reply":"2022-05-28T11:32:43.240746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.5. Machine Learning models on features of both .asm and .bytes files\n### 4.5.1. Merging both asm and byte file features","metadata":{}},{"cell_type":"code","source":"print(result.shape)\nresult.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.243972Z","iopub.status.idle":"2022-05-28T11:32:43.244695Z","shell.execute_reply.started":"2022-05-28T11:32:43.244455Z","shell.execute_reply":"2022-05-28T11:32:43.24448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(result_asm.shape)\nresult_asm.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.245872Z","iopub.status.idle":"2022-05-28T11:32:43.246483Z","shell.execute_reply.started":"2022-05-28T11:32:43.246238Z","shell.execute_reply":"2022-05-28T11:32:43.24626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_x = pd.merge(result,result_asm.drop(['Class'], axis = 1), on = 'ID', how = 'left')\nresult_y = result_x['Class']\nresult_x = result_x.drop(['ID','rtn','.BSS:','.CODE','Class'], axis = 1)\nresult_x.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.247702Z","iopub.status.idle":"2022-05-28T11:32:43.248284Z","shell.execute_reply.started":"2022-05-28T11:32:43.248067Z","shell.execute_reply":"2022-05-28T11:32:43.248089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.5.2. Multivariate Analysis on final fearures","metadata":{}},{"cell_type":"code","source":"xtsne = TSNE(perplexity=50)\nresults = xtsne.fit_transform(result_x, axis=1)\nvis_x = results[:, 0]\nvis_y = results[:, 1]\nplt.scatter(vis_x, vis_y, c=result_y, cmap=plt.cm.get_cmap(\"jet\", 9))\nplt.colorbar(ticks=range(9))\nplt.clim(0.5, 9)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.249583Z","iopub.status.idle":"2022-05-28T11:32:43.250164Z","shell.execute_reply.started":"2022-05-28T11:32:43.249946Z","shell.execute_reply":"2022-05-28T11:32:43.249967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.5.3. Train and Test split","metadata":{}},{"cell_type":"code","source":"X_train, X_test_merge, y_train, y_test_merge = train_test_split(result_x, result_y, \n    stratify=result_y, test_size=0.20)\nX_train_merge, X_cv_merge, y_train_merge, y_cv_merge = train_test_split(X_train, y_train,\n    stratify=y_train, test_size=0.20)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.251268Z","iopub.status.idle":"2022-05-28T11:32:43.251926Z","shell.execute_reply.started":"2022-05-28T11:32:43.251685Z","shell.execute_reply":"2022-05-28T11:32:43.251707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.5.4. Random Forest Classifier on final features","metadata":{}},{"cell_type":"code","source":"alpha = [10,50,100,500,1000,2000,3000]\ncv_log_error_array = []\nfrom sklearn.ensemble import RandomForestClassifier\nfor i in alpha:\n    r_cfl = RandomForestClassifier(n_estimators=i, random_state=42, n_jobs=-1)\n    r_cfl.fit(X_train_merge, y_train_merge)\n    sig_clf = CalibratedClassifierCV(r_cfl, method=\"sigmoid\")\n    sig_clf.fit(X_train_merge, y_train_merge)\n    predict_y = sig_clf.predict_proba(X_cv_merge)\n    cv_log_error_array.append(log_loss(y_cv_merge, predict_y, labels=r_cfl.classes_, eps=1e-15))\n\nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\nbest_alpha = np.argmin(cv_log_error_array)\n\nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c = 'g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i], cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nr_cfl = RandomForestClassifier(n_estimators=alpha[best_alpha], random_state=42, n_jobs=-1)\nr_cfl.fit(X_train_merge,y_train_merge)\nsig_clf = CalibratedClassifierCV(r_cfl, method=\"sigmoid\")\nsig_clf.fit(X_train_merge, y_train_merge)\n\npredict_y = sig_clf.predict_proba(X_train_merge)\nprint ('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train_merge, predict_y))\npredict_y = sig_clf.predict_proba(X_cv_merge)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv_merge, predict_y))\npredict_y = sig_clf.predict_proba(X_test_merge)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test_merge, predict_y))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.253115Z","iopub.status.idle":"2022-05-28T11:32:43.253751Z","shell.execute_reply.started":"2022-05-28T11:32:43.253523Z","shell.execute_reply":"2022-05-28T11:32:43.253545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.5.5. XgBoost Classifier on final features","metadata":{}},{"cell_type":"code","source":"# http://xgboost.readthedocs.io/en/latest/python/python_api.html?#xgboost.XGBClassifier\nalpha = [10,50,100,500,1000,2000,3000]\ncv_log_error_array = []\nfor i in alpha:\n    x_cfl = XGBClassifier(n_estimators = i)\n    x_cfl.fit(X_train_merge, y_train_merge)\n    sig_clf = CalibratedClassifierCV(x_cfl, method=\"sigmoid\")\n    sig_clf.fit(X_train_merge, y_train_merge)\n    predict_y = sig_clf.predict_proba(X_cv_merge)\n    cv_log_error_array.append(log_loss(y_cv_merge, predict_y, labels = x_cfl.classes_, eps = 1e-15))\n\nfor i in range(len(cv_log_error_array)):\n    print ('log_loss for c = ', alpha[i], 'is', cv_log_error_array[i])\nbest_alpha = np.argmin(cv_log_error_array)\n\nfig, ax = plt.subplots()\nax.plot(alpha, cv_log_error_array, c='g')\nfor i, txt in enumerate(np.round(cv_log_error_array, 3)):\n    ax.annotate((alpha[i], np.round(txt, 3)), (alpha[i] ,cv_log_error_array[i]))\nplt.grid()\nplt.title(\"Cross Validation Error for each alpha\")\nplt.xlabel(\"Alpha i's\")\nplt.ylabel(\"Error measure\")\nplt.show()\n\nx_cfl = XGBClassifier(n_estimators=3000, nthread=-1)\nx_cfl.fit(X_train_merge, y_train_merge, verbose=True)\nsig_clf = CalibratedClassifierCV(x_cfl, method=\"sigmoid\")\nsig_clf.fit(X_train_merge, y_train_merge)\n\npredict_y = sig_clf.predict_proba(X_train_merge)\nprint ('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train_merge, predict_y))\npredict_y = sig_clf.predict_proba(X_cv_merge)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv_merge, predict_y))\npredict_y = sig_clf.predict_proba(X_test_merge)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test_merge, predict_y))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.254933Z","iopub.status.idle":"2022-05-28T11:32:43.255542Z","shell.execute_reply.started":"2022-05-28T11:32:43.255288Z","shell.execute_reply":"2022-05-28T11:32:43.255327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.5.5. XgBoost Classifier on final features with best hyper parameters using Random search","metadata":{}},{"cell_type":"code","source":"x_cfl = XGBClassifier()\nparams = {\n    'learning_rate': [0.01,0.03,0.05,0.1,0.15,0.2],\n     'n_estimators': [100,200,500,1000,2000],\n     'max_depth': [3,5,10],\n    'colsample_bytree': [0.1,0.3,0.5,1],\n    'subsample': [0.1,0.3,0.5,1]\n}\nrandom_cfl = RandomizedSearchCV(x_cfl, param_distributions=params, verbose=10, n_jobs=-1)\nrandom_cfl.fit(X_train_merge, y_train_merge)\nprint (random_cfl.best_params_)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.256698Z","iopub.status.idle":"2022-05-28T11:32:43.2573Z","shell.execute_reply.started":"2022-05-28T11:32:43.257073Z","shell.execute_reply":"2022-05-28T11:32:43.257095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# http://xgboost.readthedocs.io/en/latest/python/python_api.html?#xgboost.XGBClassifier\nx_cfl = XGBClassifier(n_estimators=1000, max_depth=10, learning_rate=0.15, \n    colsample_bytree=0.3, subsample=1, nthread=-1)\nx_cfl.fit(X_train_merge, y_train_merge, verbose=True)\nsig_clf = CalibratedClassifierCV(x_cfl, method=\"sigmoid\")\nsig_clf.fit(X_train_merge, y_train_merge)\n    \npredict_y = sig_clf.predict_proba(X_train_merge)\nprint ('For values of best alpha = ', alpha[best_alpha], \"The train log loss is:\", log_loss(y_train_merge, predict_y))\npredict_y = sig_clf.predict_proba(X_cv_merge)\nprint('For values of best alpha = ', alpha[best_alpha], \"The cross validation log loss is:\", log_loss(y_cv_merge, predict_y))\npredict_y = sig_clf.predict_proba(X_test_merge)\nprint('For values of best alpha = ', alpha[best_alpha], \"The test log loss is:\", log_loss(y_test_merge, predict_y))\nplot_confusion_matrix(y_test_asm,sig_clf.predict(X_test_merge))","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:32:43.258567Z","iopub.status.idle":"2022-05-28T11:32:43.259262Z","shell.execute_reply.started":"2022-05-28T11:32:43.259034Z","shell.execute_reply":"2022-05-28T11:32:43.259056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Assignments\n- Add bi-grams and n-gram features on byte files and improve the log-loss\n- Using the 'dchad' [github](https://github.com/dchad/malware-detection) account, in order to decrease the logloss to 0.01.\n- Watch the [video](https://www.youtube.com/watch?v=VLQTRlLGz5Y ) that was in reference section and implement the image features to improve the logloss.","metadata":{}}]}