{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"provenance":[]},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":4117,"databundleVersionId":46665,"sourceType":"competition"},{"sourceId":12280887,"sourceType":"datasetVersion","datasetId":7739469},{"sourceId":12280907,"sourceType":"datasetVersion","datasetId":7739486},{"sourceId":12280940,"sourceType":"datasetVersion","datasetId":7739513}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n

\n Microsoft Malware Classification Challenge\n

","metadata":{"_uuid":"7cf0d683-5302-4988-a885-f0874203e9c0","_cell_guid":"87dcc814-8762-4393-a03c-054754c130fa","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"","metadata":{"_uuid":"c6ed254b-5b4b-4cbd-9c8d-536b01d06d22","_cell_guid":"ee3824c7-16f9-4f5f-86eb-51b3b5e1d19b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

Dataset Description

","metadata":{"_uuid":"eb3c7797-2ec8-4b9b-ae08-eb29dd4e4f44","_cell_guid":"e808e0d1-e856-467d-971f-a252d6b2d22b","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"> ⚠️ **Warning:** This dataset is almost **half a terabyte uncompressed**! \n> We have compressed the data using **7zip** to achieve the smallest file size possible. \n> **Note:** The rules do **not allow sharing** of the data outside of **Kaggle**, including **BitTorrent**.
\n> Link : https://www.kaggle.com/c/malware-classification/data","metadata":{"_uuid":"40751edb-10ef-484c-85dc-da8d89a8e662","_cell_guid":"6f27f9eb-ea26-49c7-aa8a-80f2280c5969","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 📂 Dataset Overview\n\n- **Train Set**: Hexadecimal and disassembly log files for malware samples \n- **Labels**: Provided in `trainLabels.csv` (class integers 1–9) \n- **Test Set**: Similar structure, without labels \n- **Sample Submission**: Format provided in `sampleSubmission.csv` \n- **Preview**: `dataSample.csv` allows for a sneak peek before full download","metadata":{"_uuid":"561a2b01-4a98-4b68-a39a-bf795464df46","_cell_guid":"a64cf6d2-0540-4da4-8556-92ed9d632273","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 👾 Malware Families\n\n- **Ramnit** \n- **Lollipop** \n- **Kelihos_ver3** \n- **Vundo** \n- **Simda** \n- **Tracur** \n- **Kelihos_ver1** \n- **Obfuscator.ACY** \n- **Gatak**","metadata":{"_uuid":"22f9f349-d648-4f8e-8cfd-7ca250dff730","_cell_guid":"a8181827-eeca-42f8-89ac-3f52b1247c73","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"\n

\n Problem Statement\n

","metadata":{"_uuid":"98b4b925-4f5c-419e-a250-00c8ebaae5a7","_cell_guid":"6e096de3-10c0-4c30-908d-1378367a618a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## 🛡️ Malware Risk Forecasting\r\n\r\nWith the increasing number of cyberattacks, malware remains one of the biggest threats to digital security. Traditional malware detection relies on signature-based or rule-based methods, which struggle to detect new, evolving, or zero-day threats.\r\n\r\n\r\n---\r\n\r\n### 🎯 Project Objective\r\n\r\nThe goal of this project is to develop a **machine learning-based malware risk forecasting model** that can:\r\n\r\n- Analyze patterns in **malware behavior**\r\n- **Predict future threats**\r\n- Enhance **cybersecurity readiness**\r\n\r\nBy leveraging **AI** and **threat intelligence**, we aim to provide a **proactive defense mechanism** against malware attacks.","metadata":{"_uuid":"42fbe826-bb4d-4a6d-917e-f48de2840d9a","_cell_guid":"586eed8c-1338-4906-bac4-06264fcf60ba","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"\n

Import Required Library

`","metadata":{"_uuid":"6f95b61d-d322-4a35-abba-7a02ea25bca2","_cell_guid":"df4377c3-bfe5-443c-b43e-67f1441b1e6e","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\") # Suppresses warning messages to keep the output clean\n\nimport pandas as pd # For data manipulation and analysis using DataFrames\nimport os # For interacting with the operating system (e.g., file paths, directory listing)\nimport sklearn # Scikit-learn, a machine learning library (used for modeling, preprocessing, etc.)\nimport numpy as np # For numerical operations and array handling\nimport shutil # For file operations like copying and moving files\nimport matplotlib.pyplot as plt # For creating static visualizations (plots, graphs)\nimport seaborn as sns # For advanced data visualization built on top of matplotlib\nfrom sklearn.manifold import TSNE # For dimensionality reduction and visualizing high-dimensional data in 2D/3D\nfrom sklearn.preprocessing import StandardScaler # For standardizing features (zero mean, unit variance) before modeling","metadata":{"_uuid":"3b18b41f-8fe4-4f52-8165-072b88168a80","_cell_guid":"d84be4ad-a17a-4cfa-80da-3f92627bcdbd","trusted":true,"collapsed":false,"id":"vaNFrxVOPIZr","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### In this section working with only .bytes Files","metadata":{"_uuid":"a3dd67a3-abf8-43ad-a2b6-530f69d7d729","_cell_guid":"89b13f33-1ee9-4e45-a1d8-d159cd572eed","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# Preparing the Data\n---","metadata":{"_uuid":"76bf39f9-f0e1-493e-a4b3-9c97332ff903","_cell_guid":"496b93ce-7ee2-4b17-a11f-763ab966c231","trusted":true,"collapsed":false,"id":"549abab4-7d3b-4f87-b656-89ad3751d45a","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"os.makedirs(\"byteFiles\", exist_ok=True)\nos.makedirs(\"asmFiles\", exist_ok=True)\n\ndata_path = r\"D:\\Malware\\train\\train\"\n\nfor file in os.listdir(data_path):\n file_path = os.path.join(data_path, file)\n\n if file.endswith(\".bytes\"):\n shutil.move(file_path, os.path.join(\"byteFiles\", file))\n\n elif file.endswith(\".asm\"):\n shutil.move(file_path, os.path.join(\"asmFiles\", file))","metadata":{"_uuid":"223c9e6c-3b43-4e14-b847-18835036174c","_cell_guid":"24838efe-4c9e-44e0-a35b-64f52de52292","trusted":true,"collapsed":false,"id":"fe3a9d04-805e-4bf4-b88e-831efe4066ae","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"## Getting Class labels\nclass_label = pd.read_csv(r\"D:\\Malware\\trainLabels.csv\")","metadata":{"_uuid":"e799f0de-974b-4b68-9db8-6293a4a3cf2f","_cell_guid":"fd676774-a4d3-486e-af4a-3ea30ba23e17","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class_label.info()","metadata":{"_uuid":"c4590955-98f0-4ad5-8ceb-d865f6cdd740","_cell_guid":"17d85af5-ecd4-492c-881c-9e94b1e6a93a","trusted":true,"collapsed":false,"id":"4639592f-94fe-42a3-be37-27b4003ebc6c","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class_label.head()","metadata":{"_uuid":"10b5d824-73b4-4057-a48b-8632e34fbf50","_cell_guid":"1aa4e66e-07c5-41f3-bb31-ad865a61a686","trusted":true,"collapsed":false,"id":"eb52d605-116b-4c7c-b9d0-e2c8b2dac0a2","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Making File Size as a Feature \n- because Malware behavior affects file size
\n- File families often have consistent size patterns : For example, Ramnit files might be around 800 KB, while Kelihos_ver1 might typically be larger or smaller","metadata":{"_uuid":"c616f476-9344-4b58-a9fd-f7d258387bfb","_cell_guid":"97053b2b-b5ec-47cf-a23c-3d181f1c0770","trusted":true,"collapsed":false,"id":"465e5d19-4671-4486-ab29-dde47c88f1fb","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Load ID and Class info from your label DataFrame\nid_to_class = dict(zip(class_label['Id'], class_label['Class']))\n\nfile_ids = []\nfile_sizes = []\nfile_classes = []\n\nfor filename in os.listdir('byteFiles'):\n if filename.endswith('.bytes'):\n file_id = filename.split('.')[0]\n\n if file_id in id_to_class:\n\n # Get file size in MB for consistent data\n size_mb = os.path.getsize(os.path.join('byteFiles', filename)) / (1024.0 * 1024.0)\n\n file_ids.append(file_id)\n file_sizes.append(size_mb)\n file_classes.append(id_to_class[file_id])\n\ndata_size_byte = pd.DataFrame({\n 'ID': file_ids,\n 'size': file_sizes,\n 'Class': file_classes\n})","metadata":{"_uuid":"b7721e1a-2c82-4ce2-afcf-6908f1912674","_cell_guid":"e4658b45-0f42-454c-ac5a-4a5c7f579e99","trusted":true,"collapsed":false,"id":"d4c3e4df-d766-4d7b-a50a-6aca39aff53d","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_size_byte","metadata":{"_uuid":"22f794c6-cad7-436e-968b-5ce4f235cb0d","_cell_guid":"985c62c4-d9b7-47e8-92a1-a2444de250a3","trusted":true,"collapsed":false,"id":"119d27dc-70db-4e54-ad12-1d4db2094d8e","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_size_byte[\"Class\"].value_counts()","metadata":{"_uuid":"8c80be53-2e29-45b8-997b-433158db9df2","_cell_guid":"830b0511-99b6-4f8d-91b4-fd4e3115b456","trusted":true,"collapsed":false,"id":"f612d8ed-44c3-4b0d-aa65-4d4e750598cd","outputId":"65f369cf-029e-4905-ef74-5f67f7b0c72c","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_size_byte.to_csv(\"bytes_sizes_df.csv\", index=False)","metadata":{"_uuid":"62cdf2a7-050a-4174-8243-19e0c8dbfa89","_cell_guid":"bb4b55ec-1041-46d3-a12a-0cfd8e0c8935","trusted":true,"collapsed":false,"id":"811682f8-c2be-4a48-8eb8-c6947447d8f8","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n

Simple EDA on Class Labels

`","metadata":{"_uuid":"dab1add0-dfab-4250-8752-27024b9ed5f8","_cell_guid":"3d1dcc20-9ef3-4d1d-b7a4-a810ab5f77c7","trusted":true,"collapsed":false,"id":"6bace011-a5f4-488f-9dbd-65b666165829","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"data = pd.read_csv(r\"D:\\Malware\\bytes_sizes_df.csv\")\ndata.head()","metadata":{"_uuid":"49dd0e29-ef29-4633-8ed5-43d3f7cf083c","_cell_guid":"3fabba65-0bfc-4f37-bab2-fd3bc755fb75","trusted":true,"collapsed":false,"id":"fd38149d-56b8-4cc9-88fb-61f0ea2f32d6","outputId":"4c979923-5dad-480c-bf2f-1759c58c6993","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data['Class'].value_counts()","metadata":{"_uuid":"ddb78c33-4885-4d55-983a-5958d2f9e488","_cell_guid":"2b0932cf-c6da-49df-a487-1d9ae760418c","trusted":true,"collapsed":false,"id":"92beeebf-62cc-4457-a0ee-1b3f0264171d","outputId":"5e381fb5-1e07-4b73-a3b2-9038df72eca3","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Class Distribution","metadata":{"_uuid":"1eba543c-55f1-499c-becb-145e3dcffc75","_cell_guid":"9ad92f21-e4ad-406c-9dc9-c4005eca01fb","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"import plotly.express as px\nimport pandas as pd\n\n# Calculate class counts and percentages\nclass_counts = data['Class'].value_counts().reset_index()\nclass_counts.columns = ['Class', 'Count']\nclass_counts['Percentage'] = (class_counts['Count'] / class_counts['Count'].sum() * 100).round(2)\n\n# Create bar chart with Plotly\nfig = px.bar(\n class_counts,\n x='Class',\n y='Count',\n text='Percentage',\n title='Malware Class Distribution',\n labels={'Count': 'Number of Samples'},\n color='Class' # Optional: use different colors per class\n)\n\n# Customize the text position and layout\nfig.update_traces(texttemplate='%{text}%', textposition='outside')\nfig.update_layout(\n xaxis_title='Class',\n yaxis_title='Count',\n uniformtext_minsize=8,\n uniformtext_mode='hide',\n bargap=0.3\n)\n\n# Show the interactive plot\nfig.show()","metadata":{"_uuid":"87aeaa02-e611-464c-aabc-c56309a903cb","_cell_guid":"5fe39955-508f-49bd-967b-f266fe05f67c","trusted":true,"collapsed":false,"id":"0e8c1a81-5320-48cd-b0af-6f6d4f2b823b","outputId":"434c1edb-9417-4573-b1ca-d1b061e00e51","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Insight:\n\n- We can see class 5 malware is very rare in our dataset
\n- Class 3 and 2 malware are having most occurances in our data","metadata":{"_uuid":"f7d80ad1-367a-4336-8344-d82d55493d85","_cell_guid":"d4cc0415-2609-473e-889f-28f51c819154","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"## Distribution of File Sizes across Class","metadata":{"_uuid":"bd7f9ab6-b2d6-46d9-bcc5-5e2809a70896","_cell_guid":"d6b383b5-b6d7-4bf1-a508-31f6bced326f","trusted":true,"collapsed":false,"id":"55cc9d6f-9221-4e5d-b869-088f7d86e9c8","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 6))\nsns.boxplot(data=data, x='Class', y='size', palette='Set2')\nplt.title(\"Distribution of File Sizes Across Malware Classes\")\nplt.xlabel(\"Malware Class\")\nplt.ylabel(\"File Size (MB)\")\nplt.tight_layout()\nplt.show()","metadata":{"_uuid":"fbbfafec-c6a3-4535-bfc8-acedda2cc69e","_cell_guid":"944561d2-8813-4241-9ad1-39c4bec4a9f6","trusted":true,"collapsed":false,"id":"a287ecc4-2751-48f2-85ff-97b71e18ae16","outputId":"15b91578-f090-4039-f0b0-b5c13b5f22c9","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insights\n- Class 2, 5, and 9 have high spread.\n- Class 1, 4, and 6 have a similar spread\n- Class 3 and 7 have very low spread.","metadata":{"_uuid":"82c0ca0a-14bb-4370-9db9-a5c89c013511","_cell_guid":"7a0cb279-eec9-41dd-96d8-44d3cccbe377","trusted":true,"collapsed":false,"id":"faa94439-8f9f-4def-a0ea-7d2ea5969555","jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# Average Size per Class","metadata":{"_uuid":"98235bb5-05fa-41d6-9b14-deabd39107fa","_cell_guid":"badb46c0-283e-41b7-abbe-fafde638ee96","trusted":true,"collapsed":false,"id":"9c55a846-7667-4527-a203-32b5ab6f896a","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Compute average size per class\navg_size_per_class = data.groupby('Class')['size'].mean().reset_index()\n\nplt.figure(figsize=(10,6))\nsns.barplot(data=avg_size_per_class, x='Class', y='size', palette='viridis')\nplt.title(\"Average File Size per Malware Class\")\nplt.ylabel(\"Avg Size (MB)\")\n\nfor index, row in avg_size_per_class.iterrows():\n plt.text(index, row['size'] + 0.1, f\"{row['size']:.2f} MB\", ha='center', fontsize=9)\n\nplt.tight_layout()\nplt.show()","metadata":{"_uuid":"833e0929-4d31-4061-b4fe-cfb8fe99377c","_cell_guid":"105ca157-a2d7-42ea-bdec-a2e77a33b549","trusted":true,"collapsed":false,"id":"cb2da942-3d79-45df-9c4f-7c86a18c1a08","outputId":"08419dde-2d2f-48c9-c395-11f4002864e0","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insight :\n- We can see that Class 3 having the highest average size - 8.57MB\n- Malware Classes 1 and 6 have similar average file sizes of around 1.4 MB and 1.72 MB, respectively.\n- Malware Class 8 has the smallest average file size at 0.79 MB\n---","metadata":{"_uuid":"97d46081-b06e-4019-b37c-4baf3506d30a","_cell_guid":"ae1bc839-ec4b-4a10-a3bb-a56a20d019a3","trusted":true,"collapsed":false,"id":"ea14f664-c017-4c77-97e4-9e4b9bf0799d","jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# Data Pre-processing\n- I have only use only uni-gram hexadecimal counts\r\n- \nAttempted bigram extraction, but failed due to insufficient RA\n- \r\nSo finally only using uni-gram","metadata":{"_uuid":"173d7691-3092-496d-8485-f9c792fd49f0","_cell_guid":"7ad9100c-468f-4a73-a285-f5e9ed3f4dec","trusted":true,"collapsed":false,"id":"9be38e2e-0d53-43cc-9e27-66dc04383b42","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"## This Code I use to Extract bytes files and get the occurance of each uni-gram hexa-decimal values in dataframe\n\nimport os\nfrom collections import Counter\nimport pandas as pd\nfrom concurrent.futures import ThreadPoolExecutor\nfrom tqdm import tqdm\n\nBYTE_FOLDER = r\"D:\\Malware\\byteFiles\"\nBATCH_SIZE = 500\nOUTPUT_FOLDER = \"hex_batches\"\nos.makedirs(OUTPUT_FOLDER, exist_ok=True)\n\n# Step 1: Get list of .bytes files and remove already processed\ndef get_remaining_files():\n processed_ids = set()\n for f in os.listdir(OUTPUT_FOLDER):\n if f.endswith('.csv'):\n df = pd.read_csv(os.path.join(OUTPUT_FOLDER, f), usecols=['ID'])\n processed_ids.update(df['ID'].tolist())\n\n all_files = [os.path.join(BYTE_FOLDER, f) for f in os.listdir(BYTE_FOLDER) if f.endswith('.bytes')]\n return [f for f in all_files if os.path.basename(f).split('.')[0] not in processed_ids]\n\n# Step 2: Process a single file\ndef process_file(file_path):\n file_id = os.path.basename(file_path).split('.')[0]\n freq = Counter()\n try:\n with open(file_path, 'r', errors='ignore') as file:\n for line in file:\n parts = line.strip().split()\n hex_values = parts[1:] # Skip address part\n for val in hex_values:\n freq[val.upper() if val != '??' else '??'] += 1\n except Exception as e:\n print(f\"Error in {file_id}: {e}\")\n return None\n\n hex_keys = [f'{i:02X}' for i in range(256)] + ['??']\n return [file_id] + [freq.get(k, 0) for k in hex_keys]\n\n# Step 3: Process in batches and save\ndef process_in_batches(files, batch_size):\n hex_keys = [f'{i:02X}' for i in range(256)] + ['??']\n columns = ['ID'] + hex_keys\n\n total_batches = (len(files) + batch_size - 1) // batch_size\n for i in range(total_batches):\n batch_files = files[i * batch_size: (i + 1) * batch_size]\n print(f\"\\nProcessing Batch {i + 1}/{total_batches}... ({len(batch_files)} files)\")\n\n results = []\n with ThreadPoolExecutor(max_workers=3) as executor: ## Multi threading\n for result in tqdm(executor.map(process_file, batch_files), total=len(batch_files)):\n if result:\n results.append(result)\n\n\n batch_df = pd.DataFrame(results, columns=columns)\n batch_df.to_csv(os.path.join(OUTPUT_FOLDER, f'batch_{i+1}.csv'), index=False)\n print(f\"Saved batch_{i+1}.csv\")\n\n# Step 4: Combine all batch CSVs into final DataFrame\ndef combine_batches():\n all_batches = [os.path.join(OUTPUT_FOLDER, f) for f in os.listdir(OUTPUT_FOLDER) if f.endswith('.csv')]\n final_df = pd.concat([pd.read_csv(f) for f in all_batches], ignore_index=True)\n final_df.to_csv(\"hex_frequency_all_files.csv\", index=False)\n print(\"All batches combined into 'hex_frequency_all_files.csv'\")\n\nif __name__ == \"__main__\":\n remaining_files = get_remaining_files()\n print(f\"Remaining files to process: {len(remaining_files)}\")\n process_in_batches(remaining_files, BATCH_SIZE)\n combine_batches()","metadata":{"_uuid":"125ad413-4859-4fd0-89fa-0480dbecf7e1","_cell_guid":"dabec121-c6d9-49e7-a416-830574fcff60","trusted":true,"collapsed":false,"id":"7388a0e2-f8c6-42ea-ae23-5c37b5461042","outputId":"337fb9d8-7d2f-471e-e74f-a575fbca684e","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Getting all data into single dataframe","metadata":{"_uuid":"1f463d16-3565-4d2a-8bb6-4393e29c8aa1","_cell_guid":"60f782df-f539-464a-9e5d-d3963b17f3e2","trusted":true,"collapsed":false,"id":"e45ee5b8-96cb-425c-8127-37a521c62442","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Folder containing all batch CSV files\ncsv_folder = \"hex_batches\"\n\ncsv_files = [os.path.join(csv_folder, f) for f in os.listdir(csv_folder) if f.endswith('.csv')]\n\nfinal_df = pd.concat([pd.read_csv(file) for file in csv_files], ignore_index=True)","metadata":{"_uuid":"8f0a6ad1-47b9-4dd1-a12f-12d964600827","_cell_guid":"ae8e28a2-1613-4199-b178-a5db28bf5585","trusted":true,"collapsed":false,"id":"16c84b45-0349-4b5f-8200-b4064e9ce892","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"final_df","metadata":{"_uuid":"cc0c8ce2-9632-41f5-b371-bbfb812fb64a","_cell_guid":"cbff904d-a940-4302-928b-39f7b1e7ffe8","trusted":true,"collapsed":false,"id":"43f59e67-3554-4a09-b541-9491d2388dc7","outputId":"9b558d38-e43e-4fef-cfea-9fb028f61d0f","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.read_csv(\"bytes_sizes_df.csv\")","metadata":{"_uuid":"901000a2-e49b-42d5-b2f0-b0c94cc459e3","_cell_guid":"67ae3ed5-3585-4539-bc98-e503823a78cf","trusted":true,"collapsed":false,"id":"9c945832-5146-4b47-ac39-dd0e4f957e72","outputId":"ea8face1-9fa2-4644-8937-87b710d92905","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df2 = pd.read_csv(\"bytes_sizes_df.csv\")\n\nbytes_df = final_df.merge(df2, on='ID', how='left')","metadata":{"_uuid":"9fd8fdd8-7c08-430e-91bb-0ba75671a7ba","_cell_guid":"1cef0c00-0273-409b-be93-93ad392111a2","trusted":true,"collapsed":false,"id":"b8bcea6c-fa90-4923-b460-70290431b116","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Final Dataframe","metadata":{"_uuid":"ff88fe1b-e47d-4b50-9d0f-cb8926c3b9f4","_cell_guid":"7c5f60e1-cb10-47cc-9700-31c3a3c61eb2","trusted":true,"collapsed":false,"id":"8926ac5f-9ee8-4921-9561-4439126ab344","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"bytes_df","metadata":{"_uuid":"506a0622-20fd-44c9-8e57-b26e1b2186fd","_cell_guid":"fd469464-648f-423c-a89c-0cd0de85d841","trusted":true,"collapsed":false,"id":"2b4e5916-b494-4f8c-a694-53dbd31d6f03","outputId":"7e9adb5d-c2ca-4e25-822f-264e30ec75e3","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n

EDA on Final Data

`","metadata":{"_uuid":"3c3b7090-b61d-4efb-9ee2-9c91b92e934d","_cell_guid":"516a6e56-4c64-4cd1-ae84-cf0c57191f60","trusted":true,"collapsed":false,"id":"bad3f692-4f37-4109-9251-2e4ccd93c2ee","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from google.colab import drive\ndrive.mount('/content/drive')","metadata":{"_uuid":"9674ff0d-f33a-4bce-a95a-17d2989f2791","_cell_guid":"1bd52116-0eac-4c5f-97de-968f2a1dfe81","trusted":true,"collapsed":false,"id":"dbaU4Y1upyrB","outputId":"8a354bfd-35d1-455e-cde7-5cb82efb3111","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data = pd.read_csv(r\"/content/drive/MyDrive/bytes_df.csv\")","metadata":{"_uuid":"0fcfe3cb-363f-4ca9-bca5-ac85e5f49a76","_cell_guid":"a5ff13cf-5452-4b6e-8694-8a9390a605d0","trusted":true,"collapsed":false,"id":"wTCVQ6XTp4dJ","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data = pd.read_csv(r\"C:\\Users\\udayk\\Downloads\\bytes_df (1).csv\")","metadata":{"_uuid":"057a67c9-e3c7-4379-bd08-1285ff02bddf","_cell_guid":"b97fa44f-2024-4912-a8bc-4fb80d4af10d","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.info()","metadata":{"_uuid":"c33dcead-fc02-4be1-a6af-cd565d619851","_cell_guid":"63f8fc00-a225-42f8-9d3d-bc1a7562c446","trusted":true,"collapsed":false,"id":"LFk54lgZusiO","outputId":"1aec530c-50ca-4818-bfa9-4dadbd6ff740","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = data.copy()","metadata":{"_uuid":"6e3891f8-4fef-41ad-a056-d98d34eb3bc1","_cell_guid":"0d165b4f-aca6-41ab-8303-cbe2e8b8fcea","trusted":true,"collapsed":false,"id":"KIgrO9Qqu0A3","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = pd.read_csv(r\"C:\\Users\\udayk\\Downloads\\bytes_df.csv\")","metadata":{"_uuid":"d7ab448a-3d24-46ca-8a22-3a7446307cba","_cell_guid":"86313c4c-52e0-490e-baeb-f2bc14fb42e3","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nX = df.drop(columns=['ID', 'Class'])\ny = df['Class']\n\nX_scaled = StandardScaler().fit_transform(X)\n\ntsne = TSNE(n_components=2, perplexity=30, n_iter=1000, random_state=42, n_jobs=-1)\nX_tsne = tsne.fit_transform(X_scaled)\n\ntsne_df = pd.DataFrame(X_tsne, columns=['Dim1', 'Dim2'])\ntsne_df['Class'] = y\n\nplt.figure(figsize=(10, 8))\nsns.scatterplot(data=tsne_df, x='Dim1', y='Dim2', hue='Class', palette='tab10', s=60, alpha=0.8)\nplt.title('t-SNE Visualization of Byte Frequency Features')\nplt.legend(title='Class')\nplt.show()","metadata":{"_uuid":"eea9cae2-3db9-4974-a463-cd15188d16e3","_cell_guid":"091917c1-6cc8-4cf9-a83e-7b17da026a08","trusted":true,"collapsed":false,"id":"lkgRETK3YqeZ","outputId":"461b13ae-9bcb-49a0-c895-8feb46428f29","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insight:\n- Only Class 3 (green) and Class 6 (pink) seem clearly separable. \n- The rest (e.g., Classes 1, 2, 4, 8) are heavily overlapped","metadata":{"_uuid":"848f74ab-0197-491d-ba4d-9ef45020d3ca","_cell_guid":"2602f27f-47bf-47d1-8db2-7e1110fafa0a","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"### Compare class-wise mean frequency for each hex value","metadata":{"_uuid":"d64efb0b-f0ec-415b-9b1a-f9fecdc150dd","_cell_guid":"345d5f98-d573-4f3c-8524-b048d04d5d10","trusted":true,"collapsed":false,"id":"jVbEfuei0nrr","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"# Filter out hex columns\nhex_columns = [col for col in df.columns if col not in ['ID', 'Class', 'size', '??', '00']]\n\nclasswise_sum = df.groupby('Class')[hex_columns].sum()\n\n# Extract top 3 hex values per class\nrecords = []\nfor cls in classwise_sum.index:\n top_hexes = classwise_sum.loc[cls].sort_values(ascending=False).head(3)\n for hex_val, freq in top_hexes.items():\n records.append({'Class': cls, 'Hex': hex_val, 'Frequency': freq})\n\ntop3_df = pd.DataFrame(records)\n\nplt.figure(figsize=(10, 6))\nsns.barplot(data=top3_df, x='Class', y='Frequency', hue='Hex')\nplt.title(\"Top 3 Most Frequent Hex Values per Class\")\nplt.ylabel(\"Total Frequency\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.legend(title=\"Hex Byte\")\nplt.show()","metadata":{"_uuid":"e3b88be6-16c7-4e97-9b04-119e9f62c8f2","_cell_guid":"eede22ab-5e2c-484d-8fa3-1e545a7aabae","trusted":true,"collapsed":false,"id":"H-uctmyyZI7b","outputId":"d9c61a7f-5e11-4568-aea4-9c13e208dfc1","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insight:\n\n- For **Kelihos_ver3** class (class 3) 01 hex is occuring which is not at all occuring in any other classes\n- For **Lollipop class** (class2) CC and 02 hex are mostly occuring\n- For **Ramnit class** (class1) 8B hex is occuring more than other classes","metadata":{"_uuid":"679121d5-ad74-4ae1-a9fb-055da66b2ce4","_cell_guid":"9d3cde31-d484-41f4-830b-c9522e592556","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"hex_columns = [col for col in df.columns if col not in ['ID', 'Class', 'size', '??', '00']]\n\ndf_filtered = df[~df['Class'].isin([1, 2, 3])]\n\n# Compute class-wise sum for the filtered dataframe\nclasswise_sum = df_filtered.groupby('Class')[hex_columns].sum()\n\n# Extract top 3 hex values per class\nrecords = []\nfor cls in classwise_sum.index:\n top_hexes = classwise_sum.loc[cls].sort_values(ascending=False).head(3)\n for hex_val, freq in top_hexes.items():\n records.append({'Class': cls, 'Hex': hex_val, 'Frequency': freq})\n\ntop3_df = pd.DataFrame(records)\n\nplt.figure(figsize=(10, 6))\nsns.barplot(data=top3_df, x='Class', y='Frequency', hue='Hex')\nplt.title(\"Top 3 Most Frequent Hex Values per Class (Excluding Classes 1, 2, and 3)\")\nplt.ylabel(\"Total Frequency\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.legend(title=\"Hex Byte\")\nplt.show()","metadata":{"_uuid":"fbc15c80-9d96-4ea5-886b-6d1e9e5923ca","_cell_guid":"a161cb1a-77b1-472c-ac33-c69ca50a6e3f","trusted":true,"collapsed":false,"id":"OV2W6NjeqhXT","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insight:\n\n- For **Tracur** class (class 6) 20 and 65 hex are occuring which is not at all occuring in any other classes for majority time\n- For **Kelihos_ver1** class (class 7) 80 and 10 hex are mostly occuring which also not occuring in other classes for majority time\n- 8B hex occurs in only **Obfuscator.ACY** (class 8), **Gatak** (class 9) and **Ramint** (class 1) for most frequent","metadata":{"_uuid":"cefba7f4-e903-454c-8471-18f6040d0957","_cell_guid":"b59a74d6-f5f1-4d58-9b90-3c8a23f0f044","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"bhex_columns = [col for col in df.columns if col not in ['ID', 'Class', 'size', '??', '00']]\n\ndf_filtered = df[~df['Class'].isin([1, 2, 3, 6, 7, 8, 9])]\n\n# Compute class-wise sum for the filtered dataframe\nclasswise_sum = df_filtered.groupby('Class')[hex_columns].sum()\n\n# Extract top 3 hex values per class\nrecords = []\nfor cls in classwise_sum.index:\n top_hexes = classwise_sum.loc[cls].sort_values(ascending=False).head(3)\n for hex_val, freq in top_hexes.items():\n records.append({'Class': cls, 'Hex': hex_val, 'Frequency': freq})\n\ntop3_df = pd.DataFrame(records)\n\nplt.figure(figsize=(10, 6))\nsns.barplot(data=top3_df, x='Class', y='Frequency', hue='Hex')\nplt.title(\"Top 3 Most Frequent Hex Values per Class (Including Classes 4 and 5)\")\nplt.ylabel(\"Total Frequency\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.legend(title=\"Hex Byte\")\nplt.show()","metadata":{"_uuid":"c794303a-3808-466a-ba7b-ba72be7f725b","_cell_guid":"7afe039c-1d73-4c65-9a03-0b54042ac624","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Insight\n- 01 hex is only occuring in **Simda** (class 5) and **Kelihos_ver3** (class 3) for maximun number of times , which we saw in previous plots\n- EB and 68 hex are only hex which occurs mostly in only **Vundo** (class 4)","metadata":{"_uuid":"966d8d3d-aeae-4de6-9b97-605e9f74fe40","_cell_guid":"5183f9db-f1de-4eff-9396-ebe1581fe6db","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"\n

\nData Pre-Processing

`","metadata":{"_uuid":"e5a56dbb-e621-41c7-aacf-bfbf668ccc26","_cell_guid":"30ef2e34-3fb8-486f-ba67-32dc3654837f","trusted":true,"collapsed":false,"id":"AWcv5HC0qutA","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"!pip install xgboost","metadata":{"_uuid":"4455a6a7-0bb3-4d43-a5f1-49eb3d42b858","_cell_guid":"3e75f968-ed20-4b62-96c9-2b08caafde14","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import RobustScaler, LabelEncoder\nfrom xgboost import XGBClassifier\nfrom sklearn.utils import compute_class_weight,compute_sample_weight\nfrom sklearn.metrics import log_loss, classification_report, confusion_matrix\nfrom sklearn.calibration import CalibratedClassifierCV","metadata":{"_uuid":"2c3070a4-01eb-45e3-a24f-321c6d01357d","_cell_guid":"0ed82431-8ce8-4752-864b-3280501647fd","trusted":true,"collapsed":false,"id":"lg60joUjqwoE","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Mapping 1 to 9 class labels to start them from 0 to 8\r\n\r\n- For future use while doing `inverse_transform` with `LabelEncoder()`\r\n- Some tools or metrics expect labels to start from 0, especially:\r\n - XGBoost with `multi:softprob` or `multi:softmax`\r\n - Scikit-learn classification metrics","metadata":{"_uuid":"9db81fa7-0e66-4bf7-9059-13d26a767280","_cell_guid":"510b0a54-0c4b-4f37-b15a-d937f6b33006","trusted":true,"collapsed":false,"id":"XuymH6lWsLtl","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"label_map = {\n 1: 'Ramnit',\n 2: 'Lollipop',\n 3: 'Kelihos_ver3',\n 4: 'Vundo',\n 5: 'Simda',\n 6: 'Tracur',\n 7: 'Kelihos_ver1',\n 8: 'Obfuscator.ACY',\n 9: 'Gatak'\n}\n\ndf['Class'] = df['Class'].map(label_map)","metadata":{"_uuid":"e4099ddc-c587-4d66-aca9-0240973802a2","_cell_guid":"68b7acbc-9329-4124-a99c-6fe85aa9529e","trusted":true,"collapsed":false,"id":"yYjHQDOVqze2","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df['Class'].head()","metadata":{"_uuid":"43b2b069-37ee-4436-b58e-d89236b0d00f","_cell_guid":"0bb85015-f469-423c-a141-07cdc00f33ce","trusted":true,"collapsed":false,"id":"Q_GpuBnIuqAf","outputId":"c4ff3d88-66d9-45be-faee-963966d23702","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X = df.drop(['Class', 'ID'], axis=1) # Drop ID and target\ny = df['Class']\n\nx_train, x_test, y_train, y_test = train_test_split(\n X, y,\n test_size=0.2, # 80% train, 20% test\n stratify=y,\n random_state=42\n)","metadata":{"_uuid":"4f6f7f08-aadd-4763-bf96-53f375c52f47","_cell_guid":"2eb842af-cc6a-4a7c-9849-359f1ddd4e88","trusted":true,"collapsed":false,"id":"0Qb8FVKNuvHH","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"le = LabelEncoder()\n\ny_train = le.fit_transform(y_train)\ny_test = le.transform(y_test)","metadata":{"_uuid":"31d5a603-dd5c-44c0-8e67-3f3306b68925","_cell_guid":"c7f927c9-6ef3-4247-a570-aebfa01ff3ba","trusted":true,"collapsed":false,"id":"gpBET7GXvT0D","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"re = RobustScaler()\nx_train = re.fit_transform(x_train)\nx_test = re.transform(x_test)","metadata":{"_uuid":"42adc100-bcea-46db-ad9b-942edf6af0b3","_cell_guid":"b7929c1c-f245-4c6b-bcae-1d92023a4dbb","trusted":true,"collapsed":false,"id":"1X9qgafX2RfZ","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Calculated the class weight due to imbalance data\n----","metadata":{"_uuid":"babcbf2d-037d-49f7-9cd5-a9eb125517da","_cell_guid":"b0773417-0fbe-4e36-890a-689809860f74","trusted":true,"collapsed":false,"id":"5EOq0whIvnnB","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"y_train_series = pd.Series(y_train)\n\n# Compute class weights\nclasses = np.unique(y_train_series)\nweights = compute_class_weight(class_weight='balanced', classes=classes, y=y_train_series)\nclass_weights = dict(zip(classes, weights))\n\n# Map sample weights\nsample_weights = y_train_series.map(class_weights)","metadata":{"_uuid":"a39cfc15-6761-4986-83ca-7cbe295abcca","_cell_guid":"1bd99ab7-bcd3-4af9-9ede-2bb35a5a9f58","trusted":true,"collapsed":false,"id":"_mjOvkLpvWza","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!pip install optuna","metadata":{"_uuid":"b5fa2071-be0f-4bcb-9203-5f8ed1b5a1ac","_cell_guid":"2e5f27ff-6c4c-430e-a52e-a90d09ce4a28","trusted":true,"collapsed":false,"id":"kNComZPeWWwX","outputId":"e6d23476-9aed-4177-c2b8-dc6c89cf2e43","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import optuna","metadata":{"_uuid":"0b0c1d3b-b53e-46ab-9840-7a48497960b2","_cell_guid":"0f0fbf0f-8b3d-43e6-b201-7251c29c6e33","trusted":true,"collapsed":false,"id":"mQTgLb4ZWrcy","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import sklearn\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import OneHotEncoder, RobustScaler\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom optuna.samplers import TPESampler\nfrom sklearn.model_selection import cross_validate\nfrom sklearn.metrics import log_loss\nfrom sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.ensemble import RandomForestClassifier","metadata":{"_uuid":"2719af4c-3afd-409c-9bc3-32fbead99ff4","_cell_guid":"3eb87ee1-d652-4a82-aaec-35a8e72f1fe7","trusted":true,"collapsed":false,"id":"EWYRQPhv6NZo","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Optuna using Decision Tree\n----","metadata":{"_uuid":"25be4e5a-551c-47a7-bd14-10acc83acb3d","_cell_guid":"afca1267-3275-4b3d-9a05-93602d63bd14","trusted":true,"collapsed":false,"id":"M-3O503yeQkA","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def objective(trail):\n algo=trail.suggest_categorical('algo',['dt','dt1'])\n if algo==\"dt\":\n depth=trail.suggest_int(\"max_depth\",1,25)\n min_samples_split=trail.suggest_int(\"min_samples_split\",2,50)\n min_samples_leaf=trail.suggest_int(\"min_samples_leaf\",1,50)\n min_weight_fraction_leaf=trail.suggest_float(\"min_weight_fraction_leaf\",0.0,0.15)\n min_impurity_decrease=trail.suggest_float(\"min_impurity_decrease\",0.0,1.0)\n max_features=trail.suggest_int(\"max_features\",1,7)\n model=DecisionTreeClassifier(max_depth=depth,min_samples_split=min_samples_split,min_samples_leaf=min_samples_leaf,min_weight_fraction_leaf=min_weight_fraction_leaf,min_impurity_decrease=min_impurity_decrease,max_features=max_features)\n model.fit(x_train,y_train)\n elif algo==\"dt1\":\n depth=trail.suggest_int(\"max_depth\",26,50)\n min_samples_split=trail.suggest_int(\"min_samples_split\",20,70)\n min_samples_leaf=trail.suggest_int(\"min_samples_leaf\",20,70)\n min_weight_fraction_leaf=trail.suggest_float(\"min_weight_fraction_leaf\",0.0,0.35)\n min_impurity_decrease=trail.suggest_float(\"min_impurity_decrease\",0.0,2.0)\n max_features=trail.suggest_int(\"max_features\",5,12)\n model=DecisionTreeClassifier(max_depth=depth,min_samples_split=min_samples_split,min_samples_leaf=min_samples_leaf,min_weight_fraction_leaf=min_weight_fraction_leaf,min_impurity_decrease=min_impurity_decrease,max_features=max_features)\n model.fit(x_train,y_train)\n\n\n cv_score=cross_validate(model,x_train,y_train,cv=3,scoring=\"neg_log_loss\",return_train_score=True)['test_score'].mean()\n train_score=cross_validate(model,x_train,y_train,cv=3,scoring=\"neg_log_loss\",return_train_score=True)['train_score'].mean()\n trail.set_user_attr(key=\"cv_score\",value=cv_score)\n trail.set_user_attr(key=\"train_score\",value=train_score)\n return cv_score","metadata":{"_uuid":"9086fb52-79df-40e6-b275-d7ab0888b21c","_cell_guid":"5cd5c96b-7859-48fb-8572-16e6b8dc8226","trusted":true,"collapsed":false,"id":"mQDhGfDqv5PT","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study=optuna.create_study(direction=\"maximize\",sampler=TPESampler())","metadata":{"_uuid":"caf0ff2f-36c7-43ad-9699-29390749a6bf","_cell_guid":"f6594497-caa1-43c6-a816-1c5f989764ea","trusted":true,"collapsed":false,"id":"aJYE21S3WeZn","outputId":"aa6ef561-cadf-49bf-df62-641ffdb5b1d2","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.optimize(objective,n_trials=150,n_jobs=-1)","metadata":{"_uuid":"a57b4135-6f09-4d0e-99d9-a6f411a4a35f","_cell_guid":"219fce30-fa65-4d13-b259-9792fb86b4e0","trusted":true,"collapsed":false,"id":"19VcwxiGWpMX","jupyter":{"outputs_hidden":false},"outputId":"17227aea-b80e-4e74-a91b-6019b68ca78e"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"d = study.trials_dataframe()\nd","metadata":{"_uuid":"032ca11d-1198-4739-8487-0a6f3097f42d","_cell_guid":"618f250c-98a8-4fcb-a88c-e9fd7a758ccb","trusted":true,"collapsed":false,"id":"Ax31aLWoczHK","outputId":"0adf067f-97ef-4914-8289-ddf60dfcd4d2","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(50,7))\nplt.plot(d[\"number\"],d[\"user_attrs_train_score\"],label = \"Training loss\")\nplt.plot(d[\"number\"],d[\"value\"],label = \"cv loss\")\nplt.grid()\nplt.xticks(d[\"number\"],rotation=45)\nplt.legend()\nplt.show()","metadata":{"_uuid":"5d33ab0a-678b-4bad-a70a-e0801e311941","_cell_guid":"eb8f0aa8-360f-4373-9060-2e3a439de1ae","trusted":true,"collapsed":false,"id":"lm8oZrr7dP8V","outputId":"a3de4862-b9d1-468c-d98a-ddb89cf656d5","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.best_params","metadata":{"_uuid":"1a52bb8d-9d52-4a25-a0d6-b29e2232a490","_cell_guid":"14a6027d-827e-4d11-90bb-6035cd4c4065","trusted":true,"collapsed":false,"id":"E9dcfsbUWys6","outputId":"029d44e6-6168-4483-dd43-5d28de352790","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.best_value","metadata":{"_uuid":"112e391c-dbc0-4bf0-aa7b-096c012995a0","_cell_guid":"2fec721b-f23f-44c8-a55c-1bd15a7ebba5","trusted":true,"collapsed":false,"id":"Amz-NtUQXe01","outputId":"09658776-2b33-4242-92c3-fdae1a2c820e","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dt = DecisionTreeClassifier(max_depth= 10,\n min_samples_split = 20,\n min_samples_leaf = 20,\n min_weight_fraction_leaf = 0.010734052715218798,\n min_impurity_decrease = 0.0019389454945900031,\n max_features = 6)","metadata":{"_uuid":"9dd7fc86-a7f5-4334-ba01-04b085f6bbfa","_cell_guid":"31723c6e-d1a6-43bb-9fec-cc6a10fac3bd","trusted":true,"collapsed":false,"id":"36Eqq-nAXhWS","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model=dt.fit(x_train,y_train,sample_weight=sample_weights)","metadata":{"_uuid":"e73ee680-277c-4665-9b27-650724f5cbcf","_cell_guid":"567c14b3-f5fc-4577-9d0b-04245fa696ab","trusted":true,"collapsed":false,"id":"el5NmLIPYc2X","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"log_loss(y_test,model.predict_proba(x_test))","metadata":{"_uuid":"49de962a-94dc-4337-b984-5deb4d986309","_cell_guid":"c8b8088d-18e7-4a71-bd91-8e8f4637b5f2","trusted":true,"collapsed":false,"id":"Xbje7_ADYmXc","outputId":"8990bc99-805e-400e-f828-8f6b6d01da86","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"------\n#### 🔧 Hyperparameter Tuning Result (Decision Tree)\r\n\r\nI have done hyperparameter tuning with **Optuna** using the **Decision Tree** algorithm \r\nand got a **very high LogLoss**: `0.6122972977322191`","metadata":{"_uuid":"ac6d284f-b15c-4f5e-98ec-bd4dd5195cdd","_cell_guid":"f4fb64e5-0672-4bb1-bd51-d381934bdfd6","trusted":true,"collapsed":false,"id":"4hhWkE7PZuxd","jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# Optuna using Random Forest\n----","metadata":{"_uuid":"4332a6ee-4c23-42b7-90db-89fe5445d4f9","_cell_guid":"7ed3c442-0181-4eaa-a001-9f9a30dd6c9a","trusted":true,"collapsed":false,"id":"9vzsLqmKeW8Z","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"def objective(trail):\n algo=trail.suggest_categorical('algo',['rf','rf1'])\n if algo==\"rf\":\n min_split = trail.suggest_int(\"min_samples_split\", 2, 6)\n min_leaf = trail.suggest_int(\"min_samples_leaf\", 1, 5)\n max_feature = trail.suggest_categorical(\"max_feature\",[\"log2\",\"sqrt\"])\n n_est = trail.suggest_int(\"n_est\",20,80)\n max_sample = trail.suggest_float(\"max_samples\", 0.6, 1.0)\n max_depth = trail.suggest_int(\"max_depth\", 5, 50)\n\n model=RandomForestClassifier(min_samples_split=min_split,min_samples_leaf=min_leaf,max_features=max_feature,n_estimators=n_est,max_samples=max_sample,max_depth=max_depth)\n model.fit(x_train,y_train)\n elif algo==\"rf1\":\n min_split = trail.suggest_int(\"min_samples_split\", 6, 12)\n min_leaf = trail.suggest_int(\"min_samples_leaf\", 1, 5)\n max_feature = trail.suggest_categorical(\"max_feature\",[\"log2\",\"sqrt\"])\n n_est = trail.suggest_int(\"n_est\",81,120)\n max_sample = trail.suggest_float(\"max_samples\", 0.10, 1.0)\n max_depth = trail.suggest_int(\"max_depth\", 51, 70)\n\n model=RandomForestClassifier(min_samples_split=min_split,min_samples_leaf=min_leaf,max_features=max_feature,n_estimators=n_est,max_samples=max_sample,max_depth=max_depth)\n model.fit(x_train,y_train)\n\n cv_score=cross_validate(model,x_train,y_train,cv=3,scoring=\"neg_log_loss\",return_train_score=True)['test_score'].mean()\n train_score=cross_validate(model,x_train,y_train,cv=3,scoring=\"neg_log_loss\",return_train_score=True)['train_score'].mean()\n trail.set_user_attr(key=\"cv_score\",value=cv_score)\n trail.set_user_attr(key=\"train_score\",value=train_score)\n return cv_score","metadata":{"_uuid":"d1695e5d-9513-4afc-96db-4a4ecb07f4bb","_cell_guid":"4dfbf32a-8f6a-4b67-aedf-e936409188e9","trusted":true,"collapsed":false,"id":"urxKFQ20Zrv4","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study=optuna.create_study(direction=\"maximize\",sampler=TPESampler())","metadata":{"_uuid":"47d47b2c-8b47-485d-9db8-9391d992be65","_cell_guid":"a2ac4379-2fb8-4657-8e4a-31af91c494be","trusted":true,"collapsed":false,"id":"Ic6UMnBlpEh6","outputId":"ead1f8f8-c974-48fb-804d-c7ae9e6ef252","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.optimize(objective,n_trials=150,n_jobs=-1)","metadata":{"_uuid":"59d76f22-ce3d-4862-9a02-6341f0c9ec60","_cell_guid":"56a56970-cc82-44c7-ac4b-2b9908f7a4d0","trusted":true,"collapsed":false,"id":"1bM5pqytpHrm","jupyter":{"outputs_hidden":false},"outputId":"21f7d57e-d278-451d-8049-14771d85f83e"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"d1 = study.trials_dataframe()\nd1","metadata":{"_uuid":"daad8a3d-2990-4e3d-80d0-5736f40f8eae","_cell_guid":"575c79cf-2a95-4bb2-87c4-72c917fb0d7b","trusted":true,"collapsed":false,"id":"kxNWZwJvpKa0","outputId":"21924429-1985-460a-bb57-d94a8e22dcd6","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplt.plot(d1[\"number\"],d1[\"user_attrs_train_score\"],label = \"Training loss\")\nplt.plot(d1[\"number\"],d1[\"value\"],label = \"cv loss\")\nplt.grid()\nplt.xticks(d1[\"number\"],rotation=45)\nplt.legend()\nplt.show()","metadata":{"_uuid":"cd92b332-6662-4673-88c8-cbf21e9c98ac","_cell_guid":"b16f1a0e-b091-47a0-8bdc-9bd5deb0b00c","trusted":true,"collapsed":false,"id":"ZZ8mKaxY9En9","outputId":"6f1f2d8f-e960-42ea-942c-c7fe4a6c6419","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":" study.best_params","metadata":{"_uuid":"dc3f2ac7-5085-44ea-88fa-1bd07f80fa5e","_cell_guid":"e026a21f-a25a-4726-a1b3-f58d0780fbee","trusted":true,"collapsed":false,"id":"haKGOa5o9M4P","outputId":"1a5c6e62-fde2-4321-aded-b49f74552300","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.best_value","metadata":{"_uuid":"59bdd76c-c746-498a-8d50-5859f87b65d0","_cell_guid":"ef6818a1-8ef8-4fad-ae1e-79bf28ad285a","trusted":true,"collapsed":false,"id":"NCmNbmTA9P4g","outputId":"4cadd2d7-bb02-4348-fcd6-8e4d90e74158","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"Rf = RandomForestClassifier(min_samples_split = 4,min_samples_leaf= 1,max_features ='sqrt',n_estimators= 78,max_samples = 0.8341774201566392,max_depth = 27)","metadata":{"_uuid":"9c09690e-0db3-4725-b058-2b2841593e13","_cell_guid":"49fecbbe-b523-4883-972c-f4520efa9ba2","trusted":true,"collapsed":false,"id":"teT_yjeE9d3p","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model=Rf.fit(x_train,y_train,sample_weight=sample_weights)","metadata":{"_uuid":"03dafc45-0303-452c-8c1c-04107317f69b","_cell_guid":"71bf6ef8-217a-4c9a-b62b-9bf448b7e852","trusted":true,"collapsed":false,"id":"WdjbDZAZ-AIo","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"log_loss(y_test,model.predict_proba(x_test))","metadata":{"_uuid":"5bc072c0-1046-4488-80bb-bbc72489d158","_cell_guid":"c3a08e18-a5d3-4d40-98d4-b8b39e2e693d","trusted":true,"collapsed":false,"id":"Ceypl-v_-TdN","outputId":"6ccff584-4826-46ef-cc44-fb0607b69722","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"-----\n#### Random Forest vs Decision Tree\r\n\r\nRandom Forest gave a **better log loss** than Decision Tree: `0.11735606732084684` \r\nBut it's still **not great**.\r\n\r\nIt performs better because it uses **many trees (ensemble method)**, which reduces overfitting and improves generalization.","metadata":{"_uuid":"19525d2e-6241-40ab-9132-4785a680fbac","_cell_guid":"46aa35dc-c203-4228-a89c-254703fdf2bf","trusted":true,"collapsed":false,"id":"SOM50irP-f6b","jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"# Optuna Using XG Boost","metadata":{"_uuid":"bbc3a037-3849-4254-bfdc-71cd8617a653","_cell_guid":"ea95f561-5175-45d6-9839-725d1a6b4289","trusted":true,"collapsed":false,"id":"SKpf064AeSLK","jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"from sklearn.model_selection import cross_validate\nimport optuna\nfrom sklearn.model_selection import StratifiedKFold","metadata":{"_uuid":"3489e8f7-9217-4052-94fe-778cd066939f","_cell_guid":"02d7b3ac-e445-4acd-b166-82f87fc15c46","trusted":true,"collapsed":false,"id":"z7UjFNvQXUY1","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def objective(trial):\n params = {\n 'max_depth' : trial.suggest_int('max_depth', 3, 10),\n 'learning_rate' : trial.suggest_float('learning_rate', 1e-3, 5e-2, log=True),\n 'n_estimators' : trial.suggest_int('n_estimators', 200, 1000),\n 'subsample' : trial.suggest_float('subsample', 0.6, 1.0),\n 'colsample_bytree' : trial.suggest_float('colsample_bytree', 0.6, 1.0),\n 'gamma' : trial.suggest_float('gamma', 1e-5, 1e-1, log=True),\n 'reg_alpha' : trial.suggest_float('reg_alpha', 1e-5, 1.0, log=True),\n 'reg_lambda' : trial.suggest_float('reg_lambda', 1e-5, 1.0, log=True),\n 'min_child_weight' : trial.suggest_int('min_child_weight', 1, 10),\n 'tree_method' : 'hist',\n 'predictor' : 'gpu_predictor',\n 'use_label_encoder': False,\n 'objective' : 'multi:softprob',\n 'num_class' : len(np.unique(y_train)),\n 'random_state' : 42,\n 'eval_metric' : 'mlogloss',\n }\n\n sample_weights = compute_sample_weight('balanced', y=y_train)\n skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n losses = []\n\n for tr_idx, val_idx in skf.split(x_train, y_train):\n X_tr, X_val = x_train[tr_idx], x_train[val_idx]\n y_tr, y_val = y_train[tr_idx], y_train[val_idx]\n sw_tr = sample_weights[tr_idx]\n\n\n\n model = XGBClassifier(**params)\n\n model.fit(\n X_tr,\n y_tr,\n sample_weight=sw_tr,\n eval_set=[(X_val, y_val)],\n verbose=False\n )\n\n y_pred = model.predict_proba(X_val)\n losses.append(log_loss(y_val, y_pred))\n\n trial.set_user_attr('user_attrs_train_log_loss', losses)\n return np.mean(losses)","metadata":{"_uuid":"9d1ec69e-4034-4c04-b151-5454feaae168","_cell_guid":"b0635b9b-7691-405d-ba56-3f7bb55bdfde","trusted":true,"collapsed":false,"id":"irRPxF32hzKE","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study = optuna.create_study(direction=\"minimize\", sampler= optuna.samplers.TPESampler())","metadata":{"_uuid":"9fbad6a3-d118-4591-89ca-418b56f5f444","_cell_guid":"d8e719e2-ae7c-4440-8338-83be58805cb6","trusted":true,"collapsed":false,"id":"dELfZrfaiaLL","outputId":"31ef1e8d-a6c6-4932-e18c-397a466e230c","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.optimize(objective,n_trials=50,n_jobs=-1)","metadata":{"_uuid":"df3a6767-aa8f-4c4b-a3a6-99910c8b6bea","_cell_guid":"49a1b830-a54e-4156-bf8a-3be87f1f9c2a","trusted":true,"collapsed":false,"id":"1co80fO6idqh","outputId":"57f25bfc-715d-4984-916d-26deba1c25bd","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"d2 = study.trials_dataframe()\nd2","metadata":{"_uuid":"f09d6c61-eda7-4e89-82cb-63b9ee8a3e64","_cell_guid":"4a5d5f91-c5a1-40ec-91ca-a5fffd9de2d4","trusted":true,"collapsed":false,"id":"0bIDK7EiihZw","outputId":"d51082f8-5522-4fde-8ea1-435b8c91d7cf","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":" study.best_params","metadata":{"_uuid":"c07706d6-0265-460e-8fb9-45eecae54379","_cell_guid":"52dda99a-01cc-4672-a956-6f347fc51fa1","trusted":true,"collapsed":false,"id":"wvoT-qAOLXNJ","outputId":"438e6884-1edb-43d6-fb2d-0dbc289c9821","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"study.best_value","metadata":{"_uuid":"354418a8-974e-4121-9e62-c0b6bcdea274","_cell_guid":"268c1d6c-8232-457c-927b-d9fce80ae898","trusted":true,"collapsed":false,"id":"RA0Y2R5ILd0C","outputId":"6f8e74b7-0f3f-4bf5-bea9-1c87c5eb87e8","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xgb = XGBClassifier(\n learning_rate=0.04699722743904316,\n max_depth=10,\n min_child_weight=2,\n subsample=0.6520036541391311,\n colsample_bytree= 0.811649670733542,\n gamma= 0.0037956352232668463,\n reg_alpha=0.0030466548436088757,\n reg_lambda=0.0009972779485652146,\n n_estimators=843,\n )","metadata":{"_uuid":"11a533e5-8526-400d-8542-e4d2347d3016","_cell_guid":"2c0f12aa-104f-410f-b10a-b9c0bedcdd74","trusted":true,"collapsed":false,"id":"LB7aal1MLgXI","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model=xgb.fit(x_train,y_train,sample_weight=sample_weights)","metadata":{"_uuid":"c30c43d4-d4bf-4391-b3cd-9ca7fc31c699","_cell_guid":"ba4ec942-b9d1-42ed-8ff7-32d451d61647","trusted":true,"collapsed":false,"id":"1qMvX_g9O4rX","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"log_loss(y_test,model.predict_proba(x_test))","metadata":{"_uuid":"71cb6018-89c8-41aa-a245-c5fadf4ce3ca","_cell_guid":"c0509dfc-51b7-4b08-a9cf-af5426ce8ddc","trusted":true,"collapsed":false,"id":"SchLZtkRO7pC","outputId":"24f02e09-01c7-495e-b3b4-f995a786cc09","jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### XGBoost Performance\n\nCompared to other models, **XGBoost** gave a **very good log loss**: `0.05746861827942823` \nWe can consider this the **best-performing model** so far based on log loss.","metadata":{"_uuid":"7d816ae8-1b63-4561-aba5-75fc6b7a4b13","_cell_guid":"a643f93a-4324-4145-a83a-cd83a0c03b29","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"markdown","source":"----\n### 🚧 Coming Soon: `.asm` File-Based Analysis\r\n\r\nThis project focused exclusively on hexadecimal unigram-based classification due to hardware constraints. \r\nIn the next phase, I plan to extract and analyze features from **assembly (`.asm`) files**, such as:\r\n\r\n- API call patterns \r\n- Function frequencies \r\n- Control flow structures \r\n- Opcode sequences \r\n\r\nStay tuned for **deeper static analysis** using `.asm` metadata to enhance malware family prediction. 🚀","metadata":{"_uuid":"5dd28643-ad31-4f69-80eb-c2ce011c41b1","_cell_guid":"4151d4fa-b526-43bc-b365-32b184e0b317","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}}},{"cell_type":"code","source":"","metadata":{"_uuid":"1ced8b80-ae5a-4353-8c36-02088ac40a03","_cell_guid":"76c33538-8c46-4196-a33c-f80948fa0518","trusted":true,"collapsed":false,"jupyter":{"outputs_hidden":false}},"outputs":[],"execution_count":null}]}