{"cells":[{"metadata":{"_uuid":"633511f1163a2f5ff41d3b09a7527c991eb89843","id":"6-vE4EKl_J3i","colab_type":"text"},"cell_type":"markdown","source":"# Microsoft Malware Prediction EDA Part 1\n\nIn this kernel, we'll load in the data and explore the data.\n\nWe have three aims:\n\n1. To check the variable types\n\n2. To see the relationship each variable has our response variable, ' HasDetections'\n\n3. To see how the missing values are distributed\n\nLet's get going!"},{"metadata":{"_uuid":"6a00c32731ed42b9122745eba6acb7c9f423cd5c","id":"0uVJwShh_J3m","colab_type":"text"},"cell_type":"markdown","source":"As always, let's begin by loading in useful packages....."},{"metadata":{"id":"78HkCL2HstjT","colab_type":"text","_uuid":"ccda83a6c79abf9075f6e452e6c3f00f90fd16c2"},"cell_type":"markdown","source":"# Loading in the data"},{"metadata":{"trusted":true,"_uuid":"1db1f92a748a5cd9c6191a3116ba608dbe480c49","id":"pd9kKfJs_J3n","colab_type":"code","outputId":"75c616b8-7714-4f80-c584-0946592da1aa","colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":"###Loading in useful packages\n\n#for linear algebra\nimport numpy as np\n\n#for data manipulation\nimport pandas as pd\n\n#for plotting\nimport matplotlib.pyplot as pp\nimport seaborn as sns\n%matplotlib inline\n\n#For surpressing warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n#For opening Zip files\nimport zipfile as zf","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"680bbac712081f0765e4f4ab548c7f4822a64928","id":"BqOFUaYM_J3w","colab_type":"text"},"cell_type":"markdown","source":"The first step prior to loading in the data is to specify the data types for each of the variables and the size of each data type. This will save memory because Python automatically assigns 64 bits to each numeric variables,regardless of whether it actually needs it, by specifying the size of data type, we'll save memory."},{"metadata":{"trusted":true,"_uuid":"b22d4aa4cb66ac4cd8e5ec32013aebe45c788b60","id":"37YQj7i3_J3y","colab_type":"code","colab":{}},"cell_type":"code","source":"dtypes = {\n        'MachineIdentifier':                                    'category',\n        'ProductName':                                          'category',\n        'EngineVersion':                                        'category',\n        'AppVersion':                                           'category',\n        'AvSigVersion':                                         'category',\n        'IsBeta':                                               'int8',\n        'RtpStateBitfield':                                     'float16',\n        'IsSxsPassiveMode':                                     'int8',\n        'DefaultBrowsersIdentifier':                            'float16',\n        'AVProductStatesIdentifier':                            'float32',\n        'AVProductsInstalled':                                  'float16',\n        'AVProductsEnabled':                                    'float16',\n        'HasTpm':                                               'int8',\n        'CountryIdentifier':                                    'int16',\n        'CityIdentifier':                                       'float32',\n        'OrganizationIdentifier':                               'float16',\n        'GeoNameIdentifier':                                    'float16',\n        'LocaleEnglishNameIdentifier':                          'int8',\n        'Platform':                                             'category',\n        'Processor':                                            'category',\n        'OsVer':                                                'category',\n        'OsBuild':                                              'int16',\n        'OsSuite':                                              'int16',\n        'OsPlatformSubRelease':                                 'category',\n        'OsBuildLab':                                           'category',\n        'SkuEdition':                                           'category',\n        'IsProtected':                                          'float16',\n        'AutoSampleOptIn':                                      'int8',\n        'PuaMode':                                              'category',\n        'SMode':                                                'float16',\n        'IeVerIdentifier':                                      'float16',\n        'SmartScreen':                                          'category',\n        'Firewall':                                             'float16',\n        'UacLuaenable':                                         'float32',\n        'Census_MDC2FormFactor':                                'category',\n        'Census_DeviceFamily':                                  'category',\n        'Census_OEMNameIdentifier':                             'float16',\n        'Census_OEMModelIdentifier':                            'float32',\n        'Census_ProcessorCoreCount':                            'float16',\n        'Census_ProcessorManufacturerIdentifier':               'float16',\n        'Census_ProcessorModelIdentifier':                      'float16',\n        'Census_ProcessorClass':                                'category',\n        'Census_PrimaryDiskTotalCapacity':                      'float32',\n        'Census_PrimaryDiskTypeName':                           'category',\n        'Census_SystemVolumeTotalCapacity':                     'float32',\n        'Census_HasOpticalDiskDrive':                           'int8',\n        'Census_TotalPhysicalRAM':                              'float32',\n        'Census_ChassisTypeName':                               'category',\n        'Census_InternalPrimaryDiagonalDisplaySizeInInches':    'float16',\n        'Census_InternalPrimaryDisplayResolutionHorizontal':    'float16',\n        'Census_InternalPrimaryDisplayResolutionVertical':      'float16',\n        'Census_PowerPlatformRoleName':                         'category',\n        'Census_InternalBatteryType':                           'category',\n        'Census_InternalBatteryNumberOfCharges':                'float32',\n        'Census_OSVersion':                                     'category',\n        'Census_OSArchitecture':                                'category',\n        'Census_OSBranch':                                      'category',\n        'Census_OSBuildNumber':                                 'int16',\n        'Census_OSBuildRevision':                               'int32',\n        'Census_OSEdition':                                     'category',\n        'Census_OSSkuName':                                     'category',\n        'Census_OSInstallTypeName':                             'category',\n        'Census_OSInstallLanguageIdentifier':                   'float16',\n        'Census_OSUILocaleIdentifier':                          'int16',\n        'Census_OSWUAutoUpdateOptionsName':                     'category',\n        'Census_IsPortableOperatingSystem':                     'int8',\n        'Census_GenuineStateName':                              'category',\n        'Census_ActivationChannel':                             'category',\n        'Census_IsFlightingInternal':                           'float16',\n        'Census_IsFlightsDisabled':                             'float16',\n        'Census_FlightRing':                                    'category',\n        'Census_ThresholdOptIn':                                'float16',\n        'Census_FirmwareManufacturerIdentifier':                'float16',\n        'Census_FirmwareVersionIdentifier':                     'float32',\n        'Census_IsSecureBootEnabled':                           'int8',\n        'Census_IsWIMBootEnabled':                              'float16',\n        'Census_IsVirtualDevice':                               'float16',\n        'Census_IsTouchEnabled':                                'int8',\n        'Census_IsPenCapable':                                  'int8',\n        'Census_IsAlwaysOnAlwaysConnectedCapable':              'float16',\n        'Wdft_IsGamer':                                         'float16',\n        'Wdft_RegionIdentifier':                                'float16',\n        'HasDetections':                                        'int8'\n        }\n","execution_count":null,"outputs":[]},{"metadata":{"id":"wJEqPLfD018o","colab_type":"text","_uuid":"5259ed26b887f336aa4e25aafe2e2b35268ff724"},"cell_type":"markdown","source":"When , I initally ran this on Google Colab, my next step was to unzip and save the file to somewhere else. I have done this already and, as a result, I have commented out the code."},{"metadata":{"trusted":true,"_uuid":"faa83ac24c30334fe427324dc9b7e6075bbae5ee","id":"tEHeIdEe_J34","colab_type":"code","colab":{}},"cell_type":"code","source":"#Change the training and testing sets to pickle format\n\n#I have commented these out because I have already completed this step and these files are avaliable on my drive now.\n\n#pd.read_csv(zf.ZipFile(\"/content/drive/My Drive/Microsoft/train.zip\").open(\"train.csv\"), dtype=dtypes).to_pickle(\"/content/drive/My Drive/Microsoft/train.pkl\")\n#pd.read_csv(zf.ZipFile(\"/content/drive/My Drive/Microsoft/test.zip\").open(\"test.csv\"), dtype=dtypes).to_pickle(\"/content/drive/My Drive/Microsoft/test.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19b5faecab810ba9cd3a59a8d5efc91d6082bb54","id":"Tr5_4Oqr_J38","colab_type":"code","colab":{}},"cell_type":"code","source":"#Opening the saved pickles - which contain the data :)\n\ntrain = pd.read_pickle(\"../input/mctrain/train.pkl\")\ntest =pd.read_pickle(\"../input/mctest/test.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"id":"2yJa9K6zUJpe","colab_type":"text","_uuid":"3c5612f957b6e290f5785ade64b7224d6cf13291"},"cell_type":"markdown","source":"# 1. Determining the types of variables\n"},{"metadata":{"trusted":true,"_uuid":"75c151deeb89aced06bb992a906aebc6937bc74b","id":"iM3U7Qr2_J4C","colab_type":"code","outputId":"12ab458f-43e0-41c0-ea02-dc49abefedf4","colab":{"base_uri":"https://localhost:8080/","height":51}},"cell_type":"code","source":"#Looking at shape of training data, i.e number of rows and columns\n\nprint(\"Training data dimensions\",train.shape)\nprint(\"Testing data dimension\",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"E2gntU0WHOyP","colab_type":"text","_uuid":"cc82a81e3742065d1b02126242027763db9397da"},"cell_type":"markdown","source":"Okay , so we have 82 different columns variables. Training has 83 because the it has the response variable column\n\nLet's try and distinguish the variable types"},{"metadata":{"trusted":true,"_uuid":"b08f460f59e88bda70912df3271b1973a6994871","scrolled":false,"id":"-vRMV3wl_J4G","colab_type":"code","outputId":"9d171a5a-5eea-4554-d8e5-0b919ecc6cca","colab":{"base_uri":"https://localhost:8080/","height":1513}},"cell_type":"code","source":"#Training variable data types\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{"id":"302jrl_mt1qg","colab_type":"code","outputId":"77d35415-f663-41cd-f73f-e625e02e2ce0","colab":{"base_uri":"https://localhost:8080/","height":85},"trusted":false,"_uuid":"dcc4bae7b2cfb617410b067c85e1732358b2b514"},"cell_type":"code","source":"#Make a dictionary of all the variables\nall_the_vars = list(train.drop(['MachineIdentifier','HasDetections'],axis=1))\n\n#Make a list of categorical variables\ncat_vars = [i for i in all_the_vars if (train[i].dtype.name == 'category') | (train[i].dtype.name == 'object')]\n\n#Make a list of binary variables\nbin_vars = [i for i in all_the_vars if len(train[i].value_counts()) <= 2 ]\n\n#Make a list of pure numerical variables\nnum_vars = [i for i in all_the_vars if (i not in cat_vars) & (i not in bin_vars)]\n\nprint('We have ',len(all_the_vars),' explanatory variables')\nprint('We have ',len(cat_vars),' categorical variables')\nprint('We have ',len(bin_vars),' binary variables')\nprint('We have ',len(num_vars),' numerical variables')\n","execution_count":null,"outputs":[]},{"metadata":{"id":"0jwtQLSPIbgk","colab_type":"text","_uuid":"6f4970d5c75552cc701807cf9883aa5e8b489c88"},"cell_type":"markdown","source":"We have **29 categorical variables** , **20 binary varibales**, and **33 pure numeric variables**\n\nNotice that our target variable, **HasDetections** - which is at the bottom of the list-is an integer variable. It's values are either 0 or 1.\n\n*Also note that  **MachineIdentifier** is the just the id for each computer. Not an explanatory variable*\n\n"},{"metadata":{"id":"RDXBvMu6USXq","colab_type":"text","_uuid":"8d091493226f15253438df107ace0c862ca1941a"},"cell_type":"markdown","source":"# 2. Seeing the relationship which the explanatory variables have with the response variable, 'HasDetections'\n"},{"metadata":{"id":"wvN_j2AewR5t","colab_type":"text","_uuid":"76c891323ab06ed522e93ffd67e11033d0f6ef28"},"cell_type":"markdown","source":"Let's make a bar chart of the 'HasDetections' variables, to gain an idea of how it's distributed..."},{"metadata":{"id":"VfzX9HeOjS8S","colab_type":"code","outputId":"90654d48-6598-43f0-e35b-f82b103a3aee","colab":{"base_uri":"https://localhost:8080/","height":290},"trusted":false,"_uuid":"e4405d9aa74d17f9e1a6f333978e8bdc19f5ffc8"},"cell_type":"code","source":"#Let's examine the distribution of HasDetections\nj = train['HasDetections'].value_counts()\nj = j/len(train)\nj.plot.bar()\npp.title('Distribution of HasDetections')\npp.xlabel('HasDetections values')\npp.ylabel('Proportion')\npp.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"XN4RL0Dvwlk5","colab_type":"text","_uuid":"78a9a5650c7fed31236f2e8274b8527f73f4e267"},"cell_type":"markdown","source":"Okay, so it looks like a pretty even split between not having malware detected and having malware detected.\n\nIf we just use historical training data, it seems that getting malware is a coin flip. Keeping this in mind, let's see how each features interacts with 'HasDetections' and if the probabilities of detecting malware for each feature are more insightful than a coin flip."},{"metadata":{"id":"LQPZ5ZdNYnfR","colab_type":"code","outputId":"d710362b-acb5-4183-ae21-684678630587","colab":{"base_uri":"https://localhost:8080/","height":9930},"trusted":false,"_uuid":"363eb0061f2e581902b1b3db6f954b106f21c1d2"},"cell_type":"code","source":"for i in cat_vars:\n  length = len(train)\n  j = train[i].value_counts() / length\n  j = j.sort_values(ascending=False)\n  j = j.iloc[:10]\n  x = list(j.index)\n  y = list(j.values)\n  z= list()\n\n  for j in x:\n    z.append(train['HasDetections'].loc[train[i]==j].mean())\n  \n\n  fig, ax1 = pp.subplots()\n  ax1.bar(x,y)\n  pp.xticks(x,y)\n  locs, labels = pp.xticks()\n  pp.setp(labels, rotation=90)\n  pp.title(i)\n  pp.ylabel('Proportion')\n  pp.xlabel(i + ' values')\n\n  ax2 = ax1.twinx()\n  ax2.plot(x,z,'r',linestyle='-', marker='o')\n  ax1.grid(False)\n  ax2.grid(False)\n  pp.ylabel('P(HasDetections == 1)')\n\n  pp.show()\n  \n","execution_count":null,"outputs":[]},{"metadata":{"id":"sA1og9Gs3LsP","colab_type":"text","_uuid":"b505c93cf1c5222a1a69f1c2837f2bbd04b9bdc5"},"cell_type":"markdown","source":"So my criteria for potentially useful variables or parts of variables is whether is\n| P(HasDetections ==1) - 0.5 |  > 0.04 :\n\n'ProductName_windowsintune', 'EngineVersion', 'AppVersion','AvSigVersion' , 'Platform_windows2016', 'Processor_x86', 'Processor_am64', 'OsVer','OsPlatformSubRelease', ' OsBuildLab', 'SkuEdition_Server', 'PuaMode_audit', 'SmartScreen_RequireAdmin', 'SmartScreen_ExisitsNotSet ', 'Census_MDC2FormFactor', 'Census_DeviceFamily_Windows.Server','Census_DeviceFamily_Windows', 'Censor_ProcessorClass_mid', 'Census_ProcessorClass_high', 'Census_PrimaryDiskTypeName_UNKNOWN', 'Census_PrimaryDiskTypeName_Unspecified', 'Census_ChassisTypeName_Other', 'Census_PowerPlatformRoleName_Slate', 'Census_PowerPlatformRoleName_AppliancePC' ,'Census_PowerPlatformRoleName_EntepriseServer', 'Census_InternalBatteryType_nimh', ##'Census_InternalBatteryType_log20'##, 'Census_OSVersion',  'Census_OSArchitecture_x86', ''Census_OSArchitecture_am64'', 'Census_OSBranch', 'Census_OSInstallTypeName', 'Census_OSWUAutoUpdateOptionsName_DownloadNotify', 'Census_GenuineStateName_UNKNOWN', 'Census_ActivationChannel' and 'Census_FlightRing'\n"},{"metadata":{"id":"EHRyS6FAk4xx","colab_type":"code","outputId":"4d38adb4-79af-479f-869b-52162100e5fe","colab":{"base_uri":"https://localhost:8080/","height":5557},"trusted":false,"_uuid":"9a835f29dc6bbf87332b8c8f23baa0b94eae6100"},"cell_type":"code","source":"for i in bin_vars:\n  length = len(train)\n  j = train[i].value_counts() / length\n  j = j.sort_values(ascending=False)\n  x = list(j.index)\n  y = list(j.values)\n  z= list()\n\n  for j in x:\n    z.append(train['HasDetections'].loc[train[i]==j].mean())\n  \n\n  fig, ax1 = pp.subplots()\n  ax1.bar(x,y)\n  pp.title(i)\n  pp.ylabel('Proportion')\n  pp.xlabel(i + ' values')\n\n  ax2 = ax1.twinx()\n  ax2.plot(x,z,'r',linestyle='-', marker='o')\n  ax1.grid(False)\n  ax2.grid(False)\n  pp.ylabel('P(HasDetections == 1)')\n\n  pp.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"HDmBxwrz0Kxq","colab_type":"text","_uuid":"a1dc1ff8672b40a9b9f93f511742dcf6b8b4658d"},"cell_type":"markdown","source":"For binary variables,same criteria as before, the notable areas seem to be:\n\n'Wdft_IsGamer_1',  'Census_IsAlwaysOnAlwaysConnectedCapable_1', 'Census_IsPenCapable_1', 'Census_IsTouchEnabled_1' , 'Census_IsVirtualDevice_1', ##'Cenus_IsWIMBootEnabled_1'##, ##'Census_ThresholdOptIn_1'##, 'Census_IsFlightingInternal_1', 'Census_IsPortableOperatingSystem', 'Census_HasOpticalDiskDrive' , ##'Firewall_0'## ,'SMode_1', 'PuaMode_audit' , 'AutoSampleOptIn_1 ', 'IsProtected_0', 'IsSxsPassiveMode_1'"},{"metadata":{"id":"qzjzdishr3hg","colab_type":"code","outputId":"84d06042-623c-4b56-a5e8-639d33e76996","colab":{"base_uri":"https://localhost:8080/","height":34},"trusted":false,"_uuid":"0df330ddc45c8ebf09bf11af6b4e6ebbbcc0266e"},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"id":"S9N3Ip0HmUhw","colab_type":"code","outputId":"e4ed36f0-117c-40da-f106-f2fb346948a4","colab":{"base_uri":"https://localhost:8080/","height":9158},"trusted":false,"_uuid":"deeab88fe7372a3c4b770e53f8362d7c61d51ae6"},"cell_type":"code","source":"for i in num_vars:\n  length = len(train)\n  j = train[i].value_counts() / length\n  j = j.sort_values(ascending=False)\n  x = np.array(j.index)\n  y = np.array(j.values)\n  z= list()\n\n  ax = sns.kdeplot(train[i].loc[train['HasDetections']==1],label='HasDetections ==1',color='r')\n  ax = sns.kdeplot(train[i].loc[train['HasDetections']==0],label='HasDetections ==0',color='b')\n  pp.title(i)\n  pp.ylabel('Density')\n  pp.xlabel(i+' values')\n\n  \n  pp.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"-CBVgR9KVqur","colab_type":"text","_uuid":"bc3691280a0fffe147bd1e988dec3d47e41e4a51"},"cell_type":"markdown","source":"For pure numerics, my criteria is whether there appears to at least one siginificant divergence between the lines, it's a subjective measure. the notable variables seem  to be:\n\n'DefaultBrowsersIdentifier' , 'AVProductStatesIdentifier', 'AVProductsInstalled' , 'CountryIdentifier', 'CityIdentifier', 'UacLuaenable', 'Censor_ProcessorModelIdentifier', 'Census_PrimaryDiskTotalCapacity' , 'Census_SystemVolumeTotalCapacity', 'Census_TotalPhysicalRAM', 'Census_InternalPrimaryDiagnonalDisplaySizeInInches' , 'Census_InternalPrimaryDisplayResolutionVertical', 'Census_FirmwareManufacturerIdentifier'\n "},{"metadata":{"id":"cYsgLBHQsgha","colab_type":"text","_uuid":"e7fddec2f9b83384f19d8098717f3c8937d99b99"},"cell_type":"markdown","source":"# 3. Checking Missing Values"},{"metadata":{"id":"hOmT0h4OJaDq","colab_type":"text","_uuid":"9cf4536fd91115b630882c9cb4d3ab3e5c446905"},"cell_type":"markdown","source":"Let's see the breakdown of missing values in our training data\n"},{"metadata":{"trusted":true,"_uuid":"e998589e0514b4378e6ae9038e86719748dcca41","scrolled":false,"id":"XzoipAY0_J4M","colab_type":"code","outputId":"8418a2f7-5e2f-4f2a-c4aa-e01b00b9b9ce","colab":{"base_uri":"https://localhost:8080/","height":782}},"cell_type":"code","source":"#Let's creating a series which contains the proportion of missing values for each variable\nmis_val = 100 * train.isnull().sum()/len(train)\n\n#Let's view variables with missing values\nmis_val[mis_val >0].sort_values(ascending=False)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4069081f1eef7151b10cbee631e7961b8199b96","id":"A2pGHoQI_J4R","colab_type":"text"},"cell_type":"markdown","source":"Over half of our variables have missing values :(\n\n\nPuaMode, Censor_ProcessorClass,DefaultBrowerIdentifier, Census_IsFlightingInternal,Census_InternalBatteryType,Census_ThresholdOptIn and Census_IsWIMBootEnabled have the greatest proportion of missing values.\n\nHaving dug further, PuaMode, Census_InternalBatteryType and SmartScreen seem potentially viable, intuitively. \nI'm very tempted,however, to remove the other  variables  with high ratios of missing values BUT let's  examine the data further before doing this."},{"metadata":{"id":"UYntGZnvv38E","colab_type":"text","_uuid":"d41589db0387d027bee90131ea72a466726f1267"},"cell_type":"markdown","source":"# Save the files\n\nSo we've accomplished our three aims and in our next kernels, we'll deal with missing values and possibly add new variables"},{"metadata":{"id":"8WB1cP8cAMOx","colab_type":"code","colab":{},"trusted":false,"_uuid":"6bd386372068fa6a6a1bf6779befc9e5b1058a4e"},"cell_type":"code","source":"#Save them to separate pickles\n#train.to_pickle(\"/content/drive/My Drive/Microsoft/train.pkl\")\n#test.to_pickle(\"/content/drive/My Drive/Microsoft/test.pkl\")","execution_count":null,"outputs":[]}],"metadata":{"colab":{"name":"Microsoft Malware Prediction Data Exploration Part 1.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"accelerator":"GPU","language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}