{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Neural Network - Statistical Encoding - Microsoft Malware\nThere aren't any examples of using a neural network to model Microsoft Malware, so I thought I'd post one. Also in this kernel, I show statistical one-hot-encoding where only boolean variables that are independently statistically significant are created.\n\n# Load Train.csv"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# IMPORT LIBRARIES\nimport pandas as pd, numpy as np, os, gc\n\n# LOAD AND FREQUENCY-ENCODE\nFE = ['EngineVersion','AppVersion','AvSigVersion','Census_OSVersion']\n# LOAD AND ONE-HOT-ENCODE\nOHE = [ 'RtpStateBitfield','IsSxsPassiveMode','DefaultBrowsersIdentifier',\n        'AVProductStatesIdentifier','AVProductsInstalled', 'AVProductsEnabled',\n        'CountryIdentifier', 'CityIdentifier', \n        'GeoNameIdentifier', 'LocaleEnglishNameIdentifier',\n        'Processor', 'OsBuild', 'OsSuite',\n        'SmartScreen','Census_MDC2FormFactor',\n        'Census_OEMNameIdentifier', \n        'Census_ProcessorCoreCount',\n        'Census_ProcessorModelIdentifier', \n        'Census_PrimaryDiskTotalCapacity', 'Census_PrimaryDiskTypeName',\n        'Census_HasOpticalDiskDrive',\n        'Census_TotalPhysicalRAM', 'Census_ChassisTypeName',\n        'Census_InternalPrimaryDiagonalDisplaySizeInInches',\n        'Census_InternalPrimaryDisplayResolutionHorizontal',\n        'Census_InternalPrimaryDisplayResolutionVertical',\n        'Census_PowerPlatformRoleName', 'Census_InternalBatteryType',\n        'Census_InternalBatteryNumberOfCharges',\n        'Census_OSEdition', 'Census_OSInstallLanguageIdentifier',\n        'Census_GenuineStateName','Census_ActivationChannel',\n        'Census_FirmwareManufacturerIdentifier',\n        'Census_IsTouchEnabled', 'Census_IsPenCapable',\n        'Census_IsAlwaysOnAlwaysConnectedCapable', 'Wdft_IsGamer',\n        'Wdft_RegionIdentifier']\n\n# LOAD ALL AS CATEGORIES\ndtypes = {}\nfor x in FE+OHE: dtypes[x] = 'category'\ndtypes['MachineIdentifier'] = 'str'\ndtypes['HasDetections'] = 'int8'\n\n# LOAD CSV FILE\ndf_train = pd.read_csv('../input/train.csv', usecols=dtypes.keys(), dtype=dtypes)\nprint ('Loaded',len(df_train),'rows of TRAIN.CSV!')\n\n# DOWNSAMPLE\nsm = 2000000\ndf_train = df_train.sample(sm)\nprint ('Only using',sm,'rows to train and validate')\nx=gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7a58a55b1617fefb9e9577546e11aa6252e4935"},"cell_type":"markdown","source":"# Statistically Encode Variables\nAll the variables in the Python variable list `FE` will get frequency encoded and all the variables in list `OHE` will get statistically one-hot-encoded. In total, forty-three variables are imported from the training csv while thirty-nine were ignored.\n  \nAmong all our category variables, there are a combined 211,562 values! So we can't one-hot-encode all. (Note that this is without Census_OEMModelIdentifier's 175,366 or Census_SystemVolumeTotalCapacity's 536,849) We will use a trick from statistics. First we'll assume we have a random sample. (Which we don't actually have, but let's pretend.) Then for each value, we will test the following hypotheses   \n\n $$H_0: \\text{Prob(HasDetections=1 given value is present)} = 0.5 $$ \n $$H_A: \\text{Prob(HasDetections=1 given value is present)} \\ne 0.5$$  \n    \nThe test statistic z-value equals \\\\( \\hat{p} \\\\), the observed HasDetections rate given value is present, minus 0.5 divided by the standard deviation of \\\\( \\hat{p} \\\\). The Central Limit Theorem tells us\n\n$$\\text{z-value} = \\frac{\\hat{p}-0.5}{SD(\\hat{p})} = 2 (\\hat{p} - 0.5)\\sqrt{n} $$\n\nwhere \\\\(n\\\\) is the number of occurences of the value. If the absolute value of \\\\(z\\\\) is greater than 2.0, we are 95% confident that Prob(HasDetections=1 given value is present) is not equal 0.5 and we will include a boolean for this value in our model. Actually, we'll use a \\\\(z\\\\) threshold of 5.0 and require \\\\( 10^{-7}n>0.005 \\\\). This adds 350 new boolean variables (instead of naively one-hot-encoding 211,562!).  \n  \n ## Example - Census_FirmwareManufacturerIdentifier\nIn the plots below, the dotted lines use the right y-axis and solid lines/bars use the left. The top plot below shows 20 values of variable `Census_FirmwareManufacturerIdentifier`. Notice that I consider NAN a value. Each of these values contains over 0.5% of the data. And all the variables together contain 97% of the data. Value=93 has a HasDetections rate of 52.5% while value=803 has a HasDetections rate of 35.4%. Their z-values are \\\\(22.2 = 2\\times(0.5253-0.5)\\times\\sqrt{192481} \\text{  }\\\\)  and \\\\(-71.3 = 2\\times(0.3535-0.5)\\times\\sqrt{59145}\\text{  }\\\\)   respectively! The probability that value=93 and value=803 have a HasDetections rate of 50% and what we are observing is due to chance is close to nothing. Additionally from the bottom plot, you see that these two values have consistently been high and low throughout all of the year 2018. We can trust that this trend will continue into the test set's October and November computers.  \n   \n![image](http://playagricola.com/Kaggle/Firm13019.png)\n\n## Python Code\nTo see the Python encoding functions, click 'see code' to the right.  "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1fadf38ea607fb94432c728a140c53f0f66139ca"},"cell_type":"code","source":"import math\n\n# CHECK FOR NAN\ndef nan_check(x):\n    if isinstance(x,float):\n        if math.isnan(x):\n            return True\n    return False\n\n# FREQUENCY ENCODING\ndef encode_FE(df,col,verbose=1):\n    d = df[col].value_counts(dropna=False)\n    n = col+\"_FE\"\n    df[n] = df[col].map(d)/d.max()\n    if verbose==1:\n        print('FE encoded',col)\n    return [n]\n\n# ONE-HOT-ENCODE ALL CATEGORY VALUES THAT COMPRISE MORE THAN\n# \"FILTER\" PERCENT OF TOTAL DATA AND HAS SIGNIFICANCE GREATER THAN \"ZVALUE\"\ndef encode_OHE(df, col, filter, zvalue, tar='HasDetections', m=0.5, verbose=1):\n    cv = df[col].value_counts(dropna=False)\n    cvd = cv.to_dict()\n    vals = len(cv)\n    th = filter * len(df)\n    sd = zvalue * 0.5/ math.sqrt(th)\n    #print(sd)\n    n = []; ct = 0; d = {}\n    for x in cv.index:\n        try:\n            if cv[x]<th: break\n            sd = zvalue * 0.5/ math.sqrt(cv[x])\n        except:\n            if cvd[x]<th: break\n            sd = zvalue * 0.5/ math.sqrt(cvd[x])\n        if nan_check(x): r = df[df[col].isna()][tar].mean()\n        else: r = df[df[col]==x][tar].mean()\n        if abs(r-m)>sd:\n            nm = col+'_BE_'+str(x)\n            if nan_check(x): df[nm] = (df[col].isna()).astype('int8')\n            else: df[nm] = (df[col]==x).astype('int8')\n            n.append(nm)\n            d[x] = 1\n        ct += 1\n        if (ct+1)>=vals: break\n    if verbose==1:\n        print('OHE encoded',col,'- Created',len(d),'booleans')\n    return [n,d]\n\n# ONE-HOT-ENCODING from dictionary\ndef encode_OHE_test(df,col,dt):\n    n = []\n    for x in dt: \n        n += encode_BE(df,col,x)\n    return n\n\n# BOOLEAN ENCODING\ndef encode_BE(df,col,val):\n    n = col+\"_BE_\"+str(val)\n    if nan_check(val):\n        df[n] = df[col].isna()\n    else:\n        df[n] = df[col]==val\n    df[n] = df[n].astype('int8')\n    return [n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afa1ab97768b80133161a00f0037f652f29e648b"},"cell_type":"code","source":"cols = []; dd = []\n\n# ENCODE NEW\nfor x in FE:\n    cols += encode_FE(df_train,x)\nfor x in OHE:\n    tmp = encode_OHE(df_train,x,0.005,5)\n    cols += tmp[0]; dd.append(tmp[1])\nprint('Encoded',len(cols),'new variables')\n\n# REMOVE OLD\nfor x in FE+OHE:\n    del df_train[x]\nprint('Removed original',len(FE+OHE),'variables')\nx = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30aa82afc92de7e295d11429636b5c50163c4e8c"},"cell_type":"markdown","source":"## Example - Census_OEMModelIdentifier\nBelow is variable `Census_OEMModelIdentifier`. Observe how NAN is treated like a category value and that it has consistently had the lowest HasDetections rate all of year 2018. Also notice how value=245824 has consistently been high. Finally note that value=188345 and 248045 are high and low respectively in August and September but earlier in the year their positions were reversed! What will their positions be in the test set's October and November computers??  \n  \n![image](http://playagricola.com/Kaggle/OEM13019.png)"},{"metadata":{"_uuid":"c99423ce7d2c30395ddddaf39ec7d56d87defa17"},"cell_type":"markdown","source":"# Build and Train Network\nWe will a build a 3 layer fully connected network with 100 neurons on each hidden layer. We will use ReLU activation, Batch Normalization, 40% Dropout, Adam Optimizer, and Decaying Learning Rate. Unfortunately we don't have an AUC loss function, so we will use Cross Entrophy instead. After each epoch, we will call a custom Keras callback to display the current AUC and continually save the best model."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true,"_uuid":"dc3ee8729926910d48e9543e73e7f29791dc399d"},"cell_type":"code","source":"from keras import callbacks\nfrom sklearn.metrics import roc_auc_score\n\nclass printAUC(callbacks.Callback):\n    def __init__(self, X_train, y_train):\n        super(printAUC, self).__init__()\n        self.bestAUC = 0\n        self.X_train = X_train\n        self.y_train = y_train\n        \n    def on_epoch_end(self, epoch, logs={}):\n        pred = self.model.predict(np.array(self.X_train))\n        auc = roc_auc_score(self.y_train, pred)\n        print(\"Train AUC: \" + str(auc))\n        pred = self.model.predict(self.validation_data[0])\n        auc = roc_auc_score(self.validation_data[1], pred)\n        print (\"Validation AUC: \" + str(auc))\n        if (self.bestAUC < auc) :\n            self.bestAUC = auc\n            self.model.save(\"bestNet.h5\", overwrite=True)\n        return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"611fc3d8e0d92288a6c1d558a9ebaed10cef97e5"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, BatchNormalization, Activation\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.optimizers import Adam\n\n#SPLIT TRAIN AND VALIDATION SET\nX_train, X_val, Y_train, Y_val = train_test_split(\n    df_train[cols], df_train['HasDetections'], test_size = 0.5)\n\n# BUILD MODEL\nmodel = Sequential()\nmodel.add(Dense(100,input_dim=len(cols)))\nmodel.add(Dropout(0.4))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(Dense(100))\nmodel.add(Dropout(0.4))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(Dense(1, activation='sigmoid'))\nmodel.compile(optimizer=Adam(lr=0.01), loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\nannealer = LearningRateScheduler(lambda x: 1e-2 * 0.95 ** x)\n\n# TRAIN MODEL\nmodel.fit(X_train,Y_train, batch_size=32, epochs = 20, callbacks=[annealer,\n          printAUC(X_train, Y_train)], validation_data = (X_val,Y_val), verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4dbc958dd56c5af4d74bdeded9696ff244246f02"},"cell_type":"markdown","source":"# Predict Test and Submit to Kaggle\nEven after deleting the training data, our network still needs lot of our available RAM, we'll need to load in test.csv by chunks and predict by chunks. Click 'see code' button to see how this is done."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"96e7fbf730e30707f04e4b192c839c820af5dee4"},"cell_type":"code","source":"del df_train\ndel X_train, X_val, Y_train, Y_val\nx = gc.collect()\n\n# LOAD BEST SAVED NET\nfrom keras.models import load_model\nmodel = load_model('bestNet.h5')\n\npred = np.zeros((7853253,1))\nid = 1\nchunksize = 2000000\nfor df_test in pd.read_csv('../input/test.csv', \n            chunksize = chunksize, usecols=list(dtypes.keys())[0:-1], dtype=dtypes):\n    print ('Loaded',len(df_test),'rows of TEST.CSV!')\n    # ENCODE TEST\n    cols = []\n    for x in FE:\n        cols += encode_FE(df_test,x,verbose=0)\n    for x in range(len(OHE)):\n        cols += encode_OHE_test(df_test,OHE[x],dd[x])\n    # PREDICT TEST\n    end = (id)*chunksize\n    if end>7853253: end = 7853253\n    pred[(id-1)*chunksize:end] = model.predict(df_test[cols])\n    print('  encoded and predicted part',id)\n    id += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c50d62226fb92795556adcf192b685c3e496b14e"},"cell_type":"code","source":"# SUBMIT TO KAGGLE\ndf_test = pd.read_csv('../input/test.csv', usecols=['MachineIdentifier'])\ndf_test['HasDetections'] = pred\ndf_test.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f873a4954f0f70419c331a93dcc04c2f2fb97b04"},"cell_type":"markdown","source":"![image](http://playagricola.com/Kaggle/NN13019.png)"},{"metadata":{"_uuid":"e161d9a3a1c67d29ac1e991235b6b4ed7c12f19e"},"cell_type":"markdown","source":"# Conclusion\nIn this kernel, we saw how to build and train a neural network with Keras. We also saw how to statistically one-hot-encode categorical variables. Our validation AUC was 0.703 and our LB was 0.671. So it appears that we are not time generalizing enough to the test set. Furthermore, other users are getting higher CV scores, so we should be able to improve our AUC by adding more variables and tuning our network more. \n\nIf anyone forks this kernel and improves it's AUC, let me know. All comments and suggestions are welcome. Thanks for reading."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}