{"cells":[{"metadata":{"_uuid":"c1e1d616adb350226f1017b764158d035856b62d"},"cell_type":"markdown","source":"# Lets take a look at such features as DisplayResolution and DisplaySizeInInches and see if they might be helpful\n\n**Content:**\n\n1. [Idea description](#1)\n2. [Loading data](#2)\n3. [Analyzing Display features](#3)\n\n    3.1. [New feature - resolution ratio](#4)\n    \n    3.2. [Correlation of new feature with HasDetections](#5)\n    \n    3.3. [Correlation between screen quality and HasDetections](#6)\n    \n    3.4. [Display size correlation with HasDetections](#7)\n4. [Two features interaction](#8)\n\n    4.1. [Resolution Rate / Wdft_IsGamer](#9)\n    \n    4.2. [Resolution Rate / Census_IsTouchEnabled](#10)\n    \n    4.3. [Display quality / Processor architecture](#11)\n    \n    4.4. [Display quality / Wdft_IsGamer](#12)\n\n\n<a id=\"1\"></a>\n# 1. Idea description\nThe main idea is that PC for a home use, most likely, would have medium to big sizes of displays and at least FullHD resolution. Those are some kind of 'gaming PCs' being used by people mostly for surfing and playing. And this is a kind of people who take less precautions regarding malware. Also the most popular aspect ratio amongst gamers is 16:9 (here is a pretty fresh review: https://www.gamingscan.com/best-aspect-ratio-for-gaming/) so we will take a look at aspect ratio as well by creating a new feature.\n\nOn the other had small-sized displays are mostly used either on laptops and (sometimes) servers. We are going to take a look on them as well."},{"metadata":{"_uuid":"098c0e25c204ec0286ebd3bbf4f62e975da70d54"},"cell_type":"markdown","source":"<a id=\"2\"></a>\n# 2. Loading data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import warnings\nwarnings.simplefilter('ignore')\nimport pandas as pd\nimport numpy as np\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport gc\nimport multiprocessing\nimport seaborn as sns\npd.set_option('display.max_columns', 83)\npd.set_option('display.max_rows', 83)\nplt.style.use('seaborn')\nimport os\nimport plotly.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import init_notebook_mode, iplot\nimport cufflinks\nimport plotly\nimport matplotlib\ninit_notebook_mode()\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl', offline=True)\nprint(os.listdir(\"../input\"))\nfor package in [pd, np, sns, matplotlib, plotly]:\n    print(package.__name__, 'version:', package.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5b72405cb9420901aa5ce8f2b2b6145810a3eb2","_kg_hide-input":true},"cell_type":"code","source":"dtypes = {\n        'Census_InternalPrimaryDiagonalDisplaySizeInInches':    'float16',\n        'Census_InternalPrimaryDisplayResolutionHorizontal':    'float16',\n        'Census_InternalPrimaryDisplayResolutionVertical':      'float16',\n        'Census_IsTouchEnabled':                                'int8',\n        'Census_IsPenCapable':                                  'int8',\n        'Wdft_IsGamer':                                         'float16',\n        'Processor':                                            'category',\n        'HasDetections':                                        'int8'\n        }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"137d2a35649444c327f511448b09b3c78aea0c4e"},"cell_type":"code","source":"def load_dataframe(dataset):\n    usecols = dtypes.keys()\n    if dataset == 'test':\n        usecols = [col for col in dtypes.keys() if col != 'HasDetections']\n    df = pd.read_csv(f'../input/{dataset}.csv', dtype=dtypes, usecols=usecols)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"%%time\nwith multiprocessing.Pool() as pool: \n    train, test = pool.map(load_dataframe, [\"train\", \"test\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97409af7cd31e8dfe176c1df13cc90e158658acf"},"cell_type":"markdown","source":"<a id=\"3\"></a>\n# 3. Analyzing Display features\n\nFirst lets take a look at the most popular display resoultion both vertical and horizontal"},{"metadata":{"trusted":true,"_uuid":"d7dabfd5cf4016b8c8aa9ab59a01554c261e9137"},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(18,8))\ntrain['Census_InternalPrimaryDisplayResolutionHorizontal'].value_counts().head(10).plot(kind='barh', ax=axes[0], fontsize=14).set_xlabel('Horizontal Resolution', fontsize=18)\ntrain['Census_InternalPrimaryDisplayResolutionVertical'].value_counts().head(10).plot(kind='barh', ax=axes[1], fontsize=14).set_xlabel('Vertical Resolution', fontsize=18)\naxes[0].invert_yaxis()\naxes[1].invert_yaxis()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4b2d7b1fe8d83771399aa197af67b7cb38cb047"},"cell_type":"markdown","source":"Ok, this doesn't tell us really much.\n\nLets create a new feature - ResolutionRation by dividing vertical resolution by horizontal resolution and see what we will have in result."},{"metadata":{"_uuid":"acc8e2986b50affece5b8f2b1744dffcc30f25ef"},"cell_type":"markdown","source":"<a id=\"4\"></a>\n## 3.1 New feature - resolution ratio"},{"metadata":{"trusted":true,"_uuid":"013dc46f839bc36c456c4caa964e050ec1965535"},"cell_type":"code","source":"train['ResolutionRatio'] = train['Census_InternalPrimaryDisplayResolutionVertical'] / train['Census_InternalPrimaryDisplayResolutionHorizontal']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23445489e233d490c25c3cc4779f87e0079875d2"},"cell_type":"code","source":"train['ResolutionRatio'].value_counts().head(10).plot(kind='barh', figsize=(14,8), fontsize=14);\nplt.gca().invert_yaxis()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f717897e247660300d9ae8faa53601329a4ced85"},"cell_type":"markdown","source":"So 4 most popular rations are:\n\n* 0.562011 is (mostly) a low-end laptops ratio (1366 by 768)\n* 0.5625 is 16:9 ratio. Most popular amongst gamers\n* 0.625 is 16:10 ratio. Characteristic of the old displays (https://en.wikipedia.org/wiki/Display_aspect_ratio#4:3_and_16:10)\n* 0.75 is 4:3 ratio. Also really old displays"},{"metadata":{"_uuid":"33b7f9ddb61d67bef074783f3e52a2b47f615c1f"},"cell_type":"markdown","source":"<a id=\"5\"></a>\n## 3.2 Correlation of new feature with HasDetections\n\nNow lets see dependency  between most popular resolution ratios and target value."},{"metadata":{"trusted":true,"_uuid":"d22da27b3a19c2c74002bdf6c07c99cf40569bf5","_kg_hide-input":true},"cell_type":"code","source":"ratios = train['ResolutionRatio'].value_counts().head(6).index\nfig, axes = plt.subplots(nrows=int(len(ratios) / 2), ncols=2, figsize=(16,14))\nfig.subplots_adjust(wspace=0.2, hspace=0.4)\nfor i in range(len(ratios)):\n    sns.countplot(x='ResolutionRatio', hue='HasDetections', data=train[train['ResolutionRatio'] == ratios[i]], ax=axes[i // 2,i % 2]);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0c8a715e7c2b7509e05af2eddc7265b63eafaef"},"cell_type":"markdown","source":"So 'gamers' (0.5625, which is 16:9) have more malware detections than others.\n\nWith next step lets divide all displays into 4 categories: low-definition (SD), high-definition (HD), FullHD and 4k and see detections distribution.\n\n<a id=\"6\"></a>\n## 3.3 Correlation between screen quality and HasDetections"},{"metadata":{"trusted":true,"_uuid":"9b1ef7a5ed3bfbb7ad6fdb0eac9293565f3e6b43","_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(18,10))\ntrain.loc[train['Census_InternalPrimaryDisplayResolutionVertical'] < 720, 'HasDetections'].value_counts().sort_index().plot(kind='bar', rot=0, ax=axes[0,0]).set_xlabel('SD');\ntrain.loc[(train['Census_InternalPrimaryDisplayResolutionVertical'] >= 720) & (train['Census_InternalPrimaryDisplayResolutionVertical'] < 1080), 'HasDetections'].value_counts().sort_index().plot(kind='bar', rot=0, ax=axes[0,1]).set_xlabel('HD');\ntrain.loc[(train['Census_InternalPrimaryDisplayResolutionVertical'] >= 1080) & (train['Census_InternalPrimaryDisplayResolutionVertical'] < 2160), 'HasDetections'].value_counts().sort_index().plot(kind='bar', rot=0, ax=axes[1,0]).set_xlabel('FullHD');\ntrain.loc[train['Census_InternalPrimaryDisplayResolutionVertical'] >= 2160, 'HasDetections'].value_counts().sort_index().plot(kind='bar', rot=0, ax=axes[1,1]).set_xlabel('4k');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad24a4083e97c13c58562b64ad40c1fda742c00a"},"cell_type":"markdown","source":"Same with plotly"},{"metadata":{"trusted":true,"_uuid":"9a515cd2b0688bae8eee2b8397443650ec66c5cc","_kg_hide-input":true},"cell_type":"code","source":"sd_values = train.loc[train['Census_InternalPrimaryDisplayResolutionVertical'] < 720, 'HasDetections'].value_counts().sort_index().values\nhd_values = train.loc[(train['Census_InternalPrimaryDisplayResolutionVertical'] >= 720) & (train['Census_InternalPrimaryDisplayResolutionVertical'] < 1080), 'HasDetections'].value_counts().sort_index().values\nfullhd_values = train.loc[(train['Census_InternalPrimaryDisplayResolutionVertical'] >= 1080) & (train['Census_InternalPrimaryDisplayResolutionVertical'] < 2160), 'HasDetections'].value_counts().sort_index().values\nk_values = train.loc[train['Census_InternalPrimaryDisplayResolutionVertical'] >= 2160, 'HasDetections'].value_counts().sort_index().values\nx = ['SD', 'HD', 'FullHD', '4k']\ny_0 = [sd_values[0], hd_values[0], fullhd_values[0], k_values[0]]\ny_1 = [sd_values[1], hd_values[1], fullhd_values[1], k_values[1]]\ntrace1 = go.Bar(x=x, y=y_0, name='0 (no detections)')\ntrace2 = go.Bar(x=x, y=y_1, name='1 (has detections)')\ndata = [trace1, trace2]\nlayout = go.Layout(barmode='group')\nfig = go.Figure(data=data, layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3ec26624f7c64e608e9a245390f772e732331ab"},"cell_type":"markdown","source":"The better display quality is the higher rate of malware detections."},{"metadata":{"_uuid":"f49f24829876b2588b76d23c047e6e614c7b68c3"},"cell_type":"markdown","source":"<a id=\"7\"></a>\n## 3.4 Display size correlation with HasDetections\n\nHope this plot makes sense. We can see here that the bigger display is the higher detection rate is, but also the distribution density is higher for small screens."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"98d18f5062ce1f48e4bcc572922fc400704f53a6"},"cell_type":"code","source":"def movingaverage(interval, window_size):\n    window = np.ones(int(window_size))/float(window_size)\n    return np.convolve(interval, window, 'same')\n\nplot_dict = dict()\nfor i in train['Census_InternalPrimaryDiagonalDisplaySizeInInches'].value_counts().sort_index().index:\n    try:\n        plot_dict[i] = train.loc[train['Census_InternalPrimaryDiagonalDisplaySizeInInches'] == i, 'HasDetections'].value_counts(normalize=True)[1]\n    except:\n        plot_dict[i] = 0.0\nfig, ax1 = plt.subplots(figsize=(16,7))\nax1.set_xlabel('Display Size in inches')\nax1.set_ylabel('Count', color='tab:green')\nax1.hist(plot_dict.keys(), color='tab:green', bins=int(len(plot_dict) / 20))\nax1.tick_params(axis='y', labelcolor='tab:green')\nax2 = ax1.twinx()\nax2.set_ylabel('Detection Rate', color='blue')\nax2.plot(plot_dict.keys(), movingaverage(list(plot_dict.values()), int(len(plot_dict) / 20)),color='blue', linewidth=2.0)\nax2.tick_params(axis='y', labelcolor='blue')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2c64ce0a392ecdf49596605f29a2ea60592a094"},"cell_type":"markdown","source":"<a id=\"8\"></a>\n# 4. Two features interaction\n<a id=\"9\"></a>\n## 4.1  Resolution Rate / Wdft_IsGamer\n\nNext will plot correlation between Display Resolution Rate and feature Wdft_IsGamer also with respect to a detection rate."},{"metadata":{"trusted":true,"_uuid":"0fb1e3e0f51f75515c9646d9f9a3c14afab6692e","_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=int(len(ratios) / 2), ncols=2, figsize=(18,16))\nfig.subplots_adjust(wspace=0.2, hspace=0.4)\nfor i in range(len(ratios)):\n    train.loc[train['ResolutionRatio'] == ratios[i], 'Wdft_IsGamer'].value_counts(True, dropna=False).plot(kind='bar', rot=0, ax=axes[i // 2,i % 2], fontsize=14).set_xlabel('Wdft_IsGamer', fontsize=18)\n    axes[i // 2,i % 2].plot(0, train.loc[(train['ResolutionRatio'] == ratios[i]) & (train['Wdft_IsGamer'] == 0.0), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].plot(1, train.loc[(train['ResolutionRatio'] == ratios[i]) & (train['Wdft_IsGamer'] == 1.0), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].plot(2, train.loc[(train['ResolutionRatio'] == ratios[i]) & (train['Wdft_IsGamer'].isnull()), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24) \n    axes[i // 2,i % 2].legend(['Detection rate (%)'])\n    axes[i // 2,i % 2].set_title('Ratio: ' + str(ratios[i]), fontsize=18)\nfig.suptitle('Resolution rate to Wdft_IsGamer interaction', fontsize=18);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8839736d686a1c251cae47dc5486e1b445235c59"},"cell_type":"markdown","source":"Most popular aspect ratios amongst gamers, according to the data provided, are 0.5625 (16:9) and 0.5649 (which is a 'wrong' [16:9 on laptops](https://en.wikipedia.org/wiki/Graphics_display_resolution#1360_%C3%97_768).)"},{"metadata":{"_uuid":"924e9adcb7c5a831956af6a4647975255ce3cb74"},"cell_type":"markdown","source":"<a id=\"10\"></a>\n## 4.2 Resolution rate / Census_IsTouchEnabled\nDoing the same plot for interaction of Resolution rate and Census_IsTouchEnabled features."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"994435240c36a23f77849d547f584dfccc1c24ec"},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=int(len(ratios) / 2), ncols=2, figsize=(18,16))\nfig.subplots_adjust(wspace=0.2, hspace=0.4)\nfor i in range(len(ratios)):\n    train.loc[train['ResolutionRatio'] == ratios[i], 'Census_IsTouchEnabled'].value_counts(True, dropna=False).plot(kind='bar', rot=0, ax=axes[i // 2,i % 2], fontsize=14).set_xlabel('Census_IsTouchEnabled', fontsize=18)\n    axes[i // 2,i % 2].plot(0, train.loc[(train['ResolutionRatio'] == ratios[i]) & (train['Census_IsTouchEnabled'] == 0), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].plot(1, train.loc[(train['ResolutionRatio'] == ratios[i]) & (train['Census_IsTouchEnabled'] == 1), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].legend(['Detection rate (%)'])\n    axes[i // 2,i % 2].set_title('Ratio: ' + str(ratios[i]), fontsize=18)\nfig.suptitle('Resolution rate to Census_IsTouchEnabled interaction', fontsize=18);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3bc9020f3a072162fd54edefdf84678bd44abd25"},"cell_type":"markdown","source":"I personally don't see any affect of those features interaction on HasDetections."},{"metadata":{"_uuid":"c4e519164a3cf49b223857db3e83e544278e11e4"},"cell_type":"markdown","source":"<a id=\"11\"></a>\n## 4.3 Display quality / Processor architecture"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"71df1b39bfc1c74ff98bf0d1d21abe477542f056"},"cell_type":"code","source":"train['SD'] = (train['Census_InternalPrimaryDisplayResolutionVertical'] < 720).astype('uint8')\ntrain['HD'] = (train['Census_InternalPrimaryDisplayResolutionVertical'].isin(range(720,1080))).astype('int8')\ntrain['FullHD'] = (train['Census_InternalPrimaryDisplayResolutionVertical'].isin(range(1080,2160))).astype('int8')\ntrain['4k'] = (train['Census_InternalPrimaryDisplayResolutionVertical'] >= 2160).astype('uint8')\n\nfig, axes = plt.subplots(nrows=2, ncols=2, figsize=(18,12))\nfig.subplots_adjust(wspace=0.2, hspace=0.4)\nquals = ['SD', 'HD', 'FullHD', '4k']\naxis_to_processor =  ['x86', 'x64', 'arm64']\nfor i in range(len(quals)):\n    train.loc[train[quals[i]] == 1, 'Processor'].value_counts(True).sort_index(ascending=False).plot(kind='bar', rot=0, fontsize=14, ax=axes[i // 2, i % 2]).set_xlabel('Processor', fontsize=18);\n    for j in range(len(axis_to_processor)):\n        try:\n            axes[i // 2,i % 2].plot(j, train.loc[(train[quals[i]] == 1) & (train['Processor'] == axis_to_processor[j]), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n        except:\n            pass\n    axes[i // 2,i % 2].legend(['Detection rate (%)'])\n    axes[i // 2,i % 2].set_title('Display quality: ' + quals[i], fontsize=18)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33ad75a3251dce3ae10496c0345dfa82cbe7cf61"},"cell_type":"markdown","source":"No surprise here - higher display quality owners prefer x64 processors."},{"metadata":{"_uuid":"327d8f26cb13df5fe5cc239c4ecd29fdc04544d7"},"cell_type":"markdown","source":"<a id=\"12\"></a>\n## 4.4 Display quality / Wdft_IsGamer"},{"metadata":{"trusted":true,"_uuid":"d16310f4e3fdce2b13783f9883737301985ab855","_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(18,12))\nfig.subplots_adjust(wspace=0.2, hspace=0.4)\nfor i in range(len(quals)):\n    train.loc[train[quals[i]] == 1, 'Wdft_IsGamer'].value_counts(True, dropna=False).sort_index().plot(kind='bar', rot=0, fontsize=14, ax=axes[i // 2, i % 2]).set_xlabel('Wdft_IsGamer', fontsize=18);\n    axes[i // 2,i % 2].plot(0, train.loc[(train[quals[i]] == 1) & (train['Wdft_IsGamer'] == 0), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].plot(1, train.loc[(train[quals[i]] == 1) & (train['Wdft_IsGamer'] == 1), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].plot(2, train.loc[(train[quals[i]] == 1) & (train['Wdft_IsGamer'].isnull()), 'HasDetections'].value_counts(True, dropna=False)[1], marker='.', color=\"r\", markersize=24)\n    axes[i // 2,i % 2].legend(['Detection rate (%)'])\n    axes[i // 2,i % 2].set_title('Display quality: ' + quals[i], fontsize=18)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"03cadad70f807fdcf545da1cb284fa08fda63d86"},"cell_type":"markdown","source":"Again no surprise that gamers prefer higher quality displays."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}