{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pickle\n%matplotlib inline\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"When doing EDA, I would usually like to performe the following steps:\n\n* Carefully read the description from person/entity that provided the data to understand the features and the task at hand\n* Visualize the data, quite alot of it! This is to get a sense of distribution of each feature and the lables/target values. We can also detect and fill out or disregard missing data in this step\n* Look at correlations between features and that between features and labels/target values. This gives me a sense of which features are more important, which ones can be (potentially) ignored all together.\nYou might think we should never delete features, the more data the merrier, but that is not always the case, especially when using models with low to moderatelearning capacity. \n* Perform outlier detection and removal, testing the possibility of dimensionality reduction\n* Convert  categorical features to numerical features (in most cases this is necessary), standardization of data and saving to be used fo training"},{"metadata":{"_uuid":"a1afaa2796c62bd48f894d1b81162cfffc85c17f"},"cell_type":"markdown","source":"Of course the first step is to always load the data and take a first look at the number of features that are provided. Because the data is very large, in the interest of saving time, we sample the training data and the testing data and only use 10% of these for EDA. When sampling data for EDA we need to ensure the sampled data is representative of the entire dataset."},{"metadata":{"trusted":true,"_uuid":"b282c24a60226093cfb7194360572bc1ae058bdb","scrolled":true},"cell_type":"code","source":"# for the EDA I only use 20% of the training data, randomly sampled from original dataset, I have saved this in a different .csv file to make\n# re-running of this notebook faster, for the first run un-comment the following line\n#train_data = pd.read_csv('../input/train.csv').sample(frac=0.2, axis=0)\ntrain_data = pd.read_csv('../input/springleaf-train-small/train_small.csv')\ntrain_data.drop(['Unnamed: 0'], axis=1, inplace=True)  # this feature was created during subsampling\noriginal_feat_dim = train_data.shape[1]\ntrain_data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e8795cb9ae805d45a7157e2ec845ddb2b6f4df57"},"cell_type":"code","source":"train_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f53aff391a1b1e8e7801376413fdab3f54aed27f"},"cell_type":"code","source":"# looking at categorical columns\ncategorical = train_data.select_dtypes(include=[np.object]).columns\ncategorical, len(categorical)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"960427f6176baab2292e832526c407a3e098f64b"},"cell_type":"code","source":"# looking at numerical columns\nnumerical = train_data.select_dtypes(include=[np.number]).columns\nnumerical, len(numerical)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"16ae4e119d8c2337149ae206bc8ee613cd6147b4"},"cell_type":"markdown","source":"Looks like we have 1883 numerical features (1882 really counting out ID). We are aiming to predict the 'target'. Due to the large number of features involved, this data is a prime candidate for dimensionality reduction. Lets put a pin on this and do some visualization! First stop: histogram of targets, this helps us make sure we don't have a class imbalance problem."},{"metadata":{"trusted":true,"_uuid":"9e4b49990143c6262374efe2a658cd8ecd427f1d"},"cell_type":"code","source":"plt.hist(train_data[\"target\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"13984462bb12d69cfb1b907f4d288a65f2db8de9"},"cell_type":"code","source":"print(\"Ratio of positive samples (target == 1) to negative samples (target == 0) is {}\"\\\n          .format(len(train_data[train_data[\"target\"] == 1]) / len(train_data[train_data[\"target\"] == 0])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81ca24e15da10e5b957bb2e5d9ddb7c226971e11"},"cell_type":"markdown","source":"**Note:** This means any classifier that always predicts 0 will get an accuracy of about 70%. This is a good performance benchmark to begin with. "},{"metadata":{"_uuid":"5359760432125701afda9041e2e4fd24a1853fd0"},"cell_type":"markdown","source":"# Dealling with missing data and data cleaning\n\nBefore looking at the features and trying to make sense out of them, we will first look at missing data, constant features and duplicate features.\n\nFirst, lets look at missing data and see how big of a problem that is."},{"metadata":{"trusted":true,"_uuid":"3f039ae854c7078990770d8dcceeb7d7940bad1a"},"cell_type":"code","source":"null_count = train_data.isnull().sum(axis=0).sort_values(ascending=False)\nnull_count.head(30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a38fb30859d87f8f20dbe855b14df4b578c4d16b"},"cell_type":"code","source":"# figuring out how many features have more than 10% of data missing them\nnp.sum(null_count > 0.1 * train_data.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12c8a4880d229f36930f0c6f4d5b1f6a531391e0"},"cell_type":"code","source":"# I think its safe to remove these features, I will create a to_remove list to store all features that need to be dropped\nmissing = [feature for feature in train_data.columns if null_count[feature] > 0.1 * train_data.shape[0]]\ntrain_data.drop(missing, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce24265c6e0c47b3d206e39874f4113583f037da"},"cell_type":"markdown","source":"**Dealing with the rest of missing data**\n\nWe removed the features where more than 10% of data is missing. Lets look at whats left"},{"metadata":{"trusted":true,"_uuid":"3f11dd4d4422e0840da56dc52505ec92472b36bc"},"cell_type":"code","source":"# taking a second look\ntotal = train_data.isnull().sum().sort_values(ascending=False)\npercent = total / train_data.shape[0]\nmissing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7016ed8b0b197625dc66a5f0211e94723556f4b"},"cell_type":"markdown","source":"Of the ones left, we adapt the following simple rule of thumb (we can always reconsider this)\n\n- If lest than 1% of data is missing of a given feature, just remove the training sample that is missing this feature (all of the features above except VAR_0212)\n\n- Any feature missing more thant 1% of data should be \"completed\". If the feature is numerical, we replace the NAN with mean of the feature across the training data set. Otherwise, if feature is categorical, we replace it with the mode. (more complicated extrapolation methods are abundant but we will keep things simple for now) (feature VAR_0212 above)"},{"metadata":{"trusted":true,"_uuid":"91468bb7b28e2aaf3f07f3ddb2e2acb26cec4515"},"cell_type":"code","source":"remove_missing_examples = [feat for feat in percent.index if percent[feat] < 0.01]\nfill_missing_examples = [feat for feat in percent.index if percent[feat] > 0.01]\n\n# filling in the missing data for features containing many of them\nfor col in fill_missing_examples:\n    if col in categorical:\n        # fill missing data with mode\n        train_data[col].fillna(train_data[col].mode(), inplace=True)\n    else:\n        # fill missing data with mean\n        train_data[col].fillna(train_data[col].mean(), inplace=True)\n\n# removing rows with missing data in them (only a few examples will be deleted at this point)\ntrain_data.dropna(axis=0, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"247bfc87e008125396794841f398bef7cec83b31"},"cell_type":"code","source":"# final check for missing data\ntrain_data.isnull().sum().max()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fdffc5f9569323f585244f35d1957c6b9e4429f"},"cell_type":"markdown","source":"**Deleting constant features**\n\nAnother set of features that need to be removed are the ones with constant values. We need to get rid of those too!"},{"metadata":{"trusted":true,"_uuid":"e5c5252a8f246436fdc048ba391402988d84a1ea"},"cell_type":"code","source":"values_count = train_data.nunique(dropna=False).sort_values()\nnp.sum(values_count == 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8122b39a206005d22854bd7600eabdddef875e09"},"cell_type":"code","source":"# noting features with constant values for removal\nconstants = [feature for feature in train_data.columns if values_count[feature] == 1]\ntrain_data.drop(constants, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d923c8dbf2e51c7d48f9ad2437b9ba496bf1281"},"cell_type":"markdown","source":"****Deleting duplicate features****\n\nFinally, we need to get rid of features that are duplicates of another one, this may happen very rarely but when there are a large number of features I always look for this situation. To this end, instead of comparing all features with all other features, I fist compose a list of possible duplicates by comparing mean and std values. If they are the same then I will perform element-wise comparison. \n\nBut first, lets encode categorical features into numerical features so that we can find meaningfull mean and std values. This encoding also helps us ignore the missing data when comparing duplicates, for instances, a missing piece of information might be represented differently in different columns. "},{"metadata":{"trusted":true,"_uuid":"a36f1b494fe4b8327d34530f8f5c21e6e68ad7b1"},"cell_type":"code","source":"enc_data =  pd.DataFrame(index=train_data.index)\nfor col in train_data.columns:\n    enc_data[col] = train_data[col].factorize()[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"017911d3b24d9effee795f79d373d33e5fe4c6fc"},"cell_type":"code","source":"# noting and dropping duplicated features \nduplicates = []\ndata_mean = np.mean(enc_data)\ndata_std = np.std(enc_data)\nfor i, ref in enumerate(train_data.columns[:-1]):\n    for other in train_data.columns[i + 1:-1]:\n        if other not in duplicates and data_mean[ref] == data_mean[other] and \\\n           data_std[ref] == data_std[other] and np.all(enc_data[ref] == enc_data[other]):\n            duplicates.append(other)    \ntrain_data.drop(duplicates, axis=1, inplace=True)\nlen(duplicates)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b6173594e5227d4f1eea8c83c43bde7544f15c1"},"cell_type":"code","source":"print(\"Dimension of feature space before data cleaning : %d\" % original_feat_dim)\nprint(\"Dimension of feature space after data cleaning : %d\" % train_data.shape[1])\nprint(\"Number of features removed : %d\" % (original_feat_dim - train_data.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4c280bf2af750d57c88a4b610cba338b86d564c"},"cell_type":"markdown","source":"# Going through data\n\nAlright time to look at data in hops of getting a better understanding. What we learn here will be subsequently used in another data preparation script. Here is the plan:\n\n1 - look at categorical data to see if we can create new features, especially feature crosses from them\n\n2 - Find correlation between numerical data and targets, identify top correlated features and look at their distributions, this will tell us if outlier detection or binning of features in needed\n\n3 - Do the same for categorical fetures, to find correlation we have to first encode these "},{"metadata":{"trusted":true,"_uuid":"5268e224775c69de653ce72443c523387b641925"},"cell_type":"code","source":"categorical = train_data.select_dtypes(include=[np.object]).columns\nnumerical = train_data.select_dtypes(include=[np.number]).columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f4bac7f5df8ccd37ab6a9c726281bd78c51febb"},"cell_type":"code","source":"train_data[categorical].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65b3dac167fde195a4a0570f648e7763cb282f26"},"cell_type":"markdown","source":"Observations: \n    - VAR_0226, VAR_0230 and VAR_0236 are also nearly dominated by their most frequent values so these can be removed too (making note but not removing these yet), this will reduce the dimension of the feature space,"},{"metadata":{"trusted":true,"_uuid":"b815736543fc54a58c27566718c57a350e0e1bc1"},"cell_type":"code","source":"potential_to_remove = ['VAR_0226', 'VAR_0230', 'VAR_0236']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b62b329fd17eca00445c4e9e7f9569c45bebaf2","trusted":true,"scrolled":false},"cell_type":"code","source":"train_data[categorical].head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6dfa2153e7fb75982f9592158f866119a6b8f7e6"},"cell_type":"markdown","source":"Observations:\n    - Features : VAR_0075, VAR_0204, VAR_0217 correspond to dates, keep this in mind when converting these features into numreical ones\n    - It looks like VAR_0217 is larger than VAR_0075, we may be able use the difference between the two as a good feature, lets check this.\n    - VAR_0204 dates mostly fall into late January and early February of 2014, given the date of the competition, this looks like a \"last login by user\" kinda feature, might be irrelevant to target, we have to check how it changes vs. target.\n    - Features VAR_200, VAR_0237, VAR_274 corrspond to location, first one is name of a city, the other two correspond to state\n    - Features VAR_0404, VAR_466, VAR_0467 and VAR_0493 are largely dominated by -1, have to check their histograms."},{"metadata":{"trusted":true,"_uuid":"5237fac5ad6eeeaa6ac02ba93ced1f7c8d2fb7a7"},"cell_type":"code","source":"dates = ['VAR_0075', 'VAR_0204', 'VAR_0217']\nlocations = ['VAR_200', 'VAR_0237', 'VAR_274']\n\ndate1 = pd.to_datetime(train_data[\"VAR_0075\"],format = '%d%b%y:%H:%M:%S')\ndate2 = pd.to_datetime(train_data[\"VAR_0217\"],format = '%d%b%y:%H:%M:%S')\nnp.all(date2 > date1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"445f09ea20efbab47a8209ed9d5cf95b6353e01a","scrolled":false},"cell_type":"code","source":"train_data[['VAR_0404', 'VAR_0466', 'VAR_0467', 'VAR_0493']].nunique(dropna=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b98a1f06da8b34cf544280f052748205a65c5ba2"},"cell_type":"code","source":"train_data['VAR_0466'].value_counts() / train_data.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4fca4bac7b15164d475a1a7014638e3c123d33ee"},"cell_type":"code","source":"train_data['VAR_0467'].value_counts() / train_data.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9654e34590b7e1ec5278bf7ac8701ec96255949e"},"cell_type":"markdown","source":"- I think -1 here indicates missing data in original data acquisition process, these features do not seem to carry much information.\n- Also, 'Discharge NA' may be considered as an outlier value for feature 'VAR_0467'.\nLets keep a note of this and look at the correlation between categorical data and the target. "},{"metadata":{"trusted":true,"_uuid":"4234fa6255ab00fb96c9eef3d2abf295d1dc9ad4"},"cell_type":"code","source":"# converting categorical features to encoded variables first, \n# for features represnting date, we separate year, month day, hour and minute, this is a very crude represntation, we may \n# need more interesting time features like time of the day, weekday and so on\nfrom datetime import datetime\nyear_func = lambda x: datetime.strptime(x, \"%d%b%y:%H:%M:%S\" ).year\nmonth_func = lambda x: datetime.strptime(x, \"%d%b%y:%H:%M:%S\" ).month\nday_func = lambda x: datetime.strptime(x, \"%d%b%y:%H:%M:%S\" ).day\nhour_func = lambda x: datetime.strptime(x, \"%d%b%y:%H:%M:%S\" ).hour\nminute_func = lambda x: datetime.strptime(x, \"%d%b%y:%H:%M:%S\" ).minute\n\nenc_data =  pd.DataFrame(index=train_data.index)\nfor col in categorical:\n    enc_data[col] = train_data[col].factorize()[0]\n    if col in dates:\n        enc_data[col + '_year'] = train_data[col].map(year_func)\n        enc_data[col + '_month'] = train_data[col].map(month_func)\n        enc_data[col + '_day'] = train_data[col].map(day_func)\n        enc_data[col + '_hour'] = train_data[col].map(hour_func)\n        enc_data[col + '_minute'] = train_data[col].map(minute_func)\nexpanded_categoricals = list(enc_data.columns) # saving which variables are categorical for possible one hot encoding \nenc_data[\"target\"] = train_data[\"target\"]\n# finding correlation and looking at it\ncorrmat = enc_data.corr()\ncorrmat[\"target\"].sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eef68d69d40b34244ee9b78dbfe63d57a6c4b031"},"cell_type":"markdown","source":"It looks like different components of the date have different significance in predicting the target.\nAlso, some of the features that we created are constant and need to be removed. Lets remove these, find correlation matrix again and plot it as a heatmap"},{"metadata":{"trusted":true,"_uuid":"b5d65781abdda4bf725881fdad56208d0df0b812"},"cell_type":"code","source":"enc_data.drop(['VAR_0075_hour', 'VAR_0075_minute', 'VAR_0204_year', 'VAR_0217_hour', 'VAR_0217_minute'], axis=1, inplace=True)\ncorrmat = enc_data.corr()\nf, ax = plt.subplots(figsize=(12, 9))\nsns.heatmap(corrmat, vmax=1, square=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4fd4907cdbeee54c7adc9de342480f4d3608f40"},"cell_type":"markdown","source":"Observations:\n    - There is no significant correlation between any categorical feature and the target, however, this deos not mean we can disregard these features.\n      Note: correlation only shows linear dependance between two random quantities\n    - VAR_0466 and VAR_0467 seem to be highly correlated, in fact, if we go back and look at the first 20 rows we notice that VAR_0466 has 'l' in each row where VAR_0467 has 'Discharged', lets test this out, if this is the case. Dimensionality reduction will take care of this!"},{"metadata":{"_uuid":"1f3373e60b251c09bcc33ea834e4b9db1582ca20"},"cell_type":"markdown","source":"Now lets look at the correlation between numerical features and the target, because we have a large number of numerical features, this might take a while!"},{"metadata":{"trusted":true,"_uuid":"c1825e186ef924147ae074f0811ac83efc131e02","scrolled":false},"cell_type":"code","source":"numerical = train_data.select_dtypes(include=[np.number]).columns\ncorrmat2 = pd.concat([train_data[numerical], train_data['target']], axis = 0).corr()\ncorrmat2[\"target\"].sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"195303e9348f5f813c5c09468e6e86d57f32df26"},"cell_type":"markdown","source":"Alright it looks like, unlike with categorical variables, the numerical variables show some decent correlation levels with the target. Lets \n    - look at distributions of some these feautres, looking for outliers\n    - check correlations between features themselves and see how many features have a correlation of larger than 0,8 (in absolute value terms)"},{"metadata":{"trusted":true,"_uuid":"2acb87a34b5383c22d5c65acf219e7e97956323b"},"cell_type":"code","source":"# getting more important features\ntreshold = 0.17\nimportants = [feat for feat in corrmat2.columns if abs(corrmat2['target'].loc[feat]) > treshold]\n\n# randomly selecting 6 features from importants\nsample_feats = np.random.choice(importants[:-1], 6)\n\n# looking at histograms\nf, ax = plt.subplots(figsize=(20, 10))\nfor i in range(len(sample_feats)):\n    plt.subplot(2, 3, i + 1)\n    sns.distplot(train_data[sample_feats[i]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf8301fe47a025c09f3c904957f54dd6fa46fc28"},"cell_type":"markdown","source":"Apparently some outlier detection is needed! These histograms have a very long tail. We may also have to quantize these features into bins. For now lets look at how target changes vs these sample features."},{"metadata":{"trusted":true,"_uuid":"c3a12b006ba4f28ed487e0fe8ccf02d399a0258d"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(20, 10))\nf.suptitle('Features With High Correlation (6 randomly chosen ones)', size=14)\nfor i in range(len(sample_feats)):\n    plt.subplot(2, 3, i + 1)\n    sns.boxplot(x='target', y=sample_feats[i], data=train_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5970303e144b178688d0e9f772db7dbd3575f8d"},"cell_type":"markdown","source":"Finally, lets count the highly correlated numerical features."},{"metadata":{"trusted":true,"_uuid":"dd7db041acf81d2222416a71b60434ba4fd86d96"},"cell_type":"code","source":"high_corrs = []\nfor i, c1 in enumerate(corrmat2.columns):\n    for c2 in corrmat2.columns[i + 1:]:\n        if abs(corrmat2[c1].loc[c2]) > 0.8:\n            high_corrs.append((c1, c2))\nlen(high_corrs)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7076a53494f79fc261a691aad15148ec9cefa7f2","trusted":true},"cell_type":"markdown","source":"We can see that the features are highly correlated, some dimensionality reduction is surely needed! \nBefore doing so, we save all the info that we can use for data preparation in a separate script.\n"},{"metadata":{"trusted":true,"_uuid":"ff022f08c3cb8ad148a037ac52eb2d0a551b08f0"},"cell_type":"code","source":"np.savez('./prep_info.npz', missing, fill_missing_examples, remove_missing_examples, constants, duplicates,\n         dates, locations, potential_to_remove, expanded_categoricals)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31cda62e91aaf3ef2ff9957777f913b8d73679f8"},"cell_type":"markdown","source":"# Dimensionality reduction and visualization\nIn this section I will perform dimensionality reduction to embed the data in a 2D space, this gives us an idea of how separable the data is, and, if, embedding into a lower dimenison helps.\nI try the following three methods:\n    - Principal component analysis (and kernel PCA)\n    - tSNE\n    - Diffusion mapping (with two different choices for affinity matrix)"},{"metadata":{"trusted":true,"_uuid":"3ea70e021f463c3b1161bb0549259f7a0c25693c"},"cell_type":"code","source":"from sklearn.manifold import TSNE\nfrom sklearn.manifold import SpectralEmbedding\nfrom sklearn.decomposition import PCA, KernelPCA, FastICA","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cc8e56ce0dff1d69220c4cf33914bc9fc1bdf7c"},"cell_type":"code","source":"# first step, separating features and targets, remember that we have already encoded categorical data and stored the result in enc_data \nnum_samples = 1000 # number of samples to use from each class of target for dimensionality reduction, reduce this if you are impatient!\nnumerical_feats = train_data[numerical].drop(['target'], axis=1, inplace=False)\nnew_data = pd.concat([numerical_feats, enc_data], axis=1)\npositive_samples = new_data[new_data['target'] == 1].sample(num_samples)\nnegative_samples = new_data[new_data['target'] == 0].sample(num_samples)\nto_transform = pd.concat([positive_samples, negative_samples]).sample(frac=1).reset_index(drop=True)\nto_transform.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d493e5e4ba94e37bffa082716dfe8bd976a8d3f"},"cell_type":"code","source":"features = to_transform.drop(['target'], axis=1, inplace=False)\nlabels = to_transform['target']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d0e340e7610dc4546688511fefed6f13ac36f5c"},"cell_type":"markdown","source":"**First method: the humble PCA**"},{"metadata":{"trusted":true,"_uuid":"fcabea13feac3bb39db771bd2d78b2f4b5b21ad6"},"cell_type":"code","source":"pca_embedding =  PCA(n_components=2) \npca_emb_data = pca_embedding.fit_transform(features.values)\nplt.figure(figsize=(15,15))\nplt.scatter(pca_emb_data[labels == 1, 0], pca_emb_data[labels == 1, 1], color='red', label='positive samples')\nplt.scatter(pca_emb_data[labels == 0, 0], pca_emb_data[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44817146cee02ed14e1556ac9152c78755236188"},"cell_type":"markdown","source":"**First method (b) : kernel PCA with rbf kernel**"},{"metadata":{"trusted":true,"_uuid":"e0eba566f0719f3739dda1ddee42de32f3c0b6b2"},"cell_type":"code","source":"kpca_embedding =  KernelPCA(n_components=2, kernel='rbf')\nkpca_emb_data = kpca_embedding.fit_transform(features.values)\nplt.figure(figsize=(15,15))\nplt.title('Reduced data with kernel PCA (RBF kernel)')\nplt.scatter(kpca_emb_data[labels == 1, 0], kpca_emb_data[labels == 1, 1], color='red', label='positive samples')\nplt.scatter(kpca_emb_data[labels == 0, 0], kpca_emb_data[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f50ab91d6729331a59e78af2e008ba926ee554c8"},"cell_type":"markdown","source":"**Second method: tSNE**"},{"metadata":{"trusted":true,"_uuid":"b0026e7c7f3c8f1748a5ec1bac332e0ae554042f"},"cell_type":"code","source":"tsne_embedding =  TSNE(n_components=2) \ntsne_emb_data = tsne_embedding.fit_transform(features.values)\nplt.figure(figsize=(15,15))\nplt.title('Reduced data with tSNE')\nplt.scatter(tsne_emb_data[labels == 1, 0], tsne_emb_data[labels == 1, 1], color='red', label='positive samples')\nplt.scatter(tsne_emb_data[labels == 0, 0], tsne_emb_data[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d41abdded1a5dd6191081d65f526533243f83127"},"cell_type":"markdown","source":"**Third method (a): diffusion map with RBF kernel**"},{"metadata":{"trusted":true,"_uuid":"82d4f75ef7a6133c08cf9cd26c469163762405c5"},"cell_type":"code","source":"spec_embedding = SpectralEmbedding(n_components=2, affinity='rbf')\ntransformed_data2 = spec_embedding.fit_transform(features.values)\nfig = plt.figure(figsize=(30,10))\nplt.subplot(1, 3, 1)\nplt.scatter(transformed_data2[labels == 1, 0], transformed_data2[labels == 1, 1], color='red', label='positive samples')\nplt.legend()\nplt.subplot(1, 3, 2)\nplt.scatter(transformed_data2[labels == 0, 0], transformed_data2[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()\nplt.subplot(1, 3, 3)\nplt.scatter(transformed_data2[labels == 1, 0], transformed_data2[labels == 1, 1], color='red', label='positive samples')\nplt.scatter(transformed_data2[labels == 0, 0], transformed_data2[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"905deb1e2949b2377cf280216087bd2756919f93"},"cell_type":"markdown","source":"**Third method (b): diffusion map with nearest neighbours used for affinity**"},{"metadata":{"trusted":true,"_uuid":"2a9c9b63c91dedf517e4d19f3b31404b1eba74dc"},"cell_type":"code","source":"spec_embedding2 = SpectralEmbedding(n_components=2, affinity='nearest_neighbors', n_neighbors=30)\ntransformed_data2 = spec_embedding2.fit_transform(features.values)\nfig = plt.figure(figsize=(30,10))\nplt.subplot(1, 3, 1)\nplt.scatter(transformed_data2[labels == 1, 0], transformed_data2[labels == 1, 1], color='red', label='positive samples')\nplt.legend()\nplt.subplot(1, 3, 2)\nplt.scatter(transformed_data2[labels == 0, 0], transformed_data2[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()\nplt.subplot(1, 3, 3)\nplt.scatter(transformed_data2[labels == 1, 0], transformed_data2[labels == 1, 1], color='red', label='positive samples')\nplt.scatter(transformed_data2[labels == 0, 0], transformed_data2[labels == 0, 1], color='blue', label='negative samples')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6bae2795264fe2786b130c6782e1371c6b087f14"},"cell_type":"markdown","source":"**Conclusions and next steps**\n\n* The dimension of the feature space is very large but features are highly correlated so dimensionality reduction might help\n* There are alot of duplicate feautres, constant features and features with alot of missing entires, these need to be removed\n* There are a few fetures related to the date, we can use the differences between these dates as a new feature! \n* The categoircal features do not show promising correlation with the target value, but the numerical features do\n* Outlier data detection and removal may need to be carried out (not in the first version of the model we build though)\n* Data visualisation in 2D space shows that this is a chellnging data set that may not be easily separable"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}