1. Factor analysis
2. PCA
\n
3. SVD
4. t-sne
5. Umap
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/siim-isic-melanoma-classification/train.csv\")\ntest_df = pd.read_csv(\"../input/siim-isic-melanoma-classification/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n
Resizing and converting the images to array
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess_image(image_path, desired_size=128):\n im = Image.open(image_path)\n im = im.resize((desired_size, )*2, resample=Image.LANCZOS)\n return im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_to_array(df, size=128):\n N = df.shape[0]\n x_train = np.empty((N, size, size, 3), dtype=np.uint8)\n for i, image_id in enumerate(tqdm(df['image_name'])):\n x_train[i, :, :, :] = preprocess_image(\n f'../input/siim-isic-melanoma-classification/jpeg/train/{image_id}.jpg'\n )\n return x_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### The below resized array is from @tunguz's kernel - https://www.kaggle.com/tunguz/image-resizing-128x128-train as converting it to array is taking a long time","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#x_train = convert_to_array(train_df)\nx_train = np.load('../input/x-train-128npy/x_train_128.npy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_t = x_train[:2000]\nx_t.shape\n#train = x_train.reshape((x_train.shape[0], 128*128*3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### As you can see above, it’s a 3-dimensional array. We must convert it to 1-dimension as all the upcoming techniques only take 1-dimensional input. To do this, we need to flatten the images:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image = []\nfor i in range(0,2000):\n img = x_t[i].flatten()\n image.append(img)\nimage = np.array(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feat_cols = ['pixel'+str(i) for i in range(image.shape[1])]\ndf = pd.DataFrame(image,columns=feat_cols)\ndf['label'] = train_df['benign_malignant'][:2000]\ndf['label1'] = train_df['target'][:2000]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n
1. Factor Analysis
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### In the Factor Analysis technique, variables are grouped by their correlations, i.e., all variables in a particular group will have a high correlation among themselves, but a low correlation with variables of other group(s). Here, each group is known as a factor. These factors are small in number as compared to the original dimensions of the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import FactorAnalysis\n\nFA = FactorAnalysis(n_components=2).fit_transform(df[feat_cols].values)\n\n# Here, n_components will decide the number of factors in the transformed data. After transforming the data, it’s time to visualize the results:\n\nfa_data = np.vstack((FA.T, df['label1'])).T\nfa_df = pd.DataFrame(fa_data, columns=['1st Component', '2nd Component', 'Label'])\nsns.FacetGrid(fa_df, hue=\"Label\", size=10).map(plt.scatter, \"1st Component\", \"2nd Component\").add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## For 2000 images - we can see some orange and mostly blue, highly imbalanced?","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n
2. PCA
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### PCA is a technique which helps us in extracting a new set of variables from an existing large set of variables.These newly extracted variables are called Principal Components. Some of the key points you should know about PCA before proceeding further:\n\n - A principal component is a linear combination of the original variables\n - Principal components are extracted in such a way that the first principal component explains maximum variance in the dataset\n - Second principal component tries to explain the remaining variance in the dataset and is uncorrelated to the first principal component\n - Third principal component tries to explain the variance which is not explained by the first two principal components and so on","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA\npca = PCA(n_components=2)\npca_result = pca.fit_transform(df[feat_cols].values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### In this case, n_components will decide the number of principal components in the transformed data. \n### Let’s visualize how much variance has been explained using these 2 components. We will use explained_variance_ratio_ to calculate the same.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12.5, 8))\nplt.plot(range(2), pca.explained_variance_ratio_)\nplt.plot(range(2), np.cumsum(pca.explained_variance_ratio_))\nplt.title(\"Component-wise and Cumulative Explained Variance\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### In the above graph, the blue line represents component-wise explained variance while the orange line represents the cumulative explained variance. We are able to explain around 70% variance in the dataset using just four components. Let us now try to visualize each of these decomposed components","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA \npca = PCA(n_components=2, random_state=42).fit_transform(df[feat_cols].values)\npca_data = np.vstack((pca.T, df['label1'])).T\npca_df = pd.DataFrame(pca_data, columns=['1st Component', '2nd Component', 'Label'])\nsns.FacetGrid(pca_df, hue=\"Label\", size=10).map(plt.scatter, \"1st Component\", \"2nd Component\").add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"
\n
3. SVD
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### We can also use Singular Value Decomposition (SVD) to decompose our original dataset into its constituents, resulting in dimensionality reduction. SVD decomposes the original variables into three constituent matrices. It is essentially used to remove redundant features from the dataset. It uses the concept of Eigenvalues and Eigenvectors to determine those three matrices. \n\n### Let’s implement SVD and decompose our original variables:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import TruncatedSVD \nsvd = TruncatedSVD(n_components=2, random_state=42).fit_transform(df[feat_cols].values)\nsvd_data = np.vstack((svd.T, df['label1'])).T\nsvd_df = pd.DataFrame(svd_data, columns=['1st Component', '2nd Component', 'Label'])\nsns.FacetGrid(svd_df, hue=\"Label\", size=10).map(plt.scatter, \"1st Component\", \"2nd Component\").add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The above scatter plot shows us the decomposed components.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n
4. t-sne
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### There are mainly two types of approaches we can use to map the data points:\n\n - Local approaches : They maps nearby points on the manifold to nearby points in the low dimensional representation.\n - Global approaches : They attempt to preserve geometry at all scales, i.e. mapping nearby points on manifold to nearby points in low dimensional representation as well as far away points to far away points.\n \n - t-SNE is one of the few algorithms which is capable of retaining both local and global structure of the data at the same time\n - It calculates the probability similarity of points in high dimensional space as well as in low dimensional space","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.manifold import TSNE \ntsne = TSNE(n_components=2).fit_transform(df[feat_cols].values)\nts_data = np.vstack((tsne.T, df['label1'])).T\nts_df = pd.DataFrame(ts_data, columns=['1st Component', '2nd Component', 'Label'])\nsns.FacetGrid(ts_df, hue=\"Label\", size=10).map(plt.scatter, \"1st Component\", \"2nd Component\").add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n
5. UMAP
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Uniform Manifold Approximation and Projection (UMAP) is a dimension reduction technique that can preserve as much of the local, and more of the global data structure as compared to t-SNE\n\n### Some of the key advantages of UMAP are:\n - It can handle large datasets and high dimensional data without too much difficulty\n - It combines the power of visualization with the ability to reduce the dimensions of the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import umap\numap_data = umap.UMAP(n_neighbors=5, min_dist=0.3, n_components=2).fit_transform(df[feat_cols].values)\n\n#Here,\n#n_neighbors determines the number of neighboring points used\n#min_dist controls how tightly embedding is allowed. Larger values ensure embedded points are more evenly distributed\n\numap_data = np.vstack((umap_data.T, df['label1'])).T\numap_df = pd.DataFrame(umap_data, columns=['1st Component', '2nd Component', 'Label'])\nsns.FacetGrid(umap_df, hue=\"Label\", size=10).map(plt.scatter, \"1st Component\", \"2nd Component\").add_legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Baseline model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_to_tarray(df, size=128):\n N = df.shape[0]\n x_test = np.empty((N, size, size, 3), dtype=np.uint8)\n for i, image_id in enumerate(tqdm(df['image_name'])):\n x_test[i, :, :, :] = preprocess_image(\n f'../input/siim-isic-melanoma-classification/jpeg/test/{image_id}.jpg'\n )\n return x_test\n\nx_test = convert_to_tarray(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train.reshape((x_train.shape[0], 128*128*3))\n#x_test = x_test.reshape((x_test.shape[0], 128*128*3))\ny = train_df.target.values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_oof = np.zeros((x_train.shape[0], ))\ntest_preds = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_splits = 3\n\nkf = KFold(n_splits=n_splits, random_state=137, shuffle=True)\n\nfor ij, (train_index, val_index) in enumerate(kf.split(x_train)):\n \n print(\"Fitting fold\", ij+1)\n train_features = x_train[train_index]\n train_target = y[train_index]\n \n val_features = x_train[val_index]\n val_target = y[val_index]\n \n model = RandomForestClassifier(max_depth=2, random_state=0)\n model.fit(train_features, train_target)\n \n val_pred = model.predict_proba(val_features)[:,1]\n \n train_oof[val_index] = val_pred\n \n print(\"Fold AUC:\", roc_auc_score(val_target, val_pred))\n test_preds += model.predict_proba(x_test)[:,1]/n_splits\n \n del train_features, train_target, val_features, val_target\n gc.collect()\n \nprint(roc_auc_score(y, train_oof))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')\n\nsample_submission['target'] = test_preds\n\nsample_submission.to_csv('submission_RF_01.csv', index=False)\n\nsample_submission['target'].max()\n\nsample_submission['target'].min()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}