{"cells":[{"metadata":{},"cell_type":"markdown","source":"[Rapids](https://rapids.ai) is an open-source GPU accelerated Data Science and Machine Learning library, developed and mainatained by [Nvidia](https://www.nvidia.com). It is designed to be compatible with many existing CPU tools, such as Pandas, scikit-learn, numpy, etc. It enables **massive** acceleration of many data-science and machine learning tasks, oftentimes by a factor fo 100X, or more. \n\nRapids is still undergoing developemnt, and as of right now it's not availabel in the Kaggle Docker environment. If you are interested in installing and running Rapids locally on your own machine, then you should [refer to the followong instructions](https://rapids.ai/start.html).\n\nThe purpose of this kernel is to take a look at dimensionality reduction that one gets with t-SNE and UMAP algorithms. We will apply these algorithms to the pixel-level data, in hopes of discerning some patterns.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n# INSTALL RAPIDS OFFLINE (FROM KAGGLE DATASET). TAKES 1 MINUTE :-)\nimport sys\n!cp ../input/rapids/rapids.0.13.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + [\"/opt/conda/envs/rapids/lib/python3.6\"] + [\"/opt/conda/envs/rapids/lib/python3.6/site-packages\"] + sys.path\n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cudf, cuml\nimport cupy as cp\nimport numpy as np\nimport pandas as pd\nimport os\nfrom cuml.manifold import TSNE, UMAP\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import ylim, xlim\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/siim-isic-melanoma-classification/train.csv\")\ntrain = np.load('../input/siimisic-melanoma-resized-images/x_train_32.npy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.reshape((train.shape[0], 32*32*3))\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntsne = TSNE(n_components=2)\ntrain_2D = tsne.fit_transform(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(train_2D[:,0], train_2D[:,1], c=train_df['target'].values, s = 0.8)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that the image pixel data definitley has some clustering structure. At first glance it doesn't seem that it's easy to clearly separate the target cases, partly becasue they comprise less than 2% of all the points. However, it seems like the most of them are concentrated in the lower areas, and soem clusters have more of them than the rest.\n\nLet's now take a look at what UMAP can discern.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\numap = UMAP(n_components=2)\ntrain_2D = umap.fit_transform(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(train_2D[:,0], train_2D[:,1], c=train_df['target'].values, s = 0.8)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Much more interesting and deliniated clusters. Again, no simple clustering of the positive cases, but they do show spatial concentration.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}