{"cells":[{"metadata":{},"cell_type":"markdown","source":"Having been inspired by @[Tunguz](https://www.kaggle.com/tunguz) great [kernel](https://www.kaggle.com/tunguz/mnist-2d-t-sne-with-rapids) I thought it would be nice to see a two dimensional projection of the data from [Riiid! Answer Correctness Prediction](https://www.kaggle.com/c/riiid-test-answer-prediction) "},{"metadata":{},"cell_type":"markdown","source":"If you don't know what TSNE is, you can learn it from StatQuest's great [video](https://www.youtube.com/watch?v=NEaUSP4YerM)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Installing RAPIDS\nimport sys\n!cp ../input/rapids/rapids.0.15.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom cuml.manifold import TSNE\nimport cupy, cudf\nimport os\nimport matplotlib.pyplot as plt\nimport gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, I am going to import the data from [Simple EDA and Baseline](https://www.kaggle.com/ilialar/simple-eda-and-baseline) kernel."},{"metadata":{"trusted":true},"cell_type":"code","source":"# As cudf is faster than pandas, I'm going to use that.\ndf = cudf.read_csv('../input/simple-eda-and-baseline-data-generation/train_preprocessed.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Next, I'm replacing NaNs and Infs with -9999."},{"metadata":{"trusted":true},"cell_type":"code","source":"df['prior_question_had_explanation'] = df['prior_question_had_explanation'].astype(int)\ndf = df.fillna(-9999)\ndf['prior_question_elapsed_time'] = df['prior_question_elapsed_time'].replace(['inf'], -9999)\ndf['prior_question_elapsed_time'] = df['prior_question_elapsed_time'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the data set is huge, I'm only going to use a fraction of it."},{"metadata":{"trusted":true},"cell_type":"code","source":"sampled_df = df.sample(10000)\ndel df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = sampled_df['answered_correctly']\ndel sampled_df['answered_correctly']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = target.values\nsampled_df = sampled_df.values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# I'm converting the data to numpy, as then it's easier to plot/save it etc.\ntarget = cupy.asnumpy(target)\nsampled_df = cupy.asnumpy(sampled_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntsne = TSNE(n_components=2)\ntsne_data = tsne.fit_transform(sampled_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(tsne_data[:,0], tsne_data[:,1], c = target, s = 0.6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, there's a pattern in some parts of the dataset. However, in general it's not soo easy to distinguish points fallig into different categories by eye."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}