{"cells":[{"metadata":{},"cell_type":"markdown","source":"One of the main issues that make Kaggle (and for tahat matter any other) predictive modeling tricky are the discrepancies between the training and the test datasets. In order to get an idea of the magnitude of these differences, one of the more valuable tools to use is adversarial validation. With aversariel validation we try to build an auxiliary model that predicts whether given data points belong to the train and the test set. If we can make predictions with such a model with a high degree of confidence, then that usually means that the train and test sets are significantly different, and we need to be careful to make a model that will take that into the account.\n\nIn this notebook we'll build an adversarial model on FFT features that were first used in this competition in [this Giba's notebook](https://www.kaggle.com/titericz/0-309-baseline-logisticregression-using-fft). I've created a stand-alone notebook that extracts those features, and it can be found [here](https://www.kaggle.com/tunguz/giba-s-fft-features-only).\n\nWe will make this adversarial validation notebook with the Rapids library. [Rapids](https://rapids.ai) is an open-source GPU accelerated Data Sceince and Machine Learning library, developed and mainatained by [Nvidia](https://www.nvidia.com). It is designed to be compatible with many existing CPU tools, such as Pandas, scikit-learn, numpy, etc. It enables **massive** acceleration of many data-science and machine learning tasks, oftentimes by a factor fo 100X, or even more. \n\nRapids is still undergoing developemnt, and only recently has it become possible to use RAPIDS natively in the Kaggle Docker environment. If you are interested in installing and riunning Rapids locally on your own machine, then you should [refer to the followong instructions](https://rapids.ai/start.html).\n\nFor the modeling part we'll use the latest version of XGBoost, which allows for GPU accelerated calculation of Shapely Values. We'll use these \"SHAP\" values to calculate correct feature importances."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --use-feature=2020-resolver https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.3.0_SNAPSHOT%2Bdda9e1e4879118738d9f9d5094246692c0f6123c-py3-none-manylinux2010_x86_64.whl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport cupy as cp # linear algebra\nimport cudf # data processing, CSV file I/O (e.g. cudf.read_csv)\nfrom sklearn.model_selection import train_test_split\nfrom cuml.metrics import roc_auc_score\nimport shap\nimport gc\nfrom random import shuffle\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input/giba-s-fft-features-only/'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost\nxgboost.__version__","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = cp.load('../input/giba-s-fft-features-only/TRAIN.npy')\ntest = cp.load(\"../input/giba-s-fft-features-only/TEST.npy\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = cp.hstack([cp.ones(train.shape[0]), cp.zeros(test.shape[0])])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_test = cp.vstack([train, test])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"index = list(range(train_test.shape[0]))\nshuffle(index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_test = train_test[index, :]\ntarget = target[index]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train, test, y_train, y_test = train_test_split(train_test, target, test_size=0.33, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_test\ngc.collect()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = xgboost.DMatrix(train, label=y_train)\ntest = xgboost.DMatrix(test, label=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nparam = {\n    'eta': 0.05,\n    'max_depth': 10,\n    'subsample': 0.8,\n    'colsample_bytree': 0.7,\n    'objective': 'reg:logistic',\n    'eval_metric': 'auc',\n    'tree_method': 'gpu_hist', \n    'predictor': 'gpu_predictor'\n}\nclf = xgboost.train(param, train, 600)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = clf.predict(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"roc_auc_score(y_test, preds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"AUC of 0.86 is pretty high. Let's try to see if we can find which FFT components are the most responsible for the discrepancy. In order to do this, we'll resort to calculating SHAP values, which can be done directly on GPUs with the version 1.3 of XGBoost."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nshap_preds = clf.predict(test, pred_contribs=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shap_preds.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shap_preds[:, :1000].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shap.initjs()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shap.summary_plot(shap_preds[:,:1000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shap.summary_plot(shap_preds[:,:1000], plot_type=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}