{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook we'll try to assess the degree of difference between the train and test sets using adversarial validation approach. We have already done the same exercise with the rescaled raw image data, and that approach can be found [here](https://www.kaggle.com/tunguz/adversarial-melanoma). In this notebook we'll just use the metadata and the image size data. We have already preprocessed the datasets [in this notebook](https://www.kaggle.com/tunguz/melanoma-train-test-creator/), and we'll just use the output here.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nfrom sklearn import model_selection, preprocessing, metrics\n\nfrom sklearn import preprocessing\nimport gc\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nimport os\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/melanoma-train-test-creator/train_meta_size.csv')\ntest = pd.read_csv('../input/melanoma-train-test-creator/test_meta_size.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = test.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = np.hstack([np.zeros(train.shape[0],), np.ones(test.shape[0],)])\ntrain_test = np.vstack([train[features].values, test.values])\nprint(train_test.shape)\nprint(target.shape)\ndel train, test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train, test, y_train, y_test = model_selection.train_test_split(train_test, target, test_size=0.33, random_state=42, shuffle=True)\ndel target, train_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = lgb.Dataset(train, label=y_train)\ntest = lgb.Dataset(test, label=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"param = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'binary',\n         'max_depth': 8,\n         'learning_rate': 0.05,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 56,\n         \"metric\": 'auc',\n         \"verbosity\": -1}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_round = 2000\nclf = lgb.train(param, train, num_round, valid_sets = [train, test], verbose_eval=50, early_stopping_rounds = 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For raw image data we were able to get an AUC of 0.65, while here we get 0.70. Seems that the image size and image metafeatrue data has more discrepancy between the train and test sets than the raw rescaled images.\n\nLet's look at the top features and theri relative importances.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_imp = pd.DataFrame(sorted(zip(clf.feature_importance(),features)), columns=['Value','Feature'])\n\nplt.figure(figsize=(20, 20))\nsns.barplot(x=\"Value\", y=\"Feature\", data=feature_imp.sort_values(by=\"Value\", ascending=False).head(5))\nplt.title('LightGBM Features')\nplt.tight_layout()\nplt.show()\nplt.savefig('lgbm_importances-01.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_imp.sort_values(by=\"Value\", ascending=False).head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the biggest \"culprits\" for the train/test discrepancy are the age, followed by the image width.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We'll now look at the train and test sets with more \"rigorously\" impoted missing values:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/melanoma-train-test-creator/train_meta_size_2.csv')\ntest = pd.read_csv('../input/melanoma-train-test-creator/test_meta_size_2.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = test.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = np.hstack([np.zeros(train.shape[0],), np.ones(test.shape[0],)])\ntrain_test = np.vstack([train[features].values, test.values])\nprint(train_test.shape)\nprint(target.shape)\ndel train, test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train, test, y_train, y_test = model_selection.train_test_split(train_test, target, test_size=0.33, random_state=42, shuffle=True)\ndel target, train_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = lgb.Dataset(train, label=y_train)\ntest = lgb.Dataset(test, label=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"param = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'binary',\n         'max_depth': 8,\n         'learning_rate': 0.05,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 56,\n         \"metric\": 'auc',\n         \"verbosity\": -1}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_round = 2000\nclf = lgb.train(param, train, num_round, valid_sets = [train, test], verbose_eval=50, early_stopping_rounds = 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So there is a little bit of an \"improvement\", but overall it is still a fairly significant distinction.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_imp = pd.DataFrame(sorted(zip(clf.feature_importance(),features)), columns=['Value','Feature'])\n\nplt.figure(figsize=(20, 20))\nsns.barplot(x=\"Value\", y=\"Feature\", data=feature_imp.sort_values(by=\"Value\", ascending=False).head(5))\nplt.title('LightGBM Features')\nplt.tight_layout()\nplt.show()\nplt.savefig('lgbm_importances-02.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}