{"cells":[{"metadata":{},"cell_type":"markdown","source":"So far it looks like there is not much of a gap between the local validation and the LB socres. However, there **is** a consistent gap in the experiemtns that have been run so far, and there is plenty of evidence that the training and validation datasets are statistically different.\n\nIn this notebook we'll try to assess the degree of difference using adversarial validation approach. For our features we'll just use the pixel-level data from the 32x32 resized images. This data has thus far been surprisingly informative about the images in this dataset.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold\nfrom sklearn import model_selection, preprocessing, metrics\n\nfrom sklearn import preprocessing\nimport gc\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nimport os\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = np.load('../input/siimisic-melanoma-resized-images/x_train_32.npy')\ntest = np.load('../input/siimisic-melanoma-resized-images/x_test_32.npy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = np.hstack([np.zeros(train.shape[0],), np.ones(test.shape[0],)])\ntrain_test = np.vstack([train, test])\nprint(train_test.shape)\nprint(target.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train, test\ngc.collect()\ntrain_test = train_test.reshape((train_test.shape[0], 32*32*3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train, test, y_train, y_test = model_selection.train_test_split(train_test, target, test_size=0.33, random_state=42, shuffle=True)\ndel target, train_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = lgb.Dataset(train, label=y_train)\ntest = lgb.Dataset(test, label=y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"param = {'num_leaves': 50,\n         'min_data_in_leaf': 30, \n         'objective':'binary',\n         'max_depth': 5,\n         'learning_rate': 0.05,\n         \"min_child_samples\": 20,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 56,\n         \"metric\": 'auc',\n         \"verbosity\": -1}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_round = 500\nclf = lgb.train(param, train, num_round, valid_sets = [train, test], verbose_eval=50, early_stopping_rounds = 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The adversarial validation AUC of 0.65 is not terrible, but is definitely significant. \n\nWe'll now take a look at the feature importances, although since we are just using the pixel-level data they may not be that insightful.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [f'c_{i}' for i in range(3072)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_imp = pd.DataFrame(sorted(zip(clf.feature_importance(),features)), columns=['Value','Feature'])\n\nplt.figure(figsize=(20, 20))\nsns.barplot(x=\"Value\", y=\"Feature\", data=feature_imp.sort_values(by=\"Value\", ascending=False).head(25))\nplt.title('LightGBM Features')\nplt.tight_layout()\nplt.show()\nplt.savefig('lgbm_importances-01.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}