{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Missed opportunity - 4th place solution\n\n**Scores for version-11:-**\n\n**public LB=0.9618**\n\n**private LB=0.9482** ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# This is in reference to discussion here:\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169409\n\nYesterday, I saw a notebook claiming 0.9565 score in the notebook title but it was not submitted. Thus, it was not showing in high scoring notebooks. I opened the notebook and saw he has ensembled gdbt predictions with EFNs. The gdbt alone was giving around 0.7 score. So, I submitted the predictions to see if it actually works. The kernel was made private just after 2-3 hours. So, some teams (that includes me) took advantage of it to climb public lb. Although, the notebook may be overfitting like many high scoring kernels, I feel the guilt of private sharing. So, I am making this kernel public to make things rights !! The dataset was kept private in the actual notebook.\n\n# Beware\nThis notebook along with many high scoring notebook may be just overfitting to public leaderboard. So, Making submissions with this may not work well with private data. There has been pretty drastic shakeups in many competitions recently. And at least in this competition, there is a high scope of overfitting to public LB. So, here is a small advice, whatever final submission you choose, pick a good CV technique and trust your CV score. \n\nThe main idea is to train an XG Boost model on tabular data and perform voting ensemble with efn predictions. Although, XG Boost alone scores 0.6985, when ensembled with efn predictions (0.9545), it gives a boost of 0.002 score to 0.9565. But as I said, validate this with your CV.\n\n# Scores:\n1. Without gradient boosting model: 0.9545\n2. Only gradient boosting model: 0.6985\n3. Ensemble: 0.9565\n\nI don't remember the name of kernel author so I can't give references here.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install xgboost\n\nimport numpy as np\nimport pandas as pd\n\nfrom sklearn.datasets import load_iris\nimport xgboost as xgb\nfrom sklearn.metrics import accuracy_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train= pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest= pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\nsub= pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['sex'] = train['sex'].fillna('na')\ntrain['age_approx'] = train['age_approx'].fillna(0)\ntrain['anatom_site_general_challenge'] = train['anatom_site_general_challenge'].fillna('na')\n\ntest['sex'] = test['sex'].fillna('na')\ntest['age_approx'] = test['age_approx'].fillna(0)\ntest['anatom_site_general_challenge'] = test['anatom_site_general_challenge'].fillna('na')\ntrain['sex'] = train['sex'].astype(\"category\").cat.codes +1\ntrain['anatom_site_general_challenge'] = train['anatom_site_general_challenge'].astype(\"category\").cat.codes +1\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test['sex'] = test['sex'].astype(\"category\").cat.codes +1\ntest['anatom_site_general_challenge'] = test['anatom_site_general_challenge'].astype(\"category\").cat.codes +1\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = train[['sex', 'age_approx','anatom_site_general_challenge']]\ny_train = train['target']\n\n\nx_test = test[['sex', 'age_approx','anatom_site_general_challenge']]\n# y_train = test['target']\n\n\ntrain_DMatrix = xgb.DMatrix(x_train, label= y_train)\ntest_DMatrix = xgb.DMatrix(x_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = xgb.XGBClassifier(n_estimators=2600, \n                        max_depth=15, \n                        objective='multi:softprob',\n                        seed=0,  \n                        nthread=-1, \n                        learning_rate=0.15, \n                        num_class = 2, \n                        scale_pos_weight = (32542/584))\n\nclf.fit(x_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf.predict_proba(x_test)[:,1]\n# clf.predict(x_test)\nsub.target = clf.predict_proba(x_test)[:,1]\nsub_tabular = sub.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_tabular.to_csv('submission_tabular.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ensemble \nsub_new = pd.read_csv(\"../input/siimisic-melanoma-subs/submission_b_Fork-of-ensemble_melanoma-ac9964_AjayKumar_v12_LB0p9596.csv\")\n\n#sub_mean = pd.read_csv(\"../input/siimisic-melanoma-subs/Submissions_MeanOf_26Models_LB0pt9569.csv\") # LB=0.9569\nsub_mean = pd.read_csv(\"../input/siimisic-melanoma-subs/Submissions_MeanOf_27Models_LB0pt9571.csv\") # LB=0.9571\n\nsub_tuned = pd.read_csv(\"../input/simple-public-blender-0-930/submission.csv\") # ver-86/88 LB=0.9528\n\n\nsub.target = sub_mean.target *0.2 + sub_new.target *0.5 + sub_tabular.target *0.2 + sub_tuned.target *0.1\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('ensemble_submission.csv', index = False)\nsub.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}