{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Hi everyone !","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This first starter notebook is intended to give you an insight of building a model with only patient-level information (no image processing for the moment !). This simple LightGBM baseline can also be used as a more general pipeline for other projects :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Please upvote this kernel if you found it interesting !","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Start with preparing the patient-only data","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport os\nfrom sklearn.metrics import f1_score\nimport seaborn as sns\nfrom scipy.misc import derivative\nimport lightgbm as lgb\nfrom sklearn.model_selection import StratifiedKFold, KFold\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import accuracy_score,confusion_matrix,roc_curve,roc_auc_score,classification_report","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\ntest = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/test.csv')\nsubmission = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Defining useful variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.columns, test.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will make use only of columns that are present both in train and test. It would be interesting, as future imporvement, to try and predict the column diagnosis in order to use it as an instrumental variable and help predict the target.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_only_cols = ['patient_id', 'sex', 'age_approx', 'anatom_site_general_challenge'] \npatient_only_train, patient_only_test = train[patient_only_cols+['target']].drop_duplicates(inplace=False), test[patient_only_cols].drop_duplicates(inplace=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_only_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pre-processing categorical variables","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"LightGBM requires to specify, and transform to integers, all categorical variables. Note : we don't care about missing values, LGBM handles them quite well :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"categoricals = ['sex', 'anatom_site_general_challenge']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What values do we have in the `sex` column ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"set(patient_only_train.sex.values.tolist())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Matching them to integers :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"matching_sex = {'female':1, 'male':0}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What values do we have in the `anatom_site_general_challenge` column ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"set(patient_only_train.anatom_site_general_challenge.values.tolist())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Matching them to integers :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"matching_anatom = {'head/neck':0,\n 'lower extremity':1,\n 'oral/genital':2,\n 'palms/soles':3,\n 'torso':4,\n 'upper extremity':5}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mapping those columns to their integer matched values :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_only_train.replace(to_replace={'anatom_site_general_challenge':matching_anatom, 'sex':matching_sex}, inplace=True)\npatient_only_test.replace(to_replace={'anatom_site_general_challenge':matching_anatom, 'sex':matching_sex}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's define the columns we will use for prediction :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Cols = ['sex', 'age_approx', 'anatom_site_general_challenge']\npatient_only_train[Cols] = patient_only_train[Cols].astype('int32', errors='ignore')\npatient_only_test[Cols] = patient_only_test[Cols].astype('int32', errors='ignore')\npatient_only_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LightGBM Pipeline","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Even if this competition's metric is the AUC, I always find it better to optimize on the mean average precision, especially when classes are extremely unbalanced. Mean Average Precision is more sensible to improvements of respective scoring at the top of the ranking than at the bottom, contrary to AUC which is indifferent.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def MeanAveragePrecision(y_pred, y_true):\n    y_true = y_true.get_label()\n    df = pd.DataFrame({'true': y_true, 'pred_probas': y_pred})\n    n = df.shape[0]\n    df.sort_values(by='pred_probas', ascending=False, inplace=True)\n    df['loss'] = df['true'].cumsum()/list(range(1, n+1))\n    df = df.loc[df['true']==1, 'loss']\n    return \"MeanAveragePrecision\", max(0, df.mean(axis=0)), True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def DataSetLgbm(Data,trn_idx,val_idx,target,features, categorical_features=\"\"):\n    trn_data=lgb.Dataset(Data.iloc[trn_idx][features], label=Data[target].iloc[trn_idx], categorical_feature=categorical_features)\n    val_data=lgb.Dataset(Data.iloc[val_idx][features], label=Data[target].iloc[val_idx], categorical_feature=categorical_features)\n    return trn_data,val_data\n\ndef TrainSimpleLgbm(Params,DataTrain,trn_idx,val_idx,target,features, categorical_features=\"\"): \n    trn_data,val_data=DataSetLgbm(DataTrain,trn_idx,val_idx,target,features, categorical_features=categorical_features)\n    clf=lgb.train(Params, trn_data, 30000, valid_sets = [trn_data, val_data],\n                verbose_eval=100,feval = MeanAveragePrecision, early_stopping_rounds = 500)\n    return clf\n\ndef mean_average_p(y_true, y_pred_p):\n    df = pd.DataFrame({'true': y_true, 'pred_probas': y_pred_p})\n    n = df.shape[0]\n    df.sort_values(by='pred_probas', ascending=False, inplace=True)\n    df['loss'] = df['true'].cumsum()/list(range(1, n+1))\n    df = df.loc[df['true']==1, 'loss']\n    return max(0, df.mean(axis=0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will introduce to set of parameters for 2 different lgbms. Feel free to tamper with those and makeyour own experiments !","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ParN1 = {\n    'bagging_freq': 1,\n    'bagging_fraction': 0.95,\n    'boost_from_average':'true',\n    'boost': 'gbdt',\n    'feature_fraction': 0.5,\n    'learning_rate': 0.04,\n    'max_depth': -1,\n    'metric':'auc',\n    'is_unbalance':'true',\n    'min_data_in_leaf':80,\n    'lambda_l1' :1,\n    'lambda_l2':1,\n    'num_leaves': 2000,\n    'colsample_bytree': 0.9,\n    'tree_learner': 'serial',\n    'objective': 'cross_entropy',\n    'verbosity': 1}\n\nParN2 = {\n    'bagging_freq': 20,\n    'bagging_fraction': 0.9,\n    'boost_from_average':'true',\n    'boost': 'gbdt',\n    'feature_fraction': 0.9,\n    'learning_rate': 0.04,\n    'max_depth': -1,\n    'metric':'auc',\n    'is_unbalance':'true',\n    'lambda_l1' :10,\n    'lambda_l2':10,\n    'num_leaves': 7,\n    'colsample_bytree': 0.7,\n    'tree_learner': 'serial',\n    'objective': 'cross_entropy',\n    'verbosity': 1}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = []\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=20_01_1998)\nfor fold,(trn_idx , val_idx) in enumerate(folds.split(patient_only_train[Cols],patient_only_train['target'])):\n    print(f'********************* Fitting on Fold {fold+1} ... ******************')\n    clf1=TrainSimpleLgbm(ParN1,patient_only_train,trn_idx,val_idx,\"target\",Cols, categoricals)\n    clf2=TrainSimpleLgbm(ParN2,patient_only_train,trn_idx,val_idx,\"target\",Cols, categoricals)\n    \n    pred_oof1 = clf1.predict(patient_only_train.iloc[val_idx][Cols], num_iteration=clf1.best_iteration)\n    pred_test1 = clf1.predict(patient_only_test[Cols], num_iteration=clf1.best_iteration)\n    test_preds.append(pred_test1)\n    pred_oof2 = clf2.predict(patient_only_train.iloc[val_idx][Cols], num_iteration=clf2.best_iteration)\n    pred_test2 = clf1.predict(patient_only_test[Cols], num_iteration=clf2.best_iteration)\n    test_preds.append(pred_test2)\n    mean_pred_oof=0.5*pred_oof1+0.5*pred_oof2\n    \n    m1=mean_average_p(patient_only_train[\"target\"].iloc[val_idx], pred_oof1)\n    m2=mean_average_p(patient_only_train[\"target\"].iloc[val_idx], pred_oof2)\n    m3=mean_average_p(patient_only_train[\"target\"].iloc[val_idx], mean_pred_oof)\n    \n    print(f' Mean Average M1 : {m1}  , M2 : {m2}   M3 : {m3}')\n    \n    \n    pred_oof1=(pred_oof1>=0.5).astype(int)\n    pred_oof2=(pred_oof2>=0.5).astype(int)\n    mean_pred_oof=(mean_pred_oof>=0.5).astype(int)\n    \n    print(\"*************  CR Param 1 *************************\")\n    print(classification_report(patient_only_train[\"target\"].iloc[val_idx],pred_oof1))\n    \n    print(\"*************  CR Param 2 *************************\")\n    print(classification_report(patient_only_train[\"target\"].iloc[val_idx],pred_oof2))\n    \n    print(\"*************  CR Mean *************************\")\n    print(classification_report(patient_only_train[\"target\"].iloc[val_idx],mean_pred_oof))\n   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the feature importance. The most important feature seem to be, by far, the body part in which the melanoma is located.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_importance = pd.DataFrame({'Value':clf1.feature_importance(),'Feature':Cols})\nplt.figure()\nsns.barplot(x=\"Value\", y=\"Feature\", data=feature_importance.sort_values(by=\"Value\", ascending=False))\nplt.title('Features Importance')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we're interested in what part of the body it's actually the most dangerous to get a melanoma, let's make a brief analysis :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['anatom_site_general_challenge'])['target'].mean().sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that it is on the head or neck that you get the most malignant melanoma, with 3 times as much chances as when it is located on palms or soles !","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Submission","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We had kept the `patient_id` column only to be able to perform the future merge of our results. Let's do it in two steps : first, adding our predictions to the `patient_only_test` database where we have `patient_id`, then merging it with the `test` database to match with the `image_name`, and finally using it to fill the `sample_submission` file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_only_test['target'] = np.stack(test_preds, axis=1).mean(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = submission[['image_name']].merge(test[['image_name', 'patient_id']].merge(patient_only_test[['patient_id', 'target']], how='outer', on='patient_id'), how='left', on='image_name').drop(columns='patient_id', inplace=False).drop_duplicates(subset='image_name', inplace=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.isna().describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great, the merging has performed well, we have no missing values !","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.to_csv('sample_submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}