{"cells":[{"metadata":{},"cell_type":"markdown","source":"# VGG16 + XGBoost (images + tabular data)\n\nIn this notebook I've tried a very simple approach by using VGG16 and XGBoost classification. \nIt consists of the following steps:\n1. Training a Vgg16 model using the training images; (Used [this notebook](https://www.kaggle.com/ibtesama/siim-baseline-keras-vgg16) as reference)\n2. Using the model to predict the diagnostic of each image of the training set;\n3. Add the predictions as an extra column to the training data (that contains sex, age, etc);\n4. Training an XGBoost classifier in the new dataframe.\n\nIt was inteded just as a learn-to-do-it exercise, and I think it is a good start point for newbies.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Imports","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install livelossplot","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install tornado==4.5.3","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport xgboost as xgb\nimport numpy as np\nfrom sklearn.preprocessing import MinMaxScaler, LabelEncoder\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error, accuracy_score, f1_score\n\nfrom sklearn.impute import KNNImputer\nfrom sklearn.metrics import roc_curve\nfrom sklearn.metrics import roc_auc_score\nfrom matplotlib import pyplot\nfrom sklearn.metrics import precision_recall_curve\nfrom sklearn.metrics import auc\n\n\nimport tensorflow\nimport cv2\nimport PIL\nfrom IPython.display import Image, display\nfrom keras.applications.vgg16 import VGG16,preprocess_input\n\nimport plotly.graph_objs as go\nimport plotly.graph_objects as go\nfrom sklearn.metrics import cohen_kappa_score\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import Sequential, Model, load_model\nfrom keras.applications.vgg16 import VGG16, preprocess_input\nfrom keras.applications.resnet50 import ResNet50\nfrom keras.preprocessing.image import ImageDataGenerator,load_img, img_to_array\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, MaxPooling2D, Dense, Dropout, Input, Flatten, BatchNormalization, Activation\nfrom keras.layers import GlobalMaxPooling2D\nfrom keras.models import Model\nfrom keras.optimizers import Adam, SGD, RMSprop\nfrom keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\nfrom keras.utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nimport gc\nimport skimage.io\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom tensorflow.python.keras import backend as K\nfrom livelossplot import PlotLossesKeras\n\n\ndef clean_dataset(df):\n    \"\"\"\n    Cleans data frame from NaNs, Infs and missing cells.\n    \"\"\"\n    assert isinstance(df, pd.DataFrame), \"df needs to be a pd.DataFrame\"\n    df.dropna(inplace=True)\n    indices_to_keep = ~df.isin([np.nan, np.inf, -np.inf]).any(1)\n    return df[indices_to_keep].astype(np.float64)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# VGG-16\n\nLet's start by using VGG16 to perform prediction using the images. Then we will add the results from the VGG prediction to the rest of the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_CSV_PATH = '../input/siim-isic-melanoma-classification/test.csv'\nTRAIN_CSV_PATH = '../input/siim-isic-melanoma-classification/train.csv'\nTEST_JPEG_PATH = '../input/siim-isic-melanoma-classification/jpeg/test/'\nTRAIN_JPEG_PATH = '../input/siim-isic-melanoma-classification/jpeg/train/'\n\ntrain=pd.read_csv(TRAIN_CSV_PATH)\ntest=pd.read_csv(TEST_CSV_PATH)\ntrain.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val1, val2 = train['target'].value_counts()\ndist=train['target'].value_counts()\nprint(f\"{(val1/(val1+val2))*100}% of benign data.\")\nprint(f\"{(val2/(val1+val2))*100}% of malign data.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only about 1.7% of the data consists of malign examples.\nAs we already know, that can be a problem. Check the Classifier notebook. In the Classifier notebook, we tried using SMOTE to deal with that problem. Well, that did not work that well. This time let's be more direct (and a lil bit drastic). I will just select a rather small sample of the data for training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_0=train[train['target']==0].sample(2000)\ndf_1=train[train['target']==1]\ntrain=pd.concat([df_0,df_1])\ntrain=train.reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Preparing datasets...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"labels=[]\ndata=[]\nfor i in range(train.shape[0]):\n    data.append(TRAIN_JPEG_PATH + train['image_name'].iloc[i]+'.jpg')\n    labels.append(train['target'].iloc[i])\ndf=pd.DataFrame(data)\ndf.columns=['images']\ndf['target']=labels\n\ntest_data=[]\nfor i in range(test.shape[0]):\n    test_data.append(TEST_JPEG_PATH + test['image_name'].iloc[i]+'.jpg')\ndf_test=pd.DataFrame(test_data)\ndf_test.columns=['images']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am using here .astype(np.float32) to avoid the following error:\n``TypeError: Input 'y' of 'Mul' Op has type int64 that does not match type float32 of argument 'x'``\n\nI **DID NOT** have this same problem running this notebook at Kaggle, where the tensorflow version is 2.1.0 (I am now executing the newest vertsion 2.3)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(df['images'],df['target'], test_size=0.2, random_state=1234)\n\ntrain=pd.DataFrame(X_train)\ntrain.columns=['images']\ntrain['target']=y_train.astype(np.float32)\n\nvalidation=pd.DataFrame(X_val)\nvalidation.columns=['images']\nvalidation['target']=y_val.astype(np.float32)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll do some very basic preprocessing like\n* normalizing\n* reshaping\n* augmentation(only for train data)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale=1./255,rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,horizontal_flip=True)\nval_datagen=ImageDataGenerator(rescale=1./255)\ntrain_generator = train_datagen.flow_from_dataframe(\n    train,\n    x_col='images',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=8,\n    shuffle=True,\n    class_mode='raw')\n\nvalidation_generator = val_datagen.flow_from_dataframe(\n    validation,\n    x_col='images',\n    y_col='target',\n    target_size=(224, 224),\n    shuffle=False,\n    batch_size=8,\n    class_mode='raw')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### VGG modelling + training\n\nI'm using pretrained VGG-16 and adding the last dense layer. The competition is evaluated on AUC scores, so we'll use that as a metric.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def vgg16_model(num_classes=None):\n\n    model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n    x=Flatten()(model.output)\n    output=Dense(1,activation='sigmoid')(x) # because we have to predict the AUC\n    model=Model(model.input,output)\n    \n    return model\n\nvgg_conv=vgg16_model(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also, because of class imbalance it's better to use **focal loss** rather than normal **binary_crossentropy**. You can read more about it [here](https://arxiv.org/abs/1708.02002).\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def focal_loss(alpha=0.25, gamma=2.0):\n    def focal_crossentropy(y_true, y_pred):\n        bce = K.binary_crossentropy(y_true, y_pred)\n        \n        y_pred = K.clip(y_pred, K.epsilon(), 1.- K.epsilon())\n        p_t = (y_true*y_pred) + ((1-y_true)*(1-y_pred))\n        \n        alpha_factor = 1\n        modulating_factor = 1\n\n        alpha_factor = y_true*alpha + ((1-alpha)*(1-y_true))\n        modulating_factor = K.pow((1-p_t), gamma)\n\n        # compute the final loss and return\n        return K.mean(alpha_factor*modulating_factor*bce, axis=-1)\n    return focal_crossentropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"opt = Adam(lr=1e-5)\nvgg_conv.compile(loss=focal_loss(), metrics=[tf.keras.metrics.AUC()],optimizer=opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nb_epochs = 20\nbatch_size = 16\nnb_train_steps = train.shape[0]//batch_size\nnb_val_steps=validation.shape[0]//batch_size\nprint(\"Number of training and validation steps: {} and {}\".format(nb_train_steps,nb_val_steps))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And lets create a checkout callback for the best the best model. I'll use the validation loss as monitor. That will save use some time in case something breaks during a large session of training. In such a case, we can just reload the checkout model and resume training on the weights.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"checkpoint = ModelCheckpoint(\"best_model.hdf5\", monitor='val_loss', verbose=1,\n    save_best_only=True, mode='auto', period=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cb=[PlotLossesKeras(), checkpoint]\nvgg_conv.fit_generator(\n    train_generator,\n    steps_per_epoch=nb_train_steps,\n    epochs=nb_epochs,\n    validation_data=validation_generator,\n    callbacks=cb,\n    validation_steps=nb_val_steps)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# serialize model to JSON\nmodel_json = vgg_conv.to_json()\nwith open(\"last_model.json\", \"w\") as json_file:\n    json_file.write(model_json)\n# serialize weights to HDF5\nvgg_conv.save_weights(\"last_model.h5\")\nprint(\"Saved model to disk\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# XGBoost","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Prepare data \n\n* We drop all data that is not used for classification, namely the patient id, and malign/bening (that actually just gives a name for each label, but is already represented by the target column). \n* We also have to drop the diagnosis because it defines a cancer (obvsly it is not present at the evaluation data AND using it would provoke an accuracy of 1).\n* Use one-hot encoding for sex and anatomy (the part of the body where the photo is from).\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data_df = pd.read_csv(TRAIN_CSV_PATH) \ndata_df = data_df.drop(columns=['benign_malignant', 'diagnosis', 'patient_id'])\n\n# one-hot encoding\ndata_df = pd.concat([data_df,pd.get_dummies(data_df['sex'], prefix='sex', drop_first=True)],axis=1)\ndata_df = pd.concat([data_df,pd.get_dummies(data_df['anatom_site_general_challenge'], prefix='anatom', drop_first=True)],axis=1)\ndata_df.drop(['sex', 'anatom_site_general_challenge'],axis=1, inplace=True)\n\n# normalization\ncolumn_names_to_normalize = ['age_approx']\nx = data_df[column_names_to_normalize].values\nscaler = MinMaxScaler() \nx_scaled = scaler.fit_transform(x)\ndf_temp = pd.DataFrame(x_scaled, columns=column_names_to_normalize, index = data_df.index)\ndata_df[column_names_to_normalize] = df_temp\n\ndata_df.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Predict training data (adding VGG predictions)\n\nNow we use the newly trained VGG model to extract predicitons for the training data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"loaded_vgg = load_model('./best_model.hdf5', custom_objects={'focal_crossentropy': focal_loss})\n\ndef vgg_prediction(image_name):\n    #print(str(TRAIN_JPEG_PATH + image_name + '.jpg'))\n    img = cv2.imread(str(TRAIN_JPEG_PATH + image_name + '.jpg'))\n    img = cv2.resize(img, (224,224))\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = img.astype(np.float32)/255.\n    img = np.reshape(img,(1,224,224,3))\n    prediction = loaded_vgg.predict(img)\n    return prediction[0][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_df['vgg_prediction'] = data_df['image_name'].apply(vgg_prediction)\ndata_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can take quite some time to get the predictions for all images, so lets save it in a csv file just to be sure we wont loose it.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_df.to_csv(\"train_with_vgg.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Spliting train and test data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y = data_df['target']\nX_data = data_df.drop(columns=['target', 'image_name'])\n\n# split the dataset into train and Test\nseed = 7\ntest_size = 0.3\nXtrain, Xtest, ytrain, ytest = train_test_split(X_data, y, test_size=test_size, random_state=seed)\n\nXtrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### XGBoost model definition","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dmmatrix= xgb.DMatrix(data=Xtrain,label=ytrain)\n\nworkers=4\n\nparam_bin = {\n    'nthread':workers,\n    'max_depth': 500,\n    'eta': 0.01,\n    'gamma':0,\n    'subsample':0.8,\n    'colsample_bytree':0.8,\n    'objective': 'binary:logistic'}\nepochs = 1000\n\nmodel_bin = xgb.train(param_bin, data_dmmatrix, epochs)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xgb_test = xgb.DMatrix(Xtest, label=ytest)\npredictions_bin = model_bin.predict(xgb_test)\n\nauc_bin = roc_auc_score(ytest, predictions_bin)\nprint('binary:logistic ROC AUC=%.3f' % (auc_bin))\n\n# calculate roc curves\nfpr_bin, tpr_bin, _ = roc_curve(ytest, predictions_bin)\n\n# plotting the roc curves of each xgboost objective model\npyplot.plot(fpr_bin, tpr_bin, linestyle='--', label='binary:logistic')\npyplot.xlabel('False Positive Rate')\npyplot.ylabel('True Positive Rate')\npyplot.legend()\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create submission file\nNow we use the test.csv file, get the predictions and generate a submission file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def vgg_prediction_test(image_name):\n    img = cv2.imread(str(TEST_JPEG_PATH + image_name + '.jpg'))\n    img = cv2.resize(img, (224,224))\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = img.astype(np.float32)/255.\n    img = np.reshape(img,(1,224,224,3))\n    prediction = loaded_vgg.predict(img)\n    return prediction[0][0]\n \neval_data_df = pd.read_csv(TEST_CSV_PATH)\nimageNames_lst = eval_data_df['image_name']\n\neval_data_df = pd.concat([eval_data_df,pd.get_dummies(eval_data_df['sex'], prefix='sex', drop_first=True)],axis=1)\neval_data_df = pd.concat([eval_data_df,pd.get_dummies(eval_data_df['anatom_site_general_challenge'], drop_first=True, prefix='anatom')],axis=1)\n\neval_data_df.drop(['sex', 'anatom_site_general_challenge', 'patient_id'],axis=1, inplace=True)\n\n# normalize the age...\ncolumn_names_to_normalize = ['age_approx']\nx = eval_data_df[column_names_to_normalize].values\nscaler = MinMaxScaler() \nx_scaled = scaler.fit_transform(x)\ndf_temp = pd.DataFrame(x_scaled, columns=column_names_to_normalize, index = eval_data_df.index)\neval_data_df[column_names_to_normalize] = df_temp\n\n# take the vgg prediction and add it to the dataframe...\neval_data_df['vgg_prediction'] = eval_data_df['image_name'].apply(vgg_prediction_test)\n\n# now we dont need the image name anymore...\neval_data_df = eval_data_df.drop(columns=['image_name'])\nX = eval_data_df\n\nxgb_X = xgb.DMatrix(X)\npredictions_bin = model_bin.predict(xgb_X)\n\nsub_data_bin = {'image_name': imageNames_lst, 'target':predictions_bin}\nsub_df_bin = pd.DataFrame(sub_data_bin) \nsub_df_bin","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df_bin.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}