{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Stacking model predictions and incorporating image data\n\nOne method for model stacking that I have not seen explored much on kaggle is the use of model predictions with some form of image encoding. Below is an attempt to build a model that combines the OOF preds of my best models with image embeddings that are extracted from [this](https://www.kaggle.com/cdeotte/rapids-cuml-knn-find-duplicates) notebook. \n\nI am curious as to why people do not pursue this method? And if anyone has any ideas as to how it can be improved?\n\nI have also experimented with this form of stacking using efficient to learn encodings and concatenating my predictions as though they were meta data, but my TPU hours were too valuable at the close of this competition and so I stopped pursuing that avenue. If there is any interest in that I can post the kernel. I have included at the end of this notebook the way that predictions can be read into training when using tfrecords.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\n\nfrom kaggle_datasets import KaggleDatasets\n\nfrom xgboost import XGBClassifier\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.model_selection import KFold\n\nimport random\nfrom functools import reduce\n\nimport scipy\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_df(file,name):\n    try:\n        df = pd.read_csv(file.format(name))\n    except FileNotFoundError:\n        # This is garunteed to exist if you haven't entered the filepath incorrectly\n        df = pd.read_csv(file.format(''))\n    return df\n\ndef read_oof(data_ext,name=''):\n    return [get_df('../input/'+file+'/{}oof.csv',name).set_index('image_name') for file in data_ext]\n\ndef read_pred(attach,mean,name=''):\n    df = get_df('../input/'+attach + '/{}preds_all.csv',name)\n    img_col = np.where(df.dtypes=='object')[0][0]\n    df.columns.values[img_col]='0'\n    df = pd.DataFrame(dict(image_name=df['0'], target=mean(df.drop(['0'],axis=1),axis=1)))\n    return df.set_index('image_name')\n\ndef read_preds(data_ext,mean,name=''):\n    return [read_pred(file,mean,name=name) for file in data_ext]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# List all of the data that you wish to include\n\ndata_ext = ['meta-128-assemble',\n            'meta-192',\n            'bs-384-combine',\n            'sls-assemble',\n           'meta-512-assemble',\n           'meta-768-ls-0-05-assemble',\n            'meta-1026-ls-0-05-assemble',\n           'meta-data-xgb']\n\n# Read data into lists\n# Note that preds_all.csv contains all of the predictions from each of the models that I trained, this allows me to \n# use different methods for averaging them.\nmn_func = scipy.stats.gmean\nsub_list = read_preds(data_ext,mn_func)\noof_list = read_oof(data_ext,'oof')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def collect(df_list):\n    return pd.concat(df_list,axis=1)\n\n# Collect all of the data\nsub_df = collect(sub_list)\noof_df = collect(oof_list)\n\n# Separate data into useful parts\noof_df = oof_df.dropna()\nids = oof_df.index\noof = np.array( oof_df['pred'] )\nlabels = np.array(oof_df['target'])[:,0]\nprint('There are {} values overlapping'.format(len(oof_df)))\n\nidsO = sub_df.index\ndata_sub = np.array(sub_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the embedded data\nDIM = 256; EFFN = 0; BATCH_SIZE = 128\nPATH_TO_EMBEDDINGS = '../input/embeddingsmelanoma/'\nembed = np.load(PATH_TO_EMBEDDINGS+'embed_train_%i_%i.npy'%(DIM,EFFN))\nnames = np.load(PATH_TO_EMBEDDINGS+'names_train.npy')\nembed_test = np.load(PATH_TO_EMBEDDINGS+'embed_test_%i_%i.npy'%(DIM,EFFN))\nnames_test = np.load(PATH_TO_EMBEDDINGS+'names_test.npy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD TRAIN AND TEST CSV\ntest = pd.read_csv( '../input/siim-isic-melanoma-classification/test.csv' ).set_index('image_name',drop=True)\ntest = test.loc[names_test]\ntest = pd.concat((test,sub_df),axis=1).reset_index()\nprint('Test csv shape',test.shape)\n\ntrain = pd.read_csv( '../input/melanoma-%ix%i/train.csv'%(DIM,DIM) ).set_index('image_name',drop=True)\ntrain = train.loc[names]\ntrain = pd.concat((train,oof_df['pred']),axis=1).reset_index()\ntrain.target = train.target.astype('float32')\nprint('Train csv shape',train.shape)\n\nprint('Displaying train.csv below...')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Concatenate and preprocess the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_enc = OneHotEncoder(drop='first')\n# num_enc = MinMaxScaler()\nnum_enc = StandardScaler()\nembed_enc = MinMaxScaler()\nnumeric_features = ['age_approx']\ncat_features = ['sex','anatom_site_general_challenge']\n\ncats = cat_enc.fit_transform(train[cat_features].fillna('0')).toarray()\nnums = num_enc.fit_transform(train[numeric_features].fillna(0))\nembed_ENC = embed_enc.fit_transform(embed)\nXtrain = np.concatenate((cats,nums,embed_ENC),axis=1)\nXpreds = np.array(train['pred'])\n\ncats = cat_enc.transform(test[cat_features].fillna('0')).toarray()\nnums = num_enc.transform(test[numeric_features].fillna(0))\nembed_encT = embed_enc.transform(embed_test)\nXtest = np.concatenate((cats,nums,embed_encT),axis=1)\nXtargets = np.array(test['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Take a look at the embeddings\nprint(np.shape(embed_ENC))\nfor i in range(100):\n    plt.hist(embed_ENC[i],alpha=0.5,bins=100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at how they are distributed so some estimate for augmentation can be performed.\nstd_all = np.std(embed_ENC,axis=1)\nplt.hist(std_all,bins=100)\nplt.show()\nembed_std = np.mean(std_all)\nprint('Mean standard deviation is {}'.format(embed_std))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set hyperparameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"weights = {0:1, 1:15}\nFOLDS=5\nSEED=42\nDISPLAY_PLOT = 1\nREPLICAS=1\nEPOCHS=100\nTTA=11\nbatch_size = 64\nVERBOSE=0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.random.set_seed(5);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(data,mean=0.0,std1=embed_std/10,std2=0.03):\n#     print(data.shape())\n    gauss = np.random.normal(loc=mean, scale=std1, size=Xtrain.shape[1])\n    new_data = data[0] + gauss\n    gauss = np.random.normal(loc=mean, scale=std2, size=Xpreds.shape[1])\n    new_preds = data[1] + gauss\n    return (new_data,new_preds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How to augment meta data and image embeddings properly? Ideally you would take the std from the base models for the predictions and also figure out how the image embeddings change when you augment the image and econde it.\n\nAlso, at present I am augmenting the categorical data with random noise, which isn't the best thing to do. But for illustration purposes this serves.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# helper function for loading data\ndef get_dataset(X,subs,y,augment=True,repeat=True,batch=batch_size):\n    ds = tf.data.Dataset.from_tensor_slices(((X,subs), y))\n    if repeat:\n        ds = ds.repeat()\n    if augment:\n        ds = ds.map(lambda elem,label: (data_augment(elem),label))\n        \n    ds = ds.batch(batch)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build a model that takes meta data, image embeddings and other model predictions as input","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dim = Xtrain.shape[1]\nmeta_dim = Xpreds.shape[1]\ndef build_model(ls=0.05):\n    inp = tf.keras.layers.Input(shape=(dim,))\n    x = L.Dropout(0.2)(inp)\n    x = L.Dense(int(1024), activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(int(512), activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(int(256), activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(int(128), activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(64, activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(32, activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(16, activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    x = L.Dense(12, activation='relu')(x)\n    x = L.Dropout(0.3)(x)\n    meta_inp = tf.keras.layers.Input(shape=(meta_dim,))\n    xm = L.concatenate((x,meta_inp))\n    xm = L.Dense(16, activation='relu')(xm)\n    xm = L.Dropout(0.2)(xm)\n    xm = L.Dense(12, activation='relu')(xm)\n    xm = L.Dropout(0.2)(xm)\n    xm = L.Dense(8, activation='relu')(xm)\n    xm = L.Dropout(0.1)(xm)\n    xm = L.Dense(1, activation='sigmoid')(xm)\n    model = tf.keras.Model(inputs=(inp,meta_inp), outputs=xm)\n    opt = tf.keras.optimizers.Adam(learning_rate=0.00000125* REPLICAS * batch_size)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=ls) \n    model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train the model\n\nThis has been repurposed from [this](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) notebook.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n#     lr_max     = 0.0000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 50\n    lr_sus_ep  = 0\n    lr_decay   = 0.99\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = train['index']\nlabels = train.target.astype('int32')\nidsO = test['index']\ndata_sub = Xtest\n# Default strategy for single GPU\nstrategy = tf.distribute.get_strategy()\n\n# skf = StratifiedKFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\nskf = KFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\n\noof_pred = []; oof_tar = []; oof_val = []; oof_names = []; oof_folds = [] \npreds = np.zeros((test.shape[0],1))\npreds_all = np.zeros((test.shape[0],FOLDS))\n\nfor fold,(idxT2,idxV2) in enumerate(skf.split(np.arange(15))):\n\n    idxT = train.loc[train.tfrecord.isin(idxT2)].index.values #2020 train\n    idxV = train.loc[train.tfrecord.isin(idxV2)].index.values #2020 valid\n    \n#     X = data_augment(data[idxT]); y = train.target[idxT]\n    X = Xtrain[idxT]; y = labels[idxT]; XX=Xpreds[idxT]\n    X_val = Xtrain[idxV]; y_val = labels[idxV]; XX_val=Xpreds[idxV]\n    \n    # BUILD MODEL\n    tf.keras.backend.clear_session()\n    with strategy.scope():\n        model=build_model()\n\n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n#     sv = tf.keras.callbacks.ModelCheckpoint(\n#         'fold-%i.h5'%fold, monitor='val_auc', verbose=0, save_best_only=True,\n#         save_weights_only=True, mode='min', save_freq='epoch')\n\n    # Train the model\n    history = model.fit(get_dataset(X,XX,y),epochs=EPOCHS,steps_per_epoch=X.shape[0]/batch_size//REPLICAS,\n                        verbose=VERBOSE,class_weight=weights, \n#                         callbacks=[sv,get_lr_callback(batch_size=batch_size)],\n                        callbacks=[sv],\n                       validation_data=get_dataset(X_val,XX_val, y_val,augment=False,repeat=False))\n\n    model.load_weights('fold-%i.h5'%fold)\n\n    # PREDICT OOF USING TTA\n    STEPS = TTA*X_val.shape[0]/(batch_size-1)/REPLICAS\n    pred = model.predict( get_dataset(X_val,XX_val,y_val), steps=STEPS )[:TTA*X_val.shape[0]]\n    oof_pred.append( np.mean(pred.reshape((X_val.shape[0],TTA),order='F'),axis=1) )                \n\n    # GET OOF TARGETS AND NAMES\n    oof_tar.append( y_val )\n    oof_names.append( ids[idxV] )\n    oof_folds.append( np.ones_like(oof_tar[-1],dtype='int8')*fold )\n\n    STEPS = TTA*Xtest.shape[0]/(batch_size-1)/REPLICAS\n    psub = model.predict( get_dataset(Xtest,Xtargets,np.zeros(len(Xtest))), steps=STEPS  )[:TTA*Xtest.shape[0]]\n    pstore = np.mean(psub.reshape((len(Xtest),TTA),order='F'),axis=1)\n    preds[:,0] += pstore*1/FOLDS\n\n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n    oof_val.append( np.max( history.history['val_auc'] ) )\n    print('#### FOLD %i OOF AUC without TTA = %.3f, with TTA = %.3f'%(fold+1,oof_val[-1],auc))\n\n    # PLOT TRAINING\n    if DISPLAY_PLOT:\n        plt.figure(figsize=(15,5))\n        plt.plot(np.arange(EPOCHS),history.history['auc'],'-o',label='Train AUC',color='#ff7f0e')\n        plt.plot(np.arange(EPOCHS),history.history['val_auc'],'-o',label='Val AUC',color='#1f77b4')\n        x = np.argmax( history.history['val_auc'] ); y = np.max( history.history['val_auc'] )\n        xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max auc\\n%.2f'%y,size=14)\n        plt.ylabel('AUC',size=14); plt.xlabel('Epoch',size=14)\n        plt.legend(loc=2)\n        plt2 = plt.gca().twinx()\n        plt2.plot(np.arange(EPOCHS),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n        plt2.plot(np.arange(EPOCHS),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n        x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n        ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n        plt.ylabel('Loss',size=14)\n        plt.legend(loc=3)\n        plt.show()  \n\n# COMPUTE OVERALL OOF AUC\noof_c = np.concatenate(oof_pred); true = np.concatenate(oof_tar);\nnames = np.concatenate(oof_names); folds = np.concatenate(oof_folds)\nauc = roc_auc_score(true,oof_c)\nprint('Overall OOF AUC with TTA = %.5f'%auc)\n\ndd_oof = pd.DataFrame(dict(image_name = names, target=true, pred = oof_c, fold=folds))\n\nsubmission = pd.DataFrame(dict(image_name=idsO, target=preds[:,0]))\nsubmission = submission.sort_values('image_name')\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With a linear blending approach the submissions that are passed to this model get a max OOF AUC score of 0.9417.\n\nI would be interested to know what other people think of this kernel, and would appreciate any suggestions for improvements, I did not have a huge amount of time to spend on this - and only thought of trying it in the last days of the competition - so I am sure I missed something.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Performing the same method with tfrecords","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"When using this method with a TPU you don't want to rewrite the files every time just so you can incorporate the model predictions because you would want to be updating these predictions constantly. This is true more generally I think for any time you have new data.\n\nSo when using tfrecords with the above method you have to do the following to read predictions into the training (the tfrecords contain ```example['image_name']``` which can be used as a key to read in the external data).\n\n```\nnames = pd.concat((oof_df['image_name'],sub_df['image_name']))\ndata = np.concatenate((oof_df['pred'],sub_df['target']))\n\n#Make a lookup table for the data\nwith strategy.scope():\n    get_index = tf.lookup.StaticHashTable(\n      tf.lookup.KeyValueTensorInitializer(names, np.arange( len(names) )), -1\n    )\n    METADATA = tf.constant(data)\n```\n    \nThen later you can add\n\n```\nquery = get_index.lookup(example['image_name'])\nmeta_models = tf.gather(METADATA, query)\n```\n\n(into ```read_labeled_tfrecord``` and ```read_unlabeled_tfrecord``` from AgentAuers' notebook [here](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once)) and use ``meta_models`` int the same way that ```meta_inp``` is used in this notebook in ```build_model```.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}