{"cells":[{"metadata":{},"cell_type":"markdown","source":"This Kernel was made to compare different models for tranfer learning using Tensorboard visualisation tools, all while using Keras.\nNot really knowing what architecture to go with in our model, we decided to try to compare different models that are known for image classification and compare how they did in this specific problem.\n\nThis competition was the very first we did, and so there might be errors, or incoherent things. If you think something could have been better, in a more efficient manner, or simply if you think something is wrong, please let us know! We did this competition mostly to learn as much as possible and it has been a great learning experience for the past few months! But we still have a lot to learn and even more willing to!\n\nThe logic was to craete a function, where the only input would be the model we wanted to try, and it would run it and then allow to be compared in Tensorboard."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nfrom datetime import datetime\n\nfrom keras.models import Sequential\nimport tensorflow as tf\nfrom keras import backend as K\nfrom keras.models import Model\nfrom keras_preprocessing.image import ImageDataGenerator\nfrom keras.layers import Input, Dense,Concatenate, GlobalMaxPooling2D, GlobalAveragePooling2D, Activation, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D, Dropout, BatchNormalization\nfrom keras.losses import mae, sparse_categorical_crossentropy, binary_crossentropy\nfrom keras import regularizers, optimizers\nfrom keras.optimizers import Adam\n\nfrom keras.applications import ResNet50\nfrom keras.applications.vgg19 import VGG19\nfrom keras.applications.mobilenet import MobileNet\nfrom keras.applications.inception_v3 import InceptionV3\nfrom keras.applications.vgg16 import VGG16\nfrom keras.applications.nasnet import NASNetMobile\n\n\nfrom keras.callbacks import ModelCheckpoint,EarlyStopping,TensorBoard,CSVLogger,ReduceLROnPlateau\n\nimport matplotlib.pyplot as plt \nplt.rcParams['figure.figsize'] = (20,10)\n\ndef append_ext(fn):\n    return fn+\".tif\"\n\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":1,"outputs":[{"output_type":"stream","text":"Using TensorFlow backend.\n","name":"stderr"},{"output_type":"stream","text":"['train', 'test', 'train_labels.csv', 'sample_submission.csv']\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"First we want to get the test and train dataset. Notice that we will only take 180,000 images, that is because we want to keep a balanced dataset (in the train set there are about 220,000 images, but the labels are not balanced)."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"traindf=pd.read_csv(\"../input/train_labels.csv\",dtype=str)\ntrain_size = 180000\ntraindf = traindf.sort_values(by=['label','id'])\ntraindf = traindf.iloc[:int(train_size/2)].append(traindf.iloc[-int(train_size/2):])\ntestdf=pd.read_csv(\"../input/sample_submission.csv\",dtype=str)\ntraindf[\"id\"]=traindf[\"id\"].apply(append_ext)\ntestdf[\"id\"]=testdf[\"id\"].apply(append_ext)\ndatagen=ImageDataGenerator(rescale=1./255.,validation_split=0.25)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While creating the datagens we used a batch_size of 512, because we want to go for a relatively rapid testing. In practise it seems like a lower batch_size (around 32 or 64) gives better results. But we simply want to see which model seems the best for this probelm compared to the others, so we prefer going for a code that runs fater."},{"metadata":{"trusted":true,"_uuid":"9a6c1776132938fe9d7453158405e16ecb7aab65","scrolled":true},"cell_type":"code","source":"B_size = 512\n\ntrain_generator=datagen.flow_from_dataframe(\n                                            dataframe=traindf,\n                                            directory=\"../input/train/\",\n                                            x_col=\"id\",\n                                            y_col=\"label\",\n                                            subset=\"training\",\n                                            batch_size=B_size,\n                                            seed=42,\n                                            shuffle=True,\n                                            class_mode=\"binary\",\n                                            target_size=(96, 96)\n)\n\nvalid_generator=datagen.flow_from_dataframe(\n                                            dataframe=traindf,\n                                            directory=\"../input/train/\",\n                                            x_col=\"id\",\n                                            y_col=\"label\",\n                                            subset=\"validation\",\n                                            batch_size=B_size,\n                                            seed=42,\n                                            shuffle=True,\n                                            class_mode=\"binary\",\n                                            target_size=(96, 96)\n)\n\ntest_datagen=ImageDataGenerator(rescale=1./255.)\n\ntest_generator=test_datagen.flow_from_dataframe(\n                                                dataframe=testdf,\n                                                directory=\"../input/test/\",\n                                                x_col=\"id\",\n                                                y_col=None,\n                                                batch_size=B_size,\n                                                seed=42,\n                                                shuffle=False,\n                                                class_mode=None,\n                                                target_size=(96, 96)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2206edd0a86b806d201eec4085cea486888873bd"},"cell_type":"code","source":"train_generator.n//train_generator.batch_size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The metric used for this competition was a ROC AUC, so we made a metric to be able to measure this. We seem to have gotten the wrong metric though, because most of the time in our submits, we had a validation AUC very high in our program (sometimes even going as high as 0.99) but when submitting, it would fall down to 0.94. This is something we need to look into for future competitions."},{"metadata":{"trusted":true,"_uuid":"490fb2646bf8985a4b18ef8c3d0c0ff3ff68b552"},"cell_type":"code","source":"def auc(y_true, y_pred):\n    \"\"\"ROC AUC metric evaluator\"\"\"\n    auc = tf.metrics.auc(y_true, y_pred)[1]\n    K.get_session().run(tf.local_variables_initializer())\n    return auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dfed64d51239dd4468c19b2c838e997b3eb09494"},"cell_type":"code","source":"def make_model(model_choice, model_name, input_tensor):\n    '''Function to create a model\n    Input:\n    - model_choice, for ex: VGG19(include_top=False, input_tensor=input_tensor)\n    - model_name, (str), name that will be given to the model in tensorboard\n    \n    Output:\n    - model made with keras.model.Model'''\n    \n    base_model = model_choice\n    x = base_model(input_tensor)\n    out = Flatten()(x)\n    out = Dense(1, activation=\"sigmoid\")(out)\n    model = Model(input_tensor, out)\n    \n    #The only callback we will use is TensorBoard, we could use early stopping or modifying the learning rate\n    #but we wanted to compare the models as they were, with the same parameters for each.\n    tensorboard=TensorBoard(log_dir = './logs/{}'.format(model_name),\n                            histogram_freq=0,\n                            batch_size=B_size,\n                            write_graph=True,\n                            write_grads=True,\n                            write_images=False)\n    \n    model.compile(optimizer=Adam(0.0001), loss=binary_crossentropy, metrics=['accuracy', auc])\n    model.summary()\n    \n    history = model.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=5,\n                    callbacks=[tensorboard])\n    \n\n    plt.plot(history.history['auc'])\n    plt.plot(history.history['val_auc'])\n    plt.title(model_name +  ' Model AUC')\n    plt.legend([model_name +  ' Training',model_name +  ' Validation'])\n    plt.ylabel('AUC')\n    plt.xlabel('epoch')\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\nSTEP_SIZE_TEST=test_generator.n//test_generator.batch_size","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We decided to test some of the models available with Keras that were known for image classification. In theory, it would have been interesting to test all of the available, as well as a CNN that we made ourselves, but this was our very first competition, so we mostly wanted to try making something that worked in the first place."},{"metadata":{"trusted":true,"_uuid":"a0d0250ce5c82fc437f554acc0fd3a98661f45db"},"cell_type":"code","source":"input_tensor = Input((96, 96, 3))\n\nVGG19_model = make_model(VGG19(include_top=False, weights='imagenet', input_tensor=input_tensor), 'VGG19', input_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"VGG16_model = make_model(VGG16(include_top=False, weights='imagenet', input_tensor=input_tensor), 'VGG16', input_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MobileNet_model = make_model(MobileNet(include_top=False, weights='imagenet', input_tensor=input_tensor), 'MobileNet', input_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NASNetMobile_model = make_model(NASNetMobile(include_top=False, weights='imagenet', input_tensor=input_tensor), 'NASNetMobile', input_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"InceptionV3_model = make_model(InceptionV3(include_top=False, weights='imagenet', input_tensor=input_tensor), 'InceptionV3', input_tensor)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The ResNet50 didn't seem to work with input_tensor, so instead of using our function, we just copied it and changed the inputs with input_shape, and it worked great."},{"metadata":{"trusted":true,"_uuid":"07ae1b03c810347e6c1540225535ec39add61e23"},"cell_type":"code","source":"input_shape = (96, 96, 3)\n\nResNet50_model=ResNet50(include_top=False, input_tensor=None, weights='imagenet', input_shape = input_shape)\nRx = ResNet50_model.output\nRx = Flatten()(Rx)\nprediction = Dense(1, activation=\"sigmoid\")(Rx)\nRmodel = Model(ResNet50_model.input, prediction)\n\nResNet50_tensorboard = TensorBoard(log_dir = './logs/{}'.format('ResNet50'),\n                                            histogram_freq=0,\n                                            batch_size=B_size,\n                                            write_graph=True,\n                                            write_grads=True,\n                                            write_images=False)\n\nRmodel.compile(optimizer=Adam(0.0001), loss=binary_crossentropy, metrics=['accuracy', auc])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f3734762dcd0d9e3019a5d559e948db082a026cb"},"cell_type":"code","source":"history = Rmodel.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=5,\n                    callbacks=[ResNet50_tensorboard]\n)\nend = datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b98f7d0e7b072bb518fa86d12462205364c97aa1","scrolled":true},"cell_type":"code","source":"plt.plot(history.history['acc'])\nplt.plot(history.history['val_acc'])\nplt.title('ResNet50 model accuracy')\nplt.legend(['ResNet50_training','ResNet50_validation'])\nplt.ylabel('accuracy')\nplt.xlabel('epoch')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"51506f69bfbf13007bc852f63fbc5efb498eff4a"},"cell_type":"code","source":"plt.plot(history.history['auc'])\nplt.plot(history.history['val_auc'])\nplt.title('ResNet50 model accuracy')\nplt.legend(['ResNet50_training','ResNet50_validation'])\nplt.ylabel('AUC')\nplt.xlabel('epoch')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, this last part generates a link that leads us to TensorBoard to see how each model performs according to the metric we have given him.\n\n(though it does not seem to work once the code is commited, only when running the code in the Notebook. If anybody has a solution to this, we would be very interested!)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n!unzip ngrok-stable-linux-amd64.zip\nLOG_DIR = './logs'\nget_ipython().system_raw(\n    'tensorboard --logdir {} --host 0.0.0.0 --port 8080 &'\n    .format(LOG_DIR)\n)\nget_ipython().system_raw('./ngrok http 8080 &')\n! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Conclusion**\n\nFrom the information we get from TensorBoard, it seems like the ResNet50 is the best model we could use as we have tested it here. The MobileNet and InceptionV3 also seem like good candidates, though a bit behind. Both VGG seem to be under performing compared to the other models, while the NasNet seems good, but its validation loss doesn't seem to change much, which is a bit concerning.\n\nAgain, we know that our AUC metric is a bit flawed, so these should be taken with a grain of salt. Especially since in our testings, it seems like the VGG19 outperformed the ResNet50 on the public leaderboard sumbit.\n\nIt should also be noted that we could try to tweak some things in the transfer learning, untraining more layers and adding others. The choice of an Adam optimizer over something like a SGD is also a questionnable choice.\n\nIf you have any questions and/or remarks, please let us know! This was a very interesting competition, but we also want to learn as much as possible from it now that it is over! :-)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}