{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Evaluate models for splitting the problem into 3 different problems.\n\n## Disclaimer. A beginner here!\nAs a beginner, I'm not even sure I got the problem we are facing for sure, but for my understanding maybe splitting the problem into three different problems could make the things easier. It seems that masks are a bit different depending of the type of cells we find, then selecting the type of cells in the first place and then apply three different models to find the masks.\n\nIf I'm right, what I was trying to do is to pick the better and faster model to classify each image into the problem we want to solve and then apply the correct 2nd step model to extract the correct masks. To do that, I was wondering what model could fit better and what is the minimum necessary size (parameters/speed) of the model is needed to solve this first problem. \n\n**These notebook can have a totally wrong approach or probably have errors. Please, feel free to correct me on anything that can be improved or the whole point!**\n\n## What you'll find in this notebook\n\nThis notebooks gets all the training images and trains (for 3 epochs) and evaluate on up to 26 different pre-trained models to find which one could fit better to solve what would be my first step of my final pipeline, splitting the test images to be predicted by three different models.\n\n## Starting from the end... references.\nAs a beginner I'm taking advantage of the knowledge of others who has a great experience and knowledge. Here, I'm just compiling and joining the work of others.\n\n### The dataset used:\n\nI'm using the great COCO ready annotations created by [@Slawek Biel](https://www.kaggle.com/slawekbiel) and his great series of notebooks that I'm following to just understand how the whole thing works and being my guideline to enter to this competition:\n- [Positive score with Detectron 1/3 - input data](https://www.kaggle.com/slawekbiel/positive-score-with-detectron-1-3-input-data/)\n- [Positive score with Detectron 2/3 - Training](https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training/)\n- [Positive score with Detectron 3/3 - Inference](https://www.kaggle.com/slawekbiel/positive-score-with-detectron-3-3-inference)\n\nThe dataset I'm using is the on [@Slawek Biel](https://www.kaggle.com/slawekbiel) created through his first notebook: [\"Sartorius - Cell Instance Segmentation\" COCO](https://www.kaggle.com/slawekbiel/sartorius-cell-instance-segmentation-coco)\n\n### Evaluation models\n\nI extracted (copied) the idea of evaluating  multiple models from a great blog post by [Mario Stephan Leo](https://towardsdatascience.com/how-to-choose-the-best-keras-pre-trained-model-for-image-classification-b850ca4428d4) at Medium.\nYou can visit [his Github here](https://github.com/stephenleo/keras-model-selection) \nFor further details on the evaluation models explanation, take a look at the Mario's blog post.\n\n\n## An image classification problem\n\nIn order to split the problem into 3 different problems, It is needed to classify images into different categories. This is a image classification problem. We need to solve for every image which category it contains.\n\n### Input images\n\nFor simplicity, I selected the pre-trained models that work with shape `(224, 224, 3)`. Instead of resizing the images of this competition `(702, 540)`, I decided to split images into chunks of the needed size and then having 6x images for training and 6 images (per image) for predicting and then summing up the prediction scores  for each chunk to obtain the final prediction for the whole image. Probably there're better ways to join predictions than just summing them.\n","metadata":{}},{"cell_type":"markdown","source":"#### Setting up paths","metadata":{}},{"cell_type":"code","source":"ANNOTATIONS_DIR = '../input/sartorius-cell-instance-segmentation-coco/'\nIMG_BASE_DIR = \"../input/sartorius-cell-instance-segmentation\"","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:25:52.357904Z","iopub.execute_input":"2021-11-07T11:25:52.359000Z","iopub.status.idle":"2021-11-07T11:25:52.388086Z","shell.execute_reply.started":"2021-11-07T11:25:52.358889Z","shell.execute_reply":"2021-11-07T11:25:52.387242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Preparing the dataset. Getting 6 images of shape (224,224,3) from every image \n","metadata":{}},{"cell_type":"code","source":"from tqdm.auto import tqdm\nimport json\nimport numpy as np\nfrom scipy.fftpack import dct, idct\nfrom matplotlib.pyplot import figure\nimport matplotlib.pylab as plt\nimport cv2\n\ndef get_class(item, data):\n    image_id = item['file_name'].split('/')[1].split('.')[0]\n    for annot in data['annotations']:\n        if annot['image_id']==image_id: return annot['category_id']\n    return None\n\ndef convert_images(img_path, item, category, images, classes, plot=False):\n    \n    np.set_printoptions(threshold=np.inf)\n    image = cv2.imread(img_path + '/' + item['file_name'])\n    srcImageWidth, srcImageHeight = image.shape[1], image.shape[0]\n\n    im_width = 224\n    im_height = 224\n    \n    # Extracting 6 images per original image and its classes\n    for left in range(3):\n        for top in range(2):\n            images.append(image[int(top*im_height):int(top*im_height)+im_height, left*im_width:(left*im_width)+im_width, :]/255)\n            classes.append(np.asarray(category))\n\n            if plot:\n                figure(figsize=(20, 20), dpi=80)\n                plt.gray()\n                plt.subplot(121), plt.imshow(images[len(images)-1]), plt.axis('off'), plt.title('original image', size=20)\n                plt.subplot(122), plt.imshow(image), plt.axis('off'), plt.title('original image', size=20)\n                plt.show()\n\n    return images, classes\n\ndef convert_dataset(json_filename):\n    print(f\"Converting {json_filename}\")\n\n    # Opening JSON file\n    with open(ANNOTATIONS_DIR + '/' + json_filename) as json_file:\n        data = json.load(json_file , encoding='utf-8')\n\n    images = []\n    classes = []\n    for item in tqdm(data['images']):\n        category = get_class(item, data)\n        if category is not None:\n            images, classes = convert_images(IMG_BASE_DIR, item, category, images, classes, plot=False) \n        #break\n\n    return np.asarray(images), np.asarray(classes)\n\n# Extracting data for training and testing\nX_train, y_train = convert_dataset('annotations_train.json')\nX_test, y_test = convert_dataset('annotations_val.json')","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:25:52.390227Z","iopub.execute_input":"2021-11-07T11:25:52.390806Z","iopub.status.idle":"2021-11-07T11:26:10.299018Z","shell.execute_reply.started":"2021-11-07T11:25:52.390767Z","shell.execute_reply":"2021-11-07T11:26:10.298252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shuffling the training set\nfrom sklearn.utils import shuffle\nX_train, y_train = shuffle(X_train, y_train, random_state=111)\nX_train.shape, y_train.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:26:10.300401Z","iopub.execute_input":"2021-11-07T11:26:10.300681Z","iopub.status.idle":"2021-11-07T11:26:11.829160Z","shell.execute_reply.started":"2021-11-07T11:26:10.300643Z","shell.execute_reply":"2021-11-07T11:26:11.828389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Setting up the model evaluator","metadata":{}},{"cell_type":"code","source":"# Imports\nimport tensorflow as tf\nimport tensorflow_datasets as tfds\n\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport inspect\nfrom tqdm import tqdm\n\n# Set batch size for training and validation\nbatch_size = 32","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:26:11.831163Z","iopub.execute_input":"2021-11-07T11:26:11.831504Z","iopub.status.idle":"2021-11-07T11:26:16.646079Z","shell.execute_reply.started":"2021-11-07T11:26:11.831465Z","shell.execute_reply":"2021-11-07T11:26:16.645325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a list of pre-trained available models in keras\nmodel_dictionary = {m[0]:m[1] for m in inspect.getmembers(tf.keras.applications, inspect.isfunction)}","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:26:16.647515Z","iopub.execute_input":"2021-11-07T11:26:16.647837Z","iopub.status.idle":"2021-11-07T11:26:17.438637Z","shell.execute_reply.started":"2021-11-07T11:26:16.647791Z","shell.execute_reply":"2021-11-07T11:26:17.437868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Setting up some parameters\nnum_train = X_train.shape[0]\nnum_validation = X_test.shape[0]\nnum_classes = 3\nnum_iterations = int(num_train/batch_size)\n\ndef get_joined_y(y_hot):\n    #TODO: Probably can be joined with one single line\n    y_joined = []\n    for i in range(0,y_hot.shape[0],6):\n        y_joined.append( np.argmax(np.sum(y_hot[i:i+6],axis=0)) )\n\n    return np.asarray(y_joined)\n\n# One hot enconding ys\ny_train_hot = np.zeros((y_train.shape[0], num_classes))\nfor i in range(y_train.shape[0]):\n    y_train_hot[i,:] = tf.one_hot(y_train[i]-1, depth=num_classes)\n\ny_test_hot = np.zeros((y_test.shape[0], num_classes))\nfor i in range(y_test.shape[0]):\n    y_test_hot[i,:] = tf.one_hot(y_test[i]-1, depth=num_classes)   \n    \n# Check one-hot encoding correctness\nfor i in range(y_train.shape[0]):\n    #print(y_test[i], y_test_hot[i])\n    assert(y_train_hot[i][y_train[i]-1]==1)\n    \nfor i in range(y_test.shape[0]):\n    #print(y_test[i], y_test_hot[i])\n    assert(y_test_hot[i][y_test[i]-1]==1)","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:26:17.440088Z","iopub.execute_input":"2021-11-07T11:26:17.440364Z","iopub.status.idle":"2021-11-07T11:26:21.226408Z","shell.execute_reply.started":"2021-11-07T11:26:17.440328Z","shell.execute_reply":"2021-11-07T11:26:21.225661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Evaluating on each available model","metadata":{}},{"cell_type":"code","source":"# Loop over each model available in Keras\nimport gc\ny_join = get_joined_y(y_test_hot)\n\nmodel_benchmarks = {'model_name': [], 'num_model_params': [], 'validation_accuracy': [], 'join_6_img_val_acc': []}\nfor model_name, model in tqdm(model_dictionary.items()):\n    print(f\"Evaluating on {model_name}\")\n    # Special handling for \"NASNetLarge\" since it requires input images with size (331,331)\n    if 'NASNetLarge' in model_name:\n        continue\n        #input_shape=(331,331,3)\n        #train_processed = train_processed_331\n        #validation_processed = validation_processed_331\n    #else:\n    #    input_shape=(224,224,3)\n    #    train_processed = train_processed_224\n    #    validation_processed = validation_processed_224\n\n    input_shape=(224,224,3)\n    # load the pre-trained model with global average pooling as the last layer and freeze the model weights\n    pre_trained_model = model(include_top=False, pooling='avg', input_shape=input_shape)\n    pre_trained_model.trainable = False\n\n    # custom modifications on top of pre-trained model\n    clf_model = tf.keras.models.Sequential()\n    clf_model.add(pre_trained_model)\n    clf_model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))\n    clf_model.compile(loss='categorical_crossentropy', metrics=['accuracy'])\n    history = clf_model.fit(X_train, y_train_hot, epochs=3, validation_data=(X_test, y_test_hot), steps_per_epoch=num_iterations)\n\n    predictions = clf_model.predict(X_test)\n    joined_predictions = get_joined_y(predictions)\n    \n    total_acc = joined_predictions[joined_predictions==y_join].shape[0] / joined_predictions.shape[0]\n    print(f\"Total Accuracy: {total_acc}\")\n\n    # Calculate all relevant metrics\n    model_benchmarks['model_name'].append(model_name)\n    model_benchmarks['num_model_params'].append(pre_trained_model.count_params())\n    model_benchmarks['validation_accuracy'].append(history.history['val_accuracy'][-1])\n    model_benchmarks['join_6_img_val_acc'].append(total_acc)\n\n    del pre_trained_model, clf_model\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:26:21.227615Z","iopub.execute_input":"2021-11-07T11:26:21.227902Z","iopub.status.idle":"2021-11-07T11:53:32.282875Z","shell.execute_reply.started":"2021-11-07T11:26:21.227866Z","shell.execute_reply":"2021-11-07T11:53:32.280595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluating the results","metadata":{}},{"cell_type":"code","source":"# Convert Results to DataFrame for easy viewing\nbenchmark_df = pd.DataFrame(model_benchmarks)\nbenchmark_df.sort_values('num_model_params', inplace=True) # sort in ascending order of num_model_params column\nbenchmark_df.to_csv('benchmark_df.csv', index=False) # write results to csv file\nprint(\"Sorting the benchmark by number parameters\")\nbenchmark_df","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:53:32.292589Z","iopub.execute_input":"2021-11-07T11:53:32.293761Z","iopub.status.idle":"2021-11-07T11:53:32.374216Z","shell.execute_reply.started":"2021-11-07T11:53:32.293721Z","shell.execute_reply":"2021-11-07T11:53:32.373448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Plotting the results","metadata":{}},{"cell_type":"code","source":"# Loop over each row and plot the num_model_params vs validation_accuracy\nmarkers=[\".\",\",\",\"o\",\"v\",\"^\",\"<\",\">\",\"1\",\"2\",\"3\",\"4\",\"8\",\"s\",\"p\",\"P\",\"*\",\"h\",\"H\",\"+\",\"x\",\"X\",\"D\",\"d\",\"|\",\"_\",4,5,6,7,8,9,10,11]\nplt.figure(figsize=(7,5))\nfor row in benchmark_df.itertuples():\n    plt.scatter(row.num_model_params, row.join_6_img_val_acc, label=row.model_name, marker=markers[row.Index], s=150, linewidths=2)\nplt.xscale('log')\nplt.xlabel('Number of Parameters in Model')\nplt.ylabel('Validation Accuracy after 3 Epochs')\nplt.title('Accuracy vs Model Size')\nplt.legend(bbox_to_anchor=(1, 1), loc='upper left'); # Move legend out of the plot","metadata":{"execution":{"iopub.status.busy":"2021-11-07T11:53:32.375580Z","iopub.execute_input":"2021-11-07T11:53:32.375832Z","iopub.status.idle":"2021-11-07T11:53:33.342388Z","shell.execute_reply.started":"2021-11-07T11:53:32.375796Z","shell.execute_reply":"2021-11-07T11:53:33.339616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Printing benchmark by score\")\nbenchmark_df.sort_values('join_6_img_val_acc')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\n\nReading the results I have 2 main conclusions:\n- `MobileNetv2` seems a good candidate and `ResNet50V2` having 10x parameters relative to `MobileNetv2` can be a good candidate as well. Maybe an ensembling of the top models can be a good approach.\n- These results are obtained with no data augmentation more than extracting 6 images per sample and only trained in 3 epochs. Probably scores can be improved.\n- I haven't tested training with images without splitting into 6. Maybe it gets good or better results.\n\nHappy to know what you think!!","metadata":{}}]}