{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## barebones efficientnet HPA classification training\n\n## credits to:\n https://www.kaggle.com/samusram/hpa-rgb-model-rgby-cell-level-classification/\n\nfor the generator code\n\nhttps://www.kaggle.com/glopezzz/segmentation-and-classification-with-efficientnet\n\nfor efficientnet training code\n\nfuture things:\n2 models - r+g (only microtubules) and r+b (only nucleus) - would have to re-code the labels \n \nif we decide to use the current image size (512x512) we can download the resized images locally and have less preprocessing during training. This might also have the added benefit of having augmented images in the same folder for better access during training. \n ","metadata":{}},{"cell_type":"markdown","source":"currently, the model seems to always predict the vector of all zeros - is this a product of the label matrix being sparse??? so predicting all 0 is technically pretty \"accurate\".\n\nthis can very well be that I really didn't train the model all that much.\nEDIT 0419: yeah. it wasn't actually trained. accuracy still quite bad, but at least it's giving 1s and 0s.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n\n# Input data files are available in the read-only \"../input/\" directory\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### install. segmentations and tf_explain only comes into play during \"submission\" phase so we leave it out for now. Unless later on we want to segment before training.","metadata":{}},{"cell_type":"code","source":"!pip install \"../input/keras-application/Keras_Applications-1.0.8-py3-none-any.whl\"\n!pip install \"../input/efficientnet111/efficientnet-1.1.1-py3-none-any.whl\"\n# !pip install \"../input/pycocotools/pycocotools-2.0-cp37-cp37m-linux_x86_64.whl\"\n# !pip install \"../input/hpapytorchzoozip/pytorch_zoo-master\"\n# !pip install \"../input/hpacellsegmentatormaster/HPA-Cell-Segmentation-master\"\n# !pip install \"../input/tfexplainforoffline/tf_explain-0.2.1-py3-none-any.whl\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### import. Some are probably redundant for this run ","metadata":{}},{"cell_type":"code","source":"import os, glob\nimport tensorflow as tf\ngpus = tf.config.experimental.list_physical_devices('GPU')\nfor gpu in gpus:\n    tf.config.experimental.set_memory_growth(gpu, True)\n# tf.compat.v1.disable_eager_execution()\nfrom copy import deepcopy\nimport random\nfrom sklearn.model_selection import train_test_split\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport keras\nimport keras.backend as K\nfrom keras.optimizers import Adam\nfrom keras.callbacks import Callback\n# note from OP: please note, that locally I've trained a keras.efficientnet model, but using tensorflow.keras.applications.EfficientNetB0 should lead to the same results\nfrom efficientnet.keras import EfficientNetB0\nfrom keras.layers import Dense, Flatten\nfrom keras.models import Model, load_model\nfrom keras.utils import Sequence\nfrom albumentations import Compose, VerticalFlip, HorizontalFlip, Rotate, GridDistortion\nimport matplotlib.pyplot as plt\nfrom IPython.display import Image, display\nfrom numpy.random import seed\nseed(10)\nfrom tensorflow.python.framework import ops\nimport gc\nfrom tqdm.auto import tqdm\nimport base64\nimport numpy as np\n# from pycocotools import _mask as coco_mask\nimport typing as t\nimport warnings\nwarnings.filterwarnings('ignore')\ntf.random.set_seed(10)\n%matplotlib inline","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### setup","metadata":{}},{"cell_type":"code","source":"TEST_IMGS_FOLDER = '../input/hpa-single-cell-image-classification/test/'\nTRAIN_IMGS_FOLDER = '../input/hpa-single-cell-image-classification/train/'\nIMG_HEIGHT = IMG_WIDTH = 512\nBATCH_SIZE = 32\n\n# internet must be enabled\nDOWNLOAD_PRETRAINED_WEIGHTS = True\n\ntrain_df = pd.read_csv('../input/hpa-single-cell-image-classification/train.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### splarghing with the class names","metadata":{}},{"cell_type":"code","source":"# from https://www.kaggle.com/c/hpa-single-cell-image-classification/data\n\nspecified_class_names = \"\"\"0. Nucleoplasm\n1. Nuclear membrane\n2. Nucleoli\n3. Nucleoli fibrillar center\n4. Nuclear speckles\n5. Nuclear bodies\n6. Endoplasmic reticulum\n7. Golgi apparatus\n8. Intermediate filaments\n9. Actin filaments \n10. Microtubules\n11. Mitotic spindle\n12. Centrosome\n13. Plasma membrane\n14. Mitochondria\n15. Aggresome\n16. Cytosol\n17. Vesicles and punctate cytosolic patterns\n18. Negative\"\"\"\n\nclass_names = [class_name.split('. ')[1] for class_name in specified_class_names.split('\\n')]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['Label'] = train_df['Label'].map(lambda x: set(map(int, x.split('|'))))\nfor class_i, class_name in enumerate(class_names):\n    train_df[class_name] = train_df['Label'].map(lambda x: 1 if class_i in x else 0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dictionary for image ID --> ohe vectors\nid_2_ohe_vector = {img: vec for img, vec in zip(train_df['ID'], train_df.iloc[:, 2:-1].values)}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### separate the images with unique label combinations for training","metadata":{}},{"cell_type":"code","source":"label_combinations = train_df['Label'].map(lambda x: str(sorted(list(x))))\nprint(\"There {} images with unique label combinations \".format(sum(label_combinations.value_counts()==1)))\nlabel_combinations_counts = label_combinations.value_counts()\nunique_label_combination = label_combinations_counts.index[label_combinations_counts==1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ids_unique_label_comb = train_df['ID'][train_df['Label'].map(lambda x:str(sorted(list(x))) in unique_label_combination)]\nnon_unique_label_comb_bool_idx = train_df['Label'].map(lambda x:str(sorted(list(x))) not in unique_label_combination)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ids, val_ids = train_test_split(train_df['ID'][non_unique_label_comb_bool_idx].values,\n                        test_size = 0.2,\n                        stratify = label_combinations[non_unique_label_comb_bool_idx],\n                        random_state = 42)\ntrain_ids = np.concatenate((train_ids, train_ids_unique_label_comb))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_ids),len(val_ids))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## data is too large to fit into memory, so we need a generator","metadata":{}},{"cell_type":"markdown","source":"heavily borrowed from kaggle user samusram's code. Most major change is replacing the num_channels argument with use_colors, a string of colors, eg \"rgb\", \"rgby\" etc. \nWould need to make sure that in future segmentation and cell-level classification, the channels are loaded in the same order. \n\nCurrently, the model in this notebook is 3-channel, so it defaults to rgb. Any other channel count will require remaking the model\n\nalso need to test if shuffle works since it was taken without editing","metadata":{}},{"cell_type":"code","source":"class DataGenerator(Sequence):\n    def __init__(self, id_list, id_2_ohe_vector=id_2_ohe_vector, folder_imgs=TRAIN_IMGS_FOLDER, \n                 batch_size=BATCH_SIZE, shuffle=True,\n                 resized_height=IMG_HEIGHT, resized_width=IMG_WIDTH, use_colors = \"rgb\"):\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.id_list = deepcopy(id_list)\n        self.folder_imgs = folder_imgs\n        self.len = len(self.id_list) // self.batch_size\n        self.resized_height = resized_height\n        self.resized_width = resized_width\n        self.id_2_ohe_vector = id_2_ohe_vector\n        self.is_test = not 'train' in folder_imgs\n        self.use_colors = use_colors\n        self.num_channels = len(use_colors)\n        if not self.is_test:       \n            self.num_classes = len(next(iter(id_2_ohe_vector.values())))\n        if not shuffle and not self.is_test:\n            self.labels = [id_2_ohe_vector[img] for img in self.id_list[:self.len*self.batch_size]]\n\n    def __len__(self):\n        return self.len\n    \n    def on_epoch_start(self):\n        if self.shuffle:\n            random.shuffle(self.id_list)\n            \n    # open_image adapted from https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb\n    def open_image(self, image_id): #a function that reads an image with the specified colors\n        \n        color_dict = {\"r\":\"red\",\"g\":\"green\",\"b\":\"blue\",\"y\":\"yellow\"}\n        colors = [color_dict[i] for i in self.use_colors]\n        img = [cv2.imread(os.path.join(self.folder_imgs, f'{image_id}_{color}.png'), cv2.IMREAD_GRAYSCALE)\n               for color in colors]\n        img = np.stack(img, axis=-1)\n        if img.shape[0] == self.resized_height and img.shape[1] == self.resized_width:\n            return img\n        img_resized = cv2.resize(img, (self.resized_height, self.resized_width))\n        # no need to expand dims here \n        \n        return img_resized\n    # btw id_list is the entire batch of IDs! so just pass train_ids or val_ids here. \n    # idx seems to be the batch number, created internally. \n    def __getitem__(self, idx):\n        current_batch = self.id_list[idx * self.batch_size: (idx + 1) * self.batch_size]\n        X = np.empty((self.batch_size, self.resized_height, self.resized_width, self.num_channels))\n\n        if not self.is_test:\n            y = np.empty((self.batch_size, self.num_classes))\n\n        for i, image_id in enumerate(current_batch):\n            img = self.open_image(image_id)\n            X[i, :, :, :] = img.astype(np.float32)/255.0\n            if not self.is_test:\n                y[i, :] = self.id_2_ohe_vector[image_id]\n        if not self.is_test:\n            return X, y\n        return X\n    # an extra function?? maybe only used in testing. \n    def get_labels(self):\n        if self.shuffle:\n            images_current = self.id_list[:self.len*self.batch_size]\n            labels = [self.id_2_ohe_vector[img] for img in images_current]\n        else:\n            labels = self.labels\n        return np.array(labels)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### functions to grab image data from the id -  more for looking at the data purposes. Not used for training anymore.","metadata":{"jupyter":{"outputs_hidden":true}}},{"cell_type":"code","source":"def grab_training_img(img_idx,use_colors):\n    color_dict = {\"r\":\"red\",\"g\":\"green\",\"b\":\"blue\",\"y\":\"yellow\"}\n    image_id = train_ids[img_idx]\n    \n    colors = [color_dict[i] for i in use_colors]\n    \n    img = [cv2.resize(cv2.imread(os.path.join(TRAIN_IMGS_FOLDER, f'{image_id}_{color}.png'), cv2.IMREAD_GRAYSCALE),\n                      (IMG_HEIGHT, IMG_WIDTH))\n            for color in colors]\n    img = np.stack(img, axis=-1)\n    return img, id_2_ohe_vector[image_id]\n\ndef grab_val_img(img_idx,use_colors):\n    color_dict = {\"r\":\"red\",\"g\":\"green\",\"b\":\"blue\",\"y\":\"yellow\"}\n    image_id = val_ids[img_idx]\n    colors = [color_dict[i] for i in use_colors]\n    img = [cv2.resize(cv2.imread(os.path.join(TRAIN_IMGS_FOLDER, f'{image_id}_{color}.png'), cv2.IMREAD_GRAYSCALE),\n                      (IMG_HEIGHT, IMG_WIDTH))\n            for color in colors]\n    img = np.stack(img, axis=-1)\n    return img, id_2_ohe_vector[image_id]\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(grab_val_img(6,\"rgb\")[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# make the model\n## barebones EfficientNet. Plenty of params to tune here including the base model\n### sigmoid was recommended by the internet\n\noutput is 1 less than the number of class names, since if a prediction is 0 for everything, the label would be the \"Negative\" label.","metadata":{}},{"cell_type":"markdown","source":"# EDIT 04/19: trained a model under rgb-effnet-512h5. can be reloaded and trained further.","metadata":{}},{"cell_type":"code","source":"weights_init = 'imagenet' if DOWNLOAD_PRETRAINED_WEIGHTS else None\n\n# make the parts\nimagenet_model = EfficientNetB0(weights=weights_init, include_top=False, pooling='avg',\n                               input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))\nrgb_model_output = Dense(len(class_names) - 1, activation='sigmoid')(imagenet_model.output)\n\n# make the model\nmodel_rgb = Model(inputs=imagenet_model.input, outputs=rgb_model_output)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_rgb.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"some old code. scrapped bc data couldn't fit into memory.","metadata":{}},{"cell_type":"code","source":"# training_data = []\n# for i in range(len(train_ids)):\n#     img,label = grab_training_img(i,\"rgb\")\n#     img = np.expand_dims(img, 0)\n#     label = label.reshape(1,18)\n#     training_data.append([img,label])\n    \n\n# train_ds = tf.data.Dataset.from_tensor_slices(([training_data[i][0] for i in range(len(training_data))], [training_data[i][1] for i in range(len(training_data))]))\n# len(train_ds)\n\n# val_data = []\n# for i in range(val_ids):\n#     img,label = grab_val_img(i,\"rgb\")\n#     img = np.expand_dims(img, 0)\n#     label = label.reshape(1,18)\n#     val_data.append([img,label])\n\n# val_ds = tf.data.Dataset.from_tensor_slices(([val_data[i][0] for i in range(len(val_data))], [val_data[i][1] for i in range(len(val_data))]))\n# len(val_ds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### compile the model with basic suggestions:\nfrom a brief google search, binary_crossentropy is recommended for multi-label since increasing one label's score doesn't penalize others. Can be played around with \n\nmetrics also requires some care, as some of them only check pred == ground truth, hence considering all partially correct classifications to be wrong\n","metadata":{}},{"cell_type":"code","source":"# also grabbed from sources without question. \ntf.keras.backend.clear_session()\n\nearlystopper = tf.keras.callbacks.EarlyStopping(\n    monitor='val_loss', patience=10, verbose=0, mode='min',\n    restore_best_weights=True\n)\n\nmodel_rgb.compile('adam', 'binary_crossentropy', metrics=[tf.keras.metrics.AUC(multi_label=True)])\n# model_rgb.compile('adam', 'binary_crossentropy', metrics=[\"accuracy\"])\n\n# artifact of copypasta - maybe a way to use a cluster?\n#run = wandb.init(entity='usernamehere', project='hpa', job_type='train')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"generate data, then train. \nTakes a while. At least the training initializes and runs, so that's that. ","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 32","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run generator\nmy_training_batch_generator = DataGenerator(train_ids,shuffle=True)\nmy_validation_batch_generator = DataGenerator(val_ids,shuffle=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##### this can be added to the fitter I think:\n#   callbacks=[earlystopper]\n\n# drastically truncated for demonstration. At steps_per_epoch = 500, takes 2-3 hours per epoch. \n# with GPU, 12 mins per epoch with 250 steps.\n\nhist = model_rgb.fit_generator(generator=my_training_batch_generator,\n                               steps_per_epoch = int(len(train_ids)//BATCH_SIZE),\n                               epochs=10,\n                               validation_data = my_validation_batch_generator,\n                               validation_steps = int(len(val_ids)//BATCH_SIZE),\n                               verbose=1\n                )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_rgb.save(\"rgb_effnet_full_train_0420.h5\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"model can then be used for further purposes.","metadata":{}},{"cell_type":"code","source":"look_idx = 123\n\nimg,label = grab_val_img(look_idx,\"rgb\")\npreds= model_rgb.predict(np.expand_dims(img,0))\nprint(label)\nprint(np.round(preds[0],1))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}