{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport imageio\nfrom tensorflow import keras\nfrom tqdm import tqdm\nimport glob\nimport os\nimport matplotlib.pyplot as plt\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Welcome to Recursion Cellular Image Classification competition**.\n<br>This starter kernel will guide you though our data and show you how to train a basic model."},{"metadata":{},"cell_type":"markdown","source":"## Loading the metadata"},{"metadata":{},"cell_type":"markdown","source":"![](http://)Let's start by reading train.csv:"},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = '../input'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(BASE_DIR, 'train.csv'))\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* As you can see, our train data is composed of 36515 samples from 33 different experiments\n* In each experiments we had 4 different plates, each plate contained 384 wells, not all of them are in the dataset\n* Each well has **2 samples/sites**, both marked with a single siRNA"},{"metadata":{},"cell_type":"markdown","source":"We want the starter kernel to run fast, hence, we sample only 5000 items from the dataset. When Training your model, we recommend using all samples."},{"metadata":{},"cell_type":"markdown","source":"## Pixel metadata"},{"metadata":{},"cell_type":"markdown","source":"1. We also provided statistics on all the images. This information will allow you to normalize the data, for example by reducing the mean and deviding by the standard deviation."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_pixel_stats = pd.read_csv(os.path.join(BASE_DIR, 'pixel_stats.csv')).set_index(['id_code','site', 'channel'])\ndf_pixel_stats.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Converting format"},{"metadata":{},"cell_type":"markdown","source":"On Kaggle Kernels we provided you with extracted png images, each representing a layer. If you try to load it to memory, you might encounter some IO bollteneck. This script will convert the pngs to numpy array, each representing a single sample. in the GCS extended dataset you will be able to find tfrecords files, which are a fast way to load data to Tensorflow models."},{"metadata":{"trusted":true},"cell_type":"code","source":"OUTPUT_DIR = '../output'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH_FORMAT = os.path.join(BASE_DIR, 'train/{experiment}/Plate{plate}/{well}_s{sample}_w{channel}.png')\n\ndef transform_image(sample_data, pixel_data):\n    x=[]\n    for channel in [1,2,3,4,5,6]:\n        impath = DATA_PATH_FORMAT.format(experiment=sample.experiment,\n                                        plate=sample_data.plate,\n                                        well=sample_data.well,\n                                        sample=1,# For demo only, we use sample=1, you can use also sample=2\n                                        channel=channel)\n        # normalize the channel\n        img = np.array(imageio.imread(impath)).astype(np.float64)\n        img -= pixel_data.loc[channel]['mean']\n        img /= pixel_data.loc[channel]['std']\n        img *= 255 # To keep MSB\n        \n        x.append(img)\n\n    return np.stack(x).T.astype(np.byte)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir -p {OUTPUT_DIR}/np_arrays/train/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for _, sample in tqdm(df_train.iterrows(), total=len(df_train)):\n    pixel_data = df_pixel_stats.loc[sample.id_code, 1, :].reset_index().set_index('channel')\n    x = transform_image(sample, pixel_data)\n    np.save(os.path.join(OUTPUT_DIR, 'np_arrays/train/{sample_id}.npy').format(sample_id=sample.id_code), x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Controls"},{"metadata":{},"cell_type":"markdown","source":"In each experiment, the same 30 siRNAs appear on every plate as positive controls. In addition, there is one well per plate with untreated cells as a negative control. It has the same schema as [train/test].csv, plus a well_type field denoting the type of control.\n"},{"metadata":{},"cell_type":"markdown","source":"## Train a simple model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# base sample : https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly\n\nclass DataGenerator(keras.utils.Sequence):\n    \n    def __init__(self, df, mode='train', batch_size=32, dim=(512,512), n_channels=6,  shuffle=True):\n        self.df = df\n        self.dim = dim\n        self.batch_size = batch_size\n        self.mode = mode\n        if mode == 'train':\n            self.labels = self.df['sirna'].tolist()\n            self.n_classes = self.df['sirna'].nunique()\n        self.list_IDs = self.df.index.tolist()\n        self.n_channels = n_channels\n    \n        self.shuffle = shuffle\n        if mode == 'train':\n            self.npy_data_format =  os.path.join(OUTPUT_DIR,'np_arrays/train/{sample_id}.npy')\n        elif mode == 'test':\n            self.npy_data_format = os.path.join(OUTPUT_DIR,'np_arrays/test/{sample_id}.npy')\n        \n        self.on_epoch_end()\n\n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        return int(np.floor(len(self.list_IDs) / self.batch_size))\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        # Generate indexes of the batch\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n\n        # Find list of IDs\n        list_IDs_temp = [self.list_IDs[k] for k in indexes]\n        # Generate data\n        X, y = self.__data_generation(list_IDs_temp)\n\n        return X, y\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange(len(self.list_IDs))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __data_generation(self, list_IDs_temp):\n        # Initialization\n        X = np.empty((self.batch_size, *self.dim, self.n_channels))\n        y = np.empty((self.batch_size), dtype=int)\n\n        # Generate data\n        for i, ID in enumerate(list_IDs_temp):\n            \n            # Store sample\n            sample_data = self.df.loc[ID]\n            X[i,] = np.load(self.npy_data_format.format(sample_id=sample_data.id_code))\\\n                                            .astype(np.float32) / 255.0\n            if self.mode == 'train':\n            # Store class\n                y[i] = sample_data.sirna\n            else:\n                y[i] = 0\n          \n        if self.mode == 'train':\n            return X, keras.utils.to_categorical(y, num_classes=self.n_classes)\n        else:\n            return X, y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generators\ntraining_generator = DataGenerator(df=df_train, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"n_classes = df_train['sirna'].nunique()\n\nmodel = keras.Sequential([\n    keras.layers.Conv2D(32, (3, 3), padding='same',\n                 input_shape=(512, 512, 6), activation='relu'),\n    keras.layers.Conv2D(32, (3, 3), activation='relu'),\n    keras.layers.MaxPooling2D(pool_size=(2, 2)),\n    keras.layers.Conv2D(64, (3, 3), padding='same', activation='relu'),\n    keras.layers.MaxPooling2D(pool_size=(2, 2)),\n    keras.layers.Flatten(),\n    keras.layers.Dense(128, activation='relu'),\n    keras.layers.Dense(128, activation='relu'),\n    keras.layers.Dense(n_classes, activation='softmax')\n])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='nadam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train model on dataset\nmodel.fit_generator(epochs=1, \n                    generator=training_generator,\n                    use_multiprocessing=True,\n                    workers=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clean train data\n!rm -rf {OUTPUT_DIR}/np_arrays/train/\n!mkdir -p {OUTPUT_DIR}/np_arrays/test/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv( os.path.join(BASE_DIR, 'test.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH_FORMAT = os.path.join(BASE_DIR, 'test/{experiment}/Plate{plate}/{well}_s{sample}_w{channel}.png')\n\nfor _, sample in tqdm(df_test.iterrows(), total=len(df_test)):\n    pixel_data = df_pixel_stats.loc[sample.id_code, 1, :].reset_index().set_index('channel')\n    x = transform_image(sample, pixel_data)\n    np.save(os.path.join(OUTPUT_DIR, 'np_arrays/test/{sample_id}.npy').format(sample_id=sample.id_code), x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_generator = DataGenerator(df=df_test, mode='test', shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict_generator(test_generator, steps=1000, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf {OUTPUT_DIR}/np_arrays/test/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classes = predictions.argmax(axis=1)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.3"}},"nbformat":4,"nbformat_minor":1}