{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Summary\n\nI believe this notebook can be your comprehensive guide for data preprocessing for this competition. Here we cover:\n1. Removing duplicates\n2. Label formatting\n3. Making stratified folds\n4. Data pre-augmentation\n5. Making TFRecords\n\nThis is also the very beginning of **TensorFlow** and **Keras** implementation of training loop for **Plant Pathology 2021** competition optimized for achieving maximum speed.\n\n### Also check out:\n1. The **[Training Notebook](https://www.kaggle.com/nickuzmenkov/pp2021-tpu-tf-training)** where we train the EfficientNetB4 ensemble.\n2. The **[Inference Notebook](https://www.kaggle.com/nickuzmenkov/pp2021-tpu-tf-inference)**."},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import MultiLabelBinarizer\nfrom sklearn.model_selection import StratifiedKFold\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport albumentations\nimport pandas as pd\nimport numpy as np\nimport shutil\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Configuration\nMake changes here to customize the entire notebook"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"plt.style.use('fivethirtyeight')\nprint(f'Using TensorFlow {tf.__version__}')\n\nclass CFG:\n    \n    '''\n    keep these\n    '''\n    root = '../input/plant-pathology-2021-fgvc8/train_images'\n    classes = [\n        'complex', \n        'frog_eye_leaf_spot', \n        'powdery_mildew', \n        'rust', \n        'scab',\n        'healthy']\n    strategy = tf.distribute.get_strategy()\n    batch_size = 16\n    \n    '''\n    tune these\n    '''\n    img_size = 600 # image size\n    folds = 5 # number of KFold n_splits\n    seed = 42 # random seed (only for KFold)\n    subfolds = 16 # number of .tfrec files in each fold\n    transform = True # whether to apply pre-augmentations or not\n    epochs = 5 # (>=5) number of pre-augmented dataset copies to save when transform = True\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Removing duplicates\nWe use `duplicates.csv` file containing 50 sequences of duplicates found with `image_hash` in my **[other notebook](https://www.kaggle.com/nickuzmenkov/pp2021-duplicates-revealing)**. Here for each duplicate sequence:\n1. Leave only one sample if all duplicates share the same labels\n2. Delete all duplicates if at least one of them is labeled differently"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv', index_col='image')\ninit_len = len(df)\n\nwith open('../input/pp2021-duplicates-revealing/duplicates.csv', 'r') as file:\n    duplicates = [x.strip().split(',') for x in file.readlines()]\n\nfor row in duplicates:\n    unique_labels = df.loc[row].drop_duplicates().values\n    if len(unique_labels) == 1:\n        df = df.drop(row[1:], axis=0)\n    else:\n        df = df.drop(row, axis=0)\n        \nprint(f'Dropping {init_len - len(df)} duplicate samples.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Label formatting\nThe initial format of space-separated string labels is inapplicable for model training. Here we change the format via `MultiLabelBinarizer` instance"},{"metadata":{"trusted":true},"cell_type":"code","source":"original_labels = df['labels'].values.copy()\n\ndf['labels'] = [x.split(' ') for x in df['labels']]\nlabels = MultiLabelBinarizer(classes=CFG.classes).fit_transform(df['labels'].values)\n\ndf = pd.DataFrame(columns=CFG.classes, data=labels, index=df.index)\n\ndf.to_csv('train.csv')\ndisplay(df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's better!\n## 3. Making stratified folds\nWe have some pretty rare classes here (e.g. `rust`) which can be completely lost when doing random `KFold` splits or `train_test_split`. So, we need stratification. \n\n**Sklearn**'s `StratifiedKFold` is only applicable for **multi-class** classification tasks (and current task is **multi-label**), so we simply treat the original mulit-labels as one-hot labels when applying `StratifiedKFold` and thus making label-wise stratification implicitly."},{"metadata":{"trusted":true},"cell_type":"code","source":"kfold = StratifiedKFold(n_splits=CFG.folds, shuffle=True, random_state=CFG.seed)\nfold = np.zeros((len(df),))\n\nfor i, (train_index, val_index) in enumerate(kfold.split(df.index, original_labels)):\n    fold[val_index] = i\n\nvalue_counts = lambda x: pd.Series.value_counts(x, normalize=True)\n\ndf_occurence = pd.DataFrame({\n    'origin': df.apply(value_counts).loc[1],\n    'fold_0': df[fold == 0].apply(value_counts).loc[1],\n    'fold_1': df[fold == 1].apply(value_counts).loc[1],\n    'fold_2': df[fold == 2].apply(value_counts).loc[1],\n    'fold_3': df[fold == 3].apply(value_counts).loc[1],\n    'fold_4': df[fold == 4].apply(value_counts).loc[1]})\n\nbar = df_occurence.plot.barh(figsize=[15, 5], colormap='plasma')\n\nfolds = pd.DataFrame({\n    'image': df.index,\n    'fold': fold})\n\nfolds.to_csv('folds.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Perfect! Label distributions are almost equal across all folds.\n## 4. Data pre-augmentation\nFancy data augmentation is a big pain for every **TensorFlow** user, since most of them (e.g. `CutOut`, `CLAHE`, and even random rotation (!!!) ) are implemented only for **Keras** `ImageDataGenerator` class which is super slow (of course, comparing to `tf.data.Dataset` class) or not implemented at all.\n\nSo we have to choose between rapid training on TPU with TFRecords and optimized `tf.data.Dataset` class and fancy augmentations. Or, of course, you can hard-code all those, but just look at how easy it is with `albumentations` library:"},{"metadata":{"trusted":true},"cell_type":"code","source":"if CFG.transform:\n    transform = albumentations.Compose([\n       albumentations.RandomResizedCrop(CFG.img_size, CFG.img_size, scale=(0.9, 1), p=1), \n       albumentations.HorizontalFlip(p=0.5),\n       albumentations.VerticalFlip(p=0.5),\n       albumentations.ShiftScaleRotate(p=0.5),\n       albumentations.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=10, val_shift_limit=10, p=0.7),\n       albumentations.RandomBrightnessContrast(brightness_limit=(-0.2,0.2), contrast_limit=(-0.2, 0.2), p=0.7),\n       albumentations.CLAHE(clip_limit=(1,4), p=0.5),\n       albumentations.OneOf([\n           albumentations.OpticalDistortion(distort_limit=1.0),\n           albumentations.GridDistortion(num_steps=5, distort_limit=1.),\n           albumentations.ElasticTransform(alpha=3),\n       ], p=0.2),\n       albumentations.OneOf([\n           albumentations.GaussNoise(var_limit=[10, 50]),\n           albumentations.GaussianBlur(),\n           albumentations.MotionBlur(),\n           albumentations.MedianBlur(),\n       ], p=0.2),\n      albumentations.Resize(CFG.img_size, CFG.img_size),\n      albumentations.OneOf([\n          albumentations.JpegCompression(),\n          albumentations.Downscale(scale_min=0.1, scale_max=0.15),\n      ], p=0.2),\n      albumentations.IAAPiecewiseAffine(p=0.2),\n      albumentations.IAASharpen(p=0.2),\n      albumentations.Cutout(max_h_size=int(CFG.img_size * 0.1), max_w_size=int(CFG.img_size * 0.1), num_holes=5, p=0.5),\n    ])\nelse:\n    transform = None","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All those images are actually the same image transformed by `albumentations`"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"figure, axes = plt.subplots(5, 5, figsize=[15, 15])\naxes = axes.reshape(-1,)\n\nif transform is None:\n    for i in range(len(axes)):\n        image = tf.io.read_file(os.path.join(CFG.root, df.index[i]))\n        image = tf.image.decode_jpeg(image, channels=3)\n        image = tf.image.resize(image, [CFG.img_size, CFG.img_size])\n        image = tf.cast(image, tf.uint8)\n        \n        axes[i].imshow(image.numpy())\n        axes[i].axis('off')\n\nelse:\n    image = tf.io.read_file(os.path.join(CFG.root, df.index[CFG.seed]))\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.image.resize(image, [CFG.img_size, CFG.img_size])\n    image = tf.cast(image, tf.uint8)\n\n    for i in range(len(axes)):\n        axes[i].imshow(transform(image=image.numpy())['image'])\n        axes[i].axis('off')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As mentioned by @calintimbus, we can still use the power of `albumentations` inside of `tensorflow` pipelines via `tf.py_function(func=composed_transform, inp=[image], Tout=(tf.float32))`. But the time required for downscaling really large images and applying augmentations from a relatively long list can severely slow donw the training process. If you decide to go this way, set the `transform` attribute of the `CFG` class (see the **Configurations** block at the very beginning) to `False` and apply your own augmentations on the fly. Otherwise, if you decide to save the pre-augmented dataset as a one-time payment, set it to `True`.\n## 5. Making TFRecords\nUsing `.tfrec` files instead of tensor slices can result in dramatic performance boost. Here we finish by serializing all the past work to `.tfrec` format.\n### Helper functions (serialization)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def _serialize_image(path, transform=None):\n    image = tf.io.read_file(path)\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.image.resize(image, [CFG.img_size, CFG.img_size])\n    image = tf.cast(image, tf.uint8)\n    \n    if transform is not None:\n        image = transform(image=image.numpy())['image']\n        \n    return tf.image.encode_jpeg(image).numpy()\n\n\ndef _serialize_sample(image, image_name, label):\n    feature = {\n        'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image])),\n        'image_name': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image_name])),\n        'complex': tf.train.Feature(int64_list=tf.train.Int64List(value=[label[0]])),\n        'frog_eye_leaf_spot': tf.train.Feature(int64_list=tf.train.Int64List(value=[label[1]])),\n        'powdery_mildew': tf.train.Feature(int64_list=tf.train.Int64List(value=[label[2]])),\n        'rust': tf.train.Feature(int64_list=tf.train.Int64List(value=[label[3]])),\n        'scab': tf.train.Feature(int64_list=tf.train.Int64List(value=[label[4]])),\n        'healthy': tf.train.Feature(int64_list=tf.train.Int64List(value=[label[5]]))}\n    sample = tf.train.Example(features=tf.train.Features(feature=feature))\n    return sample.SerializeToString()\n\n\ndef serialize_fold(fold, name, transform=None, bar=None):\n    samples = []\n    \n    for image_name, labels in fold.iterrows():\n        path = os.path.join(CFG.root, image_name)\n        image = _serialize_image(path, transform=transform)\n        samples.append(_serialize_sample(image, image_name.encode(), labels))\n    \n    with tf.io.TFRecordWriter(name + '.tfrec') as writer:\n        [writer.write(x) for x in samples]\n        \n    if bar is not None:\n        bar.update(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total = CFG.folds * CFG.subfolds if transform is None else CFG.folds * CFG.subfolds * CFG.epochs\n\nwith tqdm(total=total) as bar:\n\n    for i in range(CFG.folds):\n\n        df_fold = df[fold == i]\n        \n        folder = f'fold_{i}'\n        \n        try:\n            os.mkdir(folder)\n        except FileExistsError:\n            shutil.rmtree(folder)\n            os.mkdir(folder)\n        \n        if transform is None:\n            for k, subfold in enumerate(np.array_split(df_fold, CFG.subfolds)):\n                name=os.path.join(folder, '%.2i-%.3i' % (k, len(subfold)))\n                serialize_fold(subfold, name=name, bar=bar)\n        else:\n            for j in range(CFG.epochs):\n                for k, subfold in enumerate(np.array_split(df_fold, CFG.subfolds)):\n                    name=os.path.join(folder, '%.2i-%.3i' % (j * CFG.subfolds + k, len(subfold)))\n                    serialize_fold(subfold, name=name, transform=transform, bar=bar)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Run test\nLastly, we test our tfrecords and preprocessed data by training a small model with one `.tfrec` file for just one epoch. Feel free to replace this part with your training pipeline.\n### Helper functions (parsing & testing)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"feature_map = {\n    'image': tf.io.FixedLenFeature([], tf.string),\n    'image_name': tf.io.FixedLenFeature([], tf.string),\n    'complex': tf.io.FixedLenFeature([], tf.int64),\n    'frog_eye_leaf_spot': tf.io.FixedLenFeature([], tf.int64),\n    'powdery_mildew': tf.io.FixedLenFeature([], tf.int64),\n    'rust': tf.io.FixedLenFeature([], tf.int64),\n    'scab': tf.io.FixedLenFeature([], tf.int64),\n    'healthy': tf.io.FixedLenFeature([], tf.int64)}\n\n\ndef count_data_items(filenames):\n    return np.sum([int(x[:-6].split('-')[-1]) for x in filenames])\n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.reshape(image, [CFG.img_size, CFG.img_size, 3])\n    image = tf.cast(image, tf.float32) / 255.\n    return image\n\n\ndef read_tfrecord(example):\n    example = tf.io.parse_single_example(example, feature_map)\n    image = decode_image(example['image'])\n    target = [\n        tf.cast(example['complex'], tf.float32),\n        tf.cast(example['frog_eye_leaf_spot'], tf.float32),\n        tf.cast(example['healthy'], tf.float32),\n        tf.cast(example['powdery_mildew'], tf.float32),\n        tf.cast(example['rust'], tf.float32),\n        tf.cast(example['scab'], tf.float32)]\n    return image, target\n\n\ndef get_dataset(filenames):\n    auto = tf.data.experimental.AUTOTUNE\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=auto)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=auto)\n    dataset = dataset.batch(CFG.batch_size)\n    dataset = dataset.prefetch(auto)\n    return CFG.strategy.experimental_distribute_dataset(dataset)\n\n\ndef get_model():\n    model = tf.keras.models.Sequential([\n        tf.keras.applications.EfficientNetB0(\n            include_top=False,\n            input_shape=(CFG.img_size, CFG.img_size, 3),\n            weights=None,\n            pooling='avg'),\n        tf.keras.layers.Dense(len(feature_map) - 2),\n        tf.keras.layers.Activation('sigmoid', dtype='float32')\n    ], name='EfficientNetB0')\n    \n    model.compile(\n        optimizer='adam',\n        loss='categorical_crossentropy',\n        metrics=['accuracy'])\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames = tf.io.gfile.glob('./fold_0/*.tfrec')[:1]\ndataset = get_dataset(filenames)\n\nsteps_per_epoch = count_data_items(filenames) // CFG.batch_size\n\nwith CFG.strategy.scope():\n    model = get_model()\n\nmodel.summary()\n\nhistory = model.fit(\n    dataset, \n    steps_per_epoch=steps_per_epoch,\n    epochs=1,\n    verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Acknowledgements\n\n* the list of albumentations is taken from @underwearfitting **[notebook](https://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965)** with only minor changes."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}