{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Summary\n\nThis notebook is 1st step of my work. This notebook is responsible for re-organising, filtering and generating enhanced images in Yolo format for Yolo yaml config.\n\nThis will generate 4 dataset folders, which are as follows\n\n### Dataset is filtered/tranformed and divided into four parts\n- **\"original-no-annot\"** contains original, un-enhanced files of images which **do not have any annotations**\n- **\"original-annot\"** contains original, un-enhanced files of images which **have annotations**\n- **\"enhanced-no-annot\"** contains enhanced files of images which **do not have any annotations**\n- **\"enhanced-annot\"** contains enhanced files of images which **have annotations**\n\n\n> **NOTE 1 : images with no annotations will generate empty {image_id}.txt file under \"/labels\" folder**\n\n> **NOTE 2 : There are 18.5k un-annotated images. So, we will only copy/generate 1k due to memory constraints**\n\n\n### Data Folder Structure\n- dataset\n    - original-no-annot\n        - images\n        - labels\n    - original-annot\n        - images\n        - labels\n    - enhanced-no-annot\n        - images\n        - labels\n    - enhanced-annot\n        - images\n        - labels\n\n### What is enhanced image?\n   - Enhance Image has 2 filters applied to it,\n       - [CLAHE](https://en.wikipedia.org/wiki/Adaptive_histogram_equalization#Contrast_Limited_AHE)\n       - [Gamma Correction](https://en.wikipedia.org/wiki/Gamma_correction)\n\n### What is label .txt?\nA \".txt\" file is generated under \"/labels\" corresponding folder for each image. Each row is \"class x_center y_center width height\" format within that text file.","metadata":{}},{"cell_type":"markdown","source":"### Imports","metadata":{}},{"cell_type":"code","source":"!python -m pip install --upgrade pip\n!pip install tf_clahe","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-12-25T11:41:39.067399Z","iopub.execute_input":"2021-12-25T11:41:39.067700Z","iopub.status.idle":"2021-12-25T11:42:15.571573Z","shell.execute_reply.started":"2021-12-25T11:41:39.067670Z","shell.execute_reply":"2021-12-25T11:42:15.570491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%load_ext autoreload\n%autoreload 2\nfrom IPython.display import Image, display, Markdown\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib \n%matplotlib inline\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport os\nimport copy\nimport os.path as osp\nimport json5\nimport yaml\nimport shutil\nfrom pathlib import Path\nimport ast\nimport sys\nimport tensorflow as tf\nimport tf_clahe\n\n# Make numpy values easier to read.\nnp.set_printoptions(precision=3, suppress=True)\nsys.path.append('../input/tensorflow-great-barrier-reef')\nnp.random.seed(0)\ntf.version","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-12-25T11:42:15.573716Z","iopub.execute_input":"2021-12-25T11:42:15.574007Z","iopub.status.idle":"2021-12-25T11:42:21.819733Z","shell.execute_reply.started":"2021-12-25T11:42:15.573975Z","shell.execute_reply":"2021-12-25T11:42:21.818714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image Enhancement Helpers","metadata":{}},{"cell_type":"code","source":"def plot_images(img1, img2, title='', lbl1='', lbl2=''):\n    plt.close('all')\n    fig, ax = plt.subplots(1, 2, figsize=(23,13))\n    ax[0].imshow(img1, cmap = plt.get_cmap(name = 'gray'))\n    ax[0].set_axis_off()\n    ax[0].set_title(lbl1, fontsize=18)\n    ax[1].imshow(img2, cmap = plt.get_cmap(name = 'gray'))\n    ax[1].set_axis_off()\n    ax[1].set_title(lbl2, fontsize=18)\n    fig.suptitle(title, fontsize=22, y=0.81)\n    plt.tight_layout()\n    plt.show()\n\ndef tf_enhance_image_helper(img_path, gpu_optimized = False):\n    # channel 1 is grayscale and 3 is RGB\n    og_img = tf.io.decode_jpeg(tf.io.read_file(img_path), channels=3)\n     \n    correct_img = tf_clahe.clahe(og_img, tile_grid_size=(32, 32), clip_limit=3.3, gpu_optimized=gpu_optimized)\n    correct_img = tf.image.adjust_gamma(correct_img, 1.2)\n    return correct_img, og_img\n\n@tf.function(jit_compile=True)  # Enable XLA\ndef tf_enhance_image_gpu(img_path):\n    return tf_enhance_image_helper(img_path, True)\n\ndef tf_enhance_image(img_path):\n    import torch\n    if torch.cuda.is_available():\n        return tf_enhance_image_gpu(img_path)\n    return tf_enhance_image_helper(img_path, False)\n\ndef tf_save_image(img, fname):\n    enc = tf.image.encode_jpeg(img)\n    tf.io.write_file(tf.constant(fname), enc)\n    \ndef tf_save_label(annot_list, fname):\n    annots = ''\n    if annot_list:\n        annots = '\\n'.join(annot_list)\n    tf.io.write_file(tf.constant(fname), annots)","metadata":{"execution":{"iopub.status.busy":"2021-12-25T11:42:21.821639Z","iopub.execute_input":"2021-12-25T11:42:21.821989Z","iopub.status.idle":"2021-12-25T11:42:21.889544Z","shell.execute_reply.started":"2021-12-25T11:42:21.821933Z","shell.execute_reply":"2021-12-25T11:42:21.888611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### What \"Enhanced Image\" looks like","metadata":{}},{"cell_type":"code","source":"def sample_enhanced_img():\n    img_path1 = \"../input/tensorflow-great-barrier-reef/train_images/video_2/5752.jpg\"\n    img_path2 = \"../input/tensorflow-great-barrier-reef/train_images/video_2/5774.jpg\"\n\n    correct_img, og_img = tf_enhance_image(img_path1)\n    correct_img2, og_img2 = tf_enhance_image(img_path2)\n\n    plot_images(og_img.numpy(), correct_img, 'Image Comparison',\n                'Original #2_5752', 'CLAHE + Gamma Corrected #2_5752')\n    display(Markdown('## Original Image #2_5774'))\n    display(Image.fromarray(og_img2.numpy()))\n    display(Markdown('## CLAHE + GAMMA Corrected Image #2_5774'))\n    display(Image.fromarray(correct_img2.numpy()))\n\nsample_enhanced_img()\n# %timeit -n10 -r3 sample_enhanced_img(img_path1)","metadata":{"execution":{"iopub.status.busy":"2021-12-25T11:42:21.892014Z","iopub.execute_input":"2021-12-25T11:42:21.892254Z","iopub.status.idle":"2021-12-25T11:42:29.394668Z","shell.execute_reply.started":"2021-12-25T11:42:21.892228Z","shell.execute_reply":"2021-12-25T11:42:29.393558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Related Helpers","metadata":{}},{"cell_type":"code","source":"# helper to generate yolo label file\ndef generateYoloLabelFile(folder_path, file_name, label_data_list):\n    pth = f'{os.getcwd()}/{str(folder_path)}/{str(file_name)}.txt'\n    with open(pth, 'w') as f:\n        if label_data_list is None:\n            # create empty txt file\n            pass\n        else:\n            # create data txt file\n            for label in label_data_list:\n                if label:\n                    f.write(label.strip())\n                    f.write(\"\\n\")\n    return pth\n\n# helper to generate images\ndef generateImageFile(folder_path, file_name, image):\n        new_dst = f'{os.getcwd()}/{str(folder_path)}/{str(file_name)}.jpg'\n        cv2.imwrite(new_dst, image)\n        return new_dst\n    \n# Memory saving function credit to https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n\n    for col in df.columns:\n        col_type = df[col].dtype\n\n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n\n    return df\n\ndef get_path(row):\n    row['image_path'] = f'/kaggle/input/tensorflow-great-barrier-reef/train_images/video_{row.video_id}/{row.video_frame}.jpg'\n    return row\n\ndef load_labels():\n    print('fetching original csv')\n    labels = pd.read_csv(\"/kaggle/input/tensorflow-great-barrier-reef/train.csv\", skipinitialspace=True)\n    print('dropping duplicates')\n    labels.drop_duplicates(inplace=True)\n    print('dropping columns : sequence, sequence_frame')\n    labels.drop(['sequence', 'sequence_frame'], axis = 1, inplace=True)\n    print('reducing mem usage')\n    labels = reduce_mem_usage(labels)\n    print('adding image paths')\n    labels = labels.apply(get_path, axis=1)\n    print('Total rows :', len(labels))\n    return labels\n\ndef create_dataset_csv():\n    from sklearn.model_selection import train_test_split\n    labels = load_labels()\n    unannot_labels = labels[labels['annotations'] == '[]']\n    annot_labels = labels[labels['annotations'] != '[]']\n    print(f'Total annotated {len(annot_labels)}, Total un-annotated {len(unannot_labels)}\\n')\n    print('splitting data')\n    an_train, an_test = train_test_split(annot_labels, test_size=0.2)\n    un_train, un_test = train_test_split(unannot_labels.sample(1300), test_size=0.2)\n    print(f'Train annotated {len(an_train)}, Test annotated {len(an_test)}')\n    print(f'Train un-annotated {len(un_train)}, Test un-annotated {len(un_test)}\\n combining data')\n    train_data = an_train.append(un_train)\n    print(f'Total Train (annot + un-annot) {len(train_data)}')\n    test_data = an_test.append(un_test)\n    print(f'Total Test (annot + un-annot) {len(test_data)}')\n    print('writing train_new.csv')\n    train_data.to_csv('/kaggle/working/train_new.csv', encoding='utf-8', index=False)\n    print('writing test_new.csv')\n    test_data.to_csv('/kaggle/working/test_new.csv', encoding='utf-8', index=False)\n\ndef load_train_data():\n    df = pd.read_csv(\"/kaggle/working/train_new.csv\", skipinitialspace=True)\n    df = reduce_mem_usage(df)\n    return df\n\ndef load_test_data():\n    df = pd.read_csv(\"/kaggle/working/test_new.csv\", skipinitialspace=True)\n    df = reduce_mem_usage(df)\n    return df","metadata":{"execution":{"iopub.status.busy":"2021-12-25T12:29:23.758223Z","iopub.execute_input":"2021-12-25T12:29:23.759783Z","iopub.status.idle":"2021-12-25T12:29:23.830080Z","shell.execute_reply.started":"2021-12-25T12:29:23.759576Z","shell.execute_reply":"2021-12-25T12:29:23.829094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# generate train and test csv\ncreate_dataset_csv()","metadata":{"execution":{"iopub.status.busy":"2021-12-25T11:42:52.638741Z","iopub.execute_input":"2021-12-25T11:42:52.639076Z","iopub.status.idle":"2021-12-25T11:43:09.089650Z","shell.execute_reply.started":"2021-12-25T11:42:52.639044Z","shell.execute_reply":"2021-12-25T11:43:09.088452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dataset Creation","metadata":{}},{"cell_type":"code","source":"# helper to convert bbox value to yolo format\ndef convertToYoloFormat(x_min, y_min, bb_w, bb_h, im_w, im_h):\n    # find x_center\n    x_center = x_min + (bb_w/2.0)\n    # find y_center\n    y_center = y_min + (bb_h/2.0)\n    # normalize values to 0-1\n    n_x_center = x_center / im_w\n    n_y_center = y_center / im_h\n    n_bb_w = bb_w / im_w\n    n_bb_h = bb_h / im_h\n    return f'{round(n_x_center, 4)} {round(n_y_center,4)} {round(n_bb_w, 4)} {round(n_bb_h, 4)}'\n\ndef process_annot(ann_data):\n    yolo_labels = []\n    if ann_data and ann_data != '[]':\n        im_w, im_h = 1280, 720\n        ann_splits = json5.loads(ann_data)\n        for ann_split in ann_splits:\n            # load annotation data as json and get values\n            a_data = json5.loads(ann_split)\n            bb_x_min = int(a_data[\"x\"])\n            bb_y_min = int(a_data[\"y\"])\n            bb_w = int(a_data[\"width\"])\n            bb_h = int(a_data[\"height\"])\n            yolo_format_bb = convertToYoloFormat(bb_x_min, bb_y_min, bb_w, bb_h, im_w, im_h)\n            yolo_labels.append(f'0 {yolo_format_bb}')\n            del a_data, bb_x_min, bb_y_min, bb_w, bb_h, yolo_format_bb\n        del ann_splits\n    return yolo_labels\n    \ndef process_row(val, og_fp, en_fp):\n    video_id = str(val[0])\n    video_frame = str(val[1])\n    image_id = str(val[2])\n    img_path = val[4]\n    if os.path.exists(img_path) and os.path.isfile(img_path):\n        try:\n            # original dataset paths\n            og_lbl_folder = os.path.join(og_fp, 'labels')\n            og_img_folder = os.path.join(og_fp, 'images')\n            \n            # enhanced dataset paths\n            en_lbl_folder = os.path.join(en_fp, 'labels')\n            en_img_folder = os.path.join(en_fp, 'images')\n            \n            # process annotation, data\n            yolo_labels = process_annot(val[3])\n            \n            # original - generate label data \n            og_lbl_save_path = os.path.join(og_lbl_folder, f'{image_id}.txt')\n            tf_save_label(yolo_labels, og_lbl_save_path)\n            \n            # original - generate image data\n            og_save_path = os.path.join(og_img_folder, f'{image_id}.jpg')\n            shutil.copy(img_path, og_save_path)\n\n            # enhanced - generate label data\n            en_lbl_save_path = os.path.join(en_lbl_folder, f'{image_id}.txt')\n            tf_save_label(yolo_labels, en_lbl_save_path)\n            \n            # enhanced generate image data\n            en_save_path = os.path.join(en_img_folder, f'{image_id}.jpg')\n            correct_img, og_img = tf_enhance_image(img_path)\n            tf_save_image(correct_img, en_save_path)\n            \n            del yolo_labels, correct_img, og_img\n        except KeyboardInterrupt:\n            raise\n\n        except:\n            # corrupt file, skip\n            print('corrupt file')\n            raise\n    del video_id, video_frame, image_id, img_path\n            \ndef generate_dataset(data, is_train):\n    from tqdm import tqdm\n    import pandas as pd\n    import numpy as np\n    import gc\n    import os\n    import cv2\n    from joblib import Parallel, delayed\n    import multiprocessing\n    import torch\n    \n    os.chdir('/kaggle/working/dataset')\n    base_path = os.getcwd()\n    work_folder = 'train' if is_train else 'test'\n    og_fp = os.path.join(base_path, 'original', work_folder)\n    en_fp = os.path.join(base_path, 'enhanced', work_folder)\n#     for val in tqdm(data.values):\n#         torch.cuda.empty_cache()\n#         process_row(val, og_fp, en_fp)\n    Parallel(n_jobs=multiprocessing.cpu_count(), prefer='processes')(delayed(process_row)(val, og_fp, en_fp) for val in tqdm(data.values))\n    gc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-25T11:43:48.724605Z","iopub.execute_input":"2021-12-25T11:43:48.725185Z","iopub.status.idle":"2021-12-25T11:43:48.813513Z","shell.execute_reply.started":"2021-12-25T11:43:48.725125Z","shell.execute_reply":"2021-12-25T11:43:48.812877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run create dataset","metadata":{}},{"cell_type":"code","source":"# always re-create\n%cd /kaggle/working\n!rm -rf dataset\n!mkdir dataset\n!mkdir dataset/original\n!mkdir dataset/original/train\n!mkdir dataset/original/train/images\n!mkdir dataset/original/train/labels\n!mkdir dataset/original/test\n!mkdir dataset/original/test/images\n!mkdir dataset/original/test/labels\n!mkdir dataset/enhanced\n!mkdir dataset/enhanced/train\n!mkdir dataset/enhanced/train/images\n!mkdir dataset/enhanced/train/labels\n!mkdir dataset/enhanced/test\n!mkdir dataset/enhanced/test/images\n!mkdir dataset/enhanced/test/labels\n%cd /kaggle/working\n\ntry:\n    # generate train data\n    train_data = load_train_data()\n    generate_dataset(train_data, True)\n    \n    # generate test data\n    test_data = load_test_data()\n    generate_dataset(test_data, False)\nexcept KeyboardInterrupt:\n    pass","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-12-25T11:43:50.656672Z","iopub.execute_input":"2021-12-25T11:43:50.657165Z","iopub.status.idle":"2021-12-25T11:45:05.157654Z","shell.execute_reply.started":"2021-12-25T11:43:50.657108Z","shell.execute_reply":"2021-12-25T11:45:05.156034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def check_file_counts():\n    total_image_count = len(os.listdir('/kaggle/input/tensorflow-great-barrier-reef/train_images/video_0')) \\\n    + len(os.listdir('/kaggle/input/tensorflow-great-barrier-reef/train_images/video_1')) \\\n    + len(os.listdir('/kaggle/input/tensorflow-great-barrier-reef/train_images/video_2'))\n\n    print('Total image count', total_image_count)\n    \n    im_len = len(os.listdir('/kaggle/working/dataset/original/train/images'))\n    lab_len = len(os.listdir('/kaggle/working/dataset/original/train/labels'))\n    print(f'OG Train IM count {im_len}, Label Count {lab_len}')\n    \n    im_len = len(os.listdir('/kaggle/working/dataset/enhanced/train/images'))\n    lab_len = len(os.listdir('/kaggle/working/dataset/enhanced/train/labels'))\n    print(f'EN Train IM count {im_len}, Label Count {lab_len}')\n\n    im_len = len(os.listdir('/kaggle/working/dataset/original/test/images'))\n    lab_len = len(os.listdir('/kaggle/working/dataset/original/test/labels'))\n    print(f'OG Test  IM count {im_len}, Label Count {lab_len}')\n    \n    im_len = len(os.listdir('/kaggle/working/dataset/enhanced/test/images'))\n    lab_len = len(os.listdir('/kaggle/working/dataset/enhanced/test/labels'))\n    print(f'EN Test  IM count {im_len}, Label Count {lab_len}')\n\ncheck_file_counts()","metadata":{"execution":{"iopub.status.busy":"2021-12-25T12:29:32.551891Z","iopub.execute_input":"2021-12-25T12:29:32.552235Z","iopub.status.idle":"2021-12-25T12:29:32.668387Z","shell.execute_reply.started":"2021-12-25T12:29:32.552199Z","shell.execute_reply":"2021-12-25T12:29:32.666771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### References\n- Improvements learned from [Learning to Sea: Underwater img Enhancement](https://www.kaggle.com/soumya9977/learning-to-sea-underwater-img-enhancement-eda#%F0%9F%8E%AF-Main-Working-Code)\n- Improved white balance implementation by [gist](https://gist.github.com/DavidYKay/9dad6c4ab0d8d7dbf3dc#gistcomment-3025656)\n- [Image preprocessing tips](https://machinelearningmastery.com/how-to-configure-image-data-augmentation-when-training-deep-learning-neural-networks/)\n- [Why \"White Balance Correction\" is neccessary](https://openaccess.thecvf.com/content_ICCV_2019/papers/Afifi_What_Else_Can_Fool_Deep_Learning_Addressing_Color_Constancy_Errors_ICCV_2019_paper.pdf)","metadata":{}}]}