{"cells":[{"metadata":{"_uuid":"ccf02f5cdd695a57dfa3379ac4a97a843cf4f481"},"cell_type":"markdown","source":"# F2 metric for evaluation"},{"metadata":{"_uuid":"73a03f9064772720812c4a6c1516c43d44b66809"},"cell_type":"markdown","source":"A take on optimizing the calculation of F2 score by @raresbarbantan<br>\n[https://www.kaggle.com/raresbarbantan/f2-metric](http://)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import random\nfrom multiprocessing import cpu_count\ncpu_count = cpu_count()\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nfrom joblib import Parallel, delayed\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nFOLDER = '../input/'\n\nimport os\nprint(os.listdir(FOLDER))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true,"collapsed":true},"cell_type":"code","source":"df = pd.read_csv('../input/train_ship_segmentations.csv')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"accf1abc0805dcc54507d843c5b61ec34ed4d73b"},"cell_type":"markdown","source":"# Utility functions"},{"metadata":{"_uuid":"b80508c3666bb93348abf5bccbda3f43c2e9b081"},"cell_type":"markdown","source":"RLE decoding \"borrowed\" from @inversion's [Run Length Decoding - Quick Start\n](https://www.kaggle.com/inversion/run-length-decoding-quick-start )"},{"metadata":{"trusted":true,"_uuid":"6e467f05f1e207ea0647f91eec0d4419b523cd0a","collapsed":true},"cell_type":"code","source":"def rle_decode(mask_rle, shape=(768, 768)):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T  # Needed to align to RLE direction","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea3d5255e9f637ee7a93b522b6df08ab09220d67"},"cell_type":"markdown","source":"I define some utility functions for:\n* reading an image\n* reading all masks for a given image (for training and validation purposes)\n* flattening all masks (ships) for an image into a single one (for visualisation and possible training purposes)"},{"metadata":{"trusted":true,"_uuid":"229a47e5843441ebf0def4c73091892da26ff9b9","collapsed":true},"cell_type":"code","source":"def read_image(img_name, type='train'):\n    if type=='train':\n        path = '../input/train/{}'\n    else:\n        path = '../input/test/{}'\n    img = cv2.imread(path.format(img_name))\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\ndef read_masks(img_name):\n    mask_list = df.loc[df['ImageId'] == img_name, 'EncodedPixels'].tolist()\n    all_masks = np.zeros((len(mask_list), 768,768))\n    for idx, mask in enumerate(mask_list):\n        if isinstance(mask, str):\n            all_masks[idx] = rle_decode(mask)\n    return all_masks\n\ndef read_flat_mask(img_name):\n    all_masks = read_masks(img_name)\n    return np.sum(all_masks, axis=0)\n\n\nimage_with_ships = '00021ddc3.jpg'\nimage_with_no_ships = '00003e153.jpg'\n_, axarr = plt.subplots(1, 2, figsize=(15, 40))\naxarr[0].axis('off')\naxarr[1].axis('off')\naxarr[0].imshow(read_image(image_with_ships))\naxarr[0].imshow(read_flat_mask(image_with_ships), alpha=0.4)\naxarr[1].imshow(read_image(image_with_no_ships))\naxarr[1].imshow(read_flat_mask(image_with_no_ships), alpha=0.4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5923663b12d96becdd69f86be386b0fe2e42de5"},"cell_type":"markdown","source":"Compute IOU for 2 given binary masks. We need to avoid division by zero when there are no ships in ground truth and prediction.\nWe look at values for an image against itslef, all zeros and all ones masks."},{"metadata":{"trusted":true,"_uuid":"f74a67d6ff7d02e9775053ed3f5d8343c02a1a10","collapsed":true},"cell_type":"code","source":"def iou(img_true, img_pred):\n    i = np.sum((img_true*img_pred) >0)\n    u = np.sum((img_true + img_pred) >0) + 0.0000000000000000001  # avoid division by zero\n    return i/u\n\nm = read_flat_mask(image_with_ships)\nprint(iou(m, m), iou(0, np.zeros((768, 768))), iou(m, np.ones((768, 768))))\n\nm = read_flat_mask(image_with_no_ships)\nprint(iou(m, m), iou(m, np.zeros((768, 768))), iou(m, np.ones((768, 768))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87d82b0aa68c967b6d9d739876a4067bb34c104b"},"cell_type":"markdown","source":"# F2 score implementation, compared to original version"},{"metadata":{"trusted":true,"_uuid":"2e1fe71874e3e628c4f07755ec3f9ccb79a7f9fc","scrolled":true,"collapsed":true},"cell_type":"code","source":"thresholds = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]\n\n# original version of @raresbarbantan\ndef f2_prev(masks_true, masks_pred):\n    # a correct prediction on no ships in image would have F2 of zero (according to formula),\n    # but should be rewarded as 1\n    if np.sum(masks_true) == np.sum(masks_pred) == 0:\n        return 1.0\n    \n    f2_total = 0\n    for t in thresholds:\n        tp,fp,fn = 0,0,0\n        ious = {}\n        for i,mt in enumerate(masks_true):\n            found_match = False\n            for j,mp in enumerate(masks_pred):\n                miou = iou(mt, mp)\n                ious[100*i+j] = miou # save for later\n                if miou >= t:\n                    found_match = True\n            if not found_match:\n                fn += 1\n                \n        for j,mp in enumerate(masks_pred):\n            found_match = False\n            for i, mt in enumerate(masks_true):\n                miou = ious[100*i+j]\n                if miou >= t:\n                    found_match = True\n                    break\n            if found_match:\n                tp += 1\n            else:\n                fp += 1\n        f2 = (5*tp)/(5*tp + 4*fn + fp)\n        f2_total += f2\n    \n    return f2_total/len(thresholds)\n\ndef f2(masks_true, masks_pred):\n    if np.sum(masks_true) == 0:\n        return float(np.sum(masks_pred) == 0)\n    \n    ious = []\n    mp_idx_found = []\n    for mt in masks_true:\n        for mp_idx, mp in enumerate(masks_pred):\n            if mp_idx not in mp_idx_found:\n                cur_iou = iou(mt,mp)\n                if cur_iou > 0.5:\n                    ious.append(cur_iou)\n                    mp_idx_found.append(mp_idx)\n                    break\n    f2_total = 0\n    for th in thresholds:\n        tp = sum([iou > th for iou in ious])\n        fn = len(masks_true) - tp\n        fp = len(masks_pred) - tp\n        f2_total += (5*tp)/(5*tp + 4*fn + fp)    \n\n    return f2_total/len(thresholds)\n    \n    \nm = read_masks(image_with_ships)\nprint(f2_prev(m, m), f2_prev(m, [np.zeros((768, 768))]), f2_prev(m, [np.ones((768, 768))]))\nprint(f2(m, m), f2(m, [np.zeros((768, 768))]), f2(m, [np.ones((768, 768))]))\n\nm = read_masks(image_with_no_ships)\nprint(f2_prev(m, m), f2_prev(m, [np.zeros((768, 768))]), f2_prev(m, [np.ones((768, 768))]))\nprint(f2(m, m), f2(m, [np.zeros((768, 768))]), f2(m, [np.ones((768, 768))]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f390b9cbcb467a61c49c6ad8bce5eaaf82d06996"},"cell_type":"markdown","source":"# Let's compare speed of these versions:"},{"metadata":{"trusted":true,"_uuid":"da1c414720f7b5105355f47cdbb6c51ba754614f","scrolled":true,"collapsed":true},"cell_type":"code","source":"sample_size = 2000\nrandom_files = random.sample(list(df['ImageId'].unique()), sample_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd07822c063b3f4d317676040089a8975f2935bc","collapsed":true},"cell_type":"code","source":"def f2_from_fnames(fnames):\n    score_sum = 0\n    for fname in fnames:\n        mask = read_masks(fname)\n        score_sum += f2(mask, [np.zeros((768,768))])\n    return score_sum\n\ndef f2_from_fnames_prev(fnames):\n    score_sum = 0\n    for fname in fnames:\n        mask = read_masks(fname)\n        score_sum += f2_prev(mask, [np.zeros((768,768))])\n    return score_sum\n\nprint(\"Correct value is ratio of empty files: \", \n      df[df['ImageId'].isin(random_files)]['EncodedPixels'].isna().sum() / sample_size)\n\n%time scores = f2_from_fnames_prev(random_files)\nprint(scores/sample_size)\n%time scores = f2_from_fnames(random_files)\nprint(scores/sample_size)\n\n\nn_workers = cpu_count\n%time scores = Parallel(n_workers)(delayed(f2_from_fnames_prev)(random_files[i::n_workers]) for i in range(n_workers))\nscores = sum(scores)\nprint(scores/sample_size)\n%time scores = Parallel(n_workers)(delayed(f2_from_fnames)(random_files[i::n_workers]) for i in range(n_workers))\nscores = sum(scores)\nprint(scores/sample_size)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"3702c6714504b197162510e4285ea85463e7e35a"},"cell_type":"markdown","source":"There's a substantial increase due to more efficient implementation, and another increase due to parallelizing, compared to the original version of **@raresbarbantan**.<br>\nOverall it's 5+ times faster."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"04d99f58968e40beea5e14f04cf7b3cf1986595c"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}