{"cells":[{"metadata":{"_uuid":"73a03f9064772720812c4a6c1516c43d44b66809"},"cell_type":"markdown","source":"I am trying to implement the F2 scoring used for the LB, for local validation purposes. \nAny feedback is appreciated!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true},"cell_type":"code","source":"import cv2\nimport math\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\ndf = pd.read_csv('../input/train_ship_segmentations.csv')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b80508c3666bb93348abf5bccbda3f43c2e9b081"},"cell_type":"markdown","source":"RLE decoding \"borrowed\" from @inversion's [Run Length Decoding - Quick Start\n](https://www.kaggle.com/inversion/run-length-decoding-quick-start )"},{"metadata":{"trusted":true,"_uuid":"6e467f05f1e207ea0647f91eec0d4419b523cd0a"},"cell_type":"code","source":"def rle_decode(mask_rle, shape=(768, 768)):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T  # Needed to align to RLE direction","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea3d5255e9f637ee7a93b522b6df08ab09220d67"},"cell_type":"markdown","source":"I define some utility functions for:\n* reading an image\n* reading all masks for a given image (for training and validation purposes)\n* flattening all masks (ships) for an image into a single one (for visualisation and possible training purposes)"},{"metadata":{"trusted":true,"_uuid":"229a47e5843441ebf0def4c73091892da26ff9b9"},"cell_type":"code","source":"def read_image(img_name, type='train'):\n    if type=='train':\n        path = '../input/train/{}'\n    else:\n        path = '../input/test/{}'\n    img = cv2.imread(path.format(img_name))\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\ndef read_masks(img_name):\n    mask_list = df.loc[df['ImageId'] == img_name, 'EncodedPixels'].tolist()\n    all_masks = np.zeros((len(mask_list), 768,768))\n    for idx, mask in enumerate(mask_list):\n        if isinstance(mask, str):\n            all_masks[idx] = rle_decode(mask)\n    return all_masks\n\ndef read_flat_mask(img_name):\n    all_masks = read_masks(img_name)\n    return np.sum(all_masks, axis=0)\n\n\nimage_with_ships = '00021ddc3.jpg'\nimage_with_no_ships = '00003e153.jpg'\n_, axarr = plt.subplots(1, 2, figsize=(15, 40))\naxarr[0].axis('off')\naxarr[1].axis('off')\naxarr[0].imshow(read_image(image_with_ships))\naxarr[0].imshow(read_flat_mask(image_with_ships), alpha=0.4)\naxarr[1].imshow(read_image(image_with_no_ships))\naxarr[1].imshow(read_flat_mask(image_with_no_ships), alpha=0.4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5923663b12d96becdd69f86be386b0fe2e42de5"},"cell_type":"markdown","source":"Compute IOU for 2 given binary masks. We need to avoid division by zero when there are no ships in ground truth and prediction.\nWe look at values for an image against itslef, all zeros and all ones masks."},{"metadata":{"trusted":true,"_uuid":"f74a67d6ff7d02e9775053ed3f5d8343c02a1a10"},"cell_type":"code","source":"def iou(img_true, img_pred):\n    i = np.sum((img_true*img_pred) >0)\n    u = np.sum((img_true + img_pred) >0) + 0.0000000000000000001  # avoid division by zero\n    return i/u\n\nm = read_flat_mask(image_with_ships)\nprint(iou(m, m), iou(0, np.zeros((768, 768))), iou(m, np.ones((768, 768))))\n\nm = read_flat_mask(image_with_no_ships)\nprint(iou(m, m), iou(m, np.zeros((768, 768))), iou(m, np.ones((768, 768))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87d82b0aa68c967b6d9d739876a4067bb34c104b"},"cell_type":"markdown","source":"F2 score implementation, but treating the exceptional case of a correct prediction of no ship.\nAgain, we test against itself, and single predictions of ones or zeros."},{"metadata":{"trusted":true,"_uuid":"2e1fe71874e3e628c4f07755ec3f9ccb79a7f9fc","scrolled":true},"cell_type":"code","source":"thresholds = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]\n\ndef f2(masks_true, masks_pred):\n    # a correct prediction on no ships in image would have F2 of zero (according to formula),\n    # but should be rewarded as 1\n    if np.sum(masks_true) == np.sum(masks_pred) == 0:\n        return 1.0\n    \n    f2_total = 0\n    ious = {}\n    for t in thresholds:\n        tp,fp,fn = 0,0,0\n        for i,mt in enumerate(masks_true):\n            found_match = False\n            for j,mp in enumerate(masks_pred):\n                key = 100 * i + j\n                if key in ious.keys():\n                    miou = ious[key]\n                else:\n                    miou = iou(mt, mp)\n                    ious[key] = miou  # save for later\n                if miou >= t:\n                    found_match = True\n            if not found_match:\n                fn += 1\n                \n        for j,mp in enumerate(masks_pred):\n            found_match = False\n            for i, mt in enumerate(masks_true):\n                miou = ious[100*i+j]\n                if miou >= t:\n                    found_match = True\n                    break\n            if found_match:\n                tp += 1\n            else:\n                fp += 1\n        f2 = (5*tp)/(5*tp + 4*fn + fp)\n        f2_total += f2\n    \n    return f2_total/len(thresholds)\n\n\nm = read_masks(image_with_ships)\nprint(f2(m, m), f2(m, [np.zeros((768, 768))]), f2(m, [np.ones((768, 768))]))\n\nm = read_masks(image_with_no_ships)\nprint(f2(m, m), f2(m, [np.zeros((768, 768))]), f2(m, [np.ones((768, 768))]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f390b9cbcb467a61c49c6ad8bce5eaaf82d06996"},"cell_type":"markdown","source":"I compute the average F2 on a subset of ground-truths and and a single blank prediction to see if we get anywhere close to the 0.847"},{"metadata":{"trusted":true,"_uuid":"da1c414720f7b5105355f47cdbb6c51ba754614f","scrolled":true},"cell_type":"code","source":"subset_images = 2000\nrandom_files = df['ImageId'].unique()\nnp.random.shuffle(random_files)\n#print(random_files[:subset_images])\n#print(random_files)\nf2_sum = 0\nfor fname in random_files[:subset_images]:\n    mask = read_masks(fname)\n    score = f2(mask, [np.zeros((768,768))])\n    f2_sum += score\n    \nprint(f2_sum/subset_images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"3702c6714504b197162510e4285ea85463e7e35a"},"cell_type":"markdown","source":"I now get a smaller value, which could mean:\n* a slightly different distribution of the train/test datasets\n* subset is too small (due to kernel limitations) \n* my evaluation code is wrong :)\n\nAs @manuscrits very well pointed out, the expectation would be to get a value close to the ratio of images with no ship. That means 75000, but we need to take care when counting the images **with** ships, as in the training data we have a mask for each detected ship, hence an image can appear multiple times if it contains more ships. \n\nAs such, the ratio of images with no ships is "},{"metadata":{"trusted":true,"_uuid":"0bd3eaec078da0943e80b42b4d2081c0bc26ed8e"},"cell_type":"code","source":"df['EncodedPixels'].isna().sum() / len(df['ImageId'].unique())","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}