{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"87529994-556a-8231-cde3-843396a1a29d"},"source":"# Countless Sea Lions\n\nIn this challenge we are given a set of images and are asked to calculate the number of sea lions in each image, split by a few categories that are indicated in extra overlay images:\n\n* red: adult males\n* magenta: subadult males\n* brown: adult females\n* blue: juveniles\n* green: pups\n\n\n----------\n\n\nIn this notebook we will be looking in detail at:\n\n* numeric (counts of sea lions) feature exploration, both with correlation analysis and clustermaps\n* marker images and how we can use them to extract location and image patches for the marked sea lions\n* template matching to build a sea lion detector with OpenCV.\n\nLet's get started with an initial peek at the stats of the data and later plot some sample images."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"75b8f60e-0bda-5980-3fde-6a672b13ce79"},"outputs":[],"source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom subprocess import check_output\nimport matplotlib.pyplot as plt\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\nfrom glob import glob\nimport seaborn as sns\nfrom scipy import stats\n\ndf = pd.read_csv('../input/Train/train.csv')\nprint(\"{} training samples total\".format(df.shape[0]))\ndf.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"75c351eb-c001-ccf3-c93e-a410e9ef4667"},"source":"Total counts of all sea lion types over all images."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a714b048-3363-fe12-a05a-978af495238d"},"outputs":[],"source":"df[['adult_males', 'subadult_males', 'adult_females', 'juveniles', 'pups']].sum(axis=0).plot.barh()"},{"cell_type":"markdown","metadata":{"_cell_guid":"cd121877-650c-208b-70d6-c6bb2ea47b7e"},"source":"# Correlations of sea lion counts\n\nNow we will check how the counts of sea lions in the training images correlate which each other and how they are distributed."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"402b940b-5bb9-9abb-680b-006bd1a3c024"},"outputs":[],"source":"def corrfunc(x, y, **kws):\n    r, _ = stats.pearsonr(x, y)\n    ax = plt.gca()\n    ax.annotate(\"r = {:.3f}\".format(r),\n                xy=(.1, .9), xycoords=ax.transAxes)\n\ng = sns.PairGrid(df[['adult_males', 'subadult_males', 'adult_females', 'juveniles', 'pups']], palette=[\"red\"])\n#g.map_upper(plt.scatter, s=10)\ng.map_lower(plt.scatter, s=10)\ng.map_diag(sns.distplot, kde=False)\n#g.map_lower(sns.kdeplot, cmap=\"Blues_d\")\ng.map_lower(corrfunc)\n#sns.pairplot(df)"},{"cell_type":"markdown","metadata":{"_cell_guid":"4502d619-e978-7d31-ae71-d8e24e123aa1"},"source":"It intuitively makes sense that the correlation of number of adult females and pups is rather strong, slightly more than **0.81** in that case. Also there is quite a strong correlation between adult male and female counts."},{"cell_type":"markdown","metadata":{"_cell_guid":"038a62a5-87e1-5899-cfd9-7ffac8f87242"},"source":"# Distribution of sea lion counts\n\nFor that we want to get a first idea of what frequent patterns of sea lion fractions are present in the images.  We'll use seaborn's clustermap to visualize the potential clusters, but first, let's normalize the counts.\n\nWe will do that by summing over all columns for each row, that will give us the total count of sea lions for each image."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"6404c973-6392-22de-a43b-b7667c3ea82f"},"outputs":[],"source":"all_types = ['adult_males', 'subadult_males', 'adult_females', 'juveniles', 'pups']\nall_normalized_types = ['normalized_'+t for t in all_types]\nrow_counts = df[['adult_males', 'subadult_males', 'adult_females', 'juveniles', 'pups']].sum(axis=1)\n\nfor t in all_types:\n    df['normalized_'+t] = df[t].divide(row_counts)\n\ndf.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1859d899-9741-ca91-6838-dc45352bf976"},"outputs":[],"source":"sns.clustermap(\n    df[all_normalized_types].fillna(0.0),\n    col_cluster=False,\n    cmap=plt.get_cmap('viridis'),\n    figsize=(12,10)\n)"},{"cell_type":"markdown","metadata":{"_cell_guid":"36e74e40-2456-3569-f1ae-e72ffb36388f"},"source":"From the above clustermap we can see that there are a few image clusters by sea lion fractions. For example, in the bottom left corner, we have a bunch of images that mostly contain adult males. Young males don't seem to hang around adult males.\n\nPups we normaly see when there is a moderate fraction of adult females present. It seems that when there are alot of adult females around, there aren't as many pups. Still mating maybe? :)\n\nAlso, if there are a lot of juveniles, there don't seem to be a lot of adult females.\n\nAll of this information can be used in a later stage of the competition to regularize the predicted counts towards one of the clustered proportions that we've found here."},{"cell_type":"markdown","metadata":{"_cell_guid":"85aa4fbb-8a22-2f37-4cce-76ffdebf58eb"},"source":"# Sea lion clustering\n\nWe'll now compute pairwise similarities of all of the images, where each image is represented as the fraction of sea lion types present in it. This is a slightly different perspective on the clustermap from above. We can now reason about pairwise similarities of images, e.g. the thick yellow line spectrum at the bottom likely corresponds to the mostly adult male pictures."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"83a87be6-d505-54e0-031f-e319e1f46a91"},"outputs":[],"source":"from scipy.spatial.distance import pdist, squareform\n\nsq_dists = squareform(pdist(df[all_normalized_types].values))\nsq_dists[np.isnan(sq_dists)] = 0.0\nsns.clustermap(\n    sq_dists,\n    cmap=plt.get_cmap('viridis'),\n    figsize=(12,10)\n)"},{"cell_type":"markdown","metadata":{"_cell_guid":"41760144-1374-06b8-6153-3f8e0f77ef4b"},"source":"# Sample Images\n\nEach image in the training set as an unique training id and gives the counts of each sea lion category. \n\nLet's load a few images and have a look at them."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"80f24883-300a-9bae-f329-960ed8a1f52c"},"outputs":[],"source":"training_images = glob('../input/Train/*.jpg')\ntraining_dotted = glob('../input/TrainDotted/*.jpg')\nlen(training_images), len(training_dotted)"},{"cell_type":"markdown","metadata":{"_cell_guid":"1b70787f-18c7-5ff4-8b5d-73ddff7108a1"},"source":"To the kernels on kaggle there only seem to be eleven images available."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"96fd26f2-812c-0c26-3ade-2b9a37f64cca"},"outputs":[],"source":"fig = plt.figure(figsize=(16,10))\nfor i in range(4):\n    ax = fig.add_subplot(2,2,i+1)\n    plt.imshow(plt.imread(training_images[i]))"},{"cell_type":"markdown","metadata":{"_cell_guid":"75ca8d06-4d54-ab05-accf-4484230c6252"},"source":"The images seem to be really large and diverse with very different counts of sea lions on them. Let's have a look at the correlations of the sea lion counts."},{"cell_type":"markdown","metadata":{"_cell_guid":"4e56c6fc-5ee7-b890-3b38-553af8e6fbca"},"source":"# Dotted training images\n\nWe'll now load a sample file with colored dot annotations, crop the image and show the dottet and non-dotted version."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"82495d3e-17ff-ad36-08c8-80139b1fea9c"},"outputs":[],"source":"from skimage.io import imread, imshow\nfrom skimage.util import crop\nimport cv2\n\ncropped_dotted = cv2.cvtColor(cv2.imread('../input/TrainDotted/8.jpg'), cv2.COLOR_BGR2RGB)[500:1500,2000:2800,:]\ncropped_raw = cv2.cvtColor(cv2.imread('../input/Train/8.jpg'), cv2.COLOR_BGR2RGB)[500:1500,2000:2800,:]\n\nfig = plt.figure(figsize=(12,8))\nax = fig.add_subplot(1,2,1)\nplt.imshow(cropped_dotted)\nax = fig.add_subplot(1,2,2)\nplt.imshow(cropped_raw)"},{"cell_type":"markdown","metadata":{"_cell_guid":"71116866-b86f-d420-32e8-73cd4966ecf1"},"source":"# Plot the markers only\n\nThere are also brown markers which are removed by our thresholding and are also not very present in the difference image itself."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"466b1357-c53c-c455-0dfb-900dbf061de1"},"outputs":[],"source":"diff = cv2.subtract(cropped_dotted, cropped_raw)\ndiff = diff/diff.max()\nplt.figure(figsize=(12,8))\nplt.imshow((diff > 0.20).astype(float))\nplt.grid(False)"},{"cell_type":"markdown","metadata":{"_cell_guid":"2dff0a46-3820-36bb-a527-376534447565"},"source":"Thanks to [Asymptote's][1] kernel, we can have the locations of the thresholded markers and do 2D density estimation on them. This will give us an idea how sea lions like to hang around each other.\n\n\n  [1]: https://www.kaggle.com/asymptote"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"efdc4e98-167c-0826-0770-f504602571be"},"outputs":[],"source":"diff = cv2.absdiff(cropped_dotted, cropped_raw)\ngray = cv2.cvtColor(diff, cv2.COLOR_RGB2GRAY)\nret,th1 = cv2.threshold(gray,0,255,cv2.THRESH_BINARY | cv2.THRESH_OTSU)\n\ncnts = cv2.findContours(th1.copy(), cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)[-2]\nprint(\"Sea Lions Found: {}\".format(len(cnts)))\n\nx, y = [], []\n\nlion_patches = []\n\nfor loc in cnts:\n    ((xx, yy), _) = cv2.minEnclosingCircle(loc)\n\n    # store patches of some sea lions\n    if xx > 10 and xx < gray.shape[1] - 10:\n        lion_patches.append(cropped_raw[yy-10:yy+10, xx-10:xx+10])\n\n    x.append(xx)\n    y.append(yy)\n\nx = np.array(x)\ny = np.array(y)"},{"cell_type":"markdown","metadata":{"_cell_guid":"7f3a121c-4083-565c-ad48-ace3b9855448"},"source":"The code below will take each of the markers as individual coordinates and feed it to a 2D kernel density estimation using a very common kernel, the gaussian one. We then show the estimated density by showing it's contour plot.\n\nSee [stackoverflow][1] for a very detailed explanation of the code below.\n\n\n  [1]: http://stackoverflow.com/questions/36957149/density-map-heatmaps-in-matplotlib"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"be2a3cc3-2509-6b80-19e5-36f1fb7252ed"},"outputs":[],"source":"from scipy.stats.kde import gaussian_kde\n\nk = gaussian_kde(np.vstack([x, y]), bw_method=0.5)\nxi, yi = np.mgrid[x.min():x.max():x.size**0.5*1j,y.min():y.max():y.size**0.5*1j]\nzi = k(np.vstack([xi.flatten(), yi.flatten()]))"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d741f44f-ec8a-d7d8-829c-49044d61cf5a"},"outputs":[],"source":"fig = plt.figure(figsize=(12,12))\nax1 = fig.add_subplot(211)\nax2 = fig.add_subplot(212)\n\n# alpha=0.5 will make the plots semitransparent\nax1.pcolormesh(xi, yi, zi.reshape(xi.shape), alpha=0.5)\nax2.contourf(xi, yi, zi.reshape(xi.shape), alpha=0.5)\n\nax1.set_xlim(x.min(), x.max())\nax1.set_ylim(y.min(), y.max())\nax2.set_xlim(x.min(), x.max())\nax2.set_ylim(y.min(), y.max())\n\nax1.imshow(cropped_raw, extent=[x.min(), x.max(), y.min(), y.max()], aspect='auto')\nax2.imshow(cropped_raw, extent=[x.min(), x.max(), y.min(), y.max()], aspect='auto')"},{"cell_type":"markdown","metadata":{"_cell_guid":"7eba66c7-8518-31cf-c0f6-3f735fa17fca"},"source":"# Sea lion patches\n\nAbove we've also extracted a few quadratic shaped patches around each of the markers. Sea lions are not quadradic but pups come a little closer to quadratic shapes when viewed from top.\n\nWe can see that the extracted patches approximate the pups pretty good."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"db11d44e-a6ac-f13c-51c0-34e531f67cac"},"outputs":[],"source":"n_images_total = 16\nn_images_per_row = 4\n\nfig = plt.figure(figsize=(16,12))\nfor i in range(n_images_total):\n    ax = fig.add_subplot(4,n_images_per_row,i+1)\n    plt.grid(False)\n    imshow(lion_patches[i])"},{"cell_type":"markdown","metadata":{"_cell_guid":"ea90099f-d122-edce-6e8f-3099ac795176"},"source":"# Template matching\n\nGiven how large each of the images is, an ideal setting would be to have each sealion detected and properly extracted and then fed to an image classification pipeline, that can distinguish between the five classes of sealions. We would then classify each extracted sealion and simply sum them up for each category.\n\nA first step to take into that direction is to build a sealion detector. For that we can utilize template matching, readily implemented in OpenCV. This approach is rather simple and will most likely be outperformed by a properly trained CNN, but let's see.\n\nI will now select a few templates and run them all against our cropped image to see how well the templates generalize to unseen sea lions.\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c770cf1a-bf1f-20d3-bb50-a8f7d1fb9f77"},"outputs":[],"source":"# read images again for template matching code\ncropped_dotted = cv2.cvtColor(cv2.imread('../input/TrainDotted/8.jpg'), cv2.COLOR_BGR2RGB)[1000:2000,2000:2800,:]\ncropped_raw = cv2.cvtColor(cv2.imread('../input/Train/8.jpg'), cv2.COLOR_BGR2RGB)[1000:2000,2000:2800,:]\nplt.imshow(cropped_raw)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"57c47ad5-dff3-22e3-e74e-da4dadcbc1a7"},"outputs":[],"source":"plt.clf()\nsealions = [\n    cropped_raw[35:90, 505:520],\n    cropped_raw[40:60, 510:515],\n    cropped_raw[930:945, 610:665],\n    cropped_raw[935:940, 630:645],\n    cropped_raw[658:678, 395:448],\n    cropped_raw[668:673, 415:420]\n]\nfig = plt.figure(figsize=(12,8))\nfor i in range(len(sealions)):\n    ax = fig.add_subplot(1,len(sealions),i+1)\n    imshow(sealions[i])"},{"cell_type":"markdown","metadata":{"_cell_guid":"d6fd8524-bc43-3355-b471-5c7a894fa887"},"source":"The next section runs six different template matching methods over the cropped image from above. It shows the activation of each pixel, given the template. We can see, that different methods, activate differently over the cropped image.\n\nTo detect a sealion with this method, we will search for the maximum or the maxima in the activation map and indicate it with a rectangular box."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"967d5519-d1ac-7d83-80d3-f6748510f5e7"},"outputs":[],"source":"# All the 6 methods for comparison in a list\n# not using all, to let kernel finish quickly\nmethods = [\n    'cv2.TM_CCOEFF',\n    'cv2.TM_CCOEFF_NORMED',\n    'cv2.TM_CCORR',\n    #'cv2.TM_CCORR_NORMED',\n    #'cv2.TM_SQDIFF',\n    #'cv2.TM_SQDIFF_NORMED'\n]\n\ndef templateMatchFor(image, sealion):\n    w, h = sealion.shape[1], sealion.shape[0]\n    for meth in methods:\n        method = eval(meth)\n\n        # Apply template Matching\n        res = cv2.matchTemplate(image,sealion,method)\n        min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)\n\n        # If the method is TM_SQDIFF or TM_SQDIFF_NORMED, take minimum\n        if method in [cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]:\n            top_left = min_loc\n        else:\n            top_left = max_loc\n        bottom_right = (top_left[0] + w, top_left[1] + h)\n\n        cv2.rectangle(image,top_left, bottom_right, 255, 2)\n\n        plt.figure(figsize=(12,8))\n        plt.subplot(121)\n        plt.imshow(res,cmap = 'gray')\n        plt.title('Matching Result')\n        plt.xticks([]), plt.yticks([])\n        plt.subplot(122)\n        plt.imshow(image,cmap = 'gray')\n        plt.title('Detected Point')\n        plt.xticks([]), plt.yticks([])\n        plt.suptitle(meth)\n        plt.show()\n\n[templateMatchFor(cropped_raw, sealion) for sealion in sealions]\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"b30ef88a-65cc-7c4b-8acc-1e28931c4a10"},"source":"Ok, so by hand selecting a few of the sea lions as templates and matching them against our image with three different methods does yield very good generalization, in fact it did very poor, we only detected nine sea lions. :)\nBy selecting only the color as a matching criterion, we've now detected a few more sea lions. Most probably though, this method is very sensitive to noise.\nThe template matching doesn't seem right for our task, the shape and color of each of the sea lions is very different after all."},{"cell_type":"markdown","metadata":{"_cell_guid":"6efa392a-03c4-71c3-bf9a-7882866edcb8"},"source":"### to be continued.. :)\n\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"9117c3d5-597a-4fcd-e468-eb2b814698de"},"outputs":[],"source":""}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0}