{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Massive Post on Image Processing And Preparation For Deep Learning in Python\n## Manipulate and transform images at will\n\n\n \n Photo by \n Prasad Panchakshari\n on \n Unsplash\n \n","metadata":{}},{"cell_type":"markdown","source":"## Introduction","metadata":{}},{"cell_type":"markdown","source":"We are here on a sad business. Very sad, indeed. We are here to learn how to take beautiful, breathtaking images and turn them into bunch of ugly little numbers so that they are more presentable to all those soulless, mindless machines. \n\nWe will take animals and strip them of their color, making them black and white. Grab flowers with vivid colors and rob them of their beauty. We will look at disturbing images of XRays and see ways to make them even more disturbing. Sometimes, we might even have fun by drawing coins using a computer algorithm. \n\nIn other words, we will learn how to perform image processing. And our library of honor will be Scikit-Image (Skimage) throughout the article.","metadata":{}},{"cell_type":"markdown","source":"For some reason, the font size in code cells are becoming smaller and smaller towards the end. You can read the notebook as a Medium article [here](https://towardsdatascience.com/massive-tutorial-on-image-processing-and-preparation-for-deep-learning-in-python-1-e534ee42f122?source=your_stories_page----------------------------------------).","metadata":{}},{"cell_type":"markdown","source":"## Setup","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import warnings\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport skimage # pip install scikit-image\n\nwarnings.filterwarnings(\"ignore\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show(image: np.ndarray, title=\"Image\", cmap_type=\"gray\", axis=False):\n \"\"\"\n A function to display np.ndarrays as images\n \"\"\"\n plt.imshow(image, cmap=cmap_type)\n plt.title(title)\n if not axis:\n plt.axis(\"off\")\n plt.margins(0, 0)\n plt.show();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compare(\n original,\n filtered,\n title_filtered=\"Filtered\",\n cmap_type=\"gray\",\n axis=False,\n title_original=\"Original\",\n):\n fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 8), sharex=True, sharey=True)\n\n ax1.imshow(original, cmap=cmap_type)\n ax1.set_title(title_original)\n\n ax2.imshow(filtered, cmap=cmap_type)\n ax2.set_title(title_filtered)\n\n if not axis:\n ax1.axis(\"off\")\n ax2.axis(\"off\")\n plt.subplots_adjust(top=1, bottom=0, right=1, left=0, hspace=0, wspace=0.01)\n plt.margins(0, 0)\n plt.show();","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Table of Contents \n\n#### 1. [Basics](#basics)\n1. [What is an image?](#image)\n2. [Image basics with NumPy and Skimage](#numpy_basics)\n3. [Common transformations](#common_trans)\n4. [Histogram of color channels](#histogram)\n\n#### 2. [Filters](#filters)\n1. [Manual thresholding](#threshold_manual)\n2. [Thresholding - global](#global)\n3. [Thresholding - local](#local)\n4. [Edge detection](#edge)\n5. [Smoothing](#smooth)\n6. [Contrast enhancement](#contrast)\n7. [Transformations](#transformations)\n\n#### 3. [Image restoration and enhancement](#restoration)\n1. [Inpainting](#inpainting)\n2. [Noise📣](#noise)\n3. [Reducing noise - denoising](#denoise)\n4. [Superpixels and Segmentation](#superpixel)\n5. [Contours](#contour)\n\n#### 4. [Advanced operations](#advanced)\n1. [Edge detection](#edge2)\n2. [Corner detection](#corner)","metadata":{}},{"cell_type":"markdown","source":"## Basics ","metadata":{}},{"cell_type":"markdown","source":"### 1. What is an image? ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Image data is probably the most common after text. So, how does a computer understand that selfie of you in front the Eiffel Tower?\n\nIt uses a grid of small square units called pixels. A single pixel covers a small area and has a value that represents color. The more pixels in an image, the higher its quality and more memory it takes to store. \n\nThat's it. Image processing is mostly about manipulating these individual pixels (or sometimes, groups of them) so that computer vision algorithms can extract more information from them. ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 2. Image basics with NumPy and Skimage ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Images are loaded as NumPy ndarrays in both Matplotlib and Skimage. ","metadata":{}},{"cell_type":"code","source":"from skimage.io import imread # pip install scikit-image\n\nimage = imread(\"../input/notebook-images/colorful_scenery.jpg\")\n\ntype(image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As always, NumPy arrays bring flexibility, speed and power into the game. Image processing is no different. \n\nNdarrays make it easy to retrieve general details about the image, like its dimensions:","metadata":{}},{"cell_type":"code","source":"image.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image.ndim","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of pixels\nimage.size # 853 * 1280 * 3","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our hidden `image` is 853 pixels in height and 1280 in width. The third dimension denotes the value of the RGB (red, green, blue) color channel. Most common images formats are in 3D. \n\nYou can retrieve individual pixel values via regular NumPy indexing. Below, we try to index the image to retrieve each of the three color channels:","metadata":{}},{"cell_type":"code","source":"red = image[:, :, 0]\n\ncompare(image, red, \"Red Channel of the Image\", cmap_type=\"Reds_r\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"green = image[:, :, 1]\n\ncompare(image, green, \"Green Channel of the Image\", \"Greens_r\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"blue = image[:, :, 2]\n\ncompare(image, blue, \"Blue Channel of the Image\", \"Blues_r\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"0 for red, 1 for green and 2 for blue channel - easy enough. \n\nI've created two functions, [`show`](https://gist.github.com/BexTuychiev/e65c222f6fa388b22d7cf80eb561a1f4) and [`compare`]() which show an image or display two of them side by side for comparison. We will be using both functions extensively throughout the tutorial, so you might want to check out their code I hyperlinked.","metadata":{}},{"cell_type":"markdown","source":"By convention, the third dimension of the ndarray is for the color channel but this convention isn't always followed. Whenever you find it so, Skimage usually provides parameters to specify this behavior. \n\nImages are unlike the usual Matplotlib plots. Their origin isn't located in the bottom left, but at the position `(0, 0)`, the top left.","metadata":{}},{"cell_type":"code","source":"show(image, axis=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When we plot images in Matplotlib, axes denote the ordering of the pixels but we will usually be hiding them, since they don't deliver much value to the viewer.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 3. Common transformations ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"The most common image transformation we will be performing is converting color images to grayscale. Many image processing algorithms require grayscale, 2D arrays because color isn't the defining feature of images and computers can already extract enough information without it.","metadata":{}},{"cell_type":"code","source":"from skimage.color import rgb2gray\n\nimage = imread(\"../input/notebook-images/grayscale_example.jpg\")\n# Convert image to grayscale\ngray = rgb2gray(image)\n\ncompare(image, gray, \"Grayscale Image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gray.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When we convert images to grayscale, they lose their 3rd dimension - the color channel. Instead, each cell in the image array now represents an integer in `uint8` type. They range from 0 to 255, giving 256 shades of gray. ","metadata":{}},{"cell_type":"markdown","source":"You can also use NumPy functions like [`np.flipud`](https://numpy.org/doc/stable/reference/generated/numpy.flipud.html) or [`np.fliplr`](https://numpy.org/doc/stable/reference/generated/numpy.fliplr.html#numpy.fliplr) at your heart's desire to manipulate images in any way a NumPy array can be manipulated.","metadata":{}},{"cell_type":"code","source":"kitten = imread(\"../input/notebook-images/horizontal_flip.jpg\")\nhorizontal_flipped = np.fliplr(kitten)\n\ncompare(kitten, horizontal_flipped, \"Horizontally Flipped Image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ball = imread(\"../input/notebook-images/upside_down.jpg\")\nvertically_flipped = np.flipud(ball)\n\ncompare(ball, vertically_flipped, \"Vertically Flipped Image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the [`color` module](https://scikit-image.org/docs/dev/api/skimage.color.html), you can find many other transformation functions to work with colors in images.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 4. Histogram of color channels ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Sometimes, it is useful to look at the intensity of each color channel to get a feel of the color distributions. We can do so by slicing each color channel and plotting their histograms. Here is a function to perform this operation:","metadata":{}},{"cell_type":"code","source":"def plot_with_hist_channel(image, channel):\n channels = [\"red\", \"green\", \"blue\"]\n channel_idx = channels.index(channel)\n color = channels[channel_idx]\n\n extracted_channel = image[:, :, channel_idx]\n fig, (ax1, ax2) = plt.subplots(\n ncols=2, figsize=(18, 6)\n ) # , sharex=True, sharey=True)\n\n ax1.imshow(image)\n ax1.axis(\"off\")\n ax2.hist(extracted_channel.ravel(), bins=256, color=color)\n ax2.set_title(f\"{channels[channel_idx]} histogram\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Apart from the few Matplotlib details, you should pay attention to the call of the `hist` function. Once we extract the color channel and its array, we flatten it into 1D array and then pass it to the `hist` function. The number of bins should be 256, one for every pixel value - 0 being pitch black and 255 being fully white.\n\nLet's use the function for our colorful scenery image:","metadata":{}},{"cell_type":"code","source":"colorful_scenery = imread(\"../input/notebook-images/colorful_scenery.jpg\")\n\nplot_with_hist_channel(colorful_scenery, \"red\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_hist_channel(colorful_scenery, \"green\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_with_hist_channel(colorful_scenery, \"blue\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can also use histograms to find out the lightness in the image after converting it to a grayscale:","metadata":{}},{"cell_type":"code","source":"gray_color_scenery = rgb2gray(colorful_scenery)\n\nplt.hist(gray_color_scenery.ravel(), bins=256);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most pixels have lower values as the scenery image is a bit darker.\n\nWe will explore more applications of histograms in the next sections.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"## Filters ","metadata":{}},{"cell_type":"markdown","source":"### 1. Manual thresholding ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Now, we arrive at the fun stuff - filtering images. The first operation we will learn is thresholding. Let's load an example image:","metadata":{}},{"cell_type":"code","source":"stag = imread(\"../input/notebook-images/binary_example.jpg\")\n\nshow(stag)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thresholding has many applications in image segmentation, object detection, finding edges or contours, etc. It is mostly used to differentiate the background and foreground of an image.\n\nThresholding works best on high contrast grayscale images, so we will convert the stag image:","metadata":{}},{"cell_type":"code","source":"# Convert to graysacle\nstag_gray = rgb2gray(stag)\n\nshow(stag_gray)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will start with basic manual thresholding and move on to automatic. \n\nFirst, we look at the mean value of all pixels in the gray image:","metadata":{}},{"cell_type":"code","source":"stag_gray.mean()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Note that the above gray image's pixels are normalized between 0 and 1 by dividing all their values by 256. ","metadata":{}},{"cell_type":"markdown","source":"We obtain a mean of 0.2 which gives us a preliminary idea for the threshold we might want to use. \n\nNow, we use this threshold to mask the image array. If the pixel value is lower than the threshold, its value becomes 0 - black or 1 - white if otherwise. In other words, we get a black and white, binary picture:","metadata":{}},{"cell_type":"code","source":"# Set threshold\nthreshold = 0.35\n# Binarize\nbinary_image = stag_gray > threshold\n\ncompare(stag, binary_image, \"Binary image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this version, we can differentiate the outline of the stag more clearly. We can reverse the mask so that the background turns white:","metadata":{}},{"cell_type":"code","source":"inverted_binary = stag_gray <= threshold\n\ncompare(stag, inverted_binary, \"Binary image inverted\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 2. Thresholding - global ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"While it might be fun try out different thresholds and seeing their effect on the image, we usually perform thresholding by using an algorithm, which we will be more robust than our eyeball estimates. \n\nThere are many thresholding algorithms, so it might be hard to choose one. In this case, `skimage` has `try_all_threshold` function which runs 7 thresholding algorithms on the given *grayscale* image. Let's load an example and convert it:","metadata":{}},{"cell_type":"code","source":"flower = imread(\"../input/notebook-images/global_threshold_ex.jpg\")\n\nflower_gray = rgb2gray(flower)\n\ncompare(flower, flower_gray)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will see if we can refine the tulips' features by using thresholding:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import try_all_threshold\n\nfig, ax = try_all_threshold(flower_gray, figsize=(10, 8), verbose=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, some algorithms work better while others are horrible on this image. The `otsu` algorithm looks better, so we will continue using it.\n\nAt this point, I want to draw your attention back to the original tulip image:","metadata":{}},{"cell_type":"code","source":"show(flower)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The image has an uneven background because there is so much light coming from the window behind. We can confirm this by plotting a histogram of the gray tulip:","metadata":{}},{"cell_type":"code","source":"plt.hist(flower_gray.ravel(), bins=256);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, most pixels values are at the far end of the histogram, confirming that they are mostly bright. \n\nWhy is this important? Depending on the lightness of an image, the performance of thresholding algorithms also changes. For this reason, thresholding algorithms are divided into two types:\n\n1. Global - for images with even, uniform backgrounds\n2. Local - for images with different levels of brightness in different regions of the image.","metadata":{}},{"cell_type":"markdown","source":"The tulip image goes into the second category because the right part of the image is much brighter than the other half, making its background uneven. We can't use a global thresholding algorithm on it and which was the reason why the performance of all algorithms in [`try_all_threshold`](https://scikit-image.org/docs/dev/api/skimage.filters.html#skimage.filters.try_all_threshold) was so poor. \n\nWe will come back to the tulip example and local thresholding in just a bit. For now, we will load another examples with a much refined brightness and try to automatically threshold it:","metadata":{}},{"cell_type":"code","source":"spiral = imread(\"../input/notebook-images/otsu_example.jpg\")\nspiral_gray = rgb2gray(spiral)\n\ncompare(spiral, spiral_gray)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use a common global tresholding algorithm [`threshold_otsu`](https://scikit-image.org/docs/stable/api/skimage.filters.html#skimage.filters.threshold_otsu) in Skimage:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import threshold_otsu\n\n# Find optimal threshold with `threshold_otsu`\nthreshold = threshold_otsu(spiral_gray)\n\n# Binarize\nbinary_spiral = spiral_gray > threshold\n\ncompare(spiral, binary_spiral, \"Binarized Image w. Otsu Thresholding\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Works much better!","metadata":{}},{"cell_type":"markdown","source":"### 3. Thresholding - local ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Now, we will work with local thresholding algorithms. \n\nInstead of looking at the whole image, local algorithms focus on pixel neighborhoods to account for the uneven brightness in different regions. A common local algorithm in `skimage` is given as [`threshold_local`](https://scikit-image.org/docs/stable/api/skimage.filters.html#skimage.filters.threshold_local) function:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import threshold_local\n\nlocal_thresh = threshold_local(flower_gray, block_size=3, offset=0.0002)\n\nbinary_flower = flower_gray > local_thresh\n\ncompare(flower, binary_flower, \"Tresholded flower image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You have to play around with the `offset` argument to find the optimal image to your needs. `offset` is the constant that is subtracted from the mean of the local pixel neighborhood. This \"pixel neighborhood\" is determined with the `block_size` parameter in `local_threshold`, which denotes the number of pixels the algorithm looks around each point in each direction.\n\nObviously, it is a disadvantage to tune both `offset` and `block_size` but local thresholding is the only option that yields better results than manual or global thresholding. \n\nLet's try one more example:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import threshold_local\n\nhandwriting = imread(\"../input/notebook-images/chalk_writing.jpg\")\nhandwriting_gray = rgb2gray(handwriting)\n\n# Find optimal threshold using local\nlocal_thresh = threshold_local(handwriting_gray, offset=0.0003)\n\n# Binarize\nbinary_handwriting = handwriting_gray > local_thresh\n\ncompare(handwriting, binary_handwriting, \"Binarized image with local thresholding\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, the handwriting on the board is more refined after thresholding.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 4. Edge detection ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Edge detection is useful in many ways, such as identifying objects, extracting features from them, counting them and many more. We will start with the basic Sobel filter, which finds edges of objects in gray scale images. We will load an image of coins and use the Sobel filter on them:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import sobel\n\ncoins = imread(\"../input/notebook-images/coins_2.jpg\")\ncoins_gray = rgb2gray(coins)\n\ncoins_edge = sobel(coins_gray)\n\ncompare(coins, coins_edge, \"Images of coins with edges detected\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Sobel is pretty straightforward, you just have to call it on the gray image to get an output like above. We will see a more sophisticated version of Sobel in a later section. ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 5. Smoothing ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Another image filtering technique is smoothing. Many images like the chickens below, may contain random noise with no useful information to ML and DL algorithms. \n\nFor example, the hairs around the chickens add noise to the image, which may deviate the attention of ML models from the main objects themselves. In such scenarios, we use smoothing to blur the noise or edges and reduce contrast. ","metadata":{}},{"cell_type":"code","source":"chickens = imread(\"../input/notebook-images/chickens.jpg\")\n\nshow(chickens)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One of the most popular and powerful smoothing techniques is [`gaussian`](https://scikit-image.org/docs/stable/api/skimage.filters.html#skimage.filters.gaussian) smoothing:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import gaussian\n\nsmoothed = gaussian(chickens, multichannel=True, sigma=2)\n\ncompare(chickens, smoothed, \"An image smoothed with Gaussian smoothing\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can control the effect of the blur by tweaking the `sigma` argument. Don't forget to set `multichannel` to True if you are dealing with an RGB image. \n\nIf the image resolution is too high, the smoothing effect might not be visible to the naked eye but it will still be pronounced under the hood.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 6. Contrast enhancement ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Certain types of images like medical analysis results have low contrast, making it hard to spot details, like below:","metadata":{}},{"cell_type":"code","source":"xray = imread(\"../input/notebook-images/xray.jpg\")\nxray_gray = rgb2gray(xray)\n\ncompare(xray, xray_gray)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In such scenarios, we can use contrast enhancement to make the details more distinct. There are two types of contrast enhancement algorithms:\n\n1. Contrast stretching\n2. Histogram equalization\n\nWe will discuss histogram equalization in this post, which, in turn, has three types:\n\n1. Standard histogram equalization\n2. Adaptive histogram equalization\n3. Contrast Limited Adaptive Histogram Equalization (CLAHE)\n\n[Histogram equalization](https://en.wikipedia.org/wiki/Histogram_equalization) spreads out the areas with the highest contrast of an image to less bright regions, *equalizing it*.\n\n> Oh, by the way, you can calculate the contrast of an image by subtracting the lowest pixel value from the highest.","metadata":{}},{"cell_type":"code","source":"xray.max() - xray.min()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's try the standard histogram equalization from the `exposure` module:","metadata":{}},{"cell_type":"code","source":"from skimage.exposure import equalize_hist\n\nenhanced = equalize_hist(xray_gray)\n\ncompare(xray, enhanced)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can already see the details a lot more clearly. \n\nNext, we will use the CLAHE (this is a fun word to pronounce!) which computes many histograms for different pixel neighborhoods in an image, which results more detail even in the darkest of the regions:","metadata":{}},{"cell_type":"code","source":"from skimage.exposure import equalize_adapthist\n\n# Adjust clip_limit\nenhanced_adaptive = equalize_adapthist(xray_gray, clip_limit=0.4)\n\ncompare(xray, enhanced_adaptive, \"Image with contrast enhancement\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This one looks a lot better since it could show details in the background and also a couple more missing ribs in the bottom left. You can tweak `clip_limit` for more or less detail. ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 7. Transformations ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Images in your dataset might have several clashing characteristics, like different scales, unaligned rotations, etc. ML and DL algorithms expect your images to be of the same shape and dimensions. Therefore, you need to learn to how fix them.","metadata":{}},{"cell_type":"markdown","source":"**Rotations**","metadata":{}},{"cell_type":"markdown","source":"To rotate images, use the `rotate` function from the `transform` module. I've chosen actual clocks so you might remember the angle signs better:","metadata":{}},{"cell_type":"code","source":"from skimage.transform import rotate\n\nclock = imread(\"../input/notebook-images/clock.jpg\")\n\nclockwise = rotate(clock, angle=-60)\ncompare(clock, clockwise, \"Clockwise rotated image, use negative angles\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"anti_clockwise = rotate(clock, angle=33)\n\ncompare(clock, anti_clockwise, \"Anticlockwise rotated image, use positive angles\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Rescaling**","metadata":{}},{"cell_type":"markdown","source":"Another common operation is scaling images. It is mostly useful in cases where images are proportionally different from one another. \n\nWe use the similar [`rescale`](https://scikit-image.org/docs/stable/api/skimage.transform.html#skimage.transform.rescale) function for this operation:","metadata":{}},{"cell_type":"code","source":"butterflies = imread(\"../input/notebook-images/butterflies.jpg\")\nbutterflies.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from skimage.transform import rescale\n\nscaled_butterflies = rescale(butterflies, scale=3 / 4, multichannel=True)\n\ncompare(\n butterflies,\n scaled_butterflies,\n \"Butterflies scaled down by a factor of 3/4\",\n axis=True,\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When image resolution is high, downscaling it too much might result in quality loss or pixels rubbing together unceremoniously to create unexpected edges or corners. To account for this effect, you can set `anti_aliasing` to True which uses Gaussian smoothing under the hood:","metadata":{"tags":[]}},{"cell_type":"code","source":"factor_10_aa = rescale(butterflies, scale=1 / 10, multichannel=True, anti_aliasing=True)\nfactor_10_no_aa = rescale(butterflies, scale=1 / 10, multichannel=True)\n\ncompare(factor_10_aa, factor_10_no_aa, \"No anti-aliasing\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As before, the smoothing isn't noticeable but at a more granular level, it will be obvious.","metadata":{}},{"cell_type":"markdown","source":"**Resizing**","metadata":{}},{"cell_type":"markdown","source":"If you want the image to have specific width and height, rather than scaling it by a factor, you can use the [`resize`](https://scikit-image.org/docs/stable/api/skimage.transform.html#skimage.transform.resize) function by providing an `output_shape`:","metadata":{}},{"cell_type":"code","source":"from skimage.transform import resize\n\npuppies = imread(\"../input/notebook-images/puppies.jpg\")\n\n# Also possible to set anti_aliasing\npuppies_600_800 = resize(puppies, output_shape=(600, 800))\n\ncompare(puppies, puppies_600_800, \"Puppies image resized 600x800 (height, width)\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"## Image restoration and enhancement ","metadata":{}},{"cell_type":"markdown","source":"Some images might be distorted, damaged or lost during file transforms, in faulty downloads or in many other situations. Rather than giving up on the image, you can use `skimage` to account for the damage and make the image good as new. \n\nIn this section, we will discuss a few techniques for image restoration, starting with inpainting.","metadata":{}},{"cell_type":"markdown","source":"### 1. Inpainting ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"An inpainting algorithm can intelligently fill in the blanks in an image. I couldn't find a damaged image, so we will use this whale image and put a few blanks on it manually:","metadata":{}},{"cell_type":"code","source":"whale_image = imread(\"../input/happy-whale-and-dolphin/train_images/00206a224e68de.jpg\")\n\nshow(whale_image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"whale_image.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The below function creates four pitch black regions to simulate lost information on an image:","metadata":{}},{"cell_type":"code","source":"def make_mask(image):\n \"\"\"Create a mask to artificially defect the image.\"\"\"\n\n mask = np.zeros(image.shape[:-1])\n\n # Make 4 masks\n mask[250:300, 1400:1600] = 1\n mask[50:100, 300:433] = 1\n mask[300:380, 1000:1200] = 1\n mask[200:270, 750:950] = 1\n\n return mask.astype(bool)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the mask\nmask = make_mask(whale_image)\n\n# Apply the defect mask on the whale_image\nimage_defect = whale_image * ~mask[..., np.newaxis]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare(whale_image, image_defect, \"Artifically damaged image of a whale\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use the [`inpaint_biharmonic`](https://scikit-image.org/docs/stable/api/skimage.restoration.html#skimage.restoration.inpaint_biharmonic) function from the `inpaint` module to fill in the blanks, passing in the `mask` we created:","metadata":{}},{"cell_type":"code","source":"from skimage.restoration import inpaint\n\nrestored_image = inpaint.inpaint_biharmonic(\n image=image_defect, mask=mask, multichannel=True\n)\n\ncompare(\n image_defect,\n restored_image,\n \"Restored image after defects\",\n title_original=\"Faulty Image\",\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, it will be hard to tell where the defect regions are before seeing the faulty image.\n\nNow, let's make some noise📣!","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 2. Noise📣 ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"As discussed earlier, noise plays important role in image enhancement and restoration. Sometimes, you might intentionally add it to an image like below:","metadata":{}},{"cell_type":"code","source":"from skimage.util import random_noise\n\npup = imread(\"../input/notebook-images/pup.jpg\")\n\nnoisy_pup = random_noise(pup)\n\ncompare(pup, noisy_pup, \"Noise puppy image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use the [`random_noise`](https://scikit-image.org/docs/dev/api/skimage.util.html#skimage.util.random_noise) function to sprinkle an image with random specks of color. For this reason, the method is called \"salt and pepper\" technique.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 3. Reducing noise - denoising ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"But, most of the time, you want to remove noise from an image, rather than add it. There are a few types of denoising algorithms:\n\n1. Total variation (TV) filter\n2. Bilateral denoising\n3. Wavelet denoising \n4. Non-local mean denoising\n\nWe will only look at the first two in this article. Let's try TV filter first, which is available as [`denoise_tv_chambolle`](https://scikit-image.org/docs/stable/api/skimage.restoration.html#denoise-tv-chambolle):","metadata":{}},{"cell_type":"code","source":"from skimage.restoration import denoise_tv_chambolle\n\ndenoised_pup_tv = denoise_tv_chambolle(noisy_pup, weight=0.2, multichannel=True)\n\ncompare(\n noisy_pup,\n denoised_pup_tv,\n \"Total Variation Filter denoising applied\",\n title_original=\"Noisy pup\",\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The higher the resolution of the image, the longer it takes to denoise it. You can control the effect of denoising with the `weight` parameter. Now, let's try [`denoise_bilateral`](https://scikit-image.org/docs/stable/api/skimage.restoration.html#skimage.restoration.denoise_bilateral):","metadata":{}},{"cell_type":"code","source":"from skimage.restoration import denoise_bilateral\n\ndenoised_pup_bilateral = denoise_bilateral(noisy_pup, multichannel=True)\n\ncompare(noisy_pup, denoised_pup_bilateral, \"Bilateral denoising applied image\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It wasn't as effective as TV filter as can be seen below:","metadata":{}},{"cell_type":"code","source":"compare(\n denoised_pup_tv,\n denoised_pup_bilateral,\n \"Bilateral filtering\",\n title_original=\"TV filtering\",\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 4. Superpixels and Segmentation ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Image segmentation is one of the most fundamental and common topics in image processing. It is extensively used in motion and object detection, image classification and many more areas. \n\nWe've already seen an instance of segmentation - thesholding an image so that the background is extracted from the foreground. In this section, we will learn to do more than that such as segmenting image into similar areas.\n\nTo get started with segmentation, we need to understand a concept of superpixels.\n\nA pixel, on its own, just represents a small area of color. Once separated from the image, a single pixel will be useless. For this reason, segmentation algorithms use multiple groups of pixels that are similar in contrast, color or brightness. They are called superpixels. \n\nOne of the algorithms that tries to find superpixels is Simple Linear Iterative Cluster (SLIC), which uses k-Means clustering under the hood. Let's see how to use it on the coffee image available in the `skimage` library:","metadata":{}},{"cell_type":"code","source":"from skimage import data\n\ncoffee = data.coffee()\n\nshow(coffee)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use the [`slic`](https://scikit-image.org/docs/dev/api/skimage.segmentation.html?highlight=slic#skimage.segmentation.slic) function from the `segmentation` module:","metadata":{}},{"cell_type":"code","source":"from skimage.segmentation import slic\n\nsegments = slic(coffee)\nshow(segments)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`slic` finds 100 segments or labels by default. To put them back onto the image, we use the [`label2rgb`](https://scikit-image.org/docs/dev/api/skimage.color.html#skimage.color.label2rgb) function:","metadata":{}},{"cell_type":"code","source":"from skimage.color import label2rgb\n\nfinal_image = label2rgb(segments, coffee, kind=\"avg\")\nshow(final_image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's wrap this operation inside a function and try to use more segments:","metadata":{}},{"cell_type":"code","source":"from skimage.color import label2rgb\nfrom skimage.segmentation import slic\n\n\ndef segment(image, n_segments=100):\n # Obtain superpixels / segments\n superpixels = slic(coffee, n_segments=n_segments)\n\n # Put the groups on top of the original image\n segmented_image = label2rgb(superpixels, image, kind=\"avg\")\n\n return segmented_image","metadata":{"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Find 500 segments\ncoffee_segmented_2 = segment(coffee, n_segments=500)\n\ncompare(coffee, coffee_segmented_2, \"With 500 segments\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Segmentation will make it easier for computer vision algorithms to extract useful features from images.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 5. Contours ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Much of the information of an object resides in its shape. If we can detect an object's shape in lines or in contours, we can extract many useful data like object's size, its individual markings, etc.\n\nLet's see finding contours in practice using the image of dominoes. ","metadata":{}},{"cell_type":"code","source":"dominoes = imread(\"../input/notebook-images/dominoes.jpg\")\n\nshow(dominoes)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will see if we can isolate the tiles and circles using the [`find_contours`](https://scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.find_contours) function in `skimage`. This function requires a binary (black and white) image, so we have to threshold the image first. ","metadata":{}},{"cell_type":"code","source":"from skimage.measure import find_contours\n\n# Convert to grayscale\ndominoes_gray = rgb2gray(dominoes)\n# Find optimal threshold with treshold_otsu\nthresh = threshold_otsu(dominoes_gray)\n# Binarize\ndominoes_binary = dominoes_gray > thresh\n\ndomino_contours = find_contours(dominoes_binary)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The resulting array is a list of (n, 2) arrays representing the coordinates of the contour lines:","metadata":{}},{"cell_type":"code","source":"for contour in domino_contours[:5]:\n print(contour.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will wrap the operation inside a function called `mark_contours`:","metadata":{}},{"cell_type":"code","source":"from skimage.filters import threshold_otsu\nfrom skimage.measure import find_contours\n\n\ndef mark_contours(image):\n \"\"\"A function to find contours from an image\"\"\"\n gray_image = rgb2gray(image)\n # Find optimal threshold\n thresh = threshold_otsu(gray_image)\n # Mask\n binary_image = gray_image > thresh\n\n contours = find_contours(binary_image)\n\n return contours","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To plot the contour lines on the image, we will create another function called `plot_image_contours` that uses the above one:","metadata":{}},{"cell_type":"code","source":"def plot_image_contours(image):\n fig, ax = plt.subplots()\n\n ax.imshow(image, cmap=plt.cm.gray)\n\n for contour in mark_contours(image):\n ax.plot(contour[:, 1], contour[:, 0], linewidth=2, color=\"red\")\n\n ax.axis(\"off\")\n\n\nplot_image_contours(dominoes)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, we successfully detected the majority of the contours, but we can still see some random fluctuations in the center. Let's apply denoising before we pass the image of dominoes to our contour finding function:","metadata":{}},{"cell_type":"code","source":"dominoes_denoised = denoise_tv_chambolle(dominoes, multichannel=True)\n\nplot_image_contours(dominoes_denoised)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's it! We eliminated most of the noise which was causing the incorrect contour lines!","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"## Advanced operations ","metadata":{}},{"cell_type":"markdown","source":"### 1. Edge detection ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Before, we used the Sobel algorithm to detect edges of objects. Here, we will use the Canny algorithm which is more widely used for it is more faster and more accurate. As always, the function [`canny`](https://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.canny) requires a grayscale image. \n\nThis time we will use an image with more coins, hence more edges to detect:","metadata":{}},{"cell_type":"code","source":"coins_3 = imread(\"../input/notebook-images/coins_3.jpg\")\n\n# Convert to gray\ncoins_3_gray = rgb2gray(coins_3)\n\ncompare(coins_3, coins_3_gray)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To find edges, we just pass the image to the `canny` function:","metadata":{}},{"cell_type":"code","source":"from skimage.feature import canny\n\n# Find edges with canny\ncanny_edges = canny(coins_3_gray)\n\ncompare(coins_3, canny_edges, \"Edges detected with Canny algorithm\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The algorithm certainly found almost all coins edges but it is very noisy because the engravings on the coins are detected as well. We can reduce the sensitivity of `canny` by tweaking the `sigma` parameter:","metadata":{}},{"cell_type":"code","source":"canny_edges_sigma_2 = canny(coins_3_gray, sigma=2.5)\n\ncompare(coins_3, canny_edges_sigma_2, \"Edges detected using Canny with less intensity\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, `canny` now only finds the general outline of the coins. ","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### 2. Corner detection ","metadata":{"tags":[]}},{"cell_type":"markdown","source":"Another important image processing technique is corner detection. Corners can be key features of objects in image classification. \n\nTo find corners, we will use the Harris corner detection algorithm. Let's load a sample image and convert it to grayscale:","metadata":{}},{"cell_type":"code","source":"windows = imread(\"../input/notebook-images/windows.jpg\")\n\nwindows_gray = rgb2gray(windows)\n\ncompare(windows, windows_gray)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use the [`corner_harris`](https://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.corner_harris) function to produce a measure image that masks the areas where corners are. ","metadata":{}},{"cell_type":"code","source":"from skimage.feature import corner_harris\n\nmeasured_image = corner_harris(windows_gray)\n\nshow(measured_image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will pass this masked measure image to `corner_peaks` function, which returns corner coordinates this time:","metadata":{}},{"cell_type":"code","source":"from skimage.feature import corner_peaks\n\ncorner_coords = corner_peaks(measured_image, min_distance=50)\nlen(corner_coords)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The function found 79 corners using a minimum distance of 50 pixels between each corner. Let's wrap the operation up to this point in a function:","metadata":{}},{"cell_type":"code","source":"def find_corner_coords(image, min_distance=50):\n # Convert to gray\n gray_image = rgb2gray(image)\n # Produce a measure image\n measure_image = corner_harris(gray_image)\n\n # Find coords\n coords = corner_peaks(measure_image, min_distance=min_distance)\n\n return coords","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will create another function that plots each corner using the coordinates produced from the above function:","metadata":{}},{"cell_type":"code","source":"def show_image_cornered(image):\n # Find coords\n coords = find_corner_coords(image)\n\n # Plot them on top of the image\n plt.imshow(image, cmap=\"gray\")\n plt.plot(coords[:, 1], coords[:, 0], \"+b\", markersize=15)\n plt.axis(\"off\")\n\n\nshow_image_cornered(windows)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, the algorithm isn't working as expected. Rather than finding the window corners, the marks are placed at the intersection of the bricks. These intersections are noise, making them useless. Let's denoise the image and pass it to the function once again:","metadata":{}},{"cell_type":"code","source":"windows_denoised = denoise_tv_chambolle(windows, multichannel=True, weight=0.3)\n\nshow_image_cornered(windows_denoised)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, this is much better! It ignored the brick edges and found the majority of window corners.","metadata":{}},{"cell_type":"markdown","source":"Back to top🔝","metadata":{}},{"cell_type":"markdown","source":"### Conclusion","metadata":{}},{"cell_type":"markdown","source":"Phew! What a post! Both you and me deserve a pat on the back!\n\nI had quite fun writing this post. In a real computer vision problem, you won't be using all of these at once, of course. As you may have noticed, things we learned today aren't very difficult. They take a few lines of code, at most. The difficult part is applying them to a real problem and actually improving the performance of your model.\n\nThat bit comes with hard work and practice, not nicely packaged inside a single article. Thank you for reading!\n\n**You can become a premium Medium member using the link below and get access to all of my stories and thousands of others:**\n\nhttps://ibexorigin.medium.com/membership\n\n**Or subscribe to my email list:**\n\nhttps://ibexorigin.medium.com/subscribe\n\n**You can reach out to me on [LinkedIn](https://twitter.com/BexTuychiev) or [Twitter](https://twitter.com/BexTuychiev) for a friendly chat about all things data.**","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}}]}