{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Extract more informations from images πΌοΈ\n\n## TL;DR: Remap the colors of the images in a smart way (see [π³&π¬ - πͺ Enhanced dataset [Ensemble of tricks]](https://www.kaggle.com/wolfy73/enhanced-dataset-ensemble-of-tricks) to add it to your pipeline)\nIt is now well establish that a lot of the work to be done in this competition is in the data processing side of the pipeline. Models are known to work better on images of bigger resolutions (in my opinion because a lot of features useful for identification are small relative to the image). Also, numerous methods have been thought of to crop the images and maximizing the effective resolution used to achieve the task. \n\nHowever I believe it is still not enough and that we can do better.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:34.863025Z","iopub.execute_input":"2022-02-22T09:30:34.863563Z","iopub.status.idle":"2022-02-22T09:30:35.249614Z","shell.execute_reply.started":"2022-02-22T09:30:34.863350Z","shell.execute_reply":"2022-02-22T09:30:35.248725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BASE_PATH = \"../input/happywhale-enhanced-dataset-v1\"\ndata = pd.read_csv(os.path.join(BASE_PATH, \"train.csv\"))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:35.251595Z","iopub.execute_input":"2022-02-22T09:30:35.251989Z","iopub.status.idle":"2022-02-22T09:30:35.378537Z","shell.execute_reply.started":"2022-02-22T09:30:35.251936Z","shell.execute_reply":"2022-02-22T09:30:35.377638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The initial idea π‘\nThe idea of this notebook first came to me when I realized that there are some details on the individuals that are just noticable in some pictures. I hypothesized that we could enhance the details by changing the contrast of the images","metadata":{}},{"cell_type":"code","source":"image = cv2.imread(os.path.join(BASE_PATH, \"train_images\", data[\"image\"].iloc[0]))[:, :, ::-1]\nplt.figure(figsize=(2*3, 1*3))\n\nplt.subplot(1, 2, 1)\nplt.imshow(image)\nplt.title(\"Original image\")\nplt.axis(\"off\")\n\nalpha = 3\nbeta = -50\nimage = np.clip(alpha*image.astype(np.float32) + beta, 0, 255).astype(np.uint8)\nplt.subplot(1, 2, 2)\nplt.imshow(image)\nplt.title(\"More contrast, less brightness\")\nplt.axis(\"off\")\nplt.tight_layout()\nplt.savefig('contrast.png')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:35.381355Z","iopub.execute_input":"2022-02-22T09:30:35.382433Z","iopub.status.idle":"2022-02-22T09:30:35.952662Z","shell.execute_reply.started":"2022-02-22T09:30:35.382384Z","shell.execute_reply":"2022-02-22T09:30:35.951779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we start noticing some more details:\n
\n\n
\n \nGreat ! so now we just have to increase contrast on all images of the dataset and we are done ... Right ?","metadata":{}},{"cell_type":"code","source":"def plot_images_contrast(batch, row=2, col=2, alpha=3, beta=-50, base_path=os.path.join(BASE_PATH, \"train_images\")):\n \"\"\"\n Copied and adapted from https://www.kaggle.com/awsaf49/happywhale-data-distribution\n \"\"\"\n plt.figure(figsize=(col*3, row*3))\n for i in range(row*col):\n plt.subplot(row, col, i+1)\n img = cv2.imread(os.path.join(base_path, batch[\"image\"].iloc[i]))\n max_size = 400\n if img.shape[0] > max_size or img.shape[1] > max_size:\n factor = max(img.shape[0] / max_size, img.shape[1] / max_size)\n img = cv2.resize(img, (int(round(img.shape[1] / factor)), int(round(img.shape[0] / factor))))\n if img is None:\n continue\n img = img[:, :, ::-1]\n img = np.clip(alpha*img.astype(np.float32) + beta, 0, 255).astype(np.uint8)\n plt.imshow(img)\n if \"species\" in batch:\n plt.title(batch[\"species\"].iloc[i])\n plt.axis('off')\n plt.tight_layout()\n plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:35.954814Z","iopub.execute_input":"2022-02-22T09:30:35.955078Z","iopub.status.idle":"2022-02-22T09:30:35.969460Z","shell.execute_reply.started":"2022-02-22T09:30:35.955043Z","shell.execute_reply":"2022-02-22T09:30:35.968420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images_contrast(data, row=4, col=4)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T09:30:35.970640Z","iopub.execute_input":"2022-02-22T09:30:35.970982Z","iopub.status.idle":"2022-02-22T09:30:37.580337Z","shell.execute_reply.started":"2022-02-22T09:30:35.970954Z","shell.execute_reply":"2022-02-22T09:30:37.579750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well... Not really... \nTo understand what is going on, we have to take a step back","metadata":{}},{"cell_type":"markdown","source":"# A bit of theory π\nTo grasp the principle behind contrast, we must have a look at the pixels histogram of an image. Let's show an example in greyscale for simplicity:","metadata":{}},{"cell_type":"code","source":"image = cv2.imread(os.path.join(BASE_PATH, \"train_images\", data[\"image\"].iloc[1]), cv2.IMREAD_GRAYSCALE)\nplt.imshow(np.repeat(image, 3).reshape((*image.shape, 3)))\nplt.title(\"Original image\")\nplt.axis(\"off\")\nplt.show()\nplt.title(\"Histogram of the pixels of the image\")\nplt.hist(image.ravel(), bins=256)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:37.581349Z","iopub.execute_input":"2022-02-22T09:30:37.581641Z","iopub.status.idle":"2022-02-22T09:30:38.467533Z","shell.execute_reply.started":"2022-02-22T09:30:37.581608Z","shell.execute_reply":"2022-02-22T09:30:38.466690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This histogram shows the count of pixels having a given value. For this particular histogram, we see two sort of \"spikes\":\n1. One around ~50 intensity, which would correspond to the pixels of the whale\n2. A second one between ~150 and ~200 which would correspond to the rest of the image (i.e. the water)\n\nLet's see how the histogram evolve when we increase the contrast:","metadata":{}},{"cell_type":"code","source":"alpha = 3.0\nbeta = 0.0\nimage_more_contrast = np.clip(alpha*image.astype(np.float32) + beta, 0, 255).astype(np.uint8)\nplt.imshow(np.repeat(image_more_contrast, 3).reshape((*image_more_contrast.shape, 3)))\nplt.title(\"Contrast increased\")\nplt.axis(\"off\")\nplt.show()\nplt.title(\"Histogram of the pixels of the image\")\n# I removed the fully white pixels otherwise we couldn't see the rest of the distribution\nplt.hist(image_more_contrast[image_more_contrast < 255].ravel(), bins=256)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:38.468646Z","iopub.execute_input":"2022-02-22T09:30:38.468867Z","iopub.status.idle":"2022-02-22T09:30:39.209501Z","shell.execute_reply.started":"2022-02-22T09:30:38.468840Z","shell.execute_reply":"2022-02-22T09:30:39.208607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that now the image is mainly white pixels (I removed them from the histogram for clarity). If we have a look at the histogram we see that now the \"whale\" spike covers almost all the pixels values while the \"water\" spike has been transformed into white pixels. \n\nSo if we see more details, it's because the pixels of the whale are distributed across a wider range of intensities than previously. \n\n## Intermediate conclusion\n*To extract more informations from the images we have to make the pixels of the individual be spreaded across a wider range of intensities.* \n\nBut as we can see we can't do the same transformation for every images because on some of them the pixels we are interested in will be sent in the white region where nothing is distinguishable at all... ","metadata":{}},{"cell_type":"markdown","source":"# What's the plan then ? πΊοΈ\nNow we have several possibilities more or less complicated:\n1. If we have access to a pixel segmentation model, we could make an histogram of only the pixels we are interested in and transform the image such that those pixels take the total range of values\n2. Maximizing the information in the entire image, this would be less effective but at least it do not require anything other than a few lines of codes\n3. A bit of bot: If, for example, we have a way to make bounding boxes for the dataset we can crop the images such that *most* of the pixels belong to the individual we want to identify and then apply a global maximizing information method\n\nAs some effort have been made to create proper bounding boxes models for this competition, we can try the 3rd option which is probably the best balance. But if you, dear reader by any chance come across a segmentation model that works on this data, you will probably have much better results π","metadata":{}},{"cell_type":"markdown","source":"# Maximizing information in an image π¨βπ¬\nFor thoses who know a little bit about information theory, we want to remap the pixels values such that the distribution of the values of the pixels maximizes its entropy. (i.e. follows an uniform distribution) \n\nFor those who don't know about information theory: \nRemember the histogram I mentionned earlier ? We want it to look like that:","metadata":{}},{"cell_type":"code","source":"y = np.ones(256) / 256\nplt.title(\"Uniform distribution\")\nplt.plot(y)\nplt.fill_between(np.arange(256), 0, y)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:39.210722Z","iopub.execute_input":"2022-02-22T09:30:39.210927Z","iopub.status.idle":"2022-02-22T09:30:39.404114Z","shell.execute_reply.started":"2022-02-22T09:30:39.210901Z","shell.execute_reply":"2022-02-22T09:30:39.403306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## How ?\n### A bit of probabilities π \n* Let $p$ be the value of a pixel in an image and $p^{\\prime}$ be the value of the same pixel in the transformed image\n* Let $X$ be a random variable representing the intensity of a random pixel within the image\n* Let $\\mathcal{P}$ be the ensemble of pixels in the original image and $n$ its number of pixels\n\nThe transformation we are looking for is:\n$$\n p^{\\prime} = 255 \\times \\mathbb{P}\\left(X \\leq p\\right) \\\\\n p^{\\prime} = 255 \\times \\frac{1}{n}\\sum_{x \\in \\mathcal{P}}\\mathbb{1}_{x \\leq p} \\\\\n$$\n\nWe can prove that this formula has the property I mentionned:\n$$\n \\mathbb{P}\\left(p^{\\prime} \\leq k\\right) = \\mathbb{P}\\left[255 \\times \\frac{1}{n}\\sum_{x \\in \\mathcal{P}}\\mathbb{1}_{x \\leq p} \\leq k\\right] \\\\\n \\mathbb{P}\\left(p^{\\prime} \\leq k\\right) = \\mathbb{P}\\left[\\frac{1}{n}\\sum_{x \\in \\mathcal{P}}\\mathbb{1}_{x \\leq p} \\leq \\frac{k}{255}\\right]\n$$\nNote that $\\frac{1}{n}\\sum_{x \\in \\mathcal{P}}\\mathbb{1}_{x \\leq p} \\sim \\mathcal{U}(0, 1)$, which gives us:\n$$\n \\mathbb{P}\\left(p^{\\prime} \\leq k\\right) = \\mathbb{P}\\left[\\frac{1}{n}\\sum_{x \\in \\mathcal{P}}\\mathbb{1}_{x \\leq p} \\leq \\frac{k}{255}\\right] = \\frac{k}{255}\\\\\n$$\nThus, if we note $F(x)$ the repartition function of $p^{\\prime}$ and $f(x)$ its density, we have:\n$$\n F(x) = \\frac{x}{255} \\\\\n f(x) = \\frac{\\partial F}{\\partial x} = \\frac{1}{255}\n$$\nBy identification, we showed that:\n$$\n p^{\\prime} \\sim \\mathcal{U}(0, 255)\n$$\nWe can now derive the following algorithm:","metadata":{}},{"cell_type":"code","source":"def remap_channel(image):\n # Argsort of the image to keep spatial information\n ids_sorted = np.argsort((image + np.random.random(image.shape) - 0.5).ravel())\n # Shades of grey\n values = np.floor(np.linspace(0.0, 256.0, num=len(ids_sorted), endpoint=False)).astype(np.uint8)\n s = image.shape\n image = image.ravel()\n # Reorder the shades of greyto look like the original image\n image[ids_sorted] = values\n image = image.reshape(s)\n return image\n\ndef remap_colors(image):\n \"\"\"\n The remapping is equivalent to create an image with n shades of grey and move the pixels in it such that it look like the original image\n \"\"\"\n if len(image.shape) == 2:\n return remap_channel(image)\n image[:, :, 0] = remap_channel(image[:, :, 0])\n image[:, :, 1] = remap_channel(image[:, :, 1])\n image[:, :, 2] = remap_channel(image[:, :, 2])\n return image","metadata":{"execution":{"iopub.status.busy":"2022-02-22T09:30:39.405566Z","iopub.execute_input":"2022-02-22T09:30:39.405780Z","iopub.status.idle":"2022-02-22T09:30:39.414349Z","shell.execute_reply.started":"2022-02-22T09:30:39.405754Z","shell.execute_reply":"2022-02-22T09:30:39.413572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"π¨ I am not sure yet how we should apply the same principle to colored images. So I intuitively thought about performing the mapping individually on each color channel but there is no theoretical reason for that nor evidence that it's the best way to deal with them...","metadata":{}},{"cell_type":"markdown","source":"## Test this on real images πΌοΈ\nLet's see another example image:","metadata":{}},{"cell_type":"code","source":"image = cv2.imread(os.path.join(BASE_PATH, \"train_images\", data[\"image\"].iloc[16]))[:, :, ::-1]\nplt.imshow(image)\nplt.title(\"Original image\")\nplt.axis(\"off\")\nplt.show()\nplt.title(\"Histogram of the pixels of the original image\")\nplt.hist(image.ravel(), bins=256)\nplt.show()\nimage = remap_colors(image)\nplt.title(\"Transformed image\")\nplt.imshow(image)\nplt.axis(\"off\")\nplt.show()\nplt.title(\"Histogram of the pixels of the transformed image\")\nplt.hist(image.ravel(), bins=256)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:39.416726Z","iopub.execute_input":"2022-02-22T09:30:39.416970Z","iopub.status.idle":"2022-02-22T09:30:41.165449Z","shell.execute_reply.started":"2022-02-22T09:30:39.416942Z","shell.execute_reply":"2022-02-22T09:30:41.164540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Amazing ! We have succesfully remaped the colors to have a uniform histogram. And the results looks visually good. \nLet's try to apply it to more images now:","metadata":{}},{"cell_type":"code","source":"def plot_images_color_remap(batch, row=2, col=2, alpha=3, beta=-50, base_path=os.path.join(BASE_PATH, \"train_images\")):\n \"\"\"\n Copied and adapted from https://www.kaggle.com/awsaf49/happywhale-data-distribution\n \"\"\"\n plt.figure(figsize=(col*3, row*3))\n for i in range(row*col):\n plt.subplot(row, col, i+1)\n img = cv2.imread(os.path.join(base_path, batch[\"image\"].iloc[i]))\n max_size = 400\n if img.shape[0] > max_size or img.shape[1] > max_size:\n factor = max(img.shape[0] / max_size, img.shape[1] / max_size)\n img = cv2.resize(img, (int(round(img.shape[1] / factor)), int(round(img.shape[0] / factor))))\n if img is None:\n continue\n img = img[:, :, ::-1]\n img = remap_colors(img)\n plt.imshow(img)\n if \"species\" in batch:\n plt.title(batch[\"species\"].iloc[i])\n plt.axis('off')\n plt.tight_layout()\n plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:41.166476Z","iopub.execute_input":"2022-02-22T09:30:41.166673Z","iopub.status.idle":"2022-02-22T09:30:41.179191Z","shell.execute_reply.started":"2022-02-22T09:30:41.166649Z","shell.execute_reply":"2022-02-22T09:30:41.178339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images_color_remap(data, row=4, col=4)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-22T09:30:41.180936Z","iopub.execute_input":"2022-02-22T09:30:41.181575Z","iopub.status.idle":"2022-02-22T09:30:43.506117Z","shell.execute_reply.started":"2022-02-22T09:30:41.181530Z","shell.execute_reply":"2022-02-22T09:30:43.505272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I consider it good enough to give it a try in my pipeline now !","metadata":{}},{"cell_type":"markdown","source":"# Limitations π¨\nI would like though to point out a certain number of limitations:\n1. The less water in the image, the better. Ideally we would like to only take into account the pixels of the individual when computing the remaping, that's why I talked about using a segmentation model earlier\n2. Some information is lost such as the overall colors of the image so we could try pipelines that incorporate both the initial image and the remaped one\n3. To work out, this implementation needs that the pixels values within the original image are unique. To do so, I'm adding random noise, but it could be better to resize the image with interpolation method such as the cubic one, without retresholding it. I currently am not aware of an efficient implementation in python of a resizing algorithm that outputs pixels values as floating points...\n4. How to deal properly with colored images is stil unresolved, I will probably solve this question via experiments\n5. If you are using this trick with a pretrained model, keep in mind that they expect \"natural\" images, and that the transformed images are not coming from the \"natural\" images distribution, so I'm not sure what will happen","metadata":{}},{"cell_type":"markdown","source":"# Recolor the dataset π¨\nNow you may want to incorporate this trick into your pipeline. A complete code can de found in this notebook: [π³&π¬ - πͺ Enhanced dataset [Ensemble of tricks]](https://www.kaggle.com/wolfy73/enhanced-dataset-ensemble-of-tricks)","metadata":{}},{"cell_type":"markdown","source":"# Conclusion π€·\nWe discovered a method to enhance the contrast in images and maximize the information in the image that you can integrate into your pipeline. Some limitations and interrogations are still remaining, I will try to improve the method in futur versions.\n\nπ If you found this notebook helpful or learned something please consider giving an upvote, and if you disagree with the content, I'll be pleased to dicsuss it with you in the comments.\n\nπ Happy Kaggling everyone !","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}}]}
![](\"https://i.imgur.com/KBgLGxJ.png\"/)
\n