{"cells":[{"metadata":{"_uuid":"fe1ee0c221546a5e4757bdfe512c1c39ee157654","_cell_guid":"37c92f91-7569-4b03-a82a-236dff6fef9a","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":false,"col":0,"height":37,"row":0,"width":12},"report_default":{}},"version":1}}},"cell_type":"markdown","source":"\n# Ideas for Generating Image Features and Measuring Image Quality - with Multi-Processing and Data Partitioning\n\n
\n\n\n\n
\n\n[Avito](https://www.kaggle.com/c/avito-demand-prediction) is Russia's largest Advertisment firm. The quality of the advertisement image significantly affects the demand volume on an item. For both advertisers and Avito, it is important to use authentic high quality images. In this kernel, I have implemented some ideas which can be used to create new features related to images. These features are an indicatory factors about the Image Quality. Following is the list of feature ideas: \n\n\n### 1. Dullness : Is the Image Very Dull ? \n \n 1.1 Image Dullness Score\n \n\n
\n"},{"metadata":{"extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":true},"report_default":{}},"version":1}},"_uuid":"5768a3c8f6f633403efdcce8398ac3eaa74ebde1","collapsed":true,"_cell_guid":"ccdc873b-d98f-40c8-939c-9725a74f262c","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from collections import defaultdict\nfrom scipy.stats import itemfreq\nfrom scipy import ndimage as ndi\nimport matplotlib.pyplot as plt\nfrom skimage import feature\nfrom PIL import Image as IMG\nimport numpy as np\nimport pandas as pd \nimport operator\nimport cv2\nimport os \n\n# add multiprocessing library\nimport multiprocessing\nfrom multiprocessing import Pool\n\nfrom IPython.core.display import HTML \nfrom IPython.display import Image\n\nimages_path = '../input/sampleavitoimages/sample_avito_images/'\nimgs = os.listdir(images_path)\n\nfeatures = pd.DataFrame()\nfeatures['image'] = imgs","execution_count":33,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"83fd18de85734acb84b215442cf9375b877d9025"},"cell_type":"code","source":"features.head(5)","execution_count":34,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"250e2b59bdd852848560fe43436cd6d2fb54ade8"},"cell_type":"code","source":"num_partitions = 2\nnum_cores = multiprocessing.cpu_count()\n\ndef parallelize_dataframe(df, func):\n a,b = np.array_split(df, num_partitions)\n pool = Pool(num_cores)\n df = pd.concat(pool.map(func, [a,b]))\n pool.close()\n pool.join()\n return df\n","execution_count":37,"outputs":[]},{"metadata":{"_uuid":"1d372df765926103668eeaa3e1b35e6cc9f0debf","_cell_guid":"ca91d69c-1652-4565-a167-7d8f0e6cb678","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":false,"col":0,"height":6,"row":37,"width":12},"report_default":{}},"version":1}}},"cell_type":"markdown","source":"## 1. Is the image Very Dull \n\n### Feature 1 : Dullness\n\nDull Images may not be good for the advirtisment purposes. The analysis of prominent colors present in the images can indicate a lot about if the image is dull or not. In the following cell, I have added a code to measure the dullness score of the image which can be used as one of the feature in the model. \n\n"},{"metadata":{"_uuid":"e20c9c69fd613f94e3b4b94a290d59776603f5a1","collapsed":true,"_cell_guid":"459760ce-8324-48a3-ba03-a2a38d20fea7","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":true},"report_default":{}},"version":1}},"trusted":true},"cell_type":"code","source":"def color_analysis(img):\n # obtain the color palatte of the image \n palatte = defaultdict(int)\n for pixel in img.getdata():\n palatte[pixel] += 1\n \n # sort the colors present in the image \n sorted_x = sorted(palatte.items(), key=operator.itemgetter(1), reverse = True)\n light_shade, dark_shade, shade_count, pixel_limit = 0, 0, 0, 25\n for i, x in enumerate(sorted_x[:pixel_limit]):\n if all(xx <= 20 for xx in x[0][:3]): ## dull : too much darkness \n dark_shade += x[1]\n if all(xx >= 240 for xx in x[0][:3]): ## bright : too much whiteness \n light_shade += x[1]\n shade_count += x[1]\n \n light_percent = round((float(light_shade)/shade_count)*100, 2)\n dark_percent = round((float(dark_shade)/shade_count)*100, 2)\n return light_percent, dark_percent","execution_count":28,"outputs":[]},{"metadata":{"_uuid":"d9b6724b5fdb52cf9a7504012337017d49c49344","_cell_guid":"48e163bf-c867-43b9-b61f-22aece750aaa","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":false,"col":0,"height":4,"row":43,"width":4},"report_default":{}},"version":1}}},"cell_type":"markdown","source":"Lets compute the dull score for the sample images from Avito's dataset "},{"metadata":{"_uuid":"f3cec207d871a5188fe4f78dc049af477b873981","collapsed":true,"_cell_guid":"2b547611-5247-4424-b05d-02c87750c669","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":false,"col":4,"height":7,"row":43,"width":4},"report_default":{}},"version":1}},"trusted":true},"cell_type":"code","source":"def perform_color_analysis(img, flag):\n path = images_path + img \n im = IMG.open(path) #.convert(\"RGB\")\n \n # cut the images into two halves as complete average may give bias results\n size = im.size\n halves = (size[0]/2, size[1]/2)\n im1 = im.crop((0, 0, size[0], halves[1]))\n im2 = im.crop((0, halves[1], size[0], size[1]))\n\n try:\n light_percent1, dark_percent1 = color_analysis(im1)\n light_percent2, dark_percent2 = color_analysis(im2)\n except Exception as e:\n return None\n\n light_percent = (light_percent1 + light_percent2)/2 \n dark_percent = (dark_percent1 + dark_percent2)/2 \n \n if flag == 'black':\n return dark_percent\n elif flag == 'white':\n return light_percent\n else:\n return None","execution_count":29,"outputs":[]},{"metadata":{"_uuid":"c4685c6501071e90cbd70f2476e97cd9d8cdd596","_cell_guid":"e36b0c53-d99b-41b7-a9b5-b3133e77f73f","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def score_dullness(data):\n data['dullness'] = data['image'].apply(lambda x : perform_color_analysis(x, 'black')) \n return data\n\nfeatures = parallelize_dataframe(features, score_dullness)\nfeatures.head()","execution_count":39,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c863c9ea7d798ffe847f619469963e274849bc0c"},"cell_type":"code","source":"topdull = features.sort_values('dullness', ascending = False)\ntopdull.head(5)","execution_count":40,"outputs":[]},{"metadata":{"_uuid":"b392a3958d5fab56539a0ae1350fd23d8afb7de7","_cell_guid":"0e009ae9-f83f-4a42-8bf7-eeeadd6ef331","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":false,"col":8,"height":4,"row":43,"width":4},"report_default":{}},"version":1}}},"cell_type":"markdown","source":"Lets plot some of the images with very high dullness"},{"metadata":{"_uuid":"88f5ca1819a615a208489c1c29c04090b3fbe449","_cell_guid":"ceac13b7-f269-477c-9598-db0948aba06a","extensions":{"jupyter_dashboards":{"views":{"grid_default":{"hidden":false,"col":0,"height":24,"row":47,"width":4},"report_default":{}},"version":1}},"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for j,x in topdull.head(2).iterrows():\n path = images_path + x['image']\n html = \"Image : \"+x['image']+\" (Dullness : \" + str(x['dullness']) +\")
\"\n display(HTML(html))\n display(IMG.open(path).resize((300,300), IMG.ANTIALIAS))","execution_count":41,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"extensions":{"jupyter_dashboards":{"views":{"grid_default":{"name":"grid","maxColumns":12,"defaultCellHeight":20,"cellMargin":10,"type":"grid"},"report_default":{"name":"report","type":"report"}},"version":1,"activeView":"grid_default"}}},"nbformat":4,"nbformat_minor":1}