{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","collapsed":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"# [Update] Image Classification Confidence and Deal Probability\nMost publicly shared models show that the `image_top_1` feature has one of the strongest signals in predicting deal probability.  Take a look at [SRK], [kxx] and [Bojan Tunguz][bojan] among many others.  However, this is a black box feature to us.  We don't know how it was built to use it as a guide for building similar features.  I took a stab at trying image based features and found one that might work as well.\n\nIf you viewed this kernel before, skip to [the last section below](#Correlation-with-Deal-Probability) for analysis of correlation between image classification confidence and deal probability.\n\n# Image Quality\nIn this competition, the quality of the advertisement image significantly affects the demand volume on an item.  Let's extract the dataset image features and see if we can use it to help predict demand.  I found out that the image classification confidence highly correlates with the deal probability.  However, we have to keep in mind that:\n\n- Not all advertisements have images.\n- Advertisements with images tend to have a higher deal probability.\n\nSome code and sections of this notebook were adapted from:\n- https://www.kaggle.com/classtag/lightgbm-with-mean-encode-feature-0-233\n- https://keras.io/applications/#classify-imagenet-classes-with-resnet50\n\n# Image Classification with Deep Learning\n[Keras] provides pre-trained deep learning models that will save us days annotating, and training our models.  We will just load one that suits our needs and use it to classify our images.  The assumption here is that the image classification accuracy score will reflect how clear it is for a human to identify it, and affect the chance of buying it.\n\nDifferent exploratory data analyses [[1], [2], [3]] in this competition show high correlation between the `image_top_1` feature and our target `deal_probability`.  So, we are proceeding in the right direction.\n\nWe will start by preparing our workspace and copying the large pretrained model files to where Keras can find them.\n\n[keras]: https://keras.io/applications/#resnet50\n[OpenCV]: https://docs.opencv.org/3.4.1/d2/d58/tutorial_table_of_content_dnn.html\n[1]: https://www.kaggle.com/shivamb/indepth-analysis-visualisations-avito-updated\n[2]: https://www.kaggle.com/classtag/lightgbm-with-mean-encode-feature-0-233\n[3]: https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-notebook-avito\n[SRK]: https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-notebook-avito\n[kxx]: https://www.kaggle.com/kailex/xgb-text2vec-tfidf-0-2240\n[bojan]: https://www.kaggle.com/tunguz/bow-meta-text-and-dense-features-lb-0-2241"},{"metadata":{"_cell_guid":"206c03a1-5ddc-4bc9-bb10-d45380768049","collapsed":true,"_uuid":"ea55a0629f55571eea4ae324ce9283c61cdfa011","trusted":false},"cell_type":"code","source":"\"\"\"Copy Keras pre-trained model files to work directory from:\nhttps://www.kaggle.com/gaborfodor/keras-pretrained-models\n\nCode from: https://www.kaggle.com/classtag/extract-avito-image-features-via-keras-vgg16/notebook\n\"\"\"\nimport os\n\ncache_dir = os.path.expanduser(os.path.join('~', '.keras'))\nif not os.path.exists(cache_dir):\n    os.makedirs(cache_dir)\n\n# Create symbolic links for trained models.\n# Thanks to Lem Lordje Ko for the idea\n# https://www.kaggle.com/lemonkoala/pretrained-keras-models-symlinked-not-copied\nmodels_symlink = os.path.join(cache_dir, 'models')\nif not os.path.exists(models_symlink):\n    os.symlink('/kaggle/input/keras-pretrained-models/', models_symlink)\n\nimages_dir = os.path.expanduser(os.path.join('~', 'avito_images'))\nif not os.path.exists(images_dir):\n    os.makedirs(images_dir)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b5e54794-353f-4228-8df4-e6a71a8469c0","_uuid":"d52762316eff8953f49a2d08a0954f70a57b3584"},"cell_type":"markdown","source":"Due to Kaggle's disk space restrictions, we will only extract a few images to classify here.  Keep in mind that the pretrained models take almost 650 MB disk space."},{"metadata":{"_cell_guid":"9989aede-9a3b-4df8-901c-506d20a4fb2f","collapsed":true,"_uuid":"d1fcb50be83400486d6c1c3eb05a2d7890de43a9","trusted":false},"cell_type":"code","source":"\"\"\"Extract images from Avito's advertisement image zip archive.\n\nCode adapted from: https://www.kaggle.com/classtag/extract-avito-image-features-via-keras-vgg16/notebook\n\"\"\"\nimport zipfile\n\nNUM_IMAGES_TO_EXTRACT = 1000\n\nwith zipfile.ZipFile('../input/avito-demand-prediction/train_jpg.zip', 'r') as train_zip:\n    files_in_zip = sorted(train_zip.namelist())\n    for idx, file in enumerate(files_in_zip[:NUM_IMAGES_TO_EXTRACT]):\n        if file.endswith('.jpg'):\n            train_zip.extract(file, path=file.split('/')[3])\n\n!mv *.jpg/data/competition_files/train_jpg/* ~/avito_images\n!rm -rf *.jpg","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e70e1894-5723-460b-8940-4bdfc18c60d9","_uuid":"299f1fe8ac83c837d29362d5431fa51472803f45"},"cell_type":"markdown","source":"# ResNet50 *vs* InceptionV3 *vs* Xception\nLet's compare the performance of three pretrained deep learning models implmented for [Keras].\n\nThe models, [ResNet50], [InceptionV3] and [Xception], are all pre-trained on the [ImageNet] dataset.  Here we initialize them and plot a few images from our Avito's image set and the probability of their top classifications.\n\n**[ImageNet]** is a research project to develop a large image dataset with annotations, such as standard labels and descriptions.  The dataset has been used in the annual [ILSVRC] image classification challenge.  A few of the winners published their pretrained models with the research community, and we are going to use some of them here.\n\n[resnet50]: https://keras.io/applications/#resnet50\n[VGG16]: https://keras.io/applications/#vgg16\n[Xception]: https://keras.io/applications/#xception\n[InceptionV3]: https://keras.io/applications/#inceptionv3\n[Keras]: https://keras.io/applications/\n[ImageNet]: http://www.image-net.org/\n[ILSVRC]: http://image-net.org/challenges/LSVRC/2017/index"},{"metadata":{"_kg_hide-output":true,"_cell_guid":"d0a073a6-57e6-4ceb-9d81-4c65e5366635","_kg_hide-input":false,"_uuid":"f51951bb781a056f9de734a981854dfe66c8b16a","trusted":false,"collapsed":true},"cell_type":"code","source":"import os\n\nimport numpy as np\nimport pandas as pd\nfrom keras.preprocessing import image\nimport keras.applications.resnet50 as resnet50\nimport keras.applications.xception as xception\nimport keras.applications.inception_v3 as inception_v3\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\n\nsns.set_style(\"whitegrid\")\nsns.set_context(\"notebook\", font_scale=1.5, rc={\"lines.linewidth\": 2.5})\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a82406f7-0f32-44dd-985b-e2ceed6ba078","collapsed":true,"_uuid":"ba4ba5aa664f025aca86921e2e1c3e708783959f","trusted":false},"cell_type":"code","source":"resnet_model = resnet50.ResNet50(weights='imagenet')\ninception_model = inception_v3.InceptionV3(weights='imagenet')\nxception_model = xception.Xception(weights='imagenet')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2753158d-d8f7-4d70-8b61-7b307ff1a253","collapsed":true,"_uuid":"c27dd3f8918e5f738e474284de64bdc1c9e1ae07","trusted":false},"cell_type":"code","source":"def image_classify(model, pak, img, top_n=3):\n    \"\"\"Classify image and return top matches.\"\"\"\n    target_size = (224, 224)\n    if img.size != target_size:\n        img = img.resize(target_size)\n    x = image.img_to_array(img)\n    x = np.expand_dims(x, axis=0)\n    x = pak.preprocess_input(x)\n    preds = model.predict(x)\n    return pak.decode_predictions(preds, top=top_n)[0]\n\n\ndef plot_preds(img, preds_arr):\n    \"\"\"Plot image and its prediction.\"\"\"\n    sns.set_color_codes('pastel')\n    f, axarr = plt.subplots(1, len(preds_arr) + 1, figsize=(20, 5))\n    axarr[0].imshow(img)\n    axarr[0].axis('off')\n    for i in range(len(preds_arr)):\n        _, x_label, y_label = zip(*(preds_arr[i][1]))\n        plt.subplot(1, len(preds_arr) + 1, i + 2)\n        ax = sns.barplot(np.array(y_label), np.array(x_label))\n        plt.xlim(0, 1)\n        ax.set()\n        plt.xlabel(preds_arr[i][0])\n    plt.show()\n\n\ndef classify_and_plot(image_path):\n    \"\"\"Classify an image with different models.\n    Plot it and its predicitons.\n    \"\"\"\n    img = Image.open(image_path)\n    resnet_preds = image_classify(resnet_model, resnet50, img)\n    xception_preds = image_classify(xception_model, xception, img)\n    inception_preds = image_classify(inception_model, inception_v3, img)\n    preds_arr = [('Resnet50', resnet_preds), ('xception', xception_preds), ('Inception', inception_preds)]\n    plot_preds(img, preds_arr)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5c5016c2-8e4f-47b5-8a14-38a640ae4341","collapsed":true,"_uuid":"5ff0715819388e26767c62494dd2e0f98a47b795","trusted":false},"cell_type":"code","source":"image_files = [x.path for x in os.scandir(images_dir)]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2da5a956-5570-4ca3-ae50-0a6d85645dab","_uuid":"d8242cd0b2a42aecc2ad202850278bde4f774e05","scrolled":true,"trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[10])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ffce8072-0e41-4b1a-a77f-8db89568deb8","_uuid":"785296b6e889d2e22c9ae346ea127ecc404d2f3e","trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[11])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1daaaf0c-e7a6-41a7-b3cc-b4febb8ec891","_uuid":"3713a4b85e2df1cecc03c4d6aa1ff5e39cae8137","scrolled":false,"trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[12])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"65872a78-a2b2-4123-adf8-932bcb291ab5","_uuid":"2ab84b8c9c061f020d0d0ac52bbde88b96487cf7","trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[13])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"642740fb-b033-4c4c-a0e5-7468e3f672d4","_uuid":"71c5c780c4e1e6d88dd8245fc79a5d6478bf0815","trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[14])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"642740fb-b033-4c4c-a0e5-7468e3f672d4","_uuid":"71c5c780c4e1e6d88dd8245fc79a5d6478bf0815","trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[15])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"642740fb-b033-4c4c-a0e5-7468e3f672d4","_uuid":"71c5c780c4e1e6d88dd8245fc79a5d6478bf0815","trusted":false,"collapsed":true},"cell_type":"code","source":"classify_and_plot(image_files[16])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ef2743c2-061d-4aa3-a71a-305472fe7e28","_uuid":"f0f83f6e59cc6d9d0171b3adae01054f2800df0c"},"cell_type":"markdown","source":"# Correlation with Deal Probability\nFrom the few examples shown above, it seems like the classification confidence of the top class may work as a proxy for image quality.  We can say that if the image is ambiguous to the classification neural network, it will be ambiguous to a human being and unattractive as a product.\n\nTo verfify this, we can measure the correlation betwen the top class' classification score and the deal probability.  To have a reference to measure against, we can compare it to the correlation between the **advertisement's title or description lengths with deal probability**.  Analysis by [Bojan Tunguz][bojan] and others showed that these two features are strong predictors of deal probability.\n\n[bojan]: https://www.kaggle.com/tunguz/bow-meta-text-and-dense-features-lb-0-2241"},{"metadata":{"_cell_guid":"de287d84-77ea-4c97-9a21-8a8d28773cf8","collapsed":true,"_uuid":"45273aff38e2cf04cd0dd5f8f92e1c9d1f59302c","trusted":false},"cell_type":"code","source":"def classify_inception(image_path):\n    \"\"\"Classify image and return top match.\"\"\"\n    img = Image.open(image_path)\n    target_size = (224, 224)\n    if img.size != target_size:\n        img = img.resize(target_size)\n    x = image.img_to_array(img)\n    x = np.expand_dims(x, axis=0)\n    x = inception_v3.preprocess_input(x)\n    preds = inception_model.predict(x)\n    return inception_v3.decode_predictions(preds, top=1)[0][0]\n\ndef image_id_from_path(path):\n    return path.split('/')[3].split('.')[0]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b3f783ef-49b2-4447-96e9-018b245b9713","collapsed":true,"_uuid":"070550e149e35f175f4aeff5e45809f06b836543","trusted":false},"cell_type":"code","source":"train = pd.read_csv('../input/avito-demand-prediction/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e5998532-b8dc-4ad7-b188-2b4bcbce5ab7","collapsed":true,"_uuid":"3d382b1329228bf13496290474b8a5279ae33a0c","trusted":false},"cell_type":"code","source":"train['desc_len'] = train['description'].str.len()\ntrain['title_len'] = train['title'].str.len()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d497818d-45ff-4a57-819f-73341eeba6eb","_uuid":"72f85756889a3e5221b8bca6e03658896ed3f987","trusted":false,"collapsed":true},"cell_type":"code","source":"plt.figure(figsize=(10, 10))\ninception_conf = [[image_id_from_path(x), classify_inception(x)[2]] for x in image_files]\nconfidence = pd.DataFrame(inception_conf, columns=['image', 'image_confidence'])\ndf = confidence.merge(train, on='image')\ncorr = df[['image', 'image_confidence', 'deal_probability', 'desc_len', 'title_len']].corr()\nsns.heatmap(corr, annot=True)\nplt.xticks(rotation=30)\nplt.yticks(rotation=30)\nplt.title('Correlation Between Deal Probability and Strong Model Predictors')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3cae9bc8-45f6-4330-900c-6e803a2ddff2","_uuid":"26246c926a4835d334c437c80d93af161c567119"},"cell_type":"markdown","source":"# Conclusion\nThe correlation between `deal_probability` and `image_confidence` shown in the figure above is similar to or stronger than that with `title_len` (title length) or `desc_len` (description length).\n\nThis analysis was done on a relatively small sample of 1,000 images due to disk space restrictions here.  I'm looking forward to seeing someone run this on the full image data set and add it to the list of features we're experimenting with."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"file_extension":".py","mimetype":"text/x-python","pygments_lexer":"ipython3","codemirror_mode":{"name":"ipython","version":3},"name":"python","nbconvert_exporter":"python","version":"3.6.5"}},"nbformat":4,"nbformat_minor":1}