{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"*Identify specific whales and dolphins through features specific to each animal*\n\n![kaggle](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/dol1.jpeg)","metadata":{}},{"cell_type":"markdown","source":"# 1. First look of this competition \n* As a Kaggle rookie, I wanted to do a bit of research first. I found that most of the articles were similar to each other, with each falling into one of two categories:\n\n1. using the EfficientNet model with (a DOLG head) ,(GeM Pooling) and ArcFace classifier.\n\n2.   A simple ensemble of public best kernels\n>I can easily achieve the score of 0.758 but this is not what I want\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Overview of image detection/retrival\n* Based on prior work, I found that clustering embedding vectors generated by mapping the image data into a latent space generated some fairly good results. Let's map out this process with visualizations below:","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_1.png)","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_2.png)","metadata":{}},{"cell_type":"markdown","source":"# 3. A concrete approach\n* Let's use resnet50 to map the image to a vector before storing the vectors in a database. ","metadata":{}},{"cell_type":"markdown","source":"**Importing libraries**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport os\nfrom pathlib import Path\nimport torch\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nfrom pymilvus import utility","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Config Setup**","metadata":{}},{"cell_type":"code","source":"dataset_path = './Kaggle/happy-whale-and-dolphin/train_images'\nimages = []\nvectors = []\nvec_dim = len(vetcors[0])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Setup Database**\n* The vectors transformed by images need to be stored into a database which enables vector search. I use [**Milvus**](https://milvus.io/), a vector database, to store, index, and manage these massive embedding vectors. I first need to set up Docker to make Milvus work locally.","metadata":{}},{"cell_type":"code","source":"# download the latest docker-compose file\n$ wget https://github.com/milvus-io/milvus/releases/download/v2.0.0-pre-ga/milvus-standalone-docker-compose.yml -O docker-compose.yml\n# start the Milvus service\n$ docker-compose up -d\n# check the state of the containers\n$ docker-compose ps","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* With python api of Milvus, I can easily connect to our local milvus database and build a milvus collcetion.","metadata":{}},{"cell_type":"code","source":"!pip install pymilvus\nimport pymilvus as milvus\n\n# connect to local Milvus service\nmilvus.connections.connect(host='127.0.0.1', port=19530)\n\n# create collection\ncollection_name = 'reverse_image_search'\nid_field = milvus.FieldSchema(name=\"id\", dtype=milvus.DataType.INT64, is_primary=True, auto_id=True)\nvec_field = milvus.FieldSchema(name=\"vec\", dtype=milvus.DataType.FLOAT_VECTOR, dim=vec_dim)\nschema = milvus.CollectionSchema(fields=[id_field, vec_field])\ncollection = milvus.Collection(name=collection_name, schema=schema)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Images embedding**\n* Now we need to embed these images as vectors. Here I'll use **[Towhee](https://towhee.io/)**, an end-to-end library for generating embedding vectors from images.","metadata":{}},{"cell_type":"code","source":"!pip install towhee\nfrom towhee import pipeline\nembedding_pipeline = pipeline('towhee/image-embedding-resnet50')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Vectors storage and embedding in database**","metadata":{}},{"cell_type":"code","source":"for img_path in Path(dataset_path).glob('*'):\n    vec = embedding_pipeline(str(img_path))\n    norm_vec = vec / np.linalg.norm(vec)\n    vectors.append(norm_vec.tolist())\n    images.append(str(img_path.resolve()))\n    \n# insert data to Milvus\nres = collection.insert([vectors])\ncollection.load()\nimg_dict = {}\n\n# maintain mappings between primary keys and the original images for image retrieval\nfor i, key in enumerate(res.primary_keys):\n    img_dict[key] = images[i]\n    \nquery_img_path = './Kaggle/happy-whale-and-dolphin/test_images'\nquery_images = []\nquery_vectors = []\ntop_k = 5\n\nfor img_path in Path(query_img_path).glob('*'):\n    vec = embedding_pipeline(str(img_path))\n    norm_vec = vec / np.linalg.norm(vec)\n    query_vectors.append(norm_vec.tolist())\n    query_images.append(str(img_path.resolve()))\n\nquery_results = collection.search(data=query_vectors, anns_field=\"vec\", param={\"metric_type\": 'L2'}, limit=top_k)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_4.png)","metadata":{}},{"cell_type":"markdown","source":"[Reverse image search workflow](https://docs.towhee.io/tutorials/reverse-image-search/)","metadata":{}},{"cell_type":"markdown","source":"**Search results visualization**","metadata":{}},{"cell_type":"code","source":"!pip install matplotlib\nimport matplotlib.pyplot as plt\nfrom PIL import Image\n\nfor i in range(len(query_results)):\n    results = query_results[i]\n    query_file = query_images[i]\n\n    result_files = [img_dict[result.id] for result in results]\n    distances = [result.distance for result in results]\n\n    fig_query, ax_query = plt.subplots(1,1, figsize=(5,5))\n    ax_query.imshow(Image.open(query_file))\n    ax_query.set_title(\"Searched Image\\n\")\n    ax_query.axis('off')\n\n    fig, ax = plt.subplots(1,len(result_files),figsize=(20,20))\n    for x in range(len(result_files)):\n        ax[x].imshow(Image.open(result_files[x]))\n        ax[x].set_title('dist: ' + str(distances[x])[0:5])\n        ax[x].axis('off')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_5.png)","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_6.png)","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_7.png)","metadata":{}},{"cell_type":"markdown","source":"![](https://github.com/poissonyzr/dol_kaggle/raw/805fa65c76ec0f9f0354692d6b66e399d65f6a9b/Figure_8.png)","metadata":{}},{"cell_type":"markdown","source":"                     👏 IF YOU FORK THIS OR FIND THIS HELPFUL 👏\n                                 PLEASE UPVOTE!","metadata":{}}]}