{"cells":[{"metadata":{},"cell_type":"markdown","source":"#### *References: https://github.com/CSAILVision/places365*\n\n[Robin Smiths - Keras Landmark or Non-Landmark identification](https://www.kaggle.com/rsmits/keras-landmark-or-non-landmark-identification) from [Google Landmark Recognition 2019](https://www.kaggle.com/c/landmark-recognition-2019/overview)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Introduction\n\nIn this competition it is not only important to predict the landmark for an image but also to make sure that if an image is not a landmark that we don't make a prediction. \n* Correctly identifying non-landmark images in the test set will increase the score of a submission. \n* Removing as much of the non-landmark images from the training set will decrease the total amount of images that we need to train on and it will improve the 'correctness' of the model if it is trained on the landmark images and not on all the selfies, hotel-rooms and beds etcetera that were made near the landmark.\n\nA good way to start the identification is to use the Places365 dataset and models as described on this [webpage](http://places2.csail.mit.edu/). In the Places365 dataset there 365 classes and each class is also marked as either 'indoor' or 'outdoor'. We could interpret this also as 'non-landmark' or 'landmark'. It could be further optimized offcourse...an image of the inside of a castle will be marked as 'indoor' but will very likely be a legitimate 'landmark'.\n\nPrevious Landmark Reconition Challenges with winner solution have shown to use this Landmark / No Landmark Recognition Technique but instead of this they use more sophiticated techniques. This is rather a simple and not so accurate way to approach this but it works. \n\n#### Previous Comp Solution's that used this technique \n\n> Google Landmark Recognition Challenge - 2018\n* [Our solution [4th place]](https://www.kaggle.com/c/landmark-recognition-challenge/discussion/57896)\n* [Our solution and source code (0.22 Public 0.17 Private)](https://www.kaggle.com/c/landmark-recognition-challenge/discussion/57913)\n* [My single-(ok, 1.5-)model solution and source code [19th place]](https://www.kaggle.com/c/landmark-recognition-challenge/discussion/57919)\n* [Solution from Dawnbreaker on steroids in top-40](https://www.kaggle.com/c/landmark-recognition-challenge/discussion/58035)\n\n> Google Landmark Recognition Challenge - 2019\n* [27th place solution](https://www.kaggle.com/c/landmark-recognition-2019/discussion/94486)\n* [8th place solution](https://www.kaggle.com/c/landmark-recognition-2019/discussion/94512)\n* [Team JL Solution Summary](https://www.kaggle.com/c/landmark-recognition-2019/discussion/94523)\n* [20th place solution with code](https://www.kaggle.com/c/landmark-recognition-2019/discussion/94645)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Setup Dependencies","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget https://raw.githubusercontent.com/CSAILVision/places365/master/wideresnet.py","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom tqdm import tqdm\nimport cv2\nimport torch\nfrom torch.autograd import Variable as V\nimport torchvision.models as models\nfrom torchvision import transforms as trn\nfrom torch.nn import functional as F\nimport os\nimport numpy as np\nimport cv2\nfrom PIL import Image\nimport gc\ngc.enable()\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load classes and I/O labels of Places365 Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_labels():\n    # prepare all the labels\n    # scene category relevant\n    file_name_category = 'categories_places365.txt'\n    if not os.access(file_name_category, os.W_OK):\n        synset_url = 'https://raw.githubusercontent.com/csailvision/places365/master/categories_places365.txt'\n        os.system('wget ' + synset_url)\n    classes = list()\n    with open(file_name_category) as class_file:\n        for line in class_file:\n            classes.append(line.strip().split(' ')[0][3:])\n    classes = tuple(classes)\n\n    # indoor and outdoor relevant\n    file_name_IO = 'IO_places365.txt'\n    if not os.access(file_name_IO, os.W_OK):\n        synset_url = 'https://raw.githubusercontent.com/csailvision/places365/master/IO_places365.txt'\n        os.system('wget ' + synset_url)\n    with open(file_name_IO) as f:\n        lines = f.readlines()\n        labels_IO = []\n        for line in lines:\n            items = line.rstrip().split()\n            labels_IO.append(int(items[-1]) -1) # 0 is indoor, 1 is outdoor\n    labels_IO = np.array(labels_IO)\n    return classes, labels_IO","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image Transformations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def returnTF():\n# load the image transformer\n    tf = trn.Compose([\n        trn.ToPILImage(),\n        trn.Resize((224,224)),\n        trn.ToTensor(),\n        trn.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ])\n    return tf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load Pretrained Weights & Create Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def hook_feature(module, input, output):\n    features_blobs.append(np.squeeze(output.data.cpu().numpy()))\n    \ndef recursion_change_bn(module):\n    if isinstance(module, torch.nn.BatchNorm2d):\n        module.track_running_stats = 1\n    else:\n        for i, (name, module1) in enumerate(module._modules.items()):\n            module1 = recursion_change_bn(module1)\n    return module\n\ndef load_model():\n    # this model has a last conv feature map as 14x14\n    model_file = 'wideresnet18_places365.pth.tar'\n    if not os.access(model_file, os.W_OK):\n        os.system('wget http://places2.csail.mit.edu/models_places365/' + model_file)\n        os.system('wget https://raw.githubusercontent.com/csailvision/places365/master/wideresnet.py')\n\n    import wideresnet\n    model = wideresnet.resnet18(num_classes=365)\n    checkpoint = torch.load(model_file, map_location=lambda storage, loc: storage)\n    state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()}\n    model.load_state_dict(state_dict)\n    \n    # hacky way to deal with the upgraded batchnorm2D and avgpool layers...\n    for i, (name, module) in enumerate(model._modules.items()):\n        module = recursion_change_bn(model)\n    model.avgpool = torch.nn.AvgPool2d(kernel_size=14, stride=1, padding=0)\n\n    model.eval()\n    # hook the feature extractor\n    features_names = ['layer4','avgpool'] # this is the last conv layer of the resnet\n    for name in features_names:\n        model._modules.get(name).register_forward_hook(hook_feature)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Config","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"classes, labels_IO = load_labels()\nfeatures_blobs = []\ndevice = torch.device(\"cuda\") if torch.cuda.is_available() else \"cpu\"\nmodel = load_model()\nmodel = model.to(device)\n\ntf = returnTF()\n\nparams = list(model.parameters())\nweight_softmax = params[-2].data.cpu().numpy()\nweight_softmax[weight_softmax<0] = 0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Filter Train Data \n\n* Filtering images first based on minimum number of samples for a particular class will help in reducing the landmark/no landmark classification images. This speeds up the process as then we need to compute less.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MIN_SAMPLES_PER_CLASS = 50\ntrain = pd.read_csv('../input/landmark-recognition-2020/train.csv')\ntest = pd.read_csv('../input/landmark-recognition-2020/sample_submission.csv')\nprint(train.shape)\ncounts = train.landmark_id.value_counts()\nselected_classes = counts[counts >= MIN_SAMPLES_PER_CLASS].index\nnum_classes = selected_classes.shape[0]\nprint('classes with at least N samples:', num_classes)\ntrain = train.loc[train.landmark_id.isin(selected_classes)]\nprint(train.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prediction Loop","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### Test Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"io_test = []\nfor i, img_id in tqdm(enumerate(test.id), total=len(test)):\n    image_path = f\"../input/landmark-recognition-2020/test/{img_id[0]}/{img_id[1]}/{img_id[2]}/{img_id}.jpg\"\n    img = cv2.imread(image_path, cv2.IMREAD_COLOR)\n    input_img = V(tf(img).unsqueeze(0))\n    logit = model.forward(input_img.cuda())\n    h_x = F.softmax(logit, 1).data.squeeze()\n    probs, idx = h_x.sort(0, True)\n    probs = probs.cpu().numpy()\n    idx = idx.cpu().numpy()\n\n    io_image = np.mean(labels_IO[idx[:10]]) # vote for the indoor or outdoor\n    if io_image < 0.5:\n        io_test.append(0) \n    else:\n        io_test.append(1) \n        \n    del input_img\n    del img\n    del image_path\n    del logit\n    del probs\n    del idx\n    del io_image\n    if i%1000 ==0:\n        gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test['io'] = io_test\ntest.to_csv('test_io.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def return_img(img_id):\n    image_path = f\"../input/landmark-recognition-2020/test/{img_id[0]}/{img_id[1]}/{img_id[2]}/{img_id}.jpg\"\n    img = np.array(Image.open(image_path).resize((224, 224), Image.LANCZOS))\n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get 'landmark' images\nn = 16\nlandmark_images =  test[test['io'] == 1]['id'][:n]\n\nfig = plt.figure(figsize = (16, 16))\nfor i, img_id in enumerate(landmark_images):\n    image = return_img(img_id)\n    fig.add_subplot(4, 4, i+1)\n    plt.title(img_id)\n    plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get non 'landmark' images\nn = 16\nlandmark_images =  test[test['io'] == 0]['id'][:n]\n# landmark_indexes = landmark_images[:n].index.values\n\n# Plot image examples\nfig = plt.figure(figsize = (16, 16))\nfor i, img_id in enumerate(landmark_images):\n    image = return_img(img_id)\n    fig.add_subplot(4, 4, i+1)\n    plt.title(img_id)\n    plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That for sure looks promising! The majority of predicted 'non-landmarks' for sure look like they are indeed not landmarks.\n\n### More to come. Stay Tuned.!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}