{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Extracting Semantic Vectors\n\nSo here's the idea: convolutional networks for image recognition need to distill the raw, pixel information, and turn it in to a representation of what's actually in that region of the image. If we extract vectors from late in the network, perhaps these vectors will contain useful meaning about the content of that part of the network, in a form that can be compared by distance.\n\nIn this notebook, we will iterate over the training set, process every image through a convolutional neural net, extract the feature vectors from near the end of the network, and store those vectors. What we want to end up with as the output of this notebook is a file containing all those vectors for further processing.\n\nThis notebook uses some parts of Michal's excellent starter notebook here: https://www.kaggle.com/code/michaln/hotel-id-starter-classification-traning","metadata":{}},{"cell_type":"markdown","source":"We need the following packages:\n* timm for common network designs\n* pyarrow which we will be using to write a file too large for memory","metadata":{}},{"cell_type":"code","source":"!pip install timm pyarrow","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:06.07907Z","iopub.execute_input":"2022-04-10T10:38:06.079661Z","iopub.status.idle":"2022-04-10T10:38:16.924765Z","shell.execute_reply.started":"2022-04-10T10:38:06.079566Z","shell.execute_reply":"2022-04-10T10:38:16.923902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We import all of our modules...","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport random\nimport os\nimport gc\nimport math","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-04-10T10:38:16.92669Z","iopub.execute_input":"2022-04-10T10:38:16.926968Z","iopub.status.idle":"2022-04-10T10:38:16.932407Z","shell.execute_reply.started":"2022-04-10T10:38:16.926934Z","shell.execute_reply":"2022-04-10T10:38:16.931708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image as pil_image\nfrom tqdm import tqdm\n\nimport matplotlib\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:16.933637Z","iopub.execute_input":"2022-04-10T10:38:16.934138Z","iopub.status.idle":"2022-04-10T10:38:16.940821Z","shell.execute_reply.started":"2022-04-10T10:38:16.934102Z","shell.execute_reply":"2022-04-10T10:38:16.940065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader\nfrom torch.nn.parameter import Parameter\n\nimport timm","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:16.943326Z","iopub.execute_input":"2022-04-10T10:38:16.943759Z","iopub.status.idle":"2022-04-10T10:38:19.290548Z","shell.execute_reply.started":"2022-04-10T10:38:16.943723Z","shell.execute_reply":"2022-04-10T10:38:19.289725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pyarrow as pa\nfrom pyarrow.parquet import ParquetWriter\n","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:19.292171Z","iopub.execute_input":"2022-04-10T10:38:19.292445Z","iopub.status.idle":"2022-04-10T10:38:19.296632Z","shell.execute_reply.started":"2022-04-10T10:38:19.292406Z","shell.execute_reply":"2022-04-10T10:38:19.295953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Set some parameters for this run.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 256 # Images will be squares of this width and height\nSEED = 42 # Random seed for consistency\nDEVICE = 'cuda'\nBATCH_SIZE = 32\n\nVAL_SAMPLES = 1\nPROJECT_FOLDER = \"../input/hotel-id-to-combat-human-trafficking-2022-fgvc9/\"\nDATA_FOLDER = \"../input/hotelid-2022-train-images-256x256/\"\nIMAGE_FOLDER = DATA_FOLDER + \"images/\"\n\ntrain_df = pd.read_csv(os.path.join(DATA_FOLDER, 'train.csv'))","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:19.297967Z","iopub.execute_input":"2022-04-10T10:38:19.298365Z","iopub.status.idle":"2022-04-10T10:38:19.352824Z","shell.execute_reply.started":"2022-04-10T10:38:19.29833Z","shell.execute_reply":"2022-04-10T10:38:19.352073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By setting a consistent random seed, we ensure the same results with every run.","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:19.354141Z","iopub.execute_input":"2022-04-10T10:38:19.354388Z","iopub.status.idle":"2022-04-10T10:38:19.35923Z","shell.execute_reply.started":"2022-04-10T10:38:19.354354Z","shell.execute_reply":"2022-04-10T10:38:19.358558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Basic processing of images. We don't do much augmentation with this method, as every single run through the training images produces a lot of data.","metadata":{}},{"cell_type":"code","source":"import albumentations as A\nimport albumentations.pytorch as APT\nimport cv2 \n\n# used for validation dataset - only occlusions\nval_transform = A.Compose([\n    A.CoarseDropout(p=1.0, max_holes=1, \n                    min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n                    min_width=IMG_SIZE//4,  max_width=IMG_SIZE//2, \n                    fill_value=(255,0,0)),# simulating occlusions\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])\n\n# no augmentations\nbase_transform = A.Compose([\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:19.360579Z","iopub.execute_input":"2022-04-10T10:38:19.361113Z","iopub.status.idle":"2022-04-10T10:38:21.114929Z","shell.execute_reply.started":"2022-04-10T10:38:19.361075Z","shell.execute_reply":"2022-04-10T10:38:21.114219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The HotelTrainDataset class loads training images for each row in the data frame","metadata":{}},{"cell_type":"code","source":"class HotelTrainDataset:\n    def __init__(self, data, transform=None, data_path=\"train_images/\"):\n        self.data = data\n        self.data_path = data_path\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.data)\n    \n    def __getitem__(self, idx):\n        record = self.data.iloc[idx]\n        image_path = self.data_path + record[\"image_id\"]\n        image = np.array(pil_image.open(image_path)).astype(np.uint8)\n\n        if self.transform:\n            transformed = self.transform(image=image)\n            image = transformed[\"image\"]\n        \n        hotel_id = record['hotel_id']\n        \n        return {\n            \"image\" : image,\n            \"image_id\": record[\"image_id\"],\n            \"target\" : hotel_id\n        }","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:21.116246Z","iopub.execute_input":"2022-04-10T10:38:21.116508Z","iopub.status.idle":"2022-04-10T10:38:21.123176Z","shell.execute_reply.started":"2022-04-10T10:38:21.116454Z","shell.execute_reply":"2022-04-10T10:38:21.122531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is our \"model\" for the purposes of our notebook. We don't really train a model, but this is what we will be executing against each image and storing the result.\n\nThe timm library supports loading models in a \"features only\" mode for pyramid classification. This returns a series of layers from the backbone, rather than just the output of the last layer or classifier, to allow for models that need to inspect fine-grained state. We extract and return a single layer from that pyramid.\n\nOur default parameters here are to use the EfficientNet B0, with pre-trained weights, and extract the fourth layer returned. This layer has dimensions of 320x8x8 - 64 grid tiles of 320 features.","metadata":{}},{"cell_type":"code","source":"class VectorExtractor(nn.Module):\n    def __init__(self, backbone_name='efficientnet_b0', layer_to_extract=4):\n        super(VectorExtractor, self).__init__()\n        self.backbone_name = backbone_name\n        self.layer_to_extract = layer_to_extract\n        self.backbone = timm.create_model(self.backbone_name, pretrained=True, features_only=True)\n\n    def forward(self, x):\n        # Because the backbone is in \"features only\" mode, this returns an array of Tensors, from finer to coarser resolution.\n        layers = self.backbone(x)\n        # We extract and return the results from a single layer.\n        return layers[self.layer_to_extract]\n        ","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:21.126162Z","iopub.execute_input":"2022-04-10T10:38:21.126566Z","iopub.status.idle":"2022-04-10T10:38:21.138461Z","shell.execute_reply.started":"2022-04-10T10:38:21.126531Z","shell.execute_reply":"2022-04-10T10:38:21.137726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We read our training data here, split it in to a training and validation set, and then save those to disk. This allows us to ensure we're using a consistent validation set down the track.","metadata":{}},{"cell_type":"code","source":"data_df = pd.read_csv(DATA_FOLDER + \"train.csv\")\nval_df = data_df.groupby(\"hotel_id\").sample(VAL_SAMPLES, random_state=SEED)\ntrain_df = data_df[~data_df[\"image_id\"].isin(val_df[\"image_id\"])]\n\ntrain_df.to_csv('training.csv')\nval_df.to_csv('validation.csv')","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:21.139767Z","iopub.execute_input":"2022-04-10T10:38:21.140029Z","iopub.status.idle":"2022-04-10T10:38:22.433246Z","shell.execute_reply.started":"2022-04-10T10:38:21.139985Z","shell.execute_reply":"2022-04-10T10:38:22.432541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Because the results of this process will be too large to hold in memory, we need a method to write records continuously to disk.\n\nFor this purpose, I selected the Parquet format, writing it through pyarrow. The interface to pyarrow was phenomenally awkward, so I probably won't be using it again.","metadata":{}},{"cell_type":"code","source":"class TileVectorWriter():\n    def __init__(self, path, width, capacity=32768, existing=False):\n        self.path = path\n        self.width = width\n        self.capacity = capacity\n        self.record_count = 0\n        # The schema describes the columns and their types that will be written in this file\n        self.schema = pa.schema([('i', pa.int32()), ('vector', pa.binary(width * 4)), ('target', pa.int32()), ('file', pa.string()), ('x', pa.int32()), ('y', pa.int32())])\n        self.buffer = []\n        # We construct a writer to allow continuous streaming of records to disk\n        self.writer = ParquetWriter(path, self.schema, use_dictionary=['target', 'file'])\n    \n    def append(self, item):\n        vec, target, file, x, y = item\n        i = self.record_count\n        self.buffer.append({\"i\": int(i), \"vector\": vec.astype(np.float32).tobytes(), \"target\": int(target), \"file\": str(file), \"x\": int(x), \"y\": int(y)})\n        if len(self.buffer) >= self.capacity:\n            # Whenever we reach capacity, write all records to disk and clear the internal buffer\n            self.flush()\n        self.record_count += 1\n    \n    def __len__(self):\n        return self.record_count\n        \n    def flush(self):\n        # This is a very clunky and inefficient way of getting our records in to pyarrow, via pandas.\n        # Finding methods that work was hard, and pyarrow had major API changes between the version in Kaggle and the latest.\n        if len(self.buffer) > 0:\n            batch = pa.Table.from_pandas(pd.DataFrame(self.buffer), schema=self.schema)\n            self.writer.write_table(batch)\n            self.buffer = []\n    \n    def close(self):\n        self.flush()\n        self.writer.close()\n        self.writer = None\n    \n    # For anyone unfamiliar, this is a destructor. It ensures we neatly close the file when this goes out of scope.\n    def __del__(self):\n        self.close()","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:38:22.434705Z","iopub.execute_input":"2022-04-10T10:38:22.434936Z","iopub.status.idle":"2022-04-10T10:38:22.446123Z","shell.execute_reply.started":"2022-04-10T10:38:22.434903Z","shell.execute_reply":"2022-04-10T10:38:22.445434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is the workhorse function of the process we'll be following - iterate over the training set, extract our vectors, and send them to the collection.","metadata":{}},{"cell_type":"code","source":"def extract_vectors(dataset, extractor, vector_list, batch_size=32, num_workers=2, device='gpu'):\n    loader = DataLoader(dataset, num_workers=num_workers, batch_size=batch_size, shuffle=True, drop_last=True)\n    \n    i = 0\n    bar = tqdm(loader, total=len(loader))\n    # For each batch\n    for data in bar:\n        images = data['image'].to(device)\n\n        with torch.no_grad():\n            # Run the extractor over the image\n            vectors = extractor(images)\n            \n            # Get our vectors\n            vectors = vectors.detach().cpu()\n\n            # Iterate over batch\n            for b in range(vectors.shape[0]):\n                image_id = data['image_id'][b]\n                target = int(data['target'][b])\n\n                # Iterate over x and y dimensions\n                for x in range(vectors.shape[2]):\n                    for y in range(vectors.shape[3]):\n                        # Send the features to our collection, with the associated hotel, image file, and position as metadata.\n                        features = vectors[b, :, x, y].numpy()\n                        vector_list.append((features, target, image_id, x, y))","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:39:27.591336Z","iopub.execute_input":"2022-04-10T10:39:27.591619Z","iopub.status.idle":"2022-04-10T10:39:27.601869Z","shell.execute_reply.started":"2022-04-10T10:39:27.591589Z","shell.execute_reply":"2022-04-10T10:39:27.600441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Actually run everything, and we should end up with \"vectors.parquet\".","metadata":{}},{"cell_type":"code","source":"%%time\n\nseed_everything(SEED)\ndataset = HotelTrainDataset(train_df, base_transform, data_path=IMAGE_FOLDER)\nextractor = VectorExtractor().to(DEVICE)\nvectors = TileVectorWriter(\"vectors.parquet\", 320)\nextract_vectors(dataset, extractor, vectors, batch_size=BATCH_SIZE, device=DEVICE)\nvectors.close()","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:39:29.123817Z","iopub.execute_input":"2022-04-10T10:39:29.124084Z","iopub.status.idle":"2022-04-10T10:39:29.766977Z","shell.execute_reply.started":"2022-04-10T10:39:29.124054Z","shell.execute_reply":"2022-04-10T10:39:29.7661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This notebook series continues with https://www.kaggle.com/code/prubyg/hotel-id-vector-indexing/","metadata":{}}]}