{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Notebook Contents

\n\nThis is a little notebook with code for implementing a `torch` DataLoader when data is made of multiple files, each with its own number of targets. Of course the size of data is too much to be all loaded in a Notebook and that's why we need to load only a batch of it. \n\nThe inspiration for the notebook came from [this](https://www.kaggle.com/c/indoor-location-navigation) beautiful challenge and the problem of dealing with more than 55Gb of data, as you can see [here](https://www.kaggle.com/c/indoor-location-navigation/data).\n\n
\n

Notebook Contents

\n\n
\n\n

Props

\n\n**All my work** was born out of [this](https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd) wonderful article by David MacLeod, please clap it! \n\nProps also to [this](https://amaarora.github.io/2020/09/13/unet.html) for the Unet implementation, clearly explained.\n\nLet me thank also my friend [Brasnold](https://www.kaggle.com/brasnold) who's always full of bright machine learning ideas.\n\n
Edit
\n\nI'll probably restrain this notebook just to the unet application on floor images and then create another one with a deep dive on Torch datasets and dataloaders.","metadata":{}},{"cell_type":"markdown","source":"\n

0. Some Background

\n \nIn this notebook I'll use some of the pillar classes of pytorch, in particular:\n \n - Dataset\n - IterableDataset\n - DataLoader\n \nI think that `torch` documentation is all you need to read: have a look [here](https://pytorch.org/docs/stable/data.html) to quickly understand what all of the above are. \n \nOther than the article mentioned in the Props section I suggest also [this](https://medium.com/swlh/how-to-use-pytorch-dataloaders-to-work-with-enormously-large-text-files-bbd672e955a0) one which stresses how we need different classes to deal with crazy large files. ","metadata":{}},{"cell_type":"markdown","source":"\n

1. DataLoader from Multiple Files

\n\nLet's get into it quick. \n\nTo be done in general","metadata":{}},{"cell_type":"markdown","source":"\n\n

2. Example on Floor Images from the Indoor Location and Navigation Kaggle Challenge

\n\nWhat I'll show you here is how you can use the DataLoader to load just a batch of images. \n\nI'll use the floor images of the Indoor Location and Navigation Kaggle Challenge.\n\n\n\nEach site/building has its own floors and each floor has some files linked to it, including the floor image.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\npd.options.display.max_columns = 50\npd.options.display.max_colwidth = 200\nimport os\nimport multiprocessing\nfrom multiprocessing import Pool\nimport time\n\nimport colorama\nfrom colorama import Fore, Back, Style\n\ny_ = Fore.YELLOW\nr_ = Fore.RED\ng_ = Fore.GREEN\nb_ = Fore.BLUE\nm_ = Fore.MAGENTA\nc_ = Fore.CYAN\nsr_ = Style.RESET_ALL\n\nimport torch\nfrom itertools import chain, islice, cycle\nfrom torch.utils.data import Dataset, IterableDataset, DataLoader\n\ndef get_device():\n if torch.cuda.is_available():\n device = 'cuda:0'\n else:\n device = 'cpu'\n return device\n\nDEVICE = get_device()\n\nmodels_path = os.path.join(os.getcwd(), \"models\")\n\nimport json\nimport re\nfrom sys import getsizeof\nimport matplotlib\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'figure.max_open_warning': 0})\nplt.style.use('fivethirtyeight')\nimport warnings # Supress warnings \nwarnings.filterwarnings('ignore')\n\nimport cv2\nimport glob\nimport tqdm\n\nroot_path = '/kaggle/input/indoor-location-navigation'\nmetadata_path = '/kaggle/input/indoor-location-navigation/metadata'\n\nBINARY_THRESHOLD = 30\nN_CPUS = multiprocessing.cpu_count()\nN_CPUS","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"floor_images_paths = glob.glob(metadata_path+\"/*/*/floor_image.png\")\n\nN_floors = len(floor_images_paths)\nN_floors\n\ntrain_paths = np.random.choice(floor_images_paths, size = int(0.7*N_floors), replace = False)\nN_train = len(train_paths)\nval_paths = np.random.choice(list(set(floor_images_paths)-set(train_paths)), size = int(0.5*(N_floors-N_train)), replace = False)\ntest_paths = list(set(floor_images_paths)-set(val_paths)-set(train_paths))\n\nprint(N_train, len(val_paths), len(test_paths))\n\ndef chunks(lst, n):\n \"\"\"Yield successive n-sized chunks from lst.\"\"\"\n for i in range(0, len(lst), n):\n yield lst[i:i + n]\n \ndef read_image(image):\n return plt.imread(image).shape\n\ndef read_info(info):\n return pd.read_json(info).transpose()\n\nshapes = []\n\nwith Pool(N_CPUS) as p:\n n_chunk = 0\n for chunk in tqdm.tqdm(chunks(floor_images_paths, int(len(floor_images_paths)/N_CPUS))):\n shapes += p.map(read_image, chunk)\n n_chunk +=1","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"df_sizes = pd.DataFrame(shapes, columns = ['height', 'width', 'channels'])\nprint(\"{} a sample of images sizes:\".format(b_))\ndisplay(df_sizes.sample(3))\nMAX_HEIGHT = int(df_sizes.height.max())\nMAX_WIDTH = int(df_sizes.width.max())\nMAX_CHANNELS = int(df_sizes.channels.max())\nprint(\"{}Each image has its own height. All images have 800 width and 4 channels (png)\".format(b_))\nprint(\"{}Number of floors images: {}\".format(b_,len(df_sizes)))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
Let's see some images
","metadata":{}},{"cell_type":"code","source":"random_images = np.random.choice(np.arange(len(floor_images_paths)), 6).tolist()\n\ndef rgb2gray(rgb):\n return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])\n\nfig, axes = plt.subplots(3, 2, figsize = (40, 40))\nax = axes.ravel()\n\nheights = []\nwidths = []\n\nfor j, image_path in enumerate(random_images):\n \n floor_image = plt.imread(floor_images_paths[image_path])\n site, path = floor_images_paths[image_path].split(\"/\")[-3:-1]\n shape = floor_image.shape\n heights.append(shape[0])\n widths.append(shape[1])\n \n floor_gray = rgb2gray(floor_image[:, :, :3])\n\n ax[j].imshow(floor_image)\n ax[j].set_title('-'.join([site, path, str(floor_image.shape[0]), str(floor_image.shape[1])]), fontdict={'fontsize': 15, 'fontweight': 'medium'})\n \nfig.suptitle(\"Some Floors images\", fontdict = {'fontsize': 30, 'fontweight': 'medium'})","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each image has its own size. What I'll implement is a DataLoader which reads each image, resizes it such that all images have the same shape and yields the corresping tensor.","metadata":{}},{"cell_type":"markdown","source":"
Custom Dataset
","metadata":{}},{"cell_type":"code","source":"import torch\nfrom itertools import chain, islice, cycle\nfrom torch.utils.data import Dataset, IterableDataset, DataLoader\n\nclass MultiFilesDataset(Dataset):\n \n def __init__(self, data_list, proper_shape = (3336, 800, 4)):\n \n self.data_list = data_list\n self.proper_shape = proper_shape\n \n def read_and_resize(self, image_path, mask = True):\n \n out_arr = np.zeros(self.proper_shape)\n image = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)\n image = np.expand_dims(cv2.resize(image, (self.proper_shape[0], self.proper_shape[1])), 2)\n \n if mask:\n mask = cv2.threshold(image, BINARY_THRESHOLD, 255, cv2.THRESH_BINARY)[1][:, :]\n mask = np.where(mask==255, 1, mask) \n return (torch.from_numpy(image).permute(2, 0, 1)), (torch.from_numpy(np.expand_dims(mask, 2)).permute(2, 0, 1))\n else:\n return torch.from_numpy(image).permute(2, 0, 1)\n \n def __len__(self):\n return (len(self.data_list))\n \n def __getitem__(self, idx):\n \n return self.read_and_resize(self.data_list[idx])\n \ncustom_dataset = MultiFilesDataset(floor_images_paths[:10], proper_shape = (256, 256, 1))\ndata_loader = DataLoader(custom_dataset, batch_size = 3)\n\nfor batch in data_loader:\n print(batch[0].size(), batch[1].size())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Custom Iterable Dataset
","metadata":{}},{"cell_type":"code","source":"import torch\nfrom itertools import chain, islice, cycle\nfrom torch.utils.data import Dataset, IterableDataset, DataLoader\n\nclass MultiFilesIterableDataset(IterableDataset):\n \n def __init__(self, data_list, proper_shape = (3336, 800, 4)):\n \n self.data_list = data_list\n self.proper_shape = proper_shape\n \n def read_and_resize(self, image_path, mask = True):\n \n out_arr = np.zeros(self.proper_shape)\n image = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)\n image = np.expand_dims(cv2.resize(image, (self.proper_shape[0], self.proper_shape[1])), 2)\n \n if mask:\n mask = cv2.threshold(image, BINARY_THRESHOLD, 255, cv2.THRESH_BINARY)[1][:, :]\n mask = np.where(mask==255, 1, mask) \n yield (torch.from_numpy(image).permute(2, 0, 1)), (torch.from_numpy(np.expand_dims(mask, 2)).permute(2, 0, 1))\n else:\n yield torch.from_numpy(image).permute(2, 0, 1)\n \n def get_stream(self, data_list):\n return chain.from_iterable(map(lambda x: self.read_and_resize(x), data_list))\n \n def __iter__(self):\n return self.get_stream(self.data_list)\n \niterable_dataset = MultiFilesIterableDataset(floor_images_paths[:10], proper_shape = (256, 256, 1))\ndata_loader = DataLoader(iterable_dataset, batch_size = 3)\n\nfor batch in data_loader:\n print(batch[0].size(), batch[1].size())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
3. Unet Implementation on Floor Images
\n\nUnet is a neural network ideal to perform segmentation tasks. *Segmentation* is the task of classifying each pixel in an image. \n\nLook [here](https://towardsdatascience.com/understanding-semantic-segmentation-with-unet-6be4f42d4b47#:~:text=The%20UNET%20was%20developed%20by,The%20architecture%20contains%20two%20paths.&text=Thus%20it%20is%20an%20end,accept%20image%20of%20any%20size.) for a Unet explanation. \n\n\n\n\nIn this section I'll provide code to train a Unet in segmenting Floor Images.\n\n\n**Disclaimer:** being this a walkthrough I will create artificial floor masks, by taking the binary thresholded image. \n","metadata":{}},{"cell_type":"markdown","source":"\n
Model definition
","metadata":{}},{"cell_type":"code","source":"from torch import nn\nimport torchvision\n\nclass Block(nn.Module):\n def __init__(self, in_ch, out_ch, ks = 3):\n super().__init__()\n self.conv1 = nn.Conv2d(in_ch, out_ch, ks, padding = 1)\n self.relu = nn.ReLU()\n self.conv2 = nn.Conv2d(out_ch, out_ch, ks, padding = 1)\n \n def forward(self, x):\n return self.relu(self.conv2(self.relu(self.conv1(x))))\n\nclass Encoder(nn.Module):\n def __init__(self, chs=(3, 64, 128, 256, 512, 1024)):\n super().__init__()\n self.enc_blocks = nn.ModuleList([Block(chs[i], chs[i+1]) for i in range(len(chs)-1)])\n self.pool = nn.MaxPool2d(2)\n \n def forward(self, x):\n ftrs = []\n for block in self.enc_blocks:\n x = block(x)\n ftrs.append(x)\n x = self.pool(x)\n return ftrs\n\nclass Decoder(nn.Module):\n def __init__(self, chs=(1024, 512, 256, 128, 64)):\n super().__init__()\n self.chs = chs\n self.upconvs = nn.ModuleList([nn.ConvTranspose2d(chs[i], chs[i+1], 2, 2) for i in range(len(chs)-1)])\n self.dec_blocks = nn.ModuleList([Block(chs[i], chs[i+1]) for i in range(len(chs)-1)]) \n \n def forward(self, x, encoder_features):\n for i in range(len(self.chs)-1):\n x = self.upconvs[i](x)\n enc_ftrs = self.crop(encoder_features[i], x)\n x = torch.cat([x, enc_ftrs], dim=1)\n x = self.dec_blocks[i](x)\n return x\n \n def crop(self, enc_ftrs, x):\n _, _, H, W = x.shape\n enc_ftrs = torchvision.transforms.CenterCrop([H, W])(enc_ftrs)\n return enc_ftrs \n \nclass UNet(nn.Module):\n def __init__(self, enc_chs=(3, 64, 128, 256, 512, 1024), dec_chs=(1024, 512, 256, 128, 64), num_class=1, retain_dim=True, out_sz=(256,256)):\n super().__init__()\n self.encoder = Encoder(enc_chs)\n self.decoder = Decoder(dec_chs)\n self.head = nn.Conv2d(dec_chs[-1], num_class, 1)\n self.retain_dim = retain_dim\n self.out_sz = out_sz\n\n def forward(self, x):\n enc_ftrs = self.encoder(x)\n out = self.decoder(enc_ftrs[::-1][0], enc_ftrs[::-1][1:])\n out = self.head(out)\n\n return out\n ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PROPER_SHAPE = (256, 256, 1)\nmodel = UNet(enc_chs = (1, 16, 32, 64, 128, 256), dec_chs = (256, 128, 64, 32, 16), out_sz = PROPER_SHAPE[:2])\n#Loss function\ncriterion = nn.BCELoss()\n#Optimizer\noptimizer = torch.optim.Adam(model.parameters(), lr=0.001)\nmodel.to(DEVICE)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
Model Train
","metadata":{}},{"cell_type":"code","source":"SKIP_TRAIN = True #change to False if you wish to train the network\nEARLY_STOPPING_STEPS = 5\nN_EPOCHS = 20\nif SKIP_TRAIN:\n N_EPOCHS = 0\nBATCH_SIZE = 32\n\ntrain_dataset = MultiFilesDataset(train_paths, proper_shape = PROPER_SHAPE)\nval_dataset = MultiFilesDataset(val_paths, proper_shape = PROPER_SHAPE)\ntest_dataset = MultiFilesDataset(test_paths, proper_shape = PROPER_SHAPE)\n\ntrain_dataloader = DataLoader(train_dataset, batch_size = BATCH_SIZE)\nval_dataloader = DataLoader(val_dataset, batch_size = len(val_paths))\ntest_dataloader = DataLoader(test_dataset, batch_size = len(test_paths))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"early_stopping_steps = EARLY_STOPPING_STEPS\nearly_step = 0\nbest_loss = np.inf\n\nfor epoch in range(1, N_EPOCHS+1):\n # monitor training loss\n train_loss = 0.0\n model.train()\n #Training\n counter = 1\n for data in tqdm.tqdm(train_dataloader):\n images, mask = data\n images = images.to(DEVICE, dtype=torch.float32)\n mask = mask.to(DEVICE, dtype=torch.float32)\n optimizer.zero_grad()\n outputs = model(images)\n if criterion.__class__.__name__ == 'MSELoss':\n loss = criterion(outputs, mask)\n else:\n loss = criterion(F.sigmoid(outputs), mask)\n loss.backward()\n optimizer.step()\n train_loss += loss.item()*images.size(0)\n counter+=1\n \n train_loss = train_loss/counter\n for val_data in val_dataloader:\n model.eval()\n images, mask = val_data\n images = images.to(DEVICE, dtype=torch.float32)\n mask = mask.to(DEVICE, dtype=torch.float32)\n outputs = model(images)\n if criterion.__class__.__name__ == 'MSELoss':\n val_loss = criterion(outputs, mask)\n else:\n val_loss = criterion(F.sigmoid(outputs), mask)\n print('Epoch: {} \\tValidation Loss: {:.6f}'.format(epoch, val_loss))\n if val_loss < best_loss: \n best_loss = val_loss\n\n torch.save(model.state_dict(), os.path.join(models_path, 'unet_grayscale_{}_{}_{}'.format(BINARY_THRESHOLD, PROPER_SHAPE, PROPER_SHAPE)))\n\n\n elif(EARLY_STOP == True):\n\n early_step += 1\n if (early_step >= early_stopping_steps):\n break\n\n \n \n print('Epoch: {} \\tTraining Loss: {:.6f}'.format(epoch, train_loss))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#LOADING TRAINED MODEL\ntrained_model = UNet(enc_chs = (1, 16, 32, 64, 128, 256), dec_chs = (256, 128, 64, 32, 16), out_sz = PROPER_SHAPE[:2])\ntrained_model.load_state_dict(torch.load(\"../input/unet-trained/unet_grayscale_30\"))\ntrained_model.to(DEVICE)","metadata":{"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
Unet Predictions
","metadata":{}},{"cell_type":"code","source":"predictions_example_paths = np.random.choice(test_paths, size =3, replace = False)\n\nfig, axes = plt.subplots(3, 3, figsize = (20, 12))\nax = axes.ravel()\n\n\nfor j, path in enumerate(predictions_example_paths): \n image, mask = test_dataset.read_and_resize(path)\n image_color = plt.imread(path)\n \n ax[3*j].imshow(image_color)\n ax[3*j].set_title('original floor image', fontdict={'fontsize': 10, 'fontweight': 'medium'})\n ax[3*j+1].imshow(mask[0, :, :], cmap = 'gray')\n ax[3*j+1].set_title('binary thresholded image', fontdict={'fontsize': 10, 'fontweight': 'medium'})\n model.eval()\n out = trained_model(torch.from_numpy(np.expand_dims(image, 0)).to(DEVICE, dtype=torch.float32)).detach().numpy()\n ax[3*j+2].imshow(out[0, 0, :, :], cmap = 'gray')\n ax[3*j+2].set_title('unet prediction', fontdict={'fontsize': 10, 'fontweight': 'medium'})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I hope you found the notebook useful! I will tidy it up in the next days!","metadata":{}}]}