{"cells":[{"metadata":{},"cell_type":"markdown","source":"# fastai2 Tabular + Vision\n\nIn this kernel, we're going to see what it takes to combine both the Tabular and Image data from this competition into one `DataLoader`. While the resulting \"`DataLoader`\" may seem odd at first, I'll show you how it actually makes perfect sense.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First let's install grab the libraries we need:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install fastai2 pydicom --quiet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai2.tabular.all import *\nfrom fastai2.vision.all import *\nfrom fastai2.medical.imaging import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The General Approach\n\nApproaching this problem will be pretty straight forward. We're going to wrap a `TabDataLoader` and a `TfmdDL` (Or Vision `DataLoader`) together. To do this properly they need to share 3 functions:\n  * `one_batch`\n  * `shuffle_fn`\n  * `show_batch`\n  \nLet's build our two `DataLoaders` real fast and then we'll explore why. We'll make it a very small subset for the sake of example:","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"path = Path(\"../input/siim-isic-melanoma-classification\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df = pd.read_csv(path/'train.csv')\ntest_df = pd.read_csv(path/'test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I want to adjust the `image_name` to include either the `train` or `test` folder, this will help with inference:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df['image_name'] = 'train/' + df['image_name'].astype(str)\ntest_df['image_name'] = 'test/' + test_df['image_name'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tabular `DataLoader`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's build our `TabularDataLoader`. We'll want to define our `procs`, `cat` and `cont` names","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['image_name', 'sex', 'age_approx',\n       'anatom_site_general_challenge', 'diagnosis', 'benign_malignant',\n       'target']\ndf = df[cols]\nprocs = [Categorify, FillMissing, Normalize]\ncat_names  = ['sex', 'anatom_site_general_challenge']\ncont_names = ['age_approx']\nsplitter = RandomSplitter(seed=42)\nsplits = splitter(range_of(df))\nto = TabularPandas(df, procs, cat_names, cont_names,\n                  y_names='target', y_block=CategoryBlock(),\n                  splits=splits)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tab_dl = to.dataloaders(bs=8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we'll move onto Vision","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Vision","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Vision is a bit of a funky one. We're going to follow the `DataBlock` seen [here](https://www.kaggle.com/avirdee/fastai2-dicom-starter) but we're going to adjust the splitter a bit to push everything to the `train`:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"get_x = lambda x:path/f'{x[0]}.dcm'\nget_y=ColReader('target')\nbatch_tfms = aug_transforms(flip_vert=True, max_lighting=0.1, max_zoom=1.05, max_warp=0.)\nblocks = (ImageBlock(cls=PILDicom), CategoryBlock(vocab=[0,1]))\nmelanoma = DataBlock(blocks=blocks,\n                   get_x=get_x,\n                   splitter=splitter,\n                   item_tfms=Resize(128),\n                   get_y=ColReader('target'),\n                   batch_tfms=batch_tfms)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vis_dl = melanoma.dataloaders(df, bs=8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we grabbed `[0]` to get the `train` `DataLoader`:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"type(vis_dl[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Combining the `DataLoaders`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now for the part we all want and care about, how do we combine these two `DataLoaders`? We're going to build a \"mock\" `DataLoader` instead. It will have the same basic jobs as the regular `DataLoader`, but each time we call something for one, we will call it for both `DataLoaders`. We can do this as any time we iterate over the `DataLoader`, what we actually do is call `dl.one_batch` (during training, etc) so we just need to make those three functions I mentioned above. First, let's make our base.\n\nWhat we want to do is override both `DataLoader`'s `shuffle_fn`, as what this function does is return a list of index's to grab. If the index's are the same, the data is the same, and that's all there is to it. In this particular case, we're going to be calling their `shuffle_fn`, etc multiple times, so we need to keep track of who actually is being called. We'll store this in a value called `count`. When it's 0, we'll grab some index's. When it's 1, we will return those index's (this is done at the beginning of each epoch to shuffle the `DataLoader`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## MixedDL","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai2.data.load import _FakeLoader, _loaders","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class MixedDL():\n    def __init__(self, tab_dl:TabDataLoader, vis_dl:TfmdDL, device='cuda:0'):\n        \"Stores away `tab_dl` and `vis_dl`, and overrides `shuffle_fn`\"\n        self.device = device\n        tab_dl.shuffle_fn = self.shuffle_fn\n        vis_dl.shuffle_fn = self.shuffle_fn\n        self.dls = [tab_dl, vis_dl]\n        self.count = 0\n        self.fake_l = _FakeLoader(self, False, 0, 0)\n    \n    def __len__(self): return len(self.dls[0])\n        \n    def shuffle_fn(self, idxs):\n        \"Generates a new `rng` based upon which `DataLoader` is called\"\n        if self.count == 0: # if we haven't generated an rng yet\n            self.rng = self.dls[0].rng.sample(idxs, len(idxs))\n            self.count += 1\n            return self.rng\n        else:\n            self.count = 0\n            return self.rng\n        \n    def to(self, device): self.device = device","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And that's all there is to it. How do we check? Let's check the `DataLoader`'s index's","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"vis_dl[0].get_idxs()[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tab_dl[0].get_idxs()[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Right now we see they are different","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mixed_dl = MixedDL(tab_dl[0], vis_dl[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mixed_dl.dls[0].get_idxs()[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mixed_dl.dls[1].get_idxs()[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we can see our data is being shuffled the exact same way! Great! ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## One Batch\n\nWhat more can we do?\n\nWe need to get our `one_batch` properly working, so let's focus on that. We'll make use of `fastcore`'s `patch` ability to save some room.\n\nFor our `grab_batch` function, we'll want to return 3 `xs` and one `y`:\n  * Categorical\n  * Continuous\n  * Image\n  \nThe first two are stored in the first two index's of the tabular batch, the final in the x of the vision batch. Let's write that:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"But we have to do that in a few different ways. Specifically, this occurs in the `__iter__` function, and `one_batch` will grab the `first`. So let's build our `__iter__`:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"@patch\ndef __iter__(dl:MixedDL):\n    \"Iterate over your `DataLoader`\"\n    z = zip(*[_loaders[i.fake_l.num_workers==0](i.fake_l) for i in dl.dls])\n    for b in z:\n        if dl.device is not None: \n            b = to_device(b, dl.device)\n        batch = []\n        batch.extend(dl.dls[0].after_batch(b[0])[:2])\n        batch.append(dl.dls[1].after_batch(b[1][0]))\n        try: # In case the data is unlabelled\n            batch.append(b[1][1])\n            yield tuple(batch)\n        except:\n            yield tuple(batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now what did we do here? For us to iterate over everything nicely, we need to wrap it all in a `zip` (this is how the `__iter__` is on the original `DataLoader`. Then we want to return a set of `x` and `y`. \n\nBut it's a little more than that. We need to run our `after_batch` transforms here, and before we do so, as we want those to be done on the GPU if possible, we need to (recursevly) convert all of our tensors to GPU via `to_device`. Finally we run the respective `after_batch` on each item needed. So now let's do `one_batch`:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"@patch\ndef one_batch(x:MixedDL):\n    \"Grab a batch from the `DataLoader`\"\n    with x.fake_l.no_multiproc(): res = first(x)\n    if hasattr(x, 'it'): delattr(x, 'it')\n    return res","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`one_batch` will need to call the first from the `DataLoader` (which is just `next(iter(dl))`), and if we are keeping track of an iterator, delete it. Finally returning our batch","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"And now let's see if it works:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch = mixed_dl.one_batch()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch[2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch[3]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we see our batch returns what we want! What's left? Let's see if we can't merge our `show_batch`'s together:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Show Batch","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"`show_batch` is probably the easiest out of all of them. We can try to improve it by say splitting up the batch and whatnot, or we can keep it easy and simple. Let's show a batch (which operates on the same batch in both) for both of our `DataLoaders`:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"@patch\ndef show_batch(x:MixedDL):\n    \"Show a batch from multiple `DataLoaders`\"\n    for dl in x.dls:\n        dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Does it work? Let's see","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mixed_dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It does! Right away! You now have all the bits needed to fully train your model. Have fun!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Bonus: `test_dl`\n\nNow how would we do inference? Let's look at first generating our `test_dl`. It's acutally very straightforward. For each of our interior `DataLoaders` (tab + image) we want to make their own `test_dl` that we pass to our `DataLoader`","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's build one off of the `test_df`. We'll use the `test_dl` from both of the original `Tab` and `Image` `DataLoaders`:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"im_test = vis_dl.test_dl(test_df)\ntab_test = tab_dl.test_dl(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dl = MixedDL(tab_test, im_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now for inference, (note this will not actually run, you need a model), we'll use my `fastinference` library, as regular `fastai` may not like our `DataLoaders`. First let's `pip` install it:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install fastinference --quiet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And then import the inference module:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastinference.inference import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now to get your classes, all you need to do is run the commented out code below. `decoded_loss` will return your actual class values. To read more on `fastinference` see the documentation [here](muellerzr.github.io/fastinference):","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The only thing we need to do is ensure we specify the number of inputs `learn.dls` is expecting:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#learn.dls.n_inp = 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# preds = learn.get_preds(dl=test_dl, decoded_loss=True)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}