{"cells":[{"metadata":{},"cell_type":"markdown","source":"**This is an attempt to investigate the effectiveness of randomly wired cnns on this difficult multi-label task.**\n\nCode for creating the random graphs and turning them into a functional cnn comes from [this excellent github repo](https://github.com/leaderj1001/RandWireNN) , which is an implementation of [this badass paper](https://arxiv.org/pdf/1904.01569.pdf) by Xie et al., 2019.\n\nI am writing this to learn how to work with random graphs and am not an expert in graph theory, random networks, or computer vision. If you see something that could or should be changed or know something that might work better, feel free to say so. Thanks!"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install torchviz","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np, os, sys\nimport cv2\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.utils.data as data\nimport torch.nn.functional as F\nimport torchvision as vision\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nfrom pathlib import Path\nfrom PIL import Image\nfrom contextlib import contextmanager\nimport networkx as nx\nfrom joblib import Parallel, delayed\nfrom tqdm import tqdm\nimport torchviz\nfrom torchviz import make_dot, make_dot_from_trace\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class RandomGraph(object):\n    def __init__(self, node_num, p, k=4, m=5, graph_mode=\"WS\"):\n        self.node_num = node_num\n        self.p = p\n        self.k = k\n        self.m = m\n        self.graph_mode = graph_mode\n\n    def make_graph(self):\n        # reference\n        # https://networkx.github.io/documentation/networkx-1.9/reference/generators.html\n\n        # Code details,\n        # In the case of the nx.random_graphs module, we can give the random seeds as a parameter.\n        # But I have implemented it to handle it in the module.\n        if self.graph_mode is \"ER\":\n            graph = nx.random_graphs.erdos_renyi_graph(self.node_num, self.p)\n        elif self.graph_mode is \"WS\":\n            graph = nx.random_graphs.watts_strogatz_graph(self.node_num, self.k, self.p)\n        elif self.graph_mode is \"BA\":\n            graph = nx.random_graphs.barabasi_albert_graph(self.node_num, self.m)\n\n        return graph\n\n    def get_graph_info(self, graph):\n        in_edges = {}\n        in_edges[0] = []\n        nodes = [0]\n        end = []\n        for node in graph.nodes():\n            neighbors = list(graph.neighbors(node))\n            neighbors.sort()\n\n            edges = []\n            check = []\n            for neighbor in neighbors:\n                if node > neighbor:\n                    edges.append(neighbor + 1)\n                    check.append(neighbor)\n            if not edges:\n                edges.append(0)\n            in_edges[node + 1] = edges\n            if check == neighbors:\n                end.append(node + 1)\n            nodes.append(node + 1)\n        in_edges[self.node_num + 1] = end\n        nodes.append(self.node_num + 1)\n\n        return nodes, in_edges\n\n    def save_random_graph(self, graph, path):\n        if not os.path.isdir(\"saved_graph\"):\n            os.mkdir(\"saved_graph\")\n        nx.write_yaml(graph, \"./saved_graph/\" + path)\n\n    def load_random_graph(self, path):\n        return nx.read_yaml(\"./saved_graph/\" + path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code above is the base class for the generation of a random graph that will be used for constructing the random wired NN according to the given parameters. The idea is to be able to convert a random graph into a directed acyclic graph in which convolution operations are mapped onto the graph nodes while specifying the direction of information flow from input to output. Of the three graph generator methods mentioned:\n\nErdős–Rényi (really random)\nBarabási–Albert (scale-free)\nWatts–Strogatz (small world)\n\nWatts–Strogatz (WS) was found to be the most effective for the tasks examined in the paper.\n\nRandomGraph also allows saving of the random graph, which is useful if you find some network that is very effective for a specific task. We should note the paper says that the variance of different random graphs from the same generator process is low. \n"},{"metadata":{},"cell_type":"markdown","source":"\n\n**Below is a raw representation of a random WS graph representation before cnn operations are mapped to it - this graph is not the same structure as the one we will use for training. Changing the graph_mode above will show the different structures of the generators.**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"rg = RandomGraph(8, 0.75)\ngf = rg.make_graph()\nrg.get_graph_info(gf)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we can see what is being mapped to the nodes of the random graph. The basic unit here is a triplet of  ReLU - seperable 2D convolution - Batch Normalization, weights are randomly initialized."},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_init(m):\n    if isinstance(m, nn.Conv2d):\n        torch.nn.init.xavier_uniform_(m.weight)\n        if m.bias is not None:\n            torch.nn.init.zeros_(m.bias)\n\n\n# reference, Thank you.\n# https://github.com/tstandley/Xception-PyTorch/blob/master/xception.py\n# Reporting 1,\n# I don't know which one is better, between 'bias=False' and 'bias=True'\nclass SeparableConv2d(nn.Module):\n    def __init__(self, in_channels, out_channels, kernel_size=3, stride=1, padding=1, dilation=1, bias=True):\n        super(SeparableConv2d, self).__init__()\n        self.conv = nn.Conv2d(in_channels, in_channels, kernel_size, stride, padding, dilation, groups=in_channels, bias=bias)\n        self.pointwise = nn.Conv2d(in_channels, out_channels, 1, 1, 0, 1, 1, bias=bias)\n\n        # self.apply(weights_init)\n\n    def forward(self, x):\n        x = self.conv(x)\n        x = self.pointwise(x)\n        return x\n\n\n# ReLU-convolution-BN triplet\nclass Unit(nn.Module):\n    def __init__(self, in_channels, out_channels, stride=1):\n        super(Unit, self).__init__()\n\n        self.dropout_rate = 0.2\n\n        self.unit = nn.Sequential(\n            nn.ReLU(),\n            SeparableConv2d(in_channels, out_channels, stride=stride),\n            nn.BatchNorm2d(out_channels),\n            nn.Dropout(self.dropout_rate)\n        )\n\n    def forward(self, x):\n        return self.unit(x)\n\n\n# Reporting 2,\n# In the paper, they said \"The aggregation is done by weighted sum with learnable positive weights\".\nclass Node(nn.Module):\n    def __init__(self, in_degree, in_channels, out_channels, stride=1):\n        super(Node, self).__init__()\n        self.in_degree = in_degree\n        if len(self.in_degree) > 1:\n            # self.weights = nn.Parameter(torch.zeros(len(self.in_degree), requires_grad=True))\n            self.weights = nn.Parameter(torch.ones(len(self.in_degree), requires_grad=True))\n        self.unit = Unit(in_channels, out_channels, stride=stride)\n\n    def forward(self, *input):\n        if len(self.in_degree) > 1:\n            x = (input[0] * torch.sigmoid(self.weights[0]))\n            for index in range(1, len(input)):\n                x += (input[index] * torch.sigmoid(self.weights[index]))\n            out = self.unit(x)\n\n            # different paper, add identity mapping\n            # out += x\n        else:\n            out = self.unit(input[0])\n        return out\n\n\nclass RandWire(nn.Module):\n    def __init__(self, node_num, p, in_channels, out_channels, graph_mode, is_train, name):\n        super(RandWire, self).__init__()\n        self.node_num = node_num\n        self.p = p\n        self.in_channels = in_channels\n        self.out_channels = out_channels\n        self.graph_mode = graph_mode\n        self.is_train = is_train\n        self.name = name\n\n        # get graph nodes and in edges\n        graph_node = RandomGraph(self.node_num, self.p, graph_mode=graph_mode)\n        if self.is_train:\n            graph = graph_node.make_graph()\n            self.nodes, self.in_edges = graph_node.get_graph_info(graph)\n            graph_node.save_random_graph(graph, name)\n        else:\n            graph = graph_node.load_random_graph(name)\n            self.nodes, self.in_edges = graph_node.get_graph_info(graph)\n\n        # define input Node\n        self.module_list = nn.ModuleList([Node(self.in_edges[0], self.in_channels, self.out_channels, stride=2)])\n        # define the rest Node\n        self.module_list.extend([Node(self.in_edges[node], self.out_channels, self.out_channels) \n                                 for node in self.nodes if node > 0])\n\n    def forward(self, x):\n        memory = {}\n        # start vertex\n        out = self.module_list[0].forward(x)\n        memory[0] = out\n\n        # the rest vertex\n        for node in range(1, len(self.nodes) - 1):\n            # print(node, self.in_edges[node][0], self.in_edges[node])\n            if len(self.in_edges[node]) > 1:\n                out = self.module_list[node].forward(*[memory[in_vertex] for in_vertex in self.in_edges[node]])\n            else:\n                out = self.module_list[node].forward(memory[self.in_edges[node][0]])\n            memory[node] = out\n\n        # Reporting 3,\n        # How do I handle the last part?\n        # It has two kinds of methods.\n        # first, Think of the last module as a Node and collect the data by proceeding in the same way as the previous operation.\n        # second, simply sum the data and export the output.\n\n        # My Opinion\n        # out = self.module_list[self.node_num + 1].forward(*[memory[in_vertex] for in_vertex in self.in_edges[self.node_num + 1]])\n\n        # In paper\n        # print(\"self.in_edges: \", self.in_edges[self.node_num + 1], self.in_edges[self.node_num + 1][0])\n        out = memory[self.in_edges[self.node_num + 1][0]]\n        for in_vertex_index in range(1, len(self.in_edges[self.node_num + 1])):\n            out += memory[self.in_edges[self.node_num + 1][in_vertex_index]]\n        out = out / len(self.in_edges[self.node_num + 1])\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class RandNN(nn.Module):\n    def __init__(self, node_num, p, in_channels, out_channels, graph_mode, model_mode, dataset_mode, is_train):\n        super(RandNN, self).__init__()\n        self.node_num = node_num\n        self.p = p\n        self.in_channels = in_channels\n        self.out_channels = out_channels\n        self.graph_mode = graph_mode\n        self.model_mode = model_mode\n        self.is_train = is_train\n        self.dataset_mode = dataset_mode\n\n        self.num_classes = 1103\n        self.dropout_rate = 0.2\n\n        if self.dataset_mode is \"met\":\n            self.num_classes = 1103\n\n        if self.model_mode is \"met\":\n            self.REGULAR_conv1 = nn.Sequential(\n                nn.Conv2d(in_channels=3, out_channels=self.out_channels // 2, kernel_size=3, padding=1),\n                nn.BatchNorm2d(self.out_channels // 2)\n            )\n            self.REGULAR_conv2 = nn.Sequential(\n                RandWire(self.node_num // 2, self.p, self.in_channels // 2, self.out_channels, self.graph_mode, self.is_train, name=\"REGULAR_conv2\")\n            )\n            self.REGULAR_conv3 = nn.Sequential(\n                RandWire(self.node_num, self.p, self.in_channels, self.out_channels * 2, self.graph_mode, \n                         self.is_train, name=\"REGULAR_conv3\")\n            )\n            self.REGULAR_conv4 = nn.Sequential(\n                RandWire(self.node_num, self.p, self.in_channels * 2, self.out_channels * 4, self.graph_mode, \n                         self.is_train, name=\"REGULAR_conv4\")\n            )\n            self.REGULAR_conv5 = nn.Sequential(\n                RandWire(self.node_num, self.p, self.in_channels * 4, self.out_channels * 8, self.graph_mode, \n                         self.is_train, name=\"REGULAR_conv5\")\n            )\n            self.REGULAR_classifier = nn.Sequential(\n                nn.Conv2d(self.in_channels * 8, 1280, kernel_size=1),\n                nn.BatchNorm2d(1280)\n            )\n\n        self.output = nn.Sequential(\n            nn.Dropout(self.dropout_rate),\n            nn.Linear(1280, self.num_classes)\n        )\n\n    def forward(self, x):\n        if self.model_mode is \"met\":\n            out = self.REGULAR_conv1(x)\n            out = self.REGULAR_conv2(out)\n            out = self.REGULAR_conv3(out)\n            out = self.REGULAR_conv4(out)\n            out = self.REGULAR_conv5(out)\n            out = self.REGULAR_classifier(out)\n\n        # global average pooling\n        out = F.avg_pool2d(out, kernel_size=x.size()[2:])\n        out = torch.squeeze(out)\n        out = self.output(out)\n\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def rw(f=None):\n    m = RandNN(32, 0.75, 109, 109, 'WS', 'met', 'met', 'train')\n    return m","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m = rw()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#m","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr = pd.read_csv('../input/train.csv')\nte = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import fastai\nfrom fastai.vision import *\npath = Path('../input')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"SZ = 128\nBS = 32\n\ntrain, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.png') \n               for df, folder in zip([tr, te], ['train', 'test'])]\ndata = (train.split_by_rand_pct(0.1, seed=42)\n        .label_from_df(cols='attribute_ids', label_delim=' ')\n        .add_test(test)\n        .transform(get_transforms(), size=SZ, resize_method=ResizeMethod.PAD, padding_mode='border',)\n        .databunch(path=Path('.'), bs=BS).normalize(imagenet_stats))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109\nclass FocalLoss(nn.Module):\n    def __init__(self, gamma=2):\n        super().__init__()\n        self.gamma = gamma\n\n    def forward(self, logit, target):\n        target = target.float()\n        max_val = (-logit).clamp(min=0)\n        loss = logit - logit * target + max_val + \\\n               ((-max_val).exp() + (-logit - max_val).exp()).log()\n\n        invprobs = F.logsigmoid(-logit * (target * 2.0 - 1.0))\n        loss = (invprobs * self.gamma).exp() * loss\n        if len(loss.size())==2:\n            loss = loss.sum(dim=1)\n        return loss.mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, \n                    base_arch=rw, \n                    cut = 6,\n                    loss_func=FocalLoss(), \n                    metrics=fbeta)\n\nlearn = learn.to_fp16(loss_scale=64, dynamic=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#learn.lr_find()\n#learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(10, slice(1e-3,2e-2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot()\nlearn.recorder.plot_losses()\nlearn.recorder.plot_metrics()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def find_best_fixed_threshold(preds, targs, do_plot=True):\n    score = []\n    thrs = np.arange(0, 0.5, 0.01)\n    for thr in progress_bar(thrs):\n        score.append(fbeta(valid_preds[0],valid_preds[1], thresh=thr))\n    score = np.array(score)\n    pm = score.argmax()\n    best_thr, best_score = thrs[pm], score[pm].item()\n    print(f'thr={best_thr:.3f}', f'F2={best_score:.3f}')\n    if do_plot:\n        plt.plot(thrs, score)\n        plt.vlines(x=best_thr, ymin=score.min(), ymax=score.max())\n        plt.text(best_thr+0.03, best_score-0.01, f'$F_{2}=${best_score:.3f}', fontsize=14);\n        plt.show()\n    return best_thr\n\ni2c = np.array([[i, c] for c, i in learn.data.train_ds.y.c2i.items()]).astype(int) # indices to class number correspondence\n\ndef join_preds(preds, thr):\n    return [' '.join(i2c[np.where(t==1)[0],1].astype(str)) for t in (preds[0].sigmoid()>thr).long()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Validation predictions\nvalid_preds = learn.get_preds(DatasetType.Valid)\nbest_thr = find_best_fixed_threshold(*valid_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = learn.get_preds(DatasetType.Test)\nte.attribute_ids = join_preds(test_preds, best_thr)\nte.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"te.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = torch.randn(2,3,64,64).half().cuda()\nmake_dot(learn.model(x), params=dict(learn.model.named_parameters()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = torch.randn(2,3,64,64).half().cuda()\nmake_dot(learn.model[0][0:2](x), params=dict(learn.model[0][0:2].named_parameters()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}