{"cells":[{"metadata":{},"cell_type":"markdown","source":"**In my previous [kernel](https://www.kaggle.com/interneuron/randomwire) I showed how to use random graphs to build a convolutional neural network.** \n\nWhile these randomly wired networks perform quite well, they cannot (yet) compete with networks pre-trained on the ImageNet. So until someone finds a particular random net they like enough to train on ImageNet and release the weights, training from scratch is the only option. Proper references for the random network can be found in the other kernel.\n\nHere, we will try something...interesting.\n\nWhile researching about random networks, I found [this](https://github.com/vinayprabhu/Network_Science_Meets_Deep_Learning/blob/master/1_MNIST_C_Elegans.ipynb) intriguing work. Using the techniques from the RandomWire paper for converting a graph into a functional cnn, one can take, say, the known wiring diagram of the nervous system of the nematode worm *Caenorhabditis elegans*, and turn it into a convolutional network. I simply added functionality to plug the graph of the worm's nervous system into the network builder.\n\nI find this idea quite weird. \n\nWhile *C. elegans* does posess some light-sensitive cells to help it avoid dangerous stuff like UV radiation, it has no eyes, lives in the dark, and completely lacks what we understand as a visual system. What we are going to do here is repourpose the animals *entire* nervous system for image classification. \n\nWtf?\n\nNevertheless, it is a network architecture I have not seen used before and I hope others will find it as interesting as I do."},{"metadata":{},"cell_type":"markdown","source":"**First we get the graph of the nervous system**"},{"metadata":{"trusted":true},"cell_type":"code","source":"! wget https://www.cise.ufl.edu/research/sparse/MM/Newman/celegansneural.tar.gz\n! tar -xvzf celegansneural.tar.gz","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport scipy.io\nimport matplotlib.pyplot as plt\nimport networkx as nx\n\ncelegans = scipy.io.mmread('celegansneural/celegansneural.mtx').toarray()\ncelegans","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"celegans.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vfunc = np.vectorize(lambda a : 1 if a != 0 else 0)\ncelegansbin = vfunc(celegans)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"celegans_g = nx.from_numpy_array(celegansbin)\nnx.draw(celegans_g, node_size=7)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like a mess, but there must be something to it as the worms have been successfully using it for quite some time."},{"metadata":{"trusted":true},"cell_type":"code","source":"celegans_g.number_of_edges()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"celegans_g.number_of_nodes()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install torchviz","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np, os, sys\nimport cv2\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.utils.data as data\nimport torch.nn.functional as F\nimport torchvision as vision\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nfrom pathlib import Path\nfrom PIL import Image\nfrom contextlib import contextmanager\nimport networkx as nx\nfrom joblib import Parallel, delayed\nfrom tqdm import tqdm\n#import torchviz\n#from torchviz import make_dot, make_dot_from_trace\n#print(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another difference from the previous kernel, I added a function to make a second kind of **small-world** graph: the **newman_watts_strogatz_graph**."},{"metadata":{"trusted":true},"cell_type":"code","source":"class RandomGraph(object):\n    def __init__(self, node_num, p, k=4, m=5, graph_mode=\"WS\"):\n        self.node_num = node_num\n        self.p = p\n        self.k = k\n        self.m = m\n        self.graph_mode = graph_mode\n\n    def make_graph(self):\n        # reference\n        # https://networkx.github.io/documentation/networkx-1.9/reference/generators.html\n\n        # Code details,\n        # In the case of the nx.random_graphs module, we can give the random seeds as a parameter.\n        # But I have implemented it to handle it in the module.\n        if self.graph_mode is \"ER\":\n            graph = nx.random_graphs.erdos_renyi_graph(self.node_num, self.p)\n        elif self.graph_mode is \"WS\":\n            graph = nx.random_graphs.watts_strogatz_graph(self.node_num, self.k, self.p)\n        elif self.graph_mode is \"NWS\":\n            graph = nx.random_graphs.newman_watts_strogatz_graph(self.node_num, self.k, self.p)\n        elif self.graph_mode is \"BA\":\n            graph = nx.random_graphs.barabasi_albert_graph(self.node_num, self.m)\n        elif self.graph_mode is 'worm':\n            graph = celegans_g\n\n        return graph\n\n    def get_graph_info(self, graph):\n        in_edges = {}\n        in_edges[0] = []\n        nodes = [0]\n        end = []\n        if self.graph_mode is 'worm':\n            self.node_num = len(graph.nodes())\n        for node in graph.nodes():\n            neighbors = list(graph.neighbors(node))\n            neighbors.sort()\n\n            edges = []\n            check = []\n            for neighbor in neighbors:\n                if node > neighbor:\n                    edges.append(neighbor + 1)\n                    check.append(neighbor)\n            if not edges:\n                edges.append(0)\n            in_edges[node + 1] = edges\n            if check == neighbors:\n                end.append(node + 1)\n            nodes.append(node + 1)\n        in_edges[self.node_num + 1] = end\n        nodes.append(self.node_num + 1)\n\n        return nodes, in_edges\n    \n\n    def save_random_graph(self, graph, path):\n        if not os.path.isdir(\"saved_graph\"):\n            os.mkdir(\"saved_graph\")\n        nx.write_yaml(graph, \"./saved_graph/\" + path)\n\n    def load_random_graph(self, path):\n        return nx.read_yaml(\"./saved_graph/\" + path)\n    \ndef _save_random_graph(graph, path):\n    if not os.path.isdir(\"saved_graph\"):\n        os.mkdir(\"saved_graph\")\n    nx.write_yaml(graph, \"./saved_graph/\" + path)\n\ndef _load_random_graph(path):\n    return nx.read_yaml(\"./saved_graph/\" + path)\n\ndef _get_graph_info(graph):\n    in_edges = {}\n    in_edges[0] = []\n    nodes = [0]\n    end = []\n    node_num = len(graph.nodes())\n    for node in graph.nodes():\n        neighbors = list(graph.neighbors(node))\n        neighbors.sort()\n\n        edges = []\n        check = []\n        for neighbor in neighbors:\n            if node > neighbor:\n                edges.append(neighbor + 1)\n                check.append(neighbor)\n        if not edges:\n            edges.append(0)\n        in_edges[node + 1] = edges\n        if check == neighbors:\n            end.append(node + 1)\n        nodes.append(node + 1)\n    in_edges[node_num + 1] = end\n    nodes.append(node_num + 1)\n\n    return nodes, in_edges","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n**In the previous kernel I showed below a simple WS graph. In the cell below it get the same sort of representation from the worm. Uncomment the cell with in_edges to see, its quite a bit more complex than the little WS graph... **\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"rg = RandomGraph(8, 0.75)\ngf = rg.make_graph()\nrg.get_graph_info(gf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nodes, in_edges = _get_graph_info(celegans_g)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#in_edges","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_init(m):\n    if isinstance(m, nn.Conv2d):\n        torch.nn.init.xavier_uniform_(m.weight)\n        if m.bias is not None:\n            torch.nn.init.zeros_(m.bias)\n\n\n# reference, Thank you.\n# https://github.com/tstandley/Xception-PyTorch/blob/master/xception.py\n# Reporting 1,\n# I don't know which one is better, between 'bias=False' and 'bias=True'\nclass SeparableConv2d(nn.Module):\n    def __init__(self, in_channels, out_channels, kernel_size=3, stride=1, padding=1, dilation=1, bias=True):\n        super(SeparableConv2d, self).__init__()\n        self.conv = nn.Conv2d(in_channels, in_channels, kernel_size, stride, padding, dilation, groups=in_channels, bias=bias)\n        self.pointwise = nn.Conv2d(in_channels, out_channels, 1, 1, 0, 1, 1, bias=bias)\n\n        # self.apply(weights_init)\n\n    def forward(self, x):\n        x = self.conv(x)\n        x = self.pointwise(x)\n        return x\n\n\n# ReLU-convolution-BN triplet\nclass Unit(nn.Module):\n    def __init__(self, in_channels, out_channels, stride=1):\n        super(Unit, self).__init__()\n\n        self.dropout_rate = 0.2\n\n        self.unit = nn.Sequential(\n            nn.ReLU(),\n            SeparableConv2d(in_channels, out_channels, stride=stride),\n            nn.BatchNorm2d(out_channels),\n            nn.Dropout(self.dropout_rate)\n        )\n\n    def forward(self, x):\n        return self.unit(x)\n\n\n# Reporting 2,\n# In the paper, they said \"The aggregation is done by weighted sum with learnable positive weights\".\nclass Node(nn.Module):\n    def __init__(self, in_degree, in_channels, out_channels, stride=1):\n        super(Node, self).__init__()\n        self.in_degree = in_degree\n        if len(self.in_degree) > 1:\n            # self.weights = nn.Parameter(torch.zeros(len(self.in_degree), requires_grad=True))\n            self.weights = nn.Parameter(torch.ones(len(self.in_degree), requires_grad=True))\n        self.unit = Unit(in_channels, out_channels, stride=stride)\n\n    def forward(self, *input):\n        if len(self.in_degree) > 1:\n            x = (input[0] * torch.sigmoid(self.weights[0]))\n            for index in range(1, len(input)):\n                x += (input[index] * torch.sigmoid(self.weights[index]))\n            out = self.unit(x)\n\n            # different paper, add identity mapping\n            # out += x\n        else:\n            out = self.unit(input[0])\n        return out\n\n\nclass RandWire(nn.Module):\n    def __init__(self, node_num, p, in_channels, out_channels, graph_mode, is_train, name):\n        super(RandWire, self).__init__()\n        self.node_num = node_num\n        self.p = p\n        self.in_channels = in_channels\n        self.out_channels = out_channels\n        self.graph_mode = graph_mode\n        self.is_train = is_train\n        self.name = name\n\n        # get graph nodes and in edges\n        if self.graph_mode is 'worm':\n            graph_node = celegans_g\n        else:\n            graph_node = RandomGraph(self.node_num, self.p, graph_mode=graph_mode)\n        if self.is_train:\n            if self.graph_mode is 'worm':\n                graph = graph_node\n                self.nodes, self.in_edges = _get_graph_info(graph)\n                _save_random_graph(graph, name)\n            else: \n                graph = graph_node.make_graph()\n                self.nodes, self.in_edges = get_graph_info(graph)\n                graph_node.save_random_graph(graph, name)\n        else:\n            if self.graph_mode is 'worm':\n                graph = _load_random_graph(name)\n                self.nodes, self.in_edges = _get_graph_info(graph)\n            else:\n                graph = graph_node.load_random_graph(name)\n                self.nodes, self.in_edges = graph_node.get_graph_info(graph)\n\n        # define input Node\n        self.module_list = nn.ModuleList([Node(self.in_edges[0], self.in_channels, self.out_channels, stride=2)])\n        # define the rest Node\n        self.module_list.extend([Node(self.in_edges[node], self.out_channels, self.out_channels) \n                                 for node in self.nodes if node > 0])\n\n    def forward(self, x):\n        memory = {}\n        # start vertex\n        out = self.module_list[0].forward(x)\n        memory[0] = out\n\n        # the rest vertex\n        for node in range(1, len(self.nodes) - 1):\n            # print(node, self.in_edges[node][0], self.in_edges[node])\n            if len(self.in_edges[node]) > 1:\n                out = self.module_list[node].forward(*[memory[in_vertex] for in_vertex in self.in_edges[node]])\n            else:\n                out = self.module_list[node].forward(memory[self.in_edges[node][0]])\n            memory[node] = out\n\n        # Reporting 3,\n        # How do I handle the last part?\n        # It has two kinds of methods.\n        # first, Think of the last module as a Node and collect the data by proceeding in the same way as the previous operation.\n        # second, simply sum the data and export the output.\n\n        # My Opinion\n        # out = self.module_list[self.node_num + 1].forward(*[memory[in_vertex] for in_vertex in self.in_edges[self.node_num + 1]])\n\n        # In paper\n        # print(\"self.in_edges: \", self.in_edges[self.node_num + 1], self.in_edges[self.node_num + 1][0])\n        out = memory[self.in_edges[self.node_num + 1][0]]\n        for in_vertex_index in range(1, len(self.in_edges[self.node_num + 1])):\n            out += memory[self.in_edges[self.node_num + 1][in_vertex_index]]\n        out = out / len(self.in_edges[self.node_num + 1])\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class RandNN(nn.Module):\n    def __init__(self, node_num, p, in_channels, out_channels, graph_mode, model_mode, dataset_mode, is_train):\n        super(RandNN, self).__init__()\n        self.node_num = node_num\n        self.p = p\n        self.in_channels = in_channels\n        self.out_channels = out_channels\n        self.graph_mode = graph_mode\n        self.model_mode = model_mode\n        self.is_train = is_train\n        self.dataset_mode = dataset_mode\n\n        self.num_classes = 1103\n        self.dropout_rate = 0.2\n\n        if self.dataset_mode is \"met\":\n            self.num_classes = 1103\n\n        if self.model_mode is \"met\":\n            self.REGULAR_conv1 = nn.Sequential(\n                nn.Conv2d(in_channels=3, out_channels=self.out_channels // 2, kernel_size=3, padding=1),\n                nn.BatchNorm2d(self.out_channels // 2)\n            )\n            self.REGULAR_conv2 = nn.Sequential(\n                RandWire(self.node_num // 2, self.p, self.in_channels // 2, self.out_channels, self.graph_mode, self.is_train, name=\"REGULAR_conv2\")\n            )\n            self.REGULAR_conv3 = nn.Sequential(\n                RandWire(self.node_num, self.p, self.in_channels, self.out_channels * 2, self.graph_mode, \n                         self.is_train, name=\"REGULAR_conv3\")\n            )\n            self.REGULAR_conv4 = nn.Sequential(\n                RandWire(self.node_num, self.p, self.in_channels * 2, self.out_channels * 4, self.graph_mode, \n                         self.is_train, name=\"REGULAR_conv4\")\n            )\n            self.REGULAR_conv5 = nn.Sequential(\n                RandWire(self.node_num, self.p, self.in_channels * 4, self.out_channels * 8, self.graph_mode, \n                         self.is_train, name=\"REGULAR_conv5\")\n            )\n            self.REGULAR_classifier = nn.Sequential(\n                nn.Conv2d(self.in_channels * 8, 1280, kernel_size=1),\n                nn.BatchNorm2d(1280)\n            )\n\n#        self.output = nn.Sequential(\n#            nn.Dropout(self.dropout_rate),\n#            nn.Linear(1280, self.num_classes)\n#        )\n        \n        \n        elif self.model_mode is \"worm\":\n            self.worm_conv1 = nn.Sequential(\n                nn.Conv2d(in_channels=3, out_channels=self.out_channels // 2, kernel_size=3, padding=1),\n                nn.BatchNorm2d(self.out_channels // 2)\n            )\n            \n            self.worm_conv2 = nn.Sequential(\n                RandWire(self.node_num // 2, self.p, self.in_channels // 2, self.out_channels, \n                         self.graph_mode, self.is_train, name=\"worm_conv2\")\n            )\n            \n            self.worm_classifier = nn.Sequential(\n                nn.Conv2d(self.in_channels * 8, 1280, kernel_size=1),\n                nn.BatchNorm2d(1280)\n            )\n            \n        self.output = nn.Sequential(\n            nn.Dropout(self.dropout_rate),\n            nn.Linear(1280, self.num_classes)\n        )\n\n    def forward(self, x):\n        if self.model_mode is \"met\":\n            out = self.REGULAR_conv1(x)\n            out = self.REGULAR_conv2(out)\n            out = self.REGULAR_conv3(out)\n            out = self.REGULAR_conv4(out)\n            out = self.REGULAR_conv5(out)\n            out = self.REGULAR_classifier(out)\n        elif self.model_mode is 'worm':\n            out = self.worm_conv1(x)\n            out = self.worm_conv2(out)\n            out = self.worm_classifier(out)\n\n        # global average pooling\n        out = F.avg_pool2d(out, kernel_size=x.size()[2:])\n        out = torch.squeeze(out)\n        out = self.output(out)\n\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def count_parameters(model):\n    return sum(p.numel() for p in model.parameters() if p.requires_grad)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def rw(f=None):\n    m = RandNN(1, 0.75, 32, 32, 'worm', 'met', 'met', 'train')\n    return m","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"m = rw()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"count_parameters(m)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"tr = pd.read_csv('../input/train.csv')\ntr = tr.sample(frac=0.1).reset_index(drop=True)\nte = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import fastai\nfrom fastai.vision import *\npath = Path('../input/')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"SZ = 128\nBS = 16\n\ntrain, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.png') \n               for df, folder in zip([tr, te], ['train', 'test'])]\ndata = (train.split_by_rand_pct(0.1, seed=42)\n        .label_from_df(cols='attribute_ids', label_delim=' ')\n        .add_test(test)\n        .transform(get_transforms(), size=SZ, resize_method=ResizeMethod.PAD, padding_mode='border',)\n        .databunch(path=Path('.'), bs=BS).normalize(imagenet_stats))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109\nclass FocalLoss(nn.Module):\n    def __init__(self, gamma=2):\n        super().__init__()\n        self.gamma = gamma\n\n    def forward(self, logit, target):\n        target = target.float()\n        max_val = (-logit).clamp(min=0)\n        loss = logit - logit * target + max_val + \\\n               ((-max_val).exp() + (-logit - max_val).exp()).log()\n\n        invprobs = F.logsigmoid(-logit * (target * 2.0 - 1.0))\n        loss = (invprobs * self.gamma).exp() * loss\n        if len(loss.size())==2:\n            loss = loss.sum(dim=1)\n        return loss.mean()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, \n                    base_arch=rw, \n                    cut = 5,\n                    loss_func=FocalLoss(), \n                    metrics=fbeta)\n\nlearn = learn.to_fp16(loss_scale=64, dynamic=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#learn.lr_find()\n#learn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(8, slice(1e-3,2e-2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot()\nlearn.recorder.plot_losses()\nlearn.recorder.plot_metrics()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def find_best_fixed_threshold(preds, targs, do_plot=True):\n    score = []\n    thrs = np.arange(0, 0.5, 0.01)\n    for thr in progress_bar(thrs):\n        score.append(fbeta(valid_preds[0],valid_preds[1], thresh=thr))\n    score = np.array(score)\n    pm = score.argmax()\n    best_thr, best_score = thrs[pm], score[pm].item()\n    print(f'thr={best_thr:.3f}', f'F2={best_score:.3f}')\n    if do_plot:\n        plt.plot(thrs, score)\n        plt.vlines(x=best_thr, ymin=score.min(), ymax=score.max())\n        plt.text(best_thr+0.03, best_score-0.01, f'$F_{2}=${best_score:.3f}', fontsize=14);\n        plt.show()\n    return best_thr\n\ni2c = np.array([[i, c] for c, i in learn.data.train_ds.y.c2i.items()]).astype(int) # indices to class number correspondence\n\ndef join_preds(preds, thr):\n    return [' '.join(i2c[np.where(t==1)[0],1].astype(str)) for t in (preds[0].sigmoid()>thr).long()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Validation predictions\nvalid_preds = learn.get_preds(DatasetType.Valid)\nbest_thr = find_best_fixed_threshold(*valid_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = learn.get_preds(DatasetType.Test)\nte.attribute_ids = join_preds(test_preds, best_thr)\nte.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"te.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = torch.randn(2,3,64,64).half().cuda()\nmake_dot(learn.model(x), params=dict(learn.model.named_parameters()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = torch.randn(2,3,64,64).half().cuda()\nmake_dot(learn.model[0][0:2](x), params=dict(learn.model[0][0:2].named_parameters()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}