{"cells":[{"metadata":{"_uuid":"d582e2e070417ac636b9333c3f93cad006f74cb7"},"cell_type":"markdown","source":"# Minimalistic-pipeline\nThe purpose of this pipeline is to create a minimalistic pipeline, without any preprocessing and not using any pre-trained embeddings. Nor will any cross validation be used or other 'fancy' techniques. The goal is thus just to get a initial benchmark for the most simplest architecture to see how hard the problem really is. I also want to use as few 3rd-party packages as possible, just to keep everything as minimalistic as I can. Therefore I'm going to implement my own functions rather than using spacy, nltk or other packages that can help me.\nI say the \"most simplest architecture\" but here I use an LSTM which isn't so simple... This is because the highest scoring models I've seen are all some form of RNN like a LSTM, Bi-LSTM or GRU. This is the reason for my choice of using a LSTM in the initial test."},{"metadata":{"_uuid":"e80620a79b3c05528bef2950778dede5d5c7d1f0"},"cell_type":"markdown","source":"Start by importing everything we need"},{"metadata":{"trusted":true,"_uuid":"bc9d3d718b62604dee303a421afe7642b93ecd2e"},"cell_type":"code","source":"import pandas as pd\nfrom collections import Counter\nimport numpy as np\nimport torchvision\nimport torch.nn as nn\nimport torch\nimport torch.nn.functional as F\nfrom torch.autograd import Variable\nimport time\nimport copy\nfrom sklearn.metrics import f1_score, classification_report, confusion_matrix\nfrom sklearn.utils import shuffle","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7a15a84317de06c8ae344abcc8e4d0bd3d4577a"},"cell_type":"markdown","source":"Load dataset"},{"metadata":{"trusted":true,"_uuid":"1aaf868f457dcb763ae27ad173c9af24e372843c"},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_test = pd.read_csv(\"../input/test.csv\")\n\n# Fill missing values\ndf_train.fillna(\"_##_\", inplace=True)  \ndf_test.fillna(\"_##_\", inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aec4d2544965489b19b33a7368ce7182a0c83f57"},"cell_type":"markdown","source":"Extract the data we're interested in and convert questions into lists of words (instead of long strings)"},{"metadata":{"trusted":true,"_uuid":"e6a4401a663c4f9f6fb63cb0acd3c24226a36285"},"cell_type":"code","source":"def split(df):\n    split_questions = []\n    for data in df.itertuples():\n        split_q_tmp = []\n        for w in data[2].split(' '):\n            split_q_tmp.append(w)\n        split_questions.append(split_q_tmp)\n    return split_questions\n        \nX_train = split(df_train)\ny_train = df_train['target'].values.tolist()\nqid_test = df_test['qid'].values.tolist()\nX_test = split(df_test)\n\nprint(len(X_train))\nprint(len(y_train))\nprint(len(X_test))\nprint(len(qid_test))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"574cf17fd3a0ac6584bac3691ebbb47ac72d3878"},"cell_type":"markdown","source":"Get some interesting statistics from our test and validation dataset.\nWe will probably see that the data is heavlity biased and we would probably want to do something about this."},{"metadata":{"trusted":true,"_uuid":"28a1bffd647b276225eda3b0f76cf79ae48cb975"},"cell_type":"code","source":"def get_stat(X,y):\n    num_sincere = 0\n    num_insincere = 0\n    for i in range(len(X)):\n        if y[i] == 0:\n            num_sincere += 1\n        elif y[i] == 1:\n            num_insincere += 1\n        else:\n            print(\"We should not end up here! i={}\".format(i))\n    total = num_sincere + num_insincere\n    print(\"Sincere: {}, insincere: {}\".format(num_sincere, num_insincere))\n    print(\"ratio_sincere, ratio_insincere: {:.4f}, {:.4f}\".format(num_sincere / total, num_insincere / total))\n    return num_sincere, num_insincere\n\nnum_sincere, num_insincere = get_stat(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50e386a86255002ceadd759d712aede380f0786a"},"cell_type":"markdown","source":"The NN would probably guess that all questions are sincere with the bias we have in the dataset! We will not use any fancy oversampling techniques to fix this. All we'll do is just to to extract as many sincere questions as there are insincere ones to balance the dataset 50/50... Better techniques will be tested in later notebooks."},{"metadata":{"trusted":true,"_uuid":"f43a3fe30520285ee76fd401fc8baf343219294c"},"cell_type":"code","source":"def balance(X, y, n):\n    newX = []\n    newY = []\n    sincere_taken = 0\n    insincere_taken = 0\n    for i in range(len(X)):\n        if y[i] == 0 and sincere_taken <= n:\n            newX.append(X[i])\n            newY.append(y[i])\n            sincere_taken += 1\n        elif y[i] == 1 and insincere_taken <= n:\n            newX.append(X[i])\n            newY.append(y[i])\n            insincere_taken += 1\n    print(\"Sincere taken: {}, insincere taken: {}\".format(sincere_taken, insincere_taken))\n    return newX, newY\n\nprint(\"Extracting {} questions from each class!\".format(num_insincere))\n\nX_train, y_train = balance(X_train, y_train, num_insincere)\n\n# Random shuffle\nX_train, y_train = shuffle(X_train, y_train)\n\nprint(len(X_train))\nprint(len(y_train))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5448c35d3d52b7993fd482554de773de4e33fded"},"cell_type":"markdown","source":"Build a vocabulary containing all words so that we can embed each word as a unique integer later"},{"metadata":{"trusted":true,"_uuid":"8e7b1c23774e8581927b3b65f8492b5f6a991a0e"},"cell_type":"code","source":"def get_all_words(questions):\n    all_words = []\n    for question in questions:\n        for word in question:\n            all_words.append(word)\n    return all_words\n\ndef build_vocab(all_words):\n    count = Counter(all_words)\n    return sorted(count, key=count.get, reverse=True)\n\ndef vocab_to_integer(vocab):\n    ''' Map each vocab words to an integer.\n        Starts at 1 since 0 will be used for padding.'''\n    return {word: ii for ii, word in enumerate(vocab, 1)}\n\nall_words = get_all_words(X_train + X_test)\nvocab = build_vocab(all_words)\nvocab_to_int = vocab_to_integer(vocab)\n\nprint(\"Vocab size: {}\".format(len(vocab_to_int)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fff41b824fd3318f88a8a737a136228e2b2ac87c"},"cell_type":"markdown","source":"Embed every word as a integer unique to that word"},{"metadata":{"trusted":true,"_uuid":"446b5e0bc9d4d24595ed9ea406f23ef8a30c86be"},"cell_type":"code","source":"def embed_word_to_int(X, vocab_to_int):\n    embedded_X = []\n    for q in X:\n        tmp_X = []\n        for w in q:\n            tmp_X.append(vocab_to_int[w])\n        embedded_X.append(tmp_X)\n    return embedded_X\n        \nX_train = embed_word_to_int(X_train, vocab_to_int)\nX_test = embed_word_to_int(X_test, vocab_to_int)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f864df0ad74bb75c2d5c024f5daef19d0daf2f9"},"cell_type":"markdown","source":"Now it's time to create a validation set that we can use during training (we use the weights that gave us the best validation result). The train/validation split is 80/20."},{"metadata":{"trusted":true,"_uuid":"7b9fb3f6bcc6c413bb8e194ba97a58c652e95d35"},"cell_type":"code","source":"def create_validation_set(X, y, factor=0.8):\n    num_train = int(len(X) * factor)\n    X_train = X[:num_train]\n    y_train = y[:num_train]\n    X_val = X[num_train:]\n    y_val = y[num_train:]\n    return X_train, y_train, X_val, y_val\n\nX_train_split, y_train_split, X_val_split, y_val_split = create_validation_set(X_train, y_train, factor=0.8)\n\nprint(len(X_train_split))\nprint(len(y_train_split))\nprint(len(X_val_split))\nprint(len(y_val_split))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6288a5941588ae0451bbf88088cdff27040c945"},"cell_type":"markdown","source":"Now we pad the questions and convert both the questions/labels into correctly formatted numpy arrays"},{"metadata":{"trusted":true,"_uuid":"c9d7f8579d74cbc1ad80af95790fd15d375681a9"},"cell_type":"code","source":"def pad_features(questions, sequence_length=50):\n    ''' Pad each question with zeros to the same length.\n        Padding is done in the beginning of the sentence.\n        If question is truncated the truncation starts at the end of the question. '''\n    features = np.zeros((len(questions), sequence_length), dtype=int)\n    for i, row in enumerate(questions):\n        features[i, -len(row):] = np.array(row)[:sequence_length]\n    return features\n\ndef format_labels(labels):\n    ''' What we actually do is one-hot encode the labels so that a sincere question\n        gets the label [0, 1] and a insincere questions gets the label [0, 1] '''\n    y = np.zeros((len(labels), 2), dtype=int)\n    for i in range(len(labels)):\n        if labels[i] == 0:\n            y[i] = [1,0]\n        else:\n            y[i] = [0,1]\n    return y\n        \n# Calculate the max length and pad all questions to this length\nmax_train_len = max(Counter([len(x) for x in X_train]))\nmax_test_len = max(Counter([len(x) for x in X_test]))\nmax_len = max(max_train_len, max_test_len)\n\npad_length = max_len\n\n# Pad and format\nX_train_pad = pad_features(X_train_split, pad_length)\ny_train_pad = format_labels(y_train_split)\nX_val_pad = pad_features(X_val_split, pad_length)\ny_val_pad = format_labels(y_val_split)\nX_test_pad = pad_features(X_test, pad_length)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4d01c4bfa76d819f0626aa22f284618052f7732"},"cell_type":"markdown","source":"Now we convert the data to tensors so that they can be used in pytorch."},{"metadata":{"trusted":true,"_uuid":"cb7e2023ddbedcfb72de90f4746148197aa4f649"},"cell_type":"code","source":"def numpy_to_tensor(X, y=None):\n    X_tensor = Variable(torch.from_numpy(X).long(),\n                        requires_grad=False)\n    y_tensor = None\n    if y is not None:\n        y_tensor = Variable(torch.from_numpy(y).float(),\n                            requires_grad=False)\n    return X_tensor, y_tensor\n\nX_train_tensor, y_train_tensor = numpy_to_tensor(X_train_pad, y_train_pad)\nX_val_tensor, y_val_tensor = numpy_to_tensor(X_val_pad, y_val_pad)\nX_test_tensor, _ = numpy_to_tensor(X_test_pad)\n\nprint(X_train_tensor.shape)\nprint(y_train_tensor.shape)\nprint(X_val_tensor.shape)\nprint(y_val_tensor.shape)\nprint(X_test_tensor.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f000225eab693c3104909f178d063d0a98ada694"},"cell_type":"markdown","source":"Finally time to start with pytorch stuff! Start of by defining hyperparameters."},{"metadata":{"trusted":true,"_uuid":"aa06243bfd40b2ce69bf2b3e603bd263c8ab97da"},"cell_type":"code","source":"# Define hyperparams\nMINIBATCH_SIZE = 16\nLEARNING_RATE = 1e-3\nEPOCHS = 75\nSGD_MOMENTUM = 0.9\nLSTM_HIDDEN_SIZE = 100\nLSTM_EMBEDDING_SIZE = pad_length","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db0fe033fd76a0f0f9996c648594db4506410acf"},"cell_type":"markdown","source":"Create dataset and dataloaders that can be used with pytorch"},{"metadata":{"trusted":true,"_uuid":"9d67712aa7757b20d7c75d65aea44d3d672e3259"},"cell_type":"code","source":"train_dataset = torch.utils.data.TensorDataset(X_train_tensor, y_train_tensor)\nval_dataset = torch.utils.data.TensorDataset(X_val_tensor, y_val_tensor)\ntest_dataset = torch.utils.data.TensorDataset(X_test_tensor)\n\ntrain_loader = torch.utils.data.DataLoader(dataset=train_dataset,\n                                                batch_size=MINIBATCH_SIZE,\n                                                shuffle=True)\nval_loader = torch.utils.data.DataLoader(dataset=val_dataset,\n                                                batch_size=MINIBATCH_SIZE,\n                                                shuffle=True)\ntest_loader = torch.utils.data.DataLoader(dataset=test_dataset,\n                                                batch_size=MINIBATCH_SIZE,\n                                                shuffle=False)\n\nclasses = {0:\"sincere\", 1:\"insincere\"} # These are the classes we have in the dataset (labels)\n\n# Save loaders in single dict\ndataloaders_dict = {\"train\": train_loader, \"val\": val_loader, \"test\": test_loader}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5d9d30a69a5fd7eb018906096047f51212654a8"},"cell_type":"markdown","source":"Try to run pytorch on gpu if available, otherwise we'll use the cpu!"},{"metadata":{"trusted":true,"_uuid":"4068a13cb190493473596f1cb121200dce99f66c"},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\nprint(\"Running on device: {}\".format(device))\nif torch.cuda.is_available():\n    print(\"torch.cuda.current_device(): {}\".format(torch.cuda.current_device()))\n    print(\"torch.cuda.device(0): {}\".format(torch.cuda.device(0)))\n    print(\"torch.cuda.device_count(): {}\".format(torch.cuda.device_count()))\n    print(\"torch.cuda.get_device_name(0): {}\".format(torch.cuda.get_device_name(0)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1b1b30b7bf1b561c48a2e91ef2333965b252d1a"},"cell_type":"markdown","source":"Create our neural network model and initiate it"},{"metadata":{"trusted":true,"_uuid":"8854738b8f79c809e472f6eb3911455415ebe74f"},"cell_type":"code","source":"class LSTM01(nn.Module):\n    def __init__(self, batch_size, output_size, hidden_size, vocab_size, embedding_length, device):\n        super(LSTM01, self).__init__()\n        \"\"\"\n        Arguments\n        ---------\n        batch_size : Size of the batch which is same as the batch_size of the data returned by the TorchText BucketIterator\n        output_size : 2 = (sincere, insincere)\n        hidden_sie : Size of the hidden_state of the LSTM\n        vocab_size : Size of the vocabulary containing unique words\n        embedding_length : Embeddding dimension of our word embeddings (glove dimension if using glove)\n        weights : Pre-trained GloVe word_embeddings which we will use to create our word_embedding look-up table \n\n        \"\"\"\n        self.batch_size = batch_size\n        self.output_size = output_size\n        self.hidden_size = hidden_size\n        self.vocab_size = vocab_size\n        self.embedding_length = embedding_length\n\n        self.word_embeddings = nn.Embedding(vocab_size, embedding_length)# Initializing the look-up table.\n        # self.word_embeddings.weight = nn.Parameter(weights, requires_grad=False) # Assigning the look-up table to the pre-trained GloVe word embedding.\n        self.lstm = nn.LSTM(embedding_length, hidden_size, batch_first=False)\n        self.label = nn.Linear(hidden_size, output_size)\n        self.sigmoid = nn.Sigmoid() # Should use sigmoid for binary classification\n\n    def forward(self, input_sentence, batch_size=None):\n        if batch_size is not None and batch_size is not self.batch_size:\n            print(\"Got batch size {} in LSTM\".format(batch_size))\n        \"\"\" \n        Parameters\n        ----------\n        input_sentence: input_sentence of shape = (batch_size, num_sequences)\n        batch_size : default = None. Used only for prediction on a single sentence after training (batch_size = 1)\n\n        Returns\n        -------\n        Output of the linear layer containing logits for positive & negative class which receives its input as the final_hidden_state of the LSTM\n        final_output.shape = (batch_size, output_size)\n        \n        \"\"\"\n        ''' Here we will map all the indexes present in the input sequence to the corresponding word vector using our pre-trained word_embedddins.'''\n        input = self.word_embeddings(input_sentence) # embedded input of shape = (batch_size, num_sequences,  embedding_length)\n        input = input.permute(1, 0, 2) # input.size() = (num_sequences, batch_size, embedding_length)\n        if batch_size is None:\n            h_0 = Variable(torch.zeros(1, self.batch_size, self.hidden_size).to(device)) # Initial hidden state of the LSTM\n            c_0 = Variable(torch.zeros(1, self.batch_size, self.hidden_size).to(device)) # Initial cell state of the LSTM\n        else:\n            h_0 = Variable(torch.zeros(1, batch_size, self.hidden_size).to(device))\n            c_0 = Variable(torch.zeros(1, batch_size, self.hidden_size).to(device))\n        output, (final_hidden_state, final_cell_state) = self.lstm(input, (h_0, c_0))\n        output = self.label(final_hidden_state[-1]) # final_hidden_state.size() = (1, batch_size, hidden_size) & final_output.size() = (batch_size, output_size)\n        final_output = self.sigmoid(output)\n        return final_output\n    \n    def predict(self, inp, batch_size):\n        return self.forward(inp, batch_size=batch_size)\n\n    \n# Create model\nmodel = LSTM01(batch_size=MINIBATCH_SIZE, output_size=2,\n                           hidden_size=LSTM_HIDDEN_SIZE, vocab_size=len(vocab_to_int)+1,\n                            embedding_length=LSTM_EMBEDDING_SIZE, device=device)\n\n# Print the model we just instantiated\nprint(model)\n\n# Send model to deivce (GPU hopefully!)\nmodel = model.to(device)\n\n# Binary classification -> we use binary cross entropy loss func\nloss_function = nn.BCELoss()\n\n# We use the SGD optimizer algorithm\n# https://pytorch.org/docs/stable/optim.html\noptimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=SGD_MOMENTUM)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f994f76a7a9003ef730530c3ceac89d56fbf4251"},"cell_type":"markdown","source":"Create train (and validation) as well as a test function so that we can train/validate and test our model"},{"metadata":{"trusted":true,"_uuid":"5a998d27e4b72cc416f8f9be321540dc6fe243b1"},"cell_type":"code","source":"def train(model, train_loader, criterion, optimizer, minibatch_size, epoch):\n    ''' Standard pytorch procedure for training a model '''\n    print(\"Training on device: {}\".format(device))\n    \n    # declare variables\n    t0 = time.time()\n    total_epoch_loss = 0.0\n    total_epoch_acc = 0\n    steps = 0\n    model.train() # Set model to training mode\n    \n    for idx, (inputs, labels) in enumerate(train_loader):\n        inputs = inputs.to(device)\n        labels = labels.to(device)\n            \n        # zero the parameter gradients\n        optimizer.zero_grad()\n        \n        with torch.set_grad_enabled(True):\n            # forward\n            outputs = model.predict(inputs, inputs.size(0))\n            loss = criterion(outputs, labels)\n            _, preds = torch.max(outputs, 1)\n            \n            # calc accuracy\n            num_corrects = torch.sum(preds == torch.argmax(labels,1))\n            acc = 100.0 * num_corrects/inputs.size(0)\n            \n            # backward\n            loss.backward()\n            optimizer.step()\n        \n        steps += 1\n        \n        if steps % 1000 == 0:\n            time_elapsed = time.time() - t0\n            print (f\"Epoch: {epoch+1} [{time_elapsed//60:.0f}m, {time_elapsed%60:.0f}s], Idx: {idx+1}, Training Loss: {loss.item():.4f}, Training Accuracy: {acc.item(): .2f}%\")\n        \n        total_epoch_loss += loss.item()\n        total_epoch_acc += acc.item()\n             \n    return total_epoch_loss/len(train_loader), total_epoch_acc/len(train_loader)\n\ndef validate(model, val_loader, criterion, minibatch_size):\n    ''' In the validation procedure we keep track of the weights that gives the best val acc\n    In the end, we load the weights that gave us the best accuracy '''\n    print(\"Validating on device: {}\".format(device))\n\n    # declare variables\n    t0 = time.time()\n    total_epoch_loss = 0.0\n    total_epoch_acc = 0\n    model.eval() # Set model to evaluate mode\n\n    with torch.no_grad():\n        for idx, (inputs, labels) in enumerate(val_loader):\n            inputs = inputs.to(device)\n            labels = labels.to(device)\n\n            outputs = model.predict(inputs, inputs.size(0))\n            loss = criterion(outputs, labels) \n            _, preds = torch.max(outputs, 1)\n\n            # calc accuracy\n            num_corrects = torch.sum(preds == torch.argmax(labels,1))\n            acc = 100.0 * num_corrects/inputs.size(0)   \n\n            total_epoch_loss += loss.item()\n            total_epoch_acc += acc.item()\n        \n    return total_epoch_loss/len(val_loader), total_epoch_acc/len(val_loader)\n\ndef test(model, test_loader, minibatch_size, device):\n    ''' Test the model on test set and save results '''\n    print(\"Testing on device: {}\".format(device))\n    t0 = time.time()\n    results = {'qid':[], 'prediction':[]}\n    with torch.no_grad():\n        for idx, data in enumerate(test_loader):\n            inputs = data[0]            \n            inputs = inputs.to(device)\n            # get prediction\n            outputs = model.predict(inputs, inputs.size(0))\n            _, preds = torch.max(outputs, 1)\n            # save each prediction\n            for pred in preds:\n                results['prediction'].append(pred.item())               \n    time_elapsed = time.time() - t0\n    print('Testing complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n    return results","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8faaf6d14d22cef567d7310447eea823801a118c"},"cell_type":"markdown","source":"Train and test the model! (FINALLY :))"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"28d56e42a9fbccc5be1e80bc6ef7caf06e368f98"},"cell_type":"code","source":"best_wts = copy.deepcopy(model.state_dict())\nbest_acc = 0\nbest_epoch = 0\n\nfor epoch in range(EPOCHS):\n    # train\n    train_loss, train_acc = train(model, \n                                  dataloaders_dict['train'], \n                                  loss_function, optimizer, \n                                  MINIBATCH_SIZE, \n                                  epoch)\n    # validate\n    val_loss, val_acc = validate(model, \n                                 dataloaders_dict['val'],\n                                 loss_function,\n                                 MINIBATCH_SIZE)\n    # save weights that give highest val acc\n    if val_acc > best_acc:\n        best_wts = copy.deepcopy(model.state_dict())\n        best_acc = val_acc\n        best_epoch = epoch\n        \n    print(f'Epoch: {epoch+1:02}, Train Loss: {train_loss:.3f}, Train Acc: {train_acc:.2f}%, Val. Loss: {val_loss:3f}, Val. Acc: {val_acc:.2f}%')\n    print(\"Best acc: {:.4f}%, at epoch: {}\".format(best_acc, best_epoch))\n    \n# load best weights\nmodel.load_state_dict(best_wts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4282c5a56508b21f714b843dd21eb6a87872ac7e"},"cell_type":"markdown","source":"Test the model on the test set and save results to disk."},{"metadata":{"trusted":false,"_uuid":"93a8776023dfb0450a2fdd6e44f5e3d0f9acecda"},"cell_type":"code","source":"# Run test set and get results\nresults = test(model, dataloaders_dict['test'], MINIBATCH_SIZE, device)\nresults['qid'] = qid_test\n\n# Print some stats\nprint(\"Number of qids: {}, number of predictions: {}\".format(len(results['qid']), format(len(results['prediction']))))\n\n# Save results\ndf = pd.DataFrame(data=results)\ndf.to_csv('submission.csv', index=False)\nprint(\"Saved csv to disk!\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5823484b803c3286c861b2c22d7071450f7b7ef1"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}