{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Bert Toxic Text Detector\nby Marsh [ @vbookshelf ]<br>\n12 April 2020"},{"metadata":{},"cell_type":"markdown","source":"<img src=\"http://toxic.test.woza.work/assets/app_pic1.png\" width=\"350\"></img>\n<p>Fig 1 - Flask App screen shot</p>"},{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\nMy goal for this project was learn how to build and deploy a production-grade toxic text detection model. Users are able to send text to the model, via an API, and get back predictions. I'm not a frontend or backend expert. I tried to learn enough to get things working.\n\nOne way to commercialize a powerful language model like Bert is to deploy it as a microservice. This means that the model lives on a cloud server. Websites can send text to this server. The server responds by sending back a toxicity prediction for each piece of text. The system can also be set up to allow a user to send a file containing rows of text. The model will process this file and the server will return a new file with a prediction for each row.\n\nA tool like this could automatically monitor large volumes of online dialogue. It could help improve the quality of online conversations, protect children from online bullying and protect a companies brand image - but it could also be used for mass surveillance.\n\n\n## Architecture\n\nThe app frontend and backend are on different servers. They communicate using API request/response calls. \n\n### 1- Frontend\n\nThe frontend is a simple website to allow users to test the model. It takes user text as input. It then sends this text to the API server (backend) for prediction. When the prediction is received, the frontend website displays it as a toxic probability score. \n\n### 2- Backend\n\nThe application is a Docker container that's made up of Flask, Nginx and uWSgi. Flask is where the python code is stored. Nginx is the web server and uWSGI allows Flask and Nginx to talk. By using Docker, the model and all it's dependencies (incl Flask and Nginx) are in the same container.\n\nTo improve reliability, there are actually two copies of the app, each running on one of two linux servers. Both servers are connected to the same load balancer. The load balancer's job is to receive the request and then route it to each server in a round-robin sequence. There's no limit to the number of servers that can be connected in this way.\n\nEach server has 2 vCPU's and 4GB of RAM. The Bert model is larger than 500MB. I initially tried a cheap 1GB RAM server but the model would not load. With just 1 CPU the model would not predict. It seems that 2 x CPU's and 3GB of RAM are the minimum requirements to get things working.\n\n\n-x-\n\n\n\nIn this notebook I will do the following:\n\n- List the resources that I used to learn to build this app.\n- Share the model fine tuning code\n\nI trained this model using only 18,000 samples yet it achieved an LB score of 0.88. The training data included the multi-lingual competition validation data (8000 samples). \n\nSide note:<br> \nIn previous experiments I performed TTA (Test Time Augmentation) by making test predictions on test data from three sources - the competition test data, the Google translated data and the Yandex translated data. I then averaged those predictions. This improved the LB score to 0.91. This notebook does not include TTA.\n\n<hr>\n\nThis is the link to the live web app. The frontend and backend code is available on Github. \n\n> Web App<br>\n> http://toxic.test.woza.work/\n>\n> Github<br>\n> https://github.com/vbookshelf/Bert-Toxic-Text-Detector\n\nPlease note that this app will only be live for about a month - or until my free credits run out.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Learning Resources\n\nThese are the resources that I found most helpful. \n\n1. Deeplizard video tutorial<br>\nhttps://deeplizard.com/learn/video/SI1hVGvbbZ4\n\n2. What is a microservice?<br>\nhttps://www.youtube.com/watch?v=SouNISAnXlo\n\n3. Julian Nash video tutorial<br>\nhttps://www.youtube.com/watch?v=dVEjSmKFUVI\n\n4. How to use SSH to connect to a web server<br>\nhttps://www.youtube.com/watch?v=B_lZt9_9UCc\n\n5. Chris McCormick, Bert fine tuning tutorial<br>\nhttps://mccormickml.com/2019/07/22/BERT-fine-tuning/\n\n6. Free Udemy Docker course<br>\nhttps://www.udemy.com/course/docker-essentials/\n\n7. Teclado Udemy course<br>\nhttps://www.udemy.com/course/rest-api-flask-and-python/\n\n8. Practical startup advice<br>\nhttps://www.youtube.com/watch?v=BqPB88pcYTI\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build and Train the Model"},{"metadata":{},"cell_type":"markdown","source":"The training scheme used here has been adapted from a tutorial by Chris McCormick.<br>\n    https://mccormickml.com/2019/07/22/BERT-fine-tuning/"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\n\nimport torch\n\nfrom sklearn.utils import shuffle\nfrom sklearn.metrics import roc_auc_score\n\nimport transformers\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MODEL_TYPE = \"bert-base-multilingual-uncased\"\nCASE_BOOL = True # do_lower_case=CASE_BOOL\n\nMAX_LEN = 256\nNUM_EPOCHS = 2\nBATCH_SIZE = 16\nLRATE = 2e-5\n\nNUM_CORES = os.cpu_count()\n\nNUM_CORES","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define the device"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/jigsaw-multilingual-toxic-comment-classification/' + 'jigsaw-toxic-comment-train.csv'\ndf_train_toxic = pd.read_csv(path, usecols=['comment_text', 'toxic'])\n\npath = '../input/jigsaw-multilingual-toxic-comment-classification/' + 'jigsaw-unintended-bias-train.csv'\ndf_train_bias = pd.read_csv(path, usecols=['comment_text', 'toxic'])\n\npath = '../input/jigsaw-multilingual-toxic-comment-classification/' + 'validation.csv'\ndf_val = pd.read_csv(path)\n\npath = '../input/jigsaw-multilingual-toxic-comment-classification/' + 'test.csv'\ndf_test = pd.read_csv(path)\n\n\n# Rename the 'content' column\ndf_test = df_test.rename(columns={'content': 'comment_text'})\n\nprint(df_train_toxic.shape)\nprint(df_train_bias.shape)\nprint(df_val.shape)\nprint(df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create df_train"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filter out only the toxic comments from df_train_bias\ndf_1 = df_train_bias[df_train_bias['toxic'] >= 0.5]\n\ndf_1.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Combine df_1 and df_train_toxic\ndf_train = pd.concat([df_1, df_train_toxic], axis=0).reset_index(drop=True)\n\ndf_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Take a sample of 10000 rows\ndf_train = df_train.sample(n=10000, random_state=1024)\n\n# Reset the indices\ndf_train = df_train.reset_index(drop=True)\ndf_val = df_val.reset_index(drop=True)\ndf_test = df_test.reset_index(drop=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Include the val set in df_train.\n# We will be using the val set for training.\n\ndf_train = pd.concat([df_train, df_val], axis=0)\n\ndf_train = shuffle(df_train)\n\ndf_train = df_train.reset_index(drop=True)\n\ndf_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the data for Bert"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef preprocess_for_bert(sentences, MAX_LEN):\n    \n    \"\"\"\n    Preprocesses sentences to suit BERT.\n    Input:\n    sentences: numpy array\n    \n    Output:\n    Tokenized sentences, padded and truncated.\n    \n    \"\"\"\n\n    \n    # Tokenize all of the sentences and map the tokens to thier word IDs.\n    input_ids = []\n\n    # For every sentence...\n    for sent in sentences:\n        # `encode` will:\n        #   (1) Tokenize the sentence.\n        #   (2) Prepend the `[CLS]` token to the start.\n        #   (3) Append the `[SEP]` token to the end.\n        #   (4) Map tokens to their IDs.\n        encoded_sent = tokenizer.encode(\n                            sent,                      # Sentence to encode.\n                            add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n                            #max_length = 128,          # Truncate all sentences.\n                            #return_tensors = 'pt',     # Return pytorch tensors.\n                       )\n\n        # Add the encoded sentence to the list.\n        input_ids.append(encoded_sent)\n        \n    \n    # Pad the token matrix\n    \n    # **** Issue: If the length is greater than max_len then \n    # this part will cut off the [SEP] token (102), which is\n    # at the end of the long sentence.\n\n    from keras.preprocessing.sequence import pad_sequences\n\n    # Pad our input tokens with value 0.\n    # \"post\" indicates that we want to pad and truncate at the end of the sequence,\n    # as opposed to the beginning.\n    padded_input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", \n                              value=0, truncating=\"post\", padding=\"post\")\n    \n    \n    # *** This fixes the issue above.\n    # Check if the SEP token was cut off and if so put it back in.\n    # Check if the last index is 102. 102 is the SEP token.\n    # Correct the last token if needed.\n    for sent in padded_input_ids: # go row by row through the numpy 2D array.\n        length = len(sent)\n        \n        if (sent[length-1] != 0) and (sent[length-1] != 102): # 102 is the SEP token\n            sent[length-1] = 102 # set the last value to be the SEP token i.e. 102\n    \n    \n    # Create attention masks\n    attention_masks = []\n\n    # For each sentence...\n    for sent in padded_input_ids:\n\n        # Create the attention mask.\n        #   - If a token ID is 0, then it's padding, set the mask to 0.\n        #   - If a token ID is > 0, then it's a real token, set the mask to 1.\n        att_mask = [int(token_id > 0) for token_id in sent]\n\n        # Store the attention mask for this sentence.\n        attention_masks.append(att_mask)\n        \n        \n\n    return padded_input_ids, attention_masks","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Instantiate the Bert Tokenizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"from transformers import BertTokenizer\n\n# Load the BERT tokenizer.\ntokenizer = BertTokenizer.from_pretrained(MODEL_TYPE, do_lower_case=CASE_BOOL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tokenize the text"},{"metadata":{"trusted":true},"cell_type":"code","source":"sentences = df_test['comment_text'].values\nX_test, X_test_att_masks = preprocess_for_bert(sentences, MAX_LEN)\n\n\nsentences = df_train['comment_text'].values\nX_train, X_train_att_masks = preprocess_for_bert(sentences, MAX_LEN)\ny_train = df_train['toxic'].values\n\nsentences = df_val['comment_text'].values\nX_val, X_val_att_masks = preprocess_for_bert(sentences, MAX_LEN)\ny_val = df_val['toxic'].values\n\n\n\nprint(X_test.shape)\nprint(len(X_test_att_masks))\n\nprint('---')\n\nprint(X_train.shape)\nprint(len(X_train_att_masks))\nprint(y_train.shape)\n\nprint('---')\n\nprint(X_val.shape)\nprint(len(X_val_att_masks))\nprint(y_val.shape)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Convert to PyTorch datatypes"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\n\ny_train = y_train.astype('long')\ny_val = y_val.astype('long')\n\n# Convert inputs and labels into torch tensors\n\ntrain_inputs = torch.tensor(X_train)\nvalidation_inputs = torch.tensor(X_val)\n\ntrain_labels = torch.tensor(y_train)\nvalidation_labels = torch.tensor(y_val)\n\ntrain_masks = torch.tensor(X_train_att_masks)\nvalidation_masks = torch.tensor(X_val_att_masks)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n\n\n# Create the DataLoader for our training set.\ntrain_data = TensorDataset(train_inputs, train_masks, train_labels)\ntrain_sampler = RandomSampler(train_data)\ntrain_dataloader = \\\nDataLoader(train_data, sampler=train_sampler, batch_size=BATCH_SIZE, num_workers=NUM_CORES)\n\n# Create the DataLoader for our validation set.\nvalidation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\nvalidation_sampler = SequentialSampler(validation_data)\nvalidation_dataloader = \\\nDataLoader(validation_data, sampler=validation_sampler, batch_size=BATCH_SIZE, num_workers=NUM_CORES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"from transformers import BertForSequenceClassification, AdamW, BertConfig\n\n# Load BertForSequenceClassification, the pretrained BERT model with a single \n# linear classification layer on top. \nmodel = BertForSequenceClassification.from_pretrained(\n    MODEL_TYPE, \n    num_labels = 2, # The number of output labels--2 for binary classification.\n                    # You can increase this for multi-class tasks.   \n    output_attentions = False, # Whether the model returns attentions weights.\n    output_hidden_states = False, # Whether the model returns all hidden-states.\n)\n\n# Tell pytorch to run this model on the GPU.\nmodel.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Note: AdamW is a class from the huggingface library (as opposed to pytorch) \n# I believe the 'W' stands for 'Weight Decay fix\"\noptimizer = AdamW(model.parameters(),\n                  lr = LRATE, \n                  eps = 1e-8 \n                )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from transformers import get_linear_schedule_with_warmup\n\n# Total number of training steps is number of batches * number of epochs.\ntotal_steps = len(train_dataloader) * NUM_EPOCHS\n\n# Create the learning rate scheduler.\nscheduler = get_linear_schedule_with_warmup(optimizer, \n                                            num_warmup_steps = 0,\n                                            num_training_steps = total_steps)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\n# Function to calculate the accuracy of our predictions vs labels\ndef flat_accuracy(preds, labels):\n    pred_flat = np.argmax(preds, axis=1).flatten()\n    labels_flat = labels.flatten()\n    return np.sum(pred_flat == labels_flat) / len(labels_flat)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\nimport datetime\n\ndef format_time(elapsed):\n    '''\n    Takes a time in seconds and returns a string hh:mm:ss\n    '''\n    # Round to the nearest second.\n    elapsed_rounded = int(round((elapsed)))\n    \n    # Format as hh:mm:ss\n    return str(datetime.timedelta(seconds=elapsed_rounded))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\n\n# This training code is based on the `run_glue.py` script here:\n# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128\n\n\n# Set the seed value all over the place to make this reproducible.\nseed_val = 42\n\nrandom.seed(seed_val)\nnp.random.seed(seed_val)\ntorch.manual_seed(seed_val)\ntorch.cuda.manual_seed_all(seed_val)\n\n# Store the average loss after each epoch so we can plot them.\nloss_values = []\n\n# For each epoch...\nfor epoch_i in range(0, NUM_EPOCHS):\n\n\n    print(\"\")\n    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, NUM_EPOCHS))\n    print('Training...')\n\n    # Measure how long the training epoch takes.\n    t0 = time.time()\n\n    # Reset the total loss for this epoch.\n    total_loss = 0\n\n    # Put the model into training mode. \n    model.train()\n\n    # For each batch of training data...\n    for step, batch in enumerate(train_dataloader):\n\n        # Progress update every 100 batches.\n        if step % 100 == 0 and not step == 0:\n            # Calculate elapsed time in minutes.\n            elapsed = format_time(time.time() - t0)\n            \n            # Report progress.\n            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))\n\n        # Unpack this training batch from our dataloader. \n        b_input_ids = batch[0].to(device)\n        b_input_mask = batch[1].to(device)\n        b_labels = batch[2].to(device)\n\n        # Clear any previously calculated gradients\n        model.zero_grad()        \n\n        outputs = model(b_input_ids, \n                    token_type_ids=None, \n                    attention_mask=b_input_mask, \n                    labels=b_labels)\n        \n        # The call to `model` always returns a tuple, so we need to pull the \n        # loss value out of the tuple.\n        loss = outputs[0]\n\n        total_loss += loss.item()\n\n        # Perform a backward pass to calculate the gradients.\n        loss.backward()\n\n        # Clip the norm of the gradients to 1.0.\n        # This is to help prevent the \"exploding gradients\" problem.\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n\n        # Update parameters and take a step using the computed gradient.\n        optimizer.step()\n\n        # Update the learning rate.\n        scheduler.step()\n\n    # Calculate the average loss over the training data.\n    avg_train_loss = total_loss / len(train_dataloader)            \n    \n    # Store the loss value for plotting the learning curve.\n    loss_values.append(avg_train_loss)\n\n    print(\"\")\n    print(\"  Average training loss: {0:.2f}\".format(avg_train_loss))\n    print(\"  Training epcoh took: {:}\".format(format_time(time.time() - t0)))\n        \n    \n    \n    # save the model\n    torch.save(model.state_dict(), 'model.pt')\n    \n    \n\nprint(\"\")\nprint(\"Training complete!\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Make a Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert all inputs and labels into torch tensors.\ntest_inputs = torch.tensor(X_test)\ntest_masks = torch.tensor(X_test_att_masks)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create the DataLoader for our validation set.\ntest_data = TensorDataset(test_inputs, test_masks)\ntest_sampler = SequentialSampler(test_data)\ntest_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=BATCH_SIZE, num_workers=NUM_CORES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluate data for one epoch\nfor j, batch in enumerate(test_dataloader):\n\n    # Add batch to GPU\n    batch = tuple(t.to(device) for t in batch)\n\n    # Unpack the inputs from our dataloader\n    b_input_ids, b_input_mask = batch\n\n    # Telling the model not to compute or store gradients, saving memory and\n    # speeding up validation\n    with torch.no_grad():        \n\n    \n        outputs = model(b_input_ids, \n                        token_type_ids=None, \n                        attention_mask=b_input_mask)\n        \n        preds = outputs[0]\n        \n        if j == 0:\n            stacked_preds = preds\n        else:\n            stacked_preds = torch.cat((stacked_preds, preds), dim=0)\n            \n            \n# Apply the sigmoid function to the raw preds\nstacked_preds = torch.sigmoid(stacked_preds)\n\norig_np_preds = stacked_preds.cpu().numpy()\n\n# Select the second column which is class 1 i.e. toxic\npreds = orig_np_preds[:, 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create a submission csv file"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a dataframe\n\ndf_results = pd.DataFrame({'id': df_test.id,\n                             'toxic': preds\n                         }).set_index('id')\n\n\n# Create a submission csv file\n\ndf_results.to_csv('submission.csv',\n                  columns=['toxic'])\n\ndf_results.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Conclusion\n\nThis project demonstrates that a person who understands both machine learning and a bit of backend setup can build and deploy powerful applications relatively cheaply - because he or she can do most the work themselves.\n\nThank you for reading."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"0375721eaa4e470a9849450f3e0af0b4":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"07b1afe6247b4d3393eb2e1fd4fba9f4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_fdafd7ef9a784290a2e0b5127293bbde","IPY_MODEL_64f10ae7a37a45b89424f2b7108bd219"],"layout":"IPY_MODEL_8d9236081f8d405dba26c386b3954586"}},"1803de79deb848aba3b429dcff1b2a63":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"2926ba1fb29b4bfa81de2701427334c6":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"307feda082fa4308a3485bccce80f4fe":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"3a0e0c624d864296908416a6526e94ea":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"414c5f488e9c404981a5dc8f634b431f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"456b30535b2340e5bb74b3ecf4584599":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"5490f027b9a5461baeb6f9f0f8c11c06":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"592e9e226a774feda50ec2660f751048":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"64f10ae7a37a45b89424f2b7108bd219":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_f62157f87b7e4566b33d08a9d1807b4c","placeholder":"​","style":"IPY_MODEL_307feda082fa4308a3485bccce80f4fe","value":" 672M/672M [00:17&lt;00:00, 39.1MB/s]"}},"65d8c40e55464f3794546908ccf7cbda":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_e069cbfa67354f07b9d06acf9b197c22","IPY_MODEL_c104bb537a4341c5803400578a97cd15"],"layout":"IPY_MODEL_0375721eaa4e470a9849450f3e0af0b4"}},"7cf587e4fc7e404eb0e58bdcdf4ed26b":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"8d9236081f8d405dba26c386b3954586":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"8df17ac9789d4eab85c59aa66c4ca34a":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"af82f1a2978745908029e20cf9faae6e":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_414c5f488e9c404981a5dc8f634b431f","placeholder":"​","style":"IPY_MODEL_8df17ac9789d4eab85c59aa66c4ca34a","value":" 569/569 [00:17&lt;00:00, 32.3B/s]"}},"b48d4f5c74fa43b4bd25c87d33ecff6c":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_e071b212fa0b42929f75916ab6caf9fd","IPY_MODEL_af82f1a2978745908029e20cf9faae6e"],"layout":"IPY_MODEL_1803de79deb848aba3b429dcff1b2a63"}},"c104bb537a4341c5803400578a97cd15":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_c41cb767393945f684618f868146a030","placeholder":"​","style":"IPY_MODEL_c3f86e64a06e46968f86f4700cf804bf","value":" 872k/872k [00:00&lt;00:00, 4.75MB/s]"}},"c3f86e64a06e46968f86f4700cf804bf":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"c41cb767393945f684618f868146a030":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"e069cbfa67354f07b9d06acf9b197c22":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"Downloading: 100%","description_tooltip":null,"layout":"IPY_MODEL_592e9e226a774feda50ec2660f751048","max":871891,"min":0,"orientation":"horizontal","style":"IPY_MODEL_5490f027b9a5461baeb6f9f0f8c11c06","value":871891}},"e071b212fa0b42929f75916ab6caf9fd":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"Downloading: 100%","description_tooltip":null,"layout":"IPY_MODEL_2926ba1fb29b4bfa81de2701427334c6","max":569,"min":0,"orientation":"horizontal","style":"IPY_MODEL_3a0e0c624d864296908416a6526e94ea","value":569}},"f62157f87b7e4566b33d08a9d1807b4c":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"fdafd7ef9a784290a2e0b5127293bbde":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"Downloading: 100%","description_tooltip":null,"layout":"IPY_MODEL_7cf587e4fc7e404eb0e58bdcdf4ed26b","max":672271273,"min":0,"orientation":"horizontal","style":"IPY_MODEL_456b30535b2340e5bb74b3ecf4584599","value":672271273}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":4}