{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Description\n\nThe main idea here is to walk through code for loading embeddings for use in RNNs.  The code we're walking through here was used a lot throughout the \"Jigsaw Unintended Bias in Toxicity Classification\" competition.  \n\nHere are some notebooks using similar code:\n\nhttps://www.kaggle.com/gpreda/jigsaw-fast-compact-solution\n\nhttps://www.kaggle.com/christofhenkel/keras-baseline-lstm-attention-5-fold"},{"metadata":{},"cell_type":"markdown","source":"# Part 1:  Create Embedding Dictionary"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing import text, sequence","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# if we had more than 1 embedding file, we could list out the files\nEMBEDDING_FILES = [\n    '../input/jigsaw-custom-word2vec-100d-5iter/custom_word2vec_100d_5iter.txt'\n]\n# EMBEDDING_FILES = [\n#     '../input/fasttext-crawl-300d-2m/crawl-300d-2M.vec',\n#     '../input/glove840b300dtxt/glove.840B.300d.txt'\n# ]\n\n# if we have characters we want to remove before we tokenize, we can list them in a string\n# CHARS_TO_REMOVE = '!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n“”’\\'∞θ÷α•à−β∅³π‘₹´°£€\\×™√²—'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 1: code to read every line in the text file"},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        for line in f:\n            print(line)\n            # end of core code\n            # break after a few rounds\n            rounds += 1\n            if rounds==3:\n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 2: v1 + convert each line to a list"},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        for line in f:\n            # main thing that's changing here is to strip each line and split it by a blank space\n            print(line.strip().split(' '))\n            # end of core code\n            # break after a few rounds\n            rounds += 1\n            if rounds==3:\n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 3: v2 + tuple packing\n\nResources:\n\nhttps://realpython.com/defining-your-own-python-function/"},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# main thing that's changing here that we're going to pack the input into a tuple\ndef get_coefs(*mytup):\n    return mytup\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        for line in f:\n            print(get_coefs(line.strip().split(' ')))\n            # end of core code\n            # break after a few rounds\n            rounds += 1\n            if rounds==3:\n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 4: v3 + tuple unpacking\n\nWe use the * in the fuction call definition, to pack the data passed as a tuple\n\nWe use * in the function call to unpack the data passed to the function (as a tuple)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# now we pack the data passed as a tuple\ndef get_coefs(*mytup):\n    return mytup\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        for line in f:\n            # main thing that's changing here that we're going to unpack the input as a tuple\n            print(get_coefs(*line.strip().split(' ')))\n            # end of core code\n            # break after a few rounds\n            rounds += 1\n            if rounds==3:\n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 5: v4 + take the first item in the tuple and pack the rest, then return the word and the tuple"},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# the main thing that's changing here is that we take the unpacked tuple and we take the first item as 'word' \n# and then we pack the rest as a tuple into 'coefs' and we return a tuple of word,coefs\ndef get_coefs(word, *coefs):\n    return word, coefs\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        for line in f:\n            print(get_coefs(*line.strip().split(' ')))\n            # end of core code\n            # break after a few rounds\n            rounds += 1\n            if rounds==3:\n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 6: v5 + return coefs as numpy array"},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# the main thing that's changing here is we return coefs as a numpy array\ndef get_coefs(word, *coefs):\n    return word, np.asarray(coefs, dtype='float32')\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        for line in f:\n            print(get_coefs(*line.strip().split(' ')))\n            # end of core code\n            # break after a few rounds\n            rounds += 1\n            if rounds==3:\n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code Version 7: v6 + return to a dictionary"},{"metadata":{"trusted":true},"cell_type":"code","source":"# going to use this just to stop the loop after a few rounds\nrounds = 0\n\n# the main thing that's changing here is we return coefs as a numpy array\ndef get_coefs(word, *coefs):\n    return word, np.asarray(coefs, dtype='float32')\n\n# core code\nfor path in EMBEDDING_FILES:\n    with open(path) as f:\n        embedding_index = dict(get_coefs(*line.strip().split(' ')) for line in f)\n        # end of core code\n        # break after a few rounds\n\n# debug\nfor i in embedding_index:\n    print(i)\n    rounds += 1\n    if rounds==3:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 2: Initialize Embedding Matrix\n\nWe already have the embedding dictionary, so we don't need to run this again, we can move forward.\n\nFirst thing we need to do is create a tokenizer, then we can use the lenght of the tokenizer to determine the size of our embedding matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"# load data\nx_train = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv')\nx_train = x_train['comment_text'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show some of the data\nx_train[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create tokenizer object\ntokenizer = text.Tokenizer()\n# if we wanted to remove characters we could run...\n#tokenizer = text.Tokenizer(filters=CHARS_TO_REMOVE)\ntokenizer.fit_on_texts(list(x_train))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_word_index = tokenizer.word_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# debugging\nround = 0\nfor word, i in my_word_index.items():\n    print(word,i)\n    round += 1\n    if round==3:\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(my_word_index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# we add a '+ 1' because this will create rows from 0-409327, i.e. up to but not including 409328, but 0-409327 in total is 409328 rows\n# row 0 will be ignored for the most part and we'll fill 1-409327\nembedding_matrix = np.zeros((len(my_word_index) + 1, 100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_matrix.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    embedding_matrix[409328]\nexcept:\n    print(\"position does not exist\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 3:  Load weights into the proper position of the embedding matrix based on the tokenizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"# we can get the 0th row of the matrix using the following logic\nembedding_matrix[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# we can get the coefficients for any word using the following logic\nembedding_index['the']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# now we can put it all together, for loop through our tokenizer\n# and for each (word,position) we can get the coefficents using the 'word'\n# and we can set the coefficients to the correct position in the matrix using 'position'\nfor word, position in my_word_index.items():\n    try:\n        embedding_matrix[position] = embedding_index[word]\n    except KeyError:\n        pass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# debugging - showing that our matrix is loaded!\nembedding_matrix[25]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 4: Putting it all together!\n\nwe can do all of this in just a few cells and some helper functions!"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_coefs(word, *arr):\n    return word, np.asarray(arr, dtype='float32')\n\n\ndef load_embeddings(path):\n    with open(path) as f:\n        return dict(get_coefs(*line.strip().split(' ')) for line in f)\n\n\ndef build_matrix(word_index, path):\n    embedding_index = load_embeddings(path)\n    embedding_matrix = np.zeros((len(word_index) + 1, 100))\n    for word, i in word_index.items():\n        try:\n            embedding_matrix[i] = embedding_index[word]\n        except KeyError:\n            pass\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = text.Tokenizer()\ntokenizer.fit_on_texts(list(x_train))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# if we have more than 1 EMBEDDING_FILE we can concatenate the results using np.concatenate()\nembedding_matrix = np.concatenate([build_matrix(tokenizer.word_index, f) for f in EMBEDDING_FILES], axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# debugging - showing that our matrix is loaded!\nembedding_matrix[25]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}