{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "1c29b03e-310f-c64d-ed42-f9895c254c41" }, "source": [ "This script intends to be a starter script for Keras using pre-trained word embeddings.\n", "\n", "**Word embedding:**\n", "\n", "[Word embedding][1] is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. They are also called as word vectors.\n", "\n", "Two commonly used word embeddings are:\n", "\n", "1. [Google word2vec][2]\n", "2. [Stanford Glove][3]\n", "\n", "In this notebook, we will use the GloVe word vector which is downloaded from [this link][4] \n", "\n", "Let us first import the necessary packages.\n", "\n", "\n", " [1]: https://en.wikipedia.org/wiki/Word_embedding\n", " [2]: https://code.google.com/archive/p/word2vec/\n", " [3]: https://nlp.stanford.edu/projects/glove/\n", " [4]: http://nlp.stanford.edu/data/glove.6B.zip" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b481b4ef-630f-a5e1-6465-863bb15b8203" }, "outputs": [], "source": [ "import os\n", "import csv\n", "import codecs\n", "import numpy as np\n", "import pandas as pd\n", "np.random.seed(1337)\n", "\n", "from keras.preprocessing.text import Tokenizer\n", "from keras.preprocessing.sequence import pad_sequences\n", "from keras.utils.np_utils import to_categorical\n", "from keras.layers import Dense, Input, Flatten, merge, LSTM, Lambda, Dropout\n", "from keras.layers import Conv1D, MaxPooling1D, Embedding\n", "from keras.models import Model\n", "from keras.layers.wrappers import TimeDistributed, Bidirectional\n", "from keras.layers.normalization import BatchNormalization\n", "from keras import backend as K\n", "import sys" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "dc21f166-ef1f-9afe-25a2-e9d60806130a" }, "source": [ "Let us specify the constants that are needed for the model.\n", "\n", " 1. MAX_SEQUENCE_LENGTH : number of words from the question to be used\n", " 2. MAX_NB_WORDS : maximum size of the vocabulary\n", " 3. EMBEDDING_DIM : dimension of the word embeddings" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "64678483-0136-5594-4cfc-2441bfd10fd4" }, "outputs": [], "source": [ "BASE_DIR = '../input/'\n", "GLOVE_DIR = BASE_DIR + '/WordEmbeddings/Glove/'\n", "TRAIN_DATA_FILE = BASE_DIR + 'train.csv'\n", "TEST_DATA_FILE = BASE_DIR + 'test.csv'\n", "MAX_SEQUENCE_LENGTH = 30\n", "MAX_NB_WORDS = 200000\n", "EMBEDDING_DIM = 300\n", "VALIDATION_SPLIT = 0.01" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ccb2f15f-ba61-4bba-40b3-5fedf9266bc1" }, "source": [ "As the first step, let us read the word vectors text file into a dictionary where the word is the key and the 300 dimensional vector is its corresponding value.\n", "\n", "Note : This will throw an error here since the word vectors are not here in Kaggle environment." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5dbc03e4-19ca-eb82-194e-1f64613c55c1" }, "outputs": [], "source": [ "print('Indexing word vectors.')\n", "embeddings_index = {}\n", "f = codecs.open(os.path.join(GLOVE_DIR, 'glove.6B.300d.txt'), encoding='utf-8')\n", "for line in f:\n", " values = line.split(' ')\n", " word = values[0]\n", " coefs = np.asarray(values[1:], dtype='float32')\n", " embeddings_index[word] = coefs\n", "f.close()\n", "print('Found %s word vectors.' % len(embeddings_index))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "908b8685-5c18-6e0f-ff58-00c3fc3763fe" }, "source": [ "Now read the train and test questions into list of questions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "177264f6-4458-35ec-7880-ea96d00dc59c" }, "outputs": [], "source": [ "print('Processing text dataset')\n", "texts_1 = [] \n", "texts_2 = []\n", "labels = [] # list of label ids\n", "with codecs.open(TRAIN_DATA_FILE, encoding='utf-8') as f:\n", " reader = csv.reader(f, delimiter=',')\n", " header = next(reader)\n", " for values in reader:\n", " texts_1.append(values[3])\n", " texts_2.append(values[4])\n", " labels.append(int(values[5]))\n", "print('Found %s texts.' % len(texts_1))\n", "\n", "test_texts_1 = []\n", "test_texts_2 = []\n", "test_labels = [] # list of label ids\n", "with codecs.open(TEST_DATA_FILE, encoding='utf-8') as f:\n", " reader = csv.reader(f, delimiter=',')\n", " header = next(reader)\n", " for values in reader:\n", " test_texts_1.append(values[1])\n", " test_texts_2.append(values[2])\n", " test_labels.append(values[0])\n", "print('Found %s texts.' % len(test_texts_1))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "eee8b38b-63d6-0aec-c766-1f4cf8a7a28d" }, "source": [ "Using keras tokenizer to tokenize the text and then do padding the sentences to 30 words" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "cdb3f7de-05a0-c068-e65c-8e25f527e292" }, "outputs": [], "source": [ "tokenizer = Tokenizer(nb_words=MAX_NB_WORDS)\n", "tokenizer.fit_on_texts(texts_1 + texts_2 + test_texts_1 + test_texts_2)\n", "sequences_1 = tokenizer.texts_to_sequences(texts_1)\n", "sequences_2 = tokenizer.texts_to_sequences(texts_2)\n", "word_index = tokenizer.word_index\n", "print('Found %s unique tokens.' % len(word_index))\n", "\n", "test_sequences_1 = tokenizer.texts_to_sequences(test_texts_1)\n", "test_sequences_2 = tokenizer.texts_to_sequences(test_texts_2)\n", "\n", "data_1 = pad_sequences(sequences_1, maxlen=MAX_SEQUENCE_LENGTH)\n", "data_2 = pad_sequences(sequences_2, maxlen=MAX_SEQUENCE_LENGTH)\n", "labels = np.array(labels)\n", "print('Shape of data tensor:', data_1.shape)\n", "print('Shape of label tensor:', labels.shape)\n", "\n", "test_data_1 = pad_sequences(test_sequences_1, maxlen=MAX_SEQUENCE_LENGTH)\n", "test_data_2 = pad_sequences(test_sequences_2, maxlen=MAX_SEQUENCE_LENGTH)\n", "test_labels = np.array(test_labels)\n", "del test_sequences_1\n", "del test_sequences_2\n", "del sequences_1\n", "del sequences_2\n", "import gc\n", "gc.collect()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1243e57b-8a99-0ff6-ff1c-ce3f87e38b12" }, "source": [ "Now let us create the embedding matrix where each row corresponds to a word." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ca875eb8-ed26-4fbc-9f53-e38ea5f98af7" }, "outputs": [], "source": [ "print('Preparing embedding matrix.')\n", "# prepare embedding matrix\n", "nb_words = min(MAX_NB_WORDS, len(word_index))\n", "\n", "embedding_matrix = np.zeros((nb_words, EMBEDDING_DIM))\n", "for word, i in word_index.items():\n", " if i >= nb_words:\n", " continue\n", " embedding_vector = embeddings_index.get(word)\n", " if embedding_vector is not None:\n", " # words not found in embedding index will be all-zeros.\n", " embedding_matrix[i] = embedding_vector\n", "print('Null word embeddings: %d' % np.sum(np.sum(embedding_matrix, axis=1) == 0))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "56d1f6fa-b9bc-4a70-66ea-6a8acad13c46" }, "source": [ "Now its time to build the model. Let us specify the model architecture. First layer is the embedding layer." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "56eeb1b2-998e-b696-d031-9f71639c9205" }, "outputs": [], "source": [ "embedding_layer = Embedding(nb_words,\n", " EMBEDDING_DIM,\n", " weights=[embedding_matrix],\n", " input_length=MAX_SEQUENCE_LENGTH,\n", " trainable=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f163b655-f4a3-7ce6-42e8-e634ec1f39ee" }, "source": [ "In embedding layer, 'trainable' is set to False so as to not train the word embeddings during the back propogation.\n", "\n", "The neural net architecture is as follows:\n", "\n", "1. Word embeddings of each question is passed to a 1-dimensional convolution layer followed by max pooling.\n", "2. It is followed by one dense layer for each of the two questions\n", "3. The outputs from both the dense layers are merged together\n", "4. It is followed by a dense layer\n", "5. Final layer is a sigmoid layer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "278ea96d-a2f3-d93b-00e2-3f57433aa409" }, "outputs": [], "source": [ "# Model Architecture #\n", "sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n", "embedded_sequences_1 = embedding_layer(sequence_1_input)\n", "x1 = Conv1D(128, 3, activation='relu')(embedded_sequences_1)\n", "x1 = MaxPooling1D(10)(x1)\n", "x1 = Flatten()(x1)\n", "x1 = Dense(64, activation='relu')(x1)\n", "x1 = Dropout(0.2)(x1)\n", "\n", "sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n", "embedded_sequences_2 = embedding_layer(sequence_2_input)\n", "y1 = Conv1D(128, 3, activation='relu')(embedded_sequences_2)\n", "y1 = MaxPooling1D(10)(y1)\n", "y1 = Flatten()(y1)\n", "y1 = Dense(64, activation='relu')(y1)\n", "y1 = Dropout(0.2)(y1)\n", "\n", "merged = merge([x1,y1], mode='concat')\n", "merged = BatchNormalization()(merged)\n", "merged = Dense(64, activation='relu')(merged)\n", "merged = Dropout(0.2)(merged)\n", "merged = BatchNormalization()(merged)\n", "preds = Dense(1, activation='sigmoid')(merged)\n", "model = Model(input=[sequence_1_input,sequence_2_input], output=preds)\n", "model.compile(loss='binary_crossentropy',\n", " optimizer='adam',\n", " metrics=['acc'])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2e1c2121-a1ce-f9a6-bbf8-c9290bc864d4" }, "source": [ "**Model training and predictions :**\n", "\n", "Uncomment the below cell and run it in local as it is exceeding the time limits here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "19e71349-400c-f6c8-c2dd-becb6c8cb722" }, "outputs": [], "source": [ "pass\n", "#model.fit([data_1,data_2], labels, validation_split=VALIDATION_SPLIT, nb_epoch=1, batch_size=1024, shuffle=True)\n", "#preds = model.predict([test_data_1, test_data_2])\n", "#print(preds.shape)\n", "\n", "#out_df = pd.DataFrame({\"test_id\":test_labels, \"is_duplicate\":preds.ravel()})\n", "#out_df.to_csv(\"test_predictions.csv\", index=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "42eb8b2d-d666-ae90-c98a-cfa277b7400e" }, "source": [ "This scores about 0.55 when run locally using the word embedding. Got better scores using LSTM and Time Distributed layer.\n", "\n", "Try different architectures and have a happy learning." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e46ad003-246c-2bcc-bbf7-57cbacd66c34" }, "source": [ "Hope this helps to get started with keras and word embeddings in this competition." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "dd9e6c69-cdd6-87c4-2c0c-79cfd0aa466f" }, "source": [ "**References :**\n", "\n", " 1. [On word embeddings - part 1][1] by Sebastian Ruder\n", " 2. [Blog post][2] by fchollet\n", " 3. [Code][3] by Abhishek Thakur\n", " 4. [Code][4] by Bradley Pallen\n", "\n", "\n", " [1]: http://sebastianruder.com/word-embeddings-1/\n", " [2]: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html\n", " [3]: https://github.com/abhishekkrthakur/is_that_a_duplicate_quora_question\n", " [4]: https://github.com/bradleypallen/keras-quora-question-pairs" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }