{"cells":[{"metadata":{"_uuid":"c8be6feb8ca696102e4507b4f743b198f4827e0f"},"cell_type":"markdown","source":"## Easy Beginner Quora Tutorial\n\nIn this notebook will explain the main components of the Quora kernels you will read.  After reading this notebook you will much better understand some of the key components of the more complicated kernels.  Check links at bottom for helpful links."},{"metadata":{"trusted":true,"_uuid":"bd929f99b54397d697edb0ec197eb6e29c8612a3"},"cell_type":"code","source":"import numpy as np\nfrom numpy import zeros\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import Embedding","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a190705f2a00bbeae86d15ce1acb367ab7f92b8"},"cell_type":"markdown","source":"### Define a simple vocab\n\nRather than using the training data containing many thousands of words, we will keep it simple by defining a small vocab and class labels\n\nHow we can learn a word embedding while fitting a neural\nnetwork on a text classification problem. We define a small problem where we have 10\ntext documents, each with a comment about a piece of work a student submitted. Each text\ndocument is classified as positive 1 or negative 0. This is a simple sentiment analysis problem.\nFirst, we will define the documents and their class labels."},{"metadata":{"trusted":true,"_uuid":"553c178b48ac66d42538ddd01a669ea6422df1a0"},"cell_type":"code","source":"# define documents\ndocs = ['Well done!',\n'Good work',\n'Great effort',\n'nice work',\n'Excellent!',\n'Weak',\n'Poor effort!',\n'not good',\n'poor work',\n'Could have done better.']\n\n# define class labels\nlabels = np.array([1,1,1,1,1,0,0,0,0,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52d26b59b8de5f1fd52db0b0b5b25f70eb2d89a2"},"cell_type":"markdown","source":"The first step is to define the examples, encode them as integers,\nthen pad the sequences to be the same length. In this case, we need to be able to map words to\nintegers as well as integers to words. Keras provides a Tokenizer class that can be fit on the\ntraining data, can convert text to sequences consistently by calling the texts to sequences()\nmethod on the Tokenizer class, and provides access to the dictionary mapping of words to\nintegers in a word index attribute"},{"metadata":{"trusted":true,"_uuid":"5f8eca3566abc6bac7b86f30e914e2d8498d9ade"},"cell_type":"code","source":"# prepare tokenizer\nt = Tokenizer()\nt.fit_on_texts(docs)\nvocab_size = len(t.word_index) + 1\n\n# integer encode the documents\nencoded_docs = t.texts_to_sequences(docs)\nprint(encoded_docs)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"71d93515e6d2dee25554d393a3ecda80aed71c53"},"cell_type":"markdown","source":"Running the example first prints the integer encoded documents. You can see that the word `work` appears in several of the sentences and is encoded as `1` in all the encoded docs.\n\nThe sequences have different lengths and Keras prefers inputs to be vectorized and all inputs\nto have the same length. We will pad all input sequences to have the length of 4.\n"},{"metadata":{"trusted":true,"_uuid":"cb1303fcbf26043da77654eaec3e6b7513b2e128"},"cell_type":"code","source":"# pad documents to a max length of 4 words\nmax_length = 4\npadded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\nprint(padded_docs)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e597d90bced907967e4923653f660bf1d86b02a"},"cell_type":"markdown","source":"Then the padded versions of each document are printed, making them all uniform length"},{"metadata":{"_uuid":"643eda06864b54a5d556d4e1ff31e4b438c5ed79"},"cell_type":"markdown","source":"### Using Pre-Trained GloVe Embedding\n\nThe Keras Embedding layer can also use a word embedding learned elsewhere. It is common\nin the field of Natural Language Processing to learn, save, and make freely available word\nembeddings. For example, the researchers behind GloVe method provide a suite of pre-trained\nword embeddings on their website released under a public domain license\n\nIn the Quora challenge glove.840B.300d.txt is commonly used. It has a vector size of 300 dimensions - the largest available."},{"metadata":{"_uuid":"9aa9c481cf69e4c675f245bc15f196923363a48f"},"cell_type":"markdown","source":"For example, below are the first line of the embedding ASCII text file showing the embedding for `the`"},{"metadata":{"trusted":true,"_uuid":"8a28a9d82e5204d77734b365efce4ee7b94c8e87"},"cell_type":"code","source":"#look at the test file\nEMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n\ncount = 0\nimport csv\nwith open(EMBEDDING_FILE,) as f:\n    reader = csv.reader(f)\n    for row in reader:\n        if count == 2:\n            print (row)\n            break\n        count +=1\n        ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0f648ba2b40a7b6483b9c87a55e74f8dbe19d0e"},"cell_type":"markdown","source":"You can see the word `the` and its associated weights.  The neural network will use these weights when it sees this word\n\nNext, we need to load the entire GloVe word embedding file into memory as a dictionary of\nword to embedding array."},{"metadata":{"trusted":true,"_uuid":"3a64da5461f3adfc2957cd9c0c9a32cb9bc6840b"},"cell_type":"code","source":"def get_coefs(word,*arr): \n    return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE,errors='ignore'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c1fdd0da7a0896f3a25d0407c5e1956365c1d81"},"cell_type":"code","source":"print('Loaded %s word vectors.' % len(embeddings_index))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"839fdfce4ac1208fe9eb45bec7fadfc09a8928a2"},"cell_type":"markdown","source":"Next, we need to create a matrix of one embedding for each word in the training\ndataset. We can do that by enumerating all unique words in the Tokenizer.word index and\nlocating the embedding weight vector from the loaded GloVe embedding. The result is a matrix\nof weights only for words we will see during training"},{"metadata":{"trusted":true,"_uuid":"208a451f12c01e4bd18dda16574622b3de2141e4"},"cell_type":"code","source":"# create a weight matrix for words in training docs\nembedding_matrix = zeros((vocab_size, 300))\nfor word, i in t.word_index.items():\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1d524d7b592346d820fa4fad73583f1262e082d"},"cell_type":"markdown","source":"### Keras Embedding Layer\n\nKeras offers an Embedding layer that can be used for neural networks on text data. It requires\nthat the input data be integer encoded, so that each word is represented by a unique integer.\nThis data preparation step can be performed using the Tokenizer API also provided with\nKeras.\n\nHere it is used to load a pre-trained word embedding model from above\n\nThe Embedding layer is defined as the first hidden layer of a network. It must specify 3\narguments:\n- `input dim`: This is the size of the vocabulary in the text data. For example, if your data\nis integer encoded to values between 0-10, then the size of the vocabulary would be 11\nwords\n- `output dim`: This is the size of the vector space in which words will be embedded. It\ndefines the size of the output vectors from this layer for each word. For example, it could\nbe 32 or 100 or even larger. \n- `input length`: This is the length of input sequences, as you would define for any input\nlayer of a Keras model. For example, if all of your input documents are comprised of 1000\nwords, this would be 1000"},{"metadata":{"_uuid":"2592041ea1949260ad1eab02906dd75df5e9e764"},"cell_type":"markdown","source":"For example, below we define an Embedding layer with a vocabulary of 200 (e.g. integer\nencoded words from 0 to 199, inclusive), a vector space of 32 dimensions in which words will be\nembedded, and input documents that have 50 words each.\n\ne = Embedding(200, 32, input_length=50)\n"},{"metadata":{"_uuid":"43f064bc83a5c684055955638a2188d8d63d3eaf"},"cell_type":"markdown","source":"### Define a simple Model\n\nThe examples in most kernels use a more complicated neural network known as a LSTM, this is essentially a network that has memory.  Here we use a simple neural network."},{"metadata":{"_uuid":"f57c0170b5a5f0928417161a572f8621b7817dea"},"cell_type":"markdown","source":"In this example the Embedding layer has weights that are pre-learned - we use the weights from GLoVe word embedding here.   The output of the Embedding layer is a 2D vector with\none embedding for each word in the input sequence of words (input document). Here we  connect a Dense layer directly to the Embedding layer, so we first flatten the 2D output matrix to a 1D vector using the Flatten layer."},{"metadata":{"trusted":true,"_uuid":"1c6f39731654a25373575fab6d9a701547fd2ffa"},"cell_type":"code","source":"# define model\nmodel = Sequential()\ne = Embedding(vocab_size, 300, weights=[embedding_matrix], input_length=4, trainable=False)\nmodel.add(e)\nmodel.add(Flatten())\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n\n# summarize the model\nmodel.summary()\n\n# fit the model\nmodel.fit(padded_docs, labels, epochs=50, verbose=0)\n# evaluate the model\nloss, accuracy = model.evaluate(padded_docs, labels, verbose=0)\nprint('Accuracy: %f' % (accuracy*100))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f2de8020198dbbe207a61deaaff093bdd09bb69"},"cell_type":"markdown","source":"Given our very small set of words we get perfect accuracy, but this is only a small example - in reailty you will have many thousands of words in the training and test data"},{"metadata":{"_uuid":"1534b18396291c34cc5b1235763df77fca3ef699"},"cell_type":"markdown","source":"If you liked this kernel then please upvote, thanks :)"},{"metadata":{"_uuid":"6b4eb07cedebfe33890697eb3a7301b9763adbf6"},"cell_type":"markdown","source":"- http://neuralnetworksanddeeplearning.com/chap1.html\n- https://machinelearningmastery.com/introduction-machine-learning-scikit-learn/\n- https://www.kdnuggets.com/2017/07/machine-learning-exercises-python-introductory-tutorial-series.html\n- https://skymind.ai/wiki/lstm\n- https://machinelearningmastery.com/crash-course-deep-learning-natural-language-processing/\n- http://blog.kaggle.com/2017/11/27/introduction-to-neural-networks/?utm_medium=email&utm_source=mailchimp&utm_campaign=newsletter+dec+2017"},{"metadata":{"trusted":false,"_uuid":"0692d3c7cc73f28f9ff5c107c2f83a86c14634d8"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}