{"cells":[{"metadata":{"_uuid":"44d1650cab6c91e22b3fad998c4318d6e294de33"},"cell_type":"markdown","source":"# Read the file"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain = pd.read_csv(\"../input/train.csv\", nrows = 1000)\ntest = pd.read_csv(\"../input/test.csv\", nrows = 1000)\nprint(train.head())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60db9913e0b04bc45b57458f8e79e951e1099b41"},"cell_type":"markdown","source":"What is the fraction of insincere questions?"},{"metadata":{"trusted":true,"_uuid":"370ed89713ed6c1298259900185e224554573a42"},"cell_type":"code","source":"import numpy as np\n\nprint(np.mean(train.target))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"764c67b75e2778a00cab6a6ca86a22c3f7c2a4eb"},"cell_type":"markdown","source":"What is the maximum length of a sentence (in terms of number of words)?"},{"metadata":{"trusted":true,"_uuid":"9469e04fb64e6880bc701b72581f1cf59f22e25a"},"cell_type":"code","source":"max_words = np.max([len(i.split(\" \")) for i in train.question_text])\nprint(max_words)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Import the embedding"},{"metadata":{"trusted":true,"_uuid":"3ebf8f7468a9d1de34e561fead7993871cc38428"},"cell_type":"code","source":"import numpy as np\n\n# loading embedding: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\nEMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nwords_embedding = set(embeddings_index.keys())\nprint(len(embeddings_index))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c96e3b5f7ed88b770b3b3c2e10833e40679b07e2"},"cell_type":"code","source":"print(embeddings_index[\"the\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"486051f29f7cfb53ceb44b18a2734f2d0aefc97e"},"cell_type":"code","source":"print(len(embeddings_index[\"the\"]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2bf637e4579ae8f40fa8476205d581e9066396ca"},"cell_type":"markdown","source":"# Checking the coverage"},{"metadata":{"trusted":true,"_uuid":"07ad3b1ba16c4ed27ffa12d5a95a1705008707b9"},"cell_type":"code","source":"list_train_words = \" \".join(train.question_text).split(\" \")\nwords_training = set(list_train_words)\nprint(len(words_training.intersection(words_embedding))/len(words_training))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b054030aff68a4b015a0462c47c3bff1c1df6b49"},"cell_type":"markdown","source":"Printing the missing words (those not present in the embedding):"},{"metadata":{"trusted":true,"_uuid":"b2d48fd5658a670e4aadef99a2f506edb3af0d00"},"cell_type":"code","source":"print(words_training.difference(words_embedding))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfca5697893c1799bbc58770f3c935c8c531e8d8"},"cell_type":"markdown","source":"# Count the frequencies"},{"metadata":{"trusted":true,"_uuid":"a1edba0cc668794c805f02af92f63866c329a512"},"cell_type":"code","source":"from collections import Counter\n\nfrequencies = Counter(words_training.difference(words_embedding))\nprint(frequencies)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1bf3c24cb3496b86fc6f1a3442173c65d79f74f6"},"cell_type":"markdown","source":"Sorting by the number of appearences (following [stackoverflow](https://stackoverflow.com/questions/613183/how-do-i-sort-a-dictionary-by-value)):"},{"metadata":{"trusted":true,"_uuid":"f9e2e8de377b46f4774226585d6758a14429687f"},"cell_type":"code","source":"import operator\n\nsorted_frequencies = sorted(frequencies.items(), key=operator.itemgetter(1), reverse=True)\nprint(sorted_frequencies)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5229499f2577dca0bfca93a1726ccf1b06c0c84"},"cell_type":"markdown","source":"# Build the embedding matrix\n\nFollowing the [Keras blog](https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html), an embedding matrix can be created. The first step is to allocate space for the embedding matrix. The columns are the same as in the embedding index (300 in this case) and the rows are as many as the number of the training records. Words not found in embedding will be all-zeros."},{"metadata":{"trusted":true,"_uuid":"e9bc6f6155a6205d09f57e18331a581c3ea0a47c"},"cell_type":"code","source":"embedding_matrix = np.zeros((len(words_training), 300))\nmapping = {}\n\nfor index, word in enumerate(words_training):\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[index] = embedding_vector\n        mapping[word] = index\n        \nprint(embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c73cab8d27ad7cda59c467c3306e11f8dd30a8a2"},"cell_type":"code","source":"print(embedding_matrix.__class__)\nprint(embedding_matrix.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da9cd129829c264ce09218c332ac291c7e245a76"},"cell_type":"markdown","source":"# Prepare the input\n\nPrepare the data taking all tokens (for the moment, only splitting on spaces):"},{"metadata":{"trusted":true,"_uuid":"0f17fb9d75640681682800aebbbd298d198916e9"},"cell_type":"code","source":"labels = train.target\nsentences = [i.split(\" \") for i in train.question_text]\nvocab_size = len(set([item for sublist in sentences for item in sublist]))\nprint(vocab_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2d288be3523b4608ac9a3c2f5ac5c17dcaa524a"},"cell_type":"code","source":"print(sentences[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d52b512d579bf6892671bea933c843bd8d4139c"},"cell_type":"code","source":"print(mapping[sentences[0][0]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3662301fb500132227140c62bd415d345791cb22"},"cell_type":"code","source":"for i in sentences[0:2]:\n    for j in i:\n        if j in mapping.keys():\n            print(mapping[j])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0bc603932a656026b30cfc5a3dde8a88918cf71a"},"cell_type":"code","source":"for i in sentences[0:2]:\n    print([mapping[j] for j in i if j in mapping.keys()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f562a8f923f4ce873d4fb46727c453f32885d6e"},"cell_type":"code","source":"input_sequences = [[mapping[j] for j in i if j in mapping.keys()] for i in sentences]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56dc7a255b655df255fb986009d2d620d9c542fd"},"cell_type":"markdown","source":"Following [Machine Mastery](https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/), the input file is padded in order to have all input sequences of equal size padding after (*post*) until a maximum of *maxwords* (50):"},{"metadata":{"_uuid":"6bcc97c4d36e940738a2ffc47fc9d7377808a599","trusted":true},"cell_type":"code","source":"from keras.preprocessing.sequence import pad_sequences\n\npadded_docs = pad_sequences(input_sequences, maxlen=max_words, padding='post')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c342f92ae585d6f6bc3beb9886e9394d311e6bc9"},"cell_type":"markdown","source":"# Train the model\n\nFollowing [Theo Viel's kernel](https://www.kaggle.com/theoviel/improve-your-score-with-text-preprocessing-v2) the f1 metric is implemented:"},{"metadata":{"trusted":true,"_uuid":"f87447b5ab64712ad5b2687dbc59330c62ced2fd"},"cell_type":"code","source":"from keras import backend as K\n\ndef f1(y_true, y_pred):\n    def recall(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives / (predicted_positives + K.epsilon())\n        return precision\n    \n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad93e515275d0616d43e37f44e2fad74a29f975f"},"cell_type":"markdown","source":"The model can be then sketched. For the moment, a very simple network:"},{"metadata":{"trusted":true,"_uuid":"d8e82712f223552d6269fba2b3a674863c5d48cd"},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Flatten\nfrom keras.layers import Embedding\n\nmodel = Sequential()\nmodel.add(Embedding(vocab_size, 300, input_length=max_words, weights=[embedding_matrix], trainable=False))\nmodel.add(Flatten())\nmodel.add(Dense(1, activation='sigmoid'))\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=[f1])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"552ef593305d98cb0821955387d211b2cd654449"},"cell_type":"code","source":"model.fit(padded_docs, labels, epochs=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3dce4ddeb270f0b2a4fb826d5eedd0e4024229ca"},"cell_type":"code","source":"loss, f1 = model.evaluate(padded_docs, labels, verbose=0)\nprint('F1: %f' % f1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b9cd9701ddc70217918fdc333b7ae6095d5b73cb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}