{"cells":[{"metadata":{"_uuid":"df04e62e1f5d7237d201bf9f44373acecacea076"},"cell_type":"markdown","source":"In this kernel we are focused on doing two major things:\n1. Seeing if we can disambiguate words using parts of speech and if this is useful to our model\n2. Analyzing errors and understanding why they may have occurred"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# all the standard imports + spacy\nimport os\nimport time\nimport numpy as np # linear algebra                                                                                                                                                                         \nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)                                                                                                                                      \nfrom tqdm import tqdm\nimport math\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D, concatenate\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers\n\n\n\nimport matplotlib.pylab as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f1bb83cc813b48f528e6e72f57f7ad331ced945"},"cell_type":"markdown","source":"# Disambiguation\nSo what do I mean when I say dismbiguate the words using the parts of speech? If you want to dig in for real check out the wiki page for [word sense disambiguation ](https://en.wikipedia.org/wiki/Word-sense_disambiguation) , but I'll try my best to give a rudimentary example. \n\nIf you look in the dictionary many words have multiple definitions. What our word embeddings currently capture is a happy medium between the multiple different usages of the word. In theory it is likely skewing toward whatever the most common sense of the word is and just being confused by the other connotations. So for example if the word is bank then it is trying to find a reasonable location for both bank as in the financial bank and bank as in a river bank and maybe also a bank shot like in sports. Obviously these are all relatively unrelated concepts and our embedding is hopelessy trying to capture all in one 300 dimension vector.\n\nIt would be great if we had a way to disambiguate these various meanings so that they each have their own embedding that can move independently. The problem is how do we determine which sense is being used when? We don't have a perfect way of discerning this and even as humans sometimes we have misunderstandings of what specifically someone is referring to, but one potential way to make this determination is the part of speech. One thing you will notice in the dictionary is that words have their parts of speech listed and then the corresponding definitions under them. And we're in luck, automatic parts of speech tagging has been very thoroughly explored and we have libraries like spacy that can do it quickly and with a reported accuracy of 92%+. We'll do the parts of speech tagging and then I'll explain how to utilize this information further.\n\n"},{"metadata":{"trusted":true,"_uuid":"ba1b35d837317ce8231d606b098c5a745bea4f9f"},"cell_type":"code","source":"#we will import spacy and then disable the additional modules we dont need because they take a lot of compute time\nimport spacy\nnlp = spacy.load('en', disable = [\"parser\", \"ner\", \"textcat\", \"tagger\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8b534102c3e28e0c5578ae355bb89be87be6e1a"},"cell_type":"markdown","source":"**Load data**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab7f29f9212fdc7801fb0db6652603b58459fb19"},"cell_type":"markdown","source":"Here we will use the spacy pipeline just to tokenize. There are plenty of other faster/simpler tokenizers, but for the sake of consistency I am using spacy's built in. "},{"metadata":{"trusted":true,"_uuid":"a845a7980c65338d891862e816d05774b0843d07"},"cell_type":"code","source":"tokens = []\nfor doc in tqdm(nlp.pipe(train_df[\"question_text\"].values, n_threads = 16)):\n    tokens.append(\" \".join([n.text for n in doc]))\ntrain_df[\"question_text\"] = tokens","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3207e4ce22b65272811cad467c9ab5d03896a90"},"cell_type":"code","source":"results = set()\ntrain_df['question_text'].str.lower().str.split().apply(results.update)\nprint(\"Number of unique words before pos tagging:\", len(results))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e55f454ce4391d7e95dc9f5ce8341a124af93492"},"cell_type":"code","source":"print(\"What the text looks like before tagging:\",  train_df[\"question_text\"][0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa06e01989e31c92a9255631529461b8befec551"},"cell_type":"markdown","source":"Here we will use the spacy pipeline we just created in order to tag the parts of speech and then return back a string that has each word concatenated with the part of speech that was found.  "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9ef34edecb5d0637b78b04aaee6c8d0aba8b127f"},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cd85fcf4bcdf10292990890bf55381fd01305f7"},"cell_type":"code","source":"nlp = spacy.load('en', disable = [\"parser\", \"ner\", \"textcat\"])\ntokens = []\nfor doc in tqdm(nlp.pipe(train_df[\"question_text\"].values, n_threads = 16)):\n    tokens.append(\" \".join([n.text + \"_\"  + n.pos_ for n in doc]))\ntrain_df[\"question_text\"] = tokens","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96658454eab6eba2dffcbd96cfb396dfdc70b856"},"cell_type":"code","source":"print(\"after tagging:\",  train_df[\"question_text\"][0])\nresults = set()\ntrain_df['question_text'].str.lower().str.split().apply(results.update)\nprint(\"Number of unique words after pos tagging:\", len(results))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a20416ef4433fb8462242f6ec1553cf3d7ba6d9"},"cell_type":"code","source":"#number of words we have added to our vocabulary\n284281 - 219231","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8a4ecb2ec2c4bad98bd4b4cdd6e86782a337d650"},"cell_type":"code","source":"tokens = []\npos = []\nfor doc in tqdm(nlp.pipe(test_df[\"question_text\"].values, n_threads = 16)):\n    tokens.append([n.text + \"_\"  + n.pos_ for n in doc])\ntest_df[\"tokens\"] = tokens","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea0328ca5e2c0a6b8e3ca0bf9c179f4e5c4679d2"},"cell_type":"markdown","source":"**Setting up validation and training dataset**"},{"metadata":{"trusted":true,"_uuid":"b324e9197f12398425a05bf5646464b7f3cd1787"},"cell_type":"code","source":"# Cross validation - create training and testing dataset\ntrain_df, val_df = train_test_split(train_df, test_size=0.1, random_state=2018)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e692beb0fb41bd057a70508cfc7eaabf12e49d9"},"cell_type":"markdown","source":"The below is pretty much all the same standard stuff available in other public kernels. Only major thing I have changed is I removed the underscore character from the filters of the tokenizer so that our pos and word are not separated into their own tokens."},{"metadata":{"trusted":true,"_uuid":"7fca31ab0c79809c06cf8897baf22021168e501a"},"cell_type":"code","source":"# Preprocess the data\n## some config values                                                                                                                                                                                       \nembed_size = 300 # how big is each word vector                                                                                                                                                              \nmax_features = 100000 # how many unique words to use (i.e num rows in embedding vector)                                                                                                                      \nmaxlen = 20 # max number of words in a question to use                                                                                                                                                     \n\n## fill up the missing values                                                                                                                                                                               \ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\nval_X = val_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences                                                                                                                                                                                   \ntokenizer = Tokenizer(num_words=max_features, filters='!\"#$%&()*+,-./:;<=>?@[\\]^`{|}~')\ntokenizer.fit_on_texts(list(train_X))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"be26b3c8e34735e6fa92b220e7e290b2e0ee89a9"},"cell_type":"code","source":"train_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences                                                                                                                                                                                        \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\nval_X = pad_sequences(val_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\n\n## Get the target values                                                                                                                                                                                    \ntrain_y = train_df['target'].values\nval_y = val_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d9d2e845e5140966ef28e65896c8b765285dd85"},"cell_type":"markdown","source":"**Load the pretrained embeddings**"},{"metadata":{"trusted":true,"_uuid":"04a0afbec4e24f021500e45a63eff29c71adc37d"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d4482477d06e75b7549dcb44e357770dc923d76"},"cell_type":"markdown","source":"# This is where the magic happens.\nWhat we are doing here is only subtley different from what everyone else has been doing. Typically our matrix would have one entry for each word and we would fill it up by looking for that word in our pretrained embeddings. Now what we are doing is creating one entry for each word_pos pair and then looking up just the word. This will initialize bank_noun and bank_verb as the same embedding but now they will be able to be independently moved in the embedding space during training. \n\n![WSD2](https://i.imgur.com/akeenqu.png)\n\nAs you can see from my lovely paint work, now our word \"bank\" can be pushed in to separate directions for when it is used as a verb or as a noun. Might not be the best example of this, but it at least conveys the concept hopefully. Visualization is a screenshot from [projector.tensorflow.org](projector.tensorflow.org)"},{"metadata":{"trusted":true,"_uuid":"f0d062141c8dee77848a99c5ead476417924fb82"},"cell_type":"code","source":"all_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nvocab_dict = {}\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= nb_words: continue\n    word_part = word.split(\"_\")[0]\n    embedding_vector = embeddings_index.get(word_part)\n    if embedding_vector is not None: \n        if word_part in vocab_dict:\n            vocab_dict[word_part].append((word, i))\n        else:\n            vocab_dict[word_part] = [(word, i)]\n        embedding_matrix[i] = embedding_vector\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bc786e5f1e66446fc7159420cc78300d1e4bdf45"},"cell_type":"code","source":"#filter for words that with more than one part of speech\nvocab_dict = {i:vocab_dict[i] for i in vocab_dict if len(vocab_dict[i]) > 1}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d80511801a33648fad4889ab8488f94be374841a"},"cell_type":"code","source":"new_vocab_indexes = []\nnew_word_list = []\nfor word in vocab_dict.values():\n    for pos in word:\n        new_vocab_indexes.append(pos[1])\n        new_word_list.append(pos[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a187622ed56bf3774633c3f32f6c22f8f21ddf8"},"cell_type":"code","source":"from sklearn.manifold import TSNE\nfrom sklearn.decomposition import TruncatedSVD\ntsvd = TruncatedSVD(n_components=3)\n# tsne = TSNE(n_components=3)\nX_embedded = tsvd.fit_transform(embedding_matrix[new_vocab_indexes])\n# X_embedded = tsne.fit_transform(X_embedded)\nX_embedded.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a43bbaffb0b4ce7d3bcc9e39bed57cca4e42df4"},"cell_type":"markdown","source":"Using this awesome plotly interactive graph we can look at what our embeddings currently look like and see that we in fact have words with different parts of speech that are initialized to the same spot. We will check on where they are at after the embedding training. "},{"metadata":{"trusted":true,"_uuid":"85b1b6fa1ef418fe32b6f6061dd8d87b151e9ec2"},"cell_type":"code","source":"word_count = 5000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1c057eeecbc36f42e53f6e68cbfd55ed2214540c"},"cell_type":"code","source":"import plotly.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport numpy as np\ntrace1 = go.Scatter3d(\n    x=X_embedded[:word_count,0],\n    y=X_embedded[:word_count,1],\n    z=X_embedded[:word_count,2],\n    mode='markers',\n    text = new_word_list[:word_count],\n    marker=dict(\n        size=12,\n        line=dict(\n            color='rgba(217, 217, 217, 0.14)',\n            width=0.5\n        ),\n        opacity=0.8\n    )\n)\n\ndata = [trace1]\nlayout = go.Layout(\n    margin=dict(\n        l=0,\n        r=0,\n        b=0,\n        t=0\n    )\n)\nfig = go.Figure(data=data, layout=layout)\niplot(fig, filename='simple-3d-scatter')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b60b82e17961eb7359c036ff0bb3162d5193a050"},"cell_type":"code","source":"from keras.callbacks import EarlyStopping\nes = EarlyStopping(monitor='val_loss', patience=2, verbose=1, mode='min', restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c97cbcd9467670c0828738580676ce71ff9604f"},"cell_type":"markdown","source":"An insane CNN model I made just for kicks. Uncomment for an interesting graph you can show to people who doubt the complexity of your work. "},{"metadata":{"trusted":true,"_uuid":"2b8ef3c5877310d7d77fb88a362616dde9131d3f"},"cell_type":"code","source":"from keras.layers.convolutional import Convolution1D, MaxPooling1D\nfrom keras.layers import Flatten, Lambda, SpatialDropout1D\nimport keras.backend as K\n# nb_filter = 32\n# inp = Input(shape=(maxlen,))\n\n# x = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\n# x = SpatialDropout1D(.4)(x)\n# rev_x = Lambda(lambda x: K.reverse(x,axes=-1))(x)\n# conv = Convolution1D(nb_filter=nb_filter, filter_length=1,\n#                      border_mode='valid', activation='relu')(x)\n# conv1 = Convolution1D(nb_filter=nb_filter, filter_length=2,\n#                      border_mode='valid', activation='relu')(x)\n# conv2 = Convolution1D(nb_filter=nb_filter, filter_length=3,\n#                      border_mode='valid', activation='relu')(x)\n# conv3 = Convolution1D(nb_filter=nb_filter, filter_length=4,\n#                      border_mode='valid', activation='relu')(x)\n# convs = [conv, conv1, conv2, conv3]\n# convs2 = []\n# for layer in convs:\n#     conv = Convolution1D(nb_filter=nb_filter, filter_length=1,\n#                      border_mode='valid', activation='relu')(layer)\n#     conv1 = Convolution1D(nb_filter=nb_filter, filter_length=2,\n#                          border_mode='valid', activation='relu')(layer)\n#     conv2 = Convolution1D(nb_filter=nb_filter, filter_length=3,\n#                          border_mode='valid', activation='relu')(layer)\n#     conv3 = Convolution1D(nb_filter=nb_filter, filter_length=4,\n#                          border_mode='valid', activation='relu')(layer)\n#     conv = MaxPooling1D()(conv)\n#     conv1 = MaxPooling1D()(conv1)\n#     conv2 = MaxPooling1D()(conv2)\n#     conv3 = MaxPooling1D()(conv3)\n#     convs2.append(concatenate([conv, conv1, conv2, conv3], axis = 1))\n\n    \n# conv4 = concatenate(convs2, axis = 1)\n\n\n# rev_conv = Convolution1D(nb_filter=nb_filter, filter_length=1,\n#                      border_mode='valid', activation='relu')(rev_x)\n# rev_conv1 = Convolution1D(nb_filter=nb_filter, filter_length=2,\n#                      border_mode='valid', activation='relu')(rev_x)\n# rev_conv2 = Convolution1D(nb_filter=nb_filter, filter_length=3,\n#                      border_mode='valid', activation='relu')(rev_x)\n# rev_conv3 = Convolution1D(nb_filter=nb_filter, filter_length=4,\n#                      border_mode='valid', activation='relu')(rev_x)\n# rev_convs = [rev_conv, rev_conv1, rev_conv2, rev_conv3]\n# rev_convs2 = []\n# for layer in rev_convs:\n#     rev_conv = Convolution1D(nb_filter=nb_filter, filter_length=1,\n#                      border_mode='valid', activation='relu')(layer)\n#     rev_conv1 = Convolution1D(nb_filter=nb_filter, filter_length=2,\n#                          border_mode='valid', activation='relu')(layer)\n#     rev_conv2 = Convolution1D(nb_filter=nb_filter, filter_length=3,\n#                          border_mode='valid', activation='relu')(layer)\n#     rev_conv3 = Convolution1D(nb_filter=nb_filter, filter_length=4,\n#                          border_mode='valid', activation='relu')(layer)\n#     rev_conv = MaxPooling1D()(rev_conv)\n#     rev_conv1 = MaxPooling1D()(rev_conv1)\n#     rev_conv2 = MaxPooling1D()(rev_conv2)\n#     rev_conv3 = MaxPooling1D()(rev_conv3)\n    \n#     rev_convs2.append(concatenate([rev_conv, rev_conv1, rev_conv2, rev_conv3], axis = 1))\n\n    \n# rev_conv4 = concatenate(rev_convs2, axis = 1)\n# conv4 = concatenate([rev_conv4, conv4], axis = 1)\n\n# conv5 = Flatten()(conv4)\n\n# z = Dropout(0.5)(Dense(64, activation='relu')(conv5))\n# z = Dropout(0.5)(Dense(64, activation='relu')(z))\n\n# pred = Dense(1, activation='sigmoid', name='output')(z)\n\n# model = Model(inputs=inp, outputs=pred)\n\n# model.compile(loss='binary_crossentropy', optimizer='adam',\n#               metrics=['accuracy'])\n\n\n# from keras.utils.vis_utils import model_to_dot\n# from IPython.display import Image\n\n# Image(model_to_dot(model, show_shapes=True).create(prog='dot', format='png'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e8d0d85e43f7a2ec21eb31c02f7f96eebd616481"},"cell_type":"code","source":"#the model we will actually use to measure performance and analyze errors on\ninp = Input(shape=(maxlen,))\nx = Embedding(nb_words, embed_size, weights=[embedding_matrix], trainable = False)(inp)\nx = SpatialDropout1D(.4)(x)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d1bcddcd13b6f3c1258c0461877c298574c7bac"},"cell_type":"markdown","source":"We will start with frozen weights and then unfreeze them for some more tuning"},{"metadata":{"trusted":true,"_uuid":"ec19b89238d8d280f97c66bc2a08a93539f5ee0a"},"cell_type":"code","source":"model.fit(train_X, train_y, batch_size=3000, epochs=8, validation_data=(val_X, val_y), callbacks = [es])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4155ef0873c47e7c66d094cbfc59107458b8517d"},"cell_type":"code","source":"for layer in model.layers:\n    layer.trainable = False\n    if \"embedding\" in layer.name:\n        layer.trainable = True\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b889b5672b9165377829bed23835836b3bc6119"},"cell_type":"code","source":"model.fit(train_X, train_y, batch_size=3000, epochs=8, validation_data=(val_X, val_y), callbacks = [es])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cba735ace406ce828f2f708e9d35f8adc163e0d"},"cell_type":"code","source":"for layer in model.layers:\n    if \"embedding\" in layer.name:\n        new_embed = layer.get_weights()[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"abfc0e5bd3f94ef0ab57d940c457aff8ae0e61e1"},"cell_type":"markdown","source":"Now that the embeddings have been trained lets check on how they were moved and if the different POS versions were disambiguated. "},{"metadata":{"trusted":true,"_uuid":"eae14d1f57135dfc9ad85452e0da7caa17ba3dd8"},"cell_type":"code","source":"\nX_embedded = tsvd.transform(new_embed[new_vocab_indexes])\n# X_embedded = tsne.fit_transform(X_embedded)\nX_embedded.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"907f4cf82a5d04b8b921ae030de79f3c17d621df"},"cell_type":"markdown","source":"Now we can look at the cosine similarity between two versions of a word and see if they have moved. We will look at the average across all similar base words. A cosine similarity of 1 is an identical vector which is what we expect from the original embeddings"},{"metadata":{"trusted":true,"_uuid":"1d4435de3f7112a3be454262108734888b0bd810"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cfd7f4ed398e124d3e33938b6bac65b108c67405"},"cell_type":"code","source":"from sklearn.metrics.pairwise import cosine_similarity\ndef sum_cos_sim(embedding_matrix):\n    tot_sim = 0\n    sim_dict = {}\n    for word, word_parts in vocab_dict.items():\n        cos_sim = 0\n        first_embed = embedding_matrix[word_parts[0][1]].reshape(1, -1)\n        for word_part in word_parts[1:]:\n            next_embed = embedding_matrix[word_part[1]].reshape(1, -1)\n            cos_sim += cosine_similarity(first_embed, next_embed)/(len(word_parts) -1)\n        sim_dict[word] = cos_sim\n        tot_sim += cos_sim/len(vocab_dict.items())\n    return tot_sim, sim_dict\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cdb9996587c3d1aed448259b8bf41b02bf36977a"},"cell_type":"code","source":"sim_val, sim_dict = sum_cos_sim(embedding_matrix)\nprint(\"average cosine similarity of different word senses:\" , sim_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffdd4048eb6524057c66010c10c48aa399922607"},"cell_type":"code","source":"sim_val, sim_dict = sum_cos_sim(new_embed)\nprint(\"average cosine similarity of different word senses after embedding tuning:\" , sim_val)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5295c54a0175043c604374964e8203cb701633f5"},"cell_type":"markdown","source":"Looks like they were moved but only a tiny bit. Probably not enough to be useful. It is interesting to see that with more data this may be a useful method to separate the different senses of words. \n\nBelow is the same visualization as before but now with the adjusted embeddings."},{"metadata":{"trusted":true,"_uuid":"ecad47cdc69285490576161d50eec2850d672487"},"cell_type":"code","source":"import plotly.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport numpy as np\ntrace1 = go.Scatter3d(\n    x=X_embedded[:word_count,0],\n    y=X_embedded[:word_count,1],\n    z=X_embedded[:word_count,2],\n    mode='markers',\n    text = new_word_list[:word_count],\n    marker=dict(\n        size=12,\n        line=dict(\n            color='rgba(217, 217, 217, 0.14)',\n            width=0.5\n        ),\n        opacity=0.8\n    )\n)\n\ndata = [trace1]\nlayout = go.Layout(\n    margin=dict(\n        l=0,\n        r=0,\n        b=0,\n        t=0\n    )\n)\nfig = go.Figure(data=data, layout=layout)\niplot(fig, filename='simple-3d-scatter')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e4b24b6e3e90ef0aec9856ce6613d0e09192cee"},"cell_type":"markdown","source":"**Prediction on validation dataset**"},{"metadata":{"trusted":true,"_uuid":"30ed3f40a27e40d0f4be602e22da2c27d14283c9"},"cell_type":"code","source":"pred_noemb_val_y = model.predict([val_X], batch_size=1024, verbose=1)\n\nthresholds = np.arange(0.1, 0.501, 0.01)\nf1s = np.zeros(thresholds.shape[0])\n\nfor ind, thresh in np.ndenumerate(thresholds):\n    f1s[ind[0]] = metrics.f1_score(val_y, (pred_noemb_val_y > np.round(thresh, 2)).astype(int))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35dead0062146130c9ab4a9c96f62f538ff23185"},"cell_type":"code","source":"np.round(thresholds[np.argmax(f1s)], 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2b9e9e8da99a0d5f1a3f228e032e71e60636271"},"cell_type":"code","source":"import itertools\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom sklearn import svm, datasets\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n    # print(\"Normalized confusion matrix\")\n    # else:\n    # print('Confusion matrix, without normalization')\n\n    # print(cm)\n\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    # plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    fmt = '.2f' if normalize else 'd'\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, format(cm[i, j], fmt),\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"58be80e884a91bac782cbd76e702ccdd01ce77f5"},"cell_type":"code","source":"pred_noemb_val_y[:, 0].shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd145862966b3216b5a6f20c7e3b616c69188906"},"cell_type":"markdown","source":"# Error analysis\nNow that we've made predictions and found the optimal threshold we can start analyzing what is going wrong. We will start with a confusion matrix and checking out precision, recall and f1 score"},{"metadata":{"trusted":true,"_uuid":"6976566c3eb22eb3a31dda92ca2c124908b47ff4"},"cell_type":"code","source":"pred_noemb_val_y1 = pred_noemb_val_y[:, 0]\ny_test = val_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"208a100e0b28b06cf83ab3706c4f4aba0639836b"},"cell_type":"code","source":"opt_thresh = np.round(thresholds[np.argmax(f1s)], 2)\n# y_test = val_y\ny_pred = (pred_noemb_val_y > opt_thresh).astype(int)\n\ncnf_matrix = confusion_matrix(y_test, y_pred)\n\nnp.set_printoptions(precision=2)\n\n# Plot non-normalized confusion matrix\nplt.figure()\nplot_confusion_matrix(cnf_matrix, classes=['Sincere','Insincere'],\n                      title='Confusion matrix, without normalization')\n\n# Plot normalized confusion matrix\nplt.figure()\nplot_confusion_matrix(cnf_matrix, classes=['Sincere','Insincere'], normalize=True,\n                      title='Normalized confusion matrix')\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1993f1f95b0c402b527e6bf04ef3403c0728f038"},"cell_type":"code","source":"opt_thresh = np.round(thresholds[np.argmax(f1s)], 2)\ny_test = val_y\ny_pred = (pred_noemb_val_y > opt_thresh).astype(int)\n\ncnf_matrix = confusion_matrix(y_test, y_pred)\n\nnp.set_printoptions(precision=2)\n\n# Plot non-normalized confusion matrix\nplt.figure()\nplot_confusion_matrix(cnf_matrix, classes=['Sincere','Insincere'],\n                      title='Confusion matrix, without normalization')\n\n# Plot normalized confusion matrix\nplt.figure()\nplot_confusion_matrix(cnf_matrix, classes=['Sincere','Insincere'], normalize=True,\n                      title='Normalized confusion matrix')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad2b9aafcdc2aab2de521539a10145b5b19c5a57"},"cell_type":"code","source":"precision = cnf_matrix[1,1]/(cnf_matrix[1,1]+cnf_matrix[0,1])\nrecall = cnf_matrix[1,1]/(cnf_matrix[1,1]+cnf_matrix[1,0])\nprint(\"Precision: \" + str(np.round(precision, 3)))\nprint(\"Recall: \" + str(np.round(recall, 3)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4da9708498c4eddbb6b7f49cd872d4b8d20a27bc"},"cell_type":"code","source":"\"F1 Score:\", (2 * precision * recall)/(precision + recall)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1a8c9dbec48d28d679c8f17489a7b3fc23ee379"},"cell_type":"code","source":"pred_noemb_test_y = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c8d6816d581894a430b1cb711adb5ce5ee67d6bb"},"cell_type":"code","source":"original_text = val_df[\"question_text\"].fillna(\"_na_\").values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bfffd165da3f1c0bebffcbc7b14461255bfcf76"},"cell_type":"markdown","source":"Some fancy pants code my teammate Alex made for the toxic comment challenge that I've expanded on and adapted to this challenge. The gist of it is to print off the original text, then the text after embedding because that is actually what reaches our model. Everything else is just dropped, so with this we can see if potentially a distinction wasnt made because a word wasnt in our embeddings vocabulary or if it was lost in a preprocessing step. In this kernel we dont do any preprocessing, but this is a good litmus test to see exactly how that might be affecting your predictions. One other thing we can glean from this is if our questions are getting cut off by the padding size we used. In this specific case it is only 20 so that it can run quickly so many are cut off. \n\nOriginally I set this up to look at the 500 with the highest loss, but eventually I realized it might be more useful to look at the comments that are near the decision boundary threshold but not on the correct side. In theory it should be easier to move the decisions a little bit instead of taking something our model is confident is sincere all the way across the boundary to insincere.\n\nThis function prints off the correct label, the models prediction value, the loss, the original text and then the text after embedding.  "},{"metadata":{"trusted":true,"_uuid":"8582e6d6cf71fe9001b723bb023e83d6ba678546"},"cell_type":"code","source":"import operator\nfrom tqdm import tqdm\ndef analyze_model(model, num_results, reverse = False):\n    #let's see on which comments we get the biggest loss\n    train_predictions = model.predict([val_X], batch_size=250, verbose=1)\n    inverted_word_index = dict([[v,k] for k,v in word_index.items()])\n    pd.DataFrame(train_predictions).hist()\n    results = []\n    eps = 0.1 ** 64\n    for i in tqdm(range(0, len(val_y))):\n        metric = 0\n\n        for j in range(len([val_y[i]])):\n            p = train_predictions[i][j]\n            y = [val_y[i]][j]\n            metric +=  -(y * math.log(p + eps) + (1 - y) * math.log(1 - p + eps))\n        if p < opt_thresh and y == 1:\n            results.append((original_text[i], metric, val_y[i], train_predictions[i], val_X[i]))\n    results.sort(key=operator.itemgetter(1), reverse=reverse)  \n    \n    for i in range(num_results):\n        inverted_text = \"\"\n        for index in results[i][4]:\n            if index > 0:\n                word = inverted_word_index[index]\n                if not np.any(embedding_matrix[index]):\n                    word = \"_\" + word + \"_\"\n                inverted_text += word + \" \"\n\n\n        print(str(results[i][2]) + \"\\t\" + str(results[i][3]) + \"\\t\" + str(results[i][1]))\n        print(\"Original Text\")\n        print( str(results[i][0]))\n        print(\"---------------------------\")\n        print(\"Text that reached the model\")\n        print(inverted_text)\n        print(\"\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9820c605c1713989f1cd7d95f3bdbcbe58a5b60","scrolled":true},"cell_type":"code","source":"#500 highest loss comments\n#Correct Label | Model Output | Loss\n#Original Text\n#===========\n#Text that reached the model after preprocessing, tokenizing and embedding\nanalyze_model(model, 500, False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3c36a66e0411230b2261a983f3d42ee51f7cb255"},"cell_type":"markdown","source":"It appears the bigger issue is questions being cut off rather than them being missing words, but I'll let you look through them and come to your own conclusions. It seems like many of the questions have the issue of being a legitimate question, but falling under the category of being easily googled so there aren't any clear indicators in terms of vocabulary or meaning. "},{"metadata":{"trusted":true,"_uuid":"f9222a83d84913e6777392528c8eb8be8938a3bb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a16b78ac51381f4cd96f9fce12878bbb50a7ccb5"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}