{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Quora Insincere Questions Classification\n### Detect toxic content to improve online conversations\nhttps://www.kaggle.com/c/quora-insincere-questions-classification\n### Goal: \nTo develop models that identify and flag insincere questions.  These are questions that:\n - have a non-neutral tone\n - are disparaging or inflammatory\n - are not grounded in reality\n - use sexual content for shock value \n \n### Data:\n - train.csv (3 columns: qid - question identifier, question_text, and target - sincere/not sincere)\n - test.csv\n - Embeddings (these are how language data is prepped for use with neural nets)\n    - GoogleNews-vectors-negative300 - https://code.google.com/archive/p/word2vec/\n    - glove.840B.300d - https://nlp.stanford.edu/projects/glove/\n    - paragram_300_sl999 - https://cogcomp.org/page/resource_view/106\n    - wiki-news-300d-1M - https://fasttext.cc/docs/en/english-vectors.html\n    \n### Our Approach:\n - **Data Exploration** \n     - how big are the datasets? (huge!) \n     - what do the sincere and insincere questions look like?\n     - are the classes balanced? (No)\n     - are there any meta-features in the questions we can use for classification? (e.g., # words, avg word length, use of caps...)\n     - what are these Embeddings and how do we use them?\n     \n - **Model selection**\n     - Logistic Regression... tried it... why not?  Performed not so good.\n     - Next step was to jump to the big guns: Neural networks with keras\n         - how to determine network layers?  What types?"},{"metadata":{"trusted":true},"cell_type":"code","source":"import re, os, time, math, operator\nimport pandas as pd\nimport numpy as np\nimport nltk\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras import layers # Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\n                         # Bidirectional, GlobalMaxPool1D\nfrom keras.layers import CuDNNGRU, LSTM               \nfrom keras.models import Model, Sequential\nfrom keras import initializers, regularizers, constraints, optimizers, layers\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Data filenames to use\nEMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\nGLOVE_EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\nWIKI_EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\nPARAGRAM_EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\nGOOGLE_EMBEDDING_FILE = '../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin'\nTRAINING_FILE = '../input/train.csv'\nTESTING_FILE = '../input/test.csv'\nembed_file_dict = { 'glove' : GLOVE_EMBEDDING_FILE, \\\n                    'wiki' : WIKI_EMBEDDING_FILE, \\\n                    'paragram' : PARAGRAM_EMBEDDING_FILE, \\\n                    'google' : GOOGLE_EMBEDDING_FILE }\n\n# global configuration parameters\nembed_size = 300      # default size of word vector\n\n# parameters for neural net configuration (bigger #s mean more params to train!)\nmax_features = 10000  # number of unique words (features/columns) to use \nmaxlen = 72           # max number of words in a question to use\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\nimport matplotlib.pyplot as plt\n%matplotlib inline\n# Thanks : https://www.kaggle.com/aashita/word-clouds-of-various-shapes ##\n# and https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-qiqc\ndef plot_wordcloud(text, mask=None, max_words=200, max_font_size=80, figure_size=(12.0,10.0), \n                   title = None, title_size=40, image_color=False):\n    stopwords = set(STOPWORDS)\n    #more_stopwords = 'one', 'br', 'Po', 'th', 'sayi', 'fo', 'Unknown'}\n    #stopwords = stopwords.union(more_stopwords)\n\n    wordcloud = WordCloud(background_color='black',\n                    stopwords = stopwords,\n                    max_words = max_words,\n                    max_font_size = max_font_size, \n                    random_state = 42,\n                    width=800, \n                    height=400,\n                    mask = mask)\n    wordcloud.generate(str(text))\n    \n    plt.figure(figsize=figure_size)\n    if image_color:\n        image_colors = ImageColorGenerator(mask);\n        plt.imshow(wordcloud.recolor(color_func=image_colors), interpolation=\"bilinear\");\n        plt.title(title, fontdict={'size': title_size,  \n                                  'verticalalignment': 'bottom'})\n    else:\n        plt.imshow(wordcloud);\n        plt.title(title, fontdict={'size': title_size, 'color': 'black', \n                                  'verticalalignment': 'bottom'})\n    plt.axis('off');\n    plt.tight_layout()  \n\n# look at sincere vs insincere questions separately\ntrain_df = pd.read_csv(TRAINING_FILE)\ntest_df = pd.read_csv(TESTING_FILE)\ndfs = train_df[train_df[\"target\"] == 0]\ndfi = train_df[train_df[\"target\"] == 1]\nplot_wordcloud(dfi[\"question_text\"], title=\"word cloud insincere questions\")\nplot_wordcloud(dfs[\"question_text\"], title=\"word cloud sincere questions\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Total questions in training dataset:\",train_df.shape[0])\nprint(\"Percentage of insincere questions: {0:.2f}%\".format(dfi.shape[0]/train_df.shape[0]))\nprint(\"Total questions in testing dataset:\",test_df.shape[0])\n\ndef getQuestionLengths(df,colname = \"question_text\"):\n    q_lengths = []\n    for index, row in df.iterrows():\n        toks = row[colname].split()\n        q_lengths.append(len(toks))\n    return q_lengths\n\n\n# decimate training data to do more analysis\nntrain = 2000\ntrain_df = train_df.loc[np.random.choice(train_df.index, ntrain, replace=False)]\ndfs = train_df[train_df[\"target\"] == 0]\ndfi = train_df[train_df[\"target\"] == 1]\nqleni = getQuestionLengths(dfi)\nqlens = getQuestionLengths(dfs)\navgleni = np.mean(qleni)\navglens = np.mean(qlens)\n\nprint(\"Average length of sincere question: {0:.2f}\".format(avglens))\nprint(\"Average length of insincere question: {0:.2f}\".format(avgleni))\n\nprint(\"\\nSome sincere questions:\")\nprint(dfs[\"question_text\"].head().values)\n\nprint(\"\\nSome insincere questions:\")\nprint(dfi[\"question_text\"].head().values)\n\nplt.hist(qlens,density=True,alpha = .5,label = \"sincere\")\nplt.hist(qleni,density=True,alpha = .5,label = \"insincere\")\nplt.title(\"Length of questions\")\nplt.legend()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading the data\n\nAs above, we read in the training and test data using pandas read_csv method.  We then split the data into a training and validation data set, which allowed us to have a sanity check and compare our trained neural network to some \"unseen\" data.**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set of utility functions for tracking time and plotting history\ndef PrintVars(num_epochs):\n    print(\"Launching with\", max_features, \"Max Feature words\")\n    print(\"Launching with\", maxlen, \"Max words per question\")\n    print(\"Launching with\", num_epochs, \"Epoches\")\n    \n\ndef StartTime():\n    currentDT = datetime.datetime.now()\n    stime = currentDT.strftime(\"%H:%M:%S\")\n    print(\"-- Start Time: \", stime)\n    return currentDT\n\ndef EndTime(startTime):\n    currentDT = datetime.datetime.now()\n    durTime = currentDT - startTime\n    etime = currentDT.strftime(\"%H:%M:%S\")\n    print(\"-- End Time: \", etime)\n    secs = durTime.total_seconds()    \n    print(\"Total Runtime in seconds = \", secs)\n    print(\"Total Runtime in minutes = \", secs / 60)\n\n# Import Data - splitting data into training and validation data, valFrac is the percentage of the data to be split off for validation purposes\ndef ImportData(valFrac = 0.1):\n    print(\"Loading training dataset from file:\", TRAINING_FILE)\n    train_df = pd.read_csv(TRAINING_FILE)\n    train_y = train_df['target']\n    train_df = train_df.drop('target', axis=1)\n    \n    # split into train and validation dataset\n    train_df, val_df, train_y, val_y = train_test_split(train_df, train_y, test_size=valFrac, random_state=2018)\n    \n    print(\"Loading testing dataset\")\n    test_df = pd.read_csv(TESTING_FILE)\n    print(\"Datasets loaded\")\n    print(\"   Train shape: \",train_df.shape)\n    print(\"   Val shape: \", val_df.shape)\n    print(\"   Test shape: \",test_df.shape)\n    print(\"Sample of data:\\n\", train_df.head())\n    \n    return train_df, train_y, val_df, val_y, test_df ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Preparing the Data for the Model\n\nOnce the data is loaded into Pandas dataframes, it can now be preprocessed for the neural network.  The first step involves cleaning the strings to remove characters and convert the words to lower case for a more consistent tokenizing.  Then, each of the questions were passed through the tokenizer to establish the \"vocabulary\" that will be used.  However, the vocabulary size is on the order of 250K words.  For all of our submissions / tests, we kept the most frequent 10-15K words.  This was mostly due to memory constraints imposed by the kaggle kernel instance.  Later efforts were spent to optimize the memory usage / performance of the kernel, which MAY enable the use of more of the words...  TBD  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function will return the tokenizer with the requested number of most frequent words (since keras doesn't do this correctly...)\n# Adapted from https://github.com/keras-team/keras/issues/8092#issuecomment-466909653\ndef GetNumWords(tk, num_words):\n    sorted_by_word_count = sorted(tk.word_counts.items(), key=lambda kv: kv[1], reverse=True)\n    tk.word_index = {}\n    i = 0\n    for word,count in sorted_by_word_count:\n        if i == num_words:\n            break\n        tk.word_index[word] = i + 1    # <= because tokenizer is 1 indexed\n        i += 1\n    return tk\n\n# This function strips the stopwords out of the questions and converts them to lower-case\ndef get_cleaned_words_from_df(df):\n    \"\"\"\n    turn dataframe 'question_text' column into list of words \n    (with question marks, capitalizations and stopwords removed)\n    \"\"\"\n    stopwords = set(nltk.corpus.stopwords.words('english'))\n    y = list(df[\"question_text\"].values)\n    words = []\n    for entry in y:\n        words += entry.split()\n    words = [w.replace(\"?\",\"\") for w in words] # remove question marks\n    words = [w.lower() for w in words if w.lower() not in stopwords] # remove \"stopwords\"\n    return words\n\n# This function tokenizes a data set and zero pads each question, to give them all a constant length (~80 words)\ndef preprocess_set(data, tokenizer, maxlength):\n    data_out = tokenizer.texts_to_sequences(data['question_text'])\n    data_out = pad_sequences(data_out, maxlen=maxlength, padding='post' )\n    return data_out, tokenizer\n\n# This function adapted from:\n# https://realpython.com/python-keras-text-classification/#convolutional-neural-networks-cnn\n# This function creates the embedding matrix from the GloVe embedding matrix.  This function searches the embedding rows for \n# only the vocab words in the word_index.  \ndef create_embedding_matrix(filepath, word_index, embedding_dim):\n    vocab_size = len(word_index) + 1  # Adding again 1 because of reserved 0 index\n    embedding_matrix = np.zeros((vocab_size, embedding_dim))\n\n    print(\"Creating embedding matrix, with word index of length\", len(word_index), \"and vocab size\", vocab_size)\n    emb_df = pd.read_csv(filepath, sep=' ')    \n    print(\"Finished reading embedding file in from csv, generating the embedding matrix...\")\n    \n    cnt = 0\n    num_rows = emb_df.shape[0]\n    spn = 0\n    for i in range(num_rows - 1, -1, -1 ):\n        row = emb_df.iloc[i]\n        word = str(row[0])\n        w = word.lower()\n        if w in word_index:\n            idx = word_index[w]\n            embedding_matrix[idx] = np.array(row[1:], dtype=np.float32)[:embedding_dim]\n        else:\n            cnt += 1\n        \n        spn += 1\n        if (spn == 50):\n            emb_df = emb_df[:-50]\n            spn = 0\n    print(\"Pruned\", cnt, \"embedding strings...\")\n    print(\"Final embedding matrix shape: \", embedding_matrix.shape)\n    return embedding_matrix, vocab_size\n\n# This is the main function for Preprocessing the X data sets, to prepare them for the model\n# Enhanced from https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\ndef PreprocessData(train_X, val_X, test_X):\n    print(\"Starting to PreprocessData ...\")      \n    # tokenize the questions\n    # https://stackoverflow.com/questions/51956000/what-does-keras-tokenizer-method-exactly-do\n    # tokenizer.texts_to_sequences Transforms each text in texts to a sequence of integers. (takes each word in the text and replaces it corresponding integer value from the word_index dictionary_.\n    tokenizer = Tokenizer(num_words=max_features)\n    \n    tokenizer.fit_on_texts(get_cleaned_words_from_df(train_X))\n    print(\"Tokenizer word_index len (train)\", len(tokenizer.word_index))\n    tokenizer.fit_on_texts(get_cleaned_words_from_df(val_X))\n    print(\"Tokenizer word_index len (valid)\", len(tokenizer.word_index))\n    tokenizer.fit_on_texts(get_cleaned_words_from_df(test_X))\n    print(\"Tokenizer word_index len (test)\", len(tokenizer.word_index))\n    \n    print(\"Max Features: \", max_features) \n    # There is a bug in the tokenizer that doesn't limit the number of words correctly....\n    tokenizer = GetNumWords(tokenizer, max_features)\n    print(\"Tokenizer word_index len (final)\", len(tokenizer.word_index))\n    \n    train_X, tokenizer = preprocess_set(train_X, tokenizer, maxlen)\n    val_X, tokenizer = preprocess_set(val_X, tokenizer, maxlen)\n    test_X, tokenizer = preprocess_set(test_X, tokenizer, maxlen)\n    \n    return train_X, val_X, test_X, tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# What do the tokenized, padded datasets look like?\n\ntrain_df, train_y, val_df, val_y, test_df = ImportData() \ntrain_X, val_X, test_X, tokenizer = PreprocessData(train_df, val_df, test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Original training data as read in as pandas DataFrame: \",type(train_df))\nprint(\"Total # of training entries:\",train_df.shape)\nn = train_df.shape[0]\nprint(\"Processed training data converted to:\",type(train_X))\nprint(\"Entries split into train and validation set: \",train_X.shape,val_X.shape)\nprint(\"Example of what the processed data looks like:\")\ntrain_X\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Embeddings\nNeural networks want to act on a matrix of numerical data.  For natural language applications, strings of words need to be converted in a sensible way to numbers.  Embeddings are a way to do that.  This kaggle competition provided us with 4 different types of embeddings, but we only actually explored using one, GloVe.  The others included Paragram, Wiki, and GoogleNews\n\n#### GloVe = “Global Vectors [for word representation]”\n- Developed at Stanford\n- Is an unsupervised model for learning word vectors (global log-bilinear regression)\n- We were provided GloVe 300-dimensional word vectors trained on different corpus\n- [Stanford GloVe paper](https://nlp.stanford.edu/pubs/glove.pdf)\n- [Toward Data Science blog - \"What is GloVe?\"](https://towardsdatascience.com/emnlp-what-is-glove-part-i-3b6ce6a7f970)\n\n\n### How we used the Embeddings:\nFor the embeddings, our approach was borrowed from public kernels made available during the live competition.  The approach consisted of reading in the embeddings data and using it to create a weight matrix.  Due to the size of the matrix, there was considerable time spent getting this conversion process as memory efficient as possible, due to the hard limit of 16 Gb in the kaggle kernel.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_matrix, vocab_size = create_embedding_matrix('../input/embeddings/glove.840B.300d/glove.840B.300d.txt', \n                                           tokenizer.word_index, embed_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating the Models\n\nFinally, we began to build the neural network model.  We tried a few variations, with different layers, as seen below.  The \"betterModel\", which achieved the best scores on the leaderboard used the Bidirectional LSTM layer.  This layer is highly effective when analyzing text or language due to the natural meaning that exists in a sentence, even if you don't hear the final word, you can often times infer it.  In LSTM the sequence of words is evaluated in the normal order and backwards simultaneously, then the results are combined."},{"metadata":{"trusted":true},"cell_type":"code","source":"def createModel(vocab_size, embedding_matrix):\n    print(\"Setting up new model to use the embedding matrix\")\n    model = Sequential()\n    model.add(layers.Embedding(vocab_size, embed_size, \n                               weights=[embedding_matrix], \n                               input_length=max_features, \n                               trainable=True))\n                               #mask_zero=True))\n    model.add(layers.GlobalMaxPool1D())\n    model.add(layers.Dense(10, activation='relu'))\n    model.add(layers.Dense(1, activation='sigmoid'))\n    print(\"Compiling model\")\n    model.compile(optimizer='adam',\n                  loss='binary_crossentropy',\n                  metrics=['accuracy'])\n    print(model.summary())\n    return model\n\ndef createBetterModel(vocab_size, embedding_matrix):\n    inLen = min(vocab_size, max_features)\n    print(\"Vocab_size:\", vocab_size, \"Max_feature:\", max_features, \"inLen:\", inLen)\n    print(\"Setting up new model to use the embedding matrix\")\n    model = Sequential()\n    model.add(layers.Embedding(vocab_size, embed_size, \n                               weights=[embedding_matrix], \n                               input_length=maxlen, \n                               trainable=True))\n                               #mask_zero=True))\n    #model.add(layers.GlobalMaxPool1D())\n    model.add(layers.Bidirectional(LSTM(10)))\n    model.add(layers.Dense(16, activation=\"relu\"))\n    model.add(layers.Dropout(0.1))\n    model.add(layers.Dense(1, activation=\"sigmoid\"))\n    print(\"Compiling model...\")\n    model.compile(optimizer='adam',\n                  loss='binary_crossentropy',\n                  metrics=['accuracy'])\n    print(model.summary())\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training a model\n\nAgain, the primary source of ideas was the list of public kaggle kernels created during and after the live competition. In these, the use of a Bidirectional Recurrent Neural Network was a common theme.  Many of the kernels made us of a specific type: CuDNNLSTM (CUDA accelerated LSTM with fewer bells and whistles ).  We had issues successfully training the CuDNNLSTM type neural net so we reverted to just LSTM.\n\n### Layers and how they chained together\n\n- inp = layers.Input(shape=(maxlen,))\n- x = layers.Embedding(max_features, embed_size, weights = [embedding_matrix])(inp)\n- x = layers.Bidirectional(LSTM(10))(x)\n- x = layers.Dense(16, activation = \"relu\")(x)\n- x = layers.Dropout(0.1)(x)\n- x = layers.Dense(1, activation = \"sigmoid\")(x)\n- model = Model(input = inp, outputs = x)\n- model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n \n\n### What do these things mean?\n\n#### Embedding layer\n - a keras provided layer used for neural networks on text data\n - requires the input data be integer encoded (each word represented by a unique integer)\n\n#### Bidirectional\n- special network structure that allows backward and forward information about the sequence at every time step\n- [Understanding Bidirectional RNN in PyTorch](https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66)\n\n#### LSTM (Long short-term memory)\n - Artificial RNN used for deep learning, proposed in 1997\n - Has feedback connections that make it a \"general purpose computer\"\n - \"A common LSTM unit is composed of a **cell**, and **input gate**, an **output gate**, and a **forget gate**.  The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell.\"\n - **For our purposes:** A RNN using LSTM units can be trained in a supervised fashion, on a set of training sequences, using an optimization algorithm, like gradient descent, combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight. \"\n - [LSTM wikipedia](https://en.wikipedia.org/wiki/Long_short-term_memory)\n \n#### Dense and Dropout layers\n- a Dense layer represents matrix vector multiplication, can be used to change the dimension of your vector\n- a Dropout layer is part of a technique to avoid overfitting, this layer takes a parameter between 0 and 1 which is the fraction of neurons to drop \n- [Quora (sincere I think) In Keras, what is a \"dense\" and a \"dropout\" layer? ](https://www.quora.com/In-Keras-what-is-a-dense-and-a-dropout-layer)"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = createBetterModel(vocab_size, embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This plot function comes from \n# https://realpython.com/python-keras-text-classification/#convolutional-neural-networks-cnn\ndef plot_history(history):\n    acc = history.history['acc']\n    val_acc = history.history['val_acc']\n    loss = history.history['loss']\n    val_loss = history.history['val_loss']\n    x = range(1, len(acc) + 1)\n\n    plt.figure(figsize=(12, 5))\n    plt.subplot(1, 2, 1)\n    plt.plot(x, acc, 'b', label='Training acc')\n    plt.plot(x, val_acc, 'r', label='Validation acc')\n    plt.title('Training and validation accuracy')\n    plt.xlabel(\"# of Epochs\")\n    plt.legend()\n    plt.subplot(1, 2, 2)\n    plt.plot(x, loss, 'b', label='Training loss')\n    plt.plot(x, val_loss, 'r', label='Validation loss')\n    plt.title('Training and validation loss')\n    plt.xlabel(\"# of Epochs\")\n    plt.legend()\n\n# Train the given model on the data, and plot accuacy vs loss\ndef PerformFit(model, X_train, y_train, X_test, y_test, num_epochs, bat_size ):\n    history = model.fit(X_train, y_train,\n                        epochs=num_epochs,\n                        verbose=False,\n                        validation_data=(X_test, y_test),\n                        batch_size=bat_size)\n    loss, accuracy = model.evaluate(X_train, y_train, verbose=False)\n    print(\"Training Accuracy: {:.4f}\".format(accuracy))\n    loss, accuracy = model.evaluate(X_test, y_test, verbose=False)\n    print(\"Testing Accuracy:  {:.4f}\".format(accuracy))\n    plot_history(history)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the model on the training X,y\nnum_epochs = 6\nmodel = PerformFit(model, train_X, train_y, val_X, val_y, num_epochs, bat_size=512)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generating Predictions and Submitting"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function evaluates the predictions with the validation answers, to determine where is the appropriate threshold\n# to split the soft scores into hard predictions\ndef DetermineThreshold(val_y, pred_y):\n    print(\"Checking Prediction against various thresholds...\")\n    threshold = 0.1\n    best_f1score = 0\n    best_threshold = threshold\n    for thresh in np.arange(0.1, 0.501, 0.01):\n        thresh = np.round(thresh, 2)\n        f1score = metrics.f1_score(val_y, (pred_y>thresh).astype(int))\n        print(\"F1 score at threshold {0} is {1}\\n\".format(thresh, f1score))\n        if f1score > best_f1score:\n            best_f1score = f1score\n            best_threshold = thresh\n    return best_threshold\n\n# This function writes the submission file to disk to be available for scoring\ndef WriteSubmissionFile(test_df, y_pred):\n    print(\"Writing submission to file...\")\n    qid = test_df[\"qid\"]\n    sub = pd.DataFrame()\n    sub[\"qid\"] = qid\n    sub[\"prediction\"] = y_pred\n    sub.to_csv(\"submission.csv\", index=False)\n\n    print(\"Done writing submission to file\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once the model has been trained, we can begin using it to predict the classifications.  In the code below, we make predictions on the validation data set, and use those predictions to determine the \"best\" threshold to use for splitting the soft predictions.  Finally, we do a prediction of the test_X data, and using the best threshold from the validation data, convert the soft predictions to hard predictions and export the results to a submission file for scoring."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"make predictions from validation dataset\")\npred_val_y = model.predict([val_X], batch_size=1024, verbose=1)\n\nthresh = DetermineThreshold(val_y, pred_val_y)\n\nprint(\"make predictions from test dataset\")\npred_test_y = model.predict([test_X], batch_size=1024, verbose=1)\n\nprint(\"Applying threshold of {0} to predictions ...\".format(thresh))\ny_pred = (pred_test_y>thresh).astype(int)\n\nWriteSubmissionFile(test_df, y_pred)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Results\n\nOur best kernel achieved a 0.632, while the competition winner scored a 0.7132.  The competition was open for three months."},{"metadata":{},"cell_type":"markdown","source":"## Discussion / Lessons Learned\n\nThrough this project, we were exposed to a variety of ways to express and model non-numerical data in ways that can be understood and analyzed using traditional Machine Learning methods / algorithms.  A lesson learned during this project was that, as with all ML applications, getting the data prepared and clean is vital to getting the model to perform well.  Despite the fact that we did not need to collect any data for this competition, we still did need to vet the data we were given, to prune out abnormalities.\n\nAnother lesson learned was that there are a number of different neural network layers that are available, and knowing which one to use in a specific scenario is non-trivial.  Furthermore, getting the layers to work in combination with each other is even more involved.\n\nThe third lesson learned was the data set sizes for problems of this scope can be huge, and therefore, creating a practice data set from the beginning, for testing algorithms or an end-to-end pipeline, would have substantially reduced the amount of time lost due to processing time of the large data."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}