{"cells":[{"metadata":{"_uuid":"14593427e547bd499c3cbe524a081d40f3bce356","_execution_state":"idle","trusted":false},"cell_type":"markdown","source":"## Introduction\nThis notebook is a simple introduction to using Keras in R, based on the fantastic book by Francois Chollet: Deep Learning with R."},{"metadata":{"_uuid":"13ad62a1a9de3773e22c206b34e0a05c7f197357"},"cell_type":"markdown","source":"## Change Log\nVersion | LB Score | Notes\n------|--------|-------\nV1 | 0.61 | Benchmark  \nV2 | 0.64 | Changed to 90/10 split, increased epochs, added single dense layer"},{"metadata":{"_uuid":"d5a5a4e8eec27017e1ae92b98521457e5bee0611"},"cell_type":"markdown","source":"## Libraries"},{"metadata":{"trusted":true,"_uuid":"4b50846476399ed78e9da386f092c92aea2a40b3","scrolled":true},"cell_type":"code","source":"library(tidyverse) # importing, cleaning, visualising \nlibrary(tidytext) # working with text\nlibrary(keras) # deep learning with keras\nlibrary(data.table) # fast csv reading\n\noptions(scipen=999) # turn off scientific display","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04098c4fda3c3bd9c24acc2296014026be9c61b5"},"cell_type":"markdown","source":"## Import\n\nLet's have a look at the data"},{"metadata":{"trusted":true,"_uuid":"df0cad3de6f90020ad53305941668ddbb4815080"},"cell_type":"code","source":"train <- fread('../input/train.csv', data.table = FALSE)\ntest <- fread('../input/test.csv', data.table = FALSE)\n\ntest %>% head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"079e1721742f65abd1dc5668b48f6d95a045784e"},"cell_type":"markdown","source":"## Initial look\n\nLet's consider a few examples of \"insincere\" questions that Quora users might pose ..."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"62042d1621b4bb8caf2e7485a4ad69d3eb4fb4ea"},"cell_type":"code","source":"train %>% filter(target == 1) %>% sample_n(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cba387e00daa7f04f5a4e1e9550488a09413b55"},"cell_type":"markdown","source":"I can see why these questions are considered \"insincere\": they're not after a genuine answer, but tend to instead either state the questioner's beliefs as fact or try to be deliberately provocative. This seems like a really interesting challenge to me - it's hard to even pinpoint what it is about these that makes them so obviously insincere. You almost need social context beyond the immediate question. Wonder how our models wil fare?"},{"metadata":{"_uuid":"c4490a7398bfd7d4c08cbf8f3b8635587f4ae679"},"cell_type":"markdown","source":"## Tokenizing"},{"metadata":{"_uuid":"e944e7486cbde772bef770da0ec730251a689675"},"cell_type":"markdown","source":"Let's start by tokenizing the sentences."},{"metadata":{"trusted":true,"_uuid":"8b9e6321bc2120b6d196bdb2b5cf8264ddfcb85e"},"cell_type":"code","source":"# Setup some parameters\n\nmax_words <- 15000 # Maximum number of words to consider as features\nmaxlen <- 64 # Text cutoff after n words\n\n\n# Prepare to tokenize the text\n\nfull <- rbind(train %>% select(question_text), test %>% select(question_text))\ntexts <- full$question_text\n\ntokenizer <- text_tokenizer(num_words = max_words) %>% \n  fit_text_tokenizer(texts)\n\n# Tokenize - i.e. convert text into a sequence of integers\n\nsequences <- texts_to_sequences(tokenizer, texts)\nword_index <- tokenizer$word_index\n\n# Pad out texts so everything is the same length\n\ndata = pad_sequences(sequences, maxlen = maxlen)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"483e1a07aef45a22d85201c6afc6655c132f84d0"},"cell_type":"markdown","source":"## Datasplit"},{"metadata":{"trusted":true,"_uuid":"6ed6bc097aadcf859298a8ea5f287b86cd6b8d71"},"cell_type":"code","source":"\n# Split back into train and test\n\ntrain_matrix = data[1:nrow(train),]\ntest_matrix = data[(nrow(train)+1):nrow(data),]\n\n\n# Prepare training labels\n\nlabels = train$target\n\n\n# Prepare a validation set\n\nset.seed(1337)\n\ntraining_samples = nrow(train_matrix)*0.90\nvalidation_samples = nrow(train_matrix)*0.10\n\nindices = sample(1:nrow(train_matrix))\ntraining_indices = indices[1:training_samples]\nvalidation_indices = indices[(training_samples + 1): (training_samples + validation_samples)]\n\nx_train = train_matrix[training_indices,]\ny_train = labels[training_indices]\n\nx_val = train_matrix[validation_indices,]\ny_val = labels[validation_indices]\n\n# Training dimensions\n\ndim(x_train)\ntable(y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acd19e34e9574514dcb1d658f9a70b4fb25c65b3"},"cell_type":"raw","source":"Pretty major imbalance here, we will need to address this later"},{"metadata":{"_uuid":"6f007be15b199e6f39bb068bdf2338d85c2a69b8"},"cell_type":"markdown","source":"## Embeddings"},{"metadata":{"_uuid":"6c955f5daaab9218783945dbd68712d0adc71cac"},"cell_type":"markdown","source":"Our first model will be based off one of the provided word embeddings. Since we're limited to 2 hours of kernel time (6 hours if we use CPU), let's start with the smaller embedding files. I'll go with the fasttext wiki-news-300d embeddings.\n"},{"metadata":{"trusted":true,"_uuid":"0819d784b3a51f2e2112dd90c7aa33aed6138b9e","scrolled":true},"cell_type":"code","source":"lines <- readLines('../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec')\n\nfastwiki_embeddings_index = new.env(hash = TRUE, parent = emptyenv())\n\nlines <- lines[2:length(lines)]\n\npb <- txtProgressBar(min = 0, max = length(lines), style = 3)\nfor (i in 1:length(lines)){\n  line <- lines[[i]]\n  values <- strsplit(line, \" \")[[1]]\n  word<- values[[1]]\n  fastwiki_embeddings_index[[word]] = as.double(values[-1])\n  setTxtProgressBar(pb, i)\n}\n\n# Create our embedding matrix\n\nfastwiki_embedding_dim = 300\nfastwiki_embedding_matrix = array(0, c(max_words, fastwiki_embedding_dim))\n\nfor (word in names(word_index)){\n  index <- word_index[[word]]\n  if (index < max_words){\n    fastwiki_embedding_vector = fastwiki_embeddings_index[[word]]\n    if (!is.null(fastwiki_embedding_vector))\n      fastwiki_embedding_matrix[index+1,] <- fastwiki_embedding_vector # Words without an embedding are all zeros\n  }\n}\n\n\ngc()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37311c5f3e116b5e8ae7eaa022f8068dd07257ca"},"cell_type":"markdown","source":"## Model Architecture\n\nWe start with a simple LSTM topped with a single dense layer for prediction. "},{"metadata":{"trusted":true,"_uuid":"c5341ea9257cb839815ebdf695003d2852fdbffd"},"cell_type":"code","source":"\n# Setup input\n\ninput <- layer_input(\n  shape = list(NULL),\n  dtype = \"int32\",\n  name = \"input\"\n)\n\n# Model layers\n\nembedding <- input %>% \n    layer_embedding(input_dim = max_words, output_dim = fastwiki_embedding_dim, name = \"embedding\")\n\nlstm <- embedding %>% \n    layer_lstm(units = maxlen,dropout = 0.25, recurrent_dropout = 0.25, return_sequences = FALSE, name = \"lstm\")\n\ndense <- lstm %>%\n    layer_dense(units = 128, activation = \"relu\", name = \"dense\") \n\npredictions <- dense %>% \n    layer_dense(units = 1, activation = \"sigmoid\", name = \"predictions\")\n\n\n# Bring model together\n\nmodel <- keras_model(input, predictions)\n\n# Freeze the embedding weights initially to prevent updates propgating back through and ruining our embedding\n\nget_layer(model, name = \"embedding\") %>% \n  set_weights(list(fastwiki_embedding_matrix)) %>% \n  freeze_weights()\n\n\n# Compile\n\nmodel %>% compile(\n  optimizer = optimizer_adam(),\n  loss = \"binary_crossentropy\",\n  metrics = \"binary_accuracy\"\n)\n\n\n# Print architecture (plot_model isn't implemented in the R package yet)\n\nprint(model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"419d2c97743a608da47593af46d63eca744f5450"},"cell_type":"markdown","source":"## Model Training"},{"metadata":{"_uuid":"cd2caaa43f0dfc9d5bf938bcafe1b64154cd1676"},"cell_type":"markdown","source":"The embedding weights will be frozen to keep training fast for an initial benchmark model."},{"metadata":{"trusted":true,"_uuid":"cca11394994332cf2190b839d7c496b1e59d7ff0"},"cell_type":"code","source":"# Train model \n\nhistory <- model %>% fit(\n  x_train,\n  y_train,\n  batch_size = 2048,\n  validation_data = list(x_val, y_val),\n  epochs = 35,\n  view_metrics = FALSE,\n  verbose = 0\n)\n\n# Look at training results\n\nprint(history)\nplot(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f20c207c07070c9bd009efc16a08ecab25aedbb"},"cell_type":"markdown","source":"This model could easily be improved with some careful fine tuning: just unfreeze the embedding layer and train the model for a few more epochs, being careful not to overfit. "},{"metadata":{"_uuid":"ac406c03f38ad59f11d0abb04be0e864e1eac1ab"},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true,"_uuid":"69f157a94c534358ccad8cf8dcc24b4d420deefc"},"cell_type":"code","source":"# Produce and save submission\n\npredictions <- predict(model, test_matrix)\npredictions <- ifelse(predictions >= 0.5, 1, 0)\n\nsubmission = data.frame(cbind(test$qid, predictions))\nnames(submission) = c(\"qid\", \"prediction\")\n\nwrite_csv(submission, \"submission.csv\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}