{"cells":[{"metadata":{"_uuid":"ad9fe692daa64bc60058c5f6e34a6fbeefefca41"},"cell_type":"markdown","source":"# LDA Testing and F1 Wonderings\n\nThere was a short discussion in the competition forum about usefulness of LDA or topic models in general. Just recently someone posted a Gensim based topic model exploration kernel as well. Since I have used LDA/topic models for other purposes before, I thought it would be interesting to give it a try to use the topic distributions of documents (questions in this case) as features. This kernel scores very poorly but it was an interesting exercise. Let me know ideas how to improve.\n\nMuch of the basics I have taken from my wine review topic models kernel:\nhttps://www.kaggle.com/donkeys/topic-models-for-wine-reviews\n\nthis kernel just uses the topic distributions as features. Got any other ideas on how to use them here?"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom gensim import corpora\nfrom nltk.corpus import stopwords\nfrom nltk import word_tokenize\nfrom nltk.stem import WordNetLemmatizer\nimport nltk","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b4361fcc2a40fa6e1b2665cb1cdc42f493d20e8"},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4b8ded0434632ff291489db18ef400eb090403b"},"cell_type":"code","source":"df_test = pd.read_csv('../input/test.csv')\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d44131a50beaf0727be1b9b455720d8679a2196"},"cell_type":"markdown","source":"Basic stopwords to remove, along with a bit of garbage I picked from the top words for topics, when printed. Did not look into this in too much detail yet, since this is intended as a lightweight experiment:"},{"metadata":{"trusted":true,"_uuid":"8230404042657329490ddf1d523efc7e4b9f9680"},"cell_type":"code","source":"from string import punctuation\nstop_words = set(stopwords.words('english')) \nstop_words = stop_words.union(set(punctuation)) \nstop_words.update([\"\\'s\", \"n\\'t\", \"``\", \"\\'\\'\", \"“\", \"”\", \"\\'m\", \"’\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6aa7493c0b85f9f4063bef1464174dea4bc8cc0e"},"cell_type":"markdown","source":"Concat the train and test data for pre-processing:"},{"metadata":{"trusted":true,"_uuid":"f2340c81ef84b604e73147f0282401cf1eab252d"},"cell_type":"code","source":"question_texts = df_train[\"question_text\"]\ntest_texts = df_test[\"question_text\"]\nquestion_texts = pd.concat([question_texts, test_texts]).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e56d5a9eaa863ef29aff7083244bef72a5ce2b25"},"cell_type":"markdown","source":"Lemmatize to get baseforms of words."},{"metadata":{"trusted":true,"_uuid":"9b6486a065d2684c57361bca8717a5e07ca437ba"},"cell_type":"code","source":"lemmatizer = WordNetLemmatizer()\ntexts = [[lemmatizer.lemmatize(word) for word in word_tokenize(text.lower()) if word not in stop_words] \n         for text in question_texts]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4059f92433b4895073ccebce0376f7d2ec8ca85"},"cell_type":"markdown","source":"What does it all look like?"},{"metadata":{"trusted":true,"_uuid":"c70d91b66b2e33dde36e94bbd7201ea622e1afd9"},"cell_type":"code","source":"question_texts.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f96d3fb3f29dc908bc693bc64691c446b687a31"},"cell_type":"code","source":"question_texts[4]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"32db15d8b1ce6fb1280c42001a6c312ac0d7e623"},"cell_type":"code","source":"texts[4]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cef547da7d456b0a30527746d40d9f41a1d448b1"},"cell_type":"markdown","source":"Bi-grams and tri-grams for the text. Combining 2-3 words as one when common, such as mountain and bike as mountain-bike. *min_count* and *threshold* are statistics for when to consider the n-grams as one."},{"metadata":{"trusted":true,"_uuid":"a21028106777d738a92b5b3a1b304e1ab25db6ee"},"cell_type":"code","source":"import gensim\n#https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/\n# Build the bigram and trigram models\nbigram = gensim.models.Phrases(texts, min_count=5, threshold=100)\ntrigram = gensim.models.Phrases(bigram[texts], threshold=100)\n# Faster way to get a sentence clubbed as a trigram/bigram\nbigram_mod = gensim.models.phrases.Phraser(bigram) \ntrigram_mod = gensim.models.phrases.Phraser(trigram)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"530ada984c739fd1566243d9115a52a705978518"},"cell_type":"markdown","source":"Look mom, its an example, with the mountain-bike in it, uh:"},{"metadata":{"trusted":true,"_uuid":"ddbad3d1970e2014c4a9fc691de780a83a77bc34"},"cell_type":"code","source":"trigram_mod[bigram_mod[texts[4]]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e82decdc35186e6c29992082da7a699895d2f2a"},"cell_type":"markdown","source":"Just to do this (bi-gram and tri-gram generation)  for all the question texts we have available:"},{"metadata":{"trusted":true,"_uuid":"3edcbac478bbf40a8d13a16151f44bf054dafb64"},"cell_type":"code","source":"texts = [trigram_mod[bigram_mod[text]] for text in texts]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b395ed1ff845e2171a7bfbb06c42ecef75bfce08"},"cell_type":"code","source":" #id to word mapping for gensim\nid2word = corpora.Dictionary(texts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"287da3fcac880e1dea7a4650d8ac16bbccc30aa0"},"cell_type":"markdown","source":"Define a function to build an LDA model to represent the topics and their distributions. Use the Gensim multicore version to make it faster:"},{"metadata":{"trusted":true,"_uuid":"a4c203a93f28f9cc0af2b8b5b7ba11fedff381cc"},"cell_type":"code","source":"from gensim.models import LdaMulticore\n\ndef create_lda(topic_count, seed):\n    print(\"Running for topics:\"+str(topic_count))\n    print(\"creating corpus\")\n    corpus = [id2word.doc2bow(text) for text in texts] \n    print(\"creating lda model\")\n    lda_model = LdaMulticore(corpus, id2word=id2word, num_topics=topic_count, random_state=seed)\n    \n    print(\"applying lda model to texts\")\n    rows = [lda_model[id2word.doc2bow(text)] for text in texts]\n    cols = ['lda'+str(n) for n in range(1, topic_count+1)] #range is exclusive on top end, so +1 needed\n    print(\"changing lda results to weight only\")\n    rows = [[topic_weight[1] for topic_weight in topics] for topics in rows]\n    print(\"row len:\"+str(len(rows[0])))\n    print(\"row 1:\"+str(rows[0]))\n    df2 = pd.DataFrame(rows, columns=cols)\n    return df2, lda_model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1ff5762f38991605cfc4fd0db7d8e101d30bb41"},"cell_type":"markdown","source":"A function to try different thresholds on prediction probabilities to maximize the target F1 score. Plot the thresholds in relation to precision, recall, accuracy, and F1 score. Just so I can try to understand how the threshold affects it all. For the plots!"},{"metadata":{"trusted":true,"_uuid":"78d79cd6bc282b6341d753c6e28766a74b68e03e"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn  as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report, confusion_matrix\n\ndef plot_f1(target, pred):\n    best_score = 0\n    thresh = 0\n    thresholds = []\n    f1_scores = []\n    p_scores = []\n    r_scores = []\n    a_scores = []\n    for i in np.arange(0.5, 0.99, 0.01):\n        pred_bool = (pred < i)\n        temp_f1_score = f1_score(target, pred_bool)\n        thresholds.append(i)\n        f1_scores.append(temp_f1_score)\n        p_scores.append(precision_score(target, pred_bool))\n        r_scores.append(recall_score(target, pred_bool))\n        a_scores.append(accuracy_score(target, pred_bool))\n        if(temp_f1_score > best_score):\n            best_score = temp_f1_score\n            thresh = i\n            print(\"threshold:\"+str(i)+\"=\"+str(best_score))\n\n    plt.figure(figsize=(8,5))\n    plt.title(\"F1 at thresholds\")\n    plt.plot(thresholds, f1_scores, label='F1 score')\n    plt.plot(thresholds, p_scores, label='Precision score')\n    plt.plot(thresholds, r_scores, label='Recall score')\n    plt.plot(thresholds, a_scores, label='Accuracy score')\n    #https://stackoverflow.com/questions/24988448/how-to-draw-vertical-lines-on-a-given-plot-in-matplotlib\n    plt.axvline(x=thresh, color='k', linestyle='--', label='max='+\"%.2f\" % thresh)\n    plt.legend(loc=\"upper left\")\n    plt.show()\n    print(\"%.2f\" % thresh+\"=\"+\"%.2f\" % best_score)\n    return thresh, best_score\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4098f60a1e4b3c1ebad9964cc62b97662d9ce8d8"},"cell_type":"markdown","source":"I tried various numbers of topic counts and various seeds with the above functions. Settled for 5 topics as it seemed to work good for prediction features in this dataset. Larger numbers of topics seem to produce much more sparse feature sets. Not sure if that has something to do with it. Also tried a few seeds, as the seed seems to have surprisingly large impact on score. This one was ok'ish, although I am sure there are much better ones. It seems odd how much the seed can make a difference:"},{"metadata":{"trusted":true,"_uuid":"4faebd408965f554f97f0591a1568ded0960ecb6","scrolled":false},"cell_type":"code","source":"df2, lda_model = create_lda(5, 348617921)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"730d70c3c200f63cb60729f583bf9b05ee3673f1"},"cell_type":"markdown","source":"I use LGBM here, just because it is simple, popular, and works good enough for many cases. So suits me fine here. I used the hyperopt library to optimize hyperparameter values, although it did not seem to make much of a difference in this case. Maybe my features are not that great?"},{"metadata":{"trusted":true,"_uuid":"60f51749a9d5adb76720c28e7a00f1ce7bd8ca82"},"cell_type":"code","source":"import lightgbm as lgb \n\ndf_train_lda = pd.concat([df_train, df2[:len(df_train)]], axis=1)\ndf3 = df2[len(df_train):].reset_index().drop(\"index\", axis=1)\ndf_test_lda = pd.concat([df_test, df3], axis=1)\n\nX_cols = [col for col in df_train_lda.columns if col not in [\"target\", \"qid\", \"question_text\"]]\nprint(X_cols)\nX = df_train_lda[X_cols]\ny = df_train_lda[\"target\"]\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.33, random_state=42)\n\nclf = lgb.LGBMClassifier(\n                    num_leaves=116,\n                    learning_rate=0.1,\n                    feature_fraction=0.98,\n                    subsample=0.93, \n                    subsample_freq=1, #bagging_freq in some docs\n                    n_estimators=2000,\n                    boosting_type='gbdt',\n                    min_child_samples = 250,\n                    n_jobs = 4,\n                    max_depth=11,\n                    verbosity = 0)\n\nclf.fit(X_train, y_train,\n        eval_set=[(X_val, y_val)],\n        #eval_metric='accuracy',\n        eval_metric='binary_logloss',\n        categorical_feature = 'auto',\n        early_stopping_rounds=5,\n        verbose=False\n       )\n\ny_pred_prob = clf.predict_proba(X_val)\ny_pred_prob = y_pred_prob[:,0] \nthreshold, score = plot_f1(y_val, y_pred_prob)\nX_test = df_test_lda[X_cols]\n\ntopic_preds = clf.predict_proba(X_test)\ntopic_preds = topic_preds[:,0] \nbest_pred = topic_preds < threshold\nbest_probs = topic_preds\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"26e7c7f5ca6a6098eeec772d874d5d210029425f"},"cell_type":"markdown","source":"The figure above plots how the different scores change as the threshold is varied. I was hoping to get a more intuitive feel for how the F1 score changes in relation to precision and recall, and what it really represents. The vertical dashed line is where the F1 score is highest, with threshold of 81%, producing an F1 score of 0.36. I guess the main thing I learned from this plot is that when precision and recall intersect, F1 score also crosses over. For F1 score, I still have look for the intuition. And accuracy is something completely different from all these :)"},{"metadata":{"trusted":true,"_uuid":"85958e8915cb105f03fe3377de25e91f68784618"},"cell_type":"markdown","source":"What does the output look like?"},{"metadata":{"trusted":true,"_uuid":"e1bef05a79a77fb8fc18a31eb0d12aa63d7829b0"},"cell_type":"code","source":"best_pred","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8edabf7cafb07912fc30f49a46ce47dc932995d4"},"cell_type":"markdown","source":"Finally build something that can be submitted:"},{"metadata":{"trusted":true,"_uuid":"f2ee56bea83277c7d12007d4f46d0c0e9add960b"},"cell_type":"code","source":"results = pd.DataFrame()\nresults[\"pred\"] = best_pred\nresults[\"pred\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"733184b0f5daf567ba2f6c525822e818c24a7095"},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23482dccd25fa8d51fbc862c7ff897fb26140a03"},"cell_type":"code","source":"submission['prediction'] = best_pred\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7d23e1adb75c4c8e8901b509ba33a2c43285063"},"cell_type":"markdown","source":"Now that the results are done, a final look at the topic model and how it maps to some questions. First, print the top 20 words for each 5 topics:"},{"metadata":{"trusted":true,"_uuid":"8083b533e814df460c36dea1cb08bc04efab45b0"},"cell_type":"code","source":"lda_model.print_topics(num_words=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6dfb498be32475c37d0b7484b7a74792fa61d462"},"cell_type":"markdown","source":" There seems to be some underlying logic to the topic distributions, but I let you think about that. LDA topics always struct me as having some meaning behind it for some of the topics, but parts of it eluding me. Guess it is their nature as unsupervised learning.\n\nI tried to print the topic weights for the above topics for a few questions in the dataset. Many topics vs questions were a bit hard to interpret. Tried to pick a few slightly reasonable looking mappings below:"},{"metadata":{"trusted":true,"_uuid":"b77c04319b4217f86455418613d86bc463d955d2"},"cell_type":"code","source":"question_texts[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1999e4f13dc86042b6d2870dcdfc2023a3632aa9"},"cell_type":"code","source":"lda_model[id2word.doc2bow(texts[1])]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08646bbb2daa9f93c73eb357fe1c573fd4e9052f"},"cell_type":"markdown","source":"So, I have all the seeds in place, and the topic distributions don't change over runs, question 1 is about adopted dogs. It has a weight of 77% for the first topic, and 14% for the last topic. Seems to make some sense, looking at the topics. \n\nOne more:"},{"metadata":{"trusted":true,"_uuid":"28691a080c97d40d09a38574264478d73a9b841b"},"cell_type":"code","source":"question_texts[10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9db9defad58cde3951efeaeafee163a07f91e909"},"cell_type":"code","source":"lda_model[id2word.doc2bow(texts[10])]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"575d08467ca84b7b88a00337a2e4cafa3f5a8fe0"},"cell_type":"markdown","source":"This one mapped similarly high on the first topic, and spreads the rest evenly all around."},{"metadata":{"_uuid":"9b04875910b185ab91a0c40e5e68a4626295e2e0"},"cell_type":"markdown","source":"What can I say in the end? This kernel scores poorly, but it was interesting to play with. \n\nAs far as I understand, LDA maps each word in each document to a single topic. Maybe this also limits its ability to represent more complex relationships within a short document such as these? Although I am not entirely sure how Gensim counts the topic weights for a whole document at once. \n\nOne could also investigate the weights of individual words within a topic for more insights if interested to dig deeper into that. Or try other combinations, but since there are already many techniques that score good, perhaps not.\n\nAny other ideas?"},{"metadata":{"trusted":true,"_uuid":"1acb35e3e1dcfcfa9882f636eaa7862ade6f6182"},"cell_type":"markdown","source":"Thats it. Thank you for stopping by :)"},{"metadata":{"trusted":true,"_uuid":"72c05b53ae6f712f3612a0ab070ccc76d1f12681"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}