{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "225108d0-898e-9e00-5b7f-331df096a7d0" }, "source": [ "This doc will walk through Three stages of studying stackflow tags data set.\n", "\n", " **1. First stage: Data cleaning**\n", "\n", "As we notice there are columns that contain html mark ups, we want to make sure that it is taken care of and cleaned up\n", "\n", " **2. Second Stage: Feature engineering**\n", "\n", "Apply methods to engineer features from cleaned text data\n", " \n", "**3. Third Stage: Classification modeling**\n", "\n", "Using engineered features to build predictive models" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "60e5123b-0d2e-faff-5148-b6246aa9c8d8" }, "outputs": [], "source": [ "# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import nltk # natural language processing\n", "import re # regular expression\n", "from bs4 import BeautifulSoup #scraping HTML\n", "from nltk.corpus import stopwords\n", "import seaborn as sns # visualization\n", "import matplotlib.pyplot as plt\n", "from sklearn.feature_extraction.text import CountVectorizer\n", "from sklearn.feature_extraction.text import TfidfTransformer\n", "from string import punctuation\n", "\n", "\n", "\n", "\n", "# nltk workspace\n", "\n", "stop = set(stopwords.words('english'))\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f81d7e82-fd33-30dc-6d75-e049fa9aff96" }, "outputs": [], "source": [ "# read in raw data sets\n", "biology = pd.read_csv(\"../input/biology.csv\")\n", "cooking = pd.read_csv(\"../input/cooking.csv\")\n", "crypto = pd.read_csv(\"../input/crypto.csv\")\n", "diy = pd.read_csv(\"../input/diy.csv\")\n", "robotics = pd.read_csv(\"../input/robotics.csv\")\n", "travel = pd.read_csv(\"../input/travel.csv\")\n", "test = pd.read_csv(\"../input/test.csv\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a92bfbd0-08d4-8592-300d-050603b037f1" }, "outputs": [], "source": [ "## Concatenate datasets\n", "#raw = pd.concat([biology,cooking,crypto,diy,robotics,travel],axis = 0, ignore_index =True)\n", "raw = test" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e3f3d6cc-5647-f230-b018-9e6ccf8892f8" }, "outputs": [], "source": [ "# Simple Statistics\n", "print(\"This dataset has in total {} rows.\".format(raw.shape[0]))\n", "#print(\"out of which, {} rows come from train dataset and {} rows come from test dataset.\".format(raw[raw['source']=='train'].shape[0],raw[raw['source']=='test'].shape[0]))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "17b1d0a6-d27c-d3d9-2868-fea5a034d3f9" }, "source": [ "# **Stage 1: Data Cleaning**" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a2fe2f86-f49b-6eb9-e4b4-bbb9fa769760" }, "source": [ "Content column is a little bit messy since it's embedded in HTML. cleaning it up before applying any natural language processing tactics will facilitates our analysis moving forward. Beautifulsoup is a python package that helps dealing with HTML. The primary goal of this exercise is to:\n", "\n", "\n", " - Getting the clean text data;\n", " - Extract potentially important information from HTML tags. \n", "\n", " \n", "**Tags we want to pay attention to:**\n", "\n", " 1. The emphasize class: strong, em, i, li\n", " 2. The header class: h1, h2, h3\n", " 3. Link " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3727ba05-f6f4-495e-257f-0a77c3344240" }, "outputs": [], "source": [ "def parse_content(s):\n", " emphasize = []\n", " header = []\n", " link = []\n", " content = \"\"\n", " soup = BeautifulSoup(s,'html.parser')\n", " content = soup.get_text()\n", " return pd.Series({'content_parsed':content})" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "df4f6289-3856-8c5b-aff7-81953567f561" }, "outputs": [], "source": [ "# Apply parse_content onto dataframe.\n", "raw = pd.concat([raw,raw['content'].apply(parse_content)],axis = 1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "17b7d828-5ca8-32ab-98c7-efc7398f9713" }, "outputs": [], "source": [ "test[1:5]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "74f04450-b588-bab0-7db4-d08c4b14a89a" }, "source": [ "## Stage 2: A dive into the tags##\n", "\n", "Before we predict the tags, we want to study and understand the tags. Two initial thoughts are:\n", "\n", " - Studying the distribution of word counts \n", " - Studying the distribution of semantics counts" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "bfeea021-89da-6847-63ea-ed6fb1b1c72c" }, "outputs": [], "source": [ "## Study the tags Word count distribution\n", "#raw['tags'] = raw['tags'].apply(lambda x: x.split(\" \"))\n", "#raw['tags_wc'] = raw['tags'].apply(len)\n", "#sns.barplot(x='tags_wc',y='tags_wc',data=raw,estimator=lambda x: len(x),palette='Blues')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7ffcd0d8-5111-5c04-cf0e-6d4963aabc54" }, "outputs": [], "source": [ "##study the tags: semantic structure\n", "#raw['tags_token'] = raw['tags'].apply(str).apply(nltk.word_tokenize)\n", "#raw['tags_pos'] = raw['tags'].apply(nltk.pos_tag)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7773d5ca-55d2-b217-1d3a-5d6e2ff8611d" }, "outputs": [], "source": [ "# What is the distribution of different semantics?\n", "#semantics = pd.DataFrame({'semantics' : [pair[1] for col in raw['tags_pos'].tolist() for pair in col]})\n", "#semantics['count'] = 1\n", "#fig,axs = plt.subplots()\n", "#sns.barplot(x='semantics',y='count',data=semantics,estimator=lambda x: len(x),palette=sns.cubehelix_palette(8, start=.5, rot=-.75),ax=axs)\n", "#axs.set_title('semantic distribution')\n", "#axs.set_ylabel('frequency')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "108c9d98-2b02-2db1-c186-bb67f6f47562" }, "source": [ "As what we would expect, The semantic strongly skewed towards **nouns** (NN,NNS,JJ)." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9a5a14d0-10a3-17a5-7e4f-8a3f488f896c" }, "source": [ "## **Na\u00efve Try : Rule based Algorithm** ##\n", "\n", "The first Iteration is very brute. We are going to count the frequencies(but here \"adjusted\" frequencies - TF-IDF), the highest the score one word get, the more likely it's going to be included in the tags.\n", "\n", "How many words are we going to include? Since the number of words a tag can have has an upper bound 5, we are going to include 5 words." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0a7be3a3-70d3-7235-cbdc-2195e3f05d8a" }, "outputs": [], "source": [ "from string import punctuation\n", "def strip_punctuation(s):\n", " return ''.join(c for c in s if c not in punctuation)\n", "#Building NLTK pipelines\n", "def td_idf_matrix(dataset):\n", " dataset['all_text'] = dataset['title'] + dataset['content_parsed'] \n", " dataset['all_text'] = dataset['all_text'].apply(lambda x: str.lower(x).replace('\\n',' '))\n", " mydoclist = [strip_punctuation(doc) for doc in dataset['all_text'].tolist()]\n", " count_vectorizer = CountVectorizer(stop_words='english',lowercase=True,analyzer='word')\n", " term_freq_matrix = count_vectorizer.fit_transform(mydoclist)\n", " tfidf = TfidfTransformer(norm=\"l2\")\n", " tfidf.fit(term_freq_matrix)\n", " tf_idf_matrix = tfidf.transform(term_freq_matrix)\n", " pos_to_word = dict([[v,k] for k,v in count_vectorizer.vocabulary_.items()])\n", " return tf_idf_matrix, pos_to_word\n", "\n", "tf_idf_matrix, pos_to_word = td_idf_matrix(raw)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3cc35f39-fec4-82ca-f3a8-132888df640a" }, "outputs": [], "source": [ "\n", "def importance_list_row(sparse_row,n_importance):\n", " importance_list = [0]*n_importance\n", " for i in range(0,n_importance): \n", " ind = sparse_row.indices[sparse_row.data.argmax(axis=0)] if sparse_row.nnz else 0\n", " importance_list[i] = pos_to_word[ind]\n", " sparse_row[0,ind] = 0\n", " return importance_list\n", "\n", "\n", "def importance_list(sparse_matrix,n_importance):\n", " n_row = sparse_matrix.shape[0]\n", " importance_lists = [0]*n_row\n", " for row in range(0,n_row):\n", " importance_lists[row] = importance_list_row(sparse_matrix[row],n_importance)\n", " return importance_lists " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "371acc19-cf65-871b-d49d-0ba699249bb7" }, "outputs": [], "source": [ "#n_importance = 2\n", "#predict = importance_list(tf_idf_matrix,n_importance)\n", "#predict_vs_actual = pd.DataFrame({'predict':predict})\n", "#predict_vs_actual['predict'] = predict_vs_actual['predict'].apply(lambda x: \"\".join(chr+\" \") for char in x)\n", "#predict_vs_actual[0:50]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7cc8261c-5d03-154a-81c5-6c28ba540d70" }, "source": [ "The result from our naive approach is quite nasty. For the next iteration, I will work on improving it.\n", "\n", "One idea stems from the semantic distribution. The majority of tags are nouns and adjectives. therefore I'm going to tag my content and parse out words by their semantics and only focus on the **noun and the adjectives**." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c4987560-82df-5625-b7bf-d960c3543304" }, "outputs": [], "source": [ "#tokenize and tag texts. \n", "lemmatizer = nltk.stem.WordNetLemmatizer()\n", "raw['all_text'] = raw['all_text'].apply(strip_punctuation)\n", "raw['text_token'] = raw['all_text'].apply(nltk.word_tokenize)\n", "raw['text_token'] = raw['all_text'].apply(nltk.word_tokenize)\n", "raw['text_token'] = raw['text_token'].apply(lambda x:[lemmatizer.lemmatize(t) for t in x])\n", "raw['text_pos'] = raw['text_token'].apply(nltk.pos_tag)\n", "raw['text_nouns'] = raw['text_pos'].apply(lambda x: [pair[0] for pair in x if pair[1] in (\"NN\",\"NNS\",\"JJ\")])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e6c3f779-8c69-8570-0ea9-26b707b2931e" }, "outputs": [], "source": [ "raw['text_bigram'] = raw['text_pos'].apply(nltk.bigrams)\n", "raw['text_bigram'] = raw['text_bigram'].apply(list)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "520c9850-aa1f-1751-caa7-a94f820dfb66" }, "outputs": [], "source": [ "raw['word_pair'] = raw['text_bigram'].apply(findPair)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f1ce7e27-8032-2b3f-d90a-cd069ab02ac9" }, "outputs": [], "source": [ "raw[0:5]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ee749d56-1533-a076-f249-6d581d0de057" }, "outputs": [], "source": [ "def findPair(l):\n", " result = []\n", " for pair in l:\n", " if pair[1][1] in ('NN','NNS') and pair[0][1] in ('NN','NNS','JJ'):\n", " result.append(pair[0][0]+\" \"+pair[1][0])\n", " return result" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "597f9306-cfb6-b826-fd0e-8905551d3424" }, "source": [ "This seems to look much better. Let's give it a try." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c856254e-b879-6250-9947-3ed95bc2ec53" }, "outputs": [], "source": [ "mydoclist = raw['text_nouns'].apply(\" \".join).tolist()\n", "#mydoclist[0:5]\n", "count_vectorizer = CountVectorizer(stop_words='english',lowercase=True,analyzer='word',ngram_range=(1,1))\n", "term_freq_matrix = count_vectorizer.fit_transform(mydoclist)\n", "tfidf = TfidfTransformer(norm=\"l2\")\n", "tfidf.fit(term_freq_matrix)\n", "tf_idf_matrix = tfidf.transform(term_freq_matrix)\n", "pos_to_word = dict([[v,k] for k,v in count_vectorizer.vocabulary_.items()])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "03354ab4-c6ed-98d2-e5ce-d0592d23cda1" }, "outputs": [], "source": [ "n_importance = 3\n", "predict = importance_list(tf_idf_matrix,n_importance)\n", "predict_vs_actual = pd.DataFrame({'tags':predict,'id':raw['id']})\n", "predict_vs_actual['tags'] = predict_vs_actual['tags'].apply(\" \".join)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7378d643-96dc-3cff-c97f-962b320ecc2c" }, "outputs": [], "source": [ "predict_vs_actual[0:100]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5a7f3d26-1127-8797-a51b-5d5cd8ba1e99" }, "outputs": [], "source": [ "predict_vs_actual.to_csv(\"predicted.csv\",index=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9f3d4445-3158-c8be-bea9-0b5d7b06238a" }, "source": [ "# **Stage Two: Feature Engineering(TBD)**\n", "\n", "When people are posting on stack flow, they often have a **problem** to **solve**. So in the texts, what comes the most important would be the **noun** followed by the **verb**.\n", "\n", "As we are predicting unseen topics, supervised learning might not be deemed as useful here. For the first iteration I'm going to take a na\u00efve approach - by building up a very simple, **rule based** algorithm. \n", "\n", "A natural thought lands on frequency - But we also don't want to look at term frequency since we don't want our result overshadowed by high frequent but meaningless stop words. TF-IDF comes in handy.\n", "\n", "Also - from my years of experiences in writing irritatingly stupid questions on stack flow, writing the core question in the title as well as in the content will increase the odds of getting your question answers. reversely, **if a keyword shows up both on titles and on content, it might be the one we are looking for**.\n", "\n", "Other considerations - we may want to study the tag a little bit to understand how many words people normally tend to include in the tag(the word count distribution of the tags) and whether their is evident semantic structure to the tags. " ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }