{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "5afc1cb0-3968-5da7-5cfa-bb1291db252e" }, "source": [ "I've got some ideas about how I can use text data such as the description but I would like the community's advices about how I should go forward.\n", "\n", "Here what I have in my mind:\n", "\n", " 1. Do not really know how to deal ML with text data + numerical data \n", " 2. Predict the the interest_level column with only the description column\n", " 3. Use the prediction proba to influence the general ML algorithm" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "35e00467-a107-ad82-81a7-b57a93bac5cd" }, "outputs": [], "source": [ "# Import the necessary librairies for this notebook\n", "import numpy as np\n", "import pandas as pd\n", "\n", "# Machine learning librairies\n", "from sklearn.preprocessing import LabelEncoder\n", "from sklearn.linear_model import SGDClassifier\n", "from sklearn.feature_extraction.text import HashingVectorizer\n", "from sklearn.cross_validation import train_test_split\n", "\n", "# Text extraction & cleaning librairies\n", "from nltk.corpus import stopwords\n", "\n", "# General librairies\n", "from subprocess import check_output\n", "import datetime\n", "import re\n", "\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "53bb37c9-9974-5c02-6181-978eb92e7f51" }, "outputs": [], "source": [ "# First load the data\n", "df = pd.read_json('../input/train.json')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0ea5c3e0-d5b4-b8be-5f8c-b46d23ba786c" }, "source": [ "Encode the 'interest_level' column for ML purpose" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d6f51a0c-704d-7432-b1b5-050c576fbb57" }, "outputs": [], "source": [ "le_interest = LabelEncoder()\n", "df['interest_level'] = le_interest.fit_transform(df['interest_level'])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0db7cfca-b91d-61a1-ab2e-0559f6a348da" }, "source": [ "## Predict interest_level based on only the description feature" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "243739bd-a17c-57d7-5fb6-75bc6a99f34a" }, "outputs": [], "source": [ "# Define a function that we will use later to split our X_train dataframe\n", "# when executing the partial_fit function\n", "def calc_len_partial(X_train, limit=15):\n", " i=1\n", " partial_len = len(X_train)\n", " div_len=0\n", " while i:\n", " if partial_len%2:\n", " partial_len = len(X_train)/2\n", " div_len += 2\n", " elif partial_len%3:\n", " partial_len = len(X_train)/3\n", " div_len += 3\n", " elif partial_len%5:\n", " partial_len = len(X_train)/5\n", " div_len += 5\n", " elif partial_len%7:\n", " partial_len = len(X_train)/7\n", " div_len += 7\n", " elif partial_len%11:\n", " partial_len = len(X_train)/11\n", " div_len += 11\n", " else:\n", " break\n", "\n", " if div_len > limit:\n", " break\n", " \n", " return len(X_train)/div_len" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2ebd8daa-69a7-0641-13fb-34acdcfa17b1" }, "outputs": [], "source": [ "# Define a preprocessor function that will help us clean the code\n", "def preprocessor(text):\n", " text = str(text)\n", " text = re.sub('<[^>]*>', '', text)\n", " text = re.sub('[\\W]+', ' ', text.lower())\n", " text = text.rstrip().lstrip()\n", " return text" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "14091261-10d7-fc27-1976-93cad736bf9c" }, "outputs": [], "source": [ "stop = stopwords.words('english')\n", "# We can also store the stop words in a pkl file\n", "#stop = pickle.load(open('plk_objects/stopwords.pkl','rb'))\n", "\n", "# We load Hashing Vectorize that will clean and preprocess the text\n", "vect = HashingVectorizer(decode_error='ignore',\n", " n_features=2**21,\n", " preprocessor=preprocessor,\n", " stop_words=stop,\n", " ngram_range=(1, 3))\n", "\n", "# For this prediction, we will use SGDC classifier\n", "clf = SGDClassifier(loss='log',\n", " random_state=1,\n", " n_iter=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9f50e30c-971f-eaba-d2e4-b7883280834c" }, "outputs": [], "source": [ "# Split the data with a majority for the training algorithm\n", "X = df['description']\n", "y = df['interest_level']\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a3abd312-ae97-22b2-ade5-adbf42be0853" }, "outputs": [], "source": [ "print(datetime.datetime.now())\n", "\n", "# Split the X_train dataframe into 20\n", "len_partial = int(calc_len_partial(X_train=X_train, limit=20))\n", "\n", "# Use partial_fit function of SGDC Classifier\n", "# It would be too long normally and the gain in accuracy do not worth it\n", "classes = np.array([0, 1, 2])\n", "for i in range(round(len(X_train)/len_partial)):\n", " X_train_ml = X_train[i:(len_partial*(i+1))]\n", " y_train_ml = y_train[i:(len_partial*(i+1))]\n", " \n", " X_train_ml = vect.transform(X_train_ml)\n", " clf.partial_fit(X_train_ml, y_train_ml, classes=classes)\n", " \n", "print(datetime.datetime.now())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9fd1f3d8-360c-3d5a-7f73-14f4a574ca9c" }, "outputs": [], "source": [ "print(\"Training accuracy: {:.3f}%\".format(clf.score(vect.transform(X_train), y_train)*100))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "22e608e8-d486-d57f-41ee-cfeb98e741e5" }, "outputs": [], "source": [ "X_test_ml = vect.transform(X_test)\n", "print(\"Testing accuracy: {:.3f}%\".format(clf.score(X_test_ml, y_test)*100))\n", "clf = clf.partial_fit(X_test_ml, y_test)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3532d952-bbfb-b63f-5da3-154150f96c81" }, "source": [ "We can conclude that the model does not overfit the training data even if we use 90% of the total data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "50702a0b-d59a-a98f-ee9d-56cf24b8c979" }, "outputs": [], "source": [ "# Integrate it into the general dataframe\n", "df_desc_pred = clf.predict_proba(vect.transform(X))\n", "df['desc_pred_0'] = df_desc_pred[:,0]\n", "df['desc_pred_1'] = df_desc_pred[:,1]\n", "df['desc_pred_2'] = df_desc_pred[:,2]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5cd20f14-fea0-a747-760e-90af7e22c064" }, "outputs": [], "source": [ "df[1:3]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "218dc882-62cc-7c90-6798-c6f0b0f3a01d" }, "outputs": [], "source": [ "# Save it if you want\n", "np.savetxt(\"df_desc_pred.csv\", df_desc_pred, delimiter=\";\")" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4fbd4fa2-6631-71e0-3c3c-33493b169a22" }, "source": [ "# Conclusion:\n", "**\"Good\" points**\n", "\n", " - The algorithm scores around 70% which is okay\n", " - The partial_fit is quick\n", " - It returns the probability of the description's prediction for each row \n", "\n", "**My questions:**\n", "\n", " - I am new in the datascience world and this is my first public kermel. \n", " So any advice will be appreciate !\n", " - I don't know if the method i chose is good: predict the description first \n", " and then integrate the predictions into the general dataframe. \n", "What do you advice me ?\n", " - What are you general advices about predicting text and numerical variables ?" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }