{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "6e711393-7a75-17e3-539c-9169c1ae1225" }, "source": [ "It seems the current [high scoring script][1] is written in R using H2O. So let us do one in python using XGBoost. \n", "\n", "Thanks to [this script][2] for feature engineering ideas. \n", "\n", "We shall start with importing the necessary modules\n", "\n", "\n", " [1]: https://www.kaggle.com/gospursgo/two-sigma-connect-rental-listing-inquiries/h2o-starter-pack/run/835757\n", " [2]: https://www.kaggle.com/aikinogard/two-sigma-connect-rental-listing-inquiries/random-forest-starter-with-numerical-features" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1952347b-6dc9-b9f1-fa25-94587a2aee77" }, "outputs": [], "source": [ "import os\n", "import sys\n", "import operator\n", "import numpy as np\n", "import pandas as pd\n", "from scipy import sparse\n", "import xgboost as xgb\n", "from sklearn import model_selection, preprocessing, ensemble\n", "from sklearn.metrics import log_loss\n", "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d7d59f0a-0026-8e33-6236-31637173734f" }, "source": [ "Now let us write a custom function to run the xgboost model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "af6e68af-f7a8-b0ac-c565-1d04818258f9" }, "outputs": [], "source": [ "def runXGB(train_X, train_y, test_X, test_y=None, feature_names=None, seed_val=0, num_rounds=1000):\n", " param = {}\n", " param['objective'] = 'multi:softprob'\n", " param['eta'] = 0.1\n", " param['max_depth'] = 6\n", " param['silent'] = 1\n", " param['num_class'] = 3\n", " param['eval_metric'] = \"mlogloss\"\n", " param['min_child_weight'] = 1\n", " param['subsample'] = 0.7\n", " param['colsample_bytree'] = 0.7\n", " param['seed'] = seed_val\n", " num_rounds = num_rounds\n", "\n", " plst = list(param.items())\n", " xgtrain = xgb.DMatrix(train_X, label=train_y)\n", "\n", " if test_y is not None:\n", " xgtest = xgb.DMatrix(test_X, label=test_y)\n", " watchlist = [ (xgtrain,'train'), (xgtest, 'test') ]\n", " model = xgb.train(plst, xgtrain, num_rounds, watchlist, early_stopping_rounds=20)\n", " else:\n", " xgtest = xgb.DMatrix(test_X)\n", " model = xgb.train(plst, xgtrain, num_rounds)\n", "\n", " pred_test_y = model.predict(xgtest)\n", " return pred_test_y, model" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c4a69cea-cb06-5d6a-83b7-16ee8ee241f6" }, "source": [ "Let us read the train and test files and store it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0108ce34-5e84-7f49-bd6f-6562d60a9082" }, "outputs": [], "source": [ "data_path = \"../input/\"\n", "train_file = data_path + \"train.json\"\n", "test_file = data_path + \"test.json\"\n", "train_df = pd.read_json(train_file)\n", "test_df = pd.read_json(test_file)\n", "print(train_df.shape)\n", "print(test_df.shape)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2bf65ce4-7375-c8e9-97d5-621736f3338d" }, "source": [ "We do not need any pre-processing for numerical features and so create a list with those features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6462885f-97de-b2d1-2c1a-1958115c4c4d" }, "outputs": [], "source": [ "features_to_use = [\"bathrooms\", \"bedrooms\", \"latitude\", \"longitude\", \"price\"]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b7670810-6d0b-89d0-629e-f99624421229" }, "source": [ "Now let us create some new features from the given features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e3b81db5-929d-b8b8-141c-1bbb4a5eaaf3" }, "outputs": [], "source": [ "# count of photos #\n", "train_df[\"num_photos\"] = train_df[\"photos\"].apply(len)\n", "test_df[\"num_photos\"] = test_df[\"photos\"].apply(len)\n", "\n", "# count of \"features\" #\n", "train_df[\"num_features\"] = train_df[\"features\"].apply(len)\n", "test_df[\"num_features\"] = test_df[\"features\"].apply(len)\n", "\n", "# count of words present in description column #\n", "train_df[\"num_description_words\"] = train_df[\"description\"].apply(lambda x: len(x.split(\" \")))\n", "test_df[\"num_description_words\"] = test_df[\"description\"].apply(lambda x: len(x.split(\" \")))\n", "\n", "# convert the created column to datetime object so as to extract more features \n", "train_df[\"created\"] = pd.to_datetime(train_df[\"created\"])\n", "test_df[\"created\"] = pd.to_datetime(test_df[\"created\"])\n", "\n", "# Let us extract some features like year, month, day, hour from date columns #\n", "train_df[\"created_year\"] = train_df[\"created\"].dt.year\n", "test_df[\"created_year\"] = test_df[\"created\"].dt.year\n", "train_df[\"created_month\"] = train_df[\"created\"].dt.month\n", "test_df[\"created_month\"] = test_df[\"created\"].dt.month\n", "train_df[\"created_day\"] = train_df[\"created\"].dt.day\n", "test_df[\"created_day\"] = test_df[\"created\"].dt.day\n", "train_df[\"created_hour\"] = train_df[\"created\"].dt.hour\n", "test_df[\"created_hour\"] = test_df[\"created\"].dt.hour\n", "\n", "# adding all these new features to use list #\n", "features_to_use.extend([\"num_photos\", \"num_features\", \"num_description_words\",\"created_year\", \"created_month\", \"created_day\", \"listing_id\", \"created_hour\"])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3d9aa966-66a2-8ff8-2459-40e0187418a2" }, "source": [ "We have 4 categorical features in our data\n", "\n", " - display_address\n", " - manager_id\n", " - building_id\n", " - listing_id\n", "\n", "So let us label encode these features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "af410ae2-6197-adce-ee68-360aa59eff7e" }, "outputs": [], "source": [ "categorical = [\"display_address\", \"manager_id\", \"building_id\", \"street_address\"]\n", "for f in categorical:\n", " if train_df[f].dtype=='object':\n", " #print(f)\n", " lbl = preprocessing.LabelEncoder()\n", " lbl.fit(list(train_df[f].values) + list(test_df[f].values))\n", " train_df[f] = lbl.transform(list(train_df[f].values))\n", " test_df[f] = lbl.transform(list(test_df[f].values))\n", " features_to_use.append(f)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3f550f0f-0c6f-2432-2c07-d507632eaa2b" }, "source": [ "We have features column which is a list of string values. So we can first combine all the strings together to get a single string and then apply count vectorizer on top of it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d1ea3504-a12c-023a-bce6-d4f93ddb8019" }, "outputs": [], "source": [ "train_df['features'] = train_df[\"features\"].apply(lambda x: \" \".join([\"_\".join(i.split(\" \")) for i in x]))\n", "test_df['features'] = test_df[\"features\"].apply(lambda x: \" \".join([\"_\".join(i.split(\" \")) for i in x]))\n", "print(train_df[\"features\"].head())\n", "tfidf = CountVectorizer(stop_words='english', max_features=200)\n", "tr_sparse = tfidf.fit_transform(train_df[\"features\"])\n", "te_sparse = tfidf.transform(test_df[\"features\"])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2bfbcacc-e821-654b-f2b3-cda0f1a5a20b" }, "source": [ "Now let us stack both the dense and sparse features into a single dataset and also get the target variable." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9eeef912-2104-e97e-1948-c246652340e1" }, "outputs": [], "source": [ "train_X = sparse.hstack([train_df[features_to_use], tr_sparse]).tocsr()\n", "test_X = sparse.hstack([test_df[features_to_use], te_sparse]).tocsr()\n", "\n", "target_num_map = {'high':0, 'medium':1, 'low':2}\n", "train_y = np.array(train_df['interest_level'].apply(lambda x: target_num_map[x]))\n", "print(train_X.shape, test_X.shape)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7d2e5fb7-7886-68b1-326f-6db491215001" }, "source": [ "Now let us do some cross validation to check the scores. \n", "\n", "Please run it in local to get the cv scores. I am commenting it out here for time." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "13fd60b9-a8b5-c76f-1fbd-2a56219da0d2" }, "outputs": [], "source": [ "cv_scores = []\n", "kf = model_selection.KFold(n_splits=5, shuffle=True, random_state=2016)\n", "for dev_index, val_index in kf.split(range(train_X.shape[0])):\n", " dev_X, val_X = train_X[dev_index,:], train_X[val_index,:]\n", " dev_y, val_y = train_y[dev_index], train_y[val_index]\n", " preds, model = runXGB(dev_X, dev_y, val_X, val_y)\n", " cv_scores.append(log_loss(val_y, preds))\n", " print(cv_scores)\n", " break" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5cff686f-2601-321d-8f81-5fa846ef7562" }, "source": [ "Now let us build the final model and get the predictions on the test set." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4fb1954d-e3f0-9369-d50c-bd1b615c0077" }, "outputs": [], "source": [ "preds, model = runXGB(train_X, train_y, test_X, num_rounds=400)\n", "out_df = pd.DataFrame(preds)\n", "out_df.columns = [\"high\", \"medium\", \"low\"]\n", "out_df[\"listing_id\"] = test_df.listing_id.values\n", "out_df.to_csv(\"xgb_starter2.csv\", index=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b23cc080-cd12-dc7d-0877-66806a34bf4c" }, "source": [ "\n", "Hope this helps the python users as a good starting point." ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }