{ "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0, "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "6e711393-7a75-17e3-539c-9169c1ae1225", "_active": false }, "source": "It seems the current [high scoring script][1] is written in R using H2O. So let us do one in python using XGBoost. \n\nThanks to [this script][2] for feature engineering ideas. \n\nWe shall start with importing the necessary modules\n\n\n [1]: https://www.kaggle.com/gospursgo/two-sigma-connect-rental-listing-inquiries/h2o-starter-pack/run/835757\n [2]: https://www.kaggle.com/aikinogard/two-sigma-connect-rental-listing-inquiries/random-forest-starter-with-numerical-features", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1952347b-6dc9-b9f1-fa25-94587a2aee77", "_active": false }, "outputs": [], "source": "import os\nimport sys\nimport operator\nimport numpy as np\nimport pandas as pd\nfrom scipy import sparse\nimport xgboost as xgb\nfrom sklearn import model_selection, preprocessing, ensemble\nfrom sklearn.metrics import log_loss\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d7d59f0a-0026-8e33-6236-31637173734f", "_active": false }, "source": "Now let us write a custom function to run the xgboost model.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "af6e68af-f7a8-b0ac-c565-1d04818258f9", "_active": false }, "outputs": [], "source": "def runXGB(train_X, train_y, test_X, test_y=None, feature_names=None, seed_val=0, num_rounds=1000):\n param = {}\n param['objective'] = 'multi:softprob'\n param['eta'] = 0.1\n param['max_depth'] = 6\n param['silent'] = 1\n param['num_class'] = 3\n param['eval_metric'] = \"mlogloss\"\n param['min_child_weight'] = 1\n param['subsample'] = 0.7\n param['colsample_bytree'] = 0.7\n param['seed'] = seed_val\n num_rounds = num_rounds\n\n plst = list(param.items())\n xgtrain = xgb.DMatrix(train_X, label=train_y)\n\n if test_y is not None:\n xgtest = xgb.DMatrix(test_X, label=test_y)\n watchlist = [ (xgtrain,'train'), (xgtest, 'test') ]\n model = xgb.train(plst, xgtrain, num_rounds, watchlist, early_stopping_rounds=20)\n else:\n xgtest = xgb.DMatrix(test_X)\n model = xgb.train(plst, xgtrain, num_rounds)\n\n pred_test_y = model.predict(xgtest)\n return pred_test_y, model", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c4a69cea-cb06-5d6a-83b7-16ee8ee241f6", "_active": false }, "source": "Let us read the train and test files and store it.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0108ce34-5e84-7f49-bd6f-6562d60a9082", "_active": false }, "outputs": [], "source": "data_path = \"../input/\"\ntrain_file = data_path + \"train.json\"\ntest_file = data_path + \"test.json\"\ntrain_df = pd.read_json(train_file)\ntest_df = pd.read_json(test_file)\nprint(train_df.shape)\nprint(test_df.shape)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2bf65ce4-7375-c8e9-97d5-621736f3338d", "_active": false }, "source": "We do not need any pre-processing for numerical features and so create a list with those features.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6462885f-97de-b2d1-2c1a-1958115c4c4d", "_active": false, "collapsed": false }, "outputs": [], "source": "features_to_use = [\"bathrooms\", \"bedrooms\", \"latitude\", \"longitude\", \"price\"]\ntrain_df.columns", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b7670810-6d0b-89d0-629e-f99624421229", "_active": false }, "source": "Now let us create some new features from the given features.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e3b81db5-929d-b8b8-141c-1bbb4a5eaaf3", "_active": false, "collapsed": false }, "outputs": [], "source": "import datetime as dt\n# count of photos #\ntrain_df[\"num_photos\"] = train_df[\"photos\"].apply(len)\ntest_df[\"num_photos\"] = test_df[\"photos\"].apply(len)\n\n## price per bedroom\n## price per bathroom\ntrain_df[\"price_be\"] = train_df[\"price\"]/train_df[\"bedrooms\"]\ntrain_df[\"price_ba\"] = train_df[\"price\"]/train_df[\"bathrooms\"]\n\n# count of \"features\" #\ntrain_df[\"num_features\"] = train_df[\"features\"].apply(len)\ntest_df[\"num_features\"] = test_df[\"features\"].apply(len)\n\n# count of words present in description column #\ntrain_df[\"num_description_words\"] = train_df[\"description\"].apply(lambda x: len(x.split(\" \")))\ntest_df[\"num_description_words\"] = test_df[\"description\"].apply(lambda x: len(x.split(\" \")))\n\n# convert the created column to datetime object so as to extract more features \ntrain_df[\"created\"] = pd.to_datetime(train_df[\"created\"])\ntest_df[\"created\"] = pd.to_datetime(test_df[\"created\"])\n\n# Let us extract some features like year, month, day, hour from date columns #\ntrain_df[\"created_year\"] = train_df[\"created\"].dt.year\ntest_df[\"created_year\"] = test_df[\"created\"].dt.year\ntrain_df[\"created_month\"] = train_df[\"created\"].dt.month\ntest_df[\"created_month\"] = test_df[\"created\"].dt.month\ntrain_df[\"created_day\"] = train_df[\"created\"].dt.day\ntest_df[\"created_day\"] = test_df[\"created\"].dt.day\ntrain_df[\"created_hour\"] = train_df[\"created\"].dt.hour\ntest_df[\"created_hour\"] = test_df[\"created\"].dt.hour\ntrain_df['created_weekday'] = train_df['created'].dt.weekday\ntrain_df['created_week'] = train_df['created'].dt.week\ntrain_df['created_quarter'] = train_df['created'].dt.quarter\n\ntrain_df['created_weekend'] = ((train_df['created_weekday'] == 5) & (train_df['created_weekday'] == 6))\ntrain_df['created_wd'] = ((train_df['created_weekday'] != 5) & (train_df['created_weekday'] != 6))\ntrain_df['created'] = train_df['created'].map(lambda x: float((x - dt.datetime(1899, 12, 30)).days) + (float((x - dt.datetime(1899, 12, 30)).seconds) / 86400))\n\ntrain_df['x5'] = train_df['latitude'].map(lambda x : round(x,5))\ntrain_df['y5'] = train_df['longitude'].map(lambda x : round(x,5))\ntrain_df['x4'] = train_df['latitude'].map(lambda x : round(x,4))\ntrain_df['y4'] = train_df['longitude'].map(lambda x : round(x,4))\ntrain_df['x3'] = train_df['latitude'].map(lambda x : round(x,3))\ntrain_df['y3'] = train_df['longitude'].map(lambda x : round(x,3))\ntrain_df['x2'] = train_df['latitude'].map(lambda x : round(x,2))\ntrain_df['y2'] = train_df['longitude'].map(lambda x : round(x,2))\n\n# adding all these new features to use list #\nfeatures_to_use.extend([\"num_photos\", \"num_features\", \"num_description_words\",\"created_year\", \"created_month\", \"created_day\", \"listing_id\", \"created_hour\",'created_weekday'])\n#features_to_use.extend([\"created_week\",'created_quarter','created','y2','x2','y3','x3','y4','x4','y5','x5'])", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3d9aa966-66a2-8ff8-2459-40e0187418a2", "_active": false }, "source": "We have 4 categorical features in our data\n\n - display_address\n - manager_id\n - building_id\n - listing_id\n\nSo let us label encode these features.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "af410ae2-6197-adce-ee68-360aa59eff7e", "_active": true, "collapsed": false }, "outputs": [], "source": "categorical = [\"display_address\", \"manager_id\", \"building_id\", \"street_address\"]\nfor f in categorical:\n if train_df[f].dtype=='object':\n #print(f)\n lbl = preprocessing.LabelEncoder()\n lbl.fit(list(train_df[f].values) + list(test_df[f].values))\n train_df[f] = lbl.transform(list(train_df[f].values))\n test_df[f] = lbl.transform(list(test_df[f].values))\n features_to_use.append(f)", "execution_state": "idle" }, { "metadata": { "_cell_guid": "ea6fa1e4-b70e-a1e2-f182-ca7424766715", "_active": false, "collapsed": false }, "source": "#Our feature construction class will inherit from these two base classes of sklearn.\nfrom sklearn.base import BaseEstimator\nfrom sklearn.base import TransformerMixin\n\nclass manager_skill(BaseEstimator, TransformerMixin):\n \"\"\"\n Adds the column \"manager_skill\" to the dataset, based on the Kaggle kernel\n \"Improve Perfomances using Manager features\" by den3b. The function should\n be usable in scikit-learn pipelines.\n \n Parameters\n ----------\n threshold : Minimum count of rental listings a manager must have in order\n to get his \"own\" score, otherwise the mean is assigned.\n\n Attributes\n ----------\n mapping : pandas dataframe\n contains the manager_skill per manager id.\n \n mean_skill : float\n The mean skill of managers with at least as many listings as the \n threshold.\n \"\"\"\n def __init__(self, threshold = 5):\n \n self.threshold = threshold\n \n def _reset(self):\n \"\"\"Reset internal data-dependent state of the scaler, if necessary.\n \n __init__ parameters are not touched.\n \"\"\"\n # Checking one attribute is enough, becase they are all set together\n # in fit \n if hasattr(self, 'mapping_'):\n \n self.mapping_ = {}\n self.mean_skill_ = 0.0\n \n def fit(self, X,y):\n \"\"\"Compute the skill values per manager for later use.\n \n Parameters\n ----------\n X : pandas dataframe, shape [n_samples, n_features]\n The rental data. It has to contain a column named \"manager_id\".\n \n y : pandas series or numpy array, shape [n_samples]\n The corresponding target values with encoding:\n low: 0.0\n medium: 1.0\n high: 2.0\n \"\"\" \n self._reset()\n \n temp = pd.concat([X.manager_id,pd.get_dummies(y)], axis = 1).groupby('manager_id').mean()\n temp.columns = ['low_frac', 'medium_frac', 'high_frac']\n temp['count'] = X.groupby('manager_id').count().iloc[:,1]\n \n \n temp['manager_skill'] = temp['high_frac']*2 + temp['medium_frac']\n \n mean = temp.loc[temp['count'] >= self.threshold, 'manager_skill'].mean()\n \n temp.loc[temp['count'] < self.threshold, 'manager_skill'] = mean\n \n self.mapping_ = temp[['low_frac', 'medium_frac', 'high_frac','manager_skill']]\n self.mean_skill_ = mean\n \n return self\n \n def transform(self, X):\n \"\"\"Add manager skill to a new matrix.\n \n Parameters\n ----------\n X : pandas dataframe, shape [n_samples, n_features]\n Input data, has to contain \"manager_id\".\n \"\"\" \n X = pd.merge(left = X, right = self.mapping_, how = 'left', left_on = 'manager_id', right_index = True)\n X['manager_skill'].fillna(self.mean_skill_, inplace = True)\n \n return X", "execution_count": null, "cell_type": "code", "outputs": [], "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3f550f0f-0c6f-2432-2c07-d507632eaa2b", "_active": false }, "source": "We have features column which is a list of string values. So we can first combine all the strings together to get a single string and then apply count vectorizer on top of it.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "metadata": { "_cell_guid": "dbead36a-6247-c951-a20a-5a968eb552d0", "_active": false, "collapsed": false }, "source": "trans = manager_skill()\ntrain_df = trans.fit_transform(train_df, train_df['interest_level'])\ntest_df = trans.transform(test_df)\n#features_to_use.extend(['low_frac', 'medium_frac', 'high_frac','manager_skill','longitude', 'latitude',\"price_be\",'price_ba'])\nfeatures_to_use.extend(['low_frac','medium_frac','high_frac',\"price_be\",'price_ba'])", "execution_count": null, "cell_type": "code", "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d1ea3504-a12c-023a-bce6-d4f93ddb8019", "_active": false }, "outputs": [], "source": "train_df['features'] = train_df[\"features\"].apply(lambda x: \" \".join([\"_\".join(i.split(\" \")) for i in x]))\ntest_df['features'] = test_df[\"features\"].apply(lambda x: \" \".join([\"_\".join(i.split(\" \")) for i in x]))\nprint(train_df[\"features\"].head())\ntfidf = CountVectorizer(stop_words='english', max_features=200)\ntr_sparse = tfidf.fit_transform(train_df[\"features\"])\nte_sparse = tfidf.transform(test_df[\"features\"])", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2bfbcacc-e821-654b-f2b3-cda0f1a5a20b", "_active": false }, "source": "Now let us stack both the dense and sparse features into a single dataset and also get the target variable.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9eeef912-2104-e97e-1948-c246652340e1", "_active": false, "collapsed": false }, "outputs": [], "source": "train_X = sparse.hstack([train_df[features_to_use], tr_sparse]).tocsr()\n#test_X = sparse.hstack([test_df[features_to_use], te_sparse]).tocsr()\n\ntarget_num_map = {'high':0, 'medium':1, 'low':2}\ntrain_y = np.array(train_df['interest_level'].apply(lambda x: target_num_map[x]))\nprint(train_X.shape, test_X.shape)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7d2e5fb7-7886-68b1-326f-6db491215001", "_active": false }, "source": "Now let us do some cross validation to check the scores. \n\nPlease run it in local to get the cv scores. I am commenting it out here for time.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "13fd60b9-a8b5-c76f-1fbd-2a56219da0d2", "_active": false }, "outputs": [], "source": "cv_scores = []\nkf = model_selection.KFold(n_splits=5, shuffle=True, random_state=2016)\nfor dev_index, val_index in kf.split(range(train_X.shape[0])):\n dev_X, val_X = train_X[dev_index,:], train_X[val_index,:]\n dev_y, val_y = train_y[dev_index], train_y[val_index]\n preds, model = runXGB(dev_X, dev_y, val_X, val_y)\n cv_scores.append(log_loss(val_y, preds))\n print(cv_scores)\n break", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5cff686f-2601-321d-8f81-5fa846ef7562", "_active": false }, "source": "Now let us build the final model and get the predictions on the test set.", "execution_count": null, "outputs": [], "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4fb1954d-e3f0-9369-d50c-bd1b615c0077", "_active": false, "collapsed": false }, "outputs": [], "source": "#preds, model = runXGB(train_X, train_y, test_X, num_rounds=400)\n#out_df = pd.DataFrame(preds)\n#out_df.columns = [\"high\", \"medium\", \"low\"]\n#out_df[\"listing_id\"] = test_df.listing_id.values\n#out_df.to_csv(\"xgb_starter2.csv\", index=False)\ntrain_df[features_to_use].info()\n#train_df['created_weekend']", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b23cc080-cd12-dc7d-0877-66806a34bf4c", "_active": false }, "source": "\nHope this helps the python users as a good starting point.", "execution_count": null, "outputs": [], "execution_state": "idle" } ] }