{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9b7b4dd5-dfd6-4ad8-8595-006cd2cdc938" }, "outputs": [], "source": [ "# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2a37fbd8-6daf-4e1c-a035-b925172dff22" }, "source": [ "This notebook shows a \"most popular local hotel\" benchmark implemented with pandas.\n", "\n", "### Read the train data\n", "\n", "Read in the train data using only the necessary columns. \n", "Specifying dtypes helps reduce memory requirements. \n", "\n", "The file is read in chunks of 1 million rows each. In each chunk we count the number of rows and number of bookings for every destination-hotel cluster combination." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1fc865c3-b7eb-46c0-bc6a-586bcee7fdf0" }, "outputs": [], "source": [ "train = pd.read_csv('../input/train.csv',\n", " dtype={'is_booking':bool,'srch_destination_id':np.int32, 'hotel_cluster':np.int32},\n", " usecols=['srch_destination_id','is_booking','hotel_cluster'],\n", " chunksize=1000000)\n", "aggs = []\n", "print('-'*38)\n", "for chunk in train:\n", " agg = chunk.groupby(['srch_destination_id',\n", " 'hotel_cluster'])['is_booking'].agg(['sum','count'])\n", " agg.reset_index(inplace=True)\n", " aggs.append(agg)\n", " print('.',end='')\n", "print('')\n", "aggs = pd.concat(aggs, axis=0)\n", "aggs.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0d9e581b-f21e-459d-81d3-96603591e261" }, "source": [ "Next we aggregate again to compute the total number of bookings over all chunks. \n", "\n", "Compute the number of clicks by subtracting the number of bookings from total row counts.\n", "\n", "Compute the 'relevance' of a hotel cluster with a weighted sum of bookings and clicks." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "448e1da2-f29b-45b1-b06a-c1405517097a" }, "outputs": [], "source": [ "CLICK_WEIGHT = 0.05\n", "agg = aggs.groupby(['srch_destination_id','hotel_cluster']).sum().reset_index()\n", "agg['count'] -= agg['sum']\n", "agg = agg.rename(columns={'sum':'bookings','count':'clicks'})\n", "agg['relevance'] = agg['bookings'] + CLICK_WEIGHT * agg['clicks']\n", "agg.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d22117f0-aaae-4e1c-b26f-f33dd2f5aba1" }, "source": [ "### Find most popular hotel clusters by destination\n", "\n", "Define a function to get most popular hotels for a destination group.\n", "\n", "Previous version used nlargest() Series method to get indices of largest elements. \n", "But as @benjamin points out [in his fork](https://www.kaggle.com/benjaminabel/expedia-hotel-recommendations/pandas-version-of-most-popular-hotels/comments) the method is rather slow. \n", "I have updated this notebook with a version that runs faster." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c74c13bf-7102-45e3-895e-c6f633d89b25" }, "outputs": [], "source": [ "def most_popular(group, n_max=5):\n", " relevance = group['relevance'].values\n", " hotel_cluster = group['hotel_cluster'].values\n", " most_popular = hotel_cluster[np.argsort(relevance)[::-1]][:n_max]\n", " return np.array_str(most_popular)[1:-1] # remove square brackets" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "064e36a1-b8dd-4ad8-9dd9-2e9406a4cb29" }, "source": [ "Get most popular hotel clusters for all destinations." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "98f5c8e8-a831-405a-99c6-5166d709dc8b" }, "outputs": [], "source": [ "most_pop = agg.groupby(['srch_destination_id']).apply(most_popular)\n", "most_pop = pd.DataFrame(most_pop).rename(columns={0:'hotel_cluster'})\n", "most_pop.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5beb57c1-6934-43e8-a5c0-b29f9999fefe" }, "source": [ "### Predict for test data\n", "Read in the test data and merge most popular hotel clusters." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b492c927-79c0-4842-8f97-0f9c346a101b" }, "outputs": [], "source": [ "test = pd.read_csv('../input/test.csv',\n", " dtype={'srch_destination_id':np.int32},\n", " usecols=['srch_destination_id'],)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1ff5514b-06b5-492f-a902-e27cae14a725" }, "outputs": [], "source": [ "test = test.merge(most_pop, how='left',left_on='srch_destination_id',right_index=True)\n", "test.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cd1088b7-a11c-4693-adc0-023a49d92f93" }, "source": [ "Check hotel_cluster column in test for null values." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9f05e1ce-e9a4-4fa1-b215-4cb37cb79ba8" }, "outputs": [], "source": [ "test.hotel_cluster.isnull().sum()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9b51fa38-fe45-4c59-b20b-6b6557215624" }, "source": [ "Looks like there's about 14k new destinations in test. Let's fill nas with hotel clusters that are most popular overall." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a1c0ac98-668e-4156-abc0-c6705d9bfb25" }, "outputs": [], "source": [ "most_pop_all = agg.groupby('hotel_cluster')['relevance'].sum().nlargest(5).index\n", "most_pop_all = np.array_str(most_pop_all)[1:-1]\n", "most_pop_all" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "17c6ac9e-184e-41ce-a074-e9dd1ae9e9cd" }, "outputs": [], "source": [ "test.hotel_cluster.fillna(most_pop_all,inplace=True)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0409d4ac-0d8a-4c71-a00a-1dd8b78af3d0" }, "source": [ "Save the submission." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "168f31ce-af25-46c7-83b6-40d3981f9bf0" }, "outputs": [], "source": [ "test.hotel_cluster.to_csv('predicted_with_pandas.csv',header=True, index_label='id')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9e479427-49d3-4699-8124-b8a4f5afa839" }, "outputs": [], "source": [ "" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }