{"cells":[{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"86de4aee-cc16-4ec3-bd22-1d42d7005c48"},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output."},{"cell_type":"markdown","metadata":{"_cell_guid":"dda8504e-83e3-4673-9ad7-f47ab4363794"},"source":"This notebook shows a \"most popular local hotel\" benchmark implemented with pandas.\n\n### Read the train data\n\nRead in the train data using only the necessary columns. \nSpecifying dtypes helps reduce memory requirements. \n\nThe file is read in chunks of 1 million rows each. In each chunk we count the number of rows and number of bookings for every destination-hotel cluster combination."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3f5c80d8-ed1b-456d-a2cb-7d9ee174a987"},"outputs":[],"source":"train = pd.read_csv('../input/train.csv',\n                    dtype={'is_booking':bool,'srch_destination_id':np.int32, 'hotel_cluster':np.int32},\n                    usecols=['srch_destination_id','is_booking','hotel_cluster'],\n                    chunksize=1000000)\naggs = []\nprint('-'*38)\nfor chunk in train:\n    agg = chunk.groupby(['srch_destination_id',\n                         'hotel_cluster'])['is_booking'].agg(['sum','count'])\n    agg.reset_index(inplace=True)\n    aggs.append(agg)\n    print('.',end='')\nprint('')\naggs = pd.concat(aggs, axis=0)\naggs.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"7ef73cec-6f5e-45af-90e7-0dc0bce1b46d"},"source":"Next we aggregate again to compute the total number of bookings over all chunks. \n\nCompute the number of clicks by subtracting the number of bookings from total row counts.\n\nCompute the 'relevance' of a hotel cluster with a weighted sum of bookings and clicks."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8f50e5f7-b45b-4e05-b399-e5c61b977d29"},"outputs":[],"source":"CLICK_WEIGHT = 0.05\nagg = aggs.groupby(['srch_destination_id','hotel_cluster']).sum().reset_index()\nagg['count'] -= agg['sum']\nagg = agg.rename(columns={'sum':'bookings','count':'clicks'})\nagg['relevance'] = agg['bookings'] + CLICK_WEIGHT * agg['clicks']\nagg.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"07871501-0058-4d59-a261-9ffdf0c4daeb"},"source":"### Find most popular hotel clusters by destination\n\nDefine a function to get most popular hotels for a destination group.\n\nPrevious version used nlargest() Series method to get indices of largest elements. \nBut as @benjamin points out [in his fork](https://www.kaggle.com/benjaminabel/expedia-hotel-recommendations/pandas-version-of-most-popular-hotels/comments) the method is rather slow. \nI have updated this notebook with a version that runs faster."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"684e4b26-e795-4e97-8fcc-47013d9db860"},"outputs":[],"source":"def most_popular(group, n_max=5):\n    relevance = group['relevance'].values\n    hotel_cluster = group['hotel_cluster'].values\n    most_popular = hotel_cluster[np.argsort(relevance)[::-1]][:n_max]\n    return np.array_str(most_popular)[1:-1] # remove square brackets"},{"cell_type":"markdown","metadata":{"_cell_guid":"b9c6c68c-28b6-4efa-b21c-258e74519850"},"source":"Get most popular hotel clusters for all destinations."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8757fd90-470c-47ac-b132-cdfbc1418877"},"outputs":[],"source":"most_pop = agg.groupby(['srch_destination_id']).apply(most_popular)\nmost_pop = pd.DataFrame(most_pop).rename(columns={0:'hotel_cluster'})\nmost_pop.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"ac8d62cd-4e74-441f-9138-ce43af0b6ece"},"source":"### Predict for test data\nRead in the test data and merge most popular hotel clusters."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"120e91f6-af46-4f53-8b08-3c8c361c7fe0"},"outputs":[],"source":"test = pd.read_csv('../input/test.csv',\n                    dtype={'srch_destination_id':np.int32},\n                    usecols=['srch_destination_id'],)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"51041753-bce1-4077-b529-dc39a22f8735"},"outputs":[],"source":"test = test.merge(most_pop, how='left',left_on='srch_destination_id',right_index=True)\ntest.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"da526290-5fb5-4dd7-bd31-ca67fab46bd5"},"source":"Check hotel_cluster column in test for null values."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5d03c599-bcc9-4f84-a12f-6cb8459fc75e"},"outputs":[],"source":"test.hotel_cluster.isnull().sum()"},{"cell_type":"markdown","metadata":{"_cell_guid":"9fe66ac1-1dd0-499c-a4a5-b93ab3fbb95f"},"source":"Looks like there's about 14k new destinations in test. Let's fill nas with hotel clusters that are most popular overall."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"827078e3-2b8b-448b-b1e7-f8b400709d8d"},"outputs":[],"source":"most_pop_all = agg.groupby('hotel_cluster')['relevance'].sum().nlargest(5).index\nmost_pop_all = np.array_str(most_pop_all)[1:-1]\nmost_pop_all"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"656b0419-be7d-4876-b60d-38e988d4adc1"},"outputs":[],"source":"test.hotel_cluster.fillna(most_pop_all,inplace=True)"},{"cell_type":"markdown","metadata":{"_cell_guid":"b0fac186-475d-41fb-a005-ac6ae6e014e0"},"source":"Save the submission."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0a8a6fa1-fa16-4a36-9673-d9176630a786"},"outputs":[],"source":"test.hotel_cluster.to_csv('predicted_with_pandas.csv',header=True, index_label='id')"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e30c3111-ac70-49c8-902a-153a9f65ce3f"},"outputs":[],"source":""}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}