{"cells":[
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\n\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "This notebook shows a \"most popular local hotel\" benchmark implemented with pandas.\n\n### Read the train data\n\nRead in the train data using only the necessary columns. \nSpecifying dtypes helps reduce memory requirements. \n\nThe file is read in chunks of 1 million rows each. In each chunk we count the number of rows and number of bookings for every destination-hotel cluster combination."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": ""
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "### Find most popular hotel clusters by destination\n\nDefine a function to get most popular hotels for a destination group.\n\nPrevious version used nlargest() Series method to get indices of largest elements. \nBut as @benjamin points out [in his fork](https://www.kaggle.com/benjaminabel/expedia-hotel-recommendations/pandas-version-of-most-popular-hotels/comments) the method is rather slow. \nI have updated this notebook with a version that runs faster."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "def most_popular(group, n_max=5):\n    relevance = group['relevance'].values\n    hotel_cluster = group['hotel_cluster'].values\n    most_popular = hotel_cluster[np.argsort(relevance)[::-1]][:n_max]\n    return np.array_str(most_popular)[1:-1] # remove square brackets"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Get most popular hotel clusters for all destinations."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "most_pop = agg.groupby(['srch_destination_id']).apply(most_popular)\nmost_pop = pd.DataFrame(most_pop).rename(columns={0:'hotel_cluster'})\nmost_pop.head()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "### Predict for test data\nRead in the test data and merge most popular hotel clusters."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "test = pd.read_csv('../input/test.csv',\n                    dtype={'srch_destination_id':np.int32},\n                    usecols=['srch_destination_id'],)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "test = test.merge(most_pop, how='left',left_on='srch_destination_id',right_index=True)\ntest.head()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Check hotel_cluster column in test for null values."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "test.hotel_cluster.isnull().sum()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Looks like there's about 14k new destinations in test. Let's fill nas with hotel clusters that are most popular overall."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "most_pop_all = agg.groupby('hotel_cluster')['relevance'].sum().nlargest(5).index\nmost_pop_all = np.array_str(most_pop_all)[1:-1]\nmost_pop_all"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "test.hotel_cluster.fillna(most_pop_all,inplace=True)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Save the submission."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "test.hotel_cluster.to_csv('predicted_with_pandas.csv',header=True, index_label='id')"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}