{"metadata": {"language_info": {"nbconvert_exporter": "python", "version": "3.6.3", "mimetype": "text/x-python", "file_extension": ".py", "codemirror_mode": {"version": 3, "name": "ipython"}, "pygments_lexer": "ipython3", "name": "python"}, "kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}}, "nbformat": 4, "cells": [{"metadata": {"_cell_guid": "d2e03188-1a16-4fb7-a82b-d18427cdec29", "_uuid": "dfd7dcc16920ec9d2be92c197c3d3b069da28b48", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import os\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import pickle\n", "import sklearn\n", "import sklearn.metrics\n", "\n", "from sklearn.decomposition import PCA\n", "from sklearn.preprocessing import label_binarize\n", "\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "# from subprocess import check_output\n", "# print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output."]}, {"metadata": {"_cell_guid": "98935588-b9c4-44ad-8ce0-5160108a6ee4", "_uuid": "e632654e0fb8785fb72762238cb3a112ec5032d0"}, "cell_type": "markdown", "source": ["### Abstract\n", "\n", "In this Jupyter Notebook we will be demonstrating the lifecycle of the Data Science & Analysis pipeline through a Machine Learning example **(TODO: _TYPES OF ML - RANDOM FOREST, BAYES, SVM, ETC_)**. The pipeline can be broken down into five core components, each of which will be covered in this report. [The dataset we will be using is from a retired competition on Kaggle.com, courteously provided by Expedia.](https://www.kaggle.com/c/expedia-hotel-recommendations) Here is the challenge we are tasked with:\n", "            \n", "_Expedia is a proprietary Search Engine for Hotel & Travel bookings. Everytime a user initiates a search session, data is kept about the user's search. Data is also provided from Expedia's in-house hotel clustering algorithm. This data groups similar hotels into 100 different clusters, which simplifies our the machine learning portion of our task into a classification problem. Thus, given all this data, design a classifier to predict which hotel cluster a user will end up booking a hotel from, based on the user's search patterns._"]}, {"metadata": {"_cell_guid": "5c5171ad-b441-450c-b6a1-a7f41071e3b0", "_uuid": "00c272511ab4cca44ee2a7fc4ccd1f2169177e38"}, "cell_type": "markdown", "source": ["### Data Collection\n", "\n", "- Retrieve the test, training, and destinations datasets from [Kaggle](https://www.kaggle.com/c/expedia-hotel-recommendations/data).\n", "    - test.csv - Test data that we will evaluate our classifiers with. Contains n = 2528243 observations, but we we're able to load all the data without sampling it (unlike the training dataset).\n", "    - train.csv - Data about searches for hotels by users. These data are our feature vectors, complete with hotel cluster classification (labeling). Contains n = 37670293 observations. We'll need to sample these data down to a managable size.\n", "    - destinations.csv - Hotel feature data. Contains 149 anonymized features for n = 62106 hotels.\n", " \n", "\n"]}, {"metadata": {"_cell_guid": "85553e46-7e6c-48c2-8512-e1aab2148878", "_uuid": "3fd8dbc93435173b49b958f6f3a8438ac6f88afb", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# reads chunks of data from test.csv and makes a DataFrame for the test set\n", "\n", "test_df = pd.DataFrame()\n", "chunks = [test_df]\n", "\n", "for chunk in pd.read_csv('../input/test.csv', sep=',', chunksize=1e6):\n", "    chunks += [chunk]\n", "\n", "test_df = pd.concat(chunks)\n", "test_df.head()"]}, {"metadata": {"_cell_guid": "d1bcf8e9-89de-4de7-b3c0-07af219b558d", "_uuid": "101d4510911698daa696d06365b2c2e54730ec25", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# summary statistics for test data set\n", "\n", "test_df.describe()"]}, {"metadata": {"_cell_guid": "14aaee4c-d21b-49f3-92cf-ed33b339a10a", "_uuid": "26cec1653442ad03716544ade1701bcaf0be587a", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["train_df = pd.DataFrame()\n", "chunks = [train_df]\n", "\n", "sample_size = int(1e5)\n", "bin_num = 38\n", "bin_sample_size = int(sample_size // bin_num)\n", "\n", "for chunk in pd.read_csv('../input/train.csv', sep=',', chunksize=1e6):\n", "    if sample_size - (2 * bin_sample_size) < 0:\n", "        temp = chunk.sample(sample_size)\n", "    else:\n", "        temp = chunk.sample(bin_sample_size)\n", "        \n", "    print(len(chunks))\n", "    \n", "    sample_size -= bin_sample_size\n", "    chunks += [temp]\n", "\n", "\n", "train_df = pd.concat(chunks)\n", "train_df.head()"]}, {"metadata": {"_cell_guid": "1981d562-daca-4445-a580-4970dff58a54", "_uuid": "c2ddbc67fa8e779bd12354da0b07cc35cb1a2d1b", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["train_df.describe()"]}, {"metadata": {"_cell_guid": "b2d58ce6-b38d-4d47-8cd5-be289fe8285e", "_uuid": "3b30af97bf56ddabf95acc181126e8fdf574eb70", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "1866051b-e79a-4eec-8752-e315b70386d3", "_uuid": "7cfddde3d57d1d239d8149da9856856e0db64b27", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "959354ad-7b23-4d67-af40-c8c89de9b57e", "_uuid": "b2292b4fc78d56562b1b24d9bdcb02c83a8d7cd5", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["dest_df = pd.DataFrame()\n", "\n", "for chunk in pd.read_csv('../input/destinations.csv', sep=',', chunksize=1e6):\n", "    dest_df = pd.concat([dest_df, chunk])\n", "\n", "dest_df.describe()"]}, {"metadata": {"_cell_guid": "88e82c33-ba0e-4b71-a7be-1d4740c5318f", "_uuid": "0bee41318c00cd318a78169c05bf40cb711693f9", "collapsed": true}, "cell_type": "markdown", "source": ["### Explore the Data\n", "\n", "**TODO:**\n", "+ _Randomly sample from train multiple times_\n", "+ _Sample training and testing from original training_\n", "+ _Stuff about data & attributes, some graphs_\n", "+ _Convert date to date time, compare test & train dates/split by date_\n", "+ _Compare user IDs in test & train, only use user ids in train that are in test_\n", "+ _Benchmark our classifiers with premade packages_"]}, {"metadata": {"_cell_guid": "2259e21c-4944-45cd-8051-73ad9c651b71", "_uuid": "8b142de2db46751858315172c2fffeb5fd7df7cb", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# frequency count of 5 most popular hotel clusters from training sample\n", "\n", "freq_df = pd.DataFrame(train_df['hotel_cluster'].copy())\n", "freq_df.columns = ['actual']\n", "\n", "\n", "freq_predictions = train_df['hotel_cluster'].value_counts().head().index.tolist()\n", "freq_df['freq_predict'] = [freq_predictions[0] for i in range(freq_df.shape[0])]\n", "temp = [freq_predictions[0] for i in range(freq_df.shape[0])]\n", "freq_df.head()\n"]}, {"metadata": {"_cell_guid": "30a664ff-fc48-41dd-9997-71cc755c6d78", "_uuid": "5adcd4bd5a5a0f4d2f8498c2641677b647f240ee", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["y = freq_df['actual'].values\n", "y = label_binarize(y, classes=list(range(0,100)))\n", "x = label_binarize(temp, classes=list(range(0,100)))\n", "\n", "sklearn.metrics.average_precision_score(y, x)"]}, {"metadata": {"_cell_guid": "698f3435-0de2-44b7-9745-af53f19a8477", "_uuid": "9576e77a970776b67d3e057bfc060a57b16331e7"}, "cell_type": "markdown", "source": ["#### Linear Regression\n", "\n", "- _Check r coeff to see if any variables have interactions_"]}, {"metadata": {"_cell_guid": "09d4a75a-db61-4816-9b79-c397e8c5add1", "_uuid": "b553bee7b73d0034156bde7b8b0c353f84514a3d", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["# check Pearson's Correlation Coefficient (r) values for every search feature against each other\n", "\n", "rcorrs = train_df.corr()\n", "\n", "for i, r in rcorrs.iterrows():\n", "    for j in range(rcorrs.shape[1]):\n", "        if np.abs(r[j]) < 0.4 or np.abs(r[j]) == 1:\n", "            r[j] = np.NaN\n", "    \n", "rcorrs"]}, {"metadata": {"_cell_guid": "9d517b61-7145-4dfd-8e23-c6faeb51c80c", "_uuid": "c2ab49e6ea46a031089909999a8f11d76376f9fa"}, "cell_type": "markdown", "source": ["#### Destinations - Principal Component Analysis"]}, {"metadata": {"_cell_guid": "9a4e8a4f-f05a-444d-bff3-bfb55770b2b6", "_uuid": "a4880ac2061d2d731e8824deed1a86e5c988e621", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["dest_pca = PCA(n_component=5)\n"]}, {"metadata": {"_cell_guid": "41f000e6-ab32-4777-8993-2167e9dd30ca", "_uuid": "9df16486bbcc9c871f748918e2c4f15e8dc8cd82", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": ["p = list(range(1,100))\n", "print(p)"]}, {"metadata": {"_cell_guid": "e65f4d63-c2de-4118-909c-b7701dacc2d2", "_uuid": "93d913ec391bdb42ded82ce8e546100c93b07111", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "0359c12c-a3da-4872-9799-9da9838dcbe1", "_uuid": "370c43a96655f7cd4c8e24d2f0419361a8520dcc", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "316f3ead-9d6c-4778-af72-dcaa682bc096", "_uuid": "5681e514a7ee3e041040aed0bfd317082690678e", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "1c3226a5-e48c-47dd-a8d2-e4fe884bee5a", "_uuid": "50f44141ac5a01d93cd01d30943ba6b3e993633c", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "efc1465e-5469-4286-a011-5d1a3a9d3c54", "_uuid": "e6f1fc832f80e29f683df3409ed56173ecd10eea", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}, {"metadata": {"_cell_guid": "3aa811f0-8111-43bd-90e1-9a53815fa079", "_uuid": "a5143e52b49935a9584949eeb7bde33a98aacdc5", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code", "source": []}], "nbformat_minor": 1}