{"nbformat": 4, "nbformat_minor": 1, "metadata": {"kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}, "language_info": {"name": "python", "nbconvert_exporter": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "version": "3.6.3", "mimetype": "text/x-python", "pygments_lexer": "ipython3", "file_extension": ".py"}}, "cells": [{"source": ["# !pip install tensorflow"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {}}, {"source": ["# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import os\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import pickle\n", "import sklearn\n", "import sklearn.metrics\n", "import tensorflow as tf\n", "\n", "from sklearn.decomposition import PCA\n", "from sklearn.preprocessing import label_binarize\n", "from sklearn.preprocessing import OneHotEncoder\n", "from sklearn.neighbors import KNeighborsClassifier as KNN\n", "\n", "\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "# from subprocess import check_output\n", "# print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output."], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "dfd7dcc16920ec9d2be92c197c3d3b069da28b48", "_cell_guid": "d2e03188-1a16-4fb7-a82b-d18427cdec29"}}, {"cell_type": "markdown", "source": ["### Abstract\n", "\n", "In this Jupyter Notebook we will be demonstrating the lifecycle of the Data Science & Analysis pipeline through a Machine Learning example **(TODO: _TYPES OF ML - RANDOM FOREST, BAYES, SVM, ETC_)**. The pipeline can be broken down into five core components, each of which will be covered in this report. [The dataset we will be using is from a retired competition on Kaggle.com, courteously provided by Expedia.](https://www.kaggle.com/c/expedia-hotel-recommendations) Here is the challenge we are tasked with:\n", "            \n", "_Expedia is a proprietary Search Engine for Hotel & Travel bookings. Everytime a user initiates a search session, data is kept about the user's search. Data is also provided from Expedia's in-house hotel clustering algorithm. This data groups similar hotels into 100 different clusters, which simplifies our the machine learning portion of our task into a classification problem. Thus, given all this data, design a classifier to predict which hotel cluster a user will end up booking a hotel from, based on the user's search patterns._"], "metadata": {"_uuid": "e632654e0fb8785fb72762238cb3a112ec5032d0", "_cell_guid": "98935588-b9c4-44ad-8ce0-5160108a6ee4"}}, {"cell_type": "markdown", "source": ["### Data Collection\n", "\n", "- Retrieve the test, training, and destinations datasets from [Kaggle](https://www.kaggle.com/c/expedia-hotel-recommendations/data).\n", "    - test.csv - Test data that we will evaluate our classifiers with. Contains n = 2528243 observations, but we we're able to load all the data without sampling it (unlike the training dataset).\n", "    - train.csv - Data about searches for hotels by users. These data are our feature vectors, complete with hotel cluster classification (labeling). Contains n = 37670293 observations. We'll need to sample these data down to a managable size.\n", "    - destinations.csv - Hotel feature data. Contains 149 anonymized features for n = 62106 hotels.\n", " \n", "\n"], "metadata": {"_uuid": "df94c794a209117ddc1ea305727fc5b4b0749ee7", "_cell_guid": "9a3b0e8b-5f98-4566-a4a3-c482edfeab4f"}}, {"source": ["# reads chunks of data from test.csv and makes a DataFrame for the test set\n", "\n", "test_df = pd.DataFrame()\n", "chunks = [test_df]\n", "\n", "for chunk in pd.read_csv('../input/test.csv', sep=',', chunksize=1e6):\n", "    chunks += [chunk]\n", "\n", "test_df = pd.concat(chunks)\n", "test_df.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "3fd8dbc93435173b49b958f6f3a8438ac6f88afb", "_cell_guid": "85553e46-7e6c-48c2-8512-e1aab2148878"}}, {"source": ["# summary statistics for test data set\n", "\n", "test_df.describe()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "3ed9caf2435fe7de41047c2872221a9ac290858a", "_cell_guid": "a27dfbb9-1889-434a-a3c2-a687fb8356a6"}}, {"cell_type": "markdown", "source": ["## **TODO**\n", "- k-fold cross validation\n", "- repeated sampling"], "metadata": {"_uuid": "ed5e23bcbdd376ad01a871bdee036df99477d801", "_cell_guid": "a3a7065b-d652-4fc9-ab38-116fcb071574"}}, {"source": ["train_df = pd.DataFrame()\n", "chunks = [train_df]\n", "\n", "sample_size = int(1e5)\n", "bin_num = 38\n", "bin_sample_size = int(sample_size // bin_num)\n", "\n", "for chunk in pd.read_csv('../input/train.csv', sep=',', chunksize=1e6):\n", "    if sample_size - (2 * bin_sample_size) < 0:\n", "#         temp = chunk.loc[chunk['is_booking'] == 1].sample(sample_size)\n", "        temp = chunk.sample(sample_size)\n", "    else:\n", "#         temp = chunk.loc[chunk['is_booking'] == 1].sample(bin_sample_size)\n", "        temp = chunk.sample(bin_sample_size)\n", "        \n", "    print(len(chunks))\n", "    \n", "    sample_size -= bin_sample_size\n", "    chunks += [temp]\n", "\n", "\n", "train_df = pd.concat(chunks)\n", "train_df.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "6514f7c4ef1a69d5d3e4c8757619610ed552010a", "_cell_guid": "af6a32f0-75c7-49db-905f-dc60da7efaf8"}}, {"source": ["train_df.describe()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "7b620963b3c6a12754a9c213fd4e5baf35e1814b", "_cell_guid": "b473bd64-b2a8-48dd-a2df-4bd6d1119746"}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "4e1b90f3e219c40d2ca599f5a03aa0c19b3cfffb", "_cell_guid": "585a25f6-3fe6-434f-b21b-218876ab385d"}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "84757d4c438812afde4c936134313f9493ed9b8e", "_cell_guid": "5f424dcb-1e85-430e-a531-aeb2a8d5765a"}}, {"source": ["dest_df = pd.DataFrame()\n", "\n", "for chunk in pd.read_csv('../input/destinations.csv', sep=',', chunksize=1e6):\n", "    dest_df = pd.concat([dest_df, chunk])\n", "\n", "dest_df.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "b2292b4fc78d56562b1b24d9bdcb02c83a8d7cd5", "_cell_guid": "959354ad-7b23-4d67-af40-c8c89de9b57e"}}, {"source": ["dest_df.describe()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "231a0d56cfabba7817f7de9ab2bd7e8f55fd230b", "_cell_guid": "9e535ea2-11ee-4cc6-b741-7189f2d28910"}}, {"cell_type": "markdown", "source": ["### Explore the Data\n", "\n", "**TODO:**\n", "+ _Randomly sample from train multiple times_\n", "+ _Sample training and testing from original training_\n", "+ _Stuff about data & attributes, some graphs_\n", "+ _Convert date to date time, compare test & train dates/split by date_\n", "+ _Compare user IDs in test & train, only use user ids in train that are in test_\n", "+ _Benchmark our classifiers with premade packages_"], "metadata": {"collapsed": true, "_uuid": "0bee41318c00cd318a78169c05bf40cb711693f9", "_cell_guid": "88e82c33-ba0e-4b71-a7be-1d4740c5318f"}}, {"source": ["# frequency count of 5 most popular hotel clusters from training sample\n", "\n", "freq_df = pd.DataFrame(train_df['hotel_cluster'].copy())\n", "freq_df.columns = ['actual']\n", "\n", "\n", "freq_predictions = train_df['hotel_cluster'].value_counts().head().index.tolist()\n", "freq_df['freq_predict'] = [freq_predictions[0] for i in range(freq_df.shape[0])]\n", "temp = [freq_predictions[0] for i in range(freq_df.shape[0])]\n", "freq_df.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "c80bc34d0c5d382d7b37675b60ddca146f6b1a4e", "_cell_guid": "132e5ae2-2cb4-41f5-8874-0d1210f97eff"}}, {"source": ["y = freq_df['actual'].values\n", "y = label_binarize(y, classes=list(range(0,100)))\n", "x = label_binarize(temp, classes=list(range(0,100)))\n", "\n", "sklearn.metrics.average_precision_score(y, x)"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "aa4a75be476b1afaee6d0a4d85d265ccdd50c8d5", "_cell_guid": "cf6b8f31-5910-482d-9630-d42b4630ed1e"}}, {"cell_type": "markdown", "source": ["#### Linear Regression\n", "\n", "- _Check r coeff to see if any variables have interactions_"], "metadata": {"_uuid": "3639590df842e3fc388aab12217fe5c6762ce5c3", "_cell_guid": "2ae63177-91a5-4999-b2fe-07c8cc349333"}}, {"source": ["cats = train_df[['site_name', 'posa_continent', 'user_location_country', 'user_location_region', 'user_location_city', 'user_id', 'is_mobile', 'is_package', 'channel', 'srch_destination_id', 'srch_destination_type_id', 'hotel_continent', 'hotel_country', 'hotel_market', 'is_booking', 'cnt', 'hotel_cluster']]\n", "cats.corr(method='spearman')"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "a0a9f7b29b44b69b2c5249b48eba97ce3382fd7d", "_cell_guid": "9bc85a58-68e7-40e3-a11a-8d61db6e96b3"}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "489e57e17b4bfe7beeabcc3f6942943b0f0da6bf", "_cell_guid": "64c050a8-f274-4afd-98d2-6d0cb5b98df1"}}, {"source": ["# check Pearson's Correlation Coefficient (r) values for every search feature against each other\n", "\n", "rcorrs = train_df.corr()\n", "\n", "for i, r in rcorrs.iterrows():\n", "    for j in range(rcorrs.shape[1]):\n", "        if np.abs(r[j]) < 0.4 or np.abs(r[j]) == 1:\n", "            pass\n", "#             r[j] = np.NaN\n", "    \n", "rcorrs"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "92f1f0a3c1a7fddeb4b302231cf13c66283c1f92", "_cell_guid": "9e5922ea-ddb7-4c06-b439-987e57a28802"}}, {"cell_type": "markdown", "source": ["#### Destinations - Principal Component Analysis"], "metadata": {"_uuid": "2b0958ec9ebf00ac192f626e993d7767fb6f61d9", "_cell_guid": "3333fe36-1e47-4953-9d9d-c22a16157ae8"}}, {"source": ["pca = PCA(n_components=5)\n", "# dest_pca = dest_pca.fit_transform(dest_df.iloc[:,1:])\n", "dest_pca = pca.fit_transform(dest_df.iloc[:,1:])\n", "dest_pca = pd.DataFrame(dest_pca)\n", "dest_pca.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "9f77f702955336456b31d3e773e83eb840063334", "_cell_guid": "e135859e-d67f-493a-8b31-b32f8ca13563"}}, {"source": ["enc = OneHotEncoder()\n", "\n", "temp = train_df[['hotel_continent', 'site_name', 'srch_rm_cnt', 'srch_adults_cnt', 'srch_children_cnt', 'srch_destination_id', 'cnt', 'hotel_cluster']].copy().dropna()\n", "onehot = pd.get_dummies(temp[['hotel_continent', 'site_name', 'srch_rm_cnt', 'srch_adults_cnt', 'srch_children_cnt', 'srch_destination_id', 'cnt']])\n", "temp = temp.drop(['hotel_continent', 'site_name', 'srch_rm_cnt', 'srch_adults_cnt', 'srch_children_cnt', 'srch_destination_id', 'cnt'], axis=1)\n", "temp = temp.join(onehot)\n", "# dist = enc.fit([temp['orig_destination_distance']])"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "c3a8e5dc6426321842de29b2949b9db2c2bcf3e9", "_cell_guid": "fe1d6658-e3c4-461d-81fe-d6e2f56826e6"}}, {"source": ["# print(dist)\n", "# print(temp['hotel_cluster'].shape)\n", "print(temp.head())\n", "print(onehot.head())\n", "print(temp.columns)\n", "print(temp.shape)\n"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "6dd38dd04fe3d55837db98565420e380390fa893", "_cell_guid": "63e893c0-bd92-43e4-b059-ee14f1606248"}}, {"source": ["# ['site_name', 'user_location_country', 'orig_destination_distance', 'srch_adults_cnt', 'srch_rm_cnt', 'srch_destination_id', 'hotel_continent', 'hotel_country']\n", "neighbors = KNN()\n", "neighbors.fit(temp[['hotel_continent', 'site_name', 'srch_rm_cnt', 'srch_adults_cnt', 'srch_children_cnt', 'srch_destination_id', 'cnt']], train_df['hotel_cluster'])\n", "# neighbors.fit(train_df.iloc[:,:24], train_df.iloc[:,23:])"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "1b001c77cd5bb71c48ef4cb42acebb4ce65c9b24", "_cell_guid": "8f6741f2-a958-4f2e-80c3-c6a23725571a"}}, {"source": ["# # dest_df.iloc[:,1:].shape\n", "# # dest_df[[\"d{0}\".format(i + 1) for i in range(149)]].shape\n", "\n", "\n", "# # onehot = pd.get_dummies(train_df['hotel_cluster'])\n", "# temp[['hotel_continent', 'site_name', 'srch_rm_cnt', 'srch_adults_cnt', 'srch_children_cnt', 'srch_destination_id', 'cnt']].head()\n"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "8bd3e29048b403734e22648763f53f5d4af27467", "_cell_guid": "04049a2a-87ff-4422-875d-81ca893899dc"}}, {"source": ["# neighbors.predict([[0, 0, 0, 0, 0, 1]])\n", "\n", "temp_test = train_df[['hotel_continent', 'site_name', 'srch_rm_cnt', 'srch_adults_cnt', 'srch_children_cnt', 'srch_destination_id', 'cnt']].sample(temp.shape[0])\n", "print(temp_test.shape)\n", "print(temp.shape)\n", "\n", "predictions = neighbors.predict(temp_test)"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "1436b69260000eb8019fa55e02f159e7095fafda", "_cell_guid": "13bfa271-2b3c-4c33-98ad-7fb7a121b59a"}}, {"source": ["# from sklearn.multioutput import MultiOutputClassifier\n", "# from sklearn.utils import shuffle"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true}}, {"source": ["print(predictions[:5])\n", "temp_test.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "3b8dc219c46f4d0fa5441eb52165d45de7f4e8a8", "_cell_guid": "0ae86698-c03e-43d2-b2de-a1be008f9f57"}}, {"source": ["temp = train_df[['hotel_continent', 'hotel_cluster']]\n", "tempshape = temp.shape\n", "print(temp.shape)\n", "# print(temp['hotel_continent'].dtype)\n", "temp.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"_uuid": "793a1aaff6b2cf3357621009a784f0ea9a7a160d", "_cell_guid": "98761745-7cff-40b6-a5ba-d9e655694d0c"}}, {"source": ["import tensorflow as tf"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {}}, {"source": ["temp = temp.as_matrix()\n", "countries_tensor = tf.constant(temp, dtype=tf.int64, shape=tempshape)"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {}}, {"source": ["country = tf.feature_column.categorical_column_with_vocabulary_list('hotel_continent', vocabulary_list=[0, 1, 2, 3, 4, 5, 6])\n", "\n", "temp_tensor = tf.estimator.LinearClassifier(feature_columns=['hotel_continent'])\n", "# temp_tensor.train(input_fn=train_df[['hotel_continent', 'hotel_cluster']], steps=200)"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true}}, {"source": ["train_df.head()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true}}, {"source": ["import tensorflow as tf\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "tf.set_random_seed(1)\n", "np.random.seed(1)\n", "\n", "# fake data\n", "n_data = np.ones((100, 2))\n", "x0 = np.random.normal(2*n_data, 1)      # class0 x shape=(100, 2)\n", "y0 = np.zeros(100)                      # class0 y shape=(100, 1)\n", "x1 = np.random.normal(-2*n_data, 1)     # class1 x shape=(100, 2)\n", "y1 = np.ones(100)                       # class1 y shape=(100, 1)\n", "x = np.vstack((x0, x1))  # shape (200, 2) + some noise\n", "y = np.hstack((y0, y1))  # shape (200, )\n", "\n", "# plot data\n", "plt.scatter(x[:, 0], x[:, 1], c=y, s=100, lw=0, cmap='RdYlGn')\n", "plt.show()\n", "\n", "tf_x = tf.placeholder(tf.float32, x.shape)     # input x\n", "tf_y = tf.placeholder(tf.int32, y.shape)     # input y\n", "\n", "# neural network layers\n", "l1 = tf.layers.dense(tf_x, 10, tf.nn.relu)          # hidden layer\n", "output = tf.layers.dense(l1, 2)                     # output layer\n", "\n", "loss = tf.losses.sparse_softmax_cross_entropy(labels=tf_y, logits=output)           # compute cost\n", "accuracy = tf.metrics.accuracy(          # return (acc, update_op), and create 2 local variables\n", "    labels=tf.squeeze(tf_y), predictions=tf.argmax(output, axis=1),)[1]\n", "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.05)\n", "train_op = optimizer.minimize(loss)\n", "\n", "sess = tf.Session()                                                                 # control training and others\n", "init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n", "sess.run(init_op)     # initialize var in graph\n", "\n", "plt.ion()   # something about plotting\n", "for step in range(100):\n", "    # train and net output\n", "    _, acc, pred = sess.run([train_op, accuracy, output], {tf_x: x, tf_y: y})\n", "    if step % 2 == 0:\n", "        # plot and show learning process\n", "        plt.cla()\n", "        plt.scatter(x[:, 0], x[:, 1], c=pred.argmax(1), s=100, lw=0, cmap='RdYlGn')\n", "        plt.text(1.5, -4, 'Accuracy=%.2f' % acc, fontdict={'size': 20, 'color': 'red'})\n", "        plt.pause(0.1)\n", "\n", "plt.ioff()\n", "plt.show()"], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "9336f9fb731bfce34858f00e7220e1c714a83274", "_cell_guid": "f798c2cf-e7cf-4a7a-9529-876f37aa7b92"}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "a711f314f943bfbf6118cc7e6a211238f86ec25f", "_cell_guid": "643f0e2c-af4a-49e6-b66e-5d5f0b4d367c"}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "8f786fd9f3d95085cc87217a4e0af242a424f5ed", "_cell_guid": "a5c79118-e904-4aaa-ac2f-8135fc134049"}}, {"source": [], "outputs": [], "cell_type": "code", "execution_count": null, "metadata": {"collapsed": true, "_uuid": "0e2aac486e21de3721ad82f2bc1069fb3adba6a9", "_cell_guid": "bcb538cb-5837-41d9-8dbd-db69b0678c07"}}]}