{ "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0, "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "cfefd672-562f-4c14-b8a6-e369f1845c10", "_active": false }, "source": "# Machine Learning from Start to Finish with Scikit-Learn\n\nThis notebook covers the basic Machine Learning process in Python step-by-step. Go from raw data to at least 78% accuracy on the Titanic Survivors dataset. \n\n### Steps Covered\n\n\n1. Importing a DataFrame\n2. Visualize the Data\n3. Cleanup and Transform the Data\n4. Encode the Data\n5. Split Training and Test Sets\n6. Fine Tune Algorithms\n7. Cross Validate with KFold\n8. Upload to Kaggle", "outputs": [] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "25e53281-ebd2-4e8f-8ca6-496a0b78f198", "_active": false }, "source": "## CSV to DataFrame\n\nCSV files can be loaded into a dataframe by calling `pd.read_csv` . After loading the training and test files, print a `sample` to see what you're working with.", "outputs": [] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "_cell_guid": "2d80c4b5-ef2a-46c5-8d61-6a443932374a", "_active": false }, "outputs": [], "source": "import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\ndata_train = pd.read_csv('../input/train.csv')\ndata_test = pd.read_csv('../input/test.csv')\n\ndata_train.sample(3)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "535fe5e3-26f7-4da4-8fec-b3798b803392", "_active": false }, "source": "## Visualizing Data\n\nVisualizing data is crucial for recognizing underlying patterns to exploit in the model. ", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "88c02736-c7b7-4b44-a637-1498b2257ba9", "_active": true }, "outputs": [], "source": "sns.barplot(x=\"Embarked\", y=\"Survived\", hue=\"Sex\", data=data_train);" }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "915cf1bd-87d6-4a57-9394-473a6693cb88", "_active": false }, "outputs": [], "source": "sns.pointplot(x=\"Pclass\", y=\"Survived\", hue=\"Sex\", data=data_train,\n palette={\"male\": \"blue\", \"female\": \"pink\"},\n markers=[\"*\", \"o\"], linestyles=[\"-\", \"--\"]);" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d4907de3-de90-4861-8d1c-fe285191d78f", "_active": false }, "source": "## Transforming Features\n\n1. Aside from 'Sex', the 'Age' feature is second in importance. To avoid overfitting, I'm grouping people into logical human age groups. \n2. Each Cabin starts with a letter. I bet this letter is much more important than the number that follows, let's slice it off. \n3. Fare is another continuous value that should be simplified. I ran `data_train.Fare.describe()` to get the distribution of the feature, then placed them into quartile bins accordingly. \n4. Extract information from the 'Name' feature. Rather than use the full name, I extracted the last name and name prefix (Mr. Mrs. Etc.), then appended them as their own features. \n5. Lastly, drop useless features. (Ticket and Name)", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "01c87677-a0ab-4175-ae5d-52ee32b38ed7", "_active": false }, "outputs": [], "source": "def simplify_ages(df):\n df.Age = df.Age.fillna(-0.5)\n bins = (-1, 0, 5, 12, 18, 25, 35, 60, 120)\n group_names = ['Unknown', 'Baby', 'Child', 'Teenager', 'Student', 'Young Adult', 'Adult', 'Senior']\n categories = pd.cut(df.Age, bins, labels=group_names)\n df.Age = categories\n return df\n\ndef simplify_cabins(df):\n df.Cabin = df.Cabin.fillna('N')\n df.Cabin = df.Cabin.apply(lambda x: x[0])\n return df\n\ndef simplify_fares(df):\n df.Fare = df.Fare.fillna(-0.5)\n bins = (-1, 0, 8, 15, 31, 1000)\n group_names = ['Unknown', '1_quartile', '2_quartile', '3_quartile', '4_quartile']\n categories = pd.cut(df.Fare, bins, labels=group_names)\n df.Fare = categories\n return df\n\ndef format_name(df):\n df['Lname'] = df.Name.apply(lambda x: x.split(' ')[0])\n df['NamePrefix'] = df.Name.apply(lambda x: x.split(' ')[1])\n return df \n \ndef drop_features(df):\n return df.drop(['Ticket', 'Name', 'Embarked'], axis=1)\n\ndef transform_features(df):\n df = simplify_ages(df)\n df = simplify_cabins(df)\n df = simplify_fares(df)\n df = format_name(df)\n df = drop_features(df)\n return df\n\ndata_train = transform_features(data_train)\ndata_test = transform_features(data_test)\ndata_train.head()" }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "d1e33dc5-72f7-4dfc-850c-5426c43b5d24", "_active": false }, "outputs": [], "source": null }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "19b19ac3-210d-4e64-b754-a2cfc2cb9c63", "_active": false }, "outputs": [], "source": "sns.barplot(x=\"Age\", y=\"Survived\", hue=\"Sex\", data=data_train);" }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "ecae6677-32a5-4540-948c-270bc85e8e55", "_active": false }, "outputs": [], "source": "sns.barplot(x=\"Cabin\", y=\"Survived\", hue=\"Sex\", data=data_train);" }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "1fc48b15-7300-4953-a712-de976bfb83c4", "_active": false }, "outputs": [], "source": "sns.barplot(x=\"Fare\", y=\"Survived\", hue=\"Sex\", data=data_train);" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d29b7bc3-f08c-4136-a1a2-da1abf7004dc", "_active": false }, "source": "## Some Final Encoding\n\nThe last part of the preprocessing phase is to normalize labels. The LabelEncoder in Scikit-learn will convert each unique string value into a number, making out data more flexible for various algorithms. \n\nThe result is a table of numbers that looks scary to humans, but beautiful to machines. ", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "b84f44f4-77b3-4f49-8f59-ed6f49d9be09", "_active": false }, "outputs": [], "source": "from sklearn import preprocessing\ndef encode_features(df_train, df_test):\n features = ['Fare', 'Cabin', 'Age', 'Sex', 'Lname', 'NamePrefix']\n df_combined = pd.concat([df_train[features], df_test[features]])\n \n for feature in features:\n le = preprocessing.LabelEncoder()\n le = le.fit(df_combined[feature])\n df_train[feature] = le.transform(df_train[feature])\n df_test[feature] = le.transform(df_test[feature])\n return df_train, df_test\n \ndata_train, data_test = encode_features(data_train, data_test)\ndata_train.head()" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e79d53bc-7eab-44f6-8d33-dedef53c10b6", "_active": false }, "source": "## Splitting up the Training Data\n\nNow its time for some Machine Learning. \n\nFirst, separate the features(X) from the labels(y). \n\n**X_all:** All features minus the value we want to predict (Survived).\n\n**y_all:** Only the value we want to predict. \n\nSecond, use Scikit-learn to randomly shuffle this data into four variables. In this case, I'm training 80% of the data, then testing against the other 20%. \n\nLater, this data will be reorganized into a KFold pattern to validate the effectiveness of a trained algorithm. ", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "3b9001b0-f93c-4cec-98f2-7e11ce46ebc6", "_active": false }, "outputs": [], "source": "from sklearn.model_selection import train_test_split\n\nX_all = data_train.drop(['Survived', 'PassengerId'], axis=1)\ny_all = data_train['Survived']\n\nnum_test = 0.20\nX_train, X_test, y_train, y_test = train_test_split(X_all, y_all, test_size=num_test, random_state=23)" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7a8fc81f-4544-4fa7-bea8-b666b9e09d22", "_active": false }, "source": "## Fitting and Tuning an Algorithm\n\nNow it's time to figure out which algorithm is going to deliver the best model. I'm going with the RandomForestClassifier, but you can drop any other classifier here, such as Support Vector Machines or Naive Bayes. ", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "427ccaf3-5f11-42a7-8a92-e6b8187025ea", "_active": false }, "outputs": [], "source": "from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import make_scorer, accuracy_score\nfrom sklearn.model_selection import GridSearchCV\n\n# Choose the type of classifier. \nclf = RandomForestClassifier()\n\n# Choose some parameter combinations to try\nparameters = {'n_estimators': [4, 6, 9], \n 'max_features': ['log2', 'sqrt','auto'], \n 'criterion': ['entropy', 'gini'],\n 'max_depth': [2, 3, 5, 10], \n 'min_samples_split': [2, 3, 5],\n 'min_samples_leaf': [1,5,8]\n }\n\n# Type of scoring used to compare parameter combinations\nacc_scorer = make_scorer(accuracy_score)\n\n# Run the grid search\ngrid_obj = GridSearchCV(clf, parameters, scoring=acc_scorer)\ngrid_obj = grid_obj.fit(X_train, y_train)\n\n# Set the clf to the best combination of parameters\nclf = grid_obj.best_estimator_\n\n# Fit the best algorithm to the data. \nclf.fit(X_train, y_train)\n\n" }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "b0fbbb94-2d3a-44a8-97e9-d628464124a5", "_active": false }, "outputs": [], "source": "predictions = clf.predict(X_test)\nprint(accuracy_score(y_test, predictions))" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6f4fc345-4a82-457c-8e1b-da03a2f66030", "_active": false }, "source": "## Validate with KFold\n\nIs this model actually any good? It helps to verify the effectiveness of the algorithm using KFold. This will split our data into 10 buckets, then run the algorithm using a different bucket as the test set for each iteration. ", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "6b8aa496-6bc9-4ce0-b1e7-e525adcfcc11", "_active": false }, "outputs": [], "source": "from sklearn.cross_validation import KFold\n\ndef run_kfold(clf):\n kf = KFold(891, n_folds=10)\n outcomes = []\n fold = 0\n for train_index, test_index in kf:\n fold += 1\n X_train, X_test = X_all.values[train_index], X_all.values[test_index]\n y_train, y_test = y_all.values[train_index], y_all.values[test_index]\n clf.fit(X_train, y_train)\n predictions = clf.predict(X_test)\n accuracy = accuracy_score(y_test, predictions)\n outcomes.append(accuracy)\n print(\"Fold {0} accuracy: {1}\".format(fold, accuracy)) \n mean_outcome = np.mean(outcomes)\n print(\"Mean Accuracy: {0}\".format(mean_outcome)) \n\nrun_kfold(clf)\n" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7c74bd68-e4ac-41bb-a9cb-ec6ffa0c75fe", "_active": false }, "source": "## Predict the Actual Test Data\n\nAnd now for the moment of truth. Make the predictions, export the CSV file, and upload them to Kaggle.", "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "1b31ec2b-db3d-4c42-9bad-b3678f9f2312", "_active": false }, "outputs": [], "source": "ids = data_test['PassengerId']\npredictions = clf.predict(data_test.drop('PassengerId', axis=1))\n\n\noutput = pd.DataFrame({ 'PassengerId' : ids, 'Survived': predictions })\n# output.to_csv('titanic-predictions.csv', index = False)\noutput.head()" } ] }