{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<img src = \"https://databricks.com/wp-content/uploads/2018/12/PySpark-1024x164.png\"><br>\n# <center>Titanic Machine Learning Competition with PySpark</center>","metadata":{}},{"cell_type":"markdown","source":"# Introduction<br>\nI just did a course about PySpark and this notebook is my first attempt at working with it and how it can be used for EDA and machine learning.<br><br>\nPySpark is an interface for Apache Spark in Python that allows you to write Spark applications using Python APIs and is helpful for working with real-time and large-scale data. ","metadata":{}},{"cell_type":"markdown","source":"# The Titanic Machine Learning Competition<br>\nThis project is based on the Titanic dataset provided on the <a href = \"https://www.kaggle.com/competitions/titanic/overview/description\">Titanic ML challenge</a> on Kaggle. Its task is to build a machine learning model that can tell us if passengers were more likely to survive or not according to their data, such as socio-economic class, age, and gender. <br><br>\nThe evaluation method for this model will be the accuracy score i.e the total percentage of correctly predicted passengers. <br><br>\nThis is a binary classification problem and the classes used for predications are 1 for survived and 0 for deceased. \n","metadata":{}},{"cell_type":"code","source":"# Installing PySpark\n!pip install pyspark","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:39:03.422715Z","iopub.execute_input":"2022-07-22T13:39:03.423202Z","iopub.status.idle":"2022-07-22T13:39:54.490099Z","shell.execute_reply.started":"2022-07-22T13:39:03.423101Z","shell.execute_reply":"2022-07-22T13:39:54.488844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Starting a Spark session\nfrom pyspark.sql import SparkSession\nspark = SparkSession.builder.appName('Titanic-ML-Challenge').getOrCreate()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:39:54.492641Z","iopub.execute_input":"2022-07-22T13:39:54.493155Z","iopub.status.idle":"2022-07-22T13:40:00.406126Z","shell.execute_reply.started":"2022-07-22T13:39:54.493113Z","shell.execute_reply":"2022-07-22T13:40:00.404898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Importing functions for EDA\nfrom pyspark.sql.functions import count, mean, when, lit, create_map, regexp_extract, col, split\nfrom itertools import chain","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:00.407629Z","iopub.execute_input":"2022-07-22T13:40:00.407993Z","iopub.status.idle":"2022-07-22T13:40:00.414187Z","shell.execute_reply.started":"2022-07-22T13:40:00.407959Z","shell.execute_reply":"2022-07-22T13:40:00.413237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loading data\ntrain = spark.read.csv('../input/titanic/train.csv', header=True, inferSchema=True)\ntest = spark.read.csv('../input/titanic/test.csv', header=True, inferSchema=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:00.416602Z","iopub.execute_input":"2022-07-22T13:40:00.416998Z","iopub.status.idle":"2022-07-22T13:40:07.788175Z","shell.execute_reply.started":"2022-07-22T13:40:00.416962Z","shell.execute_reply":"2022-07-22T13:40:07.786725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can use printSchema() to visualize column names, data types and whether they're nullable or not\ntrain.printSchema()\ntest.printSchema()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:07.789819Z","iopub.execute_input":"2022-07-22T13:40:07.790366Z","iopub.status.idle":"2022-07-22T13:40:07.812379Z","shell.execute_reply.started":"2022-07-22T13:40:07.790311Z","shell.execute_reply":"2022-07-22T13:40:07.811258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualizing first 5 rows of each dataset\ntrain.show(5)\ntest.show(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:07.813942Z","iopub.execute_input":"2022-07-22T13:40:07.815312Z","iopub.status.idle":"2022-07-22T13:40:08.751341Z","shell.execute_reply.started":"2022-07-22T13:40:07.815259Z","shell.execute_reply":"2022-07-22T13:40:08.749659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Right away, we can see that we have some null values in our datasets. We'll deal with them later on...","metadata":{}},{"cell_type":"code","source":"# Counting the number of classes\ntrain.groupBy('Survived').count().show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:08.752503Z","iopub.execute_input":"2022-07-22T13:40:08.752898Z","iopub.status.idle":"2022-07-22T13:40:09.998345Z","shell.execute_reply.started":"2022-07-22T13:40:08.752858Z","shell.execute_reply":"2022-07-22T13:40:09.997047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most passengers didn't survive. ","metadata":{}},{"cell_type":"code","source":"# Counting average Fare value and average age for each class\ntrain.groupBy('Survived').mean('Age', 'Fare').show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:09.999702Z","iopub.execute_input":"2022-07-22T13:40:10.000177Z","iopub.status.idle":"2022-07-22T13:40:10.560932Z","shell.execute_reply.started":"2022-07-22T13:40:10.000129Z","shell.execute_reply":"2022-07-22T13:40:10.559559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On average, the passengers who did survive paid a higher fare and were younger than passengers who deceased.","metadata":{}},{"cell_type":"code","source":"# Counting survivors by sex\ntrain.groupBy('Survived').pivot('Sex').count().show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:10.562295Z","iopub.execute_input":"2022-07-22T13:40:10.562774Z","iopub.status.idle":"2022-07-22T13:40:12.094693Z","shell.execute_reply.started":"2022-07-22T13:40:10.562721Z","shell.execute_reply":"2022-07-22T13:40:12.093420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can definetly see that most females survived (68.12%) while most men died (85.24%)","metadata":{}},{"cell_type":"code","source":"# Counting survivors by ticket class\ntrain.groupBy('Survived').pivot('Pclass').count().show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:12.100544Z","iopub.execute_input":"2022-07-22T13:40:12.101401Z","iopub.status.idle":"2022-07-22T13:40:13.198687Z","shell.execute_reply.started":"2022-07-22T13:40:12.101352Z","shell.execute_reply":"2022-07-22T13:40:13.197480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the survivors were in the first class, while most of the deceased were in the third class.","metadata":{}},{"cell_type":"code","source":"# Counting survivors by the number of siblings/spouses in the titanic\ntrain.groupBy('Survived').pivot('SibSp').count().show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:13.200259Z","iopub.execute_input":"2022-07-22T13:40:13.200745Z","iopub.status.idle":"2022-07-22T13:40:14.160468Z","shell.execute_reply.started":"2022-07-22T13:40:13.200700Z","shell.execute_reply":"2022-07-22T13:40:14.159227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Counting survivors by the number of parents/children in the titanic\ntrain.groupBy('Survived').pivot('Parch').count().show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:14.161834Z","iopub.execute_input":"2022-07-22T13:40:14.162279Z","iopub.status.idle":"2022-07-22T13:40:15.078483Z","shell.execute_reply.started":"2022-07-22T13:40:14.162237Z","shell.execute_reply":"2022-07-22T13:40:15.077016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It doesn't seem like the number of siblings, spouses, children or parents in the Titanic would play a significant role in surviving the accident. Most of the survivors had no siblings with them, while most of those who were deceased were also alone. <br><br>\nThe only conclusion we could make is that it seems more likely that most large families didn't make it out alive.","metadata":{}},{"cell_type":"code","source":"# Counting survivors by the port of embarkation\ntrain.groupBy('Survived').pivot('Embarked').count().show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:15.080069Z","iopub.execute_input":"2022-07-22T13:40:15.080552Z","iopub.status.idle":"2022-07-22T13:40:15.852013Z","shell.execute_reply.started":"2022-07-22T13:40:15.080486Z","shell.execute_reply":"2022-07-22T13:40:15.850715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok! After getting some insights on survivors, we can realize that gender and socioeconomic class were the most significant features in increasing or decreasing the probability of passengers surviving the accident. <br><br>\n\nLet's now take a look at the dataset and see any changes we must do. We know we have some null data to deal with. Let's count them!","metadata":{}},{"cell_type":"code","source":"for col in train.columns:\n    print(col.ljust(20), train.filter(train[col].isNull()).count())","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:15.853327Z","iopub.execute_input":"2022-07-22T13:40:15.853805Z","iopub.status.idle":"2022-07-22T13:40:17.952145Z","shell.execute_reply.started":"2022-07-22T13:40:15.853761Z","shell.execute_reply":"2022-07-22T13:40:17.950908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Age, Cabin, and Embarked columns have some missing data.<br><br>\n\nNot only does Cabin have a lot of missing values, but it's highly related to ticket class since cabins were distributed according to these classes. We'll drop this column!<br><br>\n\nFor Age and Embarked, we will fill in missing values.","metadata":{}},{"cell_type":"code","source":"# Filling in Embarked with 'S', most repeated value\ntrain = train.fillna({'Embarked':'S'})","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:17.953414Z","iopub.execute_input":"2022-07-22T13:40:17.953875Z","iopub.status.idle":"2022-07-22T13:40:17.982022Z","shell.execute_reply.started":"2022-07-22T13:40:17.953830Z","shell.execute_reply":"2022-07-22T13:40:17.980744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For age imputation, we ought to take the title of the person in the name column and impute the average age for that specific group of people with such title.<br><br>\nMrs, for instance, tends to be older than Miss","metadata":{}},{"cell_type":"code","source":"# Counting the ages per title and getting the average age for each title\ntrain = train.withColumn('Title', regexp_extract(train.Name,'([A-Za-z]+)\\.', 1))\ntrain.groupBy('Title').agg(count('Age'), mean('Age')).sort('count(age)').show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:17.983904Z","iopub.execute_input":"2022-07-22T13:40:17.984674Z","iopub.status.idle":"2022-07-22T13:40:18.409952Z","shell.execute_reply.started":"2022-07-22T13:40:17.984623Z","shell.execute_reply":"2022-07-22T13:40:18.408638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Considering that Mr, Miss and Mrs repeat much more than other titles, we may map some other titles\n# with one of these three. For example, we may map Lady as Mrs and Capt ad Mr\n\ntitles = {'Mr':'Mr', 'Miss':'Miss', 'Mrs':'Mrs', 'Master':'Master', \\\n             'Mlle': 'Miss', 'Major': 'Mr', 'Col': 'Mr', 'Sir': 'Mr',\\\n             'Don': 'Mr', 'Mme': 'Miss', 'Jonkheer': 'Mr', 'Lady': 'Mrs',\\\n             'Capt': 'Mr', 'Countess': 'Mrs', 'Ms': 'Miss', 'Dona': 'Mrs', \\\n             'Dr':'Mr', 'Rev':'Mr'}\n\nmap = create_map([lit(x) for x in chain(*titles.items())])\ntrain = train.withColumn('Title', map[train['Title']])\ntrain.groupBy('Title').mean('Age').show() # Showing new averages for each title","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:18.411275Z","iopub.execute_input":"2022-07-22T13:40:18.412078Z","iopub.status.idle":"2022-07-22T13:40:18.886417Z","shell.execute_reply.started":"2022-07-22T13:40:18.412019Z","shell.execute_reply":"2022-07-22T13:40:18.885168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a function for age imputation\ndef age_imputer(data, title, age):\n    return data.withColumn('Age', when((data.Age.isNull()) & (data.Title==title),age).otherwise(data.Age))","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:18.887788Z","iopub.execute_input":"2022-07-22T13:40:18.889358Z","iopub.status.idle":"2022-07-22T13:40:18.898176Z","shell.execute_reply.started":"2022-07-22T13:40:18.889303Z","shell.execute_reply":"2022-07-22T13:40:18.896648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = age_imputer(train, 'Miss', 21.86)\ntrain = age_imputer(train, 'Master', 4.75)\ntrain = age_imputer(train,'Mr', 33.02)\ntrain = age_imputer(train, 'Mrs', 35.98)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:18.900379Z","iopub.execute_input":"2022-07-22T13:40:18.901320Z","iopub.status.idle":"2022-07-22T13:40:18.988282Z","shell.execute_reply.started":"2022-07-22T13:40:18.901269Z","shell.execute_reply":"2022-07-22T13:40:18.987020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a FamilySize column combining Parch and SibSp\ntrain = train.withColumn('FamilySize', train.Parch + train.SibSp).drop('Parch', 'SibSp')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:18.989669Z","iopub.execute_input":"2022-07-22T13:40:18.990230Z","iopub.status.idle":"2022-07-22T13:40:19.046572Z","shell.execute_reply.started":"2022-07-22T13:40:18.990178Z","shell.execute_reply":"2022-07-22T13:40:19.045305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Removing other unnecessary columns\ntrain = train.drop('PassengerId','Cabin','Name','Ticket','Title')\n# Visualizing data\ntrain.show(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:19.047911Z","iopub.execute_input":"2022-07-22T13:40:19.048343Z","iopub.status.idle":"2022-07-22T13:40:19.340996Z","shell.execute_reply.started":"2022-07-22T13:40:19.048300Z","shell.execute_reply":"2022-07-22T13:40:19.339682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating a Machine Learning Model","metadata":{}},{"cell_type":"code","source":"from pyspark.ml.classification import LogisticRegression,RandomForestClassifier,GBTClassifier\nfrom pyspark.ml.evaluation import MulticlassClassificationEvaluator\nfrom pyspark.ml.feature import StringIndexer, VectorAssembler\nfrom pyspark.ml import Pipeline\nfrom pyspark.ml.tuning import CrossValidator, ParamGridBuilder","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:19.342347Z","iopub.execute_input":"2022-07-22T13:40:19.342823Z","iopub.status.idle":"2022-07-22T13:40:19.499267Z","shell.execute_reply.started":"2022-07-22T13:40:19.342779Z","shell.execute_reply":"2022-07-22T13:40:19.497585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting Sex and Embarked columns to numeric index\nindexer = StringIndexer(inputCols=['Sex','Embarked'],outputCols=['SexIndex', 'EmbarkedIndex'])\nindexer_model = indexer.fit(train)\ntrain_1 = indexer_model.transform(train).drop('Sex','Embarked') # Transforming and dropping original columns\ntrain_1.show(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:19.504728Z","iopub.execute_input":"2022-07-22T13:40:19.505286Z","iopub.status.idle":"2022-07-22T13:40:20.916951Z","shell.execute_reply.started":"2022-07-22T13:40:19.505233Z","shell.execute_reply":"2022-07-22T13:40:20.915983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Separating variables into indepedent and dependent variables with VectorAssembler\nassembler = VectorAssembler(inputCols=train_1.columns[1:], outputCol = 'features')\ntrain_1 = assembler.transform(train_1).select('features','Survived')\ntrain_1.show(5, False)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:20.918147Z","iopub.execute_input":"2022-07-22T13:40:20.918506Z","iopub.status.idle":"2022-07-22T13:40:21.471405Z","shell.execute_reply.started":"2022-07-22T13:40:20.918474Z","shell.execute_reply":"2022-07-22T13:40:21.470096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Splitting data into training and validation sets\ntrain_ds, valid_ds = train_1.randomSplit([0.7,0.3]) # 70% of data will be used for training","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:21.473953Z","iopub.execute_input":"2022-07-22T13:40:21.476762Z","iopub.status.idle":"2022-07-22T13:40:22.009738Z","shell.execute_reply.started":"2022-07-22T13:40:21.476699Z","shell.execute_reply":"2022-07-22T13:40:22.008434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds.show(5, False)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:22.011084Z","iopub.execute_input":"2022-07-22T13:40:22.011535Z","iopub.status.idle":"2022-07-22T13:40:22.608610Z","shell.execute_reply.started":"2022-07-22T13:40:22.011475Z","shell.execute_reply":"2022-07-22T13:40:22.607574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_ds.show(5, False)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:22.609589Z","iopub.execute_input":"2022-07-22T13:40:22.609914Z","iopub.status.idle":"2022-07-22T13:40:22.900310Z","shell.execute_reply.started":"2022-07-22T13:40:22.609885Z","shell.execute_reply":"2022-07-22T13:40:22.898937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy = MulticlassClassificationEvaluator(labelCol='Survived', metricName = 'accuracy')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:22.913565Z","iopub.execute_input":"2022-07-22T13:40:22.916937Z","iopub.status.idle":"2022-07-22T13:40:22.942496Z","shell.execute_reply.started":"2022-07-22T13:40:22.916870Z","shell.execute_reply":"2022-07-22T13:40:22.941113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr = LogisticRegression(labelCol='Survived')\nmodel = lr.fit(train_ds)\nprediction = model.transform(valid_ds)\naccuracy.evaluate(prediction)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:22.943923Z","iopub.execute_input":"2022-07-22T13:40:22.944410Z","iopub.status.idle":"2022-07-22T13:40:26.463184Z","shell.execute_reply.started":"2022-07-22T13:40:22.944357Z","shell.execute_reply":"2022-07-22T13:40:26.460862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rf = RandomForestClassifier(labelCol='Survived')\nmodel = rf.fit(train_ds)\nprediction = model.transform(valid_ds)\naccuracy.evaluate(prediction)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:26.465141Z","iopub.execute_input":"2022-07-22T13:40:26.465797Z","iopub.status.idle":"2022-07-22T13:40:29.653961Z","shell.execute_reply.started":"2022-07-22T13:40:26.465743Z","shell.execute_reply":"2022-07-22T13:40:29.652620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gbt = GBTClassifier(labelCol='Survived')\nmodel = gbt.fit(train_ds)\npredictions = model.transform(valid_ds)\naccuracy.evaluate(predictions)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:29.656029Z","iopub.execute_input":"2022-07-22T13:40:29.656631Z","iopub.status.idle":"2022-07-22T13:40:38.836134Z","shell.execute_reply.started":"2022-07-22T13:40:29.656576Z","shell.execute_reply":"2022-07-22T13:40:38.834479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Random Forest performed better.","metadata":{}},{"cell_type":"markdown","source":"Before testing and tuning our models, we first need to clean our test dataset","metadata":{}},{"cell_type":"code","source":"test.show(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:38.843027Z","iopub.execute_input":"2022-07-22T13:40:38.844147Z","iopub.status.idle":"2022-07-22T13:40:39.001462Z","shell.execute_reply.started":"2022-07-22T13:40:38.844085Z","shell.execute_reply":"2022-07-22T13:40:39.000131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking for missing data\nfor col in test.columns:\n    print(col.ljust(20), test.filter(test[col].isNull()).count())","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:39.002816Z","iopub.execute_input":"2022-07-22T13:40:39.003292Z","iopub.status.idle":"2022-07-22T13:40:40.333776Z","shell.execute_reply.started":"2022-07-22T13:40:39.003245Z","shell.execute_reply":"2022-07-22T13:40:40.332293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filling Fare\ntest = test.fillna({'Fare':14.45})\n# Creating FamilySize Column\ntest = test.withColumn('FamilySize', test.Parch + test.SibSp). drop('Parch', 'SibSp')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:40.335059Z","iopub.execute_input":"2022-07-22T13:40:40.335500Z","iopub.status.idle":"2022-07-22T13:40:40.371370Z","shell.execute_reply.started":"2022-07-22T13:40:40.335460Z","shell.execute_reply":"2022-07-22T13:40:40.369707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For age imputing, we need to redo the same process again, getting the average values for each title\ntest = test.withColumn('Title', regexp_extract(test.Name, '([A-Za-z]+)\\.', 1))\ntest = test.withColumn('Title', map[test.Title])\ntest.groupBy('Title').agg(count('Age'), mean('Age')).sort('count(Age)').show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:40.372828Z","iopub.execute_input":"2022-07-22T13:40:40.373299Z","iopub.status.idle":"2022-07-22T13:40:40.619600Z","shell.execute_reply.started":"2022-07-22T13:40:40.373253Z","shell.execute_reply":"2022-07-22T13:40:40.618279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = age_imputer(test, 'Master', 7.40)\ntest = age_imputer(test, 'Mrs', 38.90)\ntest = age_imputer(test, 'Miss', 21.77)\ntest = age_imputer(test, 'Mr',32.34)\ntest = test.drop('Cabin', 'Name', 'Ticket', 'Title') # Keeping PassengerId for submission\ntest.show(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:40:40.620942Z","iopub.execute_input":"2022-07-22T13:40:40.621420Z","iopub.status.idle":"2022-07-22T13:40:40.846732Z","shell.execute_reply.started":"2022-07-22T13:40:40.621374Z","shell.execute_reply":"2022-07-22T13:40:40.845422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipeline = Pipeline(stages=[indexer, assembler, rf])\n\nparams = ParamGridBuilder().\\\n            addGrid(rf.maxDepth, [3, 4, 5]).\\\n            addGrid(rf.minInfoGain, [0., 0.01, 0.1]).\\\n            addGrid(rf.numTrees, [1000]).\\\n            addGrid(rf.impurity, [\"gini\", \"entropy\"]).\\\n            addGrid(rf.minInstancesPerNode, [1, 5, 10]).\\\n            addGrid(rf.maxBins, [20, 32, 50]).\\\n            build()\n\ncv = CrossValidator(estimator=pipeline, \n                                estimatorParamMaps=params, \n                                evaluator=accuracy, \n                                numFolds=5)\n\nmodel_final = cv.fit(train)\npred_train = model_final.transform(train)\naccuracy.evaluate(pred_train)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T13:41:17.330909Z","iopub.execute_input":"2022-07-22T13:41:17.331324Z","iopub.status.idle":"2022-07-22T14:35:10.127365Z","shell.execute_reply.started":"2022-07-22T13:41:17.331291Z","shell.execute_reply":"2022-07-22T14:35:10.126437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_test = model_final.transform(test)\n\npredictions = pred_test.select('PassengerId','prediction')\npredictions = predictions.withColumn('Survived', predictions['prediction'].\\\n                                    cast('integer')).drop('prediction')\npredictions.show(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T14:35:10.129111Z","iopub.execute_input":"2022-07-22T14:35:10.129705Z","iopub.status.idle":"2022-07-22T14:35:10.662884Z","shell.execute_reply.started":"2022-07-22T14:35:10.129671Z","shell.execute_reply":"2022-07-22T14:35:10.661912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Saving submission\npredictions.toPandas().to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T14:35:10.663811Z","iopub.execute_input":"2022-07-22T14:35:10.664097Z","iopub.status.idle":"2022-07-22T14:35:11.915336Z","shell.execute_reply.started":"2022-07-22T14:35:10.664071Z","shell.execute_reply":"2022-07-22T14:35:11.914051Z"},"trusted":true},"execution_count":null,"outputs":[]}]}