{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Titanic Prediction with PySpark**\n\nThe goal was to apply supervised machine learning methods to predict if a person survived or did not survived the Titanic accident using PySpark. \n\nA Logistic Regression Model was applied only to the train dataset. The main goal is to use PySpark.\n","metadata":{"id":"0lHV8cgbVnmf"}},{"cell_type":"markdown","source":"## Code Imports and Data Loading","metadata":{"id":"H5bLD1I-WxPw"}},{"cell_type":"markdown","source":"### Installing and prepping PySpark","metadata":{"id":"GQUwIDr_uutX"}},{"cell_type":"code","source":"!pip install pyspark","metadata":{"id":"mTvaz4R_4MqY","execution":{"iopub.status.busy":"2022-07-11T00:30:59.931337Z","iopub.execute_input":"2022-07-11T00:30:59.931783Z","iopub.status.idle":"2022-07-11T00:31:48.830090Z","shell.execute_reply.started":"2022-07-11T00:30:59.931693Z","shell.execute_reply":"2022-07-11T00:31:48.829154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Imports","metadata":{"id":"WxBwOZqr0hXN"}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport pyspark.pandas as ps\n\nfrom pyspark.ml.feature import Imputer\nfrom pyspark.ml.feature import StringIndexer, VectorAssembler, MinMaxScaler","metadata":{"id":"pb8METRdtj7K","outputId":"3d4fb4c1-bf97-46a0-afeb-0e739b2d73fc","execution":{"iopub.status.busy":"2022-07-11T00:34:46.227282Z","iopub.execute_input":"2022-07-11T00:34:46.227727Z","iopub.status.idle":"2022-07-11T00:34:46.234552Z","shell.execute_reply.started":"2022-07-11T00:34:46.227689Z","shell.execute_reply":"2022-07-11T00:34:46.233261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pyspark.sql import SparkSession\nspark = SparkSession.builder.appName('TitanicSpark').getOrCreate()","metadata":{"id":"rl4JFvoAwK5M","execution":{"iopub.status.busy":"2022-07-11T00:33:57.012177Z","iopub.execute_input":"2022-07-11T00:33:57.012817Z","iopub.status.idle":"2022-07-11T00:34:02.753536Z","shell.execute_reply.started":"2022-07-11T00:33:57.012785Z","shell.execute_reply":"2022-07-11T00:34:02.752114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"id":"GIs5JD-WsF-l","outputId":"b46a0d9b-5155-4852-81ec-44a029261201","execution":{"iopub.status.busy":"2022-07-11T00:34:49.472748Z","iopub.execute_input":"2022-07-11T00:34:49.473137Z","iopub.status.idle":"2022-07-11T00:34:49.479817Z","shell.execute_reply.started":"2022-07-11T00:34:49.473103Z","shell.execute_reply":"2022-07-11T00:34:49.478789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Loading and Exploration","metadata":{"id":"sqhpWF5d0lfh"}},{"cell_type":"code","source":"df_train = ps.read_csv(\"/kaggle/input/titanic/train.csv\")\ndf_train.set_index('PassengerId',inplace = True)\ndf_train.head()\n","metadata":{"id":"PTWLGkLPuGdI","outputId":"259eeada-a3fb-422b-c6dc-eccdcd0bebd2","execution":{"iopub.status.busy":"2022-07-11T00:34:53.346760Z","iopub.execute_input":"2022-07-11T00:34:53.347125Z","iopub.status.idle":"2022-07-11T00:35:01.328157Z","shell.execute_reply.started":"2022-07-11T00:34:53.347096Z","shell.execute_reply":"2022-07-11T00:35:01.327272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The dataset**\n\nVariable\t|Definition\t|Key\n-------------------|------------------|----------\nsurvival\t|Survival\t|0 = No, 1 = Yes\npclass\t|Ticket class\t|1 = 1st, 2 = 2nd, 3 = 3rd\nsex\t|Sex|\t\nAge|\tAge in years|\t\nsibsp|\t# of siblings / spouses aboard the Titanic\t|\nparch|\t# of parents / children aboard the Titanic\t|\nticket|\tTicket number\t|\nfare|\tPassenger fare\t|\ncabin|\tCabin number|\t\nembarked|\tPort of Embarkation|\tC = Cherbourg, Q = Queenstown, S = Southampton\n","metadata":{"id":"kCwMwS_i-wYF"}},{"cell_type":"markdown","source":"I do not know what can I do with ticket number and the person`s name. I will remove them and start the analysis.","metadata":{"id":"sTsTEI2X_go8"}},{"cell_type":"code","source":"interesting_cols = ['Survived','Pclass','Sex','Age','SibSp','Parch','Fare','Cabin','Embarked']\nndf_train = df_train[interesting_cols]\n\nndf_train.describe()","metadata":{"id":"rsJDBuArVV6D","outputId":"d049f083-11b7-4592-f50d-c85cbe3513dc","execution":{"iopub.status.busy":"2022-07-11T00:10:17.843058Z","iopub.status.idle":"2022-07-11T00:10:17.843487Z","shell.execute_reply.started":"2022-07-11T00:10:17.843269Z","shell.execute_reply":"2022-07-11T00:10:17.843287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train.isnull().sum(axis=0)","metadata":{"id":"EhFw5VefWID5","outputId":"1c219430-cc55-43b9-aefc-62b9c9c54ab6","execution":{"iopub.status.busy":"2022-07-11T00:10:17.845207Z","iopub.status.idle":"2022-07-11T00:10:17.845611Z","shell.execute_reply.started":"2022-07-11T00:10:17.845386Z","shell.execute_reply":"2022-07-11T00:10:17.845402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some of the age are blank and the port of embarkation. Cabin is blank for most part of the dataset. I will define the cabin as a binary feature: 1 for complete e 0 for non complete.","metadata":{"id":"hzKwTte3_9GF"}},{"cell_type":"code","source":"ndf_train.loc[df_train['Cabin'].notnull(),['Cabin']]=int(1)\nndf_train.loc[ndf_train['Cabin'].isnull(),['Cabin']]=int(0)\n\nndf_train['Cabin'] = ndf_train['Cabin'].astype('int32') ","metadata":{"id":"URx6BqT3lE1f","execution":{"iopub.status.busy":"2022-07-11T00:10:17.847101Z","iopub.status.idle":"2022-07-11T00:10:17.847507Z","shell.execute_reply.started":"2022-07-11T00:10:17.847283Z","shell.execute_reply":"2022-07-11T00:10:17.847301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The missing values for the Embarked will be completed with the its mode.","metadata":{"id":"JS0L-hjUG0dJ"}},{"cell_type":"code","source":"ndf_train.loc[ndf_train['Embarked'].isnull(),['Embarked']]=ndf_train['Embarked'].mode()[0]","metadata":{"id":"6jBo59svEhrU","execution":{"iopub.status.busy":"2022-07-11T00:10:17.849034Z","iopub.status.idle":"2022-07-11T00:10:17.849406Z","shell.execute_reply.started":"2022-07-11T00:10:17.849224Z","shell.execute_reply":"2022-07-11T00:10:17.849247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train.isnull().sum(axis=0)","metadata":{"id":"MNIl_TbzA9MM","outputId":"1b8f2b69-da35-4147-b2c5-cacfe52d7f4f","execution":{"iopub.status.busy":"2022-07-11T00:10:17.850755Z","iopub.status.idle":"2022-07-11T00:10:17.851126Z","shell.execute_reply.started":"2022-07-11T00:10:17.850947Z","shell.execute_reply":"2022-07-11T00:10:17.850963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train.loc[ndf_train['Age'].isnull(),['Survived']].sum(axis=0)/(ndf_train['Age'].isnull().sum(axis=0))","metadata":{"id":"xFw-pY0rBsjN","outputId":"a435be6d-4a69-470b-ffcf-80dab1305a11","execution":{"iopub.status.busy":"2022-07-11T00:10:17.852037Z","iopub.status.idle":"2022-07-11T00:10:17.852389Z","shell.execute_reply.started":"2022-07-11T00:10:17.852204Z","shell.execute_reply":"2022-07-11T00:10:17.852221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train.loc[ndf_train['Age'].notnull(),['Survived']].sum(axis=0)/(ndf_train['Age'].notnull().sum(axis=0))","metadata":{"id":"QzXz4BcoCo03","outputId":"9342ba50-9c22-407b-9080-74f4d4148126","execution":{"iopub.status.busy":"2022-07-11T00:10:17.854276Z","iopub.status.idle":"2022-07-11T00:10:17.854733Z","shell.execute_reply.started":"2022-07-11T00:10:17.854488Z","shell.execute_reply":"2022-07-11T00:10:17.854507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like those who do not have the 'Age' completed are more likely to have died. I will complete those instance with -20 (an impossible age). Perhaps this may help prediction.","metadata":{"id":"jEZ8PrTWDD4c"}},{"cell_type":"code","source":"ndf_train.loc[ndf_train['Age'].isnull(),['Age']]=-20","metadata":{"id":"fmEqLvoREu8I","execution":{"iopub.status.busy":"2022-07-11T00:10:17.856072Z","iopub.status.idle":"2022-07-11T00:10:17.856469Z","shell.execute_reply.started":"2022-07-11T00:10:17.856245Z","shell.execute_reply":"2022-07-11T00:10:17.856263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below I plot histograms of each feature relating to the target.","metadata":{"id":"ak0HDpm98PDn"}},{"cell_type":"code","source":"fig,axs = plt.subplots(3,3,figsize=(20,16))\ncols = ndf_train.columns\n\na = 0; b = 0;\n\nfor i in cols:\n  axs[a, b].hist( x = [ndf_train[i][ndf_train['Survived']==0].to_numpy(),\n                       ndf_train[i][ndf_train['Survived']==1].to_numpy()],\n                 label=['Died','Survived'])\n  axs[a, b].set_title(i);axs[a, b].legend(loc='upper right') \n \n  if b==2:\n    a=a+1;b=0;\n  else:\n    b=b+1;\n","metadata":{"id":"EXuyJxdfhq6m","outputId":"f8fa7de5-8484-4101-f1b9-a097dbde5480","execution":{"iopub.status.busy":"2022-07-11T00:10:17.858497Z","iopub.status.idle":"2022-07-11T00:10:17.858874Z","shell.execute_reply.started":"2022-07-11T00:10:17.858698Z","shell.execute_reply":"2022-07-11T00:10:17.858715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can notice that sex is a very important factor for survival, since most of the women succeeded in being rescued. People in class 1 had more chance of survival than class 2. People from class 2 also were more likely to survive than people class 3. Age seems to be relevant since most kids survived. Parch is also important. ","metadata":{"id":"MjDIcEqiAMAk"}},{"cell_type":"markdown","source":"It looks like Fare is highly skewed. One solution is to apply the log function in order to reduce that.","metadata":{"id":"rhzzM_5VuOxd"}},{"cell_type":"code","source":"ndf_train['Fare'].skew()","metadata":{"id":"DlzqqF_r02jk","outputId":"dbdd66c4-1b73-4192-dc84-7200fd18c286","execution":{"iopub.status.busy":"2022-07-11T00:10:17.860212Z","iopub.status.idle":"2022-07-11T00:10:17.860825Z","shell.execute_reply.started":"2022-07-11T00:10:17.860626Z","shell.execute_reply":"2022-07-11T00:10:17.860647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aux = ndf_train['Fare'].apply(lambda i: np.log(i) if i > 0 else 0)\nndf_train['Fare'] = aux","metadata":{"id":"A_Z2cRSTpiRb","execution":{"iopub.status.busy":"2022-07-11T00:10:17.862266Z","iopub.status.idle":"2022-07-11T00:10:17.862666Z","shell.execute_reply.started":"2022-07-11T00:10:17.862472Z","shell.execute_reply":"2022-07-11T00:10:17.862491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train['Fare'].skew()","metadata":{"id":"daASzNsL0-BM","outputId":"7f4cfe64-cfcc-4a64-a35a-17f9822a6d05","execution":{"iopub.status.busy":"2022-07-11T00:10:17.864011Z","iopub.status.idle":"2022-07-11T00:10:17.864387Z","shell.execute_reply.started":"2022-07-11T00:10:17.864201Z","shell.execute_reply":"2022-07-11T00:10:17.864217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i= 'Fare'\n\nfig,axs = plt.subplots(1,1,figsize=(8,6))\naxs.hist(x = [ndf_train[i][ndf_train['Survived']==0].to_numpy(),\n              ndf_train[i][ndf_train['Survived']==1].to_numpy()],\n              label=['Died','Survived'])\naxs.set_title(i);axs.legend(loc='upper right')","metadata":{"id":"6GN2-4j32_i8","outputId":"8eebb3d8-f3de-4b3e-c362-be8f216c8269","execution":{"iopub.status.busy":"2022-07-11T00:10:17.865812Z","iopub.status.idle":"2022-07-11T00:10:17.866215Z","shell.execute_reply.started":"2022-07-11T00:10:17.865995Z","shell.execute_reply":"2022-07-11T00:10:17.866014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train = ps.get_dummies(ndf_train,columns=['Sex','Embarked'])","metadata":{"id":"nSa9q_2pQTGP","outputId":"4b8cde96-e418-41af-9efd-b8e006ac87a4","execution":{"iopub.status.busy":"2022-07-11T00:10:17.867444Z","iopub.status.idle":"2022-07-11T00:10:17.867857Z","shell.execute_reply.started":"2022-07-11T00:10:17.867648Z","shell.execute_reply":"2022-07-11T00:10:17.867667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ndf_train = ndf_train.drop(columns='Sex_male')\nndf_train.head(10)","metadata":{"id":"YW28B2zvRDaG","outputId":"5b8dafd1-f043-4bf4-e8ae-910b24386dbd","execution":{"iopub.status.busy":"2022-07-11T00:10:17.869144Z","iopub.status.idle":"2022-07-11T00:10:17.869554Z","shell.execute_reply.started":"2022-07-11T00:10:17.869327Z","shell.execute_reply":"2022-07-11T00:10:17.869344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Logistic Regression with pyspark","metadata":{"id":"4KQ9cgx7Iiw8"}},{"cell_type":"markdown","source":"To use the dataframe in the MLlib PySpark library, it is necessary to convert to a Spark Datframe.","metadata":{"id":"IdmIJdtwRPqY"}},{"cell_type":"code","source":"spark_df = ndf_train.to_spark()\nspark_df.printSchema()","metadata":{"id":"nBLAx_M9gv97","outputId":"ca90cf27-def8-4811-87f1-9e0be2dbb001","execution":{"iopub.status.busy":"2022-07-11T00:10:17.870657Z","iopub.status.idle":"2022-07-11T00:10:17.871016Z","shell.execute_reply.started":"2022-07-11T00:10:17.870836Z","shell.execute_reply":"2022-07-11T00:10:17.870852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The label must be set.","metadata":{"id":"vbLXPfYaRZVB"}},{"cell_type":"code","source":"label_indexer = StringIndexer(inputCol=\"Survived\", outputCol=\"label\")\nspark_df = label_indexer.fit(spark_df).transform(spark_df)\n","metadata":{"id":"uFzHz57VTGui","execution":{"iopub.status.busy":"2022-07-11T00:10:17.872293Z","iopub.status.idle":"2022-07-11T00:10:17.872683Z","shell.execute_reply.started":"2022-07-11T00:10:17.872494Z","shell.execute_reply":"2022-07-11T00:10:17.872512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And a vector gathering all this must built.","metadata":{"id":"AV7wwtA6RdZ2"}},{"cell_type":"code","source":"dfInputCols = ndf_train.columns.drop('Survived').to_list()\nassembler = VectorAssembler(inputCols=dfInputCols, outputCol=\"features\")\noutputData = assembler.transform(spark_df).select(['features','label'])\n\noutputData.printSchema()","metadata":{"id":"QyIIQKtRKn5O","outputId":"a7099f1f-03e0-465b-b3ad-3cf6eb24db55","execution":{"iopub.status.busy":"2022-07-11T00:10:17.873781Z","iopub.status.idle":"2022-07-11T00:10:17.874141Z","shell.execute_reply.started":"2022-07-11T00:10:17.873952Z","shell.execute_reply":"2022-07-11T00:10:17.873968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will use a simple hold out technique to evaluate the Logistic Model.","metadata":{"id":"9G5zeWiwRkVO"}},{"cell_type":"code","source":"## Split Dataset\n(trainingData, testData) = outputData.randomSplit([0.7, 0.3], seed = 100)\n\nprint(\"Training Dataset Count: \" + str(trainingData.count()))\nprint(\"Test Dataset Count: \" + str(testData.count()))","metadata":{"id":"tKVwtb59T8Lc","outputId":"db6e4e38-380c-4d8d-efb2-077bf689050e","execution":{"iopub.status.busy":"2022-07-11T00:10:17.876229Z","iopub.status.idle":"2022-07-11T00:10:17.876642Z","shell.execute_reply.started":"2022-07-11T00:10:17.876438Z","shell.execute_reply":"2022-07-11T00:10:17.876456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Building the Logistic Regression model...","metadata":{"id":"KcJHLXnLRynK"}},{"cell_type":"code","source":"from pyspark.ml.classification import LogisticRegression\n\nlr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)\n\nlrModel = lr.fit(trainingData)\n\nprint(\"Coefficients: \\n\" + str(lrModel.coefficientMatrix))\nprint(\"Intercept: \" + str(lrModel.interceptVector))","metadata":{"id":"peilybI3UDUG","outputId":"6385621c-d17d-467f-f6a4-d4fb2013f9ff","execution":{"iopub.status.busy":"2022-07-11T00:10:17.878218Z","iopub.status.idle":"2022-07-11T00:10:17.878630Z","shell.execute_reply.started":"2022-07-11T00:10:17.878408Z","shell.execute_reply":"2022-07-11T00:10:17.878445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pyspark.ml.evaluation import BinaryClassificationEvaluator\n\npredictions = lrModel.transform(testData)\n\nevaluator = BinaryClassificationEvaluator(rawPredictionCol=\"rawPrediction\")\nevaluator.evaluate(predictions)","metadata":{"id":"CEcbaUz7UFp1","outputId":"3d70b3d5-7feb-4a45-8475-2740b50c9908","execution":{"iopub.status.busy":"2022-07-11T00:10:17.880022Z","iopub.status.idle":"2022-07-11T00:10:17.880392Z","shell.execute_reply.started":"2022-07-11T00:10:17.880207Z","shell.execute_reply":"2022-07-11T00:10:17.880224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With no proper tuning, the model predicted the test dataset with 74% accuracy. ","metadata":{"id":"zBgZ57gSR3jM"}},{"cell_type":"code","source":"from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n\nparamGrid = (ParamGridBuilder()\n             .addGrid(lr.regParam, [0.01, 0.5, 2.0])\n             .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])\n             .addGrid(lr.maxIter, [1, 5, 10])\n             .build())\n\ncrossval = CrossValidator(estimator=lr,\n                          estimatorParamMaps=paramGrid,\n                          evaluator=evaluator,\n                          numFolds=2)\n\ncvModel = crossval.fit(outputData)","metadata":{"id":"QFPiL0STULJC","execution":{"iopub.status.busy":"2022-07-11T00:10:17.881619Z","iopub.status.idle":"2022-07-11T00:10:17.881974Z","shell.execute_reply.started":"2022-07-11T00:10:17.881794Z","shell.execute_reply":"2022-07-11T00:10:17.881811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The cell above does a fine tuning of the model's paramenters.","metadata":{"id":"BDf4SJzZSBU5"}},{"cell_type":"code","source":"predictions = cvModel.transform(testData)\nevaluator.evaluate(predictions)","metadata":{"id":"sTK4Y2yTUYn4","outputId":"b2b577e3-a4d3-4175-88eb-e362a87997e3","execution":{"iopub.status.busy":"2022-07-11T00:10:17.883132Z","iopub.status.idle":"2022-07-11T00:10:17.883563Z","shell.execute_reply.started":"2022-07-11T00:10:17.883317Z","shell.execute_reply":"2022-07-11T00:10:17.883334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the tuning, we reached a 81% accuracy. ","metadata":{"id":"tG2tRIFwSIN_"}}]}