{"cells":[{"metadata":{"_uuid":"91d42d03c9fcae4aa0511513785c3620d3e60b5c"},"cell_type":"markdown","source":"## A Bit Different View on Real Estate"},{"metadata":{"_uuid":"56c42d3570d18dca3e4916f5f14a4189e149a06f","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"For some reasons I'm going to use apache spark ml lib.\nSo, before we begin let's import all nesessary *stuff*, that we need for our research."},{"metadata":{"_uuid":"7dc282ea57721341dc6c02e2f43da8b43a1b1891","slideshow":{"slide_type":"slide"},"trusted":true},"cell_type":"code","source":"#Initial\nimport os\nimport numpy as np\nimport pandas as pd\nimport math\n\n#Spark\nimport pyspark as spark\nfrom pyspark import SparkConf, SparkContext\n\n\nsc = SparkContext.getOrCreate()\n\n\nfrom pyspark.sql import SparkSession, SQLContext\n\nfrom pyspark.ml import Pipeline\nfrom pyspark.ml.regression import GBTRegressor, GeneralizedLinearRegression, AFTSurvivalRegression\nfrom pyspark.ml.feature import VectorIndexer\n\nfrom pyspark.ml.evaluation import RegressionEvaluator\nfrom pyspark.sql.types import DoubleType\n \nfrom pyspark.ml.feature import StringIndexer, VectorAssembler, OneHotEncoder  \n    \n#Plots    \nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bc9e1b0cc4251af748a23b0d82b145aa4d47909"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5310f97624154c6fd35e4c40b3ec62025b8f1b1"},"cell_type":"markdown","source":"### Data Exploration "},{"metadata":{"_uuid":"e6d029c62ca0f58bfdb7790fa83e12724a80ca6d"},"cell_type":"markdown","source":"At this section, I'll use tipical python datascience tools because this is more reliable and suitable tools for EDA part. (I mean using pandas in place of spark's dataframes and etc.)"},{"metadata":{"_uuid":"a1df9ebf1ba3c1b3334ca391bcb85727e8efeba6"},"cell_type":"markdown","source":"Let's take a look on what do we have here"},{"metadata":{"_uuid":"526710a11a4d8d541d48a0016ce492566fc9980e","trusted":true},"cell_type":"code","source":"dataPath = \"../input\"\n', '.join(os.listdir(dataPath))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b3918489dae5f3cd8f434b3a303ad4bf98e942a"},"cell_type":"markdown","source":"### Dataset Exploration:"},{"metadata":{"_uuid":"d3abd8b4f56d2c9c2899ae0a43cbae46d5e443d9","trusted":true},"cell_type":"code","source":"pdtrainset = pd.read_csv(\"../input/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3abd8b4f56d2c9c2899ae0a43cbae46d5e443d9","trusted":true},"cell_type":"code","source":"pdtrainset.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3abd8b4f56d2c9c2899ae0a43cbae46d5e443d9","trusted":true},"cell_type":"code","source":"pdtrainset.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cecf485510222a5df5a0c242a9b3d5b351a5bae"},"cell_type":"code","source":"pdtrainset[\"SalePrice\"] = np.log(pdtrainset[\"SalePrice\"])\npdtrainset.head(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4ed77f733705c77f315bafc133b9850de4629192"},"cell_type":"code","source":"pdtrainset.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3abd8b4f56d2c9c2899ae0a43cbae46d5e443d9","trusted":true},"cell_type":"code","source":"correl = pdtrainset[1:].corr()\ncorrel.drop([\"SalePrice\", \"Id\"], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f689472c3b9d440427ccb1eeb3d6788df44d3901"},"cell_type":"markdown","source":"Here we interrested in features which have more significant effect on sale price."},{"metadata":{"_uuid":"32d9d411ac3610ca45d498d3f103ff4625fe8a82","trusted":true},"cell_type":"code","source":"correl.iloc[-1].apply(lambda x: abs(x)).plot(kind='bar', figsize=(10, 6), title=\"Correlation with SalePrice\", grid=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e278a61e0524edab1fc58cfa74ec7fd17523638a"},"cell_type":"markdown","source":"Let's find a list of most valuable features, then sort it by descending."},{"metadata":{"_uuid":"d461a02795f8c0afb4f555e05d3e81506eb9e77f","trusted":true},"cell_type":"code","source":"salecorrel = pd.DataFrame(correl.transpose()[\"SalePrice\"])\ntop_features = salecorrel[salecorrel[\"SalePrice\"] >= 0.45].sort_values(by=[\"SalePrice\"], \n                                                                      ascending=False)\ntopcolumns = top_features.transpose().columns.values\ntopcorrelVal = top_features.values\n\n[\"{0}: {1:.4f}\".format(col, topcorrelVal[i][0]) for i,col in enumerate(topcolumns)]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02c7980d18da155bda4410e71d581cac1789b392"},"cell_type":"markdown","source":"Now compare valuable feature's correlation eachother.\n\n(Here I should note, that In my city, there are many paradoxes when, for excample, age of a building has a strong positive correlation with the quality of supporting structures...)\n \nI believe that is not everywhere, so I will solely rely  on digits.)"},{"metadata":{"_uuid":"3a02b36eb5dd29e0a54ab0a532e5018f8ba8a2e4","trusted":true},"cell_type":"code","source":"# [sns.lmplot(x=\"SalePrice\",y=x,data=pdtrainset,\n#            scatter_kws={'alpha':0.07}, aspect=2, height=4) for x in topcolumns]\n\ntopItemsCorrelation = pdtrainset[list(topcolumns)].corr().abs()\n\ntopItemsCorrelation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c97f7d99715adcd424a19e953b60308e7d131d37"},"cell_type":"code","source":"sns.heatmap(topItemsCorrelation, annot=True, fmt=\".2f\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ab9777d8e270b2287661b71c5f98395a3a906ed"},"cell_type":"markdown","source":"Let's list a pars that have correlation more then 0.8"},{"metadata":{"trusted":true,"_uuid":"face4ac54fc005c4552f33a3f9f2340edfe3e011"},"cell_type":"code","source":"utcItems = pd.DataFrame(topItemsCorrelation.unstack(), columns=[\"c\"])\nutcItems[(utcItems[\"c\"] > 0.8) & (utcItems[\"c\"] < 1)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d36f2cfff4fa0f7876dfad18a970226df9171a9e"},"cell_type":"markdown","source":"**GrLivArea** has higher correlation with SalePrice then TotRmsAbvGrd patam\nas **GarageCars** and **TotalBsmtSF**, so I'll exclude from dataset \"TotRmsAbvGrd\", \"GarageArea\" and \"1stFlrSF\" from our features research."},{"metadata":{"trusted":true,"_uuid":"bc80d2736776e4bf5124a939961b9b3a20aac414"},"cell_type":"code","source":"topcolumns = list(set(topcolumns) - set([\"TotRmsAbvGrd\", \"GarageArea\", \"1stFlrSF\", \"GarageYrBlt\"]))\ntopcolumns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"244b98a30e7556679616a1b6359e87de4468061c"},"cell_type":"markdown","source":"Totally we have seven features, two of them is year of **YearBuilt** and **YearRemodAdd** (Remodel date). \nI guess, it will good idea to use them as categorical variable splited by decades."},{"metadata":{"trusted":true,"_uuid":"5484eaa24d91e4b2517bf980be4cbe657ec98ecd"},"cell_type":"code","source":"getDecades = lambda col: col.apply(lambda x: math.ceil(float(x) / 10)*10)\n\npdtrainset[\"YearBuilt\"] = getDecades(pdtrainset[\"YearBuilt\"])\npdtrainset[\"YearRemodAdd\"] = getDecades(pdtrainset[\"YearRemodAdd\"])\npdtrainset = pdtrainset[topcolumns+[\"SalePrice\"]]\npdtrainset.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6198db5e4a6563d1902bffb818b74e2b78997249"},"cell_type":"code","source":"valuableColumns = list(pdtrainset.columns.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"660937755233ce187fd250809a70bc12eadfd766"},"cell_type":"code","source":"[sns.lmplot(x=col,y=\"SalePrice\",data=pdtrainset, \n            scatter_kws={'alpha':0.07}, aspect=3, height=5) for col in topcolumns]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf7cbed13ff6f7e480bb61860ae15dfb174d944d"},"cell_type":"markdown","source":"Before train our models, let split features on to two groups: continious and categorical"},{"metadata":{"_uuid":"14084af270795b3249aa9b83bd5e7d0553ec0ca1"},"cell_type":"markdown","source":"I won't deskribe all valuable features here, but "},{"metadata":{"_uuid":"247e6199af15c057f24dce6f8e1fe0d683285c88"},"cell_type":"markdown","source":"### Spark ML"},{"metadata":{"_uuid":"7f10c18c3cc0538214a0f4ef155bd464895acb51","trusted":true},"cell_type":"code","source":"sqlContext = SQLContext(sc)\n\nsc.setLogLevel(\"ERROR\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d15ea4b6d8dd0e8499c6fe5e8beb1ee03dcfb4d"},"cell_type":"markdown","source":"Then prepare spark's dataframes "},{"metadata":{"trusted":true,"_uuid":"a8e3792aa8479f1d4f52081b77dc9c186b7bd03e"},"cell_type":"code","source":"df = sqlContext.createDataFrame(pdtrainset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b5babaccd4647d86ee2b1e4513582640a13d104"},"cell_type":"code","source":"df.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"211f7a70ce713f2df6ab8fe924d87fa32e49c350","trusted":true},"cell_type":"code","source":"# #sptrain = sqlContext.read.csv(\"../input/train.csv\", header=True)\n\nsptrain = df.withColumn(\"label\", df.SalePrice.cast(\"double\")).cache()\n\nsptest = sqlContext.read.csv(\"../input/test.csv\", header=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9bd6db2718bc5eda8997145adbc47a822e150146","trusted":true},"cell_type":"code","source":"for col in valuableColumns[:-1]:\n    # Of cause we can't change immutable values, but we can owerwrite them\n    sptrain = sptrain.withColumn(col+\"_d\", sptrain[col].cast(\"double\"))\n    sptest = sptest.withColumn(col+\"_d\", sptest[col].cast(\"double\"))\n    \nsptrain = sptrain.fillna(-1., subset=valuableColumns)\nsptest = sptest.fillna(-1., subset=valuableColumns)\n# sptrain = sptrain.fillna(\"no\", subset=catColumn)\n# sptest = sptest.fillna(\"no\", subset=catColumn)    \nsptrain","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b20bbba383926d1117367235bf8c177cd19f8f7","trusted":true},"cell_type":"code","source":"#let add few train model \ngbt = GBTRegressor(maxIter=100, \n                   maxDepth=5)\\\n                  .setLabelCol(\"label\")\\\n                  .setFeaturesCol(\"features\")\n\n# Than let split our features on to group categorical (string) and continious (numerical)\n# indexers = [StringIndexer(inputCol=column, outputCol=column+\"_index\")\\\n#             .setHandleInvalid(\"keep\")\\\n#             .fit(sptrain) for column in catColumn]\n\n#indexers = indexers + ([OneHotEncoder(inputCol= col+\"_index\", outputCol= col+\"_oht\") for col in catColumn])\n#TODO: change all columns to a valuable only\nindexers = []\nindexers.append(VectorAssembler(\n    inputCols=[\"{0}_d\".format(col) for col in valuableColumns[:-1]], \n        outputCol=\"features\"))\nindexers.append(gbt)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"167409a4d46a8cd162b287dde9a2d6c05c334db1","trusted":true},"cell_type":"code","source":"(trainingData, subtestData) = sptrain.randomSplit([0.7, 0.3])\n\nmodelGBT = Pipeline(stages=tuple(indexers)).fit(trainingData)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"015f79c2c3a69a0c12be95bcd765d906d520da37"},"cell_type":"code","source":"predictions = modelGBT.transform(subtestData)\n\npredictions.select([\"prediction\", \"label\", \"features\"]).show(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e04b8ef4bd2e4d05db45e0c4078f47615fc6704"},"cell_type":"code","source":"evaluator = RegressionEvaluator().setLabelCol(\"label\").setPredictionCol(\"prediction\").setMetricName(\"rmse\")\nrmse = evaluator.evaluate(predictions)\nprint(\"Root Mean Squared Error (RMSE) on test data = {0}\".format(str(rmse)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f20cf72f4c664dbbbe3364db94b90a762fb0d357"},"cell_type":"code","source":"testResult = modelGBT.transform(sptest)\nsubmit = testResult.select([\"Id\",\"prediction\"])\\\n          .withColumn(\"SalePrice\", testResult[\"prediction\"])\\\n          .drop(\"prediction\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5014e040ddbf63bc092dbfddc474fac3b934c3a9"},"cell_type":"code","source":"#submit.write.format(\"com.databricks.spark.csv\").option(\"header\",\"true\").save(\"../input/output.csv\")\n\njs  = submit.dropna()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b83bcfe87e562de4242e88d4fbc3ce3a3d7e5fca"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b487802f822d42d8fe6119ab074ba0c97e5ac546","scrolled":false},"cell_type":"code","source":"#submit.coalesce(1).write.csv('../input/submission_.csv', encoding=\"utf-8\", emptyValue=-1)\n# submit.write.format(\"com.databricks.spark.csv\").option(\"header\", \"true\").save(\"../input/submit__.csv\")\nsubmit = submit.fillna(-1.0)\n#submit.show()\nsubmit.dropna().coalesce(1).write.csv(os.path.join(\"submission.csv\",dataPath), encoding=\"utf-8\", emptyValue=\"na\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"06ccf493e94339b8c94f5362f88b0f8a032b78de"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}