{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"You've learned the basics of how data sets are stored and accessed in Python. Now let's learn how to perform basic analysis and visualization by examining a data set of home sales. ","metadata":{"_uuid":"014f5c099a26d9232f9d0d6ca85d5c02b812c98a","_cell_guid":"8ca352d7-08aa-36b4-fb2d-3c9854a8d86a"}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nfrom scipy.stats import norm\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LinearRegression\nfrom scipy import stats\nimport warnings\nwarnings.filterwarnings('ignore')\n%matplotlib inline","metadata":{"_uuid":"d581f6797b9fde1580271358d484df67bf6b14a1","_cell_guid":"2df621e0-e03c-7aaa-6e08-40ed1d7dfecc","_execution_state":"idle","execution":{"iopub.status.busy":"2022-07-29T12:04:33.700044Z","iopub.execute_input":"2022-07-29T12:04:33.700706Z","iopub.status.idle":"2022-07-29T12:04:35.460161Z","shell.execute_reply.started":"2022-07-29T12:04:33.700638Z","shell.execute_reply":"2022-07-29T12:04:35.458977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read the train.csv file as a data frame called *df_train*.","metadata":{}},{"cell_type":"code","source":"#df_train = # this code is incomplete!#","metadata":{"_uuid":"827a72128cd211cf6af16b003e7c09951e3f2b1e","_cell_guid":"d56d5e71-4277-7a74-5306-7d5af4c7f263","_execution_state":"idle","execution":{"iopub.status.busy":"2022-07-29T12:04:41.220941Z","iopub.execute_input":"2022-07-29T12:04:41.221589Z","iopub.status.idle":"2022-07-29T12:04:41.254782Z","shell.execute_reply.started":"2022-07-29T12:04:41.221213Z","shell.execute_reply":"2022-07-29T12:04:41.253950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check the column names of the data frame.","metadata":{}},{"cell_type":"code","source":"# Your code here","metadata":{"_uuid":"10814ba44786b5fea5e333324c6fe54729cabf33","_cell_guid":"02250c81-7e15-c195-2e86-5adbd15c9d30","_execution_state":"idle","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us examine the SalePrice column.","metadata":{}},{"cell_type":"code","source":"# This code is incomplete! You must finish it before it works properly.\n#.describe()","metadata":{"_uuid":"5c15e1bd10b8e71c0b1d62bdb260882585a35579","_cell_guid":"54452e23-f4d3-919f-c734-80a35dc9ae08","_execution_state":"idle","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What do you think the above output represents? ","metadata":{"_uuid":"bb3005e8025ea75b0b4d1ef624927e7e21833ea1","_cell_guid":"6af460e5-1be2-6618-d624-2a4423bd501f"}},{"cell_type":"markdown","source":"Below is the code for a histogram. After running it, discuss the story it is telling you with your group members. What does the y-axis represent? What's the distinction between the blue boxes and the blue curve?","metadata":{}},{"cell_type":"code","source":"\n#histogram\nsns.distplot(df_train['SalePrice']);","metadata":{"_uuid":"2f78c77caa7290298138caf167672e62d3bc5a67","_cell_guid":"6bbea362-77b6-5385-f0a8-fb53afd088b7","_execution_state":"idle","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Look at the code below. What do you think is the purpose of the .concat() function? ","metadata":{}},{"cell_type":"code","source":"#scatter plot grlivarea/saleprice\nx_var = 'GrLivArea'\ndata = pd.concat([df_train['SalePrice'], df_train[x_var]], axis=1)\ndata.plot.scatter(x=x_var, y='SalePrice', ylim=(0,800000));","metadata":{"_uuid":"91160363898f5caeee965a1aa81eb3abb7dcd760","_cell_guid":"db040973-0adc-e126-e657-1d8934b5a5c8","_execution_state":"idle","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Does it look like there is a linear relationship between the variables?\n\nUse the previous code block as inspiration to produce a scatterplot of SalePrice versus TotalBsmtSF.","metadata":{"_uuid":"6775955416e9b2d4ac43e3fec5685c8577138455","_cell_guid":"c3dccb06-206a-20c0-6060-8d5b49f4df0b"}},{"cell_type":"code","source":"# Your code here","metadata":{"_uuid":"3ac3db51311338fcdc16014a7c506cf3d5315af7","_cell_guid":"353def35-0f26-998d-b9a4-7356f95e80ad","_execution_state":"idle","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Do you think that the relationship between SalePrice and TotalBsmtSF is linear?","metadata":{"_uuid":"4496c7d3635636e8e19296f3b5b52fdba16f7cc5","_cell_guid":"7ea69b90-de6d-c104-ff92-a49943993930"}},{"cell_type":"markdown","source":"Run the following two code blocks and examine the outputs. Then go back through the code and try to figure out what each line does. ","metadata":{}},{"cell_type":"code","source":"X = data.iloc[:, 0].values.reshape(-1, 1)  # values converts it into a numpy array\nY = data.iloc[:, 1].values.reshape(-1, 1)  # -1 means that calculate the dimension of rows, but have 1 column\nlinear_regressor = LinearRegression()  # create object for the class\nlinear_regressor.fit(X, Y)  # perform linear regression\nY_pred = linear_regressor.predict(X)  # make predictions\n","metadata":{"execution":{"iopub.status.busy":"2022-07-11T20:13:50.653188Z","iopub.execute_input":"2022-07-11T20:13:50.653735Z","iopub.status.idle":"2022-07-11T20:13:50.661882Z","shell.execute_reply.started":"2022-07-11T20:13:50.653654Z","shell.execute_reply":"2022-07-11T20:13:50.660966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nplt.scatter(X, Y)\nplt.plot(X, Y_pred, color='red')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-11T20:14:09.265315Z","iopub.execute_input":"2022-07-11T20:14:09.265894Z","iopub.status.idle":"2022-07-11T20:14:09.575975Z","shell.execute_reply.started":"2022-07-11T20:14:09.265801Z","shell.execute_reply":"2022-07-11T20:14:09.574542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extra code block if you need it","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Run the same analysis as above, but this time produce a linear regression for SalePrice versus TotalBsmtSF.","metadata":{}},{"cell_type":"code","source":"# Your code here","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you suspect that two variables don't have a linear relationship, then linear regression won't be a useful predictor. We can sometimes get a better prediction by first transforming one or both of the variables. Run the code block below and then inspect the values in the data frame. What did the code do?","metadata":{}},{"cell_type":"code","source":"data['LogSalePrice'] = np.log(data['SalePrice'])","metadata":{"execution":{"iopub.status.busy":"2022-07-29T12:10:14.819614Z","iopub.execute_input":"2022-07-29T12:10:14.819976Z","iopub.status.idle":"2022-07-29T12:10:14.871890Z","shell.execute_reply.started":"2022-07-29T12:10:14.819917Z","shell.execute_reply":"2022-07-29T12:10:14.871038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a new linear regression of the logarithm of SalePrice versus TotalBsmtSF. ","metadata":{}},{"cell_type":"code","source":"# Your code here","metadata":{},"execution_count":null,"outputs":[]}]}