{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport plotly.express as px\n\nimport plotly.graph_objects as go\nimport numpy as np\n\nimport seaborn as sns\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-03T00:40:41.155108Z","iopub.execute_input":"2022-08-03T00:40:41.155502Z","iopub.status.idle":"2022-08-03T00:40:42.434331Z","shell.execute_reply.started":"2022-08-03T00:40:41.155456Z","shell.execute_reply":"2022-08-03T00:40:42.433346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_df = pd.read_csv('/kaggle/input/housing-affordability-in-canada/housing-supply-price-rental.csv')","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:42.436250Z","iopub.execute_input":"2022-08-03T00:40:42.436763Z","iopub.status.idle":"2022-08-03T00:40:42.460113Z","shell.execute_reply.started":"2022-08-03T00:40:42.436726Z","shell.execute_reply":"2022-08-03T00:40:42.459194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Insights","metadata":{}},{"cell_type":"markdown","source":"### Concepts and Definitions provided by CMHC\n\n* A **“start”** for the purposes of the Starts and Completions Survey, is defined as the beginning of\nconstruction work on a building, usually when the concrete has been poured for the whole of the\nfooting around the structure, or an equivalent stage where a basement will not be part of the structure.\n\n* A __“completion”__ is defined as the stage at which all proposed construction work on the building has\nbeen performed, although under some circumstances a building may be counted as completed where up\nto 10 percent of the proposed work remains to be done.\n\n* For multiple-dwelling structures, the definition of a Start or a Completion applies to the structure\nrather than to the individual dwelling units therein.\n\n* The number of units __“under construction”__ as at the end of the period shown, takes into account\ncertain adjustments which are necessary for various reasons. For example, after a start on a dwelling\nhas commenced construction may cease, or a structure when completed may contain more or fewer\ndwelling units than were reported at start.\n\n* A __dwelling__ is defined as being “absorbed” when a binding, non-conditional agreement is made to buy\nthe dwelling.\nOnly new self-contained dwelling units are enumerated in the Starts and Completions\nSurvey, such units being designed for non-transient and year-round occupancy.\n\n* Seasonal dwellings, such as: summer cottages, hunting and ski cabins, trailers and boat houses; and\nhostel accommodation, such as: hospitals, nursing homes, penal institutions, convents,\nmonasteries, military and industrial camps, and collective types of accommodation such as: hotels,\nclubs, and lodging homes are excluded from all residential housing surveys.\n\n* Mobile Homes are included in the surveys. A mobile home is a type of manufactured house that is\ncompletely assembled in a factory, then moved to a foundation before it is occupied.\n\n* Trailers or any other movable dwelling (the larger often referred to as a mobile home) with no\npermanent foundation are excluded from the surveys.\n\n* Market housing is defined as housing that is marketed to the general public for sale or rent.\n\n* A “dwelling unit” is defined as a structurally separate set of living premises with a private entrance\neither outside the building or from a common hall, lobby, vestibule or stairway inside the building. The\nentrance must be one that can be used without passing through anyone else’s living quarters.\n\n* Seasonally Adjusted at Annual Rate (SAAR) is the result of adjusting monthly or quarterly\nstatistics to provide an indication of the annual total which would be achieved if activity in all other\nmonths or quarters were at the same level of performance relative to past seasonal patterns.\n\n\n","metadata":{}},{"cell_type":"code","source":"housing_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:42.461639Z","iopub.execute_input":"2022-08-03T00:40:42.462269Z","iopub.status.idle":"2022-08-03T00:40:42.508786Z","shell.execute_reply.started":"2022-08-03T00:40:42.462235Z","shell.execute_reply":"2022-08-03T00:40:42.507511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_df.tail()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:42.805528Z","iopub.execute_input":"2022-08-03T00:40:42.805947Z","iopub.status.idle":"2022-08-03T00:40:42.841675Z","shell.execute_reply.started":"2022-08-03T00:40:42.805911Z","shell.execute_reply":"2022-08-03T00:40:42.840546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:43.141427Z","iopub.execute_input":"2022-08-03T00:40:43.142736Z","iopub.status.idle":"2022-08-03T00:40:43.148282Z","shell.execute_reply.started":"2022-08-03T00:40:43.142688Z","shell.execute_reply":"2022-08-03T00:40:43.147491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Drop last two rows which has additional info","metadata":{}},{"cell_type":"code","source":"housing_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:43.677553Z","iopub.execute_input":"2022-08-03T00:40:43.678747Z","iopub.status.idle":"2022-08-03T00:40:43.707664Z","shell.execute_reply.started":"2022-08-03T00:40:43.678700Z","shell.execute_reply":"2022-08-03T00:40:43.706402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Understanding the features\n\n* total_dwelling(construction started)  - single_detached + multiple \n* multiple        - semi_detached + row + apartment\n* total_dwelling_market - homeownership_freehold + rental + homeownership_condo + other\n* completed (construction completed total)\n* res_building_permit - count\n* res_building_permit_amount - value/price in 1000s (dollars)\n* completed_but_unabsorbed_homes - (new_single_and_semi_detached + new_rows_and_apartment )\n* rental_vacancy_rate\n* rental_avilability_rate\n* vacancy_rate_seniors \n* vacancy_rate_condo  \n* HPI_change - housing price index (percentage change)\n* CPI_change - Consumer price index (percentage change)\n* owned_accommodation_costs_change (percentage change)\n* rental_accommodation_costs_change (percentage change)\n* bachelor - Average rent\n* one_bedroom   - Average rent                \n* two_bedroom   - Average rent    \n* three_bedroom - Average rent of 3+ bedrroms\n* population    - in thousands\n* labour_participation_rate  - percentage\n* employment_change          - percentage change       \n* unemployment_rate          - percentage\n* disposable_income_change   - percentage change   \n* migration                  - immigration\n* region       - provinces and Census Metropolitan Areas (CMAs) across Canada\n\n\n\n\n","metadata":{}},{"cell_type":"code","source":"def percentage_missing(df):\n    no_of_missing_values = df.isnull().sum()\n    percent_of_missing_values = no_of_missing_values/df.shape[0] * 100\n\n    table = pd.DataFrame(percent_of_missing_values , columns=['percentage of missing values'] )   # Create dataframe with percentage of missing values\n\n    return table[table.any(axis=1)]    # remove the rows with zeros\n    \n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:44.452741Z","iopub.execute_input":"2022-08-03T00:40:44.453144Z","iopub.status.idle":"2022-08-03T00:40:44.460322Z","shell.execute_reply.started":"2022-08-03T00:40:44.453111Z","shell.execute_reply":"2022-08-03T00:40:44.459094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_columns = housing_df.columns[housing_df.isnull().any()]","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:44.823573Z","iopub.execute_input":"2022-08-03T00:40:44.824397Z","iopub.status.idle":"2022-08-03T00:40:44.831444Z","shell.execute_reply.started":"2022-08-03T00:40:44.824357Z","shell.execute_reply":"2022-08-03T00:40:44.830175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"percentage_missing(housing_df)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:45.265552Z","iopub.execute_input":"2022-08-03T00:40:45.266297Z","iopub.status.idle":"2022-08-03T00:40:45.278602Z","shell.execute_reply.started":"2022-08-03T00:40:45.266260Z","shell.execute_reply":"2022-08-03T00:40:45.277137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt \n\n# Visual representation of columns with missing values\nimport missingno as mno\nmno.matrix(housing_df[missing_columns], figsize = (10, 5))\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:46.369174Z","iopub.execute_input":"2022-08-03T00:40:46.370214Z","iopub.status.idle":"2022-08-03T00:40:46.726799Z","shell.execute_reply.started":"2022-08-03T00:40:46.370169Z","shell.execute_reply":"2022-08-03T00:40:46.725572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# for col in missing_columns:\n#     print(\"dataframe with NaN values in \" + col)\n#     display( housing_df[housing_df[col].isna()] )","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:47.217859Z","iopub.execute_input":"2022-08-03T00:40:47.218274Z","iopub.status.idle":"2022-08-03T00:40:47.223910Z","shell.execute_reply.started":"2022-08-03T00:40:47.218232Z","shell.execute_reply":"2022-08-03T00:40:47.222626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Imputing missing values by interpolating according to year","metadata":{}},{"cell_type":"code","source":"housing_df.interpolate(method='linear', limit_direction='forward', axis=0, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:49.076740Z","iopub.execute_input":"2022-08-03T00:40:49.077398Z","iopub.status.idle":"2022-08-03T00:40:49.089394Z","shell.execute_reply.started":"2022-08-03T00:40:49.077348Z","shell.execute_reply":"2022-08-03T00:40:49.088299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#housing_df.set_index('year').interpolate(method=\"linear\", inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:49.626306Z","iopub.execute_input":"2022-08-03T00:40:49.627287Z","iopub.status.idle":"2022-08-03T00:40:49.632458Z","shell.execute_reply.started":"2022-08-03T00:40:49.627239Z","shell.execute_reply":"2022-08-03T00:40:49.631548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:50.006761Z","iopub.execute_input":"2022-08-03T00:40:50.007192Z","iopub.status.idle":"2022-08-03T00:40:50.025249Z","shell.execute_reply.started":"2022-08-03T00:40:50.007156Z","shell.execute_reply":"2022-08-03T00:40:50.024072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sanity check for NaN values\npercentage_missing(housing_df)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:50.545761Z","iopub.execute_input":"2022-08-03T00:40:50.546177Z","iopub.status.idle":"2022-08-03T00:40:50.558620Z","shell.execute_reply.started":"2022-08-03T00:40:50.546139Z","shell.execute_reply":"2022-08-03T00:40:50.557399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dropping column 1 which is just index\nhousing_df.drop(['Unnamed: 0'], axis=1, inplace= True)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:50.964201Z","iopub.execute_input":"2022-08-03T00:40:50.965363Z","iopub.status.idle":"2022-08-03T00:40:50.971154Z","shell.execute_reply.started":"2022-08-03T00:40:50.965318Z","shell.execute_reply":"2022-08-03T00:40:50.970063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:51.390253Z","iopub.execute_input":"2022-08-03T00:40:51.391606Z","iopub.status.idle":"2022-08-03T00:40:51.416586Z","shell.execute_reply.started":"2022-08-03T00:40:51.391554Z","shell.execute_reply":"2022-08-03T00:40:51.415722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Analysis on Supply and Demand","metadata":{}},{"cell_type":"code","source":"# seperate dataframe for construction details\n\n\nconstn_df = housing_df.drop(housing_df.columns[24: -1], axis=1)\n\nconstn_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:52.673745Z","iopub.execute_input":"2022-08-03T00:40:52.674403Z","iopub.status.idle":"2022-08-03T00:40:52.683912Z","shell.execute_reply.started":"2022-08-03T00:40:52.674363Z","shell.execute_reply":"2022-08-03T00:40:52.682566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"constn_df.rename(columns={'total_dwelling' :'total_constn_start', \n                          'total_dwelling_market' : 'contn_start_int_market',\n                          'completed' : 'total_constn_complete', \n                         }, inplace = True)\n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:53.155705Z","iopub.execute_input":"2022-08-03T00:40:53.156141Z","iopub.status.idle":"2022-08-03T00:40:53.162337Z","shell.execute_reply.started":"2022-08-03T00:40:53.156105Z","shell.execute_reply":"2022-08-03T00:40:53.161066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"constn_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:53.630541Z","iopub.execute_input":"2022-08-03T00:40:53.631143Z","iopub.status.idle":"2022-08-03T00:40:53.636794Z","shell.execute_reply.started":"2022-08-03T00:40:53.631108Z","shell.execute_reply":"2022-08-03T00:40:53.635976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.line(constn_df, x=\"year\", y=\"total_constn_start\", color = 'region', title='Constructions started year by year')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:54.080571Z","iopub.execute_input":"2022-08-03T00:40:54.080977Z","iopub.status.idle":"2022-08-03T00:40:55.388106Z","shell.execute_reply.started":"2022-08-03T00:40:54.080945Z","shell.execute_reply":"2022-08-03T00:40:55.386731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The above chart shows that Ontario has highest number of constructions.\n* In 1995, there was a considerable dip in new constructions. Then it increased and reached the peak in 2003.In 2009 there is little drop. Till 2016 the construction are steadily increasing.\n* We can see the same trend in other provinces as well(Quebec, Alberta)","metadata":{}},{"cell_type":"code","source":"provinces = ['alberta', 'ontario', 'quebec','prince_edward', 'manitoba','new_brunswick', 'saskatchewan', 'nova_scotia' ]","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:55.390701Z","iopub.execute_input":"2022-08-03T00:40:55.391202Z","iopub.status.idle":"2022-08-03T00:40:55.396857Z","shell.execute_reply.started":"2022-08-03T00:40:55.391145Z","shell.execute_reply":"2022-08-03T00:40:55.395731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plotting bar chart for the provinces alone\nfig = px.bar(constn_df[constn_df['region'].isin(provinces)], x=\"year\", y = \"total_constn_start\", color = 'region',\n             barmode='group',height=500, title='Constructions started year by year for Canadian Provinces')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:55.676431Z","iopub.execute_input":"2022-08-03T00:40:55.676881Z","iopub.status.idle":"2022-08-03T00:40:55.794865Z","shell.execute_reply.started":"2022-08-03T00:40:55.676843Z","shell.execute_reply":"2022-08-03T00:40:55.793691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The figure above shows Ontario, Quebec and Alberta high construction rate compared to other provinces.\n* The housing crisis in USA was in 2008, In 2009, number constructions decreased but again picked up in next year.\n* In 1990, Ontario and Quebec had significatly higher number of constructions but after 2003, the numbers increased almost similar to Quebec\n","metadata":{}},{"cell_type":"markdown","source":"## More detailed comparion for Constructions Started vs Completed Vs Unabsorbed by year Vs Residential Permit for  Census Metropolitan Areas - Canada","metadata":{}},{"cell_type":"code","source":"# regions = housing_df[\"region\"].unique().tolist()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:58.168928Z","iopub.execute_input":"2022-08-03T00:40:58.169740Z","iopub.status.idle":"2022-08-03T00:40:58.174300Z","shell.execute_reply.started":"2022-08-03T00:40:58.169697Z","shell.execute_reply":"2022-08-03T00:40:58.173041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# set with all the region in ascending order\nregions = sorted(set(housing_df[\"region\"]))\n\n# Initialize figure, plots , buttons and default region to be displayed\nfig1=go.Figure()\nregion_plot_names = []\nbuttons=[]\ndefault_region = regions[0]\n\nfor region_name in regions:\n    reg_df = constn_df[constn_df['region']== region_name]\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.total_constn_complete,  \n                        mode='lines + markers', name='Construction Completed',\n                            visible=(region_name==default_region)))\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.completed_but_unabsorbed_homes,\n                            mode='lines+markers', name='Units unabsorbed',\n                            visible=(region_name==default_region)))\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.res_building_permit,  \n                        mode='lines+markers', name='residential building Permit',\n                            visible=(region_name==default_region)))\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.total_constn_start,\n                            mode='lines+markers', name='Construction Started',\n                            visible=(region_name==default_region)))\n    \n    region_plot_names.extend([region_name]*4)\n    \nfor region_name in regions:\n    buttons.append(dict(method='update',\n                        label=region_name,\n                        args = [{'visible': [region_name==r for r in region_plot_names]}]))\n    \n# Add dropdown menus to the figure\nfig1.update_layout(title = \"Construction Started vs Completed Vs Unabsorbed by year Vs Residential Permit for  Census Metropolitan Areas - Canada\",\n                  yaxis_title = \"number of dwelling units\",\n                  xaxis_title = \"Year\",\n                  showlegend=True, \n                  updatemenus=[{\"buttons\": buttons, \n                                \"direction\": \"down\", \n                                \"active\": regions.index(default_region), \n                                \"showactive\": True, \n                                \n                                \"x\": 0.5, \n                                \"y\": 1.15}\n                              ])\nfig1.show()\n    ","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:58.708070Z","iopub.execute_input":"2022-08-03T00:40:58.708933Z","iopub.status.idle":"2022-08-03T00:40:58.893439Z","shell.execute_reply.started":"2022-08-03T00:40:58.708883Z","shell.execute_reply":"2022-08-03T00:40:58.892536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* In almost all the places, the unabsorbed homes are very low compared to the new homes constructed every year and residential building permit. This shows that demand is pretty high.\n\n* In all regions there is a big drop in construction in 1995 and 2008\n    - 1990 - 1992 was recession, the period of economic downturn affecting much of the Western world( Canada was affected more than US).\n    - 2008 housing crisis in US, not that much influence in Canada, the construction started raising from 2010\n * Number of Constructions started, completed and building permits closely arround the same range. \n * When comparing unabsorbed homes(construction finished but not sold yet) with the total number of completed units, unabsorbed homes are less.The unabsorbed units are very low, which shows the demand is very high","metadata":{}},{"cell_type":"code","source":"# provinces = ['alberta', 'ontario', 'quebec','prince_edward', 'manitoba','new_brunswick', 'saskatchewan', 'nova_scotia' ]","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:40:59.759912Z","iopub.execute_input":"2022-08-03T00:40:59.760753Z","iopub.status.idle":"2022-08-03T00:40:59.766089Z","shell.execute_reply.started":"2022-08-03T00:40:59.760703Z","shell.execute_reply":"2022-08-03T00:40:59.764984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Population growth vs Construction growth","metadata":{}},{"cell_type":"code","source":"housing_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:01.070052Z","iopub.execute_input":"2022-08-03T00:41:01.070458Z","iopub.status.idle":"2022-08-03T00:41:01.078277Z","shell.execute_reply.started":"2022-08-03T00:41:01.070421Z","shell.execute_reply":"2022-08-03T00:41:01.076977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rate_df= housing_df[ ['year','completed', 'rental_vacancy_rate', 'rental_avilability_rate', 'vacancy_rate_seniors', 'vacancy_rate_condo',\n       'HPI_change', 'CPI_change', 'owned_accommodation_costs_change','rental_accommodation_costs_change', 'migration','population', 'region'] ]","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:01.612980Z","iopub.execute_input":"2022-08-03T00:41:01.613824Z","iopub.status.idle":"2022-08-03T00:41:01.620212Z","shell.execute_reply.started":"2022-08-03T00:41:01.613777Z","shell.execute_reply":"2022-08-03T00:41:01.619155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rate_change_df = rate_df.copy()\n#completed_percent = rate_change_df.loc[:, ('completed')].pct_change()\n#df = df.sort_values(['Item', 'Year']).reset_index(drop=True)\nrate_change_df.sort_values('year').reset_index(drop=True)\nconstruction_percent = rate_change_df.groupby('region', sort=False)['completed'].apply(\n                                    lambda x: x.pct_change())\npopn_percent = rate_change_df.groupby('region', sort=False)['population'].apply(\n                                    lambda x: x.pct_change())\nmigration_percent = rate_change_df.groupby('region', sort=False)['migration'].apply(\n                                    lambda x: x.pct_change())\n#new_percent = (rate_change_df.groupby('region')['completed'].apply(pd.Series.pct_change) )\n\n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:02.229064Z","iopub.execute_input":"2022-08-03T00:41:02.229708Z","iopub.status.idle":"2022-08-03T00:41:02.299633Z","shell.execute_reply.started":"2022-08-03T00:41:02.229672Z","shell.execute_reply":"2022-08-03T00:41:02.298530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rate_change_df.loc[:,'construction_compn_rate'] = construction_percent \nrate_change_df.loc[:,'popn_rate'] = popn_percent \nrate_change_df.loc[:,'migration_rate'] = migration_percent \n# rate_change_df.loc[:,'new_construction_compn_rate'] = new_percent \n#.to_numpy()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:02.832410Z","iopub.execute_input":"2022-08-03T00:41:02.832894Z","iopub.status.idle":"2022-08-03T00:41:02.841625Z","shell.execute_reply.started":"2022-08-03T00:41:02.832858Z","shell.execute_reply":"2022-08-03T00:41:02.840580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rate_change_df[rate_change_df['region'] == 'ontario']","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:03.785029Z","iopub.execute_input":"2022-08-03T00:41:03.785437Z","iopub.status.idle":"2022-08-03T00:41:03.824469Z","shell.execute_reply.started":"2022-08-03T00:41:03.785399Z","shell.execute_reply":"2022-08-03T00:41:03.823229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# rate_change_df.drop(['completed', 'population', 'migration'], axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:04.757426Z","iopub.execute_input":"2022-08-03T00:41:04.758320Z","iopub.status.idle":"2022-08-03T00:41:04.763206Z","shell.execute_reply.started":"2022-08-03T00:41:04.758277Z","shell.execute_reply":"2022-08-03T00:41:04.761919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# rate_change_df = pd.DataFrame( data, columns=['prod_desc','activity_month','prod_count'] )\n \n# product_df['pct_ch'] = product_df.groupby('prod_desc')['prod_count'].pct_change() + 1\n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:07.485014Z","iopub.execute_input":"2022-08-03T00:41:07.485737Z","iopub.status.idle":"2022-08-03T00:41:07.490661Z","shell.execute_reply.started":"2022-08-03T00:41:07.485686Z","shell.execute_reply":"2022-08-03T00:41:07.489532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# set with all the region in ascending order\nregions = sorted(set(rate_change_df[\"region\"]))\n\n# Initialize figure, plots , buttons and default region to be displayed\nfig1=go.Figure()\nregion_plot_names = []\nbuttons=[]\ndefault_region = regions[0]\n\nfor region_name in regions:\n    reg_df = rate_change_df[rate_change_df['region']== region_name]\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.construction_compn_rate,  \n                        mode='lines + markers', name='Rate of Construction Completed',\n                            visible=(region_name==default_region)))\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.popn_rate,\n                            mode='lines+markers', name='Population growth rate',\n                            visible=(region_name==default_region)))\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.migration_rate,  \n                        mode='lines+markers', name='Immigration rate',\n                            visible=(region_name==default_region)))\n    \n#     fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.HPI_change,\n#                             mode='lines+markers', name='HPI rate',\n#                             visible=(region_name==default_region)))\n    \n    region_plot_names.extend([region_name]*3)\n    \nfor region_name in regions:\n    buttons.append(dict(method='update',\n                        label=region_name,\n                        args = [{'visible': [region_name==r for r in region_plot_names]}]))\n    \n# Add dropdown menus to the figure\nfig1.update_layout(title = \"Pecentage of Construction, Population and Immigration  for  Census Metropolitan Areas - Canada\",\n                  yaxis_title = \"Percentage change\",\n                  xaxis_title = \"Year\",\n                  showlegend=True, \n                  updatemenus=[{\"buttons\": buttons, \n                                \"direction\": \"down\", \n                                \"active\": regions.index(default_region), \n                                \"showactive\": True, \n                                \n                                \"x\": 0.5, \n                                \"y\": 1.15}\n                              ])\nfig1.show()\n    ","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:08.377309Z","iopub.execute_input":"2022-08-03T00:41:08.378649Z","iopub.status.idle":"2022-08-03T00:41:08.508092Z","shell.execute_reply.started":"2022-08-03T00:41:08.378589Z","shell.execute_reply":"2022-08-03T00:41:08.506842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The graph above is not a good comparison. \n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Population vs total construction","metadata":{}},{"cell_type":"code","source":"# set with all the region in ascending order\nregions = sorted(set(housing_df[\"region\"]))\n\n# Initialize figure, plots , buttons and default region to be displayed\nfig1=go.Figure()\nregion_plot_names = []\nbuttons=[]\ndefault_region = regions[0]\n\nfor region_name in regions:\n    reg_df = rate_change_df[housing_df['region']== region_name]\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.population*1000,  \n                        mode='lines + markers', name='Population ',\n                            visible=(region_name==default_region)))\n    \n    fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.migration,\n                            mode='lines+markers', name='Migration',\n                            visible=(region_name==default_region)))\n    \n#     fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.migration_rate,  \n#                         mode='lines+markers', name='Immigration rate',\n#                             visible=(region_name==default_region)))\n    \n#     fig1.add_trace(go.Scatter(x=reg_df.year, y=reg_df.HPI_change,\n#                             mode='lines+markers', name='HPI rate',\n#                             visible=(region_name==default_region)))\n    \n    region_plot_names.extend([region_name]*2)\n    \nfor region_name in regions:\n    buttons.append(dict(method='update',\n                        label=region_name,\n                        args = [{'visible': [region_name==r for r in region_plot_names]}]))\n    \n# Add dropdown menus to the figure\nfig1.update_layout(title = \"Construction Vs Population for  Census Metropolitan Areas - Canada\",\n                  yaxis_title = \"Count\",\n                  xaxis_title = \"Year\",\n                  showlegend=True, \n                  updatemenus=[{\"buttons\": buttons, \n                                \"direction\": \"down\", \n                                \"active\": regions.index(default_region), \n                                \"showactive\": True, \n                                \n                                \"x\": 0.5, \n                                \"y\": 1.15}\n                              ])\nfig1.show()\n    ","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:11.409434Z","iopub.execute_input":"2022-08-03T00:41:11.409866Z","iopub.status.idle":"2022-08-03T00:41:11.518603Z","shell.execute_reply.started":"2022-08-03T00:41:11.409832Z","shell.execute_reply":"2022-08-03T00:41:11.517443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plotting Correlation matrix for Ontario, Quebec and Alberta(top 3 provinces)","metadata":{}},{"cell_type":"code","source":"\ncorr_mat = constn_df[ constn_df['region'] == 'ontario'].corr()\nfig = px.imshow(round(corr_mat,2), text_auto=True, height = 1000, width = 1000, title = \"Correlation of construction data in Ontario\")\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:16.887404Z","iopub.execute_input":"2022-08-03T00:41:16.887873Z","iopub.status.idle":"2022-08-03T00:41:16.965014Z","shell.execute_reply.started":"2022-08-03T00:41:16.887832Z","shell.execute_reply":"2022-08-03T00:41:16.963766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_mat['HPI_change'].sort_values(ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:19.209969Z","iopub.execute_input":"2022-08-03T00:41:19.210800Z","iopub.status.idle":"2022-08-03T00:41:19.220828Z","shell.execute_reply.started":"2022-08-03T00:41:19.210756Z","shell.execute_reply":"2022-08-03T00:41:19.219791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_mat = constn_df[ constn_df['region'] == 'quebec'].corr()\nfig = px.imshow(round(corr_mat,2), text_auto=True, height = 1000, width = 1000, title = \"Correlation of construction data in Quebec\")\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:19.321101Z","iopub.execute_input":"2022-08-03T00:41:19.322209Z","iopub.status.idle":"2022-08-03T00:41:19.378906Z","shell.execute_reply.started":"2022-08-03T00:41:19.322164Z","shell.execute_reply":"2022-08-03T00:41:19.377561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_mat['HPI_change'].sort_values(ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:21.247934Z","iopub.execute_input":"2022-08-03T00:41:21.248836Z","iopub.status.idle":"2022-08-03T00:41:21.259265Z","shell.execute_reply.started":"2022-08-03T00:41:21.248776Z","shell.execute_reply":"2022-08-03T00:41:21.257799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_mat = constn_df[ constn_df['region'] == 'alberta'].corr()\nfig = px.imshow(round(corr_mat,2), text_auto=True, height = 1000, width = 1000, title = \"Correlation of construction data in Alberta\")\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:21.404574Z","iopub.execute_input":"2022-08-03T00:41:21.405409Z","iopub.status.idle":"2022-08-03T00:41:21.469343Z","shell.execute_reply.started":"2022-08-03T00:41:21.405359Z","shell.execute_reply":"2022-08-03T00:41:21.467967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_mat['HPI_change'].sort_values(ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:23.287213Z","iopub.execute_input":"2022-08-03T00:41:23.287781Z","iopub.status.idle":"2022-08-03T00:41:23.298507Z","shell.execute_reply.started":"2022-08-03T00:41:23.287743Z","shell.execute_reply":"2022-08-03T00:41:23.297416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"rental ,total_constn_complete,total_constn_start, completed_but_unabsorbed_homes has pretty good correlation with HPI","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Code taken from Ankit's notebook\n#converting years into integer\n# df_master = housing_df.copy()\n\nfrom datetime import datetime\n\n\nhousing_df = housing_df[housing_df['year'] != 1993.1].astype({'year': int})\n\n#converting year into datetime\nhousing_df['year'] = housing_df['year'].transform(lambda x : datetime.strptime(str(x), '%Y'))\n\nhousing_df.year.unique()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:25.853444Z","iopub.execute_input":"2022-08-03T00:41:25.854101Z","iopub.status.idle":"2022-08-03T00:41:25.879988Z","shell.execute_reply.started":"2022-08-03T00:41:25.854063Z","shell.execute_reply":"2022-08-03T00:41:25.879025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Code taken from Ankit's notebook\nimport plotly.express as px\nimport plotly.graph_objects as go\n\nfig = px.bar(housing_df, x = housing_df['year'])\nbuttonlist1 = []\n\nfor col in housing_df.columns:\n    buttonlist1.append(\n        dict(\n        args = ['y', [housing_df[str(col)]]],\n        label = str(col),\n        method ='restyle'\n        ))\n\nfig.update_layout(\n        title = \"Housing features by year\",\n        yaxis_title = \"value\",\n        xaxis_title = \"Year\",\n        updatemenus = [\n            go.layout.Updatemenu(\n            buttons = buttonlist1,\n            direction = \"down\",\n            pad = {\"r\":10, \"t\":10},\n            showactive = True,\n            x = 0.1,\n            xanchor = \"left\",\n            y = 1.1,\n            yanchor = \"top\"\n            )\n        ],\n    autosize = True\n)\n\n","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:31.834916Z","iopub.execute_input":"2022-08-03T00:41:31.835783Z","iopub.status.idle":"2022-08-03T00:41:32.365154Z","shell.execute_reply.started":"2022-08-03T00:41:31.835724Z","shell.execute_reply":"2022-08-03T00:41:32.363920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The total dwelling market - “intended market” is the tenure (own or rent) in which the unit is being offered. \n\n--\n\nChanging the column name just for my understanding\n","metadata":{}},{"cell_type":"code","source":"# 'single_detached': 'detached_start',\n#                           'multiple'       : 'multiple_start',\n#                           'semi_detached'  : 'semi_start', \n#                           'row'            : 'row_start'\n#                           'apartment'      : 'appartment_start'","metadata":{"execution":{"iopub.status.busy":"2022-08-03T00:41:34.814148Z","iopub.execute_input":"2022-08-03T00:41:34.814880Z","iopub.status.idle":"2022-08-03T00:41:34.822067Z","shell.execute_reply.started":"2022-08-03T00:41:34.814834Z","shell.execute_reply":"2022-08-03T00:41:34.820789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# constn_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-24T00:31:36.115551Z","iopub.status.idle":"2022-07-24T00:31:36.116059Z","shell.execute_reply.started":"2022-07-24T00:31:36.115816Z","shell.execute_reply":"2022-07-24T00:31:36.115835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reference:\n1. https://eppdscrmssa01.blob.core.windows.net/cmhcprodcontainer/sf/project/archive/housing_markets/housinginformationmonthly/61504_2019_m10.pdf\n\n2. Survey methodologies by CMHC - https://www.cmhc-schl.gc.ca/en/professionals/housing-markets-data-and-research/housing-research/surveys/methods/methodologies-starts-completions-market-absorption-survey\n\n3. https://www.analyticsvidhya.com/blog/2021/06/power-of-interpolation-in-python-to-fill-missing-values/\n\n4. https://github.com/OmdenaAI/philadelphia-climate-change-buildings/blob/main/src/tasks/task-2-EDA/task-2-EDA.ipynb\n\n5. https://www.kaggle.com/code/benhamner/python-plotly-dropdown-demo/notebook","metadata":{}}]}