{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"It might be interesting to see where the deals are happening in geographic space. Here I wanted to show the deals on an interactive map and color code them by deal probability. \n\nI also wanted to get more familiar with the PyViz tools, which allow for large scale plotting. By combining several of these tools you can plot all 1.5M deals in the train set and interact with it in your browser, even on a plain laptop. That's nothing by the way - I saw a demonstration using 1B data points! https://anaconda.org/jbednar/osm-1billion/notebook\n\nHere are two static views of the map, one overall and one zoomed in around Stavropol. You can see the effect of terrain on population and deals. This view uses a 'reversed fire' color scheme with yellow showing low deal probabilities and dark red showing the highest. \n\nThe map down below the code should be fully interactive. You can pan, zoom, etc. Getting this all to work in Kaggle kernels is still a bit difficult though. The map tiles are loading, which is great, but the datashader part isn't adjusting the pixels for zoomed-in views. You may see big fat pixels when zooming in as opposed to the nice adjusted ones like in the static image.\n\n![map](https://s3.amazonaws.com/nonwebstorage/russia.png)\n\n![zoomed](https://s3.amazonaws.com/nonwebstorage/russia+zoom.png)"},{"metadata":{"_cell_guid":"8e46ac3b-b7fb-4089-befa-a6bcee3d5ee6","_uuid":"bb490b5e9eaa50046d363d95e51106aed6efdb25"},"cell_type":"markdown","source":"Here's the code to make the map. Getting coordinates from google maps doesn't work directly here so I use a csv file with the results. "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom functools import partial\n\nimport colorcet as cc\nfrom bokeh.models import BoxZoomTool\nfrom bokeh.plotting import figure, output_notebook, show\nfrom bokeh.tile_providers import STAMEN_TERRAIN_RETINA\nimport datashader as ds\nfrom datashader import transfer_functions as txf\nfrom datashader.bokeh_ext import InteractiveImage\nfrom datashader.utils import export_image","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ebdf7fe0a4f2ea44ffee6c2a09f44dd932a9fe2","_cell_guid":"4f17f1d2-dabe-4719-9e84-03309805e652","_kg_hide-output":true,"trusted":true,"collapsed":true},"cell_type":"code","source":"#import data\ntrain = pd.read_csv('../input/avito-demand-prediction/train.csv', usecols = ['city',  'region', 'deal_probability'])\ntrain['location'] = train['city'] + ', ' + train['region']\n\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"_uuid":"d51e5c4cb3668ae5129d592593f955f24d315dd8"},"cell_type":"markdown","source":"Here's the code to make the map. Getting coordinates from google maps doesn't work directly here so I use a csv file with the results. "},{"metadata":{"_uuid":"b1406c6e2280805720b7077916110c4da1b8bc15","_kg_hide-input":true,"_cell_guid":"25724830-ddaf-4323-b55a-98ee68ed87c9","_kg_hide-output":true,"collapsed":true,"trusted":true},"cell_type":"code","source":"# # get coordinates from Google - for running locally\n\n# import googlemaps\n# gmaps = googlemaps.Client(key='yourAPIkey')\n# locations = train['location'].unique()\n\n# queries = []\n# for l in tqdm(locations):\n#     coords = gmaps.geocode(l, language='ru', region='ru')\n#     queries.append(coords)\n\n# coordlist = []\n# for i in range(len(queries)):\n#     if not queries[i]:\n#         coordlist.append([50, 50])\n#     else:\n#         q=queries[i][0]\n#         qpair = (list(q['geometry']['location'].values()))\n#         coordlist.append(qpair)\n\n# lats = [c[0] for c in coordlist]\n# lons = [c[1] for c in coordlist]\n\n# locs_df = pd.DataFrame(np.column_stack([locations, lons, lats]), columns = ['location', 'lon', 'lat'])\n\n# locs_df['lon'] = pd.to_numeric(locs_df['lon'])\n# locs_df['lat'] = pd.to_numeric(locs_df['lat'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d80d5159292e0ff46fe714111b93a29880d298a9"},"cell_type":"code","source":"# get coordinates from file\nlocs_df = pd.read_csv('../input/russian-cities/city_latlons.csv')\n\n# merge dataframes and convert to Mercator\ntrain = train.merge(locs_df, how='left', on='location')\n\ndef merc_from_arrays(lons, lats):\n    r_major = 6378137.000\n    x = r_major * np.radians(lons)\n    scale = x/lons\n    y = 180.0/np.pi * np.log(np.tan(np.pi/4.0 + lats * (np.pi/180.0)/2.0)) * scale\n    return (x, y)\n\ntrain['deal_x'], train['deal_y'] = merc_from_arrays(train['lon'].values, train['lat'].values)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e896711a-32ec-436f-9934-a603c304595f","_uuid":"b58495301cbe0586026223fb637ecd8a48e7c202","trusted":true},"cell_type":"code","source":"# make the map\noutput_notebook()\nRU = x_range, y_range = ((2050000, 12720000), (5250000, 12000000))\nplot_width  = int(750)\nplot_height = int(plot_width//1.6)\n\ndef base_plot(tools='pan,wheel_zoom,reset',plot_width=plot_width, plot_height=plot_height, **plot_args):\n    p = figure(tools=tools, plot_width=plot_width, plot_height=plot_height,\n        x_range=x_range, y_range=y_range, outline_line_color=None,\n        min_border=0, min_border_left=0, min_border_right=0,\n        min_border_top=0, min_border_bottom=0, **plot_args)\n    p.axis.visible = False\n    p.xgrid.grid_line_color = None\n    p.ygrid.grid_line_color = None\n    p.add_tools(BoxZoomTool(match_aspect=True))\n    return p\n\nbackground = \"black\"\nexport = partial(export_image, export_path=\"export\", background=background)\n\ndef colorized_images(x_range, y_range, w=plot_width, h=plot_height):   \n    cvs = ds.Canvas(plot_width=w, plot_height=h, x_range=x_range, y_range=y_range)\n    agg = cvs.points(train, 'deal_x', 'deal_y', ds.count('deal_probability'))   # reference to data\n    img = txf.shade(agg, cmap=list(reversed(cc.fire)), how='eq_hist')   \n    return txf.dynspread(img, threshold=0.3, max_px=4)\n\np = base_plot(background_fill_color=background)\np.add_tile(STAMEN_TERRAIN_RETINA)\nexport(colorized_images(*RU),\"Avito_Deals\")\nInteractiveImage(p, colorized_images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"8095d49d8821e062ffbf4ff6ddc6ac143639e81a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}