{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom geopy import distance","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:42.502011Z","iopub.execute_input":"2022-08-07T17:26:42.503111Z","iopub.status.idle":"2022-08-07T17:26:42.717437Z","shell.execute_reply.started":"2022-08-07T17:26:42.503003Z","shell.execute_reply":"2022-08-07T17:26:42.716219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generally, numerical features contain the same information when their values are very close and do not require special processing. For example, if we want to use a person's height to predict something about him, it is not important whether he is 1.8m or 1.799m tall. For both heights, in most cases, we expect our model to treat them equally.\n\nHowever, if we want to know whether two locations, one with coordinates (32.988, 55.291) and the other (32.987, 55.294), refer to the same point of interest on a map, this small difference is relevant. Although apparently small, this difference in the third decimal corresponds to 300m of distance, enough to determine that probably two locations do not refer to the same point of interest.","metadata":{}},{"cell_type":"code","source":"distance.distance((32.988, 55.291), (32.987, 55.294)).km","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:42.719553Z","iopub.execute_input":"2022-08-07T17:26:42.720162Z","iopub.status.idle":"2022-08-07T17:26:42.733402Z","shell.execute_reply.started":"2022-08-07T17:26:42.720128Z","shell.execute_reply":"2022-08-07T17:26:42.732218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Therefore, these high-precision features, where several of their decimal numbers are relevant, require special treatment for some types of ML models. For example, a linear regression or a neural network will not be able to correctly model this type of features.\n\nTo facilitate its use by this kind of models, it is necessary to apply some kind of preprocessing. The one proposed in this notebook is based on positional encoding applied in transformers to provide positional information to the tokens. Some related resources:\n* https://www.tensorflow.org/text/tutorials/transformer#positional_encoding\n* https://www.youtube.com/watch?v=dichIcUZfOw&ab_channel=HeduAI\n\nThis method consists of applying sinusoidal functions to the token position at different frequencies. A similar method is used in this notebook to encode latitude and longitude.","metadata":{}},{"cell_type":"code","source":"def encode_high_precision(x_hp, emb_size=20, precision=1e6):\n    m = np.exp(np.log(precision) / emb_size)\n\n    angle_freq = m ** np.arange(emb_size)\n    angle_freq = np.expand_dims(angle_freq, axis=0)\n\n    x_hp = np.expand_dims(x_hp, axis=1)\n\n    x_enc = x_hp * angle_freq\n    x_enc[:, 0::2] = np.cos(x_enc[:, 0::2])\n    x_enc[:, 1::2] = np.sin(x_enc[:, 1::2])\n\n    return x_enc\n\n\ndef plot(latitudes_encoded, latitudes):\n    plt.figure(figsize=(16, 4))\n    plt.pcolormesh(latitudes_encoded, cmap=\"RdBu\")\n    plt.ylabel(\"latitudes\")\n    plt.xlabel(\"embedding\")\n    plt.colorbar()\n    plt.yticks([0.5, 1.5], [latitudes[0], latitudes[1]])\n    plt.xticks([], [])\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:42.735237Z","iopub.execute_input":"2022-08-07T17:26:42.735976Z","iopub.status.idle":"2022-08-07T17:26:42.746439Z","shell.execute_reply.started":"2022-08-07T17:26:42.735918Z","shell.execute_reply":"2022-08-07T17:26:42.745367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe how in the first positions of the embedding, which are the ones that use less frequency and capture less granularity, the values are similar. As we move to the right the sensitivity to small changes increases and the embedded values are different. In this way the difference between the feature values at different scales is captured.","metadata":{}},{"cell_type":"code","source":"latitudes = [78.988, -31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-07T17:26:42.748671Z","iopub.execute_input":"2022-08-07T17:26:42.749553Z","iopub.status.idle":"2022-08-07T17:26:43.017406Z","shell.execute_reply.started":"2022-08-07T17:26:42.749497Z","shell.execute_reply":"2022-08-07T17:26:43.015996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latitudes = [35.988, 31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:43.019366Z","iopub.execute_input":"2022-08-07T17:26:43.020239Z","iopub.status.idle":"2022-08-07T17:26:43.226316Z","shell.execute_reply.started":"2022-08-07T17:26:43.020203Z","shell.execute_reply":"2022-08-07T17:26:43.225220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latitudes = [32.988, 31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:43.227552Z","iopub.execute_input":"2022-08-07T17:26:43.228880Z","iopub.status.idle":"2022-08-07T17:26:43.438594Z","shell.execute_reply.started":"2022-08-07T17:26:43.228845Z","shell.execute_reply":"2022-08-07T17:26:43.437136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latitudes = [31.5, 31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:43.440365Z","iopub.execute_input":"2022-08-07T17:26:43.441027Z","iopub.status.idle":"2022-08-07T17:26:43.764624Z","shell.execute_reply.started":"2022-08-07T17:26:43.440993Z","shell.execute_reply":"2022-08-07T17:26:43.763772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latitudes = [31.9, 31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:43.765826Z","iopub.execute_input":"2022-08-07T17:26:43.766555Z","iopub.status.idle":"2022-08-07T17:26:43.997038Z","shell.execute_reply.started":"2022-08-07T17:26:43.766523Z","shell.execute_reply":"2022-08-07T17:26:43.995658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latitudes = [31.98, 31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:43.998564Z","iopub.execute_input":"2022-08-07T17:26:43.999468Z","iopub.status.idle":"2022-08-07T17:26:44.238815Z","shell.execute_reply.started":"2022-08-07T17:26:43.999415Z","shell.execute_reply":"2022-08-07T17:26:44.237730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latitudes = [31.985, 31.987]\nlatitudes_scaled = np.array(latitudes) / 90  \nlatitudes_encoded = encode_high_precision(latitudes_scaled)\n\nplot(latitudes_encoded, latitudes)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T17:26:44.241390Z","iopub.execute_input":"2022-08-07T17:26:44.241866Z","iopub.status.idle":"2022-08-07T17:26:44.473849Z","shell.execute_reply.started":"2022-08-07T17:26:44.241833Z","shell.execute_reply":"2022-08-07T17:26:44.472804Z"},"trusted":true},"execution_count":null,"outputs":[]}]}