{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"The competition hosts want to use this doc to record tips, issues, and notes about the datasets. This doc will be evolving as the competition goes. These notes are not mandatory to know for you to tackle the competition, but they might help you understand the competition better. \n\nSo far, the hosts are aware of three tips that can make the matching process between the `_derived.csv` and the `_GnssLog.txt` more convenient. Note that it's not a must to match them, but each of these two files has information that the other file doesn't contain, so merging them may give you new features. These tips are listed as follows. After the list, a function to match the two files is provided. \n\n**Tip 1**: `millisSinceGpsEpoch` in `_derived.csv` refers to the timestamp of the next epoch not the current epoch. \n\n- **Important Note**: this may affect your final results if you are only using `_derived.csv` in your positioning algorithm.\n\n**Tip 2**: QZSS svids in `_GnssLog.txt` do not equal the svids in `_derived.csv`. The former is in the range of 193-200, as defined in https://developer.android.com/reference/android/location/GnssStatus#getSvid(int). The latter is converted values ranging from 1 to 4, as per https://qzss.go.jp/en/technical/satellites/index.html#QZSS. \n\n**Tip 3**: `CarrierFrequencyHz` is provided in `_GnssLog.txt` but not in `_derived.csv`. This makes the matching process a lot harder. Actually, the carrier frequency field is digested inside the `SignalType` field in the `_derived.csv`. A mapping function is provided below.\n\n**Tip 4**: The altitude `heightAboveWgs84EllipsoidM` in the ground_truth.csv has a bias of ~61 meters. For example, if you see 35.00 meters, its real value should be about -26.00 meters. However, the evaluation of this competition only cares about horizontal distance and will not use `heightAboveWgs84EllipsoidM`. This tip doesn't affect the performance of your positioning algorithm, but it may help you for debugging if you are computing a 3D position.\n\n**Tip 5**: Some rows in `_derived.csv` are duplicated with their previous row and they might introduce outliers in the position solutions. They shouldn't be reported in the file. A sample code to remove them is provided here:\n\n> delta_millis = df_signals['MillisSinceGpsEpoch'] - df_signals['ReceivedSvTimeInGpsNanos'] / 1e6\n\n> where_good_signals = (delta_millis > 0) & (delta_millis < 300)\n\n> df_signals = df_signals[where_good_signals].copy()\n\n\nA suggested way addressing Tips 1-3 to match an epoch from `_derived.csv` and `_GnssLog.txt` is provided as follows:\n\n","metadata":{}},{"cell_type":"code","source":"# Assume we've loaded a dataframe from _GnssLog.txt for only lines beginning with \"Raw\", we denote this df_raw. Next, assume we've loaded a dataframe from _derived.csv. We denote this df_derived.\n\n# Create a new column in df_raw that corresponds to df_derived['MillisSinceGpsEpoch']\ndf_raw['MillisSinceGpsEpoch'] = np.floor( (df_raw['TimeNanos'] - df_raw['FullBiasNanos']) / 1000000.0).astype(int)\n\n# Change each value in df_derived['MillisSinceGpsEpoch'] to be the prior epoch.\nraw_timestamps = df_raw['MillisSinceGpsEpoch'].unique()\nderived_timestamps = df_derived['MillisSinceGpsEpoch'].unique()\n\n# The timestamps in derived are one epoch ahead. We need to map each epoch\n# in derived to the prior one (in Raw).\nindexes = np.searchsorted(raw_timestamps, derived_timestamps)\nfrom_t_to_fix_derived = dict(zip(derived_timestamps, raw_timestamps[indexes-1]))\ndf_derived['MillisSinceGpsEpoch'] = np.array(list(map(lambda v: from_t_to_fix_derived[v], df_derived['MillisSinceGpsEpoch'])))\n\n# Compute signal_type in df_raw.\n# Map from constellation id to frequencies and signals.\nCONSTEL_FREQ_TABLE = {\n    0: {'UNKNOWN': (0, 999999999999)},\n    1: {\n        'GPS_L1': (1563000000, 1587000000),\n        'GPS_L2': (1215000000, 1240000000),\n        'GPS_L5': (1164000000, 1189000000)\n    },\n    3: {\n        'GLO_G1': (1593000000, 1610000000),\n        'GLO_G2': (1237000000, 1254000000)\n    },\n    4: {\n        'QZS_J1': (1563000000, 1587000000),\n        'QZS_J2': (1215000000, 1240000000),\n        'QZS_J5': (1164000000, 1189000000)\n    },\n    5: {\n        'BDS_B1C': (1569000000, 1583000000),\n        'BDS_B1I': (1553000000, 1568990000),\n        'BDS_B2A': (1164000000, 1189000000),\n        'BDS_B2B': (1189000000, 1225000000)\n    },\n    6: {\n        'GAL_E1': (1559000000, 1591000000),\n        'GAL_E5A': (1164000000, 1189000000),\n        'GAL_E5B': (1189000000, 1218000000),\n        'GAL_E6': (1258000000, 1300000000)\n    },\n    7: {\n        'IRN_S': (2472000000, 2512000000),\n        'IRN_L5': (1164000000, 1189000000)\n    },\n}\n\ndef SignalTypeFromConstellationAndFequency(constel, freq_hz):\n  'Returns the signal type as a string for the given constellation and frequency.'\n  freqs = CONSTEL_FREQ_TABLE.get(constel, {})\n  for id_freq_range in freqs.items():\n    rng = id_freq_range[1]\n    if rng[0] <= freq_hz <= rng[1]:\n      return id_freq_range[0]\n  return 'UNKNOWN'\n\nsignal_types = itertools.chain(*[c.keys() for c in CONSTEL_FREQ_TABLE.values()])\nsig_type_cat = pd.api.types.CategoricalDtype(categories=signal_types)\ndf_raw['SignalType'] = df_raw.apply(lambda r: SignalTypeFromConstellationAndFequency(r.ConstellationType, r.CarrierFrequencyHz), axis=1).astype(sig_type_cat)\n\n# Fix QZS Svids issue. \n\n# The SVID of any QZS sat in derived may be changed. Since it may be a many to one relationship, we'll need to adjust the values in Raw.\nnew_to_old = {1:(183, 193), 2:(184, 194, 196), 3:(187, 189, 197, 199), 4:(185, 195, 200)}\n# Maps original svid to new svid for only ConstellationType=4.\nold_to_new={}\nfor new_svid, old_svids in new_to_old.items():\n  for s in old_svids:\n    old_to_new[s] = new_svid\ndf_raw['Svid'] = df_raw.apply(lambda r: old_to_new.get(r.Svid, r.Svid) if r.ConstellationType == 4 else r.Svid, axis=1)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]}]}