{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Bias Correction**\n\nThe excellent notebooks [GSDC - Bias EDA](https://www.kaggle.com/code/saitodevel01/gsdc-bias-eda) & [GSDC - Bias Correction](https://www.kaggle.com/code/saitodevel01/gsdc-bias-correction) by @saitodevel01, written for the previous GSDC competition in 2021, demonstrate how the positions of the phone and driver in the car create biases between the recorded and actual positions in both the front-back and left-right directions.\n\nIn outline, the phone has better sightlines to satellites located in front of the car, since it is placed close to the to front windscreen, whereas sightlines of signals from satellites located behind the car may be compromised. Similarly, the driver sits to the left of the phone (these data came from the USA) and may block signals from satellites located to the left, whereas those to the right should have better communication. This would be expected to lead to an artefactual error locating the phone too far forward and too far to the right; hence the anticipated correction would move the phone backwards and leftwards.\n\nI have analysed @saitodevel01's work from the 2021 competition, here is an example plot showing @saitodevel01's latitude correction against the cosine of the angle between the car's current direction and due north:\n\n<img src=\"https://i.imgur.com/HjF7Nmh.png\" width=\"400\" />\n\nThe corresponding plot for longitude looks broadly similar in form. My simplification is to treat the ellipse as the signal I want to model, and ignore all other points. That leads to a constant \"lever arm\" type correction of these biases, which I express as as simple trigonometric functions of the direction the car is pointing in any one time step. Thus, I can compute latitude and longitude corrections for each predicted position. \n\nMy analysis of the prior work suggests that the predicted uniform generic correction is:\n\n40 cm from front to back;\n27 cm from right to left.\n\nI added these corrections corrections to the originally predicted latitude and longitude to generate a trial postprocessed submission.\n\nHowever, some trial and error showed that the best public LB scores come from a somewhat larger front-back correction of about 1.3 times that previously suggested. This works out as about 52 cm front to back correction.\n\nFor the left-right correction, surprisingly the best empirical LB results came from a small correction of about 8 cm from left to right, that is in the counterintuitive direction. However, for now, I will set this generic left-right correction to zero.\n\nThus, I first implement the generic front-back correction.","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('../input/decimeter-challenge-2-submissions/2682_rtklib_submission.csv')\nsub = sub.sort_values(['tripId','UnixTimeMillis'])\nsub1 = pd.read_csv('../input/decimeter-challenge-2-submissions/2682_2gbc_fblr.csv')\nsub1 = sub1.sort_values(['tripId','UnixTimeMillis'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FB = 1.3\nLR = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub['LatitudeDegrees'] = sub['LatitudeDegrees'] + FB*sub1['FB_Lat']*sub1['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + FB*sub1['FB_Lng']*sub1['Include']\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + LR*sub1['LR_Lat']*sub1['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + LR*sub1['LR_Lng']*sub1['Include']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 'Include' parameter allows certain trajectories to be excluded from the correction. In this case, 34/36 trajectories came from RKTLIB but for 2 (public LB) trips it was unable to generate satisfactory solutions. Those were replaced with my best alternative solutions for those journeys, which had already been bias-corrected. This 'Include' parameter was set to zero to exclude these from the operation.\n\nThe 'FB_Lat', 'FB_Lng', 'LR_Lat' and 'LR_Lng' parameters are the 40 or 27 cm corrections, multiplied by the appropriate cosines and converted from distance to degrees.","metadata":{}},{"cell_type":"markdown","source":"**Phone-specific Bias Correction**\n\nSome consideration of possible relevant effects led me to consider whether the bias correction might depend on the model of phone. There are various possible explanations for this. \n\nPossibly some aspect of the phone's hardware or software, or alternatively some quirk of the individual devices used in the experiment, meant that some phones were better able than others to communicate with satellites via compromised sightlines or through atmospheric or environmental interference. Hence, corrections could vary between phones.\n\nAlternatively, if each phone has a designated holder or position in the car, its results may be affected by sightlines and/or lever arm correction errors related to that position. For example. one phone holder may have better views backwards or to the left than another, depending on the extent to which the driver, seats, or experimental equipment block the view. \n\nThis seemed worth investigating, so a brief analysis was made using training set data.","metadata":{}},{"cell_type":"markdown","source":"**Training Results**\n\nDataset\t\nForward error\t-0.0159 m (median)\nLeft error\t-0.3248 m (median)\n\t\n\t\nGooglePixel4\t\nForward error\t-0.1798\nRelative to dataset\t-0.1639\nLeft error\t-0.3383\nRelative to dataset\t-0.0136\n\t\n\t\nGooglePixel4XL\t\nForward error\t0.1375\nRelative to dataset\t0.1535\nLeft error\t-0.2301\nRelative to dataset\t0.0947\n\t\n\t\nGooglePixel5\t\nForward error\t-0.7068\nRelative to dataset\t-0.6909\nLeft error\t0.2203\nRelative to dataset\t0.5450\n\t\n\t\nSamsungGalaxyS20Ultra\t\nForward error\t0.1469\nRelative to dataset\t0.1628\nLeft error\t-1.0847\nRelative to dataset\t-0.7599\n\t\n\t\nXiaomiMi8\t\nForward error\t0.4559\nRelative to dataset\t0.4718\nLeft error\t-1.0740\nRelative to dataset\t-0.7493","metadata":{}},{"cell_type":"markdown","source":"So it seems that it is worth trying a phone-specific set of corrections. Firstly, we will look at the front-back corrections. For the front-back component, I will only accept these corrections if the resulting overall correction is in the same direction as the training results, and the specific correction is no larger than the original generic correction. We can do this for the four phone models that are common to the training and test data; GooglePixel4, GooglePixel5, XiaomiMi8 and SamsungGalaxyS20Ultra. \n\nFor the left-right correction, based on my previous experience of bias correction, I suggest the following. I will accept changes to the left up to the 27 cm of the original generic lever arm correction or up to the training-suggested value if greater. In the 'wrong' direction I will accept changes up to the 8 cm of my first fit or up to the training-suggested value if greater..\n\nThere is some trial and error involved here; the direction and size constraints mentioned above are designed to limit public-score chasing.","metadata":{}},{"cell_type":"code","source":"sub4 = pd.read_csv('../input/decimeter-challenge-2-submissions/GooglePixel4_x2682_fblri.csv')\nsub5 = pd.read_csv('../input/decimeter-challenge-2-submissions/GooglePixel5_x2682_fblri.csv')\nsub8 = pd.read_csv('../input/decimeter-challenge-2-submissions/XiaomiMi8_x2682_fblri.csv')\nsub20 = pd.read_csv('../input/decimeter-challenge-2-submissions/SamsungGalaxyS20Ultra_x2682_fblri.csv')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FBG4 = -1.3 # 0.0 -> 2.595; 0.2 -> 2.594; 0.4 -> 2.591; 0.6 -> 2.588;  0.8 -> 2.584; -1.0 -> 2.581; -1.3 -> 2.577 (best)\nFBG5 = 0.0 # 0.0 -> 2.577 (best); 0.15 (worse); 0.3 -> 2.577 (worse)\nFBX8 = 0.6 # 0.0 -> 2.577; 0.3 -> 2.566; 0.6 -> 2.561 (best); 0.8 -> 2.563\nFBS20 = 0.2 # 0.0 -> 2.561; 0.2 -> 2.560 (best); 0.4 -> 2.560","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub['LatitudeDegrees'] = sub['LatitudeDegrees'] + FBG4*sub4['FB_Lat']*sub4['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + FBG4*sub4['FB_Lng']*sub4['Include']\n\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + FBG5*sub5['FB_Lat']*sub5['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + FBG5*sub5['FB_Lng']*sub5['Include']\n\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + FBX8*sub8['FB_Lat']*sub8['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + FBX8*sub8['FB_Lng']*sub8['Include']\n\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + FBS20*sub20['FB_Lat']*sub20['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + FBS20*sub20['FB_Lng']*sub20['Include']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus the front-back corrections in the forward direction are: <br>\n\nGooglePixel4: -52 cm (generic) + 52 cm (specific) = 0 cm <br>\nGooglePixel5: -52 cm (generic) + 0 cm (specific) = -52 cm <br>\nXiaomiMi8: -52 cm (generic) -  24 cm (specific) = -76 cm <br>\nSamsungGalaxyS20Ultra: -52 cm (generic) -  8 cm (specific) = -60 cm <br>\nGooglePixel6Pro: -52 cm (generic) = -52 cm  (no specific correction as no training data)<br>","metadata":{}},{"cell_type":"markdown","source":"Now we apply phone-specific left-right corrections.","metadata":{}},{"cell_type":"code","source":"LRG4 = 0.2 # 0.0 -> 2.560; 0.2 -> 2.560 (best); 0.4 -> 2.561\nLRG5 = -2.7 # 0.0 -> 2.560; -0.2 -> 2.554 ; -1.0 -> 2.534 ; -2.0 -> 2.520; -2.5 -> 2.515; -2.7 -> 2.515 (best)\nLRX8 = -0.15 # -0.3 -> 2.514; -0.15 -> 2.514 (best); 0.0 -> 2.515; 0.3 -> 2.518; 1.0 -> 2.528\nLRS20 = 1.8 # 0.0 -> 2.514; 0.3 -> 2.509; 1.0 -> 2.496; 1.5 -> 2.488; 1.8 -> 2.486 (best); 2.0 -> 2.486","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus the left-right corrections in the left-facing direction are: <br>\n\nGooglePixel4: 5 cm (training & prior fits suggest -8 to 34 cm) <br>\nGooglePixel5: -74 cm (training & prior fits suggest -22 to 27 cm) <br>\nXiaomiMi8: -3 cm (training & prior fits suggest -8 to 107 cm) <br> \nSamsungGalaxyS20Ultra: 49 cm (training suggests -8 to 108 cm) <br>\nGooglePixel6Pro: 0 cm (no specific correction as no training data)\n\nThis suggests that LRG5 needs to be clipped:","metadata":{}},{"cell_type":"code","source":"LRG5 = -0.8","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub['LatitudeDegrees'] = sub['LatitudeDegrees'] + LRG4*sub4['LR_Lat']*sub4['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + LRG4*sub4['LR_Lng']*sub4['Include']\n\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + LRG5*sub5['LR_Lat']*sub5['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + LRG5*sub5['LR_Lng']*sub5['Include']\n\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + LRX8*sub8['LR_Lat']*sub8['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + LRX8*sub8['LR_Lng']*sub8['Include']\n\nsub['LatitudeDegrees'] = sub['LatitudeDegrees'] + LRS20*sub20['LR_Lat']*sub20['Include']\nsub['LongitudeDegrees'] = sub['LongitudeDegrees'] + LRS20*sub20['LR_Lng']*sub20['Include']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)\nsub.head(10)","metadata":{},"execution_count":null,"outputs":[]}]}