{
  "id": 247238,
  "title": "Current methodologies benchmark analysis",
  "url": "/competitions/google-smartphone-decimeter-challenge/discussion/247238",
  "author_name": "Vlad Vaduva",
  "post_date": "2021-06-18T18:53:37.385000",
  "votes": 21,
  "comment_count": 0,
  "views": 0,
  "content": "<p>We can definitely say that this competition is different from the most of the classical Kaggle challenges. As a matter of fact, my personal machine learning approaches were not successfully so far. Instead of the machine learning, in the discussion and kernels there are several interesting techniques which I benchmarked<br>\nI have grouped this techniques into 3 categories and made a analysis of how much improvement there is in each successive step of the methods. Keep in mind that the simulation is done on training data, so it can be benchmarked vs the real coordinates and that the methodologies are successive, the output predictions of the first method are the input of the second one and so on. </p>\n<p><strong>First processing</strong> <br>\nTechniques used in this step:</p>\n<ul>\n<li>Outlier correction ( determined best threshold for consider a sample to be an outlier)</li>\n<li>Kalman filter ( determined best Kalman filter parameters)</li>\n<li>Phones mean  </li>\n</ul>\n<p>Using scikit-optimize I have run 200 iterations of finding the best parameters for outlier corrections and Kalman filter. Each iteration used 30 calls for finding the best parameters. So practically, I have run 200 times a operation of finding the best parameters where each operation uses 30 iteration to find those parameters<br>\nThe parameters which were tunned were:</p>\n<ul>\n<li>Threshold for outlier correction</li>\n<li>Noise for Kalman filter</li>\n<li>Observation noise for Kalman filter</li>\n<li>Time variable for Kalman filter</li>\n</ul>\n<p>In the figure below are the distribution of the parameters found in this 200 iterations:</p>\n<p>Threshold for outlier correction<br>\n<img src=\"https://imgur.com/EZOLKXq.png\" alt=\"https://imgur.com/EZOLKXq\"></p>\n<p>Time for Kalman filter<br>\n<img src=\"https://imgur.com/TwZ7enz.png\" alt=\"\"></p>\n<p>Noise for Kalman filter<br>\n<img src=\"https://imgur.com/gkfCN0s.png\" alt=\"\"></p>\n<p>Obs noise for Kalman filter<br>\n<img src=\"https://imgur.com/CgndCJP.jpg\" alt=\"\"></p>\n<p>Observations:</p>\n<ul>\n<li>Most of the optimum time found is on the interval upper limit, so on the next parameter tunning I will need to extend the interval, there are chances that above 3 there can be optimum values</li>\n<li>Both noises are grouped pretty good around a common value</li>\n</ul>\n<p>Score after first processing<br>\n<img src=\"https://imgur.com/prpeZv4.png\" alt=\"\"></p>\n<p>While from the 200 tries, most of the results are grouped above 4.5 and especially 4.6, there are some good combinations lower than 4.4 and even 4.3.<br>\nThe scores mean is 4.59 and the standard deviation is 0.10 which means that there are pretty closely grouped</p>\n<p><strong>Second processing</strong> <br>\nTechniques used in this step:</p>\n<ul>\n<li>removing the Samsung S20Ultra  </li>\n</ul>\n<p>As input data, were the 200 predictions which were the output of the first processing step.<br>\nIn this step, I have removed the Samsung S20Ultra phone and interpolate the predictions using the other phones. <br>\n<img src=\"https://imgur.com/qg7JZ7O.png\" alt=\"\"><br>\nThe scores mean is 4.46(a gain of 0.13 from the first step) and the standard deviation is 0.10.<br>\nWe can easily see that the whole histogram shifted with aprox. 0.1 to the left side, fact which confirms the improvement of this step.</p>\n<p><strong>Third processing</strong> <br>\nTechniques used in this step:</p>\n<ul>\n<li>predictions shift  </li>\n</ul>\n<p>In this step I have used the 200 data from the second processing, to tune with optuna the best  distance for shifting the predictions. There were used 10 trials in each of the 200 optuna searches. Let's see what were the best distances and how the score was influenced by this step.</p>\n<p>Alpha parameter of shifting<br>\n<img src=\"https://imgur.com/mn7Aw0f.png\" alt=\"\"></p>\n<p>Score<br>\n<img src=\"https://imgur.com/4cauWcF.png\" alt=\"\"></p>\n<p>The scores mean is 4.37(a gain of 0.09 from the second step) and the standard deviation is 0.09 <br>\nSo we can see that each step has better results than the previous,</p>\n<p>I have made the same successive methodology with the test set, using the best 200 parameters found on the first step (threshold for outlier, noise, observation noise, time), than removed the Samsung S20Ultra phone, and on the 3rd step use the 200 parameters for the optimum shift distance.<br>\nThan the submission was the mean for those 200 prediction. <br>\nSurprisingly, or not, this blending of 200 prediction with optimum thresholds was no better than using one submission consisting in the best from the 200 predictions (using the training set for estimating).<br>\nI am still curious if anybody had any success with blending or any form of combining multiple submission.</p>",
  "messages": [
    {
      "id": 1356142,
      "postDate": "2021-06-18T18:53:37.387Z",
      "content": "<p>We can definitely say that this competition is different from the most of the classical Kaggle challenges. As a matter of fact, my personal machine learning approaches were not successfully so far. Instead of the machine learning, in the discussion and kernels there are several interesting techniques which I benchmarked<br>\nI have grouped this techniques into 3 categories and made a analysis of how much improvement there is in each successive step of the methods. Keep in mind that the simulation is done on training data, so it can be benchmarked vs the real coordinates and that the methodologies are successive, the output predictions of the first method are the input of the second one and so on. </p>\n<p><strong>First processing</strong> <br>\nTechniques used in this step:</p>\n<ul>\n<li>Outlier correction ( determined best threshold for consider a sample to be an outlier)</li>\n<li>Kalman filter ( determined best Kalman filter parameters)</li>\n<li>Phones mean  </li>\n</ul>\n<p>Using scikit-optimize I have run 200 iterations of finding the best parameters for outlier corrections and Kalman filter. Each iteration used 30 calls for finding the best parameters. So practically, I have run 200 times a operation of finding the best parameters where each operation uses 30 iteration to find those parameters<br>\nThe parameters which were tunned were:</p>\n<ul>\n<li>Threshold for outlier correction</li>\n<li>Noise for Kalman filter</li>\n<li>Observation noise for Kalman filter</li>\n<li>Time variable for Kalman filter</li>\n</ul>\n<p>In the figure below are the distribution of the parameters found in this 200 iterations:</p>\n<p>Threshold for outlier correction<br>\n<img src=\"https://imgur.com/EZOLKXq.png\" alt=\"https://imgur.com/EZOLKXq\"></p>\n<p>Time for Kalman filter<br>\n<img src=\"https://imgur.com/TwZ7enz.png\" alt=\"\"></p>\n<p>Noise for Kalman filter<br>\n<img src=\"https://imgur.com/gkfCN0s.png\" alt=\"\"></p>\n<p>Obs noise for Kalman filter<br>\n<img src=\"https://imgur.com/CgndCJP.jpg\" alt=\"\"></p>\n<p>Observations:</p>\n<ul>\n<li>Most of the optimum time found is on the interval upper limit, so on the next parameter tunning I will need to extend the interval, there are chances that above 3 there can be optimum values</li>\n<li>Both noises are grouped pretty good around a common value</li>\n</ul>\n<p>Score after first processing<br>\n<img src=\"https://imgur.com/prpeZv4.png\" alt=\"\"></p>\n<p>While from the 200 tries, most of the results are grouped above 4.5 and especially 4.6, there are some good combinations lower than 4.4 and even 4.3.<br>\nThe scores mean is 4.59 and the standard deviation is 0.10 which means that there are pretty closely grouped</p>\n<p><strong>Second processing</strong> <br>\nTechniques used in this step:</p>\n<ul>\n<li>removing the Samsung S20Ultra  </li>\n</ul>\n<p>As input data, were the 200 predictions which were the output of the first processing step.<br>\nIn this step, I have removed the Samsung S20Ultra phone and interpolate the predictions using the other phones. <br>\n<img src=\"https://imgur.com/qg7JZ7O.png\" alt=\"\"><br>\nThe scores mean is 4.46(a gain of 0.13 from the first step) and the standard deviation is 0.10.<br>\nWe can easily see that the whole histogram shifted with aprox. 0.1 to the left side, fact which confirms the improvement of this step.</p>\n<p><strong>Third processing</strong> <br>\nTechniques used in this step:</p>\n<ul>\n<li>predictions shift  </li>\n</ul>\n<p>In this step I have used the 200 data from the second processing, to tune with optuna the best  distance for shifting the predictions. There were used 10 trials in each of the 200 optuna searches. Let's see what were the best distances and how the score was influenced by this step.</p>\n<p>Alpha parameter of shifting<br>\n<img src=\"https://imgur.com/mn7Aw0f.png\" alt=\"\"></p>\n<p>Score<br>\n<img src=\"https://imgur.com/4cauWcF.png\" alt=\"\"></p>\n<p>The scores mean is 4.37(a gain of 0.09 from the second step) and the standard deviation is 0.09 <br>\nSo we can see that each step has better results than the previous,</p>\n<p>I have made the same successive methodology with the test set, using the best 200 parameters found on the first step (threshold for outlier, noise, observation noise, time), than removed the Samsung S20Ultra phone, and on the 3rd step use the 200 parameters for the optimum shift distance.<br>\nThan the submission was the mean for those 200 prediction. <br>\nSurprisingly, or not, this blending of 200 prediction with optimum thresholds was no better than using one submission consisting in the best from the 200 predictions (using the training set for estimating).<br>\nI am still curious if anybody had any success with blending or any form of combining multiple submission.</p>",
      "rawMarkdown": "We can definitely say that this competition is different from the most of the classical Kaggle challenges. As a matter of fact, my personal machine learning approaches were not successfully so far. Instead of the machine learning, in the discussion and kernels there are several interesting techniques which I benchmarked\nI have grouped this techniques into 3 categories and made a analysis of how much improvement there is in each successive step of the methods. Keep in mind that the simulation is done on training data, so it can be benchmarked vs the real coordinates and that the methodologies are successive, the output predictions of the first method are the input of the second one and so on. \n\n**First processing** \nTechniques used in this step:\n- Outlier correction ( determined best threshold for consider a sample to be an outlier)\n- Kalman filter ( determined best Kalman filter parameters)\n- Phones mean  \n\nUsing scikit-optimize I have run 200 iterations of finding the best parameters for outlier corrections and Kalman filter. Each iteration used 30 calls for finding the best parameters. So practically, I have run 200 times a operation of finding the best parameters where each operation uses 30 iteration to find those parameters\nThe parameters which were tunned were:\n- Threshold for outlier correction\n- Noise for Kalman filter\n- Observation noise for Kalman filter\n- Time variable for Kalman filter\n\nIn the figure below are the distribution of the parameters found in this 200 iterations:\n\nThreshold for outlier correction\n![https://imgur.com/EZOLKXq](https://imgur.com/EZOLKXq.png)\n\nTime for Kalman filter\n![](https://imgur.com/TwZ7enz.png)\n\nNoise for Kalman filter\n![](https://imgur.com/gkfCN0s.png)\n\nObs noise for Kalman filter\n![](https://imgur.com/CgndCJP.jpg)\n\nObservations:\n- Most of the optimum time found is on the interval upper limit, so on the next parameter tunning I will need to extend the interval, there are chances that above 3 there can be optimum values\n- Both noises are grouped pretty good around a common value\n\nScore after first processing\n![](https://imgur.com/prpeZv4.png)\n\nWhile from the 200 tries, most of the results are grouped above 4.5 and especially 4.6, there are some good combinations lower than 4.4 and even 4.3.\nThe scores mean is 4.59 and the standard deviation is 0.10 which means that there are pretty closely grouped\n\n**Second processing** \nTechniques used in this step:\n- removing the Samsung S20Ultra  \n\nAs input data, were the 200 predictions which were the output of the first processing step.\nIn this step, I have removed the Samsung S20Ultra phone and interpolate the predictions using the other phones. \n![](https://imgur.com/qg7JZ7O.png)\nThe scores mean is 4.46(a gain of 0.13 from the first step) and the standard deviation is 0.10.\nWe can easily see that the whole histogram shifted with aprox. 0.1 to the left side, fact which confirms the improvement of this step.\n\n**Third processing** \nTechniques used in this step:\n- predictions shift  \n\nIn this step I have used the 200 data from the second processing, to tune with optuna the best  distance for shifting the predictions. There were used 10 trials in each of the 200 optuna searches. Let's see what were the best distances and how the score was influenced by this step.\n\nAlpha parameter of shifting\n![](https://imgur.com/mn7Aw0f.png)\n\nScore\n![](https://imgur.com/4cauWcF.png)\n\nThe scores mean is 4.37(a gain of 0.09 from the second step) and the standard deviation is 0.09 \nSo we can see that each step has better results than the previous,\n\nI have made the same successive methodology with the test set, using the best 200 parameters found on the first step (threshold for outlier, noise, observation noise, time), than removed the Samsung S20Ultra phone, and on the 3rd step use the 200 parameters for the optimum shift distance.\nThan the submission was the mean for those 200 prediction. \nSurprisingly, or not, this blending of 200 prediction with optimum thresholds was no better than using one submission consisting in the best from the 200 predictions (using the training set for estimating).\nI am still curious if anybody had any success with blending or any form of combining multiple submission.\n\n",
      "votes": 21
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1356142": "We can definitely say that this competition is different from the most of the classical Kaggle challenges. As a matter of fact, my personal machine learning approaches were not successfully so far. Instead of the machine learning, in the discussion and kernels there are several interesting techniques which I benchmarked\nI have grouped this techniques into 3 categories and made a analysis of how much improvement there is in each successive step of the methods. Keep in mind that the simulation is done on training data, so it can be benchmarked vs the real coordinates and that the methodologies are successive, the output predictions of the first method are the input of the second one and so on. \n\n**First processing** \nTechniques used in this step:\n- Outlier correction ( determined best threshold for consider a sample to be an outlier)\n- Kalman filter ( determined best Kalman filter parameters)\n- Phones mean  \n\nUsing scikit-optimize I have run 200 iterations of finding the best parameters for outlier corrections and Kalman filter. Each iteration used 30 calls for finding the best parameters. So practically, I have run 200 times a operation of finding the best parameters where each operation uses 30 iteration to find those parameters\nThe parameters which were tunned were:\n- Threshold for outlier correction\n- Noise for Kalman filter\n- Observation noise for Kalman filter\n- Time variable for Kalman filter\n\nIn the figure below are the distribution of the parameters found in this 200 iterations:\n\nThreshold for outlier correction\n![https://imgur.com/EZOLKXq](https://imgur.com/EZOLKXq.png)\n\nTime for Kalman filter\n![](https://imgur.com/TwZ7enz.png)\n\nNoise for Kalman filter\n![](https://imgur.com/gkfCN0s.png)\n\nObs noise for Kalman filter\n![](https://imgur.com/CgndCJP.jpg)\n\nObservations:\n- Most of the optimum time found is on the interval upper limit, so on the next parameter tunning I will need to extend the interval, there are chances that above 3 there can be optimum values\n- Both noises are grouped pretty good around a common value\n\nScore after first processing\n![](https://imgur.com/prpeZv4.png)\n\nWhile from the 200 tries, most of the results are grouped above 4.5 and especially 4.6, there are some good combinations lower than 4.4 and even 4.3.\nThe scores mean is 4.59 and the standard deviation is 0.10 which means that there are pretty closely grouped\n\n**Second processing** \nTechniques used in this step:\n- removing the Samsung S20Ultra  \n\nAs input data, were the 200 predictions which were the output of the first processing step.\nIn this step, I have removed the Samsung S20Ultra phone and interpolate the predictions using the other phones. \n![](https://imgur.com/qg7JZ7O.png)\nThe scores mean is 4.46(a gain of 0.13 from the first step) and the standard deviation is 0.10.\nWe can easily see that the whole histogram shifted with aprox. 0.1 to the left side, fact which confirms the improvement of this step.\n\n**Third processing** \nTechniques used in this step:\n- predictions shift  \n\nIn this step I have used the 200 data from the second processing, to tune with optuna the best  distance for shifting the predictions. There were used 10 trials in each of the 200 optuna searches. Let's see what were the best distances and how the score was influenced by this step.\n\nAlpha parameter of shifting\n![](https://imgur.com/mn7Aw0f.png)\n\nScore\n![](https://imgur.com/4cauWcF.png)\n\nThe scores mean is 4.37(a gain of 0.09 from the second step) and the standard deviation is 0.09 \nSo we can see that each step has better results than the previous,\n\nI have made the same successive methodology with the test set, using the best 200 parameters found on the first step (threshold for outlier, noise, observation noise, time), than removed the Samsung S20Ultra phone, and on the 3rd step use the 200 parameters for the optimum shift distance.\nThan the submission was the mean for those 200 prediction. \nSurprisingly, or not, this blending of 200 prediction with optimum thresholds was no better than using one submission consisting in the best from the 200 predictions (using the training set for estimating).\nI am still curious if anybody had any success with blending or any form of combining multiple submission.\n\n"
  }
}