Hey everyone. The evaluation metric for this competition is:
\n$$
\nmean\\_position\\_error =
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\n\\left(
\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +
\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert
\n\\right)
\n$$
Which can be expanded as:
\n$$
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert
\n$$
With this in mind, we may split our work into two models: one to predict coordinates, and the second to predict floor levels. I have been taking this approach and I am starting with floor level prediction.
\nI'm curious to see how well you are all doing at floor level prediction?
\nDuring cross-validation, I am currently around 65% accuracy and +7 in terms of the evaluation metric, that is:
\n$$
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert
\n= 7
\n$$
That said, I am quite impressed to see the leaderboard is already pushing a score of 8! Is anyone else monitoring this metric individually? How are you doing?
", "messages": [ { "id": "1188065", "postDate": "02/05/2021 23:17:35", "content": "Hey everyone. The evaluation metric for this competition is:
\n$$
\nmean\\_position\\_error =
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\n\\left(
\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +
\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert
\n\\right)
\n$$
Which can be expanded as:
\n$$
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert
\n$$
With this in mind, we may split our work into two models: one to predict coordinates, and the second to predict floor levels. I have been taking this approach and I am starting with floor level prediction.
\nI'm curious to see how well you are all doing at floor level prediction?
\nDuring cross-validation, I am currently around 65% accuracy and +7 in terms of the evaluation metric, that is:
\n$$
\n\\frac{1}{N}
\n\\sum_{i=1}^N
\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert
\n= 7
\n$$
That said, I am quite impressed to see the leaderboard is already pushing a score of 8! Is anyone else monitoring this metric individually? How are you doing?
", "rawMarkdown": "Hey everyone. The evaluation metric for this competition is:\n\n$$\nmean\\\\_position\\\\_error = \n\\frac{1}{N} \n\\sum_{i=1}^N \n\\left(\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert \n\\right)\n$$\n\nWhich can be expanded as:\n\n$$\n\\frac{1}{N} \n\\sum_{i=1}^N\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\n\\frac{1}{N} \n\\sum_{i=1}^N\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert\n$$\n\nWith this in mind, we may split our work into two models: one to predict coordinates, and the second to predict floor levels. I have been taking this approach and I am starting with floor level prediction. \n\nI'm curious to see how well you are all doing at floor level prediction?\n\nDuring cross-validation, I am currently around **65%** accuracy and **+7** in terms of the evaluation metric, that is:\n\n$$\n\\frac{1}{N} \n\\sum_{i=1}^N\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert\n= 7\n$$\n\nThat said, I am quite impressed to see the leaderboard is already pushing a score of 8! Is anyone else monitoring this metric individually? How are you doing?", "votes": null }, { "id": "1188066", "postDate": "02/05/2021 23:19:37", "content": "In case you are curious, I built the equations using MathJax.
\nThe first equation is as follows:
\n$$\nmean\\\\_position\\\\_error = \n\\frac{1}{N} \n\\sum_{i=1}^N \\left(\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert \\right)\n$$\n@npa02012 - I can monitor floor predictions in relative terms using the public LB dataset by deliberately making submissions that contain the same xy predictions and differing floor predictions. From this I can score the floor predictions of the different public kernels that some of the leaders have kindly released against mine. Zero is ammarali32's kernel, and lower numbers are better.
\nDevin Anzelmo v6 +1.345
\nJiwei Liu v1 +1.306
\nammarali32 v1 0.000
\nMy submission Feb 5th -0.265
However, I don't see a way to retrieve an absolute value like +7 against the public test set - clearly you could do this against internal validation data from the training set.
", "rawMarkdown": "npa02012 - I can monitor floor predictions in relative terms using the public LB dataset by deliberately making submissions that contain the same xy predictions and differing floor predictions. From this I can score the floor predictions of the different public kernels that some of the leaders have kindly released against mine. Zero is ammarali32's kernel, and lower numbers are better.\n\nDevin Anzelmo v6 +1.345\nJiwei Liu v1 +1.306\nammarali32 v1 0.000\nMy submission Feb 5th -0.265\n\nHowever, I don't see a way to retrieve an absolute value like +7 against the public test set - clearly you could do this against internal validation data from the training set.", "votes": null }, { "id": "1189119", "postDate": "02/06/2021 18:19:26", "content": "I've been using the same approach as @jbomitchell for comparison. Against ammarali32's submission I score -0.246 so far.
", "rawMarkdown": "I've been using the same approach as @jbomitchell for comparison. Against ammarali32's submission I score -0.246 so far.", "votes": null }, { "id": "1189265", "postDate": "02/06/2021 21:12:11", "content": "@jbomitchell @franciscojosandrades
\nThanks!
I edited my post: the +7 is during cross-validation, though I haven't spent much time dialing in that process.
\nAnyways, I appreciate the insight! I didn't think to compare against public notebooks; interesting perspective and perfect for us to use as baseline.
\nI'll be reporting back when I get a test pipeline built out!
", "rawMarkdown": "jbomitchell @franciscojosandrades \nThanks!\n\nI edited my post: the **+7** is during cross-validation, though I haven't spent much time dialing in that process.\n\nAnyways, I appreciate the insight! I didn't think to compare against public notebooks; interesting perspective and perfect for us to use as baseline. \n\nI'll be reporting back when I get a test pipeline built out!", "votes": null }, { "id": "1240841", "postDate": "03/16/2021 16:51:38", "content": "Have you tried comparing your floor prediction model against https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model?
", "rawMarkdown": "Have you tried comparing your floor prediction model against https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model?", "votes": null }, { "id": "1241160", "postDate": "03/16/2021 23:40:01", "content": "@olaf2000 - I score exactly the same as @nigelhenry's model on the public leaderboard, but we differ on 11 paths comprising about 200 points in the private leaderboard dataset. At some point, I will study the relevant floor maps in detail to see how many of these differences can be resolved that way.
", "rawMarkdown": "olaf2000 - I score exactly the same as @nigelhenry's model on the public leaderboard, but we differ on 11 paths comprising about 200 points in the private leaderboard dataset. At some point, I will study the relevant floor maps in detail to see how many of these differences can be resolved that way.", "votes": null } ], "comments": [ { "id": 1188066, "author_name": "npa02012", "author_url": "", "post_date": "02/05/2021 23:19:37", "content": "In case you are curious, I built the equations using MathJax.
\nThe first equation is as follows:
\n$$\nmean\\\\_position\\\\_error = \n\\frac{1}{N} \n\\sum_{i=1}^N \\left(\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert \\right)\n$$\n@npa02012 - I can monitor floor predictions in relative terms using the public LB dataset by deliberately making submissions that contain the same xy predictions and differing floor predictions. From this I can score the floor predictions of the different public kernels that some of the leaders have kindly released against mine. Zero is ammarali32's kernel, and lower numbers are better.
\nDevin Anzelmo v6 +1.345
\nJiwei Liu v1 +1.306
\nammarali32 v1 0.000
\nMy submission Feb 5th -0.265
However, I don't see a way to retrieve an absolute value like +7 against the public test set - clearly you could do this against internal validation data from the training set.
", "votes": null, "replies": [ { "id": 1189265, "author_name": "npa02012", "author_url": "", "post_date": "02/06/2021 21:12:11", "content": "@jbomitchell @franciscojosandrades
\nThanks!
I edited my post: the +7 is during cross-validation, though I haven't spent much time dialing in that process.
\nAnyways, I appreciate the insight! I didn't think to compare against public notebooks; interesting perspective and perfect for us to use as baseline.
\nI'll be reporting back when I get a test pipeline built out!
", "votes": null, "replies": [] }, { "id": 1240841, "author_name": "olaf2000", "author_url": "", "post_date": "03/16/2021 16:51:38", "content": "Have you tried comparing your floor prediction model against https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model?
", "votes": null, "replies": [] }, { "id": 1241160, "author_name": "jbomitchell", "author_url": "", "post_date": "03/16/2021 23:40:01", "content": "@olaf2000 - I score exactly the same as @nigelhenry's model on the public leaderboard, but we differ on 11 paths comprising about 200 points in the private leaderboard dataset. At some point, I will study the relevant floor maps in detail to see how many of these differences can be resolved that way.
", "votes": null, "replies": [] } ] }, { "id": 1189119, "author_name": "franciscojosandrades", "author_url": "", "post_date": "02/06/2021 18:19:26", "content": "I've been using the same approach as @jbomitchell for comparison. Against ammarali32's submission I score -0.246 so far.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1188065": "Hey everyone. The evaluation metric for this competition is:\n\n$$\nmean\\\\_position\\\\_error = \n\\frac{1}{N} \n\\sum_{i=1}^N \n\\left(\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert \n\\right)\n$$\n\nWhich can be expanded as:\n\n$$\n\\frac{1}{N} \n\\sum_{i=1}^N\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\n\\frac{1}{N} \n\\sum_{i=1}^N\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert\n$$\n\nWith this in mind, we may split our work into two models: one to predict coordinates, and the second to predict floor levels. I have been taking this approach and I am starting with floor level prediction. \n\nI'm curious to see how well you are all doing at floor level prediction?\n\nDuring cross-validation, I am currently around **65%** accuracy and **+7** in terms of the evaluation metric, that is:\n\n$$\n\\frac{1}{N} \n\\sum_{i=1}^N\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert\n= 7\n$$\n\nThat said, I am quite impressed to see the leaderboard is already pushing a score of 8! Is anyone else monitoring this metric individually? How are you doing?", "1188066": "In case you are curious, I built the equations using [MathJax](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference).\n\nThe first equation is as follows:\n```\n$$\nmean\\\\_position\\\\_error = \n\\frac{1}{N} \n\\sum_{i=1}^N \\left(\n\\sqrt{(\\hat{x_i} - x_i)^2 + (\\hat{y_i} - y_i)^2 } +\np \\cdot \\lvert \\hat{f_i} - f_i \\rvert \\right)\n$$\n```", "1188865": "npa02012 - I can monitor floor predictions in relative terms using the public LB dataset by deliberately making submissions that contain the same xy predictions and differing floor predictions. From this I can score the floor predictions of the different public kernels that some of the leaders have kindly released against mine. Zero is ammarali32's kernel, and lower numbers are better.\n\nDevin Anzelmo v6 +1.345\nJiwei Liu v1 +1.306\nammarali32 v1 0.000\nMy submission Feb 5th -0.265\n\nHowever, I don't see a way to retrieve an absolute value like +7 against the public test set - clearly you could do this against internal validation data from the training set.", "1189119": "I've been using the same approach as @jbomitchell for comparison. Against ammarali32's submission I score -0.246 so far.", "1189265": "jbomitchell @franciscojosandrades \nThanks!\n\nI edited my post: the **+7** is during cross-validation, though I haven't spent much time dialing in that process.\n\nAnyways, I appreciate the insight! I didn't think to compare against public notebooks; interesting perspective and perfect for us to use as baseline. \n\nI'll be reporting back when I get a test pipeline built out!", "1240841": "Have you tried comparing your floor prediction model against https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model?", "1241160": "olaf2000 - I score exactly the same as @nigelhenry's model on the public leaderboard, but we differ on 11 paths comprising about 200 points in the private leaderboard dataset. At some point, I will study the relevant floor maps in detail to see how many of these differences can be resolved that way." }, "source": "meta" }