{
  "id": 61561,
  "title": "Ratio of non-helix traces is at the order of 15%?",
  "url": "/competitions/trackml-particle-identification/discussion/61561",
  "author_name": "",
  "post_date": "2018-07-20T22:57:38.373897300Z",
  "votes": null,
  "comment_count": 6,
  "views": 0,
  "content": "<p><a href=\"https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal\">norm</a> is a non-negative quantity, \ndescribing the fitting of helix to a trace.\nThe smaller, the better the fit.</p>\n\n<p>I plot the distribution of log10(norm) for an event.\n<img src=\"https://jian2017.files.wordpress.com/2018/07/download-1.png\" alt=\"distribution of log10(norm)\">\nIf log10(norm) &gt;0 , then we consider it as non-helix trace.\nWe can see the percentage is very high, about 15% in the above histogram.</p>\n\n<p>Physically, the non-helix trace is caused by secondary scattering into random direction. And there is no way to track random motions.</p>\n\n<p><strong>So, in other words, 85% is approximately the limit of the best score?</strong></p>",
  "messages": [
    {
      "id": "359849",
      "postDate": "07/20/2018 22:57:38",
      "content": "<p><a href=\"https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal\">norm</a> is a non-negative quantity, \ndescribing the fitting of helix to a trace.\nThe smaller, the better the fit.</p>\n\n<p>I plot the distribution of log10(norm) for an event.\n<img src=\"https://jian2017.files.wordpress.com/2018/07/download-1.png\" alt=\"distribution of log10(norm)\">\nIf log10(norm) &gt;0 , then we consider it as non-helix trace.\nWe can see the percentage is very high, about 15% in the above histogram.</p>\n\n<p>Physically, the non-helix trace is caused by secondary scattering into random direction. And there is no way to track random motions.</p>\n\n<p><strong>So, in other words, 85% is approximately the limit of the best score?</strong></p>",
      "rawMarkdown": "[norm][1] is a non-negative quantity, \ndescribing the fitting of helix to a trace.\nThe smaller, the better the fit.\n\nI plot the distribution of log10(norm) for an event.\n![distribution of log10(norm)][2]\nIf log10(norm) &gt;0 , then we consider it as non-helix trace.\nWe can see the percentage is very high, about 15% in the above histogram.\n\nPhysically, the non-helix trace is caused by secondary scattering into random direction. And there is no way to track random motions.\n\n**So, in other words, 85% is approximately the limit of the best score?**\n\n\n  [1]: https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal\n  [2]: https://jian2017.files.wordpress.com/2018/07/download-1.png",
      "votes": null
    },
    {
      "id": "360013",
      "postDate": "07/21/2018 09:20:44",
      "content": "<p>@hello_world, how do you calculate the fitting?</p>",
      "rawMarkdown": "hello_world, how do you calculate the fitting?",
      "votes": null
    },
    {
      "id": "360029",
      "postDate": "07/21/2018 10:01:53",
      "content": "<p>Are you looking for helix originating near the z axis only?  Looks like it.</p>",
      "rawMarkdown": "Are you looking for helix originating near the z axis only?  Looks like it.",
      "votes": null
    },
    {
      "id": "360225",
      "postDate": "07/21/2018 22:49:39",
      "content": "<p>The fitting assume the points are on a quadratic surface.\nA cylinder is a quadratic surface, and a helix is on a cylinder.</p>\n\n<p>I think it fits not only helix near z axis, because that quadratic fitting contains transnational constants.</p>\n\n<p>following this paper  <a href=\"https://www.sciencedirect.com/science/article/pii/S0167839696000581\">Fitting helices to data by total least squares by Yves Nievergelt</a>\nor  the chapter 2 of <a href=\"https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal\">this kernel</a></p>",
      "rawMarkdown": "The fitting assume the points are on a quadratic surface.\nA cylinder is a quadratic surface, and a helix is on a cylinder.\n\nI think it fits not only helix near z axis, because that quadratic fitting contains transnational constants.\n\n\nfollowing this paper  [Fitting helices to data by total least squares by Yves Nievergelt][1]\nor  the chapter 2 of [this kernel][2]\n\n\n  [1]: https://www.sciencedirect.com/science/article/pii/S0167839696000581\n  [2]: https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal",
      "votes": null
    },
    {
      "id": "360285",
      "postDate": "07/22/2018 04:22:03",
      "content": "<p>This is very interesting.  Note however that we can get some positive score from the bad tracks as long as we identify more than half of the hits.  It may be that some of the bad tracks are pretty helix up to some point where they start diverging.  Have you looked into that?</p>",
      "rawMarkdown": "This is very interesting.  Note however that we can get some positive score from the bad tracks as long as we identify more than half of the hits.  It may be that some of the bad tracks are pretty helix up to some point where they start diverging.  Have you looked into that?",
      "votes": null
    },
    {
      "id": "365532",
      "postDate": "08/02/2018 20:56:36",
      "content": "<p>I think you are in the right ballpark with your number regarding \"nice\" helical tracks, but it is not a strong limit for scores.</p>\n\n<p>Incidentally my own score is currently quite close to your prediction, but it is not such a good fit: For example, my approach has a hard time with secondary vertices, so I am certainly not finding all of the 85% you predicted. On the other hand, it means that I'm finding a significant number of tracks that you would not have included in your 85%.</p>\n\n<p>My feeling is that 0.90 is almost impossible to exceed, but after seeing <a href=\"/demelian\">@demelian</a>'s speed of improvement, I'm no longer quite certain.</p>",
      "rawMarkdown": "I think you are in the right ballpark with your number regarding \"nice\" helical tracks, but it is not a strong limit for scores.\n\nIncidentally my own score is currently quite close to your prediction, but it is not such a good fit: For example, my approach has a hard time with secondary vertices, so I am certainly not finding all of the 85% you predicted. On the other hand, it means that I'm finding a significant number of tracks that you would not have included in your 85%.\n\nMy feeling is that 0.90 is almost impossible to exceed, but after seeing @demelian's speed of improvement, I'm no longer quite certain.",
      "votes": null
    },
    {
      "id": "366217",
      "postDate": "08/04/2018 11:46:36",
      "content": "<p>Congratulations to <a href=\"/icecuber\">@icecuber</a> and @ersol for invalidating my prediction about 0.90! It's amazing!</p>",
      "rawMarkdown": "Congratulations to @icecuber and @ersol for invalidating my prediction about 0.90! It's amazing!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 360013,
      "author_name": "yuval6967",
      "author_url": "",
      "post_date": "07/21/2018 09:20:44",
      "content": "<p>@hello_world, how do you calculate the fitting?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 360029,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/21/2018 10:01:53",
      "content": "<p>Are you looking for helix originating near the z axis only?  Looks like it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 360225,
      "author_name": "jian2017",
      "author_url": "",
      "post_date": "07/21/2018 22:49:39",
      "content": "<p>The fitting assume the points are on a quadratic surface.\nA cylinder is a quadratic surface, and a helix is on a cylinder.</p>\n\n<p>I think it fits not only helix near z axis, because that quadratic fitting contains transnational constants.</p>\n\n<p>following this paper  <a href=\"https://www.sciencedirect.com/science/article/pii/S0167839696000581\">Fitting helices to data by total least squares by Yves Nievergelt</a>\nor  the chapter 2 of <a href=\"https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal\">this kernel</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 360285,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "07/22/2018 04:22:03",
          "content": "<p>This is very interesting.  Note however that we can get some positive score from the bad tracks as long as we identify more than half of the hits.  It may be that some of the bad tracks are pretty helix up to some point where they start diverging.  Have you looked into that?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 365532,
      "author_name": "edwinst",
      "author_url": "",
      "post_date": "08/02/2018 20:56:36",
      "content": "<p>I think you are in the right ballpark with your number regarding \"nice\" helical tracks, but it is not a strong limit for scores.</p>\n\n<p>Incidentally my own score is currently quite close to your prediction, but it is not such a good fit: For example, my approach has a hard time with secondary vertices, so I am certainly not finding all of the 85% you predicted. On the other hand, it means that I'm finding a significant number of tracks that you would not have included in your 85%.</p>\n\n<p>My feeling is that 0.90 is almost impossible to exceed, but after seeing <a href=\"/demelian\">@demelian</a>'s speed of improvement, I'm no longer quite certain.</p>",
      "votes": null,
      "replies": [
        {
          "id": 366217,
          "author_name": "edwinst",
          "author_url": "",
          "post_date": "08/04/2018 11:46:36",
          "content": "<p>Congratulations to <a href=\"/icecuber\">@icecuber</a> and @ersol for invalidating my prediction about 0.90! It's amazing!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "359849": "[norm][1] is a non-negative quantity, \ndescribing the fitting of helix to a trace.\nThe smaller, the better the fit.\n\nI plot the distribution of log10(norm) for an event.\n![distribution of log10(norm)][2]\nIf log10(norm) &gt;0 , then we consider it as non-helix trace.\nWe can see the percentage is very high, about 15% in the above histogram.\n\nPhysically, the non-helix trace is caused by secondary scattering into random direction. And there is no way to track random motions.\n\n**So, in other words, 85% is approximately the limit of the best score?**\n\n\n  [1]: https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal\n  [2]: https://jian2017.files.wordpress.com/2018/07/download-1.png",
    "360013": "hello_world, how do you calculate the fitting?",
    "360029": "Are you looking for helix originating near the z axis only?  Looks like it.",
    "360225": "The fitting assume the points are on a quadratic surface.\nA cylinder is a quadratic surface, and a helix is on a cylinder.\n\nI think it fits not only helix near z axis, because that quadratic fitting contains transnational constants.\n\n\nfollowing this paper  [Fitting helices to data by total least squares by Yves Nievergelt][1]\nor  the chapter 2 of [this kernel][2]\n\n\n  [1]: https://www.sciencedirect.com/science/article/pii/S0167839696000581\n  [2]: https://www.kaggle.com/mindcool/unrolling-of-helices-outliers-removal",
    "360285": "This is very interesting.  Note however that we can get some positive score from the bad tracks as long as we identify more than half of the hits.  It may be that some of the bad tracks are pretty helix up to some point where they start diverging.  Have you looked into that?",
    "365532": "I think you are in the right ballpark with your number regarding \"nice\" helical tracks, but it is not a strong limit for scores.\n\nIncidentally my own score is currently quite close to your prediction, but it is not such a good fit: For example, my approach has a hard time with secondary vertices, so I am certainly not finding all of the 85% you predicted. On the other hand, it means that I'm finding a significant number of tracks that you would not have included in your 85%.\n\nMy feeling is that 0.90 is almost impossible to exceed, but after seeing @demelian's speed of improvement, I'm no longer quite certain.",
    "366217": "Congratulations to @icecuber and @ersol for invalidating my prediction about 0.90! It's amazing!"
  },
  "source": "meta"
}