{
  "id": 55801,
  "title": "A Stamp Collector's Take (first observations)",
  "url": "/competitions/trackml-particle-identification/discussion/55801",
  "author_name": "",
  "post_date": "2018-05-01T23:03:11.852269200Z",
  "votes": 3,
  "comment_count": 8,
  "views": 0,
  "content": "<p>“All science is either physics or stamp collecting.” -Lord Rutherford</p>\n\n<p>I have a few quick observations on the challenge that might be helpful as you start looking at the data.  A quick recap in case you've forgotten: The data comes from the particle colliders at CERN. Since I'm the stamp collector type I'm going to explain it in terms I understand.  They accelerate particles up to appreciable fractions of the speed of light (AKA really, <em>really</em> fast) in opposite directions then let them slam into each other. The resulting explosion blows the particles into their constituent bits and sprays them out in every direction.  Around the collider they have sensors that can detect where each bit goes after it gets exploded.  The sensors can register where (spatially) a hit occurs, and looking at all the sensors in order you can trace the path a particle follows as it hits each sensor in turn.   Our task is to sort through the raw data and find which hits correspond to each particle as it passes through the sensor array.</p>\n\n<p>Here's what I've been able to glean:</p>\n\n<ul>\n<li><p>There's a lot more in the Truth/Particle files than we're being asked to provide.</p>\n\n<p>The Truth and Particle files have a lot of columns, but the submission format only allows for Event ID, Hit ID, and Track ID.  The extra columns are things that are derived from the path data we are being asked to find.  For submissions the event and hit IDs are given, all we are being asked to do is provide (for each event) a list of particles and the hits that are associated with them. </p></li>\n<li><p>Any model you build should <em>only</em> use data originating in the Cells and Hits files</p>\n\n<p>As in the above point, all we have to go on in the test data are where each hit occurs. There's spatial data in the x,y,z coordinates of the hits (potentially augmented by the supplementary data on the location of the sensors), as well as the magnitude of the impacts from the Value column in the Cells file. The other values might be useful (probably are since the folks at CERN cared enough to calculate them) but to build a useful model <em>we must derive them for ourselves</em> -they are not a given.</p></li>\n<li><p>Tracks = Particles</p>\n\n<p>This one is a minor thing, but we are tasked with providing a Track ID. In the Particle/Truth files this appears as the Particle ID.  What I think they are asking for is: Each hit that corresponds with a particular particle (not all hits do) is assigned to the same track (Track ID).  Ideally each known particle will correspond 1:1 with a track, where each track contains every hit that particle made -and none that it didn't.</p></li>\n<li><p>The \"Origin\" for the spatial coordinates is relative to the sensors <strong>not</strong> the tracks.</p>\n\n<p>The collisions happen where and when they happen, that's not always going to be in the precise spot we'd like them to be.  I believe the coordinate system we are given is based around the sensors.  The origins of each of the tracks should converge on the location where the generating collision occurred, but this will not be at 0,0,0. </p></li>\n<li><p>We don't know how many particles are generated by each event.</p>\n\n<p>Yes, in the training data we can know how many particles there are, but it's not like there will be exactly 4,392 tracks in every event file.  Any model we build will need to take into account that there can be any number of possible tracks.</p></li>\n</ul>\n\n<p>That's how I'm understanding the challenge.  Now, I'm left wondering a few things.</p>\n\n<ol>\n<li><p>Do the particles lose energy after each hit?  If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?</p></li>\n<li><p>Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.</p></li>\n<li><p>Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred.  Down the line this might help to cluster hits that follow viable tracks that converge on the same point.</p></li>\n</ol>\n\n<p>Anyone who knows better please let me know if I'm off base on any of this (I'll gladly edit, hope to in fact).  Also, I'd like to join a team if anyone will have me ;)</p>",
  "messages": [
    {
      "id": "321796",
      "postDate": "05/01/2018 23:03:11",
      "content": "<p>“All science is either physics or stamp collecting.” -Lord Rutherford</p>\n\n<p>I have a few quick observations on the challenge that might be helpful as you start looking at the data.  A quick recap in case you've forgotten: The data comes from the particle colliders at CERN. Since I'm the stamp collector type I'm going to explain it in terms I understand.  They accelerate particles up to appreciable fractions of the speed of light (AKA really, <em>really</em> fast) in opposite directions then let them slam into each other. The resulting explosion blows the particles into their constituent bits and sprays them out in every direction.  Around the collider they have sensors that can detect where each bit goes after it gets exploded.  The sensors can register where (spatially) a hit occurs, and looking at all the sensors in order you can trace the path a particle follows as it hits each sensor in turn.   Our task is to sort through the raw data and find which hits correspond to each particle as it passes through the sensor array.</p>\n\n<p>Here's what I've been able to glean:</p>\n\n<ul>\n<li><p>There's a lot more in the Truth/Particle files than we're being asked to provide.</p>\n\n<p>The Truth and Particle files have a lot of columns, but the submission format only allows for Event ID, Hit ID, and Track ID.  The extra columns are things that are derived from the path data we are being asked to find.  For submissions the event and hit IDs are given, all we are being asked to do is provide (for each event) a list of particles and the hits that are associated with them. </p></li>\n<li><p>Any model you build should <em>only</em> use data originating in the Cells and Hits files</p>\n\n<p>As in the above point, all we have to go on in the test data are where each hit occurs. There's spatial data in the x,y,z coordinates of the hits (potentially augmented by the supplementary data on the location of the sensors), as well as the magnitude of the impacts from the Value column in the Cells file. The other values might be useful (probably are since the folks at CERN cared enough to calculate them) but to build a useful model <em>we must derive them for ourselves</em> -they are not a given.</p></li>\n<li><p>Tracks = Particles</p>\n\n<p>This one is a minor thing, but we are tasked with providing a Track ID. In the Particle/Truth files this appears as the Particle ID.  What I think they are asking for is: Each hit that corresponds with a particular particle (not all hits do) is assigned to the same track (Track ID).  Ideally each known particle will correspond 1:1 with a track, where each track contains every hit that particle made -and none that it didn't.</p></li>\n<li><p>The \"Origin\" for the spatial coordinates is relative to the sensors <strong>not</strong> the tracks.</p>\n\n<p>The collisions happen where and when they happen, that's not always going to be in the precise spot we'd like them to be.  I believe the coordinate system we are given is based around the sensors.  The origins of each of the tracks should converge on the location where the generating collision occurred, but this will not be at 0,0,0. </p></li>\n<li><p>We don't know how many particles are generated by each event.</p>\n\n<p>Yes, in the training data we can know how many particles there are, but it's not like there will be exactly 4,392 tracks in every event file.  Any model we build will need to take into account that there can be any number of possible tracks.</p></li>\n</ul>\n\n<p>That's how I'm understanding the challenge.  Now, I'm left wondering a few things.</p>\n\n<ol>\n<li><p>Do the particles lose energy after each hit?  If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?</p></li>\n<li><p>Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.</p></li>\n<li><p>Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred.  Down the line this might help to cluster hits that follow viable tracks that converge on the same point.</p></li>\n</ol>\n\n<p>Anyone who knows better please let me know if I'm off base on any of this (I'll gladly edit, hope to in fact).  Also, I'd like to join a team if anyone will have me ;)</p>",
      "rawMarkdown": "“All science is either physics or stamp collecting.” -Lord Rutherford\n\nI have a few quick observations on the challenge that might be helpful as you start looking at the data.  A quick recap in case you've forgotten: The data comes from the particle colliders at CERN. Since I'm the stamp collector type I'm going to explain it in terms I understand.  They accelerate particles up to appreciable fractions of the speed of light (AKA really, *really* fast) in opposite directions then let them slam into each other. The resulting explosion blows the particles into their constituent bits and sprays them out in every direction.  Around the collider they have sensors that can detect where each bit goes after it gets exploded.  The sensors can register where (spatially) a hit occurs, and looking at all the sensors in order you can trace the path a particle follows as it hits each sensor in turn.   Our task is to sort through the raw data and find which hits correspond to each particle as it passes through the sensor array.\n\nHere's what I've been able to glean:\n\n* There's a lot more in the Truth/Particle files than we're being asked to provide.\n\n     The Truth and Particle files have a lot of columns, but the submission format only allows for Event ID, Hit ID, and Track ID.  The extra columns are things that are derived from the path data we are being asked to find.  For submissions the event and hit IDs are given, all we are being asked to do is provide (for each event) a list of particles and the hits that are associated with them. \n\n* Any model you build should *only* use data originating in the Cells and Hits files\n\n     As in the above point, all we have to go on in the test data are where each hit occurs. There's spatial data in the x,y,z coordinates of the hits (potentially augmented by the supplementary data on the location of the sensors), as well as the magnitude of the impacts from the Value column in the Cells file. The other values might be useful (probably are since the folks at CERN cared enough to calculate them) but to build a useful model *we must derive them for ourselves* -they are not a given.\n\n* Tracks = Particles\n\n     This one is a minor thing, but we are tasked with providing a Track ID. In the Particle/Truth files this appears as the Particle ID.  What I think they are asking for is: Each hit that corresponds with a particular particle (not all hits do) is assigned to the same track (Track ID).  Ideally each known particle will correspond 1:1 with a track, where each track contains every hit that particle made -and none that it didn't.\n\n* The \"Origin\" for the spatial coordinates is relative to the sensors **not** the tracks.\n\n     The collisions happen where and when they happen, that's not always going to be in the precise spot we'd like them to be.  I believe the coordinate system we are given is based around the sensors.  The origins of each of the tracks should converge on the location where the generating collision occurred, but this will not be at 0,0,0. \n\n* We don't know how many particles are generated by each event.\n\n     Yes, in the training data we can know how many particles there are, but it's not like there will be exactly 4,392 tracks in every event file.  Any model we build will need to take into account that there can be any number of possible tracks.\n\nThat's how I'm understanding the challenge.  Now, I'm left wondering a few things.\n\n1. Do the particles lose energy after each hit?  If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?\n\n2. Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.\n\n3.  Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred.  Down the line this might help to cluster hits that follow viable tracks that converge on the same point.\n\n\nAnyone who knows better please let me know if I'm off base on any of this (I'll gladly edit, hope to in fact).  Also, I'd like to join a team if anyone will have me ;)",
      "votes": null
    },
    {
      "id": "321816",
      "postDate": "05/02/2018 00:39:56",
      "content": "<p>I am not a physicist.</p>\n\n<ol>\n<li><p>My recollection of this is that \"It depends.\"  The particles file has a column for initial particle energy but it doesn't list what type of particle it is.  In general, the different particles interact differently, so they can lose energy depending on the interaction type as they cause a signal in a detector.  I haven't seen any information that lists which particles the tracks are for.  I hope they mention which particles the tracks are for, if the cross-sections of the interactions differ, or if the 'q' column in the particles files denote different particles.</p></li>\n<li><p>AFAIK every 'hit' can cause the particle to change direction but I also think the particles have a well-defined path through the detector otherwise. See disclaimer.</p></li>\n</ol>",
      "rawMarkdown": "I am not a physicist.\n\n1. My recollection of this is that \"It depends.\"  The particles file has a column for initial particle energy but it doesn't list what type of particle it is.  In general, the different particles interact differently, so they can lose energy depending on the interaction type as they cause a signal in a detector.  I haven't seen any information that lists which particles the tracks are for.  I hope they mention which particles the tracks are for, if the cross-sections of the interactions differ, or if the 'q' column in the particles files denote different particles.\n\n2. AFAIK every 'hit' can cause the particle to change direction but I also think the particles have a well-defined path through the detector otherwise. See disclaimer.",
      "votes": null
    },
    {
      "id": "321847",
      "postDate": "05/02/2018 02:32:32",
      "content": "<p>Hi Mark,</p>\n\n<p>I'm not a physicist either!</p>\n\n<p>I think you're right that the energy can vary particle to particle. That's just based on intuition though. There are a few kernels up now that show the paths in 3D space and they are definitely curved, but maybe that combined with the change in energy might help with identification?</p>\n\n<p>In terms of q and particle type, I don't think we have that available. It's like you have sheets of paper spaced at even intervals then fire a shotgun into them. We have the sheets of paper (and that's it) and our job is to trace the paths of the pellets.</p>\n\n<p>I think in the morning I'm going to take a look into energy loss along the z axis and see if I can fit curves to the known paths -maybe something interesting will jump out?</p>",
      "rawMarkdown": "Hi Mark,\n\nI'm not a physicist either!\n\nI think you're right that the energy can vary particle to particle. That's just based on intuition though. There are a few kernels up now that show the paths in 3D space and they are definitely curved, but maybe that combined with the change in energy might help with identification?\n\nIn terms of q and particle type, I don't think we have that available. It's like you have sheets of paper spaced at even intervals then fire a shotgun into them. We have the sheets of paper (and that's it) and our job is to trace the paths of the pellets.\n\nI think in the morning I'm going to take a look into energy loss along the z axis and see if I can fit curves to the known paths -maybe something interesting will jump out?",
      "votes": null
    },
    {
      "id": "321861",
      "postDate": "05/02/2018 03:42:32",
      "content": "<p>A curved path is still a well-defined path. I bet someone will post the equation eventually.</p>\n\n<p>I also don't think we will get the particle types they simulated so if there are multiple types I hope they all behave the same.</p>",
      "rawMarkdown": "A curved path is still a well-defined path. I bet someone will post the equation eventually.\n\nI also don't think we will get the particle types they simulated so if there are multiple types I hope they all behave the same.",
      "votes": null
    },
    {
      "id": "321884",
      "postDate": "05/02/2018 04:51:44",
      "content": "<p>I found the post I was thinking of:</p>\n\n<p>\"At first glance one could assume all particles to progress along a helical trajectory - which is the movement of a charged particle in constant magnetic field.\"</p>\n\n<p><a href=\"https://www.kaggle.com/c/trackml-particle-identification/forums/t/55699/welcome?forumMessageId=321531#post321531\">https://www.kaggle.com/c/trackml-particle-identification/forums/t/55699/welcome?forumMessageId=321531#post321531</a></p>",
      "rawMarkdown": "I found the post I was thinking of:\n\n\"At first glance one could assume all particles to progress along a helical trajectory - which is the movement of a charged particle in constant magnetic field.\"\n\nhttps://www.kaggle.com/c/trackml-particle-identification/forums/t/55699/welcome?forumMessageId=321531#post321531",
      "votes": null
    },
    {
      "id": "321932",
      "postDate": "05/02/2018 07:10:53",
      "content": "<p>Your questions are addressed in (the beta-version of) this document:\n<a href=\"https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw\">https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw</a>\n(Once finalized, I will add the document as a final version to the TrackML page).</p>\n\n<p>Let me address your questions\"</p>\n\n<blockquote>\n  <p>Do the particles lose energy after each hit? If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?</p>\n</blockquote>\n\n<p>Yes, but the energy loss is rather small, we only included ionization loss, which is a small effect for particles in question.\nThe bigger effect is scattering at the detector material.</p>\n\n<blockquote>\n  <p>Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.</p>\n</blockquote>\n\n<p>It is indeed a fixable curved path, though not an analytically defined path as the magnetic field is not constant and scattering and energy loss effects are stochastic.</p>\n\n<blockquote>\n  <p>Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred. Down the line this might help to cluster hits that follow viable tracks that converge on the same point.</p>\n</blockquote>\n\n<p>No, there are roughly 200 collisions per event, which means there are something like 200 <strong>primary</strong>  vertices where the particles emerge from per event.</p>",
      "rawMarkdown": "Your questions are addressed in (the beta-version of) this document:\nhttps://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw\n(Once finalized, I will add the document as a final version to the TrackML page).\n\nLet me address your questions\"\n\n&gt; Do the particles lose energy after each hit? If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?\n\nYes, but the energy loss is rather small, we only included ionization loss, which is a small effect for particles in question.\nThe bigger effect is scattering at the detector material.\n\n&gt; Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.\n\nIt is indeed a fixable curved path, though not an analytically defined path as the magnetic field is not constant and scattering and energy loss effects are stochastic.\n\n&gt; Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred. Down the line this might help to cluster hits that follow viable tracks that converge on the same point.\n\nNo, there are roughly 200 collisions per event, which means there are something like 200 **primary**  vertices where the particles emerge from per event.",
      "votes": null
    },
    {
      "id": "322216",
      "postDate": "05/02/2018 15:02:24",
      "content": "<p>Thanks Andreas, that's a very helpful document! </p>",
      "rawMarkdown": "Thanks Andreas, that's a very helpful document!",
      "votes": null
    },
    {
      "id": "322220",
      "postDate": "05/02/2018 15:08:08",
      "content": "<p>I was toying around with the data last night and it seems like the vast majority of the particles are well behaved. Andreas Salzburger linked a <a href=\"https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw\">document</a> that suggests the region where the collisions occur (the <em>luminous region</em>) is roughly 15x15x55 (x,y,z) centered at (0,0,0), and regressing with a second order polynomial bears that out beautifully!</p>",
      "rawMarkdown": "I was toying around with the data last night and it seems like the vast majority of the particles are well behaved. Andreas Salzburger linked a [document](https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw) that suggests the region where the collisions occur (the *luminous region*) is roughly 15x15x55 (x,y,z) centered at (0,0,0), and regressing with a second order polynomial bears that out beautifully!",
      "votes": null
    },
    {
      "id": "322228",
      "postDate": "05/02/2018 15:28:42",
      "content": "<p>Thanks, Jason ! </p>\n\n<p>I am a physicist, and thus there might be quite some physicist jargon in it, come back to me with questions and suggestions if you want to. I am happy to explain more and use other words in the document as well ...</p>",
      "rawMarkdown": "Thanks, Jason ! \n\nI am a physicist, and thus there might be quite some physicist jargon in it, come back to me with questions and suggestions if you want to. I am happy to explain more and use other words in the document as well ...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 321816,
      "author_name": "mcvicuna",
      "author_url": "",
      "post_date": "05/02/2018 00:39:56",
      "content": "<p>I am not a physicist.</p>\n\n<ol>\n<li><p>My recollection of this is that \"It depends.\"  The particles file has a column for initial particle energy but it doesn't list what type of particle it is.  In general, the different particles interact differently, so they can lose energy depending on the interaction type as they cause a signal in a detector.  I haven't seen any information that lists which particles the tracks are for.  I hope they mention which particles the tracks are for, if the cross-sections of the interactions differ, or if the 'q' column in the particles files denote different particles.</p></li>\n<li><p>AFAIK every 'hit' can cause the particle to change direction but I also think the particles have a well-defined path through the detector otherwise. See disclaimer.</p></li>\n</ol>",
      "votes": null,
      "replies": [
        {
          "id": 321847,
          "author_name": "smytjf11",
          "author_url": "",
          "post_date": "05/02/2018 02:32:32",
          "content": "<p>Hi Mark,</p>\n\n<p>I'm not a physicist either!</p>\n\n<p>I think you're right that the energy can vary particle to particle. That's just based on intuition though. There are a few kernels up now that show the paths in 3D space and they are definitely curved, but maybe that combined with the change in energy might help with identification?</p>\n\n<p>In terms of q and particle type, I don't think we have that available. It's like you have sheets of paper spaced at even intervals then fire a shotgun into them. We have the sheets of paper (and that's it) and our job is to trace the paths of the pellets.</p>\n\n<p>I think in the morning I'm going to take a look into energy loss along the z axis and see if I can fit curves to the known paths -maybe something interesting will jump out?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 321861,
          "author_name": "mcvicuna",
          "author_url": "",
          "post_date": "05/02/2018 03:42:32",
          "content": "<p>A curved path is still a well-defined path. I bet someone will post the equation eventually.</p>\n\n<p>I also don't think we will get the particle types they simulated so if there are multiple types I hope they all behave the same.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 321884,
          "author_name": "mcvicuna",
          "author_url": "",
          "post_date": "05/02/2018 04:51:44",
          "content": "<p>I found the post I was thinking of:</p>\n\n<p>\"At first glance one could assume all particles to progress along a helical trajectory - which is the movement of a charged particle in constant magnetic field.\"</p>\n\n<p><a href=\"https://www.kaggle.com/c/trackml-particle-identification/forums/t/55699/welcome?forumMessageId=321531#post321531\">https://www.kaggle.com/c/trackml-particle-identification/forums/t/55699/welcome?forumMessageId=321531#post321531</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 322220,
          "author_name": "smytjf11",
          "author_url": "",
          "post_date": "05/02/2018 15:08:08",
          "content": "<p>I was toying around with the data last night and it seems like the vast majority of the particles are well behaved. Andreas Salzburger linked a <a href=\"https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw\">document</a> that suggests the region where the collisions occur (the <em>luminous region</em>) is roughly 15x15x55 (x,y,z) centered at (0,0,0), and regressing with a second order polynomial bears that out beautifully!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 321932,
      "author_name": "asalzburger",
      "author_url": "",
      "post_date": "05/02/2018 07:10:53",
      "content": "<p>Your questions are addressed in (the beta-version of) this document:\n<a href=\"https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw\">https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw</a>\n(Once finalized, I will add the document as a final version to the TrackML page).</p>\n\n<p>Let me address your questions\"</p>\n\n<blockquote>\n  <p>Do the particles lose energy after each hit? If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?</p>\n</blockquote>\n\n<p>Yes, but the energy loss is rather small, we only included ionization loss, which is a small effect for particles in question.\nThe bigger effect is scattering at the detector material.</p>\n\n<blockquote>\n  <p>Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.</p>\n</blockquote>\n\n<p>It is indeed a fixable curved path, though not an analytically defined path as the magnetic field is not constant and scattering and energy loss effects are stochastic.</p>\n\n<blockquote>\n  <p>Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred. Down the line this might help to cluster hits that follow viable tracks that converge on the same point.</p>\n</blockquote>\n\n<p>No, there are roughly 200 collisions per event, which means there are something like 200 <strong>primary</strong>  vertices where the particles emerge from per event.</p>",
      "votes": null,
      "replies": [
        {
          "id": 322216,
          "author_name": "smytjf11",
          "author_url": "",
          "post_date": "05/02/2018 15:02:24",
          "content": "<p>Thanks Andreas, that's a very helpful document! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 322228,
          "author_name": "asalzburger",
          "author_url": "",
          "post_date": "05/02/2018 15:28:42",
          "content": "<p>Thanks, Jason ! </p>\n\n<p>I am a physicist, and thus there might be quite some physicist jargon in it, come back to me with questions and suggestions if you want to. I am happy to explain more and use other words in the document as well ...</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "321796": "“All science is either physics or stamp collecting.” -Lord Rutherford\n\nI have a few quick observations on the challenge that might be helpful as you start looking at the data.  A quick recap in case you've forgotten: The data comes from the particle colliders at CERN. Since I'm the stamp collector type I'm going to explain it in terms I understand.  They accelerate particles up to appreciable fractions of the speed of light (AKA really, *really* fast) in opposite directions then let them slam into each other. The resulting explosion blows the particles into their constituent bits and sprays them out in every direction.  Around the collider they have sensors that can detect where each bit goes after it gets exploded.  The sensors can register where (spatially) a hit occurs, and looking at all the sensors in order you can trace the path a particle follows as it hits each sensor in turn.   Our task is to sort through the raw data and find which hits correspond to each particle as it passes through the sensor array.\n\nHere's what I've been able to glean:\n\n* There's a lot more in the Truth/Particle files than we're being asked to provide.\n\n     The Truth and Particle files have a lot of columns, but the submission format only allows for Event ID, Hit ID, and Track ID.  The extra columns are things that are derived from the path data we are being asked to find.  For submissions the event and hit IDs are given, all we are being asked to do is provide (for each event) a list of particles and the hits that are associated with them. \n\n* Any model you build should *only* use data originating in the Cells and Hits files\n\n     As in the above point, all we have to go on in the test data are where each hit occurs. There's spatial data in the x,y,z coordinates of the hits (potentially augmented by the supplementary data on the location of the sensors), as well as the magnitude of the impacts from the Value column in the Cells file. The other values might be useful (probably are since the folks at CERN cared enough to calculate them) but to build a useful model *we must derive them for ourselves* -they are not a given.\n\n* Tracks = Particles\n\n     This one is a minor thing, but we are tasked with providing a Track ID. In the Particle/Truth files this appears as the Particle ID.  What I think they are asking for is: Each hit that corresponds with a particular particle (not all hits do) is assigned to the same track (Track ID).  Ideally each known particle will correspond 1:1 with a track, where each track contains every hit that particle made -and none that it didn't.\n\n* The \"Origin\" for the spatial coordinates is relative to the sensors **not** the tracks.\n\n     The collisions happen where and when they happen, that's not always going to be in the precise spot we'd like them to be.  I believe the coordinate system we are given is based around the sensors.  The origins of each of the tracks should converge on the location where the generating collision occurred, but this will not be at 0,0,0. \n\n* We don't know how many particles are generated by each event.\n\n     Yes, in the training data we can know how many particles there are, but it's not like there will be exactly 4,392 tracks in every event file.  Any model we build will need to take into account that there can be any number of possible tracks.\n\nThat's how I'm understanding the challenge.  Now, I'm left wondering a few things.\n\n1. Do the particles lose energy after each hit?  If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?\n\n2. Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.\n\n3.  Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred.  Down the line this might help to cluster hits that follow viable tracks that converge on the same point.\n\n\nAnyone who knows better please let me know if I'm off base on any of this (I'll gladly edit, hope to in fact).  Also, I'd like to join a team if anyone will have me ;)",
    "321816": "I am not a physicist.\n\n1. My recollection of this is that \"It depends.\"  The particles file has a column for initial particle energy but it doesn't list what type of particle it is.  In general, the different particles interact differently, so they can lose energy depending on the interaction type as they cause a signal in a detector.  I haven't seen any information that lists which particles the tracks are for.  I hope they mention which particles the tracks are for, if the cross-sections of the interactions differ, or if the 'q' column in the particles files denote different particles.\n\n2. AFAIK every 'hit' can cause the particle to change direction but I also think the particles have a well-defined path through the detector otherwise. See disclaimer.",
    "321847": "Hi Mark,\n\nI'm not a physicist either!\n\nI think you're right that the energy can vary particle to particle. That's just based on intuition though. There are a few kernels up now that show the paths in 3D space and they are definitely curved, but maybe that combined with the change in energy might help with identification?\n\nIn terms of q and particle type, I don't think we have that available. It's like you have sheets of paper spaced at even intervals then fire a shotgun into them. We have the sheets of paper (and that's it) and our job is to trace the paths of the pellets.\n\nI think in the morning I'm going to take a look into energy loss along the z axis and see if I can fit curves to the known paths -maybe something interesting will jump out?",
    "321861": "A curved path is still a well-defined path. I bet someone will post the equation eventually.\n\nI also don't think we will get the particle types they simulated so if there are multiple types I hope they all behave the same.",
    "321884": "I found the post I was thinking of:\n\n\"At first glance one could assume all particles to progress along a helical trajectory - which is the movement of a charged particle in constant magnetic field.\"\n\nhttps://www.kaggle.com/c/trackml-particle-identification/forums/t/55699/welcome?forumMessageId=321531#post321531",
    "321932": "Your questions are addressed in (the beta-version of) this document:\nhttps://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw\n(Once finalized, I will add the document as a final version to the TrackML page).\n\nLet me address your questions\"\n\n&gt; Do the particles lose energy after each hit? If we follow each particle from sensor to sensor do we see an appreciable drop in the energy level?\n\nYes, but the energy loss is rather small, we only included ionization loss, which is a small effect for particles in question.\nThe bigger effect is scattering at the detector material.\n\n&gt; Do the particles follow a straight (or fit-ably curved) path? -That might make an algorithm that clusters hits into tracks viable.\n\nIt is indeed a fixable curved path, though not an analytically defined path as the magnetic field is not constant and scattering and energy loss effects are stochastic.\n\n&gt; Is each event a single collision? It might be neat to trace each of the known tracks back to their origin to see if we can determine where each collision occurred. Down the line this might help to cluster hits that follow viable tracks that converge on the same point.\n\nNo, there are roughly 200 collisions per event, which means there are something like 200 **primary**  vertices where the particles emerge from per event.",
    "322216": "Thanks Andreas, that's a very helpful document!",
    "322220": "I was toying around with the data last night and it seems like the vast majority of the particles are well behaved. Andreas Salzburger linked a [document](https://cernbox.cern.ch/index.php/s/tkF9YFCrYu8auWw) that suggests the region where the collisions occur (the *luminous region*) is roughly 15x15x55 (x,y,z) centered at (0,0,0), and regressing with a second order polynomial bears that out beautifully!",
    "322228": "Thanks, Jason ! \n\nI am a physicist, and thus there might be quite some physicist jargon in it, come back to me with questions and suggestions if you want to. I am happy to explain more and use other words in the document as well ..."
  },
  "source": "meta"
}