![\"LHC\"](\"https://www.metroweekly.com/articles/attachments/2014-03-20_film_8935_8813.jpg\")
Hi !
\nWe are a team of machine learning and particle physics scientists who have teamed up for three years to set up this challenge which we hope will excite your creativity! We do need you!
\nAll the necessary information is on the site, but to go a bit deeper we're providing here:
\nOf course we'll monitor the forum and strive to respond quickly.
\nA few tips, from the most common questions w've seen so far:
\nGood luck!
\nThe TrackML team : Sabrina Amrouche, Paolo Calafiura, Victor Estrade \"Sorme\", Steven Farrell, CecileGermain, Vava Gligorov, Tobias Golling, Heather Gray, Isabelle Guyon \"Isabelle\", Vincenzo Innocente, Mikhail Hushhyn, Moritz Kiehn, Ed Moyse, David Rousseau, Andreas Salzburger, Andrey Ustyuzhanin, Jean-Roch Vlimant \"jr\", Yetkin Yilmaz
\nPS : Follow us on twitter
\nAddendum : The final write-up of this Kaggle Accuracy phase for the TrackML challenge is published :
\nS.Amrouche,L.Basara,P.Calafiura,V.Estrade,S.Farrell,D.R. Ferreira, L. Finnie, N. Finnie, C. Germain, V. V. Gligorov, T. Golling, S. Gorbunov, H. Gray, I. Guyon, M. Hushchyn, V. Innocente, M. Kiehn, E. Moyse, J.-F. Puget, Y. Reina, D. Rousseau, A. Salzburger, A. Ustyuzhanin, J.-R. Vlimant, J. S. Wind, T. Xylouris, and Y. Yilmaz, \"The tracking machine learning challenge: Accuracy phase\", in The NeurIPS 2018 Competition, pp. 231–264. Springer International Publishing, Nov., 2019. arXiv:1904.06778 [hep-ex].
\ndoi:10.1007/978-3-030-29135-8_9
The final write-up of the Codalab Throughput phase is being finalized.
", "messages": [ { "id": 321278, "postDate": "2018-04-30T23:12:03.443Z", "content": "Hi !
\nWe are a team of machine learning and particle physics scientists who have teamed up for three years to set up this challenge which we hope will excite your creativity! We do need you!
\nAll the necessary information is on the site, but to go a bit deeper we're providing here:
\nOf course we'll monitor the forum and strive to respond quickly.
\nA few tips, from the most common questions w've seen so far:
\nGood luck!
\nThe TrackML team : Sabrina Amrouche, Paolo Calafiura, Victor Estrade \"Sorme\", Steven Farrell, CecileGermain, Vava Gligorov, Tobias Golling, Heather Gray, Isabelle Guyon \"Isabelle\", Vincenzo Innocente, Mikhail Hushhyn, Moritz Kiehn, Ed Moyse, David Rousseau, Andreas Salzburger, Andrey Ustyuzhanin, Jean-Roch Vlimant \"jr\", Yetkin Yilmaz
\nPS : Follow us on twitter
\nAddendum : The final write-up of this Kaggle Accuracy phase for the TrackML challenge is published :
\nS.Amrouche,L.Basara,P.Calafiura,V.Estrade,S.Farrell,D.R. Ferreira, L. Finnie, N. Finnie, C. Germain, V. V. Gligorov, T. Golling, S. Gorbunov, H. Gray, I. Guyon, M. Hushchyn, V. Innocente, M. Kiehn, E. Moyse, J.-F. Puget, Y. Reina, D. Rousseau, A. Salzburger, A. Ustyuzhanin, J.-R. Vlimant, J. S. Wind, T. Xylouris, and Y. Yilmaz, \"The tracking machine learning challenge: Accuracy phase\", in The NeurIPS 2018 Competition, pp. 231–264. Springer International Publishing, Nov., 2019. arXiv:1904.06778 [hep-ex].
\ndoi:10.1007/978-3-030-29135-8_9
The final write-up of the Codalab Throughput phase is being finalized.
", "rawMarkdown": "Hi !\n\nWe are a team of machine learning and particle physics scientists who have teamed up for three years to set up this challenge which we hope will excite your creativity! We do need you!\n\nAll the necessary information is on the site, but to go a bit deeper we're providing here:\n\n- a introduction document for a non-physicist audience with more details and figures (attached)\n- a few starting notebooks are being submitted as kernels and referenced here. Kept deliberately minimal and unoptimised, they show different techniques:\n - [DBSCAN][1] with a few lines of preprocessing and one line calling sklearn DBSCAN clusterer one can get the non trivial score of 20%\n - [Hough Transform][2] which is a mapping between the x,y,z point space to the space of all possible helix parameter going through these points, where the clustering is done\n - [kNN][3] which does require some training\n - [this kernel show how to use the cells][4] and recompute global position\n- Wesam has provided a nice [data visualisation kernel][5]\n- [a 3D event viewer][6] where one can upload files, then rotate, zoom etc... (but some people are already having fun [with their own viewer][7])\n- and a reminder : we recommend usage of our [helper library][8] to access the data, and compute the score on a training sample. Check [here][9] on details how to use it in kernels (Thanks Wesam!). \n- the submission file should be sorted by event_id, so something like : submission.sort_values(by = [\"event_id\", \"hit_id\"], \n- a [GoLang version of the helper library and Hough transform][10] is now available\n\nOf course we'll monitor the forum and strive to respond quickly.\n\nA few tips, from the most common questions w've seen so far:\n\n - there is not as much physics knowledge involved as it may appear. There is 3D geometry for sure, but not much beyond this. The core of the challenge is to connect all the points with arcs of helices in 3D.\n- when projected on the (x,y) plane an helix is an arc of circle\n- we define r=sqrt(x^2+y^2). In the (r,z) plane, an helix is to a good approximation a straight line starting **around** (0,0). \n - an analogy : it is like sorting beans (==hit_id) into piles, and then labelling (==track_id) the piles 1 2 3 4 or 123 456 678 679 , does not matter as long as each pile is given a unique positive integer\n - don't be overwhelmed by the amount of data, we provided it because we could, but we just don't know whether such a large amount of data is needed. Trying with just 100 events in train_sample is not a waste of time, see [discussion][11]\n - we expect the cells file to be useful for the last 5-10% (if we knew exactly we would not do this challenge!). So the cells file can be ignored at the beginning.\n - we ask every hit_id to be associated to a track_id, but it is completely fine to define a garbage track (with track_id 0 maybe) will all the hits your algorithm could not assigned. This garbage track will contribute zero to the score of course, but this will make a valid contribution.\n - all events are independent entities (it would be useless to look for correlation across event of the hit or particle numbering)\n - all events from the training and testing datasets have been generated in an identical way (except the random seed of course)\n - particles do not interact with each others. A particle trajectory is not influenced in any way by other close-by particles.\n\nGood luck!\n\nThe [TrackML team][12] : Sabrina Amrouche, Paolo Calafiura, Victor Estrade \"Sorme\", Steven Farrell, CecileGermain, Vava Gligorov, Tobias Golling, Heather Gray, Isabelle Guyon \"Isabelle\", Vincenzo Innocente, Mikhail Hushhyn, Moritz Kiehn, Ed Moyse, David Rousseau, Andreas Salzburger, Andrey Ustyuzhanin, Jean-Roch Vlimant \"jr\", Yetkin Yilmaz\n\nPS : [Follow us on twitter][13] \n\nAddendum : The final write-up of this Kaggle Accuracy phase for the TrackML challenge is published : \nS.Amrouche,L.Basara,P.Calafiura,V.Estrade,S.Farrell,D.R. Ferreira, L. Finnie, N. Finnie, C. Germain, V. V. Gligorov, T. Golling, S. Gorbunov, H. Gray, I. Guyon, M. Hushchyn, V. Innocente, M. Kiehn, E. Moyse, J.-F. Puget, Y. Reina, D. Rousseau, A. Salzburger, A. Ustyuzhanin, J.-R. Vlimant, J. S. Wind, T. Xylouris, and Y. Yilmaz, \"The tracking machine learning challenge: Accuracy phase\", in The NeurIPS 2018 Competition, pp. 231–264. Springer International Publishing, Nov., 2019. arXiv:[1904.06778](https://arxiv.org/abs/1904.06778) [hep-ex].\n[doi:10.1007/978-3-030-29135-8_9](https://doi.org/10.1007/978-3-030-29135-8_9) \n\nThe final write-up of the Codalab Throughput phase is being finalized. \n\n\n [1]: https://www.kaggle.com/mikhailhushchyn/dbscan-benchmark\n [2]: https://www.kaggle.com/mikhailhushchyn/hough-transform\n [3]: https://www.kaggle.com/mikhailhushchyn/knn-approach\n [4]: https://www.kaggle.com/asalzburger/pixel-detector-cells\n [5]: https://www.kaggle.com/wesamelshamy/trackml-problem-explanation-and-data-exploration/comments\n [6]: https://emoyse.web.cern.ch/emoyse/WebEventDisplay/jsdisplay_TrackML.html\n [7]: https://www.kaggle.com/c/trackml-particle-identification/discussion/55725\n [8]: https://github.com/LAL/trackml-library\n [9]: https://www.kaggle.com/c/trackml-particle-identification/discussion/55753\n [10]: https://www.kaggle.com/c/trackml-particle-identification/discussion/56858\n [11]: https://www.kaggle.com/c/trackml-particle-identification/discussion/55754\n [12]: https://sites.google.com/site/trackmlparticle/organisation\n [13]: https://twitter.com/trackmllhc", "votes": 79 }, { "id": 336546, "postDate": "2018-05-31T23:02:14.223Z", "content": "On a non-technical note... There is a wonderful documentary about the LHC and CERN called Particle Fever which I rewatched today. Highly recommended to put all this in context!
\n\nhttps://www.youtube.com/watch?v=PHObwzAg7Q0
\n\n
@Marketneutral thanks, I just watched it. I have seen documentaries about the LHC before but have not seen this one.
", "rawMarkdown": "@Marketneutral thanks, I just watched it. I have seen documentaries about the LHC before but have not seen this one." } ] }, { "id": 321432, "postDate": "2018-05-01T09:12:02.833Z", "content": "Hi Vadim,
\n\nyes, that is the metric used in this challenge.
\n\nMoritz
", "rawMarkdown": "Hi Vadim,\n\nyes, that is the metric used in this challenge.\n\nMoritz", "votes": 6, "replies": [ { "id": 321444, "postDate": "2018-05-01T09:54:01.480Z", "content": "... Moritz is the main author of this library.
", "rawMarkdown": "... Moritz is the main author of this library.", "votes": 4 }, { "id": 321528, "postDate": "2018-05-01T13:36:58.443Z", "content": "The library is awesome, thank you Moritz. Still trying to get it to run locally but it works like a dream on the site.
", "rawMarkdown": "The library is awesome, thank you Moritz. Still trying to get it to run locally but it works like a dream on the site. ", "votes": 1 } ] }, { "id": 335516, "postDate": "2018-05-29T21:12:29.363Z", "content": "I am curious as to the performance of the kalman filter approach. Do the organizers have an estimate of the score that this method would get?
", "rawMarkdown": "I am curious as to the performance of the kalman filter approach. Do the organizers have an estimate of the score that this method would get?", "votes": 3 }, { "id": 322426, "postDate": "2018-05-02T22:42:57.030Z", "content": "I am curious - how did you label the data?
", "rawMarkdown": "I am curious - how did you label the data?", "votes": 3, "replies": [ { "id": 322430, "postDate": "2018-05-02T23:05:41.877Z", "content": "This is simulated data so we know exactly which particle has been where. Most of the 3 years spent on the preparation of this challenge was in setting up this simulation, which is sufficiently detailed that we are quite convinced that an algorithm performing well on the simulation will perform well (after adjustment) on real data coming from actual proton collision.
", "rawMarkdown": "This is simulated data so we know exactly which particle has been where. Most of the 3 years spent on the preparation of this challenge was in setting up this simulation, which is sufficiently detailed that we are quite convinced that an algorithm performing well on the simulation will perform well (after adjustment) on real data coming from actual proton collision.", "votes": 14 }, { "id": 322432, "postDate": "2018-05-02T23:22:47.217Z", "content": "Very interesting, thanks!
", "rawMarkdown": "Very interesting, thanks!" }, { "id": 322677, "postDate": "2018-05-03T12:19:22.660Z", "content": "Does the same particle_id across events point to similar particles ?
", "rawMarkdown": "Does the same particle_id across events point to similar particles ? ", "votes": 1 }, { "id": 322679, "postDate": "2018-05-03T12:23:59.157Z", "content": "No, the events are independent - the particle_id is just a numbering schema
", "rawMarkdown": "No, the events are independent - the particle_id is just a numbering schema ", "votes": 2 }, { "id": 322701, "postDate": "2018-05-03T13:00:50.790Z", "content": "Thanks.... Using the \"truth\" file and the \"particle\" file and by calculating the distance traveled and the target location can we identify which \"particle_id\" across events are similar ?
", "rawMarkdown": "Thanks.... Using the \"truth\" file and the \"particle\" file and by calculating the distance traveled and the target location can we identify which \"particle_id\" across events are similar ?" }, { "id": 322711, "postDate": "2018-05-03T13:38:38.253Z", "content": "Somehow, yes. If you want; you can learn how trajectories are of particles with similar start parameters across events, those similarities will, however, not map onto particle_id
Hi, are the momentum values tpx, tpy, tpz for the trajectory right before the hit, or right after the hit? I'm asking because the trajectory can be modified via Multiple Coulomb Scattering, which means that momentum right before and right after the hit can be different.
", "rawMarkdown": "Hi, are the momentum values tpx, tpy, tpz for the trajectory right before the hit, or right after the hit? I'm asking because the trajectory can be modified via Multiple Coulomb Scattering, which means that momentum right before and right after the hit can be different.", "votes": 1, "replies": [ { "id": 352535, "postDate": "2018-07-04T15:29:48.720Z", "content": "Hi,
\n\nfor the pixel detector, the scattering material (dominant is the support material, the module material is minimal) is after the hit creation, for the strip detector the module material is before the hit creation.
\n\nThe idea is to mimic a pixel detector with inwards facing Silicon modules mounted on an outside shell - and for the the strips with cylinder/disks where the modules are mounted.
\n\nIn addition, there is a material cylinder between pixel and strips pretending to be a pixel support tube.
", "rawMarkdown": "Hi,\n\nfor the pixel detector, the scattering material (dominant is the support material, the module material is minimal) is *after* the hit creation, for the strip detector the module material is *before* the hit creation.\n\nThe idea is to mimic a pixel detector with inwards facing Silicon modules mounted on an outside shell - and for the the strips with cylinder/disks where the modules are mounted.\n\nIn addition, there is a material cylinder between pixel and strips pretending to be a pixel support tube.\n", "votes": 1 }, { "id": 352537, "postDate": "2018-07-04T15:32:37.670Z", "content": "Thank you for the fast answer. If I get you correctly, momentum can change right after a hit in the pixel detector, while momentum can change right before a hit in the strip detector.
", "rawMarkdown": "Thank you for the fast answer. If I get you correctly, momentum can change right after a hit in the pixel detector, while momentum can change right before a hit in the strip detector." } ] }, { "id": 347333, "postDate": "2018-06-24T00:20:00.903Z", "content": "The participant document contains misleading information about the submission format [my emphasis]:
\n\n\n\n\nParticipants are advised to compress the file (with zip, bzip2, gzip)\n before submission.
\n
However, bzip2-compressed submission files are rejected with an error message during scoring. I found this out the hard way today. Luckily gzip worked. Would be nice to really support bzip2 as it saves a few MB of upload.
", "rawMarkdown": "The participant document contains misleading information about the submission format [my emphasis]: \n\n> Participants are advised to compress the file (with zip, **bzip2**, gzip)\n> before submission.\n\nHowever, bzip2-compressed submission files are rejected with an error message during scoring. I found this out the hard way today. Luckily gzip worked. Would be nice to really support bzip2 as it saves a few MB of upload.", "votes": 1, "replies": [ { "id": 347461, "postDate": "2018-06-24T12:24:24.917Z", "content": "7z works fine, it compresses files more than other available options.
", "rawMarkdown": "7z works fine, it compresses files more than other available options.", "votes": 1 }, { "id": 352174, "postDate": "2018-07-03T20:15:08.553Z", "content": "Thanks! The 7z worked and it compressed a lot better than bzip2, actually.
", "rawMarkdown": "Thanks! The 7z worked and it compressed a lot better than bzip2, actually.", "votes": 1 }, { "id": 352682, "postDate": "2018-07-04T21:49:03.540Z", "content": "Top Kaggler secrets leeked! 7z for 0.8x!
", "rawMarkdown": "Top Kaggler secrets leeked! 7z for 0.8x!", "votes": 2 } ] }, { "id": 321408, "postDate": "2018-05-01T07:50:04.633Z", "content": "Hi David,
\n\nthank y'all for the competion!
\n\nCould you please verify this is the correct metric?
\n\nhttps://github.com/LAL/trackml-library/blob/master/trackml/score.py
", "rawMarkdown": "Hi David,\n\nthank y'all for the competion! \n\nCould you please verify this is the correct metric?\n\nhttps://github.com/LAL/trackml-library/blob/master/trackml/score.py", "votes": 1, "replies": [ { "id": 321756, "postDate": "2018-05-01T21:02:52.503Z", "content": "yes it is. If you have some doubt, please be more specific
", "rawMarkdown": "yes it is. If you have some doubt, please be more specific" } ] }, { "id": 328274, "postDate": "2018-05-13T21:12:24.143Z", "content": "I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?
\n\nThanks!
", "rawMarkdown": "I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?\n\nThanks!", "votes": 2, "replies": [ { "id": 330264, "postDate": "2018-05-18T12:42:19.693Z", "content": "I am also struggling to find this and the companion document that talks more about the physics of the simulation (mentioned on page two of the document). It says it's on the competition website but it's certainly not.
", "rawMarkdown": "I am also struggling to find this and the companion document that talks more about the physics of the simulation (mentioned on page two of the document). It says it's on the competition website but it's certainly not.", "votes": 2 } ] }, { "id": 324148, "postDate": "2018-05-07T09:09:43.760Z", "content": "This post is very helpful, specially the DBSCAN example with 20% success rate gave me more grip on the problem.
", "rawMarkdown": "This post is very helpful, specially the DBSCAN example with 20% success rate gave me more grip on the problem.", "votes": 2 }, { "id": 322925, "postDate": "2018-05-03T23:57:21.263Z", "content": "@David, can we assume the remaining charge is difference between the original charge - deposited charge after the particle traverses the pixel detector, or how should we calculate the remaining charge? Some of values are 1s, does that mean the charge is fully deposited on the cell and doesn't penetrate the cell? I couldn't find the answer from the Atlas paper due to a lack of domain knowledge.\nhttps://atlas.cern/updates/physics-briefing/charged-particle-reconstruction-energy-frontier\nhttp://de.arxiv.org/pdf/1406.7690
Sorry, I think that's a bit confusing, let me explain:
\n\n1 appear.I will release shortly a Kernel explaining that.
", "rawMarkdown": "Sorry, I think that's a bit confusing, let me explain: \n\n- the particle charge never changes, it is +1 or -1 (this is in terms of unit charge *e*), particles with charge 0 do not leave hits/traces\n- now, when they traverse Silicon, they induce charge in the Silicon, which is then read out, the effect is called *ionization*, but does not alter the charge of the traversing particles (it does decrease its energy a bit though)\n- the pixel detector is the only one (volume 7,8,9) where we emulate a readout that can actually measure this induced charge, the outer detectors only measure 1 (on) or 0 (off), that's why the values ``1`` appear.\n\nI will release shortly a Kernel explaining that.", "votes": 4 }, { "id": 327110, "postDate": "2018-05-10T20:42:21.300Z", "content": "Hi,\nI'm trying to put cuts on 'particles' objects to make it easier to try some models. As I remember, the units in ATLAS were mm and MeV. \nI made histograms of total momentum from the particles dataframe and find the maximum value to be 400 in the event I'm looking. That looks too small for LHC if my recollection of the units is correct.\nCan you please confirm the units in the momentum values?\nThanks
", "rawMarkdown": "Hi,\nI'm trying to put cuts on 'particles' objects to make it easier to try some models. As I remember, the units in ATLAS were mm and MeV. \nI made histograms of total momentum from the particles dataframe and find the maximum value to be 400 in the event I'm looking. That looks too small for LHC if my recollection of the units is correct.\nCan you please confirm the units in the momentum values?\nThanks" }, { "id": 327288, "postDate": "2018-05-11T07:28:56.393Z", "content": "The length unit is indeed mm but here we use GeV/c as the unit for momentum. Feel free to also have a look at the detailed description on the data page. It should list the units used for all quantities (if not let us know).
", "rawMarkdown": "The length unit is indeed mm but here we use GeV/c as the unit for momentum. Feel free to also have a look at the detailed description on the data page. It should list the units used for all quantities (if not let us know).", "votes": 1 }, { "id": 327364, "postDate": "2018-05-11T11:28:58.877Z", "content": "Hi Moritz, thanks for the reply. I went back to the document:
\n\n\n\nAnd found all the relevant information about units.
", "rawMarkdown": "Hi Moritz, thanks for the reply. I went back to the document:\n\nhttps://kaggle2.blob.core.windows.net/forum-message-attachments/321278/9331/trackml-participant-document-particle-v1.0.pdf\n\nAnd found all the relevant information about units." } ] }, { "id": 322731, "postDate": "2018-05-03T14:21:05.330Z", "content": "@David Rousseau Hi, will the organizer also provide a baseline code for hough transform? What is the score for using hough transform?
", "rawMarkdown": "@David Rousseau Hi, will the organizer also provide a baseline code for hough transform? What is the score for using hough transform?", "votes": 2, "replies": [ { "id": 322864, "postDate": "2018-05-03T20:03:23.630Z", "content": "coming soon...
", "rawMarkdown": "coming soon...", "votes": 1 }, { "id": 323071, "postDate": "2018-05-04T09:36:06.430Z", "content": "Hi @David Rousseau , just to get an idea if you know, what's the score of state of art algorithm on this dataset and what would be considered a meaningful realistic score (<1) to be useful in practice ?
", "rawMarkdown": "Hi @David Rousseau , just to get an idea if you know, what's the score of state of art algorithm on this dataset and what would be considered a meaningful _realistic_ score (<1) to be useful in practice ?", "votes": 3 }, { "id": 323249, "postDate": "2018-05-04T17:39:19.973Z", "content": "Hough transform posted!
", "rawMarkdown": "[Hough transform][1] posted!\n\n\n [1]: https://www.kaggle.com/mikhailhushchyn/hough-transform", "votes": 2 }, { "id": 323251, "postDate": "2018-05-04T17:42:27.710Z", "content": "@David Rousseau
\n Thanks!
@TeraFlops At the moment we do not have a score for an algorithm currently used in one of the running experiments. The code for these tend to be deeply integrated into the experiment software packages and are usually hard to run standalone. However, we hope to get a benchmark score for these algorithms later during the challenge.
\n\nMy personal guess is that they will end up with a score somewhere around 0.8-0.9, but this is really just an (educated) guess.
", "rawMarkdown": "@TeraFlops At the moment we do not have a score for an algorithm currently used in one of the running experiments. The code for these tend to be deeply integrated into the experiment software packages and are usually hard to run standalone. However, we hope to get a benchmark score for these algorithms later during the challenge.\n\nMy personal guess is that they will end up with a score somewhere around 0.8-0.9, but this is really just an (educated) guess.", "votes": 2 }, { "id": 326746, "postDate": "2018-05-10T09:08:54.873Z", "content": "@Moritz, The score equal 0.8-0.9 seems to be too optimistic to me. Not because of my lack of faith in Machine Learning, but because of the metrics used. In such a dense system, each of a few neighbouring pixels may be equally good for each of a few crossing tracks (when the differences between them are lower than the detector error), but according to the metrics used a hit/pixel either belongs to the track or does not. I do not know how \"dense\" is the system, but such groups of very similar hits/pixels (regarding the space position and signal) may be many.
", "rawMarkdown": "@Moritz, The score equal 0.8-0.9 seems to be too optimistic to me. Not because of my lack of faith in Machine Learning, but because of the metrics used. In such a dense system, each of a few neighbouring pixels may be equally good for each of a few crossing tracks (when the differences between them are lower than the detector error), but according to the metrics used a hit/pixel either belongs to the track or does not. I do not know how \"dense\" is the system, but such groups of very similar hits/pixels (regarding the space position and signal) may be many.", "votes": 2 }, { "id": 326817, "postDate": "2018-05-10T11:35:15.233Z", "content": "The key thing is that the density is not large when compared to the point precision. In fact, @Moritz has checked that the probability that the closest measured point from a given true point was not its corresponding measured point was a few per mille.
", "rawMarkdown": "The key thing is that the density is not large when compared to the point precision. In fact, @Moritz has checked that the probability that the closest measured point from a given true point was **not** its corresponding measured point was a few per mille.", "votes": 1 }, { "id": 326838, "postDate": "2018-05-10T12:08:53.813Z", "content": "@David, It is worth to know, however it is a pity - I thought the aim of the competition is to help to solve the system more complicated, not more simplified.
", "rawMarkdown": "@David, It is worth to know, however it is a pity - I thought the aim of the competition is to help to solve the system more complicated, not more simplified. ", "votes": 1 }, { "id": 327429, "postDate": "2018-05-11T14:37:25.997Z", "content": "The main complication is that the trajectories are built from a set of points which have to be self consistent. The randomisation due to measured x y z versus true x y z is \"blurring\" this self consistency. It is not the main complication, but it adds to it.
", "rawMarkdown": "The main complication is that the trajectories are built from a set of points which have to be self consistent. The randomisation due to measured x y z versus true x y z is \"blurring\" this self consistency. It is not the main complication, but it adds to it." }, { "id": 339029, "postDate": "2018-06-06T07:06:48.317Z", "content": "\n\n\nMy personal guess is that they will end up with a score somewhere around 0.8-0.9, but this is really just an (educated) guess.
\n
Are you saying you currently can get a score above 0.8 on this data?
\n\nJust curious to see if we are useful to you.
", "rawMarkdown": "> My personal guess is that they will end up with a score somewhere around 0.8-0.9, but this is really just an (educated) guess.\n\nAre you saying you currently can get a score above 0.8 on this data? \n\nJust curious to see if we are useful to you." } ] }, { "id": 1470660, "postDate": "2021-08-13T16:37:25.890Z", "content": "Nice, good job!
", "rawMarkdown": "Nice, good job!" }, { "id": 1149834, "postDate": "2021-01-12T07:26:50.817Z", "content": "where can i find the attachment?
\na introduction document for a non-physicist audience with more details and figures (attached)
The very last line of the first post here \"trackml-participant-document-particle-v1.0.pdf\" is clickable
", "rawMarkdown": " The very last line of the first post here \"trackml-participant-document-particle-v1.0.pdf\" is clickable", "votes": 1 }, { "id": 1149971, "postDate": "2021-01-12T09:16:51.870Z", "content": "Found it , thanks a lot
", "rawMarkdown": "Found it , thanks a lot" } ] }, { "id": 385942, "postDate": "2018-09-11T20:53:49.017Z", "content": "Second, \"Throughput\" phase is online, see this post
", "rawMarkdown": "Second, \"Throughput\" phase is online, see [this post][1]\n\n\n [1]: https://www.kaggle.com/c/trackml-particle-identification/discussion/65525" }, { "id": 374790, "postDate": "2018-08-23T20:14:30.777Z", "content": "The TrackML Kaggle competition has ended. Any participant (not necessarily with the highest score) who think they made valuable contributions with innovative algorithms (in particular if they have shared insights on the forum) are welcome to release publicly their code with an open source license and lightweight structured documentation. This will allow to compete for the « HEP meets ML » jury prizes: a NVidia V100 GPU, and two invitations to NIPS 2018 or the spring 2019 CERN workshop.See details in forum topic:
\n\nhttps://www.kaggle.com/c/trackml-particle-identification/discussion/63289
", "rawMarkdown": "The TrackML Kaggle competition has ended. Any participant (not necessarily with the highest score) who think they made valuable contributions with innovative algorithms (in particular if they have shared insights on the forum) are welcome to release publicly their code with an open source license and lightweight structured documentation. This will allow to compete for the « HEP meets ML » jury prizes: a NVidia V100 GPU, and two invitations to NIPS 2018 or the spring 2019 CERN workshop.See details in forum topic:\n\nhttps://www.kaggle.com/c/trackml-particle-identification/discussion/63289" }, { "id": 347288, "postDate": "2018-06-23T20:01:41.383Z", "content": "hi David\nI have a track_id attached to this message is ggplot of this. I was wondering is the tracker supposed to unify micro and macro events, because the only way I was able to make a positive tracker was using something like the Legendrian so the tracker is not north of strangeness and spontaneous. Does this come close to what you need? How do I complete my work if I do not have access to training sets 2-5? I called the plot something like tracking the destiny of future language because the \"macro\" events tether a Morse like line. I also think the curve looks like the two body Coulomb attraction/repulsion curve over distance. help me out if it is worth it and tell me what you think.
", "rawMarkdown": "hi David\nI have a track_id attached to this message is ggplot of this. I was wondering is the tracker supposed to unify micro and macro events, because the only way I was able to make a positive tracker was using something like the Legendrian so the tracker is not north of strangeness and spontaneous. Does this come close to what you need? How do I complete my work if I do not have access to training sets 2-5? I called the plot something like tracking the destiny of future language because the \"macro\" events tether a Morse like line. I also think the curve looks like the two body Coulomb attraction/repulsion curve over distance. help me out if it is worth it and tell me what you think." }, { "id": 337460, "postDate": "2018-06-02T22:06:54.540Z", "content": "Hi,
\n\nYou mention in the pdf that heavy particles are more important to find. Are particles likely to leave a charge in the detector proportional to their weight ? Is there a defined relation between charge left on a pixel and particle weight ? I fail to find details about that.
\n\nSimilarly, I'm interested in more details about helix parameters and momentum (which you mention the particles will shed a bit of every time they cross silicon). \nThe pdf says \"The relation between the parameters of the helix and the momentum are explained in appendix ??.\" I fail to find the ?? appendix :)
\n\nCheers !
", "rawMarkdown": "Hi,\n\nYou mention in the pdf that heavy particles are more important to find. Are particles likely to leave a charge in the detector proportional to their weight ? Is there a defined relation between charge left on a pixel and particle weight ? I fail to find details about that.\n\nSimilarly, I'm interested in more details about helix parameters and momentum (which you mention the particles will shed a bit of every time they cross silicon). \nThe pdf says \"The relation between the parameters of the helix and the momentum are explained in appendix ??.\" I fail to find the ?? appendix :)\n\nCheers !", "replies": [ { "id": 338617, "postDate": "2018-06-05T12:52:38.873Z", "content": "Apologies, in my first paragraph I meant to say not heavy particles but high energy particles. To quote the pdf \"weight_pt The high energy particles (large transverse momentum pT ) are the most\ninteresting ones.\" Those ones ...
\n\nSo to rephrase said paragraph : Are particles likely to leave a charge in the detector proportional to the amount of momentum or energy they have ? (roughly EnergyxWeight = momentum if I am not mistaken). Is there a defined relation between charge left on a pixel and particle energy ? I fail to find details about that.
\n\nThanks for the good work on the doc, I liked reading it. Looking forward to the next version.
", "rawMarkdown": "Apologies, in my first paragraph I meant to say not heavy particles but high energy particles. To quote the pdf \"weight_pt The high energy particles (large transverse momentum pT ) are the most\ninteresting ones.\" Those ones ...\n\nSo to rephrase said paragraph : Are particles likely to leave a charge in the detector proportional to the amount of momentum or energy they have ? (roughly EnergyxWeight = momentum if I am not mistaken). Is there a defined relation between charge left on a pixel and particle energy ? I fail to find details about that.\n\nThanks for the good work on the doc, I liked reading it. Looking forward to the next version.\n" }, { "id": 339083, "postDate": "2018-06-06T09:03:37.897Z", "content": "Multiple Coulomb Scattering is the phenomenon behind energy loss. you can google Bethe-Bloch energy loss,![\"bethe](\"http://meroli.web.cern.ch/img/lecture/straggling/Lectur1.gif\")
We're mostly interested in the domain of few hundreds MeV of momentum till a handful of GeV's, thus you can say the dEdx (aka \"stopping power\") is proportional to the momentum. Additionally make a note that different types of particles deposit different amount of energy in different materials.
\n\n![\"energy](\"https://i.imgur.com/J88JK09.png\")
Thanks ! I'm not much of a physicist beyond Newton, so I dont grok all of it :) but at least enough to see that it won't necessarily be easy to deduce much hint about corelation between charge left on the detector and trajectory \"straighness\" going from the theory. I'll see if I have better luck using the data set !
\n\nOne last question : does the tube also act like silicon in fuzzing the trajectory of particles ? or is it more \"transparent\", ie is Multiple Coulomb Scattering happening there as well ?
", "rawMarkdown": "Thanks ! I'm not much of a physicist beyond Newton, so I dont grok all of it :) but at least enough to see that it won't necessarily be easy to deduce much hint about corelation between charge left on the detector and trajectory \"straighness\" going from the theory. I'll see if I have better luck using the data set !\n\nOne last question : does the tube also act like silicon in fuzzing the trajectory of particles ? or is it more \"transparent\", ie is Multiple Coulomb Scattering happening there as well ?", "votes": 1 }, { "id": 339714, "postDate": "2018-06-07T13:26:56.173Z", "content": "what do you mean by tube? if you mean the beam pipe, then in the HEP experiment I was working on it was made out of Beryllium
", "rawMarkdown": "what do you mean by tube? if you mean the beam pipe, then in the HEP experiment I was working on it was made out of Beryllium" }, { "id": 339730, "postDate": "2018-06-07T14:22:35.493Z", "content": "Yes the beam pipe thanks. So does beryllium have the same properties than silicon when it comes to slightly changing the trajectories of particles ?
", "rawMarkdown": "Yes the beam pipe thanks. So does beryllium have the same properties than silicon when it comes to slightly changing the trajectories of particles ?" }, { "id": 339735, "postDate": "2018-06-07T14:28:11.360Z", "content": "I would not care about the beam in the particular case. I don't know if they put it inside the simulation. Generally yes! it does affect the trajectory (hint: that's why they try n make it as thin and sturdy as possible and)
", "rawMarkdown": "I would not care about the beam in the particular case. I don't know if they put it inside the simulation. Generally yes! it does affect the trajectory (hint: that's why they try n make it as thin and sturdy as possible and)" }, { "id": 339741, "postDate": "2018-06-07T14:36:04.703Z", "content": "yeah thanks, would have been interesting to know if it is accounted for in the sim for 2 reasons : \nOne: it makes it easier to decide if the collisions points that are found in the data can be precisely relied on\nTwo: I guess it has large effects on particle with trajectories somewhat tangential to the pipe, those will be very hard to link to a collision if so.
\n\nI think I can work it out looking at the truth data, but I thought I would ask as well :) a yes/no from the person who did the sim would probably save a day or two of data wrangling .. :)
", "rawMarkdown": "yeah thanks, would have been interesting to know if it is accounted for in the sim for 2 reasons : \nOne: it makes it easier to decide if the collisions points that are found in the data can be precisely relied on\nTwo: I guess it has large effects on particle with trajectories somewhat tangential to the pipe, those will be very hard to link to a collision if so.\n\nI think I can work it out looking at the truth data, but I thought I would ask as well :) a yes/no from the person who did the sim would probably save a day or two of data wrangling .. :)\n", "votes": 1 }, { "id": 339745, "postDate": "2018-06-07T14:44:29.323Z", "content": "@agerom, thanks for the references.
\n\nMy physics is old memories hence I am asking the question here. I am not sure the energy lost in thin detectors is really proportional to the energy of the particle, see figure 27.6 of the same paper you extracted your pictures from (http://pdg.lbl.gov/2006/reviews/passagerpp.pdf)
\n\nThe curves are pretty flat.
", "rawMarkdown": "@agerom, thanks for the references.\n\nMy physics is old memories hence I am asking the question here. I am not sure the energy lost in thin detectors is really proportional to the energy of the particle, see figure 27.6 of the same paper you extracted your pictures from (http://pdg.lbl.gov/2006/reviews/passagerpp.pdf)\n\nThe curves are pretty flat.", "votes": 1 } ] }, { "id": 336083, "postDate": "2018-05-31T03:09:54.307Z", "content": "Can you explain why there is no timestamp for each hit in the data?
", "rawMarkdown": "Can you explain why there is no timestamp for each hit in the data?", "replies": [ { "id": 336162, "postDate": "2018-05-31T06:54:01.513Z", "content": "why do you need that?
", "rawMarkdown": "why do you need that?" }, { "id": 336269, "postDate": "2018-05-31T10:42:40.967Z", "content": "We don't have timing information from the Silicon detector we are using. Since all particles are flying inside out at almost the speed of light, timing info would not be useful, unless one can get it with much better than 1 nanosecond precision (light travels at 30cm per nanosecond). In fact, in real experiments we are considering to have very high precision timing detectors but this would be technically possible in only a very small subset, like one additional layer on the outside.
", "rawMarkdown": "We don't have timing information from the Silicon detector we are using. Since all particles are flying inside out at almost the speed of light, timing info would not be useful, unless one can get it with much better than 1 nanosecond precision (light travels at 30cm per nanosecond). In fact, in real experiments we are considering to have very high precision timing detectors but this would be technically possible in only a very small subset, like one additional layer on the outside.", "votes": 1 }, { "id": 336295, "postDate": "2018-05-31T11:39:30.093Z", "content": "@Vadim, I was was hoping there could be some opportunity for feature engineering with the hit time -- the simplest example being: if two hits had the same timestamp then they could not be of the same track.
\n\n@David, Thanks for the reply and explanation.
", "rawMarkdown": "@Vadim, I was was hoping there could be some opportunity for feature engineering with the hit time -- the simplest example being: if two hits had the same timestamp then they could not be of the same track.\n\n@David, Thanks for the reply and explanation.\n\n" } ] }, { "id": 333171, "postDate": "2018-05-24T14:49:27.203Z", "content": "The \"Description\" page was incorrectly giving \"July 2018\" for the end of the first phase, in contradiction with the \"Timeline\" and \"Rules\" (which participants have signed) saying 13th August. This has just been fixed. We hope this did not confuse anyone.
", "rawMarkdown": "The \"Description\" page was incorrectly giving \"July 2018\" for the end of the first phase, in contradiction with the \"Timeline\" and \"Rules\" (which participants have signed) saying 13th August. This has just been fixed. We hope this did not confuse anyone." }, { "id": 332506, "postDate": "2018-05-23T09:17:29.447Z", "content": "In some cases in the Train dataset particles particle_id and truth particle_id do not match. I mean there is a discrepancy between the two files. Can you please elaborate?
", "rawMarkdown": "In some cases in the Train dataset *particles* particle_id and *truth* particle_id do not match. I mean there is a discrepancy between the two files. Can you please elaborate? ", "replies": [ { "id": 345006, "postDate": "2018-06-19T04:02:59.027Z", "content": "Did you get an answer why dataset particles particle_id and truth particle_id do not match? I found particle_id = 0 in truth, but there are none in the particle set with that ID. I was directed to this thread, and the trackml-participant-document, but I do not see an explanation.
", "rawMarkdown": "Did you get an answer why dataset particles particle_id and truth particle_id do not match? I found particle_id = 0 in truth, but there are none in the particle set with that ID. I was directed to this thread, and the trackml-participant-document, but I do not see an explanation.", "votes": 1 }, { "id": 345256, "postDate": "2018-06-19T14:09:18.127Z", "content": "Unfortunately I did not. I proceed with my analysis by creating an inner join of truth and particles based on the particle_id
", "rawMarkdown": "Unfortunately I did not. I proceed with my analysis by creating an inner join of truth and particles based on the particle_id" }, { "id": 345264, "postDate": "2018-06-19T14:28:24.120Z", "content": "To my knowledge, the only particle_id found in truth files that cannot be found in the particle file are all id 0. This is not clearly stated anywhere, but my belief is that it's noise that your algorythm has to be able to wade thru.
\n\nIOW: all hits with particle ID 0 in the truth file should be identified and discarded by your algo.
\n\nNot sure what physics process is behind the generation of this noise, seems there are multiple candidate sources when looking at it closely. Perhaps one bit of explanation is this line in the pdf :
\n\n\"In most cases, the particles come from the production points which are in the luminous region, however a small fraction of the particles come from short-lived unstable heavy particles flying a few millimeter.\"
\n\nThe fact the hits don't have an ID could then be explained by the fact that they are not generated by a particle part of the collision byproduct simulation, but part of some noise simulation ...
", "rawMarkdown": "To my knowledge, the only particle_id found in truth files that cannot be found in the particle file are all id 0. This is not clearly stated anywhere, but my belief is that it's noise that your algorythm has to be able to wade thru.\n\nIOW: all hits with particle ID 0 in the truth file should be identified and discarded by your algo.\n\nNot sure what physics process is behind the generation of this noise, seems there are multiple candidate sources when looking at it closely. Perhaps one bit of explanation is this line in the pdf :\n\n\"In most cases, the particles come from the production points which are in the luminous region, however a small fraction of the particles come from short-lived unstable heavy particles flying a few millimeter.\"\n\nThe fact the hits don't have an ID could then be explained by the fact that they are not generated by a particle part of the collision byproduct simulation, but part of some noise simulation ...\n\n" }, { "id": 345577, "postDate": "2018-06-20T04:27:13.630Z", "content": "I found one reference on the data page that explains particle_id=0.
\nhttps://www.kaggle.com/c/trackml-particle-identification/discussion/59071
@David is there a theoretical limit to the number of tracks or number of particles?
", "rawMarkdown": "@David is there a theoretical limit to the number of tracks or number of particles?", "replies": [ { "id": 331445, "postDate": "2018-05-21T08:50:14.420Z", "content": "No, but given that the test events have been randomly selected from the same master dataset as the training events, you can easily infer the distribution of number of particles.
", "rawMarkdown": "No, but given that the test events have been randomly selected from the same master dataset as the training events, you can easily infer the distribution of number of particles." }, { "id": 331494, "postDate": "2018-05-21T12:01:46.517Z", "content": "Sounds good, Thanks!
", "rawMarkdown": "Sounds good, Thanks!\n" } ] }, { "id": 328271, "postDate": "2018-05-13T21:04:20.370Z", "content": "Sorry if this is discussed elsewhere, but is there any ordering to the training files? For example if I took only the first half of train_1.zip, would I be able to get started with a relatively representative dataset, or would I have to randomly sample from the 5 different files?
", "rawMarkdown": "Sorry if this is discussed elsewhere, but is there any ordering to the training files? For example if I took only the first half of train_1.zip, would I be able to get started with a relatively representative dataset, or would I have to randomly sample from the 5 different files? ", "replies": [ { "id": 328272, "postDate": "2018-05-13T21:08:49.040Z", "content": "The event in training (or testing) datasets are completely randomized, any subset is representative of the whole.
", "rawMarkdown": "The event in training (or testing) datasets are completely randomized, any subset is representative of the whole.", "votes": 1 }, { "id": 328293, "postDate": "2018-05-13T22:08:40.160Z", "content": "Excellent, thanks for the quick response!
", "rawMarkdown": "Excellent, thanks for the quick response! " } ] }, { "id": 327513, "postDate": "2018-05-11T18:18:13.493Z", "content": "@David Is the increase in hit_ids in each event aligned with the passage of time? And also I think the close by particles have scattering effect on each other. Am I right? And what about the detectors, do the have scatteing effect on particles?
", "rawMarkdown": "@David Is the increase in hit_ids in each event aligned with the passage of time? And also I think the close by particles have scattering effect on each other. Am I right? And what about the detectors, do the have scatteing effect on particles?", "replies": [ { "id": 328273, "postDate": "2018-05-13T21:10:41.403Z", "content": "There is no information in the hit_id themselves. They might somehow reflect the passage of time, but this is because they are ordered geometrically.
", "rawMarkdown": "There is no information in the hit_id themselves. They might somehow reflect the passage of time, but this is because they are ordered geometrically.", "votes": 2 }, { "id": 328275, "postDate": "2018-05-13T21:13:01.587Z", "content": "There is no scattering of particles with other particles. Their trajectory cannot be influenced by close by other particles. Detectors have a small scattering effect on the particles, see doc.
", "rawMarkdown": "There is no scattering of particles with other particles. Their trajectory cannot be influenced by close by other particles. Detectors have a small scattering effect on the particles, see doc.", "votes": 1 }, { "id": 328632, "postDate": "2018-05-14T18:59:52.057Z", "content": "Thank you.
", "rawMarkdown": "Thank you." } ] }, { "id": 327142, "postDate": "2018-05-10T22:34:49.707Z", "content": "Hi, seems that particle velocity and momentum and charge are not associated to the hit_ids.\nso the input is only hits(possibly also cells) and output is their associated track ids?
", "rawMarkdown": "Hi, seems that particle velocity and momentum and charge are not associated to the hit_ids.\nso the input is only hits(possibly also cells) and output is their associated track ids? ", "replies": [ { "id": 327290, "postDate": "2018-05-11T07:34:38.967Z", "content": "Indeed - particle velocity and momentum are only available for the training dataset. \nThe input is: hits and cells.
", "rawMarkdown": "Indeed - particle velocity and momentum are only available for the training dataset. \nThe input is: hits and cells.", "votes": 1 }, { "id": 327357, "postDate": "2018-05-11T11:10:06.993Z", "content": "Thanks for the asnwer, i have a few more questions:\nhow does particle's initial position (contained in Particles file) relate to hit position on the particle detector(contained in Hits file )?
\nwhat is the number of hit? i mean is it number of hit generated by this particle along the track?
The initial position and momentum (from the particles file) are the physical reasons for the particle to end up on a certain detector element. This is almost deterministic, although there's a randomness caused by the interaction of the particle with the detector material.
The hit number is just a unique identification of every single hit in one event.
", "rawMarkdown": "The initial position and momentum (from the ``particles`` file) are the physical reasons for the particle to end up on a certain detector element. This is *almost* deterministic, although there's a randomness caused by the interaction of the particle with the detector material.\n\nThe hit number is just a unique identification of every single hit in one event.", "votes": 2 } ] }, { "id": 326906, "postDate": "2018-05-10T13:53:07.807Z", "content": "@David Rousseau I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?
\n\nThanks!
", "rawMarkdown": "@David Rousseau I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?\n\nThanks!" }, { "id": 326305, "postDate": "2018-05-09T14:38:19.840Z", "content": "Just a minor infliction: there is an error in Tab. 2 in \"Particle Tracking Machine Learning Challenge: Detector and Dataset\", D Rousseau et al. Ring 1 in volume 12 & 14 should have 54 modules, not 52.
\n\n[Deleted, please disregard] Barrel volume 17, layer 2 (r = 820 mm) coincides with end cap volume 9, layer 6 (z = 820 mm).
", "rawMarkdown": "Just a minor infliction: there is an error in Tab. 2 in \"*Particle Tracking Machine Learning Challenge: Detector and Dataset*\", D Rousseau et al. Ring 1 in volume 12 & 14 should have 54 modules, not 52.\n\n[Deleted, please disregard] Barrel volume 17, layer 2 (r = 820 mm) coincides with end cap volume 9, layer 6 (z = 820 mm).\n\n" }, { "id": 324626, "postDate": "2018-05-07T21:34:45.053Z", "content": "Posted kNN kernel
", "rawMarkdown": "Posted kNN kernel" }, { "id": 324124, "postDate": "2018-05-07T07:50:54.990Z", "content": "Wow that's A LOT of data! Thanks for your efforts!
", "rawMarkdown": "Wow that's A LOT of data! Thanks for your efforts!" }, { "id": 323911, "postDate": "2018-05-06T16:39:56.723Z", "content": "Hi, there will not be a Julia kernel for this competition?
", "rawMarkdown": "Hi, there will not be a Julia kernel for this competition? ", "replies": [ { "id": 323927, "postDate": "2018-05-06T17:22:49.690Z", "content": "Not from the organisers. But if a participant is providing one we will reference it here. We re aware a golang one is brewing.
", "rawMarkdown": "Not from the organisers. But if a participant is providing one we will reference it here. We re aware a golang one is brewing. ", "votes": 1 } ] }, { "id": 323521, "postDate": "2018-05-05T10:51:28.013Z", "content": "", "rawMarkdown": "[Nice coverage by nature !][1] \n\n\n [1]: https://www.nature.com/articles/d41586-018-05084-2" }, { "id": 323520, "postDate": "2018-05-05T10:49:48.013Z", "content": "Added mention of Hough transform kernel
", "rawMarkdown": "[Added mention of Hough transform kernel][1]\n\n\n [1]: https://www.kaggle.com/mikhailhushchyn/hough-transform" }, { "id": 323517, "postDate": "2018-05-05T10:45:00.737Z", "content": "Hi @David Rousseau, I want to read the contents of the train_1.zip file into my kernel for that I have used this command
\nzf = zipfile.ZipFile( '../input/train_1.zip' , 'r' ) but it gives me an error no such file or directory. what should I do?
...probably train_1.zip is already unzipped...which we recommend any way, given that there are several types of files there. Also better start with train_sample.
", "rawMarkdown": "...probably train_1.zip is already unzipped...which we recommend any way, given that there are several types of files there. Also better start with train_sample.", "votes": 1 }, { "id": 339759, "postDate": "2018-06-07T15:24:12.920Z", "content": "I believe this is because the train 2-5 files aren't available in the kernel, at least they weren't last time I checked.
\n\nTry list.files() to check
", "rawMarkdown": "I believe this is because the train 2-5 files aren't available in the kernel, at least they weren't last time I checked. \n\nTry list.files() to check " } ] }, { "id": 322411, "postDate": "2018-05-02T22:09:58.193Z", "content": "Added mention of DBScan notebook we provide
", "rawMarkdown": "Added mention of DBScan notebook we provide" }, { "id": 338696, "postDate": "2018-06-05T15:56:16.380Z", "rawMarkdown": "", "isDeleted": true }, { "id": 331572, "postDate": "2018-05-21T13:59:30.970Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 331739, "postDate": "2018-05-21T19:54:38.527Z", "content": "Your submission is required to assign a track number for all the hits (of all events). If your algorithm already does this naturally, fine. But if it refuses to cluster hits appearing unambiguous, then you can just assign them all to one track_id. Then your submission will be valid.
", "rawMarkdown": "Your submission is required to assign a track number for all the hits (of all events). If your algorithm already does this naturally, fine. But if it refuses to cluster hits appearing unambiguous, then you can just assign them all to one track_id. Then your submission will be valid." }, { "id": 331779, "postDate": "2018-05-21T20:38:02.160Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 340348, "postDate": "2018-06-09T00:57:40.863Z", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": 1 } ], "comments": [ { "id": 336546, "author_name": "marketneutral", "author_url": "", "post_date": "2018-05-31T23:02:14.223000", "content": "On a non-technical note... There is a wonderful documentary about the LHC and CERN called Particle Fever which I rewatched today. Highly recommended to put all this in context!
\n\nhttps://www.youtube.com/watch?v=PHObwzAg7Q0
\n\n
@Marketneutral thanks, I just watched it. I have seen documentaries about the LHC before but have not seen this one.
", "votes": 0, "replies": [] } ] }, { "id": 321432, "author_name": "Moritz Kiehn", "author_url": "", "post_date": "2018-05-01T09:12:02.833000", "content": "Hi Vadim,
\n\nyes, that is the metric used in this challenge.
\n\nMoritz
", "votes": 6, "replies": [ { "id": 321444, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-01T09:54:01.480000", "content": "... Moritz is the main author of this library.
", "votes": 4, "replies": [] }, { "id": 321528, "author_name": "AlanVitullo", "author_url": "", "post_date": "2018-05-01T13:36:58.443000", "content": "The library is awesome, thank you Moritz. Still trying to get it to run locally but it works like a dream on the site.
", "votes": 1, "replies": [] } ] }, { "id": 335516, "author_name": "pete", "author_url": "", "post_date": "2018-05-29T21:12:29.363000", "content": "I am curious as to the performance of the kalman filter approach. Do the organizers have an estimate of the score that this method would get?
", "votes": 3, "replies": [] }, { "id": 322426, "author_name": "anokas", "author_url": "", "post_date": "2018-05-02T22:42:57.030000", "content": "I am curious - how did you label the data?
", "votes": 3, "replies": [ { "id": 322430, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-02T23:05:41.877000", "content": "This is simulated data so we know exactly which particle has been where. Most of the 3 years spent on the preparation of this challenge was in setting up this simulation, which is sufficiently detailed that we are quite convinced that an algorithm performing well on the simulation will perform well (after adjustment) on real data coming from actual proton collision.
", "votes": 14, "replies": [] }, { "id": 322432, "author_name": "anokas", "author_url": "", "post_date": "2018-05-02T23:22:47.217000", "content": "Very interesting, thanks!
", "votes": 0, "replies": [] }, { "id": 322677, "author_name": "mrxnew", "author_url": "", "post_date": "2018-05-03T12:19:22.660000", "content": "Does the same particle_id across events point to similar particles ?
", "votes": 1, "replies": [] }, { "id": 322679, "author_name": "Andreas Salzburger", "author_url": "", "post_date": "2018-05-03T12:23:59.157000", "content": "No, the events are independent - the particle_id is just a numbering schema
", "votes": 2, "replies": [] }, { "id": 322701, "author_name": "mrxnew", "author_url": "", "post_date": "2018-05-03T13:00:50.790000", "content": "Thanks.... Using the \"truth\" file and the \"particle\" file and by calculating the distance traveled and the target location can we identify which \"particle_id\" across events are similar ?
", "votes": 0, "replies": [] }, { "id": 322711, "author_name": "Andreas Salzburger", "author_url": "", "post_date": "2018-05-03T13:38:38.253000", "content": "Somehow, yes. If you want; you can learn how trajectories are of particles with similar start parameters across events, those similarities will, however, not map onto particle_id
Hi, are the momentum values tpx, tpy, tpz for the trajectory right before the hit, or right after the hit? I'm asking because the trajectory can be modified via Multiple Coulomb Scattering, which means that momentum right before and right after the hit can be different.
", "votes": 1, "replies": [ { "id": 352535, "author_name": "Andreas Salzburger", "author_url": "", "post_date": "2018-07-04T15:29:48.720000", "content": "Hi,
\n\nfor the pixel detector, the scattering material (dominant is the support material, the module material is minimal) is after the hit creation, for the strip detector the module material is before the hit creation.
\n\nThe idea is to mimic a pixel detector with inwards facing Silicon modules mounted on an outside shell - and for the the strips with cylinder/disks where the modules are mounted.
\n\nIn addition, there is a material cylinder between pixel and strips pretending to be a pixel support tube.
", "votes": 1, "replies": [] }, { "id": 352537, "author_name": "CPMP", "author_url": "", "post_date": "2018-07-04T15:32:37.670000", "content": "Thank you for the fast answer. If I get you correctly, momentum can change right after a hit in the pixel detector, while momentum can change right before a hit in the strip detector.
", "votes": 0, "replies": [] } ] }, { "id": 347333, "author_name": "Edwin Steiner", "author_url": "", "post_date": "2018-06-24T00:20:00.903000", "content": "The participant document contains misleading information about the submission format [my emphasis]:
\n\n\n\n\nParticipants are advised to compress the file (with zip, bzip2, gzip)\n before submission.
\n
However, bzip2-compressed submission files are rejected with an error message during scoring. I found this out the hard way today. Luckily gzip worked. Would be nice to really support bzip2 as it saves a few MB of upload.
", "votes": 1, "replies": [ { "id": 347461, "author_name": "CPMP", "author_url": "", "post_date": "2018-06-24T12:24:24.917000", "content": "7z works fine, it compresses files more than other available options.
", "votes": 1, "replies": [] }, { "id": 352174, "author_name": "Edwin Steiner", "author_url": "", "post_date": "2018-07-03T20:15:08.553000", "content": "Thanks! The 7z worked and it compressed a lot better than bzip2, actually.
", "votes": 1, "replies": [] }, { "id": 352682, "author_name": "macfarll", "author_url": "", "post_date": "2018-07-04T21:49:03.540000", "content": "Top Kaggler secrets leeked! 7z for 0.8x!
", "votes": 2, "replies": [] } ] }, { "id": 321408, "author_name": "Vadim Borisov", "author_url": "", "post_date": "2018-05-01T07:50:04.633000", "content": "Hi David,
\n\nthank y'all for the competion!
\n\nCould you please verify this is the correct metric?
\n\nhttps://github.com/LAL/trackml-library/blob/master/trackml/score.py
", "votes": 1, "replies": [ { "id": 321756, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-01T21:02:52.503000", "content": "yes it is. If you have some doubt, please be more specific
", "votes": 0, "replies": [] } ] }, { "id": 328274, "author_name": "AnanthS", "author_url": "", "post_date": "2018-05-13T21:12:24.143000", "content": "I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?
\n\nThanks!
", "votes": 2, "replies": [ { "id": 330264, "author_name": "Matt", "author_url": "", "post_date": "2018-05-18T12:42:19.693000", "content": "I am also struggling to find this and the companion document that talks more about the physics of the simulation (mentioned on page two of the document). It says it's on the competition website but it's certainly not.
", "votes": 2, "replies": [] } ] }, { "id": 324148, "author_name": "Arjun Agarwal", "author_url": "", "post_date": "2018-05-07T09:09:43.760000", "content": "This post is very helpful, specially the DBSCAN example with 20% success rate gave me more grip on the problem.
", "votes": 2, "replies": [] }, { "id": 322925, "author_name": "Nicole Finnie", "author_url": "", "post_date": "2018-05-03T23:57:21.263000", "content": "@David, can we assume the remaining charge is difference between the original charge - deposited charge after the particle traverses the pixel detector, or how should we calculate the remaining charge? Some of values are 1s, does that mean the charge is fully deposited on the cell and doesn't penetrate the cell? I couldn't find the answer from the Atlas paper due to a lack of domain knowledge.\nhttps://atlas.cern/updates/physics-briefing/charged-particle-reconstruction-energy-frontier\nhttp://de.arxiv.org/pdf/1406.7690
Sorry, I think that's a bit confusing, let me explain:
\n\n1 appear.I will release shortly a Kernel explaining that.
", "votes": 4, "replies": [] }, { "id": 327110, "author_name": "Ramiro Debbe", "author_url": "", "post_date": "2018-05-10T20:42:21.300000", "content": "Hi,\nI'm trying to put cuts on 'particles' objects to make it easier to try some models. As I remember, the units in ATLAS were mm and MeV. \nI made histograms of total momentum from the particles dataframe and find the maximum value to be 400 in the event I'm looking. That looks too small for LHC if my recollection of the units is correct.\nCan you please confirm the units in the momentum values?\nThanks
", "votes": 0, "replies": [] }, { "id": 327288, "author_name": "Moritz Kiehn", "author_url": "", "post_date": "2018-05-11T07:28:56.393000", "content": "The length unit is indeed mm but here we use GeV/c as the unit for momentum. Feel free to also have a look at the detailed description on the data page. It should list the units used for all quantities (if not let us know).
", "votes": 1, "replies": [] }, { "id": 327364, "author_name": "Ramiro Debbe", "author_url": "", "post_date": "2018-05-11T11:28:58.877000", "content": "Hi Moritz, thanks for the reply. I went back to the document:
\n\n\n\nAnd found all the relevant information about units.
", "votes": 0, "replies": [] } ] }, { "id": 322731, "author_name": "hengck23", "author_url": "", "post_date": "2018-05-03T14:21:05.330000", "content": "@David Rousseau Hi, will the organizer also provide a baseline code for hough transform? What is the score for using hough transform?
", "votes": 2, "replies": [ { "id": 322864, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-03T20:03:23.630000", "content": "coming soon...
", "votes": 1, "replies": [] }, { "id": 323071, "author_name": "TeraFlops", "author_url": "", "post_date": "2018-05-04T09:36:06.430000", "content": "Hi @David Rousseau , just to get an idea if you know, what's the score of state of art algorithm on this dataset and what would be considered a meaningful realistic score (<1) to be useful in practice ?
", "votes": 3, "replies": [] }, { "id": 323249, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-04T17:39:19.973000", "content": "Hough transform posted!
", "votes": 2, "replies": [] }, { "id": 323251, "author_name": "hengck23", "author_url": "", "post_date": "2018-05-04T17:42:27.710000", "content": "@David Rousseau
\n Thanks!
@TeraFlops At the moment we do not have a score for an algorithm currently used in one of the running experiments. The code for these tend to be deeply integrated into the experiment software packages and are usually hard to run standalone. However, we hope to get a benchmark score for these algorithms later during the challenge.
\n\nMy personal guess is that they will end up with a score somewhere around 0.8-0.9, but this is really just an (educated) guess.
", "votes": 2, "replies": [] }, { "id": 326746, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2018-05-10T09:08:54.873000", "content": "@Moritz, The score equal 0.8-0.9 seems to be too optimistic to me. Not because of my lack of faith in Machine Learning, but because of the metrics used. In such a dense system, each of a few neighbouring pixels may be equally good for each of a few crossing tracks (when the differences between them are lower than the detector error), but according to the metrics used a hit/pixel either belongs to the track or does not. I do not know how \"dense\" is the system, but such groups of very similar hits/pixels (regarding the space position and signal) may be many.
", "votes": 2, "replies": [] }, { "id": 326817, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-10T11:35:15.233000", "content": "The key thing is that the density is not large when compared to the point precision. In fact, @Moritz has checked that the probability that the closest measured point from a given true point was not its corresponding measured point was a few per mille.
", "votes": 1, "replies": [] }, { "id": 326838, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2018-05-10T12:08:53.813000", "content": "@David, It is worth to know, however it is a pity - I thought the aim of the competition is to help to solve the system more complicated, not more simplified.
", "votes": 1, "replies": [] }, { "id": 327429, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-11T14:37:25.997000", "content": "The main complication is that the trajectories are built from a set of points which have to be self consistent. The randomisation due to measured x y z versus true x y z is \"blurring\" this self consistency. It is not the main complication, but it adds to it.
", "votes": 0, "replies": [] }, { "id": 339029, "author_name": "CPMP", "author_url": "", "post_date": "2018-06-06T07:06:48.317000", "content": "\n\n\nMy personal guess is that they will end up with a score somewhere around 0.8-0.9, but this is really just an (educated) guess.
\n
Are you saying you currently can get a score above 0.8 on this data?
\n\nJust curious to see if we are useful to you.
", "votes": 0, "replies": [] } ] }, { "id": 1470660, "author_name": "Zhastay Yeltay", "author_url": "", "post_date": "2021-08-13T16:37:25.890000", "content": "Nice, good job!
", "votes": 0, "replies": [] }, { "id": 1149834, "author_name": "Kasis Lundia", "author_url": "", "post_date": "2021-01-12T07:26:50.817000", "content": "where can i find the attachment?
\na introduction document for a non-physicist audience with more details and figures (attached)
The very last line of the first post here \"trackml-participant-document-particle-v1.0.pdf\" is clickable
", "votes": 1, "replies": [] }, { "id": 1149971, "author_name": "Kasis Lundia", "author_url": "", "post_date": "2021-01-12T09:16:51.870000", "content": "Found it , thanks a lot
", "votes": 0, "replies": [] } ] }, { "id": 385942, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-09-11T20:53:49.017000", "content": "Second, \"Throughput\" phase is online, see this post
", "votes": 0, "replies": [] }, { "id": 374790, "author_name": "JR", "author_url": "", "post_date": "2018-08-23T20:14:30.777000", "content": "The TrackML Kaggle competition has ended. Any participant (not necessarily with the highest score) who think they made valuable contributions with innovative algorithms (in particular if they have shared insights on the forum) are welcome to release publicly their code with an open source license and lightweight structured documentation. This will allow to compete for the « HEP meets ML » jury prizes: a NVidia V100 GPU, and two invitations to NIPS 2018 or the spring 2019 CERN workshop.See details in forum topic:
\n\nhttps://www.kaggle.com/c/trackml-particle-identification/discussion/63289
", "votes": 0, "replies": [] }, { "id": 347288, "author_name": "Aimé T Shangula", "author_url": "", "post_date": "2018-06-23T20:01:41.383000", "content": "hi David\nI have a track_id attached to this message is ggplot of this. I was wondering is the tracker supposed to unify micro and macro events, because the only way I was able to make a positive tracker was using something like the Legendrian so the tracker is not north of strangeness and spontaneous. Does this come close to what you need? How do I complete my work if I do not have access to training sets 2-5? I called the plot something like tracking the destiny of future language because the \"macro\" events tether a Morse like line. I also think the curve looks like the two body Coulomb attraction/repulsion curve over distance. help me out if it is worth it and tell me what you think.
", "votes": 0, "replies": [] }, { "id": 337460, "author_name": "charlie", "author_url": "", "post_date": "2018-06-02T22:06:54.540000", "content": "Hi,
\n\nYou mention in the pdf that heavy particles are more important to find. Are particles likely to leave a charge in the detector proportional to their weight ? Is there a defined relation between charge left on a pixel and particle weight ? I fail to find details about that.
\n\nSimilarly, I'm interested in more details about helix parameters and momentum (which you mention the particles will shed a bit of every time they cross silicon). \nThe pdf says \"The relation between the parameters of the helix and the momentum are explained in appendix ??.\" I fail to find the ?? appendix :)
\n\nCheers !
", "votes": 0, "replies": [ { "id": 338617, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-06-05T12:52:38.873000", "content": "Apologies, in my first paragraph I meant to say not heavy particles but high energy particles. To quote the pdf \"weight_pt The high energy particles (large transverse momentum pT ) are the most\ninteresting ones.\" Those ones ...
\n\nSo to rephrase said paragraph : Are particles likely to leave a charge in the detector proportional to the amount of momentum or energy they have ? (roughly EnergyxWeight = momentum if I am not mistaken). Is there a defined relation between charge left on a pixel and particle energy ? I fail to find details about that.
\n\nThanks for the good work on the doc, I liked reading it. Looking forward to the next version.
", "votes": 0, "replies": [] }, { "id": 339083, "author_name": "Art", "author_url": "", "post_date": "2018-06-06T09:03:37.897000", "content": "Multiple Coulomb Scattering is the phenomenon behind energy loss. you can google Bethe-Bloch energy loss,
We're mostly interested in the domain of few hundreds MeV of momentum till a handful of GeV's, thus you can say the dEdx (aka \"stopping power\") is proportional to the momentum. Additionally make a note that different types of particles deposit different amount of energy in different materials.
\n\n
Thanks ! I'm not much of a physicist beyond Newton, so I dont grok all of it :) but at least enough to see that it won't necessarily be easy to deduce much hint about corelation between charge left on the detector and trajectory \"straighness\" going from the theory. I'll see if I have better luck using the data set !
\n\nOne last question : does the tube also act like silicon in fuzzing the trajectory of particles ? or is it more \"transparent\", ie is Multiple Coulomb Scattering happening there as well ?
", "votes": 1, "replies": [] }, { "id": 339714, "author_name": "Art", "author_url": "", "post_date": "2018-06-07T13:26:56.173000", "content": "what do you mean by tube? if you mean the beam pipe, then in the HEP experiment I was working on it was made out of Beryllium
", "votes": 0, "replies": [] }, { "id": 339730, "author_name": "charlie", "author_url": "", "post_date": "2018-06-07T14:22:35.493000", "content": "Yes the beam pipe thanks. So does beryllium have the same properties than silicon when it comes to slightly changing the trajectories of particles ?
", "votes": 0, "replies": [] }, { "id": 339735, "author_name": "Art", "author_url": "", "post_date": "2018-06-07T14:28:11.360000", "content": "I would not care about the beam in the particular case. I don't know if they put it inside the simulation. Generally yes! it does affect the trajectory (hint: that's why they try n make it as thin and sturdy as possible and)
", "votes": 0, "replies": [] }, { "id": 339741, "author_name": "charlie", "author_url": "", "post_date": "2018-06-07T14:36:04.703000", "content": "yeah thanks, would have been interesting to know if it is accounted for in the sim for 2 reasons : \nOne: it makes it easier to decide if the collisions points that are found in the data can be precisely relied on\nTwo: I guess it has large effects on particle with trajectories somewhat tangential to the pipe, those will be very hard to link to a collision if so.
\n\nI think I can work it out looking at the truth data, but I thought I would ask as well :) a yes/no from the person who did the sim would probably save a day or two of data wrangling .. :)
", "votes": 1, "replies": [] }, { "id": 339745, "author_name": "CPMP", "author_url": "", "post_date": "2018-06-07T14:44:29.323000", "content": "@agerom, thanks for the references.
\n\nMy physics is old memories hence I am asking the question here. I am not sure the energy lost in thin detectors is really proportional to the energy of the particle, see figure 27.6 of the same paper you extracted your pictures from (http://pdg.lbl.gov/2006/reviews/passagerpp.pdf)
\n\nThe curves are pretty flat.
", "votes": 1, "replies": [] } ] }, { "id": 336083, "author_name": "marketneutral", "author_url": "", "post_date": "2018-05-31T03:09:54.307000", "content": "Can you explain why there is no timestamp for each hit in the data?
", "votes": 0, "replies": [ { "id": 336162, "author_name": "Vadim Borisov", "author_url": "", "post_date": "2018-05-31T06:54:01.513000", "content": "why do you need that?
", "votes": 0, "replies": [] }, { "id": 336269, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-31T10:42:40.967000", "content": "We don't have timing information from the Silicon detector we are using. Since all particles are flying inside out at almost the speed of light, timing info would not be useful, unless one can get it with much better than 1 nanosecond precision (light travels at 30cm per nanosecond). In fact, in real experiments we are considering to have very high precision timing detectors but this would be technically possible in only a very small subset, like one additional layer on the outside.
", "votes": 1, "replies": [] }, { "id": 336295, "author_name": "marketneutral", "author_url": "", "post_date": "2018-05-31T11:39:30.093000", "content": "@Vadim, I was was hoping there could be some opportunity for feature engineering with the hit time -- the simplest example being: if two hits had the same timestamp then they could not be of the same track.
\n\n@David, Thanks for the reply and explanation.
", "votes": 0, "replies": [] } ] }, { "id": 333171, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-24T14:49:27.203000", "content": "The \"Description\" page was incorrectly giving \"July 2018\" for the end of the first phase, in contradiction with the \"Timeline\" and \"Rules\" (which participants have signed) saying 13th August. This has just been fixed. We hope this did not confuse anyone.
", "votes": 0, "replies": [] }, { "id": 332506, "author_name": "Art", "author_url": "", "post_date": "2018-05-23T09:17:29.447000", "content": "In some cases in the Train dataset particles particle_id and truth particle_id do not match. I mean there is a discrepancy between the two files. Can you please elaborate?
", "votes": 0, "replies": [ { "id": 345006, "author_name": "Corey Carlson", "author_url": "", "post_date": "2018-06-19T04:02:59.027000", "content": "Did you get an answer why dataset particles particle_id and truth particle_id do not match? I found particle_id = 0 in truth, but there are none in the particle set with that ID. I was directed to this thread, and the trackml-participant-document, but I do not see an explanation.
", "votes": 1, "replies": [] }, { "id": 345256, "author_name": "Art", "author_url": "", "post_date": "2018-06-19T14:09:18.127000", "content": "Unfortunately I did not. I proceed with my analysis by creating an inner join of truth and particles based on the particle_id
", "votes": 0, "replies": [] }, { "id": 345264, "author_name": "charlie", "author_url": "", "post_date": "2018-06-19T14:28:24.120000", "content": "To my knowledge, the only particle_id found in truth files that cannot be found in the particle file are all id 0. This is not clearly stated anywhere, but my belief is that it's noise that your algorythm has to be able to wade thru.
\n\nIOW: all hits with particle ID 0 in the truth file should be identified and discarded by your algo.
\n\nNot sure what physics process is behind the generation of this noise, seems there are multiple candidate sources when looking at it closely. Perhaps one bit of explanation is this line in the pdf :
\n\n\"In most cases, the particles come from the production points which are in the luminous region, however a small fraction of the particles come from short-lived unstable heavy particles flying a few millimeter.\"
\n\nThe fact the hits don't have an ID could then be explained by the fact that they are not generated by a particle part of the collision byproduct simulation, but part of some noise simulation ...
", "votes": 0, "replies": [] }, { "id": 345577, "author_name": "Corey Carlson", "author_url": "", "post_date": "2018-06-20T04:27:13.630000", "content": "I found one reference on the data page that explains particle_id=0.
\nhttps://www.kaggle.com/c/trackml-particle-identification/discussion/59071
@David is there a theoretical limit to the number of tracks or number of particles?
", "votes": 0, "replies": [ { "id": 331445, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-21T08:50:14.420000", "content": "No, but given that the test events have been randomly selected from the same master dataset as the training events, you can easily infer the distribution of number of particles.
", "votes": 0, "replies": [] }, { "id": 331494, "author_name": "Rajat Goel", "author_url": "", "post_date": "2018-05-21T12:01:46.517000", "content": "Sounds good, Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 328271, "author_name": "macfarll", "author_url": "", "post_date": "2018-05-13T21:04:20.370000", "content": "Sorry if this is discussed elsewhere, but is there any ordering to the training files? For example if I took only the first half of train_1.zip, would I be able to get started with a relatively representative dataset, or would I have to randomly sample from the 5 different files?
", "votes": 0, "replies": [ { "id": 328272, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-13T21:08:49.040000", "content": "The event in training (or testing) datasets are completely randomized, any subset is representative of the whole.
", "votes": 1, "replies": [] }, { "id": 328293, "author_name": "macfarll", "author_url": "", "post_date": "2018-05-13T22:08:40.160000", "content": "Excellent, thanks for the quick response!
", "votes": 0, "replies": [] } ] }, { "id": 327513, "author_name": "bhartha", "author_url": "", "post_date": "2018-05-11T18:18:13.493000", "content": "@David Is the increase in hit_ids in each event aligned with the passage of time? And also I think the close by particles have scattering effect on each other. Am I right? And what about the detectors, do the have scatteing effect on particles?
", "votes": 0, "replies": [ { "id": 328273, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-13T21:10:41.403000", "content": "There is no information in the hit_id themselves. They might somehow reflect the passage of time, but this is because they are ordered geometrically.
", "votes": 2, "replies": [] }, { "id": 328275, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-13T21:13:01.587000", "content": "There is no scattering of particles with other particles. Their trajectory cannot be influenced by close by other particles. Detectors have a small scattering effect on the particles, see doc.
", "votes": 1, "replies": [] }, { "id": 328632, "author_name": "bhartha", "author_url": "", "post_date": "2018-05-14T18:59:52.057000", "content": "Thank you.
", "votes": 0, "replies": [] } ] }, { "id": 327142, "author_name": "Heisenberg", "author_url": "", "post_date": "2018-05-10T22:34:49.707000", "content": "Hi, seems that particle velocity and momentum and charge are not associated to the hit_ids.\nso the input is only hits(possibly also cells) and output is their associated track ids?
", "votes": 0, "replies": [ { "id": 327290, "author_name": "Andreas Salzburger", "author_url": "", "post_date": "2018-05-11T07:34:38.967000", "content": "Indeed - particle velocity and momentum are only available for the training dataset. \nThe input is: hits and cells.
", "votes": 1, "replies": [] }, { "id": 327357, "author_name": "Heisenberg", "author_url": "", "post_date": "2018-05-11T11:10:06.993000", "content": "Thanks for the asnwer, i have a few more questions:\nhow does particle's initial position (contained in Particles file) relate to hit position on the particle detector(contained in Hits file )?
\nwhat is the number of hit? i mean is it number of hit generated by this particle along the track?
The initial position and momentum (from the particles file) are the physical reasons for the particle to end up on a certain detector element. This is almost deterministic, although there's a randomness caused by the interaction of the particle with the detector material.
The hit number is just a unique identification of every single hit in one event.
", "votes": 2, "replies": [] } ] }, { "id": 326906, "author_name": "AnanthS", "author_url": "", "post_date": "2018-05-10T13:53:07.807000", "content": "@David Rousseau I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?
\n\nThanks!
", "votes": 0, "replies": [] }, { "id": 326305, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2018-05-09T14:38:19.840000", "content": "Just a minor infliction: there is an error in Tab. 2 in \"Particle Tracking Machine Learning Challenge: Detector and Dataset\", D Rousseau et al. Ring 1 in volume 12 & 14 should have 54 modules, not 52.
\n\n[Deleted, please disregard] Barrel volume 17, layer 2 (r = 820 mm) coincides with end cap volume 9, layer 6 (z = 820 mm).
", "votes": 0, "replies": [] }, { "id": 324626, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-07T21:34:45.053000", "content": "Posted kNN kernel
", "votes": 0, "replies": [] }, { "id": 324124, "author_name": "Viacheslav Patcera", "author_url": "", "post_date": "2018-05-07T07:50:54.990000", "content": "Wow that's A LOT of data! Thanks for your efforts!
", "votes": 0, "replies": [] }, { "id": 323911, "author_name": "Lazarus", "author_url": "", "post_date": "2018-05-06T16:39:56.723000", "content": "Hi, there will not be a Julia kernel for this competition?
", "votes": 0, "replies": [ { "id": 323927, "author_name": "David Rousseau", "author_url": "", "post_date": "2018-05-06T17:22:49.690000", "content": "Not from the organisers. But if a participant is providing one we will reference it here. We re aware a golang one is brewing.
", "votes": 1, "replies": [] } ] }, { "id": 323521, "author_name": "", "author_url": "", "post_date": "2018-05-05T10:51:28.013000", "content": "", "votes": 0, "replies": [] }, { "id": 323520, "author_name": "", "author_url": "", "post_date": "2018-05-05T10:49:48.013000", "content": "", "votes": 0, "replies": [] }, { "id": 323517, "author_name": "", "author_url": "", "post_date": "2018-05-05T10:45:00.737000", "content": "", "votes": 0, "replies": [ { "id": 323550, "author_name": "", "author_url": "", "post_date": "2018-05-05T13:06:46.447000", "content": "", "votes": 1, "replies": [] }, { "id": 339759, "author_name": "", "author_url": "", "post_date": "2018-06-07T15:24:12.920000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 322411, "author_name": "", "author_url": "", "post_date": "2018-05-02T22:09:58.193000", "content": "", "votes": 0, "replies": [] }, { "id": 338696, "author_name": "", "author_url": "", "post_date": "2018-06-05T15:56:16.380000", "content": "", "votes": 0, "replies": [] }, { "id": 331572, "author_name": "", "author_url": "", "post_date": "2018-05-21T13:59:30.970000", "content": "", "votes": 1, "replies": [ { "id": 331739, "author_name": "", "author_url": "", "post_date": "2018-05-21T19:54:38.527000", "content": "", "votes": 0, "replies": [] }, { "id": 331779, "author_name": "", "author_url": "", "post_date": "2018-05-21T20:38:02.160000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 340348, "author_name": "", "author_url": "", "post_date": "2018-06-09T00:57:40.863000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "321278": "Hi !\n\nWe are a team of machine learning and particle physics scientists who have teamed up for three years to set up this challenge which we hope will excite your creativity! We do need you!\n\nAll the necessary information is on the site, but to go a bit deeper we're providing here:\n\n- a introduction document for a non-physicist audience with more details and figures (attached)\n- a few starting notebooks are being submitted as kernels and referenced here. Kept deliberately minimal and unoptimised, they show different techniques:\n - [DBSCAN][1] with a few lines of preprocessing and one line calling sklearn DBSCAN clusterer one can get the non trivial score of 20%\n - [Hough Transform][2] which is a mapping between the x,y,z point space to the space of all possible helix parameter going through these points, where the clustering is done\n - [kNN][3] which does require some training\n - [this kernel show how to use the cells][4] and recompute global position\n- Wesam has provided a nice [data visualisation kernel][5]\n- [a 3D event viewer][6] where one can upload files, then rotate, zoom etc... (but some people are already having fun [with their own viewer][7])\n- and a reminder : we recommend usage of our [helper library][8] to access the data, and compute the score on a training sample. Check [here][9] on details how to use it in kernels (Thanks Wesam!). \n- the submission file should be sorted by event_id, so something like : submission.sort_values(by = [\"event_id\", \"hit_id\"], \n- a [GoLang version of the helper library and Hough transform][10] is now available\n\nOf course we'll monitor the forum and strive to respond quickly.\n\nA few tips, from the most common questions w've seen so far:\n\n - there is not as much physics knowledge involved as it may appear. There is 3D geometry for sure, but not much beyond this. The core of the challenge is to connect all the points with arcs of helices in 3D.\n- when projected on the (x,y) plane an helix is an arc of circle\n- we define r=sqrt(x^2+y^2). In the (r,z) plane, an helix is to a good approximation a straight line starting **around** (0,0). \n - an analogy : it is like sorting beans (==hit_id) into piles, and then labelling (==track_id) the piles 1 2 3 4 or 123 456 678 679 , does not matter as long as each pile is given a unique positive integer\n - don't be overwhelmed by the amount of data, we provided it because we could, but we just don't know whether such a large amount of data is needed. Trying with just 100 events in train_sample is not a waste of time, see [discussion][11]\n - we expect the cells file to be useful for the last 5-10% (if we knew exactly we would not do this challenge!). So the cells file can be ignored at the beginning.\n - we ask every hit_id to be associated to a track_id, but it is completely fine to define a garbage track (with track_id 0 maybe) will all the hits your algorithm could not assigned. This garbage track will contribute zero to the score of course, but this will make a valid contribution.\n - all events are independent entities (it would be useless to look for correlation across event of the hit or particle numbering)\n - all events from the training and testing datasets have been generated in an identical way (except the random seed of course)\n - particles do not interact with each others. A particle trajectory is not influenced in any way by other close-by particles.\n\nGood luck!\n\nThe [TrackML team][12] : Sabrina Amrouche, Paolo Calafiura, Victor Estrade \"Sorme\", Steven Farrell, CecileGermain, Vava Gligorov, Tobias Golling, Heather Gray, Isabelle Guyon \"Isabelle\", Vincenzo Innocente, Mikhail Hushhyn, Moritz Kiehn, Ed Moyse, David Rousseau, Andreas Salzburger, Andrey Ustyuzhanin, Jean-Roch Vlimant \"jr\", Yetkin Yilmaz\n\nPS : [Follow us on twitter][13] \n\nAddendum : The final write-up of this Kaggle Accuracy phase for the TrackML challenge is published : \nS.Amrouche,L.Basara,P.Calafiura,V.Estrade,S.Farrell,D.R. Ferreira, L. Finnie, N. Finnie, C. Germain, V. V. Gligorov, T. Golling, S. Gorbunov, H. Gray, I. Guyon, M. Hushchyn, V. Innocente, M. Kiehn, E. Moyse, J.-F. Puget, Y. Reina, D. Rousseau, A. Salzburger, A. Ustyuzhanin, J.-R. Vlimant, J. S. Wind, T. Xylouris, and Y. Yilmaz, \"The tracking machine learning challenge: Accuracy phase\", in The NeurIPS 2018 Competition, pp. 231–264. Springer International Publishing, Nov., 2019. arXiv:[1904.06778](https://arxiv.org/abs/1904.06778) [hep-ex].\n[doi:10.1007/978-3-030-29135-8_9](https://doi.org/10.1007/978-3-030-29135-8_9) \n\nThe final write-up of the Codalab Throughput phase is being finalized. \n\n\n [1]: https://www.kaggle.com/mikhailhushchyn/dbscan-benchmark\n [2]: https://www.kaggle.com/mikhailhushchyn/hough-transform\n [3]: https://www.kaggle.com/mikhailhushchyn/knn-approach\n [4]: https://www.kaggle.com/asalzburger/pixel-detector-cells\n [5]: https://www.kaggle.com/wesamelshamy/trackml-problem-explanation-and-data-exploration/comments\n [6]: https://emoyse.web.cern.ch/emoyse/WebEventDisplay/jsdisplay_TrackML.html\n [7]: https://www.kaggle.com/c/trackml-particle-identification/discussion/55725\n [8]: https://github.com/LAL/trackml-library\n [9]: https://www.kaggle.com/c/trackml-particle-identification/discussion/55753\n [10]: https://www.kaggle.com/c/trackml-particle-identification/discussion/56858\n [11]: https://www.kaggle.com/c/trackml-particle-identification/discussion/55754\n [12]: https://sites.google.com/site/trackmlparticle/organisation\n [13]: https://twitter.com/trackmllhc", "336546": "On a non-technical note... There is a wonderful documentary about the LHC and CERN called **Particle Fever** which I rewatched today. Highly recommended to put all this in context!\n\nhttps://www.youtube.com/watch?v=PHObwzAg7Q0\n\n\n![LHC][2]\n\n [2]: https://www.metroweekly.com/articles/attachments/2014-03-20_film_8935_8813.jpg", "321432": "Hi Vadim,\n\nyes, that is the metric used in this challenge.\n\nMoritz", "335516": "I am curious as to the performance of the kalman filter approach. Do the organizers have an estimate of the score that this method would get?", "322426": "I am curious - how did you label the data?", "352517": "Hi, are the momentum values tpx, tpy, tpz for the trajectory right before the hit, or right after the hit? I'm asking because the trajectory can be modified via Multiple Coulomb Scattering, which means that momentum right before and right after the hit can be different.", "347333": "The participant document contains misleading information about the submission format [my emphasis]: \n\n> Participants are advised to compress the file (with zip, **bzip2**, gzip)\n> before submission.\n\nHowever, bzip2-compressed submission files are rejected with an error message during scoring. I found this out the hard way today. Luckily gzip worked. Would be nice to really support bzip2 as it saves a few MB of upload.", "321408": "Hi David,\n\nthank y'all for the competion! \n\nCould you please verify this is the correct metric?\n\nhttps://github.com/LAL/trackml-library/blob/master/trackml/score.py", "328274": "I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?\n\nThanks!", "324148": "This post is very helpful, specially the DBSCAN example with 20% success rate gave me more grip on the problem.", "322925": "@David, can we assume the remaining charge is difference between the `original charge - deposited charge` after the particle traverses the pixel detector, or how should we calculate the remaining charge? Some of values are `1`s, does that mean the charge is fully deposited on the cell and doesn't penetrate the cell? I couldn't find the answer from the Atlas paper due to a lack of domain knowledge.\nhttps://atlas.cern/updates/physics-briefing/charged-particle-reconstruction-energy-frontier\nhttp://de.arxiv.org/pdf/1406.7690\n", "322731": "@David Rousseau Hi, will the organizer also provide a baseline code for hough transform? What is the score for using hough transform?", "1470660": "Nice, good job!", "1149834": "where can i find the attachment?\na introduction document for a non-physicist audience with more details and figures (attached)", "385942": "Second, \"Throughput\" phase is online, see [this post][1]\n\n\n [1]: https://www.kaggle.com/c/trackml-particle-identification/discussion/65525", "374790": "The TrackML Kaggle competition has ended. Any participant (not necessarily with the highest score) who think they made valuable contributions with innovative algorithms (in particular if they have shared insights on the forum) are welcome to release publicly their code with an open source license and lightweight structured documentation. This will allow to compete for the « HEP meets ML » jury prizes: a NVidia V100 GPU, and two invitations to NIPS 2018 or the spring 2019 CERN workshop.See details in forum topic:\n\nhttps://www.kaggle.com/c/trackml-particle-identification/discussion/63289", "347288": "hi David\nI have a track_id attached to this message is ggplot of this. I was wondering is the tracker supposed to unify micro and macro events, because the only way I was able to make a positive tracker was using something like the Legendrian so the tracker is not north of strangeness and spontaneous. Does this come close to what you need? How do I complete my work if I do not have access to training sets 2-5? I called the plot something like tracking the destiny of future language because the \"macro\" events tether a Morse like line. I also think the curve looks like the two body Coulomb attraction/repulsion curve over distance. help me out if it is worth it and tell me what you think.", "337460": "Hi,\n\nYou mention in the pdf that heavy particles are more important to find. Are particles likely to leave a charge in the detector proportional to their weight ? Is there a defined relation between charge left on a pixel and particle weight ? I fail to find details about that.\n\nSimilarly, I'm interested in more details about helix parameters and momentum (which you mention the particles will shed a bit of every time they cross silicon). \nThe pdf says \"The relation between the parameters of the helix and the momentum are explained in appendix ??.\" I fail to find the ?? appendix :)\n\nCheers !", "336083": "Can you explain why there is no timestamp for each hit in the data?", "333171": "The \"Description\" page was incorrectly giving \"July 2018\" for the end of the first phase, in contradiction with the \"Timeline\" and \"Rules\" (which participants have signed) saying 13th August. This has just been fixed. We hope this did not confuse anyone.", "332506": "In some cases in the Train dataset *particles* particle_id and *truth* particle_id do not match. I mean there is a discrepancy between the two files. Can you please elaborate? ", "331178": "@David is there a theoretical limit to the number of tracks or number of particles?", "328271": "Sorry if this is discussed elsewhere, but is there any ordering to the training files? For example if I took only the first half of train_1.zip, would I be able to get started with a relatively representative dataset, or would I have to randomly sample from the 5 different files? ", "327513": "@David Is the increase in hit_ids in each event aligned with the passage of time? And also I think the close by particles have scattering effect on each other. Am I right? And what about the detectors, do the have scatteing effect on particles?", "327142": "Hi, seems that particle velocity and momentum and charge are not associated to the hit_ids.\nso the input is only hits(possibly also cells) and output is their associated track ids? ", "326906": "@David Rousseau I can't seem to find the relation between the parameters of the helix and momentum in the appendix as mentioned in section 3.4 of the document. Can you please help me find it?\n\nThanks!", "326305": "Just a minor infliction: there is an error in Tab. 2 in \"*Particle Tracking Machine Learning Challenge: Detector and Dataset*\", D Rousseau et al. Ring 1 in volume 12 & 14 should have 54 modules, not 52.\n\n[Deleted, please disregard] Barrel volume 17, layer 2 (r = 820 mm) coincides with end cap volume 9, layer 6 (z = 820 mm).\n\n", "324626": "Posted kNN kernel", "324124": "Wow that's A LOT of data! Thanks for your efforts!", "323911": "Hi, there will not be a Julia kernel for this competition? ", "323521": "[Nice coverage by nature !][1] \n\n\n [1]: https://www.nature.com/articles/d41586-018-05084-2", "323520": "[Added mention of Hough transform kernel][1]\n\n\n [1]: https://www.kaggle.com/mikhailhushchyn/hough-transform", "323517": "Hi @David Rousseau, I want to read the contents of the train_1.zip file into my kernel for that I have used this command \nzf = zipfile.ZipFile( '../input/train_1.zip' , 'r' ) but it gives me an error no such file or directory. what should I do?", "322411": "Added mention of DBScan notebook we provide", "338696": "", "331572": "", "340348": "Thanks!" } }