In the data set provided to us we have event ids and batch ids. I wanted to understand what this information means, especially the batch ids.
\nIf I understand it correctly, an event in a particle physics detector is an \"event of interaction\". What physically happens in this moment is that some particle starts an interaction which can be registered by the particle detector. What people see is some combination of signals on the detector, which show that something happened.
\nHowever, it is unclear to me what a batch is in this case. For example, in accelerator-based particle physics experiments, the particles are \"moving\" in the accelerators not one by one, but in batches, which have some spread, dencity, time intervals etc. But what about this case? Here one tries to detect the interactions of single particles from space, which on a very general level are not always related to each other. How one groups the events to batches then? By the amount of events? By possible physics behind the events? Or just by time?
\nThe main aim of this question is also to understand if the batch id is \"just an id\", or if it has some interesting nature behind?
\nThank you in advance!
", "messages": [ { "id": "2111097", "postDate": "01/22/2023 16:00:08", "content": "In the data set provided to us we have event ids and batch ids. I wanted to understand what this information means, especially the batch ids.
\nIf I understand it correctly, an event in a particle physics detector is an \"event of interaction\". What physically happens in this moment is that some particle starts an interaction which can be registered by the particle detector. What people see is some combination of signals on the detector, which show that something happened.
\nHowever, it is unclear to me what a batch is in this case. For example, in accelerator-based particle physics experiments, the particles are \"moving\" in the accelerators not one by one, but in batches, which have some spread, dencity, time intervals etc. But what about this case? Here one tries to detect the interactions of single particles from space, which on a very general level are not always related to each other. How one groups the events to batches then? By the amount of events? By possible physics behind the events? Or just by time?
\nThe main aim of this question is also to understand if the batch id is \"just an id\", or if it has some interesting nature behind?
\nThank you in advance!
", "rawMarkdown": "In the data set provided to us we have event ids and batch ids. I wanted to understand what this information means, especially the batch ids.\n\nIf I understand it correctly, an event in a particle physics detector is an \"event of interaction\". What physically happens in this moment is that some particle starts an interaction which can be registered by the particle detector. What people see is some combination of signals on the detector, which show that something happened.\n\nHowever, it is unclear to me what a batch is in this case. For example, in accelerator-based particle physics experiments, the particles are \"moving\" in the accelerators not one by one, but in batches, which have some spread, dencity, time intervals etc. But what about this case? Here one tries to detect the interactions of single particles from space, which on a very general level are not always related to each other. How one groups the events to batches then? By the amount of events? By possible physics behind the events? Or just by time?\n\nThe main aim of this question is also to understand if the batch id is \"just an id\", or if it has some interesting nature behind?\n\nThank you in advance!", "votes": null }, { "id": "2111142", "postDate": "01/22/2023 16:21:04", "content": "My inference was the batch id are just batches of non related events (chunked in smaller batches for convenience sake) as they have not explicitly stated otherwise.
", "rawMarkdown": "My inference was the batch id are just batches of non related events (chunked in smaller batches for convenience sake) as they have not explicitly stated otherwise.", "votes": null }, { "id": "2111143", "postDate": "01/22/2023 16:22:32", "content": "I think it's a very good question. Section 3. of https://arxiv.org/pdf/1612.05093.pdf mentions yearly calibration of the in-ice detectors. Even if there's no physics reason behind the batching, can we assume that the same calibration holds within one batch?
", "rawMarkdown": "I think it's a very good question. Section 3. of https://arxiv.org/pdf/1612.05093.pdf mentions yearly calibration of the in-ice detectors. Even if there's no physics reason behind the batching, can we assume that the same calibration holds within one batch?", "votes": null }, { "id": "2111156", "postDate": "01/22/2023 16:30:30", "content": "interesting - thanks for sharing
", "rawMarkdown": "interesting - thanks for sharing", "votes": null }, { "id": "2111192", "postDate": "01/22/2023 17:05:19", "content": "Yes, I agree. However, then it is unclear to me why we have this information. I assume that an event_id is unique, because the result of the prediction should only contain the event id. But I will check this for sure in an EDA.
\nThat is why it puzzles me a bit. There is no explanation if the events are batched by time, by memory or however. Maybe batch ids make sence for data collection. Then we have it just to match the different data sources
Kaggle sometimes batches large datasets to make things easier to navigate.
\nIn image competitions, for example, having 100 folders with 10,000 images is easier to manage and navigate than a single folder with 1,000,000 images.
\nIn this case, they have chosen to break the dataset into 660 smaller parts instead of a single monolithic parquet file that you would need huge amounts of RAM to open. I don't think there is any other meaning behind the batch ids
", "rawMarkdown": "Kaggle sometimes batches large datasets to make things easier to navigate. \n\nIn image competitions, for example, having 100 folders with 10,000 images is easier to manage and navigate than a single folder with 1,000,000 images.\n\nIn this case, they have chosen to break the dataset into 660 smaller parts instead of a single monolithic parquet file that you would need huge amounts of RAM to open. I don't think there is any other meaning behind the batch ids", "votes": null }, { "id": "2111240", "postDate": "01/22/2023 18:04:12", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "2111808", "postDate": "01/23/2023 07:59:21", "content": "This is indeed the case. The batches are random sub-samples provided for convenience.
", "rawMarkdown": "This is indeed the case. The batches are random sub-samples provided for convenience.", "votes": null }, { "id": "2112765", "postDate": "01/23/2023 20:12:14", "content": "Others have already answered in terms of batches. Further, we know from discussion response by organizer that all the data is simulated. I believe it is a safe assumption all train and test data in all batches was simulated in the same exact manner.
\nThe rest is purely my own assumptions from what we know:
\nThe meaning of an event, then, in this context, is that it is a generated simulated event including 1 neutrino event from a neutrino with specific angle (labeled in train data, hidden in test data). AND also including 0 to N non-neutrino 'noise' events and 0 to M other non-event based noise. At least that's my very basic assumption. For example, there could be one or several event(s) from a non-neutrino source adding pattern-based noise, on top of ambient noise that triggers sensors in a random or random-ish fashion.
", "rawMarkdown": "Others have already answered in terms of batches. Further, we know from discussion response by organizer that all the data is simulated. I believe it is a safe assumption all train and test data in all batches was simulated in the same exact manner.\n\nThe rest is purely my own assumptions from what we know:\n\nThe meaning of an event, then, in this context, is that it is a generated simulated event including 1 neutrino event from a neutrino with specific angle (labeled in train data, hidden in test data). AND also including 0 to N non-neutrino 'noise' events and 0 to M other non-event based noise. At least that's my very basic assumption. For example, there could be one or several event(s) from a non-neutrino source adding pattern-based noise, on top of ambient noise that triggers sensors in a random or random-ish fashion.", "votes": null } ], "comments": [ { "id": 2111142, "author_name": "seanmacmaghnusa", "author_url": "", "post_date": "01/22/2023 16:21:04", "content": "My inference was the batch id are just batches of non related events (chunked in smaller batches for convenience sake) as they have not explicitly stated otherwise.
", "votes": null, "replies": [ { "id": 2111192, "author_name": "gannadolinska", "author_url": "", "post_date": "01/22/2023 17:05:19", "content": "Yes, I agree. However, then it is unclear to me why we have this information. I assume that an event_id is unique, because the result of the prediction should only contain the event id. But I will check this for sure in an EDA.
\nThat is why it puzzles me a bit. There is no explanation if the events are batched by time, by memory or however. Maybe batch ids make sence for data collection. Then we have it just to match the different data sources
Kaggle sometimes batches large datasets to make things easier to navigate.
\nIn image competitions, for example, having 100 folders with 10,000 images is easier to manage and navigate than a single folder with 1,000,000 images.
\nIn this case, they have chosen to break the dataset into 660 smaller parts instead of a single monolithic parquet file that you would need huge amounts of RAM to open. I don't think there is any other meaning behind the batch ids
", "votes": null, "replies": [ { "id": 2111240, "author_name": "gannadolinska", "author_url": "", "post_date": "01/22/2023 18:04:12", "content": "Thank you!
", "votes": null, "replies": [] }, { "id": 2111808, "author_name": "rasmusrse", "author_url": "", "post_date": "01/23/2023 07:59:21", "content": "This is indeed the case. The batches are random sub-samples provided for convenience.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2111143, "author_name": "balintfodor", "author_url": "", "post_date": "01/22/2023 16:22:32", "content": "I think it's a very good question. Section 3. of https://arxiv.org/pdf/1612.05093.pdf mentions yearly calibration of the in-ice detectors. Even if there's no physics reason behind the batching, can we assume that the same calibration holds within one batch?
", "votes": null, "replies": [ { "id": 2111156, "author_name": "seanmacmaghnusa", "author_url": "", "post_date": "01/22/2023 16:30:30", "content": "interesting - thanks for sharing
", "votes": null, "replies": [] } ] }, { "id": 2112765, "author_name": "roberthatch", "author_url": "", "post_date": "01/23/2023 20:12:14", "content": "Others have already answered in terms of batches. Further, we know from discussion response by organizer that all the data is simulated. I believe it is a safe assumption all train and test data in all batches was simulated in the same exact manner.
\nThe rest is purely my own assumptions from what we know:
\nThe meaning of an event, then, in this context, is that it is a generated simulated event including 1 neutrino event from a neutrino with specific angle (labeled in train data, hidden in test data). AND also including 0 to N non-neutrino 'noise' events and 0 to M other non-event based noise. At least that's my very basic assumption. For example, there could be one or several event(s) from a non-neutrino source adding pattern-based noise, on top of ambient noise that triggers sensors in a random or random-ish fashion.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2111097": "In the data set provided to us we have event ids and batch ids. I wanted to understand what this information means, especially the batch ids.\n\nIf I understand it correctly, an event in a particle physics detector is an \"event of interaction\". What physically happens in this moment is that some particle starts an interaction which can be registered by the particle detector. What people see is some combination of signals on the detector, which show that something happened.\n\nHowever, it is unclear to me what a batch is in this case. For example, in accelerator-based particle physics experiments, the particles are \"moving\" in the accelerators not one by one, but in batches, which have some spread, dencity, time intervals etc. But what about this case? Here one tries to detect the interactions of single particles from space, which on a very general level are not always related to each other. How one groups the events to batches then? By the amount of events? By possible physics behind the events? Or just by time?\n\nThe main aim of this question is also to understand if the batch id is \"just an id\", or if it has some interesting nature behind?\n\nThank you in advance!", "2111142": "My inference was the batch id are just batches of non related events (chunked in smaller batches for convenience sake) as they have not explicitly stated otherwise.", "2111143": "I think it's a very good question. Section 3. of https://arxiv.org/pdf/1612.05093.pdf mentions yearly calibration of the in-ice detectors. Even if there's no physics reason behind the batching, can we assume that the same calibration holds within one batch?", "2111156": "interesting - thanks for sharing", "2111192": "Yes, I agree. However, then it is unclear to me why we have this information. I assume that an event_id is unique, because the result of the prediction should only contain the event id. But I will check this for sure in an EDA. \nThat is why it puzzles me a bit. There is no explanation if the events are batched by time, by memory or however. Maybe batch ids make sence for data collection. Then we have it just to match the different data sources", "2111220": "Kaggle sometimes batches large datasets to make things easier to navigate. \n\nIn image competitions, for example, having 100 folders with 10,000 images is easier to manage and navigate than a single folder with 1,000,000 images.\n\nIn this case, they have chosen to break the dataset into 660 smaller parts instead of a single monolithic parquet file that you would need huge amounts of RAM to open. I don't think there is any other meaning behind the batch ids", "2111240": "Thank you!", "2111808": "This is indeed the case. The batches are random sub-samples provided for convenience.", "2112765": "Others have already answered in terms of batches. Further, we know from discussion response by organizer that all the data is simulated. I believe it is a safe assumption all train and test data in all batches was simulated in the same exact manner.\n\nThe rest is purely my own assumptions from what we know:\n\nThe meaning of an event, then, in this context, is that it is a generated simulated event including 1 neutrino event from a neutrino with specific angle (labeled in train data, hidden in test data). AND also including 0 to N non-neutrino 'noise' events and 0 to M other non-event based noise. At least that's my very basic assumption. For example, there could be one or several event(s) from a non-neutrino source adding pattern-based noise, on top of ambient noise that triggers sensors in a random or random-ish fashion." }, "source": "meta" }