{ "id": 394792, "title": "Information available during \"inference\"", "url": "/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/394792", "author_name": "", "post_date": "2023-03-14T21:39:14.411310900Z", "votes": 4, "comment_count": 10, "views": 0, "content": "

Dear hosts,

\n

I very much appreciate that you provide a lot of information on the data page. However, I want to make sure that I understand what kind of raw variables are available when predicting the samples from the test set for the public/private LB.

\n
    \n
  1. An \"Id\" occurs only in exactly one of those sets: train, public test set, private test set? (Asking this because 003f117e14 occurs in the train AND test set in the interactive notebook version)
    2. \n
  2. For every \"Id\", irrespective in which of the 3 set it occurs (train, public test set, private test set), I can merge subjects.csv to it and tdcsfog_metadata.csv or defog_metadata.csv subject to the data set the \"Id\" is coming from. Correct?
    3. \n
  3. Can you please elaborate on \"The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets.\"? The words public/hidden confuses me, because we have a public version of these csv-files when opening a kaggle notebook and loading the data set. Then there might by a different public version for scoring the public LB and a third hidden/private version for scoring the private LB.
    4. \n
\n

Thank you very much!

", "messages": [ { "id": "2182001", "postDate": "03/14/2023 21:39:14", "content": "

Dear hosts,

\n

I very much appreciate that you provide a lot of information on the data page. However, I want to make sure that I understand what kind of raw variables are available when predicting the samples from the test set for the public/private LB.

\n
    \n
  1. An \"Id\" occurs only in exactly one of those sets: train, public test set, private test set? (Asking this because 003f117e14 occurs in the train AND test set in the interactive notebook version)
    2. \n
  2. For every \"Id\", irrespective in which of the 3 set it occurs (train, public test set, private test set), I can merge subjects.csv to it and tdcsfog_metadata.csv or defog_metadata.csv subject to the data set the \"Id\" is coming from. Correct?
    3. \n
  3. Can you please elaborate on \"The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets.\"? The words public/hidden confuses me, because we have a public version of these csv-files when opening a kaggle notebook and loading the data set. Then there might by a different public version for scoring the public LB and a third hidden/private version for scoring the private LB.
    4. \n
\n

Thank you very much!

", "rawMarkdown": "Dear hosts,\n\nI very much appreciate that you provide a lot of information on the [data](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/data) page. However, I want to make sure that I understand what kind of raw variables are available when predicting the samples from the test set for the public/private LB. \n\n1. An \"Id\" occurs only in exactly one of those sets: train, public test set, private test set? (Asking this because 003f117e14 occurs in the train AND test set in the interactive notebook version)\n2. For every \"Id\", irrespective in which of the 3 set it occurs (train, public test set, private test set), I can merge subjects.csv to it and tdcsfog_metadata.csv or defog_metadata.csv subject to the data set the \"Id\" is coming from. Correct?\n3. Can you please elaborate on \"The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets.\"? The words public/hidden confuses me, because we have a public version of these csv-files when opening a kaggle notebook and loading the data set. Then there might by a different public version for scoring the public LB and a third hidden/private version for scoring the private LB.\n\nThank you very much!", "votes": null }, { "id": "2183008", "postDate": "03/15/2023 12:29:07", "content": "

Hi @dv0048 ,

\n

There are two versions of the dataset:

\n\n

The hidden (or private) dataset has the entire test data. The words \"public\" and \"private\" here have nothing to do with the public and private leaderboards. You don't know which test id goes with the public or private leaderboard.

\n
    \n
  1. In the hidden dataset, every id occurs only once, either in train or in test (or in unlabeled).
    2. \n
  2. Correct. This metadata is available for the test set series.
    3. \n
  3. This is just saying that these files don't contain any information about test set series, only the training set. The training set is the same in the public and hidden versions, so these files are also the same in both versions.
    4. \n
\n

Hope this answers your questions!

", "rawMarkdown": "Hi @dv0048 ,\n\nThere are two versions of the dataset:\n- The *public* version, which you see on the Data page and is available to you in interactive notebook sessions. The public version only contains training data. The files in `test` are only examples in the right format to help you write your code; you should ignore the ids and the actual contents.\n- The *private* or *hidden* version, which you cannot see but is available to your notebook when you submit. This version contains both the training data and the actual test data. To get the ids of the test series, you can either read them out of the `test` folder or out of the `sample_submission.csv` file.\n\nThe hidden (or private) dataset has the entire test data. The words \"public\" and \"private\" here have nothing to do with the public and private leaderboards. You don't know which test id goes with the public or private leaderboard.\n\n1. In the hidden dataset, every id occurs only once, either in `train` or in `test` (or in `unlabeled`).\n2. Correct. This metadata is available for the test set series.\n3. This is just saying that these files don't contain any information about test set series, only the training set. The training set is the same in the public and hidden versions, so these files are also the same in both versions.\n\nHope this answers your questions!", "votes": null }, { "id": "2183361", "postDate": "03/15/2023 15:52:11", "content": "

Yes. Thank you very much for the quick reply!

", "rawMarkdown": "Yes. Thank you very much for the quick reply!", "votes": null }, { "id": "2223037", "postDate": "04/15/2023 18:41:47", "content": "

Hello @ryanholbrook

\n

When you say:

\n
\n

\"The private or hidden version, which you cannot see but is available to your notebook when you submit. This version contains both the training data and the actual test data. To get the ids of the test series, you can either read them out of the test folder or out of the sample_submission.csv file.\"

\n
\n

I only see 2 unique trial \"Id\" values, 02ab235146 or 003f117e14 from the test folder and the sample_submission.csv file, but we are told in the Data Splits section of the \"Data\" tab that:

\n
\n

\"The test set contains about 250 data series. The series from the tdcsfog and defog are in a proportion similar to that of the training set. These series have subjects that are entirely distinct from those in the training set.\"

\n
\n

So then how can we know what the Id's are for the 250 trials of the private testing set? How then will we accurately name the \"Id\" column in our outputted submission.csv file if we do not know which Id is related to the private test data that we predicting for? Can we obtain the metadata for these trial Id's or subject Id's? What are we actually predicting? (When I say \"trial\" or \"trial id\" I mean the event of time in which the subject is walking, turning, going through door ways, etc).

\n

I have successfully submitted two notebooks (and received poor scores) which simply have zeros for the \"StartHesitation\", \"Turn\", and \"Walking\" columns (essentially predicting no FOG events throughout the trials). I noticed that the submission score is only based on the submission.csv output that my notebook saves as a .csv with the shape of (286370, 4). I have no statistical model in my notebook submissions. How then is my submission.csv file used to determine the precision for all 250 hidden trials?

\n

I also noticed that the \"Data Split\" section states the following:

\n
\n

\"When your submission is scored, this example test data will be replaced with the full test set.\"

\n
\n

What does this mean? What is being replaced? How are the 250 trials of the private test data being predicted on if we do not even have the accelerometer data for those 250 trials?

\n

I feel as though I am missing something very clear here, even after reading the documents thoroughly. I sincerely appreciate your time and effort in clarifying my questions.

", "rawMarkdown": "Hello @ryanholbrook \n\nWhen you say:\n\n>\"The private or hidden version, which you cannot see but is available to your notebook when you submit. This version contains both the training data and the actual test data. To get the ids of the test series, you can either read them out of the test folder or out of the sample_submission.csv file.\"\n\nI only see 2 unique trial \"Id\" values, 02ab235146 or 003f117e14 from the `test` folder and the `sample_submission.csv` file, but we are told in the **Data Splits** section of the \"Data\" tab that:\n\n>\"The test set contains about 250 data series. The series from the tdcsfog and defog are in a proportion similar to that of the training set. These series have subjects that are entirely distinct from those in the training set.\"\n\nSo then how can we know what the Id's are for the 250 trials of the private testing set? How then will we accurately name the \"Id\" column in our outputted `submission.csv` file if we do not know which Id is related to the *private test* data that we predicting for? Can we obtain the metadata for these trial Id's or subject Id's? What are we actually predicting? (When I say \"trial\" or \"trial id\" I mean the event of time in which the subject is walking, turning, going through door ways, etc).\n\nI have successfully submitted two notebooks (and received poor scores) which simply have zeros for the \"StartHesitation\", \"Turn\", and \"Walking\" columns (essentially predicting no FOG events throughout the trials). I noticed that the submission score is only based on the `submission.csv` output that my notebook saves as a .csv with the shape of (286370, 4). I have no statistical model in my notebook submissions. How then is my `submission.csv` file used to determine the precision for all 250 hidden trials?\n\nI also noticed that the \"Data Split\" section states the following:\n\n>\"When your submission is scored, this example test data will be replaced with the full test set.\"\n\nWhat does this mean? What is being replaced? How are the 250 trials of the private test data being predicted on if we do not even have the accelerometer data for those 250 trials? \n\nI feel as though I am missing something very clear here, even after reading the documents thoroughly. I sincerely appreciate your time and effort in clarifying my questions.", "votes": null }, { "id": "2224497", "postDate": "04/17/2023 11:38:26", "content": "

Hi @williammcintoshpdx ,

\n

The files you see in the test set aren't the actual test set. They are just examples in the right format to help you write code. When you submit your notebook, we rerun it against a hidden test set, which you cannot see. This hidden test set has a test folder with all ~250 files like:

\n
test/defog/\n   abc012.csv\n   def345.csv\n   ...\ntest/tdcsfog/\n   ghi678.csv\n   ...\n
\n

Since you don't know the actual names of the files, you have to read them programmatically. (You could use os.glob or pathlib.Path.glob, say.)

\n

Hope this answers your question!

\n

Ryan

", "rawMarkdown": "Hi @williammcintoshpdx ,\n\nThe files you see in the `test` set aren't the actual test set. They are just examples in the right format to help you write code. When you submit your notebook, we rerun it against a hidden test set, which you cannot see. This hidden test set has a `test` folder with all ~250 files like:\n\n```\ntest/defog/\n abc012.csv\n def345.csv\n ...\ntest/tdcsfog/\n ghi678.csv\n ...\n```\n\nSince you don't know the actual names of the files, you have to read them programmatically. (You could use `os.glob` or `pathlib.Path.glob`, say.)\n\nHope this answers your question!\n\nRyan", "votes": null }, { "id": "2224738", "postDate": "04/17/2023 16:05:08", "content": "

OH! I think I better understand by your response. When we submit our notebook and set its corresponding output submission.csv file, the contents of the test \"folder\" are replaced during inference time. Is that correct?

", "rawMarkdown": "OH! I think I better understand by your response. When we submit our notebook and set its corresponding output submission.csv file, the contents of the test \"folder\" are replaced during inference time. Is that correct?", "votes": null }, { "id": "2224828", "postDate": "04/17/2023 17:23:45", "content": "

Yep! Some of the other files change too if they contain info about the test set. The hidden sample_submission.csv for instance has the full set of ids needed for your submission.csv. See the Data Splits section in the data description for the details.

", "rawMarkdown": "Yep! Some of the other files change too if they contain info about the test set. The hidden `sample_submission.csv` for instance has the full set of ids needed for your `submission.csv`. See the *Data Splits* section in the data description for the details.", "votes": null }, { "id": "2225381", "postDate": "04/18/2023 06:11:55", "content": "

Thks for asking such questions, my confusion is same as yours. this discussion really helps me a lot😄

", "rawMarkdown": "Thks for asking such questions, my confusion is same as yours. this discussion really helps me a lot😄", "votes": null }, { "id": "2228827", "postDate": "04/20/2023 21:05:50", "content": "

Thank you @ryanholbrook!

\n

Does this also mean that for the defog private test datasets that we will also have access to the Tasks table for those experiments during submission (inference) time?

", "rawMarkdown": "Thank you @ryanholbrook!\n\nDoes this also mean that for the defog private test datasets that we will also have access to the `Tasks` table for those experiments during submission (inference) time?", "votes": null }, { "id": "2228873", "postDate": "04/20/2023 22:09:50", "content": "

No, as I mentioned above The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets. from the data description implies that info from these files is only available for the training set.

", "rawMarkdown": "No, as I mentioned above `The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets.` from the data description implies that info from these files is only available for the training set.", "votes": null }, { "id": "2229756", "postDate": "04/21/2023 16:47:44", "content": "

okay thank you for the further clarification!

", "rawMarkdown": "okay thank you for the further clarification!", "votes": null } ], "comments": [ { "id": 2183008, "author_name": "ryanholbrook", "author_url": "", "post_date": "03/15/2023 12:29:07", "content": "

Hi @dv0048 ,

\n

There are two versions of the dataset:

\n\n

The hidden (or private) dataset has the entire test data. The words \"public\" and \"private\" here have nothing to do with the public and private leaderboards. You don't know which test id goes with the public or private leaderboard.

\n
    \n
  1. In the hidden dataset, every id occurs only once, either in train or in test (or in unlabeled).
    2. \n
  2. Correct. This metadata is available for the test set series.
    3. \n
  3. This is just saying that these files don't contain any information about test set series, only the training set. The training set is the same in the public and hidden versions, so these files are also the same in both versions.
    4. \n
\n

Hope this answers your questions!

", "votes": null, "replies": [ { "id": 2183361, "author_name": "dv0048", "author_url": "", "post_date": "03/15/2023 15:52:11", "content": "

Yes. Thank you very much for the quick reply!

", "votes": null, "replies": [] }, { "id": 2223037, "author_name": "williammcintoshpdx", "author_url": "", "post_date": "04/15/2023 18:41:47", "content": "

Hello @ryanholbrook

\n

When you say:

\n
\n

\"The private or hidden version, which you cannot see but is available to your notebook when you submit. This version contains both the training data and the actual test data. To get the ids of the test series, you can either read them out of the test folder or out of the sample_submission.csv file.\"

\n
\n

I only see 2 unique trial \"Id\" values, 02ab235146 or 003f117e14 from the test folder and the sample_submission.csv file, but we are told in the Data Splits section of the \"Data\" tab that:

\n
\n

\"The test set contains about 250 data series. The series from the tdcsfog and defog are in a proportion similar to that of the training set. These series have subjects that are entirely distinct from those in the training set.\"

\n
\n

So then how can we know what the Id's are for the 250 trials of the private testing set? How then will we accurately name the \"Id\" column in our outputted submission.csv file if we do not know which Id is related to the private test data that we predicting for? Can we obtain the metadata for these trial Id's or subject Id's? What are we actually predicting? (When I say \"trial\" or \"trial id\" I mean the event of time in which the subject is walking, turning, going through door ways, etc).

\n

I have successfully submitted two notebooks (and received poor scores) which simply have zeros for the \"StartHesitation\", \"Turn\", and \"Walking\" columns (essentially predicting no FOG events throughout the trials). I noticed that the submission score is only based on the submission.csv output that my notebook saves as a .csv with the shape of (286370, 4). I have no statistical model in my notebook submissions. How then is my submission.csv file used to determine the precision for all 250 hidden trials?

\n

I also noticed that the \"Data Split\" section states the following:

\n
\n

\"When your submission is scored, this example test data will be replaced with the full test set.\"

\n
\n

What does this mean? What is being replaced? How are the 250 trials of the private test data being predicted on if we do not even have the accelerometer data for those 250 trials?

\n

I feel as though I am missing something very clear here, even after reading the documents thoroughly. I sincerely appreciate your time and effort in clarifying my questions.

", "votes": null, "replies": [ { "id": 2224497, "author_name": "ryanholbrook", "author_url": "", "post_date": "04/17/2023 11:38:26", "content": "

Hi @williammcintoshpdx ,

\n

The files you see in the test set aren't the actual test set. They are just examples in the right format to help you write code. When you submit your notebook, we rerun it against a hidden test set, which you cannot see. This hidden test set has a test folder with all ~250 files like:

\n
test/defog/\n   abc012.csv\n   def345.csv\n   ...\ntest/tdcsfog/\n   ghi678.csv\n   ...\n
\n

Since you don't know the actual names of the files, you have to read them programmatically. (You could use os.glob or pathlib.Path.glob, say.)

\n

Hope this answers your question!

\n

Ryan

", "votes": null, "replies": [ { "id": 2224738, "author_name": "williammcintoshpdx", "author_url": "", "post_date": "04/17/2023 16:05:08", "content": "

OH! I think I better understand by your response. When we submit our notebook and set its corresponding output submission.csv file, the contents of the test \"folder\" are replaced during inference time. Is that correct?

", "votes": null, "replies": [ { "id": 2224828, "author_name": "ryanholbrook", "author_url": "", "post_date": "04/17/2023 17:23:45", "content": "

Yep! Some of the other files change too if they contain info about the test set. The hidden sample_submission.csv for instance has the full set of ids needed for your submission.csv. See the Data Splits section in the data description for the details.

", "votes": null, "replies": [ { "id": 2228827, "author_name": "williammcintoshpdx", "author_url": "", "post_date": "04/20/2023 21:05:50", "content": "

Thank you @ryanholbrook!

\n

Does this also mean that for the defog private test datasets that we will also have access to the Tasks table for those experiments during submission (inference) time?

", "votes": null, "replies": [ { "id": 2228873, "author_name": "ryanholbrook", "author_url": "", "post_date": "04/20/2023 22:09:50", "content": "

No, as I mentioned above The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets. from the data description implies that info from these files is only available for the training set.

", "votes": null, "replies": [ { "id": 2229756, "author_name": "williammcintoshpdx", "author_url": "", "post_date": "04/21/2023 16:47:44", "content": "

okay thank you for the further clarification!

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2225381, "author_name": "roger92", "author_url": "", "post_date": "04/18/2023 06:11:55", "content": "

Thks for asking such questions, my confusion is same as yours. this discussion really helps me a lot😄

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2182001": "Dear hosts,\n\nI very much appreciate that you provide a lot of information on the [data](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/data) page. However, I want to make sure that I understand what kind of raw variables are available when predicting the samples from the test set for the public/private LB. \n\n1. An \"Id\" occurs only in exactly one of those sets: train, public test set, private test set? (Asking this because 003f117e14 occurs in the train AND test set in the interactive notebook version)\n2. For every \"Id\", irrespective in which of the 3 set it occurs (train, public test set, private test set), I can merge subjects.csv to it and tdcsfog_metadata.csv or defog_metadata.csv subject to the data set the \"Id\" is coming from. Correct?\n3. Can you please elaborate on \"The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets.\"? The words public/hidden confuses me, because we have a public version of these csv-files when opening a kaggle notebook and loading the data set. Then there might by a different public version for scoring the public LB and a third hidden/private version for scoring the private LB.\n\nThank you very much!", "2183008": "Hi @dv0048 ,\n\nThere are two versions of the dataset:\n- The *public* version, which you see on the Data page and is available to you in interactive notebook sessions. The public version only contains training data. The files in `test` are only examples in the right format to help you write your code; you should ignore the ids and the actual contents.\n- The *private* or *hidden* version, which you cannot see but is available to your notebook when you submit. This version contains both the training data and the actual test data. To get the ids of the test series, you can either read them out of the `test` folder or out of the `sample_submission.csv` file.\n\nThe hidden (or private) dataset has the entire test data. The words \"public\" and \"private\" here have nothing to do with the public and private leaderboards. You don't know which test id goes with the public or private leaderboard.\n\n1. In the hidden dataset, every id occurs only once, either in `train` or in `test` (or in `unlabeled`).\n2. Correct. This metadata is available for the test set series.\n3. This is just saying that these files don't contain any information about test set series, only the training set. The training set is the same in the public and hidden versions, so these files are also the same in both versions.\n\nHope this answers your questions!", "2183361": "Yes. Thank you very much for the quick reply!", "2223037": "Hello @ryanholbrook \n\nWhen you say:\n\n>\"The private or hidden version, which you cannot see but is available to your notebook when you submit. This version contains both the training data and the actual test data. To get the ids of the test series, you can either read them out of the test folder or out of the sample_submission.csv file.\"\n\nI only see 2 unique trial \"Id\" values, 02ab235146 or 003f117e14 from the `test` folder and the `sample_submission.csv` file, but we are told in the **Data Splits** section of the \"Data\" tab that:\n\n>\"The test set contains about 250 data series. The series from the tdcsfog and defog are in a proportion similar to that of the training set. These series have subjects that are entirely distinct from those in the training set.\"\n\nSo then how can we know what the Id's are for the 250 trials of the private testing set? How then will we accurately name the \"Id\" column in our outputted `submission.csv` file if we do not know which Id is related to the *private test* data that we predicting for? Can we obtain the metadata for these trial Id's or subject Id's? What are we actually predicting? (When I say \"trial\" or \"trial id\" I mean the event of time in which the subject is walking, turning, going through door ways, etc).\n\nI have successfully submitted two notebooks (and received poor scores) which simply have zeros for the \"StartHesitation\", \"Turn\", and \"Walking\" columns (essentially predicting no FOG events throughout the trials). I noticed that the submission score is only based on the `submission.csv` output that my notebook saves as a .csv with the shape of (286370, 4). I have no statistical model in my notebook submissions. How then is my `submission.csv` file used to determine the precision for all 250 hidden trials?\n\nI also noticed that the \"Data Split\" section states the following:\n\n>\"When your submission is scored, this example test data will be replaced with the full test set.\"\n\nWhat does this mean? What is being replaced? How are the 250 trials of the private test data being predicted on if we do not even have the accelerometer data for those 250 trials? \n\nI feel as though I am missing something very clear here, even after reading the documents thoroughly. I sincerely appreciate your time and effort in clarifying my questions.", "2224497": "Hi @williammcintoshpdx ,\n\nThe files you see in the `test` set aren't the actual test set. They are just examples in the right format to help you write code. When you submit your notebook, we rerun it against a hidden test set, which you cannot see. This hidden test set has a `test` folder with all ~250 files like:\n\n```\ntest/defog/\n abc012.csv\n def345.csv\n ...\ntest/tdcsfog/\n ghi678.csv\n ...\n```\n\nSince you don't know the actual names of the files, you have to read them programmatically. (You could use `os.glob` or `pathlib.Path.glob`, say.)\n\nHope this answers your question!\n\nRyan", "2224738": "OH! I think I better understand by your response. When we submit our notebook and set its corresponding output submission.csv file, the contents of the test \"folder\" are replaced during inference time. Is that correct?", "2224828": "Yep! Some of the other files change too if they contain info about the test set. The hidden `sample_submission.csv` for instance has the full set of ids needed for your `submission.csv`. See the *Data Splits* section in the data description for the details.", "2225381": "Thks for asking such questions, my confusion is same as yours. this discussion really helps me a lot😄", "2228827": "Thank you @ryanholbrook!\n\nDoes this also mean that for the defog private test datasets that we will also have access to the `Tasks` table for those experiments during submission (inference) time?", "2228873": "No, as I mentioned above `The events.csv, defog_tasks.csv, and daily_metadata.csv files are the same in both the public and hidden datasets.` from the data description implies that info from these files is only available for the training set.", "2229756": "okay thank you for the further clarification!" }, "source": "meta" }