{
  "id": 401582,
  "title": "Need Help -  Submission Scoring Error??",
  "url": "/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/401582",
  "author_name": "",
  "post_date": "2023-04-14T00:20:47.035540600Z",
  "votes": 1,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I submit notebook but got submission scoring error but do not know which part is wrong?? (wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected)</p>\n<p>I made a test submission with fake data and I see the output (attached parkinson.jpg) still have submission scoring error!</p>\n<p>tdcsfog_test_names: ['003f117e14.csv']</p>\n<p>defog_test_names: ['02ab235146.csv']</p>\n<p>output_combined:                        Id  StartHesitation  Turn  Walking<br>\n0            003f117e14_0              0.2   0.3     0.08<br>\n1            003f117e14_1              0.2   0.3     0.08<br>\n2            003f117e14_2              0.7   0.4     0.10<br>\n3            003f117e14_3              0.7   0.4     0.10<br>\n4            003f117e14_4              0.7   0.4     0.10<br>\n…                   …              …   …      …<br>\n281683  02ab235146_281683              0.2   0.4     0.07<br>\n281684  02ab235146_281684              0.2   0.4     0.07<br>\n281685  02ab235146_281685              0.2   0.4     0.07<br>\n281686  02ab235146_281686              0.2   0.4     0.07<br>\n281687  02ab235146_281687              0.2   0.4     0.07</p>\n<p>[286370 rows x 4 columns]</p>\n<p>I do not know what is wrong with this….. Any help appreciated!</p>",
  "messages": [
    {
      "id": "2221064",
      "postDate": "04/14/2023 00:20:47",
      "content": "<p>I submit notebook but got submission scoring error but do not know which part is wrong?? (wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected)</p>\n<p>I made a test submission with fake data and I see the output (attached parkinson.jpg) still have submission scoring error!</p>\n<p>tdcsfog_test_names: ['003f117e14.csv']</p>\n<p>defog_test_names: ['02ab235146.csv']</p>\n<p>output_combined:                        Id  StartHesitation  Turn  Walking<br>\n0            003f117e14_0              0.2   0.3     0.08<br>\n1            003f117e14_1              0.2   0.3     0.08<br>\n2            003f117e14_2              0.7   0.4     0.10<br>\n3            003f117e14_3              0.7   0.4     0.10<br>\n4            003f117e14_4              0.7   0.4     0.10<br>\n…                   …              …   …      …<br>\n281683  02ab235146_281683              0.2   0.4     0.07<br>\n281684  02ab235146_281684              0.2   0.4     0.07<br>\n281685  02ab235146_281685              0.2   0.4     0.07<br>\n281686  02ab235146_281686              0.2   0.4     0.07<br>\n281687  02ab235146_281687              0.2   0.4     0.07</p>\n<p>[286370 rows x 4 columns]</p>\n<p>I do not know what is wrong with this….. Any help appreciated!</p>",
      "rawMarkdown": "I submit notebook but got submission scoring error but do not know which part is wrong?? (wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected)\n\n\nI made a test submission with fake data and I see the output (attached parkinson.jpg) still have submission scoring error!\n\ntdcsfog_test_names: ['003f117e14.csv']\n\ndefog_test_names: ['02ab235146.csv']\n\n\n\noutput_combined:                        Id  StartHesitation  Turn  Walking\n0            003f117e14_0              0.2   0.3     0.08\n1            003f117e14_1              0.2   0.3     0.08\n2            003f117e14_2              0.7   0.4     0.10\n3            003f117e14_3              0.7   0.4     0.10\n4            003f117e14_4              0.7   0.4     0.10\n...                   ...              ...   ...      ...\n281683  02ab235146_281683              0.2   0.4     0.07\n281684  02ab235146_281684              0.2   0.4     0.07\n281685  02ab235146_281685              0.2   0.4     0.07\n281686  02ab235146_281686              0.2   0.4     0.07\n281687  02ab235146_281687              0.2   0.4     0.07\n\n[286370 rows x 4 columns]\n\n\n\n\n\n\n\nI do not know what is wrong with this..... Any help appreciated!",
      "votes": null
    },
    {
      "id": "2223470",
      "postDate": "04/16/2023 09:44:11",
      "content": "<p>Hi, I had same error, but it seems that there are other csv files in the test folders and we should consider them, These two csv files are just sample. you can write a script that reads all csv files in the two test folders and build a test data set.<br>\nI hope this can help you.</p>",
      "rawMarkdown": "Hi, I had same error, but it seems that there are other csv files in the test folders and we should consider them, These two csv files are just sample. you can write a script that reads all csv files in the two test folders and build a test data set.\nI hope this can help you.",
      "votes": null
    },
    {
      "id": "2223743",
      "postDate": "04/16/2023 15:17:39",
      "content": "<p>Hello Iraj, thanks for your reply. I used the below to get test files, are they correct? I can only 1 file under these two directories in the notebook.</p>\n<p>defog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog/'<br>\ntdcsfog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/tdcsfog/'</p>\n<p>tdcsfog_test_names = os.listdir(tdcsfog_test_root)<br>\nprint(\"tdcsfog_test_names:\",tdcsfog_test_names)<br>\ntdcsfog_test_names: ['003f117e14.csv']</p>\n<p>defog_test_names = os.listdir(defog_test_root)<br>\nprint(\"defog_test_names:\",defog_test_names)<br>\ndefog_test_names: ['02ab235146.csv']</p>",
      "rawMarkdown": "Hello Iraj, thanks for your reply. I used the below to get test files, are they correct? I can only 1 file under these two directories in the notebook.\n\n\ndefog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog/'\ntdcsfog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/tdcsfog/'\n\ntdcsfog_test_names = os.listdir(tdcsfog_test_root)\nprint(\"tdcsfog_test_names:\",tdcsfog_test_names)\ntdcsfog_test_names: ['003f117e14.csv']\n\ndefog_test_names = os.listdir(defog_test_root)\nprint(\"defog_test_names:\",defog_test_names)\ndefog_test_names: ['02ab235146.csv']",
      "votes": null
    },
    {
      "id": "2223785",
      "postDate": "04/16/2023 16:02:39",
      "content": "<p>yes, this is ok for reading some files when we are coding, but you can assume than when your code being evaluated, some files in the test folders will be added-while in normal we can not see them- and your code will be executed.<br>\nfor example this is my code, i know that my code is not perfect but it is a solution</p>\n<p>Set the directory path<br>\ndirectory_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog'</p>\n<p>df_list = []</p>\n<p>for filename in os.listdir(directory_path):<br>\n    if filename.endswith('.csv'):</p>\n<pre><code>    df = pd.read_csv(os.path.join(directory_path, filename))\n\n\n    file_id = filename.replace('.csv', '') # remove .csv from the filename\n    df['Id'] = df.apply(lambda row: f\"{file_id}_{int(row['Time'])}\", axis=1)\n\n\n    df_list.append(df)\n</code></pre>\n<p>test_defog_df = pd.concat(df_list, ignore_index=True)</p>\n<p>column_names = test_defog_df.columns.tolist()<br>\ncolumn_names = [column_names[-1]] + column_names[:-1] # move last column to first position<br>\ntest_defog_df = test_defog_df.reindex(columns=column_names)</p>\n<p>when my code is being evaluated, all of csv files including hidden files will be consider.</p>",
      "rawMarkdown": "yes, this is ok for reading some files when we are coding, but you can assume than when your code being evaluated, some files in the test folders will be added-while in normal we can not see them- and your code will be executed.\nfor example this is my code, i know that my code is not perfect but it is a solution\n\n\n Set the directory path\ndirectory_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog'\n\ndf_list = []\n\n\nfor filename in os.listdir(directory_path):\n    if filename.endswith('.csv'):\n       \n        df = pd.read_csv(os.path.join(directory_path, filename))\n        \n        \n        file_id = filename.replace('.csv', '') # remove .csv from the filename\n        df['Id'] = df.apply(lambda row: f\"{file_id}_{int(row['Time'])}\", axis=1)\n        \n      \n        df_list.append(df)\n\n\ntest_defog_df = pd.concat(df_list, ignore_index=True)\n\n\ncolumn_names = test_defog_df.columns.tolist()\ncolumn_names = [column_names[-1]] + column_names[:-1] # move last column to first position\ntest_defog_df = test_defog_df.reindex(columns=column_names)\n\n\nwhen my code is being evaluated, all of csv files including hidden files will be consider.",
      "votes": null
    },
    {
      "id": "2224185",
      "postDate": "04/17/2023 04:22:46",
      "content": "<p>Thanks, I use a similar method to list all relevant csv files in the test folder, but still got the submission scoring error. I add a print statement, and submit the notebook, I see only 1 csv file name printed out for defog and tdcsfog separately. I am wondering when you submit notebook, are you able to see the full test csv files?    </p>",
      "rawMarkdown": "Thanks, I use a similar method to list all relevant csv files in the test folder, but still got the submission scoring error. I add a print statement, and submit the notebook, I see only 1 csv file name printed out for defog and tdcsfog separately. I am wondering when you submit notebook, are you able to see the full test csv files?",
      "votes": null
    },
    {
      "id": "2224298",
      "postDate": "04/17/2023 07:29:12",
      "content": "<p>which error do you see? memory error or notebook threw exception?</p>",
      "rawMarkdown": "which error do you see? memory error or notebook threw exception?",
      "votes": null
    },
    {
      "id": "2225169",
      "postDate": "04/18/2023 00:45:46",
      "content": "<p>I still see this: \"Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. \"</p>\n<p>The output for 2 sample files look ok to me, I have no idea which format is wrong…</p>",
      "rawMarkdown": "I still see this: \"Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. \"\n\nThe output for 2 sample files look ok to me, I have no idea which format is wrong...",
      "votes": null
    },
    {
      "id": "2225174",
      "postDate": "04/18/2023 00:51:38",
      "content": "<p>Is it possible to see your submission section of the code?</p>",
      "rawMarkdown": "Is it possible to see your submission section of the code?",
      "votes": null
    },
    {
      "id": "2225257",
      "postDate": "04/18/2023 03:11:04",
      "content": "<p>Sure, I share the public notebook with you (<a href=\"https://www.kaggle.com/code/backpack30/notebookdfc5704c4f?scriptVersionId=126049995)\" target=\"_blank\">https://www.kaggle.com/code/backpack30/notebookdfc5704c4f?scriptVersionId=126049995)</a>.<br>\nMy notebook is a bit unorganized, I do tdcsfog and defog separately, generate two df for each and then combined them to generate submission.csv.</p>\n<p>tdcsfog_test_names = []<br>\nfor dirname, _, filenames in os.walk(tdcsfog_test_root):<br>\n    for filename in filenames:<br>\n        if filename.endswith('.csv'):<br>\n            tdcsfog_test_names.append(filename)</p>\n<p>print(\"tdcsfog_test_names:\",tdcsfog_test_names)</p>\n<p>tdcsfog_test_names: ['003f117e14.csv']</p>\n<p>================================================<br>\ndefog_test_names = []<br>\nfor dirname, _, filenames in os.walk(defog_test_root):<br>\n    for filename in filenames:<br>\n        if filename.endswith('.csv'):<br>\n            defog_test_names.append(filename)</p>\n<p>print(\"defog_test_names:\",defog_test_names)<br>\ndefog_test_names: ['02ab235146.csv']</p>\n<p>================================================</p>\n<h1>Combine the two dfs</h1>\n<p>output_combined = pd.concat([output,output_defog],axis=0)<br>\nprint(\"output_combined.shape[0]:\",output_combined.shape[0])<br>\noutput_combined = pd.concat([output,output_defog],axis=0)<br>\noutput_combined = output_combined.astype({'Id':'str','StartHesitation':'float64','Turn':'float64','Walking':'float64'})<br>\noutput_combined['Id'] = output_combined['Id'].astype(str)<br>\nprint(\"output_combined.dtypes:\",output_combined.dtypes)<br>\noutput_combined.to_csv('submission.csv',index=False)<br>\nprint(\"output_combined:\",output_combined)<br>\noutput_combined.shape[0]: 286370<br>\noutput_combined.dtypes: Id                  object<br>\nStartHesitation    float64<br>\nTurn               float64<br>\nWalking            float64<br>\ndtype: object<br>\noutput_combined:                        Id  StartHesitation  Turn  Walking<br>\n0            003f117e14_0              0.2   0.3     0.08<br>\n1            003f117e14_1              0.2   0.3     0.08<br>\n2            003f117e14_2              0.7   0.4     0.10<br>\n3            003f117e14_3              0.7   0.4     0.10<br>\n4            003f117e14_4              0.7   0.4     0.10<br>\n…                   …              …   …      …<br>\n281683  02ab235146_281683              0.2   0.4     0.07<br>\n281684  02ab235146_281684              0.2   0.4     0.07<br>\n281685  02ab235146_281685              0.2   0.4     0.07<br>\n281686  02ab235146_281686              0.2   0.4     0.07<br>\n281687  02ab235146_281687              0.2   0.4     0.07</p>\n<p>[286370 rows x 4 columns]</p>",
      "rawMarkdown": "Sure, I share the public notebook with you (https://www.kaggle.com/code/backpack30/notebookdfc5704c4f?scriptVersionId=126049995).\nMy notebook is a bit unorganized, I do tdcsfog and defog separately, generate two df for each and then combined them to generate submission.csv.\n\ntdcsfog_test_names = []\nfor dirname, _, filenames in os.walk(tdcsfog_test_root):\n    for filename in filenames:\n        if filename.endswith('.csv'):\n            tdcsfog_test_names.append(filename)\n\nprint(\"tdcsfog_test_names:\",tdcsfog_test_names)\n\n\ntdcsfog_test_names: ['003f117e14.csv']\n\n\n\n================================================\ndefog_test_names = []\nfor dirname, _, filenames in os.walk(defog_test_root):\n    for filename in filenames:\n        if filename.endswith('.csv'):\n            defog_test_names.append(filename)\n\nprint(\"defog_test_names:\",defog_test_names)\ndefog_test_names: ['02ab235146.csv']\n\n================================================\n\n# Combine the two dfs\noutput_combined = pd.concat([output,output_defog],axis=0)\nprint(\"output_combined.shape[0]:\",output_combined.shape[0])\noutput_combined = pd.concat([output,output_defog],axis=0)\noutput_combined = output_combined.astype({'Id':'str','StartHesitation':'float64','Turn':'float64','Walking':'float64'})\noutput_combined['Id'] = output_combined['Id'].astype(str)\nprint(\"output_combined.dtypes:\",output_combined.dtypes)\noutput_combined.to_csv('submission.csv',index=False)\nprint(\"output_combined:\",output_combined)\noutput_combined.shape[0]: 286370\noutput_combined.dtypes: Id                  object\nStartHesitation    float64\nTurn               float64\nWalking            float64\ndtype: object\noutput_combined:                        Id  StartHesitation  Turn  Walking\n0            003f117e14_0              0.2   0.3     0.08\n1            003f117e14_1              0.2   0.3     0.08\n2            003f117e14_2              0.7   0.4     0.10\n3            003f117e14_3              0.7   0.4     0.10\n4            003f117e14_4              0.7   0.4     0.10\n...                   ...              ...   ...      ...\n281683  02ab235146_281683              0.2   0.4     0.07\n281684  02ab235146_281684              0.2   0.4     0.07\n281685  02ab235146_281685              0.2   0.4     0.07\n281686  02ab235146_281686              0.2   0.4     0.07\n281687  02ab235146_281687              0.2   0.4     0.07\n\n[286370 rows x 4 columns]",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2223470,
      "author_name": "irajahangari",
      "author_url": "",
      "post_date": "04/16/2023 09:44:11",
      "content": "<p>Hi, I had same error, but it seems that there are other csv files in the test folders and we should consider them, These two csv files are just sample. you can write a script that reads all csv files in the two test folders and build a test data set.<br>\nI hope this can help you.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2223743,
          "author_name": "backpack30",
          "author_url": "",
          "post_date": "04/16/2023 15:17:39",
          "content": "<p>Hello Iraj, thanks for your reply. I used the below to get test files, are they correct? I can only 1 file under these two directories in the notebook.</p>\n<p>defog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog/'<br>\ntdcsfog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/tdcsfog/'</p>\n<p>tdcsfog_test_names = os.listdir(tdcsfog_test_root)<br>\nprint(\"tdcsfog_test_names:\",tdcsfog_test_names)<br>\ntdcsfog_test_names: ['003f117e14.csv']</p>\n<p>defog_test_names = os.listdir(defog_test_root)<br>\nprint(\"defog_test_names:\",defog_test_names)<br>\ndefog_test_names: ['02ab235146.csv']</p>",
          "votes": null,
          "replies": [
            {
              "id": 2223785,
              "author_name": "irajahangari",
              "author_url": "",
              "post_date": "04/16/2023 16:02:39",
              "content": "<p>yes, this is ok for reading some files when we are coding, but you can assume than when your code being evaluated, some files in the test folders will be added-while in normal we can not see them- and your code will be executed.<br>\nfor example this is my code, i know that my code is not perfect but it is a solution</p>\n<p>Set the directory path<br>\ndirectory_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog'</p>\n<p>df_list = []</p>\n<p>for filename in os.listdir(directory_path):<br>\n    if filename.endswith('.csv'):</p>\n<pre><code>    df = pd.read_csv(os.path.join(directory_path, filename))\n\n\n    file_id = filename.replace('.csv', '') # remove .csv from the filename\n    df['Id'] = df.apply(lambda row: f\"{file_id}_{int(row['Time'])}\", axis=1)\n\n\n    df_list.append(df)\n</code></pre>\n<p>test_defog_df = pd.concat(df_list, ignore_index=True)</p>\n<p>column_names = test_defog_df.columns.tolist()<br>\ncolumn_names = [column_names[-1]] + column_names[:-1] # move last column to first position<br>\ntest_defog_df = test_defog_df.reindex(columns=column_names)</p>\n<p>when my code is being evaluated, all of csv files including hidden files will be consider.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2224185,
                  "author_name": "backpack30",
                  "author_url": "",
                  "post_date": "04/17/2023 04:22:46",
                  "content": "<p>Thanks, I use a similar method to list all relevant csv files in the test folder, but still got the submission scoring error. I add a print statement, and submit the notebook, I see only 1 csv file name printed out for defog and tdcsfog separately. I am wondering when you submit notebook, are you able to see the full test csv files?    </p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2224298,
                      "author_name": "irajahangari",
                      "author_url": "",
                      "post_date": "04/17/2023 07:29:12",
                      "content": "<p>which error do you see? memory error or notebook threw exception?</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 2225169,
                          "author_name": "backpack30",
                          "author_url": "",
                          "post_date": "04/18/2023 00:45:46",
                          "content": "<p>I still see this: \"Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. \"</p>\n<p>The output for 2 sample files look ok to me, I have no idea which format is wrong…</p>",
                          "votes": null,
                          "replies": [
                            {
                              "id": 2225174,
                              "author_name": "irajahangari",
                              "author_url": "",
                              "post_date": "04/18/2023 00:51:38",
                              "content": "<p>Is it possible to see your submission section of the code?</p>",
                              "votes": null,
                              "replies": [
                                {
                                  "id": 2225257,
                                  "author_name": "backpack30",
                                  "author_url": "",
                                  "post_date": "04/18/2023 03:11:04",
                                  "content": "<p>Sure, I share the public notebook with you (<a href=\"https://www.kaggle.com/code/backpack30/notebookdfc5704c4f?scriptVersionId=126049995)\" target=\"_blank\">https://www.kaggle.com/code/backpack30/notebookdfc5704c4f?scriptVersionId=126049995)</a>.<br>\nMy notebook is a bit unorganized, I do tdcsfog and defog separately, generate two df for each and then combined them to generate submission.csv.</p>\n<p>tdcsfog_test_names = []<br>\nfor dirname, _, filenames in os.walk(tdcsfog_test_root):<br>\n    for filename in filenames:<br>\n        if filename.endswith('.csv'):<br>\n            tdcsfog_test_names.append(filename)</p>\n<p>print(\"tdcsfog_test_names:\",tdcsfog_test_names)</p>\n<p>tdcsfog_test_names: ['003f117e14.csv']</p>\n<p>================================================<br>\ndefog_test_names = []<br>\nfor dirname, _, filenames in os.walk(defog_test_root):<br>\n    for filename in filenames:<br>\n        if filename.endswith('.csv'):<br>\n            defog_test_names.append(filename)</p>\n<p>print(\"defog_test_names:\",defog_test_names)<br>\ndefog_test_names: ['02ab235146.csv']</p>\n<p>================================================</p>\n<h1>Combine the two dfs</h1>\n<p>output_combined = pd.concat([output,output_defog],axis=0)<br>\nprint(\"output_combined.shape[0]:\",output_combined.shape[0])<br>\noutput_combined = pd.concat([output,output_defog],axis=0)<br>\noutput_combined = output_combined.astype({'Id':'str','StartHesitation':'float64','Turn':'float64','Walking':'float64'})<br>\noutput_combined['Id'] = output_combined['Id'].astype(str)<br>\nprint(\"output_combined.dtypes:\",output_combined.dtypes)<br>\noutput_combined.to_csv('submission.csv',index=False)<br>\nprint(\"output_combined:\",output_combined)<br>\noutput_combined.shape[0]: 286370<br>\noutput_combined.dtypes: Id                  object<br>\nStartHesitation    float64<br>\nTurn               float64<br>\nWalking            float64<br>\ndtype: object<br>\noutput_combined:                        Id  StartHesitation  Turn  Walking<br>\n0            003f117e14_0              0.2   0.3     0.08<br>\n1            003f117e14_1              0.2   0.3     0.08<br>\n2            003f117e14_2              0.7   0.4     0.10<br>\n3            003f117e14_3              0.7   0.4     0.10<br>\n4            003f117e14_4              0.7   0.4     0.10<br>\n…                   …              …   …      …<br>\n281683  02ab235146_281683              0.2   0.4     0.07<br>\n281684  02ab235146_281684              0.2   0.4     0.07<br>\n281685  02ab235146_281685              0.2   0.4     0.07<br>\n281686  02ab235146_281686              0.2   0.4     0.07<br>\n281687  02ab235146_281687              0.2   0.4     0.07</p>\n<p>[286370 rows x 4 columns]</p>",
                                  "votes": null,
                                  "replies": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2221064": "I submit notebook but got submission scoring error but do not know which part is wrong?? (wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected)\n\n\nI made a test submission with fake data and I see the output (attached parkinson.jpg) still have submission scoring error!\n\ntdcsfog_test_names: ['003f117e14.csv']\n\ndefog_test_names: ['02ab235146.csv']\n\n\n\noutput_combined:                        Id  StartHesitation  Turn  Walking\n0            003f117e14_0              0.2   0.3     0.08\n1            003f117e14_1              0.2   0.3     0.08\n2            003f117e14_2              0.7   0.4     0.10\n3            003f117e14_3              0.7   0.4     0.10\n4            003f117e14_4              0.7   0.4     0.10\n...                   ...              ...   ...      ...\n281683  02ab235146_281683              0.2   0.4     0.07\n281684  02ab235146_281684              0.2   0.4     0.07\n281685  02ab235146_281685              0.2   0.4     0.07\n281686  02ab235146_281686              0.2   0.4     0.07\n281687  02ab235146_281687              0.2   0.4     0.07\n\n[286370 rows x 4 columns]\n\n\n\n\n\n\n\nI do not know what is wrong with this..... Any help appreciated!",
    "2223470": "Hi, I had same error, but it seems that there are other csv files in the test folders and we should consider them, These two csv files are just sample. you can write a script that reads all csv files in the two test folders and build a test data set.\nI hope this can help you.",
    "2223743": "Hello Iraj, thanks for your reply. I used the below to get test files, are they correct? I can only 1 file under these two directories in the notebook.\n\n\ndefog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog/'\ntdcsfog_test_root = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/tdcsfog/'\n\ntdcsfog_test_names = os.listdir(tdcsfog_test_root)\nprint(\"tdcsfog_test_names:\",tdcsfog_test_names)\ntdcsfog_test_names: ['003f117e14.csv']\n\ndefog_test_names = os.listdir(defog_test_root)\nprint(\"defog_test_names:\",defog_test_names)\ndefog_test_names: ['02ab235146.csv']",
    "2223785": "yes, this is ok for reading some files when we are coding, but you can assume than when your code being evaluated, some files in the test folders will be added-while in normal we can not see them- and your code will be executed.\nfor example this is my code, i know that my code is not perfect but it is a solution\n\n\n Set the directory path\ndirectory_path = '/kaggle/input/tlvmc-parkinsons-freezing-gait-prediction/test/defog'\n\ndf_list = []\n\n\nfor filename in os.listdir(directory_path):\n    if filename.endswith('.csv'):\n       \n        df = pd.read_csv(os.path.join(directory_path, filename))\n        \n        \n        file_id = filename.replace('.csv', '') # remove .csv from the filename\n        df['Id'] = df.apply(lambda row: f\"{file_id}_{int(row['Time'])}\", axis=1)\n        \n      \n        df_list.append(df)\n\n\ntest_defog_df = pd.concat(df_list, ignore_index=True)\n\n\ncolumn_names = test_defog_df.columns.tolist()\ncolumn_names = [column_names[-1]] + column_names[:-1] # move last column to first position\ntest_defog_df = test_defog_df.reindex(columns=column_names)\n\n\nwhen my code is being evaluated, all of csv files including hidden files will be consider.",
    "2224185": "Thanks, I use a similar method to list all relevant csv files in the test folder, but still got the submission scoring error. I add a print statement, and submit the notebook, I see only 1 csv file name printed out for defog and tdcsfog separately. I am wondering when you submit notebook, are you able to see the full test csv files?",
    "2224298": "which error do you see? memory error or notebook threw exception?",
    "2225169": "I still see this: \"Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. \"\n\nThe output for 2 sample files look ok to me, I have no idea which format is wrong...",
    "2225174": "Is it possible to see your submission section of the code?",
    "2225257": "Sure, I share the public notebook with you (https://www.kaggle.com/code/backpack30/notebookdfc5704c4f?scriptVersionId=126049995).\nMy notebook is a bit unorganized, I do tdcsfog and defog separately, generate two df for each and then combined them to generate submission.csv.\n\ntdcsfog_test_names = []\nfor dirname, _, filenames in os.walk(tdcsfog_test_root):\n    for filename in filenames:\n        if filename.endswith('.csv'):\n            tdcsfog_test_names.append(filename)\n\nprint(\"tdcsfog_test_names:\",tdcsfog_test_names)\n\n\ntdcsfog_test_names: ['003f117e14.csv']\n\n\n\n================================================\ndefog_test_names = []\nfor dirname, _, filenames in os.walk(defog_test_root):\n    for filename in filenames:\n        if filename.endswith('.csv'):\n            defog_test_names.append(filename)\n\nprint(\"defog_test_names:\",defog_test_names)\ndefog_test_names: ['02ab235146.csv']\n\n================================================\n\n# Combine the two dfs\noutput_combined = pd.concat([output,output_defog],axis=0)\nprint(\"output_combined.shape[0]:\",output_combined.shape[0])\noutput_combined = pd.concat([output,output_defog],axis=0)\noutput_combined = output_combined.astype({'Id':'str','StartHesitation':'float64','Turn':'float64','Walking':'float64'})\noutput_combined['Id'] = output_combined['Id'].astype(str)\nprint(\"output_combined.dtypes:\",output_combined.dtypes)\noutput_combined.to_csv('submission.csv',index=False)\nprint(\"output_combined:\",output_combined)\noutput_combined.shape[0]: 286370\noutput_combined.dtypes: Id                  object\nStartHesitation    float64\nTurn               float64\nWalking            float64\ndtype: object\noutput_combined:                        Id  StartHesitation  Turn  Walking\n0            003f117e14_0              0.2   0.3     0.08\n1            003f117e14_1              0.2   0.3     0.08\n2            003f117e14_2              0.7   0.4     0.10\n3            003f117e14_3              0.7   0.4     0.10\n4            003f117e14_4              0.7   0.4     0.10\n...                   ...              ...   ...      ...\n281683  02ab235146_281683              0.2   0.4     0.07\n281684  02ab235146_281684              0.2   0.4     0.07\n281685  02ab235146_281685              0.2   0.4     0.07\n281686  02ab235146_281686              0.2   0.4     0.07\n281687  02ab235146_281687              0.2   0.4     0.07\n\n[286370 rows x 4 columns]"
  },
  "source": "meta"
}