![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1916132%2F6b6535827f4d6955869f41a26e7c669f%2FScreenshot%20from%202023-05-05%2012-48-42.png?generation=1683316153709111&alt=media\")
The test data folder is not automatically populated when submitting my notebook. Here's part of my code
The Img_name list is empty during submissions. However, when I replace test with train, I can load all the data. Can someone help here?
", "messages": [ { "id": "2247197", "postDate": "05/05/2023 19:45:06", "content": "The test data folder is not automatically populated when submitting my notebook. Here's part of my code
The Img_name list is empty during submissions. However, when I replace test with train, I can load all the data. Can someone help here?
", "rawMarkdown": "The test data folder is not automatically populated when submitting my notebook. Here's part of my code\n\nThe Img_name list is empty during submissions. However, when I replace test with train, I can load all the data. Can someone help here?", "votes": null }, { "id": "2247233", "postDate": "05/05/2023 20:19:12", "content": "As you can see in the \"Submission example notebook\", you need to read the sample_submission.csv to get the path of the test scenes. Please see here: https://www.kaggle.com/code/eduardtrulls/imc-2023-submission-example?scriptVersionId=125665081&cellId=11
\n\ndata_dict = {}\n (, ) f:\n i, l (f):\n \n l i > :\n image, dataset, scene, _, _ = l.strip().split()\n dataset data_dict:\n data_dict[dataset] = {}\n scene data_dict[dataset]:\n data_dict[dataset][scene] = []\n data_dict[dataset][scene].append(image)\nI tried that as well![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1916132%2F408f6388731aaf997edbb3ef5645a7b0%2FScreenshot%20from%202023-05-05%2013-38-57.png?generation=1683319189164753&alt=media\")
My output dataset is still empty ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1916132%2F3e915b00d8b3f58e564a7a1c67a14cf5%2FScreenshot%20from%202023-05-05%2013-39-56.png?generation=1683319227462981&alt=media\")
The test data is hidden and will be populated once you hit \"submit\". The 'sample_submission.csv' file will be replaced during submission with the real test data. Your notebook can only see the test data during the submission process.
", "rawMarkdown": "The test data is hidden and will be populated once you hit \"submit\". The 'sample_submission.csv' file will be replaced during submission with the real test data. Your notebook can only see the test data during the submission process.", "votes": null }, { "id": "2247293", "postDate": "05/05/2023 21:19:05", "content": "Maybe Im missing something here. So I hit \"submit\", my notebook runs successfully. But the competition submission still fails (as my submission.csv is empty). Is there some intermediate step Im missing?
", "rawMarkdown": "Maybe Im missing something here. So I hit \"submit\", my notebook runs successfully. But the competition submission still fails (as my submission.csv is empty). Is there some intermediate step Im missing?", "votes": null }, { "id": "2247297", "postDate": "05/05/2023 21:22:48", "content": "It is hard to diagnose without seeing your code. But you should look carefully into the submission example above. Your notebook needs to first read the 'sample_submission.csv' to get the path to the test data (which you cannot see yourself). It is all in the submission example.
", "rawMarkdown": "It is hard to diagnose without seeing your code. But you should look carefully into the submission example above. Your notebook needs to first read the 'sample_submission.csv' to get the path to the test data (which you cannot see yourself). It is all in the submission example.", "votes": null }, { "id": "2247396", "postDate": "05/06/2023 00:53:07", "content": "See here:
\nhttps://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/401566#2221444
The data is not empty, it's just not populated until you disable internet and hit submit: keeping it hidden is the point, and you'll never be able to inspect it. If your notebook fails quickly (as opposed to crashing after a few hours, or running out of compute), it is doing something wrong. Please check the examples.
", "rawMarkdown": "The data is not empty, it's just not populated until you disable internet and hit submit: keeping it hidden is the point, and you'll never be able to inspect it. If your notebook fails quickly (as opposed to crashing after a few hours, or running out of compute), it is doing something wrong. Please check the examples.", "votes": null }, { "id": "2259918", "postDate": "05/15/2023 10:18:55", "content": "@thomasrochefort - I am facing a similar issue, too ! @rishab26, Did you manage to solve it ?
\nHave a look at the code snippet attached which shows that I am reading the sample submission.csv, first then fetching image_path from it. I am trying to read the image upon submission which should have been populated as per the full length sample_submission.csv expected during submission. Unfortunately, It yields the error of image being empty.
\nKindly have a look at the snippets, Been stuck here for a while !
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9201105%2F5477eb26e9f5997373b24c24d4d3cd8d%2FScreenshot%20from%202023-05-15%2015-46-24.png?generation=1684145877295846&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9201105%2Fb39ccdca9af72affc3e2011555fb19d6%2FScreenshot%20from%202023-05-15%2015-47-31.png?generation=1684145900658495&alt=media\")
As an alternative to the above,
\nI traversed the entire path inside the test directory to read the image, too ! Extracted image_path, dataset, scene from it via split functions but this also doesn't work because images don't seem to be there in
\n'/kaggle/input/image-matching-challenge-2023/test/dataset_name/scene_name/images/' directory
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9201105%2Fb4e961e6e6d4a3f7315b8d002ec023b2%2FScreenshot%20from%202023-05-15%2016-31-42.png?generation=1684148642511173&alt=media\")
Where's the catch ? What am I missing ? Please help
", "rawMarkdown": "thomasrochefort - I am facing a similar issue, too ! @rishab26, Did you manage to solve it ?\n\nHave a look at the code snippet attached which shows that I am reading the sample submission.csv, first then fetching image_path from it. I am trying to read the image upon submission which should have been populated as per the full length sample_submission.csv expected during submission. Unfortunately, It yields the error of image being empty.\n\nKindly have a look at the snippets, Been stuck here for a while ! \n\n\n\n\n\n\nAs an alternative to the above,\n\nI traversed the entire path inside the test directory to read the image, too ! Extracted image_path, dataset, scene from it via split functions but this also doesn't work because images don't seem to be there in \n\n'/kaggle/input/image-matching-challenge-2023/test/dataset_name/scene_name/images/' directory\n\n\n\n\n\nWhere's the catch ? What am I missing ? Please help", "votes": null }, { "id": "2261689", "postDate": "05/16/2023 13:59:40", "content": "@suraj520 Please see the answer from @eduardtrulls here: https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/407263#2249222
\nYou cannot inspect the test data that's the whole point. The test data will be populated once you deactivate internet access and hit submit.
", "rawMarkdown": "suraj520 Please see the answer from @eduardtrulls here: https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/407263#2249222\n\nYou cannot inspect the test data that's the whole point. The test data will be populated once you deactivate internet access and hit submit.", "votes": null }, { "id": "2261771", "postDate": "05/16/2023 14:40:41", "content": "Yeah, I tried that @thomasrochefort ! Like in the 2 code snippets that I have attached above, You can see two approaches that I have taken to fill predictions which is then being converted to submission.csv in next cell.
\nThe former extracts absolute path of images stored in '/kaggle/input/image-matching-challenge-2023/test/dataset_name/scene_name/images/' directory via sample_submission.csv's image_path, dataset and scene columns which as per README file inside the test folder claims to be populated with full path names. Unfortunately, It doesn't execute and yield Notebook Threw Exception error ! I acknowledge that the error that it yields related to reading the image is because test images are populated in the backend during submission.
\nThe latter approach traverses inside the /kaggle/input/image-matching-challenge-2023/test dataset and once it finds an image in the folder structure hierarchy then it reads it, processes it to extract the R,T matrices and use the path string to extract dataset, scene names via split operations. A kinda reverse approach. Upon submission, It also doesn't find an image in the path and quickly reverts back with Notebook Threw Exception error.
\nI specifically validated the latter approach in my local system by creating a similar folder hierarchy and placing the images inside. It works there ! The trailing print statement is from that validation process, only !
\nBoth of the approaches quoted above don't work upon switching off the internet and clicking the submission as they yield Notebook Threw Exception Error.
\nI think I am missing a piece in this ! Idk, what !? :(
", "rawMarkdown": "Yeah, I tried that @thomasrochefort ! Like in the 2 code snippets that I have attached above, You can see two approaches that I have taken to fill predictions which is then being converted to submission.csv in next cell.\n\nThe former extracts absolute path of images stored in '/kaggle/input/image-matching-challenge-2023/test/dataset_name/scene_name/images/' directory via sample_submission.csv's image_path, dataset and scene columns which as per README file inside the test folder claims to be populated with full path names. Unfortunately, It doesn't execute and yield Notebook Threw Exception error ! I acknowledge that the error that it yields related to reading the image is because test images are populated in the backend during submission.\n\nThe latter approach traverses inside the /kaggle/input/image-matching-challenge-2023/test dataset and once it finds an image in the folder structure hierarchy then it reads it, processes it to extract the R,T matrices and use the path string to extract dataset, scene names via split operations. A kinda reverse approach. Upon submission, It also doesn't find an image in the path and quickly reverts back with Notebook Threw Exception error. \n\nI specifically validated the latter approach in my local system by creating a similar folder hierarchy and placing the images inside. It works there ! The trailing print statement is from that validation process, only !\n\nBoth of the approaches quoted above don't work upon switching off the internet and clicking the submission as they yield Notebook Threw Exception Error.\n\nI think I am missing a piece in this ! Idk, what !? :(", "votes": null }, { "id": "2263502", "postDate": "05/17/2023 16:34:21", "content": "I have finally fixed it. Found the sample_submission.csv file during test time using the following code and fed the image path to trained model.
\nimport os\nimport pandas as pd\n\n# List the files in the input directory\ninput_dir = '/kaggle/input/image-matching-challenge-2023'\nfiles = []\nfor root, _, filenames in os.walk(input_dir):\n for filename in filenames:\n files.append(os.path.join(root, filename))\n\n# Search for the sample submission file\nsample_submission_path = None\nfor file in files:\n if 'sample_submission' in file:\n df = pd.read_csv(file)\n if len(df) >= 3:\n sample_submission_path = file\n break\n\nif sample_submission_path is not None:\n print(\"Path to sample submission:\", sample_submission_path)\nelse:\n print(\"Sample submission file not found.\")\nThanks !
", "rawMarkdown": "I have finally fixed it. Found the sample_submission.csv file during test time using the following code and fed the image path to trained model.\n\n```\nimport os\nimport pandas as pd\n\n# List the files in the input directory\ninput_dir = '/kaggle/input/image-matching-challenge-2023'\nfiles = []\nfor root, _, filenames in os.walk(input_dir):\n for filename in filenames:\n files.append(os.path.join(root, filename))\n\n# Search for the sample submission file\nsample_submission_path = None\nfor file in files:\n if 'sample_submission' in file:\n df = pd.read_csv(file)\n if len(df) >= 3:\n sample_submission_path = file\n break\n\nif sample_submission_path is not None:\n print(\"Path to sample submission:\", sample_submission_path)\nelse:\n print(\"Sample submission file not found.\")\n```\nThanks !", "votes": null } ], "comments": [ { "id": 2247233, "author_name": "thomasrochefort", "author_url": "", "post_date": "05/05/2023 20:19:12", "content": "As you can see in the \"Submission example notebook\", you need to read the sample_submission.csv to get the path of the test scenes. Please see here: https://www.kaggle.com/code/eduardtrulls/imc-2023-submission-example?scriptVersionId=125665081&cellId=11
\n\ndata_dict = {}\n (, ) f:\n i, l (f):\n \n l i > :\n image, dataset, scene, _, _ = l.strip().split()\n dataset data_dict:\n data_dict[dataset] = {}\n scene data_dict[dataset]:\n data_dict[dataset][scene] = []\n data_dict[dataset][scene].append(image)\nI tried that as well
My output dataset is still empty
The test data is hidden and will be populated once you hit \"submit\". The 'sample_submission.csv' file will be replaced during submission with the real test data. Your notebook can only see the test data during the submission process.
", "votes": null, "replies": [ { "id": 2247293, "author_name": "rishab26", "author_url": "", "post_date": "05/05/2023 21:19:05", "content": "Maybe Im missing something here. So I hit \"submit\", my notebook runs successfully. But the competition submission still fails (as my submission.csv is empty). Is there some intermediate step Im missing?
", "votes": null, "replies": [ { "id": 2247297, "author_name": "thomasrochefort", "author_url": "", "post_date": "05/05/2023 21:22:48", "content": "It is hard to diagnose without seeing your code. But you should look carefully into the submission example above. Your notebook needs to first read the 'sample_submission.csv' to get the path to the test data (which you cannot see yourself). It is all in the submission example.
", "votes": null, "replies": [ { "id": 2259918, "author_name": "suraj520", "author_url": "", "post_date": "05/15/2023 10:18:55", "content": "@thomasrochefort - I am facing a similar issue, too ! @rishab26, Did you manage to solve it ?
\nHave a look at the code snippet attached which shows that I am reading the sample submission.csv, first then fetching image_path from it. I am trying to read the image upon submission which should have been populated as per the full length sample_submission.csv expected during submission. Unfortunately, It yields the error of image being empty.
\nKindly have a look at the snippets, Been stuck here for a while !
\n
As an alternative to the above,
\nI traversed the entire path inside the test directory to read the image, too ! Extracted image_path, dataset, scene from it via split functions but this also doesn't work because images don't seem to be there in
\n'/kaggle/input/image-matching-challenge-2023/test/dataset_name/scene_name/images/' directory
\n
Where's the catch ? What am I missing ? Please help
", "votes": null, "replies": [ { "id": 2261689, "author_name": "thomasrochefort", "author_url": "", "post_date": "05/16/2023 13:59:40", "content": "@suraj520 Please see the answer from @eduardtrulls here: https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/407263#2249222
\nYou cannot inspect the test data that's the whole point. The test data will be populated once you deactivate internet access and hit submit.
", "votes": null, "replies": [ { "id": 2261771, "author_name": "suraj520", "author_url": "", "post_date": "05/16/2023 14:40:41", "content": "Yeah, I tried that @thomasrochefort ! Like in the 2 code snippets that I have attached above, You can see two approaches that I have taken to fill predictions which is then being converted to submission.csv in next cell.
\nThe former extracts absolute path of images stored in '/kaggle/input/image-matching-challenge-2023/test/dataset_name/scene_name/images/' directory via sample_submission.csv's image_path, dataset and scene columns which as per README file inside the test folder claims to be populated with full path names. Unfortunately, It doesn't execute and yield Notebook Threw Exception error ! I acknowledge that the error that it yields related to reading the image is because test images are populated in the backend during submission.
\nThe latter approach traverses inside the /kaggle/input/image-matching-challenge-2023/test dataset and once it finds an image in the folder structure hierarchy then it reads it, processes it to extract the R,T matrices and use the path string to extract dataset, scene names via split operations. A kinda reverse approach. Upon submission, It also doesn't find an image in the path and quickly reverts back with Notebook Threw Exception error.
\nI specifically validated the latter approach in my local system by creating a similar folder hierarchy and placing the images inside. It works there ! The trailing print statement is from that validation process, only !
\nBoth of the approaches quoted above don't work upon switching off the internet and clicking the submission as they yield Notebook Threw Exception Error.
\nI think I am missing a piece in this ! Idk, what !? :(
", "votes": null, "replies": [ { "id": 2263502, "author_name": "suraj520", "author_url": "", "post_date": "05/17/2023 16:34:21", "content": "I have finally fixed it. Found the sample_submission.csv file during test time using the following code and fed the image path to trained model.
\nimport os\nimport pandas as pd\n\n# List the files in the input directory\ninput_dir = '/kaggle/input/image-matching-challenge-2023'\nfiles = []\nfor root, _, filenames in os.walk(input_dir):\n for filename in filenames:\n files.append(os.path.join(root, filename))\n\n# Search for the sample submission file\nsample_submission_path = None\nfor file in files:\n if 'sample_submission' in file:\n df = pd.read_csv(file)\n if len(df) >= 3:\n sample_submission_path = file\n break\n\nif sample_submission_path is not None:\n print(\"Path to sample submission:\", sample_submission_path)\nelse:\n print(\"Sample submission file not found.\")\nThanks !
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2249222, "author_name": "eduardtrulls", "author_url": "", "post_date": "05/07/2023 15:02:27", "content": "The data is not empty, it's just not populated until you disable internet and hit submit: keeping it hidden is the point, and you'll never be able to inspect it. If your notebook fails quickly (as opposed to crashing after a few hours, or running out of compute), it is doing something wrong. Please check the examples.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2247396, "author_name": "thomasrochefort", "author_url": "", "post_date": "05/06/2023 00:53:07", "content": "See here:
\nhttps://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/401566#2221444