how can we join training and validation folders to one and later split according to our split ratio
", "messages": [ { "id": 1217484, "postDate": "2021-02-25T05:42:49.417Z", "content": "how can we join training and validation folders to one and later split according to our split ratio
", "rawMarkdown": "how can we join training and validation folders to one and later split according to our split ratio", "votes": 3 }, { "id": 1249240, "postDate": "2021-03-23T07:36:58.360Z", "content": "The data are in the input folder. I don't think you could move the files or combine the folders. If you really want to put them in the same folder, you could copy them into your /kaggle/working/ directory. The following is from my notebook showing an example of copying the sample_submission.csv over to your working folder. You could use this command: !gsutil cp
# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved \n# as output when you create a version using \"Save & Run All\" \n!gsutil cp /kaggle/input/tpu-getting-started/sample_submission.csv /kaggle/working/test.csv\nfor dirpath, _, filenames in os.walk('/kaggle/working'):\n for filename in filenames:\n print(os.path.join(dirpath, filename)) \nprint('list of entries contained in /kaggle/working/:',tf.io.gfile.listdir('/kaggle/working')) \nThanks a lot.😃
\nI will check them out👍.
Are you going to use the combined set to train with cross validation? I think working with the indices are the better approach. For each fold, I would split the indices of the training set into train_1 and val_1 and the indices of validation set into train_2 and val_2. Then you could combine train_1 and train_2 indices for training and val_1 and val_2. Good idea. You get more images for training this way.
\nI found these external images that we could use for this challenge:
\ntf_flower_photo_tfrec (jpeg) created from @hengck23 data. Most duplicates removed and images converted to tfrecords for convenience. Thanks to @hengck23 for creating and to Kirill Blinov for converting and sharing! They definitely help boost the public score of my notebook. Please check it out!
Hi, I just came across these codes from Tensorflow. They could make partitioning your combined data a lot easier. The codes show how to split the val data to create test data. Good luck!
\nAs the original dataset doesn't contains a test set, you will create one. To do so, determine how many batches of data are available in the validation set using tf.data.experimental.cardinality, then move 20% of them to a test set.
\nval_batches = tf.data.experimental.cardinality(validation_dataset)\ntest_dataset = validation_dataset.take(val_batches // 5)\nvalidation_dataset = validation_dataset.skip(val_batches // 5)\nThank you😊, Great idea to add https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec to the training set. It really improved the training and F1 score.
", "rawMarkdown": "Thank you😊, Great idea to add https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec to the training set. It really improved the training and F1 score.\n\n", "votes": 1 } ] }, { "id": 1249226, "postDate": "2021-03-23T07:23:50.147Z", "content": "Hi, I have never thought of doing the challenge this way. But, I have a proposal:
\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(\"tpu-getting-started\") # Google Cloud Storage\nprint(GCS_DS_PATH)\nprint('Entries in the bucket:')\n!gsutil ls $GCS_DS_PATH # list items in the bucket \nI believe you could do the following to combine the list of file paths for training and validation datasets of the current competition:
\nTRAIN_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/train/*.tfrec') \nVAL_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/val/*.tfrec') \nCOMBINED_FILENAMES = TRAIN_FILENAMES + VAL_FILENAMES\nThe indexes of the COMBINED_FILENAMES list ranges from 0 to len(COMBINED_FILENAMES) - 1. You could split them into two sets of indexes for the training and the validation sets according to your desired ratio and take it from there. Note that each file path is for the .tfrec file which may consist different number of flower images (as shown in the last three characters of the tfrec filename). All but the last training tfrec files consist of 798 images while all validation tfrec files consist of 232 images.
\nI hope this help.
", "rawMarkdown": "Hi, I have never thought of doing the challenge this way. But, I have a proposal:\n\n```\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(\"tpu-getting-started\") # Google Cloud Storage\nprint(GCS_DS_PATH)\nprint('Entries in the bucket:')\n!gsutil ls $GCS_DS_PATH # list items in the bucket \n```\nI believe you could do the following to combine the list of file paths for training and validation datasets of the current competition:\n\n```\nTRAIN_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/train/*.tfrec') \nVAL_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/val/*.tfrec') \nCOMBINED_FILENAMES = TRAIN_FILENAMES + VAL_FILENAMES\n\n```\nThe indexes of the COMBINED_FILENAMES list ranges from 0 to len(COMBINED_FILENAMES) - 1. You could split them into two sets of indexes for the training and the validation sets according to your desired ratio and take it from there. Note that each file path is for the .tfrec file which may consist different number of flower images (as shown in the last three characters of the tfrec filename). All but the last training tfrec files consist of 798 images while all validation tfrec files consist of 232 images.\n\nI hope this help. \n", "votes": 1 } ], "comments": [ { "id": 1249240, "author_name": "Sau Kha", "author_url": "", "post_date": "2021-03-23T07:36:58.360000", "content": "The data are in the input folder. I don't think you could move the files or combine the folders. If you really want to put them in the same folder, you could copy them into your /kaggle/working/ directory. The following is from my notebook showing an example of copying the sample_submission.csv over to your working folder. You could use this command: !gsutil cp
# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved \n# as output when you create a version using \"Save & Run All\" \n!gsutil cp /kaggle/input/tpu-getting-started/sample_submission.csv /kaggle/working/test.csv\nfor dirpath, _, filenames in os.walk('/kaggle/working'):\n for filename in filenames:\n print(os.path.join(dirpath, filename)) \nprint('list of entries contained in /kaggle/working/:',tf.io.gfile.listdir('/kaggle/working')) \nThanks a lot.😃
\nI will check them out👍.
Are you going to use the combined set to train with cross validation? I think working with the indices are the better approach. For each fold, I would split the indices of the training set into train_1 and val_1 and the indices of validation set into train_2 and val_2. Then you could combine train_1 and train_2 indices for training and val_1 and val_2. Good idea. You get more images for training this way.
\nI found these external images that we could use for this challenge:
\ntf_flower_photo_tfrec (jpeg) created from @hengck23 data. Most duplicates removed and images converted to tfrecords for convenience. Thanks to @hengck23 for creating and to Kirill Blinov for converting and sharing! They definitely help boost the public score of my notebook. Please check it out!
Hi, I just came across these codes from Tensorflow. They could make partitioning your combined data a lot easier. The codes show how to split the val data to create test data. Good luck!
\nAs the original dataset doesn't contains a test set, you will create one. To do so, determine how many batches of data are available in the validation set using tf.data.experimental.cardinality, then move 20% of them to a test set.
\nval_batches = tf.data.experimental.cardinality(validation_dataset)\ntest_dataset = validation_dataset.take(val_batches // 5)\nvalidation_dataset = validation_dataset.skip(val_batches // 5)\nThank you😊, Great idea to add https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec to the training set. It really improved the training and F1 score.
", "votes": 1, "replies": [] } ] }, { "id": 1249226, "author_name": "Sau Kha", "author_url": "", "post_date": "2021-03-23T07:23:50.147000", "content": "Hi, I have never thought of doing the challenge this way. But, I have a proposal:
\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(\"tpu-getting-started\") # Google Cloud Storage\nprint(GCS_DS_PATH)\nprint('Entries in the bucket:')\n!gsutil ls $GCS_DS_PATH # list items in the bucket \nI believe you could do the following to combine the list of file paths for training and validation datasets of the current competition:
\nTRAIN_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/train/*.tfrec') \nVAL_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/val/*.tfrec') \nCOMBINED_FILENAMES = TRAIN_FILENAMES + VAL_FILENAMES\nThe indexes of the COMBINED_FILENAMES list ranges from 0 to len(COMBINED_FILENAMES) - 1. You could split them into two sets of indexes for the training and the validation sets according to your desired ratio and take it from there. Note that each file path is for the .tfrec file which may consist different number of flower images (as shown in the last three characters of the tfrec filename). All but the last training tfrec files consist of 798 images while all validation tfrec files consist of 232 images.
\nI hope this help.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1217484": "how can we join training and validation folders to one and later split according to our split ratio", "1249240": "The data are in the input folder. I don't think you could move the files or combine the folders. If you really want to put them in the same folder, you could copy them into your /kaggle/working/ directory. The following is from [my notebook](https://www.kaggle.com/saukha/petals-to-the-metals-flower-classification/edit/run/57501789) showing an example of copying the sample_submission.csv over to your working folder. You could use this command: `!gsutil cp`\n\n```\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved \n# as output when you create a version using \"Save & Run All\" \n!gsutil cp /kaggle/input/tpu-getting-started/sample_submission.csv /kaggle/working/test.csv\nfor dirpath, _, filenames in os.walk('/kaggle/working'):\n for filename in filenames:\n print(os.path.join(dirpath, filename)) \nprint('list of entries contained in /kaggle/working/:',tf.io.gfile.listdir('/kaggle/working')) \n```", "1249226": "Hi, I have never thought of doing the challenge this way. But, I have a proposal:\n\n```\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(\"tpu-getting-started\") # Google Cloud Storage\nprint(GCS_DS_PATH)\nprint('Entries in the bucket:')\n!gsutil ls $GCS_DS_PATH # list items in the bucket \n```\nI believe you could do the following to combine the list of file paths for training and validation datasets of the current competition:\n\n```\nTRAIN_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/train/*.tfrec') \nVAL_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-512x512/val/*.tfrec') \nCOMBINED_FILENAMES = TRAIN_FILENAMES + VAL_FILENAMES\n\n```\nThe indexes of the COMBINED_FILENAMES list ranges from 0 to len(COMBINED_FILENAMES) - 1. You could split them into two sets of indexes for the training and the validation sets according to your desired ratio and take it from there. Note that each file path is for the .tfrec file which may consist different number of flower images (as shown in the last three characters of the tfrec filename). All but the last training tfrec files consist of 798 images while all validation tfrec files consist of 232 images.\n\nI hope this help. \n" } }