Hello kaggle,\nI'm a beginner at Deep Learning, Kaggle, Tensorflow and I'm trying out this competition for experience and hopefully learning something new. I've read a few notebooks on this competition and I could understand the ones where they used .jpg files for input, but I really want to use the TFRecord files, and I came across another question regarding that, and it solved most of my initial doubt but I am still confused about how are we supposed to split the train/test data and also make splits for cross validation? and what do the numbers on the name of the files signify?
", "messages": [ { "id": "871801", "postDate": "06/02/2020 16:34:04", "content": "Hello kaggle,\nI'm a beginner at Deep Learning, Kaggle, Tensorflow and I'm trying out this competition for experience and hopefully learning something new. I've read a few notebooks on this competition and I could understand the ones where they used .jpg files for input, but I really want to use the TFRecord files, and I came across another question regarding that, and it solved most of my initial doubt but I am still confused about how are we supposed to split the train/test data and also make splits for cross validation? and what do the numbers on the name of the files signify?
", "rawMarkdown": "Hello kaggle,\nI'm a beginner at Deep Learning, Kaggle, Tensorflow and I'm trying out this competition for experience and hopefully learning something new. I've read a few notebooks on this competition and I could understand the ones where they used .jpg files for input, but I really want to use the TFRecord files, and I came across another question regarding that, and it solved most of my initial doubt but I am still confused about **how are we supposed to split the train/test data and also make splits for cross validation? and what do the numbers on the name of the files signify?**", "votes": null }, { "id": "871834", "postDate": "06/02/2020 17:09:02", "content": "You split train and test by filename. For example TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec') and TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test*.tfrec'). You can also make validation sets by choosing specific numbers between 0 and 15. For example
VAL = [0,5,8,12]\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train%.2i-2071.tfrec'%VAL[0])\nfor k in VAL[1:]:\n VALIDATION_FILENAMES += tf.io.gfile.glob(GCS_PATH + '/train%.2i-2071.tfrec'%k)\nIn the filename train00-2071.tfrec, the 00 says this is 0 out of 16th file. And 2071 says there are 2071 records (images i.e. samples) within.
To learn more, review this Flower Comp notebook here
", "rawMarkdown": "You split train and test by filename. For example `TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')` and `TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')`. You can also make validation sets by choosing specific numbers between 0 and 15. For example \n\n VAL = [0,5,8,12]\n VALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train%.2i-2071.tfrec'%VAL[0])\n for k in VAL[1:]:\n VALIDATION_FILENAMES += tf.io.gfile.glob(GCS_PATH + '/train%.2i-2071.tfrec'%k)\n\nIn the filename `train00-2071.tfrec`, the `00` says this is 0 out of 16th file. And 2071 says there are 2071 records (images i.e. samples) within.\n\nTo learn more, review this Flower Comp notebook [here][1]\n\n[1]: https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu", "votes": null }, { "id": "872352", "postDate": "06/03/2020 05:43:04", "content": "Thank you so much for the detailed explanation! I think I finally understood what I have to do!
", "rawMarkdown": "Thank you so much for the detailed explanation! I think I finally understood what I have to do!", "votes": null }, { "id": "872377", "postDate": "06/03/2020 06:17:37", "content": "Also, I was wondering that does the given TFRecords in the competition only have two features? Which in case of training, would be image and target ? Because since the distribution of targets in heavily skewed, a Stratified KFold CV is recommended by Abhishek Thakur, in this Youtube video. So would that be possible with TFRecords ? I also saw that you uploaded multiple datasets with different sizes, and additional features, so can you help me on how I would conditionally slice the dataset or records, if that's possible?
You split train and test by filename. For example TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec') and TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test*.tfrec'). You can also make validation sets by choosing specific numbers between 0 and 15. For example
VAL = [0,5,8,12]\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train%.2i-2071.tfrec'%VAL[0])\nfor k in VAL[1:]:\n VALIDATION_FILENAMES += tf.io.gfile.glob(GCS_PATH + '/train%.2i-2071.tfrec'%k)\nIn the filename train00-2071.tfrec, the 00 says this is 0 out of 16th file. And 2071 says there are 2071 records (images i.e. samples) within.
To learn more, review this Flower Comp notebook here
", "votes": null, "replies": [ { "id": 872352, "author_name": "sayantankarmakar", "author_url": "", "post_date": "06/03/2020 05:43:04", "content": "Thank you so much for the detailed explanation! I think I finally understood what I have to do!
", "votes": null, "replies": [] }, { "id": 872377, "author_name": "sayantankarmakar", "author_url": "", "post_date": "06/03/2020 06:17:37", "content": "Also, I was wondering that does the given TFRecords in the competition only have two features? Which in case of training, would be image and target ? Because since the distribution of targets in heavily skewed, a Stratified KFold CV is recommended by Abhishek Thakur, in this Youtube video. So would that be possible with TFRecords ? I also saw that you uploaded multiple datasets with different sizes, and additional features, so can you help me on how I would conditionally slice the dataset or records, if that's possible?