Here are my tf records for different image size and different object detection models.
\nAll bounding boxed were extracted from public discussions and notebook so it is not something new but you may find all one place.
\nDetic:
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-512
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-768
Yolov5:
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-512
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-768
Yolov5 (Backfin)
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-512
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-768
Yolov5 (Full Body)
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-fullbody-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-fullbody-768
Cheers and have fun kaggling.
", "messages": [ { "id": 1737759, "postDate": "2022-03-28T17:28:47.463Z", "content": "Here are my tf records for different image size and different object detection models.
\nAll bounding boxed were extracted from public discussions and notebook so it is not something new but you may find all one place.
\nDetic:
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-512
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-768
Yolov5:
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-512
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-768
Yolov5 (Backfin)
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-512
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-768
Yolov5 (Full Body)
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-fullbody-256
\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-fullbody-768
Cheers and have fun kaggling.
", "rawMarkdown": "Here are my tf records for different image size and different object detection models.\n\nAll bounding boxed were extracted from public discussions and notebook so it is not something new but you may find all one place.\n\nDetic:\n\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-256\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-512\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-detic-box-768\n\nYolov5:\n\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-256\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-512\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-yolov5-box-768\n\nYolov5 (Backfin)\n\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-256\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-512\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-backfin-768\n\nYolov5 (Full Body)\n\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-fullbody-256\nhttps://www.kaggle.com/datasets/ragnar123/happywhale-tfrecords-fullbody-768\n\nCheers and have fun kaggling.", "votes": 74 }, { "id": 1739474, "postDate": "2022-03-30T04:43:16.027Z", "content": "how to get tfrecord format
\nfilenames = ['gs://kds-af44e5315e385a2ad58825ddb8a1cdad1f3ba94ccf3783349018dff8/train-0-1702.tfrec']
\nraw_dataset = tf.data.TFRecordDataset(filenames)
\nfor raw_record in raw_dataset.take(1):
\n e42xample = tf.train.Example()
\n example.ParseFromString(raw_record.numpy())
\n print(example)
thank for reply. lack of experiences in tfrecord processing.
", "rawMarkdown": "thank for reply. lack of experiences in tfrecord processing.", "votes": 1 } ] }, { "id": 1738099, "postDate": "2022-03-29T03:10:43.067Z", "content": "thanks for sharing. It could save lot of time for others.
\nIf with original notebook info would be better.
Hi! I wrote separate functions for reading testset and trainset. I started training, the model learned, everything is fine, but on Lb I showed an accuracy of about 0.1. The mistake is obvious with reading the testset please tell me what I did wrong.
\ndef read_labeled_tfrecord(example):
\n LABELED_TFREC_FORMAT = {
\n \"image_id\": tf.io.FixedLenFeature([], tf.string),
\n \"image\": tf.io.FixedLenFeature([], tf.string),
\n \"individual_id\": tf.io.FixedLenFeature([], tf.int64),
\n }
example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\nposting_id = example['image_id']\nimage = decode_image(example['image'])\n\nlabel_group = tf.cast(example['individual_id'], tf.int32)\nmatches = 1\nreturn posting_id, image, label_group, matches\ndef read_labeled_tfrecord_test(example):
\n LABELED_TFREC_FORMAT = {
\n \"image_id\": tf.io.FixedLenFeature([], tf.string),
\n \"image\": tf.io.FixedLenFeature([], tf.string),
\n #\"individual_id\": tf.io.FixedLenFeature([], tf.int64),
\n }
example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\nposting_id = example['image_id']\nimage = decode_image(example['image'])\nlabel_group = 0\n#label_group = tf.cast(example['individual_id'], tf.int32)\nmatches = 1\nreturn posting_id, image, label_group, matches\nHey Andrij. Did you map the individual_ids according to the train_encoded.csv?
", "rawMarkdown": "Hey Andrij. Did you map the individual_ids according to the train_encoded.csv?", "votes": 1 }, { "id": 1752067, "postDate": "2022-04-11T11:21:21.047Z", "content": "Thank you!!! I realized
", "rawMarkdown": "Thank you!!! I realized" } ] }, { "id": 1741431, "postDate": "2022-03-31T19:53:08.680Z", "content": "would you mind sharing which of these four datasets works best in your case?
", "rawMarkdown": "would you mind sharing which of these four datasets works best in your case?", "votes": 1, "replies": [ { "id": 1742291, "postDate": "2022-04-01T16:52:14.617Z", "content": "In my case full body dataset works best
", "rawMarkdown": "In my case full body dataset works best", "votes": 2 }, { "id": 1742329, "postDate": "2022-04-01T17:55:25.870Z", "content": "Thank you !
", "rawMarkdown": "Thank you !" } ] }, { "id": 1738954, "postDate": "2022-03-29T16:34:23.993Z", "content": "I try to use the full body dataset. However Its format is not same. Can you please give the generating notebook, or the tfrecord format?
", "rawMarkdown": "I try to use the full body dataset. However Its format is not same. Can you please give the generating notebook, or the tfrecord format?", "votes": 2, "replies": [ { "id": 1740307, "postDate": "2022-03-30T19:39:45.730Z", "content": "The format is the same, it has the same keys
\nTrain: [\"image_id\", \"image\", \"species\",\"individual_id\"]
\nTest: [\"image_id\", \"image\"]
\nFor mapping the classes you can use the csv file store in the same dataset named train_encoded.csv
Excellent. I've been stuck at LB = 0.779 for single fold. Hopefully I can break through now before I sink into oblivion.
", "rawMarkdown": "Excellent. I've been stuck at LB = 0.779 for single fold. Hopefully I can break through now before I sink into oblivion.", "votes": 2 }, { "id": 1742996, "postDate": "2022-04-02T13:26:43.113Z", "content": "If possible, could you please publish a simple notebook with the above tfrecord.
", "rawMarkdown": "If possible, could you please publish a simple notebook with the above tfrecord.", "votes": 1 }, { "id": 1749184, "postDate": "2022-04-08T10:07:05.597Z", "content": "Can you please tell me how much folds does the tfrecords have???
", "rawMarkdown": "Can you please tell me how much folds does the tfrecords have???" }, { "id": 1744361, "postDate": "2022-04-03T23:04:21.373Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1738219, "postDate": "2022-03-29T05:50:05.053Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1737959, "postDate": "2022-03-28T22:37:55.310Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1743867, "postDate": "2022-04-03T11:26:23.743Z", "content": "Thanks for sharing!
\nGreat job!
how to get tfrecord format
\nfilenames = ['gs://kds-af44e5315e385a2ad58825ddb8a1cdad1f3ba94ccf3783349018dff8/train-0-1702.tfrec']
\nraw_dataset = tf.data.TFRecordDataset(filenames)
\nfor raw_record in raw_dataset.take(1):
\n e42xample = tf.train.Example()
\n example.ParseFromString(raw_record.numpy())
\n print(example)
thank for reply. lack of experiences in tfrecord processing.
", "votes": 1, "replies": [] } ] }, { "id": 1738099, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-03-29T03:10:43.067000", "content": "thanks for sharing. It could save lot of time for others.
\nIf with original notebook info would be better.
Hi! I wrote separate functions for reading testset and trainset. I started training, the model learned, everything is fine, but on Lb I showed an accuracy of about 0.1. The mistake is obvious with reading the testset please tell me what I did wrong.
\ndef read_labeled_tfrecord(example):
\n LABELED_TFREC_FORMAT = {
\n \"image_id\": tf.io.FixedLenFeature([], tf.string),
\n \"image\": tf.io.FixedLenFeature([], tf.string),
\n \"individual_id\": tf.io.FixedLenFeature([], tf.int64),
\n }
example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\nposting_id = example['image_id']\nimage = decode_image(example['image'])\n\nlabel_group = tf.cast(example['individual_id'], tf.int32)\nmatches = 1\nreturn posting_id, image, label_group, matches\ndef read_labeled_tfrecord_test(example):
\n LABELED_TFREC_FORMAT = {
\n \"image_id\": tf.io.FixedLenFeature([], tf.string),
\n \"image\": tf.io.FixedLenFeature([], tf.string),
\n #\"individual_id\": tf.io.FixedLenFeature([], tf.int64),
\n }
example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\nposting_id = example['image_id']\nimage = decode_image(example['image'])\nlabel_group = 0\n#label_group = tf.cast(example['individual_id'], tf.int32)\nmatches = 1\nreturn posting_id, image, label_group, matches\nHey Andrij. Did you map the individual_ids according to the train_encoded.csv?
", "votes": 1, "replies": [] }, { "id": 1752067, "author_name": "Andrij", "author_url": "", "post_date": "2022-04-11T11:21:21.047000", "content": "Thank you!!! I realized
", "votes": 0, "replies": [] } ] }, { "id": 1741431, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-03-31T19:53:08.680000", "content": "would you mind sharing which of these four datasets works best in your case?
", "votes": 1, "replies": [ { "id": 1742291, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2022-04-01T16:52:14.617000", "content": "In my case full body dataset works best
", "votes": 2, "replies": [] }, { "id": 1742329, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2022-04-01T17:55:25.870000", "content": "Thank you !
", "votes": 0, "replies": [] } ] }, { "id": 1738954, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-03-29T16:34:23.993000", "content": "I try to use the full body dataset. However Its format is not same. Can you please give the generating notebook, or the tfrecord format?
", "votes": 2, "replies": [ { "id": 1740307, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2022-03-30T19:39:45.730000", "content": "The format is the same, it has the same keys
\nTrain: [\"image_id\", \"image\", \"species\",\"individual_id\"]
\nTest: [\"image_id\", \"image\"]
\nFor mapping the classes you can use the csv file store in the same dataset named train_encoded.csv
Excellent. I've been stuck at LB = 0.779 for single fold. Hopefully I can break through now before I sink into oblivion.
", "votes": 2, "replies": [] }, { "id": 1742996, "author_name": "tatsun", "author_url": "", "post_date": "2022-04-02T13:26:43.113000", "content": "If possible, could you please publish a simple notebook with the above tfrecord.
", "votes": 1, "replies": [] }, { "id": 1749184, "author_name": "Ifti", "author_url": "", "post_date": "2022-04-08T10:07:05.597000", "content": "Can you please tell me how much folds does the tfrecords have???
", "votes": 0, "replies": [] }, { "id": 1744361, "author_name": "", "author_url": "", "post_date": "2022-04-03T23:04:21.373000", "content": "", "votes": 0, "replies": [] }, { "id": 1738219, "author_name": "", "author_url": "", "post_date": "2022-03-29T05:50:05.053000", "content": "", "votes": 1, "replies": [] }, { "id": 1737959, "author_name": "", "author_url": "", "post_date": "2022-03-28T22:37:55.310000", "content": "", "votes": 0, "replies": [] }, { "id": 1743867, "author_name": "Atom_Hu", "author_url": "", "post_date": "2022-04-03T11:26:23.743000", "content": "Thanks for sharing!
\nGreat job!