![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2048731%2F509abbb69edb130c783a87f0449f1142%2Flyft-level5-site.png?generation=1599296306729582&alt=media\")
Hi, I came across this page on the level5 site.
\n
Is there even more training data available on level5 site than what is available through kaggle? The training set part2 download is 70 GB. Are we permitted to use this too?
", "messages": [ { "id": 999000, "postDate": "2020-09-05T09:00:44.633Z", "content": "Hi, I came across this page on the level5 site.
\n
Is there even more training data available on level5 site than what is available through kaggle? The training set part2 download is 70 GB. Are we permitted to use this too?
", "rawMarkdown": "Hi, I came across this page on the level5 site. \n\n\n\nIs there even more training data available on level5 site than what is available through kaggle? The training set part2 download is 70 GB. Are we permitted to use this too?\n", "votes": 6 }, { "id": 1003092, "postDate": "2020-09-08T16:50:42.447Z", "content": "Just to clarify for future kagglers:
\nData on Kaggle match the ones on Lyft's website, so no problems in using those as they are the same :)
I haven't invested too much time investigating the differences, but going by the raw size of the sets, how can they be the same?
\nKaggle: 22 GB
\nLyft: 86 GB
@ilu000 You can find the \"missing\" data here
", "rawMarkdown": "@ilu000 You can find the \"missing\" data [here](https://www.kaggle.com/philculliton/lyft-full-training-set)", "votes": 2 }, { "id": 1003176, "postDate": "2020-09-08T17:54:09.563Z", "content": "thanks @pestipeti , yeah I didn't add that to the list as I wasn't sure about possible overlaps.
\nkaggle full_train: 71.7 GB
so, there is still some overlap. And i'd love to know where from the hosts (@lucabergamini ), so I don't have to investigate myself ;)
\nKaggle train/val/test + kaggle full train = ~94 GB
\nKaggle only train + kaggle full train = ~86 GB?
\nLyft: 86 GB
so is Kaggle train + kaggle full train the same as the data on the lyft website including validation? If so, what is validation on kaggle?
", "rawMarkdown": "thanks @pestipeti , yeah I didn't add that to the list as I wasn't sure about possible overlaps.\nkaggle full_train: 71.7 GB\n\nso, there is still some overlap. And i'd love to know where from the hosts (@lucabergamini ), so I don't have to investigate myself ;)\n\nKaggle train/val/test + kaggle full train = ~94 GB\nKaggle only train + kaggle full train = ~86 GB? \nLyft: 86 GB\n\nso is Kaggle train + kaggle full train the same as the data on the lyft website including validation? If so, what is validation on kaggle?" }, { "id": 1003202, "postDate": "2020-09-08T18:26:42.557Z", "content": "@ilu000 should be kaggle full train + kaggle val = lyft dataset?
", "rawMarkdown": "@ilu000 should be kaggle full train + kaggle val = lyft dataset?" }, { "id": 1003206, "postDate": "2020-09-08T18:31:31.323Z", "content": "Kaggle validate 9 GB\nKaggle full_train 71.7 GB\n---------------------------\nTotal: 80.7 GB\n\nLyft dataset including validation = 86.6 GB\nIt's close but not equal.
", "rawMarkdown": "```\nKaggle validate 9 GB\nKaggle full_train 71.7 GB\n---------------------------\nTotal: 80.7 GB\n\nLyft dataset including validation = 86.6 GB\n```\nIt's close but not equal." }, { "id": 1003254, "postDate": "2020-09-08T19:15:39.920Z", "content": "Update to release 1.1 of the dataset:
\nLyft Website
\nKaggle
\nThanks a lot @lucabergamini
\n\n\nValidation is not included on Lyft's website
\n
What is validation dataset (8.2 GB) on Lyft website then?
", "rawMarkdown": "Thanks a lot @lucabergamini \n\n> Validation is not included on Lyft's website\n\nWhat is validation dataset (8.2 GB) on Lyft website then?" }, { "id": 1003283, "postDate": "2020-09-08T19:37:36.120Z", "content": "Ah I should have checked it! We changed it in the last release. I'll edit my answer!
", "rawMarkdown": "Ah I should have checked it! We changed it in the last release. I'll edit my answer!", "votes": 1 }, { "id": 1003294, "postDate": "2020-09-08T19:47:20.793Z", "content": "Thank you for the clarification @lucabergamini , highly appreciated!
\nSo:
\nLyft train part 1 is train on kaggle
\nLyft train part 2 is train_full on kaggle
\nLyft validation is validation on kaggle
edit: is sample.zarr included in the training set?
\nand despite the name, train_full is actually not the full training set? it's just part 2 (admittedly the much larger part) of the training set?
sample.zarr is just a subset of train part 1 (first 100 scenes I think), so train_1 also includes scenes from sample.zarr.
\ntrain_full is the full train set actually (train part 1 is the first part of it).
we should probably clarify it even further somewhere as can I see people are getting a little bit confused by this
", "rawMarkdown": "`sample.zarr` is just a subset of train part 1 (first 100 scenes I think), so train_1 also includes scenes from `sample.zarr`.\n`train_full` is the full train set actually (train part 1 is the first part of it).\n\nwe should probably clarify it even further somewhere as can I see people are getting a little bit confused by this", "votes": 2 }, { "id": 1004308, "postDate": "2020-09-09T16:00:32.550Z", "content": "yes, complete clarification would be great.
\nSpecifically, is part1 (lyft website) included in part2 (lyft website)? Is part2 (lyft website) the same as train_full (kaggle dataset)?
I'm just wondering what the best place for that would be….
\nIn the meantime:
\n\"part1 is included in part2\" is a surprise))
", "rawMarkdown": "\"part1 is included in part2\" is a surprise))", "votes": 1 }, { "id": 1004623, "postDate": "2020-09-09T21:25:40.567Z", "content": "yeah, somewhat misleading wording here
", "rawMarkdown": "yeah, somewhat misleading wording here" }, { "id": 1004678, "postDate": "2020-09-09T23:39:27.983Z", "content": "@lucabergamini Yes I second @zaharch and @ilu000 the numbers are bit confusing. Grateful for the dataset and competition but even on the published papers numbers don't add up. The claim is there are 170k scene. But tabular description on paper says 162k scene. Only thing constant is hrs and Km.Think we need to find out for ourselves. I recommend a separate thread created by host to clarify size and the number of scene.
", "rawMarkdown": "@lucabergamini Yes I second @zaharch and @ilu000 the numbers are bit confusing. Grateful for the dataset and competition but even on the published papers numbers don't add up. The claim is there are 170k scene. But tabular description on paper says 162k scene. Only thing constant is hrs and Km.Think we need to find out for ourselves. I recommend a separate thread created by host to clarify size and the number of scene. ", "votes": 1 }, { "id": 1005247, "postDate": "2020-09-10T10:59:52.600Z", "content": "We re-sourced the dataset between the paper release and the competition start to include also traffic lights. We tried to retain the original splits, but it was not trivial with more than 100k scenes.
\nI'll start a post to clarify the current structure :)
", "rawMarkdown": "We re-sourced the dataset between the paper release and the competition start to include also traffic lights. We tried to retain the original splits, but it was not trivial with more than 100k scenes. \n\nI'll start a post to clarify the current structure :) ", "votes": 2 } ] }, { "id": 999216, "postDate": "2020-09-05T13:38:16.057Z", "content": "This should be the one mentioned under the \"Additional Files\" paragraph in the \"Data\"-Tab, also hosted on kaggle
\nGiven that it is mentioned under the Data-Tab, I would certainly assume that we can use it for training.
wow, I completely missed that. Thank you @hanshaas
", "rawMarkdown": "wow, I completely missed that. Thank you @hanshaas ", "votes": 1 } ] }, { "id": 999365, "postDate": "2020-09-05T15:39:41.097Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1003092, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-08T16:50:42.447000", "content": "Just to clarify for future kagglers:
\nData on Kaggle match the ones on Lyft's website, so no problems in using those as they are the same :)
I haven't invested too much time investigating the differences, but going by the raw size of the sets, how can they be the same?
\nKaggle: 22 GB
\nLyft: 86 GB
@ilu000 You can find the \"missing\" data here
", "votes": 2, "replies": [] }, { "id": 1003176, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-08T17:54:09.563000", "content": "thanks @pestipeti , yeah I didn't add that to the list as I wasn't sure about possible overlaps.
\nkaggle full_train: 71.7 GB
so, there is still some overlap. And i'd love to know where from the hosts (@lucabergamini ), so I don't have to investigate myself ;)
\nKaggle train/val/test + kaggle full train = ~94 GB
\nKaggle only train + kaggle full train = ~86 GB?
\nLyft: 86 GB
so is Kaggle train + kaggle full train the same as the data on the lyft website including validation? If so, what is validation on kaggle?
", "votes": 0, "replies": [] }, { "id": 1003202, "author_name": "fergusoci", "author_url": "", "post_date": "2020-09-08T18:26:42.557000", "content": "@ilu000 should be kaggle full train + kaggle val = lyft dataset?
", "votes": 0, "replies": [] }, { "id": 1003206, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-08T18:31:31.323000", "content": "Kaggle validate 9 GB\nKaggle full_train 71.7 GB\n---------------------------\nTotal: 80.7 GB\n\nLyft dataset including validation = 86.6 GB\nIt's close but not equal.
", "votes": 0, "replies": [] }, { "id": 1003254, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-08T19:15:39.920000", "content": "Update to release 1.1 of the dataset:
\nLyft Website
\nKaggle
\nThanks a lot @lucabergamini
\n\n\nValidation is not included on Lyft's website
\n
What is validation dataset (8.2 GB) on Lyft website then?
", "votes": 0, "replies": [] }, { "id": 1003283, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-08T19:37:36.120000", "content": "Ah I should have checked it! We changed it in the last release. I'll edit my answer!
", "votes": 1, "replies": [] }, { "id": 1003294, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-08T19:47:20.793000", "content": "Thank you for the clarification @lucabergamini , highly appreciated!
\nSo:
\nLyft train part 1 is train on kaggle
\nLyft train part 2 is train_full on kaggle
\nLyft validation is validation on kaggle
edit: is sample.zarr included in the training set?
\nand despite the name, train_full is actually not the full training set? it's just part 2 (admittedly the much larger part) of the training set?
sample.zarr is just a subset of train part 1 (first 100 scenes I think), so train_1 also includes scenes from sample.zarr.
\ntrain_full is the full train set actually (train part 1 is the first part of it).
we should probably clarify it even further somewhere as can I see people are getting a little bit confused by this
", "votes": 2, "replies": [] }, { "id": 1004308, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-09T16:00:32.550000", "content": "yes, complete clarification would be great.
\nSpecifically, is part1 (lyft website) included in part2 (lyft website)? Is part2 (lyft website) the same as train_full (kaggle dataset)?
I'm just wondering what the best place for that would be….
\nIn the meantime:
\n\"part1 is included in part2\" is a surprise))
", "votes": 1, "replies": [] }, { "id": 1004623, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-09T21:25:40.567000", "content": "yeah, somewhat misleading wording here
", "votes": 0, "replies": [] }, { "id": 1004678, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-09T23:39:27.983000", "content": "@lucabergamini Yes I second @zaharch and @ilu000 the numbers are bit confusing. Grateful for the dataset and competition but even on the published papers numbers don't add up. The claim is there are 170k scene. But tabular description on paper says 162k scene. Only thing constant is hrs and Km.Think we need to find out for ourselves. I recommend a separate thread created by host to clarify size and the number of scene.
", "votes": 1, "replies": [] }, { "id": 1005247, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-10T10:59:52.600000", "content": "We re-sourced the dataset between the paper release and the competition start to include also traffic lights. We tried to retain the original splits, but it was not trivial with more than 100k scenes.
\nI'll start a post to clarify the current structure :)
", "votes": 2, "replies": [] } ] }, { "id": 999216, "author_name": "Johann Haas", "author_url": "", "post_date": "2020-09-05T13:38:16.057000", "content": "This should be the one mentioned under the \"Additional Files\" paragraph in the \"Data\"-Tab, also hosted on kaggle
\nGiven that it is mentioned under the Data-Tab, I would certainly assume that we can use it for training.
wow, I completely missed that. Thank you @hanshaas
", "votes": 1, "replies": [] } ] }, { "id": 999365, "author_name": "", "author_url": "", "post_date": "2020-09-05T15:39:41.097000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "999000": "Hi, I came across this page on the level5 site. \n\n\n\nIs there even more training data available on level5 site than what is available through kaggle? The training set part2 download is 70 GB. Are we permitted to use this too?\n", "1003092": "Just to clarify for future kagglers:\nData on Kaggle match the ones on Lyft's website, so no problems in using those as they are the same :) ", "999216": "This should be the one mentioned under the \"Additional Files\" paragraph in the \"Data\"-Tab, also hosted on [kaggle](https://www.kaggle.com/philculliton/lyft-full-training-set)\nGiven that it is mentioned under the Data-Tab, I would certainly assume that we can use it for training.", "999365": "" } }