has anybody else see some issues with the cloud in file train_lidar/host-a011_lidar1_1233090652702363606.bin ? the array needs to have a number of elements divisble by 5 for the python code to work (else an error https://github.com/lyft/nuscenes-devkit/blob/master/lyft_dataset_sdk/utils/data_classes.py#L286 )
In the kaggle zip file, I get 265728 values:\n```
\n\n\n\n\n\n\n\n\narr = np.fromfile('/lyft_kaggle_root/train_lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n arr.shape\n (265728,)\n ```
\n
But from the Lyft Level 5 download site ( https://level5.lyft.com/dataset/ -- which also must have this cloud, because it's part of the training set and not the kaggle-only test set), I see 341800 values:\n```
\n\n\n\n\n\n\n\n\narr = np.fromfile('/lyft_level_5_root/train/lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n arr.shape\n (341800,)\n ```
\n
It looks like the cloud file might have gotten truncated when added to the Kaggle tarball?
\n\nI ran a job over the 30,744 lidar files in the kaggle training set, and this is the only file that causes an error.
\n\nI ran a job over the 27,468 lidar files in the kaggle test set, and everything looks ok (or at least divisible by 5 ...)...
", "messages": [ { "id": "664178", "postDate": "11/03/2019 09:30:37", "content": "has anybody else see some issues with the cloud in file train_lidar/host-a011_lidar1_1233090652702363606.bin ? the array needs to have a number of elements divisble by 5 for the python code to work (else an error https://github.com/lyft/nuscenes-devkit/blob/master/lyft_dataset_sdk/utils/data_classes.py#L286 )
In the kaggle zip file, I get 265728 values:\n```
\n\n\n\n\n\n\n\n\narr = np.fromfile('/lyft_kaggle_root/train_lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n arr.shape\n (265728,)\n ```
\n
But from the Lyft Level 5 download site ( https://level5.lyft.com/dataset/ -- which also must have this cloud, because it's part of the training set and not the kaggle-only test set), I see 341800 values:\n```
\n\n\n\n\n\n\n\n\narr = np.fromfile('/lyft_level_5_root/train/lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n arr.shape\n (341800,)\n ```
\n
It looks like the cloud file might have gotten truncated when added to the Kaggle tarball?
\n\nI ran a job over the 30,744 lidar files in the kaggle training set, and this is the only file that causes an error.
\n\nI ran a job over the 27,468 lidar files in the kaggle test set, and everything looks ok (or at least divisible by 5 ...)...
", "rawMarkdown": "has anybody else see some issues with the cloud in file `train_lidar/host-a011_lidar1_1233090652702363606.bin` ? the array needs to have a number of elements divisble by 5 for the python code to work (else an error https://github.com/lyft/nuscenes-devkit/blob/master/lyft_dataset_sdk/utils/data_classes.py#L286 )\n\nIn the kaggle zip file, I get 265728 values:\n```\n>>> arr = np.fromfile('/lyft_kaggle_root/train_lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n>>> arr.shape\n(265728,)\n```\n\nBut from the Lyft Level 5 download site ( https://level5.lyft.com/dataset/ -- which also must have this cloud, because it's part of the training set and not the kaggle-only test set), I see 341800 values:\n```\n>>> arr = np.fromfile('/lyft_level_5_root/train/lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n>>> arr.shape\n(341800,)\n```\n\nIt looks like the cloud file might have gotten truncated when added to the Kaggle tarball?\n\n\nI ran a job over the 30,744 lidar files in the kaggle training set, and this is the only file that causes an error. \n\nI ran a job over the 27,468 lidar files in the kaggle *test* set, and everything looks ok (or at least divisible by 5 ...)...", "votes": null }, { "id": "664201", "postDate": "11/03/2019 10:17:19", "content": "this issue has already been discussed in one of the threads.
", "rawMarkdown": "this issue has already been discussed in one of the threads.", "votes": null }, { "id": "664698", "postDate": "11/04/2019 04:42:47", "content": "@rishabhiitbhu thank you! derp could you link to that thread? or might you just know if the result was that the kaggle data was updated / fixed? I admit I appear to have an old copy of the zipfile.
", "rawMarkdown": "rishabhiitbhu thank you! derp could you link to that thread? or might you just know if the result was that the kaggle data was updated / fixed? I admit I appear to have an old copy of the zipfile.", "votes": null }, { "id": "664725", "postDate": "11/04/2019 06:08:09", "content": "Here's the fixed version:
", "rawMarkdown": "Here's the fixed version:", "votes": null }, { "id": "664827", "postDate": "11/04/2019 09:29:08", "content": "oh thank you! yes I have that though from https://level5.lyft.com/dataset/ I was just curious if they fixed the kaggle download perhaps
", "rawMarkdown": "oh thank you! yes I have that though from https://level5.lyft.com/dataset/ I was just curious if they fixed the kaggle download perhaps", "votes": null } ], "comments": [ { "id": 664201, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "11/03/2019 10:17:19", "content": "this issue has already been discussed in one of the threads.
", "votes": null, "replies": [] }, { "id": 664698, "author_name": "oarphme", "author_url": "", "post_date": "11/04/2019 04:42:47", "content": "@rishabhiitbhu thank you! derp could you link to that thread? or might you just know if the result was that the kaggle data was updated / fixed? I admit I appear to have an old copy of the zipfile.
", "votes": null, "replies": [ { "id": 664725, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "11/04/2019 06:08:09", "content": "Here's the fixed version:
", "votes": null, "replies": [] }, { "id": 664827, "author_name": "oarphme", "author_url": "", "post_date": "11/04/2019 09:29:08", "content": "oh thank you! yes I have that though from https://level5.lyft.com/dataset/ I was just curious if they fixed the kaggle download perhaps
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "664178": "has anybody else see some issues with the cloud in file `train_lidar/host-a011_lidar1_1233090652702363606.bin` ? the array needs to have a number of elements divisble by 5 for the python code to work (else an error https://github.com/lyft/nuscenes-devkit/blob/master/lyft_dataset_sdk/utils/data_classes.py#L286 )\n\nIn the kaggle zip file, I get 265728 values:\n```\n>>> arr = np.fromfile('/lyft_kaggle_root/train_lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n>>> arr.shape\n(265728,)\n```\n\nBut from the Lyft Level 5 download site ( https://level5.lyft.com/dataset/ -- which also must have this cloud, because it's part of the training set and not the kaggle-only test set), I see 341800 values:\n```\n>>> arr = np.fromfile('/lyft_level_5_root/train/lidar/host-a011_lidar1_1233090652702363606.bin', dtype=np.float32)\n>>> arr.shape\n(341800,)\n```\n\nIt looks like the cloud file might have gotten truncated when added to the Kaggle tarball?\n\n\nI ran a job over the 30,744 lidar files in the kaggle training set, and this is the only file that causes an error. \n\nI ran a job over the 27,468 lidar files in the kaggle *test* set, and everything looks ok (or at least divisible by 5 ...)...", "664201": "this issue has already been discussed in one of the threads.", "664698": "rishabhiitbhu thank you! derp could you link to that thread? or might you just know if the result was that the kaggle data was updated / fixed? I admit I appear to have an old copy of the zipfile.", "664725": "Here's the fixed version:", "664827": "oh thank you! yes I have that though from https://level5.lyft.com/dataset/ I was just curious if they fixed the kaggle download perhaps" }, "source": "meta" }