{
  "id": 112439,
  "title": "Camera-lidar sync and comparison with NuScenes",
  "url": "/competitions/3d-object-detection-for-autonomous-vehicles/discussion/112439",
  "author_name": "",
  "post_date": "2019-10-12T23:41:54.820369200Z",
  "votes": 6,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I was curious about the quality of the camera-lidar sync and how it compares to NuScenes ( <a href=\"https://www.nuscenes.org/\">https://www.nuscenes.org/</a> ).  I wrote a function that you can add as a member function to your <code>LyftDataset</code> or <code>NuScenes</code> class (code: <a href=\"https://pastebin.com/HRuEFTjd\">https://pastebin.com/HRuEFTjd</a> ).</p>\n\n<p>Major findings:\n  * Lyft Level 5 appears to <em>only</em> include labeled sensor data, with labels at 5Hz.  In contrast, NuScenes includes some unlabeled sensor data, with labels at 2Hz.  (Aside: I investigated interpolating NuScenes labels to the full sensor sample rate, and results looked qualitatively OK.  FWIW, the LyftDataset and NuScenes code appear to automatically project labels through the world frame, so I think you get motion correction of labels for free using either codebase).</p>\n\n<ul>\n<li><p>Lyft Level 5 lidars are synchronized to each other, but do not appear synchronized to any particular camera.  In particular, there appears to be a 50ms offset between lidar sweeps and the front camera.  In comparison, NuScenes single lidar is roughly synchronized with <code>CAM_BACK_LEFT</code>, and there's only a ~15ms offset to the front camera.  </p></li>\n<li><p>Oddly, Lyft Level 5 appears to have perfect lidar-localization sync (i.e. diff with <code>ego_pose</code>), but NuScenes has a consistent 11ms offset.  I'd assume that both Lyft Level 5 and NuScenes are using lidar-enhanced localization, so perhaps the NuScenes <code>ego_pose</code> timestamp is just wrong.</p></li>\n<li><p>NuScenes runs their Velodyne at full 20Hz, but only record cameras at about 12Hz.  </p></li>\n</ul>\n\n<p>So it should be possible to convert NuScenes to a 5Hz (or even perhaps ~12Hz) dataset similar to Lyft Level 5.  Note that the Lyft lidars have substantially better coverage, especially for pedestrians and bikes.</p>\n\n<p>Sample results, 3 scenes from each:\n```</p>\n\n<h2>NuScenes:</h2>\n\n<hr>\n\n<p>Scene scene-1000 09f67057dd8346388b28f79d9bb1cf04\nStart 2018-11-14 11:01:41       Duration 19.922956943511963 sec\nNum Annos 493 (Tracks 27)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.738035          10.554540  19.850000      234\n1       CAM_BACK_LEFT   11.536524           0.918133  19.850000      230\n2      CAM_BACK_RIGHT   11.788413          20.070969  19.850000      235\n3           CAM_FRONT   11.637280          14.986247  19.850000      232\n4      CAM_FRONT_LEFT   11.536524           7.586523  19.850000      230\n5     CAM_FRONT_RIGHT   11.536524          22.528135  19.850000      230\n6           LIDAR_TOP   19.747937           0.000000  19.850175      393\n7     RADAR_BACK_LEFT   12.593898          12.635898  19.850883      251\n8    RADAR_BACK_RIGHT   13.211602          12.786931  19.831054      263\n9         RADAR_FRONT   12.976290          12.728528  19.882416      259\n10   RADAR_FRONT_LEFT   13.510020          12.710849  19.911148      270\n11  RADAR_FRONT_RIGHT   13.724216          12.571387  19.891846      274\n12           ego_pose  155.599393          11.128198  19.922957     3101\n13     sample (annos)    2.015096           0.000000  19.850175       41</p>\n\n<h2>14        sample_data  155.599393          11.128198  19.922957     3101</h2>\n\n<hr>\n\n<p>Scene scene-0293 6308d6d934074a028fc3145eedf3e65f\nStart 2018-08-31 15:25:42       Duration 19.525898933410645 sec\nNum Annos 3548 (Tracks 277)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.773779          10.441362  19.450000      230\n1       CAM_BACK_LEFT   11.825193           1.573582  19.450000      231\n2      CAM_BACK_RIGHT   11.928021          19.624993  19.450000      233\n3           CAM_FRONT   11.876607          14.872485  19.450000      232\n4      CAM_FRONT_LEFT   11.876607           7.329719  19.450000      232\n5     CAM_FRONT_RIGHT   11.979434          22.875966  19.450000      234\n6           LIDAR_TOP   19.844216           0.000000  19.451512      387\n7     RADAR_BACK_LEFT   13.157897          12.698666  19.455997      257\n8    RADAR_BACK_RIGHT   13.490288          12.647669  19.421380      263\n9         RADAR_FRONT   13.082394          12.616018  19.491845      256\n10   RADAR_FRONT_LEFT   13.305139          12.709008  19.466163      260\n11  RADAR_FRONT_RIGHT   13.209300          12.723777  19.455989      258\n12           ego_pose  157.329504          11.144872  19.525899     3073\n13     sample (annos)    2.004986           0.000000  19.451512       40</p>\n\n<h2>14        sample_data  157.329504          11.144872  19.525899     3073</h2>\n\n<hr>\n\n<p>Scene scene-1107 89f20737ec344aa48b543a9e005a38ca\nStart 2018-11-21 11:59:53       Duration 19.820924997329712 sec\nNum Annos 496 (Tracks 47)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.696203          11.014590  19.750000      232\n1       CAM_BACK_LEFT   11.848101           1.382666  19.750000      235\n2      CAM_BACK_RIGHT   11.746835          20.483792  19.750000      233\n3           CAM_FRONT   11.848103          14.481831  19.749997      235\n4      CAM_FRONT_LEFT   11.898734           7.063236  19.750000      236\n5     CAM_FRONT_RIGHT   11.898734          22.159666  19.750000      236\n6           LIDAR_TOP   19.797377           0.000000  19.750091      392\n7     RADAR_BACK_LEFT   13.578124          12.646209  19.811279      270\n8    RADAR_BACK_RIGHT   13.271911          12.721395  19.740940      263\n9         RADAR_FRONT   13.198245          12.553021  19.775357      262\n10   RADAR_FRONT_LEFT   13.730931          12.645984  19.736462      272\n11  RADAR_FRONT_RIGHT   13.778595          12.488227  19.740764      273\n12           ego_pose  158.317536          11.102567  19.820925     3139\n13     sample (annos)    2.025307           0.000000  19.750091       41\n14        sample_data  158.317536          11.102567  19.820925     3139</p>\n\n<h2>Lyft Level 5:</h2>\n\n<hr>\n\n<p>Scene host-a015-lidar0-1235423635198474636-1235423660098038666 755e4564756ad5c92243b7f77039d07ab1cce40662a6a19b67c820647666a3ef\nStart 2019-02-28 21:13:55       Duration 24.99979877471924 sec\nNum Annos 1637 (Tracks 44)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.882582  24.900000      126\n1      CAM_BACK_LEFT   5.020080          16.919276  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          82.887411  24.900000      126\n3          CAM_FRONT   5.020080          50.027272  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.542296  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.427920  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.096270  24.900000      126\n7          LIDAR_TOP   5.020156           0.000000  24.899626      126\n8           ego_pose  40.442375           0.000000  24.899626     1008\n9     sample (annos)   5.020156           0.000000  24.899626      126</p>\n\n<h2>10       sample_data  40.280324          49.847878  24.999799     1008</h2>\n\n<hr>\n\n<p>Scene host-a004-lidar0-1233947108297817786-1233947133198765096 114b780b2efd6f73f134fc3a8f9db628e43131dc47f90e9b5dfdb886400d70f2\nStart 2019-02-11 19:05:08       Duration 25.000741004943848 sec\nNum Annos 4155 (Tracks 137)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.790201  24.900000      126\n1      CAM_BACK_LEFT   5.020080          17.030725  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          83.033195  24.900000      126\n3          CAM_FRONT   5.020080          50.151564  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.667718  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.649443  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.265691  24.900000      126\n7          LIDAR_TOP   5.019934           0.000000  24.900724      126\n8           ego_pose  40.440592           0.000000  24.900724     1008\n9     sample (annos)   5.019934           0.000000  24.900724      126</p>\n\n<h2>10       sample_data  40.278806          49.948567  25.000741     1008</h2>\n\n<hr>\n\n<p>Scene host-a101-lidar0-1241886983298988182-1241887008198992182 7b4640d63a9c62d07a8551d4b430d0acd88eaba8249c843248feb888f4630070\nStart 2019-05-14 16:36:23       Duration 25.002139806747437 sec\nNum Annos 4777 (Tracks 173)\n               Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   5.020080          93.825297  24.900000      126\n1       CAM_BACK_LEFT   5.020080          85.205165  24.900000      126\n2      CAM_BACK_RIGHT   5.020080          19.099243  24.900000      126\n3           CAM_FRONT   5.020080          52.394347  24.900000      126\n4      CAM_FRONT_LEFT   5.020080          68.799780  24.900000      126\n5     CAM_FRONT_RIGHT   5.020080          35.769232  24.900000      126\n6    CAM_FRONT_ZOOMED   5.020080          52.394347  24.900000      126\n7    LIDAR_FRONT_LEFT   5.020060           0.000000  24.900101      126\n8   LIDAR_FRONT_RIGHT   5.020060           0.000000  24.900101      126\n9           LIDAR_TOP   5.020060           0.000000  24.900101      126\n10           ego_pose  50.562044           0.000000  24.900101     1260\n11     sample (annos)   5.020060           0.000000  24.900101      126\n12        sample_data  50.355690          40.748741  25.002140     1260\n```</p>",
  "messages": [
    {
      "id": "647603",
      "postDate": "10/12/2019 23:41:54",
      "content": "<p>I was curious about the quality of the camera-lidar sync and how it compares to NuScenes ( <a href=\"https://www.nuscenes.org/\">https://www.nuscenes.org/</a> ).  I wrote a function that you can add as a member function to your <code>LyftDataset</code> or <code>NuScenes</code> class (code: <a href=\"https://pastebin.com/HRuEFTjd\">https://pastebin.com/HRuEFTjd</a> ).</p>\n\n<p>Major findings:\n  * Lyft Level 5 appears to <em>only</em> include labeled sensor data, with labels at 5Hz.  In contrast, NuScenes includes some unlabeled sensor data, with labels at 2Hz.  (Aside: I investigated interpolating NuScenes labels to the full sensor sample rate, and results looked qualitatively OK.  FWIW, the LyftDataset and NuScenes code appear to automatically project labels through the world frame, so I think you get motion correction of labels for free using either codebase).</p>\n\n<ul>\n<li><p>Lyft Level 5 lidars are synchronized to each other, but do not appear synchronized to any particular camera.  In particular, there appears to be a 50ms offset between lidar sweeps and the front camera.  In comparison, NuScenes single lidar is roughly synchronized with <code>CAM_BACK_LEFT</code>, and there's only a ~15ms offset to the front camera.  </p></li>\n<li><p>Oddly, Lyft Level 5 appears to have perfect lidar-localization sync (i.e. diff with <code>ego_pose</code>), but NuScenes has a consistent 11ms offset.  I'd assume that both Lyft Level 5 and NuScenes are using lidar-enhanced localization, so perhaps the NuScenes <code>ego_pose</code> timestamp is just wrong.</p></li>\n<li><p>NuScenes runs their Velodyne at full 20Hz, but only record cameras at about 12Hz.  </p></li>\n</ul>\n\n<p>So it should be possible to convert NuScenes to a 5Hz (or even perhaps ~12Hz) dataset similar to Lyft Level 5.  Note that the Lyft lidars have substantially better coverage, especially for pedestrians and bikes.</p>\n\n<p>Sample results, 3 scenes from each:\n```</p>\n\n<h2>NuScenes:</h2>\n\n<hr>\n\n<p>Scene scene-1000 09f67057dd8346388b28f79d9bb1cf04\nStart 2018-11-14 11:01:41       Duration 19.922956943511963 sec\nNum Annos 493 (Tracks 27)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.738035          10.554540  19.850000      234\n1       CAM_BACK_LEFT   11.536524           0.918133  19.850000      230\n2      CAM_BACK_RIGHT   11.788413          20.070969  19.850000      235\n3           CAM_FRONT   11.637280          14.986247  19.850000      232\n4      CAM_FRONT_LEFT   11.536524           7.586523  19.850000      230\n5     CAM_FRONT_RIGHT   11.536524          22.528135  19.850000      230\n6           LIDAR_TOP   19.747937           0.000000  19.850175      393\n7     RADAR_BACK_LEFT   12.593898          12.635898  19.850883      251\n8    RADAR_BACK_RIGHT   13.211602          12.786931  19.831054      263\n9         RADAR_FRONT   12.976290          12.728528  19.882416      259\n10   RADAR_FRONT_LEFT   13.510020          12.710849  19.911148      270\n11  RADAR_FRONT_RIGHT   13.724216          12.571387  19.891846      274\n12           ego_pose  155.599393          11.128198  19.922957     3101\n13     sample (annos)    2.015096           0.000000  19.850175       41</p>\n\n<h2>14        sample_data  155.599393          11.128198  19.922957     3101</h2>\n\n<hr>\n\n<p>Scene scene-0293 6308d6d934074a028fc3145eedf3e65f\nStart 2018-08-31 15:25:42       Duration 19.525898933410645 sec\nNum Annos 3548 (Tracks 277)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.773779          10.441362  19.450000      230\n1       CAM_BACK_LEFT   11.825193           1.573582  19.450000      231\n2      CAM_BACK_RIGHT   11.928021          19.624993  19.450000      233\n3           CAM_FRONT   11.876607          14.872485  19.450000      232\n4      CAM_FRONT_LEFT   11.876607           7.329719  19.450000      232\n5     CAM_FRONT_RIGHT   11.979434          22.875966  19.450000      234\n6           LIDAR_TOP   19.844216           0.000000  19.451512      387\n7     RADAR_BACK_LEFT   13.157897          12.698666  19.455997      257\n8    RADAR_BACK_RIGHT   13.490288          12.647669  19.421380      263\n9         RADAR_FRONT   13.082394          12.616018  19.491845      256\n10   RADAR_FRONT_LEFT   13.305139          12.709008  19.466163      260\n11  RADAR_FRONT_RIGHT   13.209300          12.723777  19.455989      258\n12           ego_pose  157.329504          11.144872  19.525899     3073\n13     sample (annos)    2.004986           0.000000  19.451512       40</p>\n\n<h2>14        sample_data  157.329504          11.144872  19.525899     3073</h2>\n\n<hr>\n\n<p>Scene scene-1107 89f20737ec344aa48b543a9e005a38ca\nStart 2018-11-21 11:59:53       Duration 19.820924997329712 sec\nNum Annos 496 (Tracks 47)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.696203          11.014590  19.750000      232\n1       CAM_BACK_LEFT   11.848101           1.382666  19.750000      235\n2      CAM_BACK_RIGHT   11.746835          20.483792  19.750000      233\n3           CAM_FRONT   11.848103          14.481831  19.749997      235\n4      CAM_FRONT_LEFT   11.898734           7.063236  19.750000      236\n5     CAM_FRONT_RIGHT   11.898734          22.159666  19.750000      236\n6           LIDAR_TOP   19.797377           0.000000  19.750091      392\n7     RADAR_BACK_LEFT   13.578124          12.646209  19.811279      270\n8    RADAR_BACK_RIGHT   13.271911          12.721395  19.740940      263\n9         RADAR_FRONT   13.198245          12.553021  19.775357      262\n10   RADAR_FRONT_LEFT   13.730931          12.645984  19.736462      272\n11  RADAR_FRONT_RIGHT   13.778595          12.488227  19.740764      273\n12           ego_pose  158.317536          11.102567  19.820925     3139\n13     sample (annos)    2.025307           0.000000  19.750091       41\n14        sample_data  158.317536          11.102567  19.820925     3139</p>\n\n<h2>Lyft Level 5:</h2>\n\n<hr>\n\n<p>Scene host-a015-lidar0-1235423635198474636-1235423660098038666 755e4564756ad5c92243b7f77039d07ab1cce40662a6a19b67c820647666a3ef\nStart 2019-02-28 21:13:55       Duration 24.99979877471924 sec\nNum Annos 1637 (Tracks 44)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.882582  24.900000      126\n1      CAM_BACK_LEFT   5.020080          16.919276  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          82.887411  24.900000      126\n3          CAM_FRONT   5.020080          50.027272  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.542296  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.427920  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.096270  24.900000      126\n7          LIDAR_TOP   5.020156           0.000000  24.899626      126\n8           ego_pose  40.442375           0.000000  24.899626     1008\n9     sample (annos)   5.020156           0.000000  24.899626      126</p>\n\n<h2>10       sample_data  40.280324          49.847878  24.999799     1008</h2>\n\n<hr>\n\n<p>Scene host-a004-lidar0-1233947108297817786-1233947133198765096 114b780b2efd6f73f134fc3a8f9db628e43131dc47f90e9b5dfdb886400d70f2\nStart 2019-02-11 19:05:08       Duration 25.000741004943848 sec\nNum Annos 4155 (Tracks 137)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.790201  24.900000      126\n1      CAM_BACK_LEFT   5.020080          17.030725  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          83.033195  24.900000      126\n3          CAM_FRONT   5.020080          50.151564  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.667718  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.649443  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.265691  24.900000      126\n7          LIDAR_TOP   5.019934           0.000000  24.900724      126\n8           ego_pose  40.440592           0.000000  24.900724     1008\n9     sample (annos)   5.019934           0.000000  24.900724      126</p>\n\n<h2>10       sample_data  40.278806          49.948567  25.000741     1008</h2>\n\n<hr>\n\n<p>Scene host-a101-lidar0-1241886983298988182-1241887008198992182 7b4640d63a9c62d07a8551d4b430d0acd88eaba8249c843248feb888f4630070\nStart 2019-05-14 16:36:23       Duration 25.002139806747437 sec\nNum Annos 4777 (Tracks 173)\n               Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   5.020080          93.825297  24.900000      126\n1       CAM_BACK_LEFT   5.020080          85.205165  24.900000      126\n2      CAM_BACK_RIGHT   5.020080          19.099243  24.900000      126\n3           CAM_FRONT   5.020080          52.394347  24.900000      126\n4      CAM_FRONT_LEFT   5.020080          68.799780  24.900000      126\n5     CAM_FRONT_RIGHT   5.020080          35.769232  24.900000      126\n6    CAM_FRONT_ZOOMED   5.020080          52.394347  24.900000      126\n7    LIDAR_FRONT_LEFT   5.020060           0.000000  24.900101      126\n8   LIDAR_FRONT_RIGHT   5.020060           0.000000  24.900101      126\n9           LIDAR_TOP   5.020060           0.000000  24.900101      126\n10           ego_pose  50.562044           0.000000  24.900101     1260\n11     sample (annos)   5.020060           0.000000  24.900101      126\n12        sample_data  50.355690          40.748741  25.002140     1260\n```</p>",
      "rawMarkdown": "I was curious about the quality of the camera-lidar sync and how it compares to NuScenes ( https://www.nuscenes.org/ ).  I wrote a function that you can add as a member function to your `LyftDataset ` or `NuScenes` class (code: https://pastebin.com/HRuEFTjd ).\n\nMajor findings:\n  * Lyft Level 5 appears to *only* include labeled sensor data, with labels at 5Hz.  In contrast, NuScenes includes some unlabeled sensor data, with labels at 2Hz.  (Aside: I investigated interpolating NuScenes labels to the full sensor sample rate, and results looked qualitatively OK.  FWIW, the LyftDataset and NuScenes code appear to automatically project labels through the world frame, so I think you get motion correction of labels for free using either codebase).\n\n  * Lyft Level 5 lidars are synchronized to each other, but do not appear synchronized to any particular camera.  In particular, there appears to be a 50ms offset between lidar sweeps and the front camera.  In comparison, NuScenes single lidar is roughly synchronized with `CAM_BACK_LEFT`, and there's only a ~15ms offset to the front camera.  \n\n  * Oddly, Lyft Level 5 appears to have perfect lidar-localization sync (i.e. diff with `ego_pose`), but NuScenes has a consistent 11ms offset.  I'd assume that both Lyft Level 5 and NuScenes are using lidar-enhanced localization, so perhaps the NuScenes `ego_pose` timestamp is just wrong.\n\n  * NuScenes runs their Velodyne at full 20Hz, but only record cameras at about 12Hz.  \n\nSo it should be possible to convert NuScenes to a 5Hz (or even perhaps ~12Hz) dataset similar to Lyft Level 5.  Note that the Lyft lidars have substantially better coverage, especially for pedestrians and bikes.\n\nSample results, 3 scenes from each:\n```\nNuScenes:\n---\n---\nScene scene-1000 09f67057dd8346388b28f79d9bb1cf04\nStart 2018-11-14 11:01:41       Duration 19.922956943511963 sec\nNum Annos 493 (Tracks 27)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.738035          10.554540  19.850000      234\n1       CAM_BACK_LEFT   11.536524           0.918133  19.850000      230\n2      CAM_BACK_RIGHT   11.788413          20.070969  19.850000      235\n3           CAM_FRONT   11.637280          14.986247  19.850000      232\n4      CAM_FRONT_LEFT   11.536524           7.586523  19.850000      230\n5     CAM_FRONT_RIGHT   11.536524          22.528135  19.850000      230\n6           LIDAR_TOP   19.747937           0.000000  19.850175      393\n7     RADAR_BACK_LEFT   12.593898          12.635898  19.850883      251\n8    RADAR_BACK_RIGHT   13.211602          12.786931  19.831054      263\n9         RADAR_FRONT   12.976290          12.728528  19.882416      259\n10   RADAR_FRONT_LEFT   13.510020          12.710849  19.911148      270\n11  RADAR_FRONT_RIGHT   13.724216          12.571387  19.891846      274\n12           ego_pose  155.599393          11.128198  19.922957     3101\n13     sample (annos)    2.015096           0.000000  19.850175       41\n14        sample_data  155.599393          11.128198  19.922957     3101\n---\n---\nScene scene-0293 6308d6d934074a028fc3145eedf3e65f\nStart 2018-08-31 15:25:42       Duration 19.525898933410645 sec\nNum Annos 3548 (Tracks 277)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.773779          10.441362  19.450000      230\n1       CAM_BACK_LEFT   11.825193           1.573582  19.450000      231\n2      CAM_BACK_RIGHT   11.928021          19.624993  19.450000      233\n3           CAM_FRONT   11.876607          14.872485  19.450000      232\n4      CAM_FRONT_LEFT   11.876607           7.329719  19.450000      232\n5     CAM_FRONT_RIGHT   11.979434          22.875966  19.450000      234\n6           LIDAR_TOP   19.844216           0.000000  19.451512      387\n7     RADAR_BACK_LEFT   13.157897          12.698666  19.455997      257\n8    RADAR_BACK_RIGHT   13.490288          12.647669  19.421380      263\n9         RADAR_FRONT   13.082394          12.616018  19.491845      256\n10   RADAR_FRONT_LEFT   13.305139          12.709008  19.466163      260\n11  RADAR_FRONT_RIGHT   13.209300          12.723777  19.455989      258\n12           ego_pose  157.329504          11.144872  19.525899     3073\n13     sample (annos)    2.004986           0.000000  19.451512       40\n14        sample_data  157.329504          11.144872  19.525899     3073\n---\n---\nScene scene-1107 89f20737ec344aa48b543a9e005a38ca\nStart 2018-11-21 11:59:53       Duration 19.820924997329712 sec\nNum Annos 496 (Tracks 47)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.696203          11.014590  19.750000      232\n1       CAM_BACK_LEFT   11.848101           1.382666  19.750000      235\n2      CAM_BACK_RIGHT   11.746835          20.483792  19.750000      233\n3           CAM_FRONT   11.848103          14.481831  19.749997      235\n4      CAM_FRONT_LEFT   11.898734           7.063236  19.750000      236\n5     CAM_FRONT_RIGHT   11.898734          22.159666  19.750000      236\n6           LIDAR_TOP   19.797377           0.000000  19.750091      392\n7     RADAR_BACK_LEFT   13.578124          12.646209  19.811279      270\n8    RADAR_BACK_RIGHT   13.271911          12.721395  19.740940      263\n9         RADAR_FRONT   13.198245          12.553021  19.775357      262\n10   RADAR_FRONT_LEFT   13.730931          12.645984  19.736462      272\n11  RADAR_FRONT_RIGHT   13.778595          12.488227  19.740764      273\n12           ego_pose  158.317536          11.102567  19.820925     3139\n13     sample (annos)    2.025307           0.000000  19.750091       41\n14        sample_data  158.317536          11.102567  19.820925     3139\n\nLyft Level 5:\n---\n---\nScene host-a015-lidar0-1235423635198474636-1235423660098038666 755e4564756ad5c92243b7f77039d07ab1cce40662a6a19b67c820647666a3ef\nStart 2019-02-28 21:13:55       Duration 24.99979877471924 sec\nNum Annos 1637 (Tracks 44)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.882582  24.900000      126\n1      CAM_BACK_LEFT   5.020080          16.919276  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          82.887411  24.900000      126\n3          CAM_FRONT   5.020080          50.027272  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.542296  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.427920  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.096270  24.900000      126\n7          LIDAR_TOP   5.020156           0.000000  24.899626      126\n8           ego_pose  40.442375           0.000000  24.899626     1008\n9     sample (annos)   5.020156           0.000000  24.899626      126\n10       sample_data  40.280324          49.847878  24.999799     1008\n---\n---\nScene host-a004-lidar0-1233947108297817786-1233947133198765096 114b780b2efd6f73f134fc3a8f9db628e43131dc47f90e9b5dfdb886400d70f2\nStart 2019-02-11 19:05:08       Duration 25.000741004943848 sec\nNum Annos 4155 (Tracks 137)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.790201  24.900000      126\n1      CAM_BACK_LEFT   5.020080          17.030725  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          83.033195  24.900000      126\n3          CAM_FRONT   5.020080          50.151564  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.667718  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.649443  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.265691  24.900000      126\n7          LIDAR_TOP   5.019934           0.000000  24.900724      126\n8           ego_pose  40.440592           0.000000  24.900724     1008\n9     sample (annos)   5.019934           0.000000  24.900724      126\n10       sample_data  40.278806          49.948567  25.000741     1008\n---\n---\nScene host-a101-lidar0-1241886983298988182-1241887008198992182 7b4640d63a9c62d07a8551d4b430d0acd88eaba8249c843248feb888f4630070\nStart 2019-05-14 16:36:23       Duration 25.002139806747437 sec\nNum Annos 4777 (Tracks 173)\n               Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   5.020080          93.825297  24.900000      126\n1       CAM_BACK_LEFT   5.020080          85.205165  24.900000      126\n2      CAM_BACK_RIGHT   5.020080          19.099243  24.900000      126\n3           CAM_FRONT   5.020080          52.394347  24.900000      126\n4      CAM_FRONT_LEFT   5.020080          68.799780  24.900000      126\n5     CAM_FRONT_RIGHT   5.020080          35.769232  24.900000      126\n6    CAM_FRONT_ZOOMED   5.020080          52.394347  24.900000      126\n7    LIDAR_FRONT_LEFT   5.020060           0.000000  24.900101      126\n8   LIDAR_FRONT_RIGHT   5.020060           0.000000  24.900101      126\n9           LIDAR_TOP   5.020060           0.000000  24.900101      126\n10           ego_pose  50.562044           0.000000  24.900101     1260\n11     sample (annos)   5.020060           0.000000  24.900101      126\n12        sample_data  50.355690          40.748741  25.002140     1260\n```",
      "votes": null
    },
    {
      "id": "670307",
      "postDate": "11/11/2019 09:52:13",
      "content": "<p>The reason for better sync is \nThe Lyft system triggers a particular camera when LiDAR is executing a scan in its Field of View. </p>",
      "rawMarkdown": "The reason for better sync is \nThe Lyft system triggers a particular camera when LiDAR is executing a scan in its Field of View.",
      "votes": null
    },
    {
      "id": "670426",
      "postDate": "11/11/2019 13:02:00",
      "content": "<p>nuscenes isn’t a great baseline because the velodyne 64 gives the most points at 20hz and yet it probably wasn’t possible for them to record all the cameras at 20hz at the time the data was collected due to disk constraints.  if you look at argoverse or waymo open you also see good shutter-lidar sync.</p>\n\n<p>i wish lyft went with dual top lidars with 180 out-of-sync like argoverse did.  there’s a ton of drift for cars going 30-40mph if you only get scans at 10hz versus an effective 20hz.</p>\n\n<p>also, the test set lacks the side lidars, which is kinda sad :(</p>",
      "rawMarkdown": "nuscenes isn’t a great baseline because the velodyne 64 gives the most points at 20hz and yet it probably wasn’t possible for them to record all the cameras at 20hz at the time the data was collected due to disk constraints.  if you look at argoverse or waymo open you also see good shutter-lidar sync.\n\ni wish lyft went with dual top lidars with 180 out-of-sync like argoverse did.  there’s a ton of drift for cars going 30-40mph if you only get scans at 10hz versus an effective 20hz.\n\nalso, the test set lacks the side lidars, which is kinda sad :(",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 670307,
      "author_name": "",
      "author_url": "",
      "post_date": "11/11/2019 09:52:13",
      "content": "<p>The reason for better sync is \nThe Lyft system triggers a particular camera when LiDAR is executing a scan in its Field of View. </p>",
      "votes": null,
      "replies": [
        {
          "id": 670426,
          "author_name": "oarphme",
          "author_url": "",
          "post_date": "11/11/2019 13:02:00",
          "content": "<p>nuscenes isn’t a great baseline because the velodyne 64 gives the most points at 20hz and yet it probably wasn’t possible for them to record all the cameras at 20hz at the time the data was collected due to disk constraints.  if you look at argoverse or waymo open you also see good shutter-lidar sync.</p>\n\n<p>i wish lyft went with dual top lidars with 180 out-of-sync like argoverse did.  there’s a ton of drift for cars going 30-40mph if you only get scans at 10hz versus an effective 20hz.</p>\n\n<p>also, the test set lacks the side lidars, which is kinda sad :(</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "647603": "I was curious about the quality of the camera-lidar sync and how it compares to NuScenes ( https://www.nuscenes.org/ ).  I wrote a function that you can add as a member function to your `LyftDataset ` or `NuScenes` class (code: https://pastebin.com/HRuEFTjd ).\n\nMajor findings:\n  * Lyft Level 5 appears to *only* include labeled sensor data, with labels at 5Hz.  In contrast, NuScenes includes some unlabeled sensor data, with labels at 2Hz.  (Aside: I investigated interpolating NuScenes labels to the full sensor sample rate, and results looked qualitatively OK.  FWIW, the LyftDataset and NuScenes code appear to automatically project labels through the world frame, so I think you get motion correction of labels for free using either codebase).\n\n  * Lyft Level 5 lidars are synchronized to each other, but do not appear synchronized to any particular camera.  In particular, there appears to be a 50ms offset between lidar sweeps and the front camera.  In comparison, NuScenes single lidar is roughly synchronized with `CAM_BACK_LEFT`, and there's only a ~15ms offset to the front camera.  \n\n  * Oddly, Lyft Level 5 appears to have perfect lidar-localization sync (i.e. diff with `ego_pose`), but NuScenes has a consistent 11ms offset.  I'd assume that both Lyft Level 5 and NuScenes are using lidar-enhanced localization, so perhaps the NuScenes `ego_pose` timestamp is just wrong.\n\n  * NuScenes runs their Velodyne at full 20Hz, but only record cameras at about 12Hz.  \n\nSo it should be possible to convert NuScenes to a 5Hz (or even perhaps ~12Hz) dataset similar to Lyft Level 5.  Note that the Lyft lidars have substantially better coverage, especially for pedestrians and bikes.\n\nSample results, 3 scenes from each:\n```\nNuScenes:\n---\n---\nScene scene-1000 09f67057dd8346388b28f79d9bb1cf04\nStart 2018-11-14 11:01:41       Duration 19.922956943511963 sec\nNum Annos 493 (Tracks 27)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.738035          10.554540  19.850000      234\n1       CAM_BACK_LEFT   11.536524           0.918133  19.850000      230\n2      CAM_BACK_RIGHT   11.788413          20.070969  19.850000      235\n3           CAM_FRONT   11.637280          14.986247  19.850000      232\n4      CAM_FRONT_LEFT   11.536524           7.586523  19.850000      230\n5     CAM_FRONT_RIGHT   11.536524          22.528135  19.850000      230\n6           LIDAR_TOP   19.747937           0.000000  19.850175      393\n7     RADAR_BACK_LEFT   12.593898          12.635898  19.850883      251\n8    RADAR_BACK_RIGHT   13.211602          12.786931  19.831054      263\n9         RADAR_FRONT   12.976290          12.728528  19.882416      259\n10   RADAR_FRONT_LEFT   13.510020          12.710849  19.911148      270\n11  RADAR_FRONT_RIGHT   13.724216          12.571387  19.891846      274\n12           ego_pose  155.599393          11.128198  19.922957     3101\n13     sample (annos)    2.015096           0.000000  19.850175       41\n14        sample_data  155.599393          11.128198  19.922957     3101\n---\n---\nScene scene-0293 6308d6d934074a028fc3145eedf3e65f\nStart 2018-08-31 15:25:42       Duration 19.525898933410645 sec\nNum Annos 3548 (Tracks 277)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.773779          10.441362  19.450000      230\n1       CAM_BACK_LEFT   11.825193           1.573582  19.450000      231\n2      CAM_BACK_RIGHT   11.928021          19.624993  19.450000      233\n3           CAM_FRONT   11.876607          14.872485  19.450000      232\n4      CAM_FRONT_LEFT   11.876607           7.329719  19.450000      232\n5     CAM_FRONT_RIGHT   11.979434          22.875966  19.450000      234\n6           LIDAR_TOP   19.844216           0.000000  19.451512      387\n7     RADAR_BACK_LEFT   13.157897          12.698666  19.455997      257\n8    RADAR_BACK_RIGHT   13.490288          12.647669  19.421380      263\n9         RADAR_FRONT   13.082394          12.616018  19.491845      256\n10   RADAR_FRONT_LEFT   13.305139          12.709008  19.466163      260\n11  RADAR_FRONT_RIGHT   13.209300          12.723777  19.455989      258\n12           ego_pose  157.329504          11.144872  19.525899     3073\n13     sample (annos)    2.004986           0.000000  19.451512       40\n14        sample_data  157.329504          11.144872  19.525899     3073\n---\n---\nScene scene-1107 89f20737ec344aa48b543a9e005a38ca\nStart 2018-11-21 11:59:53       Duration 19.820924997329712 sec\nNum Annos 496 (Tracks 47)\n               Series     Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   11.696203          11.014590  19.750000      232\n1       CAM_BACK_LEFT   11.848101           1.382666  19.750000      235\n2      CAM_BACK_RIGHT   11.746835          20.483792  19.750000      233\n3           CAM_FRONT   11.848103          14.481831  19.749997      235\n4      CAM_FRONT_LEFT   11.898734           7.063236  19.750000      236\n5     CAM_FRONT_RIGHT   11.898734          22.159666  19.750000      236\n6           LIDAR_TOP   19.797377           0.000000  19.750091      392\n7     RADAR_BACK_LEFT   13.578124          12.646209  19.811279      270\n8    RADAR_BACK_RIGHT   13.271911          12.721395  19.740940      263\n9         RADAR_FRONT   13.198245          12.553021  19.775357      262\n10   RADAR_FRONT_LEFT   13.730931          12.645984  19.736462      272\n11  RADAR_FRONT_RIGHT   13.778595          12.488227  19.740764      273\n12           ego_pose  158.317536          11.102567  19.820925     3139\n13     sample (annos)    2.025307           0.000000  19.750091       41\n14        sample_data  158.317536          11.102567  19.820925     3139\n\nLyft Level 5:\n---\n---\nScene host-a015-lidar0-1235423635198474636-1235423660098038666 755e4564756ad5c92243b7f77039d07ab1cce40662a6a19b67c820647666a3ef\nStart 2019-02-28 21:13:55       Duration 24.99979877471924 sec\nNum Annos 1637 (Tracks 44)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.882582  24.900000      126\n1      CAM_BACK_LEFT   5.020080          16.919276  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          82.887411  24.900000      126\n3          CAM_FRONT   5.020080          50.027272  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.542296  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.427920  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.096270  24.900000      126\n7          LIDAR_TOP   5.020156           0.000000  24.899626      126\n8           ego_pose  40.442375           0.000000  24.899626     1008\n9     sample (annos)   5.020156           0.000000  24.899626      126\n10       sample_data  40.280324          49.847878  24.999799     1008\n---\n---\nScene host-a004-lidar0-1233947108297817786-1233947133198765096 114b780b2efd6f73f134fc3a8f9db628e43131dc47f90e9b5dfdb886400d70f2\nStart 2019-02-11 19:05:08       Duration 25.000741004943848 sec\nNum Annos 4155 (Tracks 137)\n              Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0           CAM_BACK   5.020080          98.790201  24.900000      126\n1      CAM_BACK_LEFT   5.020080          17.030725  24.900000      126\n2     CAM_BACK_RIGHT   5.020080          83.033195  24.900000      126\n3          CAM_FRONT   5.020080          50.151564  24.900000      126\n4     CAM_FRONT_LEFT   5.020080          33.667718  24.900000      126\n5    CAM_FRONT_RIGHT   5.020080          66.649443  24.900000      126\n6   CAM_FRONT_ZOOMED   5.020080          50.265691  24.900000      126\n7          LIDAR_TOP   5.019934           0.000000  24.900724      126\n8           ego_pose  40.440592           0.000000  24.900724     1008\n9     sample (annos)   5.019934           0.000000  24.900724      126\n10       sample_data  40.278806          49.948567  25.000741     1008\n---\n---\nScene host-a101-lidar0-1241886983298988182-1241887008198992182 7b4640d63a9c62d07a8551d4b430d0acd88eaba8249c843248feb888f4630070\nStart 2019-05-14 16:36:23       Duration 25.002139806747437 sec\nNum Annos 4777 (Tracks 173)\n               Series    Freq Hz  Diff Lidar (msec)   Duration  Support\n0            CAM_BACK   5.020080          93.825297  24.900000      126\n1       CAM_BACK_LEFT   5.020080          85.205165  24.900000      126\n2      CAM_BACK_RIGHT   5.020080          19.099243  24.900000      126\n3           CAM_FRONT   5.020080          52.394347  24.900000      126\n4      CAM_FRONT_LEFT   5.020080          68.799780  24.900000      126\n5     CAM_FRONT_RIGHT   5.020080          35.769232  24.900000      126\n6    CAM_FRONT_ZOOMED   5.020080          52.394347  24.900000      126\n7    LIDAR_FRONT_LEFT   5.020060           0.000000  24.900101      126\n8   LIDAR_FRONT_RIGHT   5.020060           0.000000  24.900101      126\n9           LIDAR_TOP   5.020060           0.000000  24.900101      126\n10           ego_pose  50.562044           0.000000  24.900101     1260\n11     sample (annos)   5.020060           0.000000  24.900101      126\n12        sample_data  50.355690          40.748741  25.002140     1260\n```",
    "670307": "The reason for better sync is \nThe Lyft system triggers a particular camera when LiDAR is executing a scan in its Field of View.",
    "670426": "nuscenes isn’t a great baseline because the velodyne 64 gives the most points at 20hz and yet it probably wasn’t possible for them to record all the cameras at 20hz at the time the data was collected due to disk constraints.  if you look at argoverse or waymo open you also see good shutter-lidar sync.\n\ni wish lyft went with dual top lidars with 180 out-of-sync like argoverse did.  there’s a ton of drift for cars going 30-40mph if you only get scans at 10hz versus an effective 20hz.\n\nalso, the test set lacks the side lidars, which is kinda sad :("
  },
  "source": "meta"
}