{
  "id": 199498,
  "title": "Looking into the future",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/199498",
  "author_name": "",
  "post_date": "2020-11-26T00:34:44.671243100Z",
  "votes": 9,
  "comment_count": 7,
  "views": 0,
  "content": "<p>One thing that can be exploited in this competition is that the first frame of the next scene is only 15 seconds away.</p>\n<p>The images below is an example. In each image I combined several channels for demonstration. The left picture is the latest frame for which we need to make a prediction. In my rasterization I mark the AV with a black hole in the middle, you can see it on the pictures. The red car is the agent for which we need to predict. It is standing just in front.</p>\n<p>The right picture is the first frame of the next scene, 15 seconds away. <code>87%</code> of the test scenes have another scene following it immediately without a time delay. A green line on the right picture indicates what path the AV has made, it points from its location on the left image to the future location. </p>\n<p>One can see that in 15 seconds the cars have moved only slightly. Moreover, we can guess where is the agent that we predict on the right picture.</p>\n<p><img src=\"https://imgur.com/38xK2oS.png\" alt=\"seen into the future\"></p>\n<p>This example is cherry-picked, in the majority of cases the AV has already moved too far, or just does not give any new information for other reasons. My best model validation gives score <code>12.02</code> with this additional future frame, and <code>12.14</code> without it. So I was not able to fully exploit it, but my guess is that one can get much more from it. </p>",
  "messages": [
    {
      "id": "1091348",
      "postDate": "11/26/2020 00:34:44",
      "content": "<p>One thing that can be exploited in this competition is that the first frame of the next scene is only 15 seconds away.</p>\n<p>The images below is an example. In each image I combined several channels for demonstration. The left picture is the latest frame for which we need to make a prediction. In my rasterization I mark the AV with a black hole in the middle, you can see it on the pictures. The red car is the agent for which we need to predict. It is standing just in front.</p>\n<p>The right picture is the first frame of the next scene, 15 seconds away. <code>87%</code> of the test scenes have another scene following it immediately without a time delay. A green line on the right picture indicates what path the AV has made, it points from its location on the left image to the future location. </p>\n<p>One can see that in 15 seconds the cars have moved only slightly. Moreover, we can guess where is the agent that we predict on the right picture.</p>\n<p><img src=\"https://imgur.com/38xK2oS.png\" alt=\"seen into the future\"></p>\n<p>This example is cherry-picked, in the majority of cases the AV has already moved too far, or just does not give any new information for other reasons. My best model validation gives score <code>12.02</code> with this additional future frame, and <code>12.14</code> without it. So I was not able to fully exploit it, but my guess is that one can get much more from it. </p>",
      "rawMarkdown": "One thing that can be exploited in this competition is that the first frame of the next scene is only 15 seconds away.\n\nThe images below is an example. In each image I combined several channels for demonstration. The left picture is the latest frame for which we need to make a prediction. In my rasterization I mark the AV with a black hole in the middle, you can see it on the pictures. The red car is the agent for which we need to predict. It is standing just in front.\n\nThe right picture is the first frame of the next scene, 15 seconds away. `87%` of the test scenes have another scene following it immediately without a time delay. A green line on the right picture indicates what path the AV has made, it points from its location on the left image to the future location. \n\nOne can see that in 15 seconds the cars have moved only slightly. Moreover, we can guess where is the agent that we predict on the right picture.\n\n![seen into the future](https://imgur.com/38xK2oS.png)\n\nThis example is cherry-picked, in the majority of cases the AV has already moved too far, or just does not give any new information for other reasons. My best model validation gives score `12.02` with this additional future frame, and `12.14` without it. So I was not able to fully exploit it, but my guess is that one can get much more from it.",
      "votes": null
    },
    {
      "id": "1091352",
      "postDate": "11/26/2020 00:37:30",
      "content": "<p>in theory, you can train a NN to predict the original dataset from chopped dataset.<br>\nthen you would have all the continuous trajectory</p>",
      "rawMarkdown": "in theory, you can train a NN to predict the original dataset from chopped dataset.\nthen you would have all the continuous trajectory",
      "votes": null
    },
    {
      "id": "1091357",
      "postDate": "11/26/2020 00:44:40",
      "content": "<p>Sorry, I miss your point. Can you please elaborate?</p>",
      "rawMarkdown": "Sorry, I miss your point. Can you please elaborate?",
      "votes": null
    },
    {
      "id": "1091364",
      "postDate": "11/26/2020 00:52:01",
      "content": "<p>i think we start withe the orginal zarr array. when making chopped dataset some of the data are chopped, or masked zero (you have to read the doc or code to see how it chop the data).</p>\n<p>to predict the original array data from the masked version, you can think of it as inpainting or masked token in BERT</p>",
      "rawMarkdown": "i think we start withe the orginal zarr array. when making chopped dataset some of the data are chopped, or masked zero (you have to read the doc or code to see how it chop the data).\n\nto predict the original array data from the masked version, you can think of it as inpainting or masked token in BERT",
      "votes": null
    },
    {
      "id": "1091370",
      "postDate": "11/26/2020 00:58:47",
      "content": "<p>Oh, yeah, I got you. Yes, your approach is similar in spirit.</p>",
      "rawMarkdown": "Oh, yeah, I got you. Yes, your approach is similar in spirit.",
      "votes": null
    },
    {
      "id": "1091539",
      "postDate": "11/26/2020 05:14:52",
      "content": "<p>Can you share the code how you plotted that raster image</p>",
      "rawMarkdown": "Can you share the code how you plotted that raster image",
      "votes": null
    },
    {
      "id": "1091825",
      "postDate": "11/26/2020 10:13:30",
      "content": "<p>Sure, to create this specific image I used this code</p>\n<pre><code>fig, axs = plt.subplots(1, 2, figsize=(15,7))\nlanes = data['image'][25:28]\nim1 = lanes.copy()\nim1[0] = np.maximum(im1[0], data['image'][12])\nim1[1] = np.maximum(im1[1], data['image'][0])\naxs[0].imshow(im1.transpose((1,2,0))[::-1])\naxs[0].set_title('time 0s, frame 99')\nim2 = lanes.copy()\nim2[1] = np.maximum(im2[0], data['image'][24])\naxs[1].imshow(im2.transpose((1,2,0))[::-1])\naxs[1].set_title('time 15s, frame 0 of the next scene') \n</code></pre>\n<p>but the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.</p>",
      "rawMarkdown": "Sure, to create this specific image I used this code\n\n```\nfig, axs = plt.subplots(1, 2, figsize=(15,7))\nlanes = data['image'][25:28]\nim1 = lanes.copy()\nim1[0] = np.maximum(im1[0], data['image'][12])\nim1[1] = np.maximum(im1[1], data['image'][0])\naxs[0].imshow(im1.transpose((1,2,0))[::-1])\naxs[0].set_title('time 0s, frame 99')\nim2 = lanes.copy()\nim2[1] = np.maximum(im2[0], data['image'][24])\naxs[1].imshow(im2.transpose((1,2,0))[::-1])\naxs[1].set_title('time 15s, frame 0 of the next scene') \n```\n\nbut the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.",
      "votes": null
    },
    {
      "id": "1091837",
      "postDate": "11/26/2020 10:29:14",
      "content": "<blockquote>\n  <p>but the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.</p>\n</blockquote>\n<p>You mean, is it optimized version</p>",
      "rawMarkdown": "> but the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.\n\nYou mean, is it optimized version",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1091352,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "11/26/2020 00:37:30",
      "content": "<p>in theory, you can train a NN to predict the original dataset from chopped dataset.<br>\nthen you would have all the continuous trajectory</p>",
      "votes": null,
      "replies": [
        {
          "id": 1091357,
          "author_name": "zaharch",
          "author_url": "",
          "post_date": "11/26/2020 00:44:40",
          "content": "<p>Sorry, I miss your point. Can you please elaborate?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1091364,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "11/26/2020 00:52:01",
          "content": "<p>i think we start withe the orginal zarr array. when making chopped dataset some of the data are chopped, or masked zero (you have to read the doc or code to see how it chop the data).</p>\n<p>to predict the original array data from the masked version, you can think of it as inpainting or masked token in BERT</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1091370,
          "author_name": "zaharch",
          "author_url": "",
          "post_date": "11/26/2020 00:58:47",
          "content": "<p>Oh, yeah, I got you. Yes, your approach is similar in spirit.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1091539,
      "author_name": "morizin",
      "author_url": "",
      "post_date": "11/26/2020 05:14:52",
      "content": "<p>Can you share the code how you plotted that raster image</p>",
      "votes": null,
      "replies": [
        {
          "id": 1091825,
          "author_name": "zaharch",
          "author_url": "",
          "post_date": "11/26/2020 10:13:30",
          "content": "<p>Sure, to create this specific image I used this code</p>\n<pre><code>fig, axs = plt.subplots(1, 2, figsize=(15,7))\nlanes = data['image'][25:28]\nim1 = lanes.copy()\nim1[0] = np.maximum(im1[0], data['image'][12])\nim1[1] = np.maximum(im1[1], data['image'][0])\naxs[0].imshow(im1.transpose((1,2,0))[::-1])\naxs[0].set_title('time 0s, frame 99')\nim2 = lanes.copy()\nim2[1] = np.maximum(im2[0], data['image'][24])\naxs[1].imshow(im2.transpose((1,2,0))[::-1])\naxs[1].set_title('time 15s, frame 0 of the next scene') \n</code></pre>\n<p>but the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1091837,
          "author_name": "morizin",
          "author_url": "",
          "post_date": "11/26/2020 10:29:14",
          "content": "<blockquote>\n  <p>but the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.</p>\n</blockquote>\n<p>You mean, is it optimized version</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1091348": "One thing that can be exploited in this competition is that the first frame of the next scene is only 15 seconds away.\n\nThe images below is an example. In each image I combined several channels for demonstration. The left picture is the latest frame for which we need to make a prediction. In my rasterization I mark the AV with a black hole in the middle, you can see it on the pictures. The red car is the agent for which we need to predict. It is standing just in front.\n\nThe right picture is the first frame of the next scene, 15 seconds away. `87%` of the test scenes have another scene following it immediately without a time delay. A green line on the right picture indicates what path the AV has made, it points from its location on the left image to the future location. \n\nOne can see that in 15 seconds the cars have moved only slightly. Moreover, we can guess where is the agent that we predict on the right picture.\n\n![seen into the future](https://imgur.com/38xK2oS.png)\n\nThis example is cherry-picked, in the majority of cases the AV has already moved too far, or just does not give any new information for other reasons. My best model validation gives score `12.02` with this additional future frame, and `12.14` without it. So I was not able to fully exploit it, but my guess is that one can get much more from it.",
    "1091352": "in theory, you can train a NN to predict the original dataset from chopped dataset.\nthen you would have all the continuous trajectory",
    "1091357": "Sorry, I miss your point. Can you please elaborate?",
    "1091364": "i think we start withe the orginal zarr array. when making chopped dataset some of the data are chopped, or masked zero (you have to read the doc or code to see how it chop the data).\n\nto predict the original array data from the masked version, you can think of it as inpainting or masked token in BERT",
    "1091370": "Oh, yeah, I got you. Yes, your approach is similar in spirit.",
    "1091539": "Can you share the code how you plotted that raster image",
    "1091825": "Sure, to create this specific image I used this code\n\n```\nfig, axs = plt.subplots(1, 2, figsize=(15,7))\nlanes = data['image'][25:28]\nim1 = lanes.copy()\nim1[0] = np.maximum(im1[0], data['image'][12])\nim1[1] = np.maximum(im1[1], data['image'][0])\naxs[0].imshow(im1.transpose((1,2,0))[::-1])\naxs[0].set_title('time 0s, frame 99')\nim2 = lanes.copy()\nim2[1] = np.maximum(im2[0], data['image'][24])\naxs[1].imshow(im2.transpose((1,2,0))[::-1])\naxs[1].set_title('time 15s, frame 0 of the next scene') \n```\n\nbut the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.",
    "1091837": "> but the hole in AV and the line are changes that I made in l5kit package, so not that easy to share.\n\nYou mean, is it optimized version"
  },
  "source": "meta"
}