{
  "id": 192458,
  "title": "Is there any success in using solely Kaggle/Colab for training?",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/192458",
  "author_name": "Jiayu",
  "post_date": "2020-10-21T15:38:45.151000",
  "votes": 5,
  "comment_count": 17,
  "views": 0,
  "content": "<p>Or some more powerful CPU (either desktop or GCE) is definitely needed?</p>\n<p>I can only get down to 3% batches within an epoch (around 5000-10000 steps) before notebook times out. I wonder if that's some approach I should give up sooner.</p>\n<p>Thanks!</p>",
  "messages": [
    {
      "id": 1056326,
      "postDate": "2020-10-21T15:38:45.150Z",
      "content": "<p>Or some more powerful CPU (either desktop or GCE) is definitely needed?</p>\n<p>I can only get down to 3% batches within an epoch (around 5000-10000 steps) before notebook times out. I wonder if that's some approach I should give up sooner.</p>\n<p>Thanks!</p>",
      "rawMarkdown": "Or some more powerful CPU (either desktop or GCE) is definitely needed?\n\nI can only get down to 3% batches within an epoch (around 5000-10000 steps) before notebook times out. I wonder if that's some approach I should give up sooner.\n\nThanks!",
      "votes": 5
    },
    {
      "id": 1059283,
      "postDate": "2020-10-24T21:22:19.163Z",
      "content": "<p>i am training on Colab Pro. I don't have own GPU so Colab Pro is nice choice</p>",
      "rawMarkdown": "i am training on Colab Pro. I don't have own GPU so Colab Pro is nice choice",
      "votes": 2,
      "replies": [
        {
          "id": 1059345,
          "postDate": "2020-10-25T00:38:15.657Z",
          "content": "<p>Same with me. How long it took for you train.zarr <a href=\"https://www.kaggle.com/doanquanvietnamca\" target=\"_blank\">@doanquanvietnamca</a> ?</p>",
          "rawMarkdown": "Same with me. How long it took for you train.zarr @doanquanvietnamca ?"
        },
        {
          "id": 1060635,
          "postDate": "2020-10-26T12:24:38.920Z",
          "content": "<p>How can you load that huge train.zarr in Colab? Even I use Colab pro, I still can not load train.zarr from Google Drive. The only way is to download the dataset on the VM locally. But if I disconnect, I have to download the dataset again, which literally killing me…  </p>",
          "rawMarkdown": "How can you load that huge train.zarr in Colab? Even I use Colab pro, I still can not load train.zarr from Google Drive. The only way is to download the dataset on the VM locally. But if I disconnect, I have to download the dataset again, which literally killing me...  "
        },
        {
          "id": 1060979,
          "postDate": "2020-10-26T17:05:58.997Z",
          "content": "<p>I use kaggle API to download dataset<br>\nI also try to download using GCS, but download speed was slow in my case.</p>\n<p><a href=\"https://www.kaggle.com/docs/api\" target=\"_blank\">https://www.kaggle.com/docs/api</a></p>\n<pre><code>from google.colab import files\nfiles.upload() #upload your kaggle.json\n</code></pre>\n<pre><code>!pip uninstall -y kaggle -q\n!pip install --upgrade pip -q\n!pip install -q kaggle==1.5.6\n!mkdir -p ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!ls ~/.kaggle\n!chmod 600 /root/.kaggle/kaggle.json\n</code></pre>\n<pre><code>!kaggle competitions download -c lyft-motion-prediction-autonomous-vehicles -p ./dataset -q # download to ./dataset\n</code></pre>\n<pre><code>!unzip -q ./dataset/lyft-motion-prediction-autonomous-vehicles.zip -d ./dataset\n</code></pre>",
          "rawMarkdown": "I use kaggle API to download dataset\nI also try to download using GCS, but download speed was slow in my case.\n\n[https://www.kaggle.com/docs/api](https://www.kaggle.com/docs/api)\n\n\n```\nfrom google.colab import files\nfiles.upload() #upload your kaggle.json\n```\n\n```\n!pip uninstall -y kaggle -q\n!pip install --upgrade pip -q\n!pip install -q kaggle==1.5.6\n!mkdir -p ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!ls ~/.kaggle\n!chmod 600 /root/.kaggle/kaggle.json\n```\n```\n!kaggle competitions download -c lyft-motion-prediction-autonomous-vehicles -p ./dataset -q # download to ./dataset\n```\n\n```\n!unzip -q ./dataset/lyft-motion-prediction-autonomous-vehicles.zip -d ./dataset\n```"
        },
        {
          "id": 1061051,
          "postDate": "2020-10-26T17:59:12.077Z",
          "content": "<p>i use the same method. It took 20min to download but you can train it 24hours if not disconnect. Colab Pro is good choice in this competition. I got this position with colab pro, train 3-4days</p>",
          "rawMarkdown": "i use the same method. It took 20min to download but you can train it 24hours if not disconnect. Colab Pro is good choice in this competition. I got this position with colab pro, train 3-4days",
          "votes": 1
        },
        {
          "id": 1061617,
          "postDate": "2020-10-27T06:25:10.697Z",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/doanquanvietnamca\" target=\"_blank\">@doanquanvietnamca</a>, the Colab pro page says that - \"With Colab Pro your notebooks can stay connected for up to 24 hours\" as you said as well. Does this mean that it will get disconnected after 24hrs even if my training is on? </p>",
          "rawMarkdown": "Hi @doanquanvietnamca, the Colab pro page says that - \"With Colab Pro your notebooks can stay connected for up to 24 hours\" as you said as well. Does this mean that it will get disconnected after 24hrs even if my training is on? "
        },
        {
          "id": 1061760,
          "postDate": "2020-10-27T09:54:58.827Z",
          "content": "<p>yeap. You will get new session and all data and training checkpoint delete</p>",
          "rawMarkdown": "yeap. You will get new session and all data and training checkpoint delete\n",
          "votes": 1
        },
        {
          "id": 1061767,
          "postDate": "2020-10-27T10:04:28.410Z",
          "content": "<p>So how did you train for 3-4 days on colab pro as you mentioned in your previous comment <a href=\"https://www.kaggle.com/doanquanvietnamca\" target=\"_blank\">@doanquanvietnamca</a> ?</p>",
          "rawMarkdown": "So how did you train for 3-4 days on colab pro as you mentioned in your previous comment @doanquanvietnamca ?"
        },
        {
          "id": 1061899,
          "postDate": "2020-10-27T12:21:02.977Z",
          "content": "<p>spectacular endurance!</p>",
          "rawMarkdown": "spectacular endurance!"
        },
        {
          "id": 1062147,
          "postDate": "2020-10-27T15:51:39.587Z",
          "content": "<p><a href=\"https://www.kaggle.com/arpitrf\" target=\"_blank\">@arpitrf</a> check the discussion thread <a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/192458\" target=\"_blank\">here</a> and the notebook <a href=\"https://www.kaggle.com/kool777/ultimate-google-colab-training-batch-size-64\" target=\"_blank\">here</a> the walk through by <a href=\"https://www.kaggle.com/kool777\" target=\"_blank\">@kool777</a> </p>",
          "rawMarkdown": "@arpitrf check the discussion thread [here](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/192458) and the notebook [here](https://www.kaggle.com/kool777/ultimate-google-colab-training-batch-size-64) the walk through by @kool777 "
        },
        {
          "id": 1062229,
          "postDate": "2020-10-27T17:13:15.857Z",
          "content": "<p>save checkpoint :)</p>",
          "rawMarkdown": "save checkpoint :)",
          "votes": 3
        },
        {
          "id": 1063559,
          "postDate": "2020-10-29T04:18:16.700Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1057112,
      "postDate": "2020-10-22T11:32:52.300Z",
      "content": "<p>I did some experiments with Kaggle and Colab Pro.<br>\nAs you know, on Colab Pro you can choose high-memory  VM.</p>\n<p>Colab Pro normal VM for GPU: CPU cores = 2<br>\nColab Pro normal VM for TPU: CPU cores = 2, TPU cores = 8</p>\n<p>Colab Pro high-memory VM for GPU: CPU cores = 4<br>\nColab Pro high-memory VM for TPU: CPU cores = 40, TPU cores = 8</p>\n<p>For example, with batch size=16<br>\nAssume it will take 1 iteration / second for training on Kaggle.</p>\n<p>Colab Pro high-memory VM for GPU: 2.7 it/s<br>\nColab Pro high-memory VM for TPU with Pytorch XLA<br>\nsingle core of TPU: ~0.7 it/s<br>\n8 cores of TPU: 1.2 it/s</p>\n<p>So if you train on colab pro high-memory vm of gpu, you may shorten train time x2.7 than training on kaggle.</p>",
      "rawMarkdown": "I did some experiments with Kaggle and Colab Pro.\nAs you know, on Colab Pro you can choose high-memory  VM.\n\nColab Pro normal VM for GPU: CPU cores = 2\nColab Pro normal VM for TPU: CPU cores = 2, TPU cores = 8\n\nColab Pro high-memory VM for GPU: CPU cores = 4\nColab Pro high-memory VM for TPU: CPU cores = 40, TPU cores = 8\n\nFor example, with batch size=16\nAssume it will take 1 iteration / second for training on Kaggle.\n\nColab Pro high-memory VM for GPU: 2.7 it/s\nColab Pro high-memory VM for TPU with Pytorch XLA\nsingle core of TPU: ~0.7 it/s\n8 cores of TPU: 1.2 it/s\n\nSo if you train on colab pro high-memory vm of gpu, you may shorten train time x2.7 than training on kaggle.",
      "votes": 2,
      "replies": [
        {
          "id": 1059302,
          "postDate": "2020-10-24T21:48:48.493Z",
          "content": "<p>Did you also try TPU with TensorFlow? PyTorch XLA is not well optimized still, unfortunately.</p>",
          "rawMarkdown": "Did you also try TPU with TensorFlow? PyTorch XLA is not well optimized still, unfortunately."
        },
        {
          "id": 1059342,
          "postDate": "2020-10-25T00:23:28.323Z",
          "content": "<p>At first I had plan to train on TPU with TensorFlow. But I couldn't try. Because making TFRecord takes too long time. I think it is hard to do experiment with different image size.</p>",
          "rawMarkdown": "At first I had plan to train on TPU with TensorFlow. But I couldn't try. Because making TFRecord takes too long time. I think it is hard to do experiment with different image size."
        }
      ]
    },
    {
      "id": 1071749,
      "postDate": "2020-11-07T11:15:39.510Z",
      "content": "<p>I was able to train 1,600,000 datapoints (of ~22.5M) (100,000 iterations of batch size 16) in 38-40 hours on Colab pro. It seems to me that it took unusually long time for this amount of data! Is that so? How many samples/datapoints are you guys being able to train in around this much amount of time?</p>\n<p>I had the following configurations - </p>\n<pre><code>cfg = {\n    'format_version': 4,\n    'model_params': {\n        'history_num_frames': 2,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1,\n        'model_name': \"model_resnet34_output\",\n        'weight_path': None\n    },\n\n    'raster_params': {\n        'raster_size': [667, 300],\n        'pixel_size': [0.2, 0.2],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n  'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 16,\n        'shuffle': True,\n        'num_workers': 3\n    },\n}\n</code></pre>",
      "rawMarkdown": "I was able to train 1,600,000 datapoints (of ~22.5M) (100,000 iterations of batch size 16) in 38-40 hours on Colab pro. It seems to me that it took unusually long time for this amount of data! Is that so? How many samples/datapoints are you guys being able to train in around this much amount of time?\n\nI had the following configurations - \n```\ncfg = {\n    'format_version': 4,\n    'model_params': {\n        'history_num_frames': 2,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1,\n        'model_name': \"model_resnet34_output\",\n        'weight_path': None\n    },\n    \n    'raster_params': {\n        'raster_size': [667, 300],\n        'pixel_size': [0.2, 0.2],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n  'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 16,\n        'shuffle': True,\n        'num_workers': 3\n    },\n}\n```"
    },
    {
      "id": 1056467,
      "postDate": "2020-10-21T18:19:22Z",
      "content": "<p>Try incremental training by running through kaggle notebooks multiple times. You can find some ideas here:<br>\n<a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/189996\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/189996</a></p>",
      "rawMarkdown": "Try incremental training by running through kaggle notebooks multiple times. You can find some ideas here:\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/189996"
    }
  ],
  "comments": [
    {
      "id": 1059283,
      "author_name": "Manh Lab",
      "author_url": "",
      "post_date": "2020-10-24T21:22:19.163000",
      "content": "<p>i am training on Colab Pro. I don't have own GPU so Colab Pro is nice choice</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1059345,
          "author_name": "SeshuRaju 🧘‍♂️",
          "author_url": "",
          "post_date": "2020-10-25T00:38:15.657000",
          "content": "<p>Same with me. How long it took for you train.zarr <a href=\"https://www.kaggle.com/doanquanvietnamca\" target=\"_blank\">@doanquanvietnamca</a> ?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1060635,
          "author_name": "Yannik",
          "author_url": "",
          "post_date": "2020-10-26T12:24:38.920000",
          "content": "<p>How can you load that huge train.zarr in Colab? Even I use Colab pro, I still can not load train.zarr from Google Drive. The only way is to download the dataset on the VM locally. But if I disconnect, I have to download the dataset again, which literally killing me…  </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1060979,
          "author_name": "Sunghyun Jun",
          "author_url": "",
          "post_date": "2020-10-26T17:05:58.997000",
          "content": "<p>I use kaggle API to download dataset<br>\nI also try to download using GCS, but download speed was slow in my case.</p>\n<p><a href=\"https://www.kaggle.com/docs/api\" target=\"_blank\">https://www.kaggle.com/docs/api</a></p>\n<pre><code>from google.colab import files\nfiles.upload() #upload your kaggle.json\n</code></pre>\n<pre><code>!pip uninstall -y kaggle -q\n!pip install --upgrade pip -q\n!pip install -q kaggle==1.5.6\n!mkdir -p ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!ls ~/.kaggle\n!chmod 600 /root/.kaggle/kaggle.json\n</code></pre>\n<pre><code>!kaggle competitions download -c lyft-motion-prediction-autonomous-vehicles -p ./dataset -q # download to ./dataset\n</code></pre>\n<pre><code>!unzip -q ./dataset/lyft-motion-prediction-autonomous-vehicles.zip -d ./dataset\n</code></pre>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1061051,
          "author_name": "Manh Lab",
          "author_url": "",
          "post_date": "2020-10-26T17:59:12.077000",
          "content": "<p>i use the same method. It took 20min to download but you can train it 24hours if not disconnect. Colab Pro is good choice in this competition. I got this position with colab pro, train 3-4days</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1061617,
          "author_name": "Arpit",
          "author_url": "",
          "post_date": "2020-10-27T06:25:10.697000",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/doanquanvietnamca\" target=\"_blank\">@doanquanvietnamca</a>, the Colab pro page says that - \"With Colab Pro your notebooks can stay connected for up to 24 hours\" as you said as well. Does this mean that it will get disconnected after 24hrs even if my training is on? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1061760,
          "author_name": "Manh Lab",
          "author_url": "",
          "post_date": "2020-10-27T09:54:58.827000",
          "content": "<p>yeap. You will get new session and all data and training checkpoint delete</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1061767,
          "author_name": "Arpit",
          "author_url": "",
          "post_date": "2020-10-27T10:04:28.410000",
          "content": "<p>So how did you train for 3-4 days on colab pro as you mentioned in your previous comment <a href=\"https://www.kaggle.com/doanquanvietnamca\" target=\"_blank\">@doanquanvietnamca</a> ?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1061899,
          "author_name": "Yannik",
          "author_url": "",
          "post_date": "2020-10-27T12:21:02.977000",
          "content": "<p>spectacular endurance!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1062147,
          "author_name": "Akash",
          "author_url": "",
          "post_date": "2020-10-27T15:51:39.587000",
          "content": "<p><a href=\"https://www.kaggle.com/arpitrf\" target=\"_blank\">@arpitrf</a> check the discussion thread <a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/192458\" target=\"_blank\">here</a> and the notebook <a href=\"https://www.kaggle.com/kool777/ultimate-google-colab-training-batch-size-64\" target=\"_blank\">here</a> the walk through by <a href=\"https://www.kaggle.com/kool777\" target=\"_blank\">@kool777</a> </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1062229,
          "author_name": "Manh Lab",
          "author_url": "",
          "post_date": "2020-10-27T17:13:15.857000",
          "content": "<p>save checkpoint :)</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1063559,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-10-29T04:18:16.700000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1057112,
      "author_name": "Sunghyun Jun",
      "author_url": "",
      "post_date": "2020-10-22T11:32:52.300000",
      "content": "<p>I did some experiments with Kaggle and Colab Pro.<br>\nAs you know, on Colab Pro you can choose high-memory  VM.</p>\n<p>Colab Pro normal VM for GPU: CPU cores = 2<br>\nColab Pro normal VM for TPU: CPU cores = 2, TPU cores = 8</p>\n<p>Colab Pro high-memory VM for GPU: CPU cores = 4<br>\nColab Pro high-memory VM for TPU: CPU cores = 40, TPU cores = 8</p>\n<p>For example, with batch size=16<br>\nAssume it will take 1 iteration / second for training on Kaggle.</p>\n<p>Colab Pro high-memory VM for GPU: 2.7 it/s<br>\nColab Pro high-memory VM for TPU with Pytorch XLA<br>\nsingle core of TPU: ~0.7 it/s<br>\n8 cores of TPU: 1.2 it/s</p>\n<p>So if you train on colab pro high-memory vm of gpu, you may shorten train time x2.7 than training on kaggle.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1059302,
          "author_name": "Ali Abdin",
          "author_url": "",
          "post_date": "2020-10-24T21:48:48.493000",
          "content": "<p>Did you also try TPU with TensorFlow? PyTorch XLA is not well optimized still, unfortunately.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1059342,
          "author_name": "Sunghyun Jun",
          "author_url": "",
          "post_date": "2020-10-25T00:23:28.323000",
          "content": "<p>At first I had plan to train on TPU with TensorFlow. But I couldn't try. Because making TFRecord takes too long time. I think it is hard to do experiment with different image size.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1071749,
      "author_name": "Arpit",
      "author_url": "",
      "post_date": "2020-11-07T11:15:39.510000",
      "content": "<p>I was able to train 1,600,000 datapoints (of ~22.5M) (100,000 iterations of batch size 16) in 38-40 hours on Colab pro. It seems to me that it took unusually long time for this amount of data! Is that so? How many samples/datapoints are you guys being able to train in around this much amount of time?</p>\n<p>I had the following configurations - </p>\n<pre><code>cfg = {\n    'format_version': 4,\n    'model_params': {\n        'history_num_frames': 2,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1,\n        'model_name': \"model_resnet34_output\",\n        'weight_path': None\n    },\n\n    'raster_params': {\n        'raster_size': [667, 300],\n        'pixel_size': [0.2, 0.2],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n  'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 16,\n        'shuffle': True,\n        'num_workers': 3\n    },\n}\n</code></pre>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1056467,
      "author_name": "Ali Abdin",
      "author_url": "",
      "post_date": "2020-10-21T18:19:22",
      "content": "<p>Try incremental training by running through kaggle notebooks multiple times. You can find some ideas here:<br>\n<a href=\"https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/189996\" target=\"_blank\">https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/189996</a></p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1056326": "Or some more powerful CPU (either desktop or GCE) is definitely needed?\n\nI can only get down to 3% batches within an epoch (around 5000-10000 steps) before notebook times out. I wonder if that's some approach I should give up sooner.\n\nThanks!",
    "1059283": "i am training on Colab Pro. I don't have own GPU so Colab Pro is nice choice",
    "1057112": "I did some experiments with Kaggle and Colab Pro.\nAs you know, on Colab Pro you can choose high-memory  VM.\n\nColab Pro normal VM for GPU: CPU cores = 2\nColab Pro normal VM for TPU: CPU cores = 2, TPU cores = 8\n\nColab Pro high-memory VM for GPU: CPU cores = 4\nColab Pro high-memory VM for TPU: CPU cores = 40, TPU cores = 8\n\nFor example, with batch size=16\nAssume it will take 1 iteration / second for training on Kaggle.\n\nColab Pro high-memory VM for GPU: 2.7 it/s\nColab Pro high-memory VM for TPU with Pytorch XLA\nsingle core of TPU: ~0.7 it/s\n8 cores of TPU: 1.2 it/s\n\nSo if you train on colab pro high-memory vm of gpu, you may shorten train time x2.7 than training on kaggle.",
    "1071749": "I was able to train 1,600,000 datapoints (of ~22.5M) (100,000 iterations of batch size 16) in 38-40 hours on Colab pro. It seems to me that it took unusually long time for this amount of data! Is that so? How many samples/datapoints are you guys being able to train in around this much amount of time?\n\nI had the following configurations - \n```\ncfg = {\n    'format_version': 4,\n    'model_params': {\n        'history_num_frames': 2,\n        'history_step_size': 1,\n        'history_delta_time': 0.1,\n        'future_num_frames': 50,\n        'future_step_size': 1,\n        'future_delta_time': 0.1,\n        'model_name': \"model_resnet34_output\",\n        'weight_path': None\n    },\n    \n    'raster_params': {\n        'raster_size': [667, 300],\n        'pixel_size': [0.2, 0.2],\n        'ego_center': [0.25, 0.5],\n        'map_type': 'py_semantic',\n        'satellite_map_key': 'aerial_map/aerial_map.png',\n        'semantic_map_key': 'semantic_map/semantic_map.pb',\n        'dataset_meta_key': 'meta.json',\n        'filter_agents_threshold': 0.5\n    },\n  'train_data_loader': {\n        'key': 'scenes/train.zarr',\n        'batch_size': 16,\n        'shuffle': True,\n        'num_workers': 3\n    },\n}\n```",
    "1056467": "Try incremental training by running through kaggle notebooks multiple times. You can find some ideas here:\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/189996"
  }
}