{
  "id": 104257,
  "title": "Computing resources used by the current gold and silver medalists?",
  "url": "/competitions/recursion-cellular-image-classification/discussion/104257",
  "author_name": "",
  "post_date": "2019-08-15T15:00:43.844914400Z",
  "votes": 5,
  "comment_count": 13,
  "views": 0,
  "content": "<p>I am training a model with the current setup:   </p>\n\n<hr>\n\n<p>model:resnet18</p>\n\n<p>inputs: 2*256x256 RGB w/ [baseline bag of tricks] (<a href=\"https://arxiv.org/pdf/1812.01187.pdf\">https://arxiv.org/pdf/1812.01187.pdf</a>) for train and validation sets</p>\n\n<p>lr: Cosine Annealing Scheduler from 3e-4 to 1e-7 (based on previous experiments and lr finder)\noptimizer: Adam   </p>\n\n<p>loss: CrossEntropy   </p>\n\n<p>epochs: 120 w/ early stopping after 50 epochs   </p>\n\n<p>resources: Tesla K80 (12 GB Memory | 61 GB RAM | 100 GB SSD)</p>\n\n<p><strong>estimate training time: ~8 hours</strong>  </p>\n\n<hr>\n\n<p>8 hours, in my view, seems a bit long to iterate fast. Since I'm trying to learn from the experts in this competition, I wonder what resources current medalists are using to arrive [faster] at such high LB scores? How do you iterate fast and to what extent the resources you use help you iterate that fast?</p>",
  "messages": [
    {
      "id": "600036",
      "postDate": "08/15/2019 15:00:43",
      "content": "<p>I am training a model with the current setup:   </p>\n\n<hr>\n\n<p>model:resnet18</p>\n\n<p>inputs: 2*256x256 RGB w/ [baseline bag of tricks] (<a href=\"https://arxiv.org/pdf/1812.01187.pdf\">https://arxiv.org/pdf/1812.01187.pdf</a>) for train and validation sets</p>\n\n<p>lr: Cosine Annealing Scheduler from 3e-4 to 1e-7 (based on previous experiments and lr finder)\noptimizer: Adam   </p>\n\n<p>loss: CrossEntropy   </p>\n\n<p>epochs: 120 w/ early stopping after 50 epochs   </p>\n\n<p>resources: Tesla K80 (12 GB Memory | 61 GB RAM | 100 GB SSD)</p>\n\n<p><strong>estimate training time: ~8 hours</strong>  </p>\n\n<hr>\n\n<p>8 hours, in my view, seems a bit long to iterate fast. Since I'm trying to learn from the experts in this competition, I wonder what resources current medalists are using to arrive [faster] at such high LB scores? How do you iterate fast and to what extent the resources you use help you iterate that fast?</p>",
      "rawMarkdown": "I am training a model with the current setup:   \n\n-----------------\n\nmodel:resnet18\n\ninputs: 2*256x256 RGB w/ [baseline bag of tricks] (https://arxiv.org/pdf/1812.01187.pdf) for train and validation sets\n\nlr: Cosine Annealing Scheduler from 3e-4 to 1e-7 (based on previous experiments and lr finder)\noptimizer: Adam   \n\nloss: CrossEntropy   \n\nepochs: 120 w/ early stopping after 50 epochs   \n\nresources: Tesla K80 (12 GB Memory | 61 GB RAM | 100 GB SSD)\n\n**estimate training time: ~8 hours**  \n\n----------------\n\n8 hours, in my view, seems a bit long to iterate fast. Since I'm trying to learn from the experts in this competition, I wonder what resources current medalists are using to arrive [faster] at such high LB scores? How do you iterate fast and to what extent the resources you use help you iterate that fast?",
      "votes": null
    },
    {
      "id": "600041",
      "postDate": "08/15/2019 15:10:53",
      "content": "<p>Thanks for sharing, Michel. I run my experiments (ResNet18) on GCP instances with Tesla P4. I notice that K80 is slower than P4. Did you sign up for GCP credits? May I also ask what do you mean by \"2-head resnet\"?  Thanks!</p>",
      "rawMarkdown": "Thanks for sharing, Michel. I run my experiments (ResNet18) on GCP instances with Tesla P4. I notice that K80 is slower than P4. Did you sign up for GCP credits? May I also ask what do you mean by \"2-head resnet\"?  Thanks!",
      "votes": null
    },
    {
      "id": "600069",
      "postDate": "08/15/2019 15:56:25",
      "content": "<p>How long does it take you to train such a model on a P4 <a href=\"/wjshenggggg\">@wjshenggggg</a> ? And, over how many epochs?   </p>\n\n<p>Two-head is probably the wrong terminology, I meant that my model takes two inputs (one for each site of a well) that are passed separately to a resnet18 \"layer\" without its final linear layer, these two independent resnet outputs are then concatenated and passed to a final linear layer afterward.</p>",
      "rawMarkdown": "How long does it take you to train such a model on a P4 @wjshenggggg ? And, over how many epochs?   \n\nTwo-head is probably the wrong terminology, I meant that my model takes two inputs (one for each site of a well) that are passed separately to a resnet18 \"layer\" without its final linear layer, these two independent resnet outputs are then concatenated and passed to a final linear layer afterward.",
      "votes": null
    },
    {
      "id": "600101",
      "postDate": "08/15/2019 16:38:22",
      "content": "<p>With an image size of 320, it takes ~10 mins per epoch excluding validation time. I have been running 30 epochs and it is still improving. I am currently trying to overfit to it to find a cutoff point for the epoch. But it is hard to quantify and compare my case as apples to apples to yours.</p>",
      "rawMarkdown": "With an image size of 320, it takes ~10 mins per epoch excluding validation time. I have been running 30 epochs and it is still improving. I am currently trying to overfit to it to find a cutoff point for the epoch. But it is hard to quantify and compare my case as apples to apples to yours.",
      "votes": null
    },
    {
      "id": "600134",
      "postDate": "08/15/2019 17:32:13",
      "content": "<p>Sure it's hard to compare apple to apple, but it helps! 10 minutes per epoch * 30 = 300 minutes = 300 minutes / 60 = 5 hours ... seems like a training process as slow as mine (no offense :) ). I'd be curious to hear other people on this one.</p>",
      "rawMarkdown": "Sure it's hard to compare apple to apple, but it helps! 10 minutes per epoch * 30 = 300 minutes = 300 minutes / 60 = 5 hours ... seems like a training process as slow as mine (no offense :) ). I'd be curious to hear other people on this one.",
      "votes": null
    },
    {
      "id": "603635",
      "postDate": "08/20/2019 14:06:12",
      "content": "<p>Are you resizing on the fly?</p>",
      "rawMarkdown": "Are you resizing on the fly?",
      "votes": null
    },
    {
      "id": "604007",
      "postDate": "08/20/2019 23:57:43",
      "content": "<p>No I'm not <a href=\"/christofhenkel\">@christofhenkel</a>  ! I'm data-augmenting on the fly if we can say so, here's a snippet of my transformations \"on the fly\" at the time of writing this question:  </p>\n\n<p>For training and cv respectively:   </p>\n\n<p>```\n    # taken textbook from <a href=\"https://arxiv.org/pdf/1812.01187.pdf\">https://arxiv.org/pdf/1812.01187.pdf</a>\n    transform_train = transforms.Compose([\n        transforms.RandomResizedCrop(224),\n        transforms.ColorJitter(brightness=jitter, contrast=jitter, saturation=jitter, hue=.1),\n        transforms.RandomHorizontalFlip(p=0.5),\n        # PCA Noise should go here,\n        transforms.ToTensor(),\n        transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n    ])</p>\n\n<pre><code>transform_validation = transforms.Compose([\n    transforms.CenterCrop(224),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n])  \n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "No I'm not @christofhenkel  ! I'm data-augmenting on the fly if we can say so, here's a snippet of my transformations \"on the fly\" at the time of writing this question:  \n\nFor training and cv respectively:   \n\n```\n    # taken textbook from https://arxiv.org/pdf/1812.01187.pdf\n    transform_train = transforms.Compose([\n        transforms.RandomResizedCrop(224),\n        transforms.ColorJitter(brightness=jitter, contrast=jitter, saturation=jitter, hue=.1),\n        transforms.RandomHorizontalFlip(p=0.5),\n        # PCA Noise should go here,\n        transforms.ToTensor(),\n        transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n    ])\n    \n    transform_validation = transforms.Compose([\n        transforms.CenterCrop(224),\n        transforms.ToTensor(),\n        transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n    ])  \n```",
      "votes": null
    },
    {
      "id": "604111",
      "postDate": "08/21/2019 03:37:38",
      "content": "<p>Are you still using ResNet18? Thanks.</p>",
      "rawMarkdown": "Are you still using ResNet18? Thanks.",
      "votes": null
    },
    {
      "id": "604114",
      "postDate": "08/21/2019 03:47:34",
      "content": "<p>I see you're using the mean and std from the paper. Is this giving you better performances?</p>",
      "rawMarkdown": "I see you're using the mean and std from the paper. Is this giving you better performances?",
      "votes": null
    },
    {
      "id": "604687",
      "postDate": "08/21/2019 16:49:51",
      "content": "<p>I haven't put much thought into that yet to be honest, I really applied the baseline augmentation of the paper as-is without the PCA noise part because for now it is beyond my comprehension. </p>",
      "rawMarkdown": "I haven't put much thought into that yet to be honest, I really applied the baseline augmentation of the paper as-is without the PCA noise part because for now it is beyond my comprehension.",
      "votes": null
    },
    {
      "id": "604695",
      "postDate": "08/21/2019 16:56:26",
      "content": "<p>I moved to efficientnet(s), my reasoning was that 1) <a href=\"https://arxiv.org/pdf/1905.11946.pdf\">they are faster to train for the accuracy you get</a> and 2) it seemed easier to scale to more complex models (efficientnet-0 to efficientnet-[1,2,3,4,5,6,7]) granularly as they have pretty much the same architecture except minor tweaks. I would test all models architecture if I'd have unlimited resources though - some leaders seem to use densenet(s).</p>\n\n<p>I have to tell you - I consider myself still a noob, so you take this information at your own risk :D . Nevertheless, I can tell you I was able to reach a LB score of .313 with an efficientnet-3 working with 448 crops on 512px RGB images of both sites - and this score went up to LB .423 when I applied the <a href=\"https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak\">plates leak</a> .</p>\n\n<p>P.S. I'd loved to be challenged on the model architecture choice reasoning if someone would have interesting insights on that end.</p>",
      "rawMarkdown": "I moved to efficientnet(s), my reasoning was that 1) [they are faster to train for the accuracy you get](https://arxiv.org/pdf/1905.11946.pdf) and 2) it seemed easier to scale to more complex models (efficientnet-0 to efficientnet-[1,2,3,4,5,6,7]) granularly as they have pretty much the same architecture except minor tweaks. I would test all models architecture if I'd have unlimited resources though - some leaders seem to use densenet(s).\n\nI have to tell you - I consider myself still a noob, so you take this information at your own risk :D . Nevertheless, I can tell you I was able to reach a LB score of .313 with an efficientnet-3 working with 448 crops on 512px RGB images of both sites - and this score went up to LB .423 when I applied the [plates leak](https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak) .\n\nP.S. I'd loved to be challenged on the model architecture choice reasoning if someone would have interesting insights on that end.",
      "votes": null
    },
    {
      "id": "604698",
      "postDate": "08/21/2019 16:59:43",
      "content": "<p>Thanks. Was just asking since I'm stuck at a low score so I'm trying to gather as much info as possible!</p>",
      "rawMarkdown": "Thanks. Was just asking since I'm stuck at a low score so I'm trying to gather as much info as possible!",
      "votes": null
    },
    {
      "id": "606282",
      "postDate": "08/23/2019 11:43:50",
      "content": "<p>Hi <a href=\"/michelml\">@michelml</a> , how many images per second do you train with your setup? I am getting 110 now, see <a href=\"https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&amp;sk=695f9605874c7306837645ff814e7fc9\">my setup description here</a>.</p>",
      "rawMarkdown": "Hi @michelml , how many images per second do you train with your setup? I am getting 110 now, see [my setup description here](https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&amp;sk=695f9605874c7306837645ff814e7fc9).",
      "votes": null
    },
    {
      "id": "606452",
      "postDate": "08/23/2019 15:41:18",
      "content": "<p>I presume that is on full sized images ? Around 10 minutes per epoch I guess - that is great. Thanks for sharing the details</p>",
      "rawMarkdown": "I presume that is on full sized images ? Around 10 minutes per epoch I guess - that is great. Thanks for sharing the details",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 600041,
      "author_name": "wjshenggggg",
      "author_url": "",
      "post_date": "08/15/2019 15:10:53",
      "content": "<p>Thanks for sharing, Michel. I run my experiments (ResNet18) on GCP instances with Tesla P4. I notice that K80 is slower than P4. Did you sign up for GCP credits? May I also ask what do you mean by \"2-head resnet\"?  Thanks!</p>",
      "votes": null,
      "replies": [
        {
          "id": 600069,
          "author_name": "michelml",
          "author_url": "",
          "post_date": "08/15/2019 15:56:25",
          "content": "<p>How long does it take you to train such a model on a P4 <a href=\"/wjshenggggg\">@wjshenggggg</a> ? And, over how many epochs?   </p>\n\n<p>Two-head is probably the wrong terminology, I meant that my model takes two inputs (one for each site of a well) that are passed separately to a resnet18 \"layer\" without its final linear layer, these two independent resnet outputs are then concatenated and passed to a final linear layer afterward.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 600101,
          "author_name": "wjshenggggg",
          "author_url": "",
          "post_date": "08/15/2019 16:38:22",
          "content": "<p>With an image size of 320, it takes ~10 mins per epoch excluding validation time. I have been running 30 epochs and it is still improving. I am currently trying to overfit to it to find a cutoff point for the epoch. But it is hard to quantify and compare my case as apples to apples to yours.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 600134,
          "author_name": "michelml",
          "author_url": "",
          "post_date": "08/15/2019 17:32:13",
          "content": "<p>Sure it's hard to compare apple to apple, but it helps! 10 minutes per epoch * 30 = 300 minutes = 300 minutes / 60 = 5 hours ... seems like a training process as slow as mine (no offense :) ). I'd be curious to hear other people on this one.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 603635,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "08/20/2019 14:06:12",
      "content": "<p>Are you resizing on the fly?</p>",
      "votes": null,
      "replies": [
        {
          "id": 604007,
          "author_name": "michelml",
          "author_url": "",
          "post_date": "08/20/2019 23:57:43",
          "content": "<p>No I'm not <a href=\"/christofhenkel\">@christofhenkel</a>  ! I'm data-augmenting on the fly if we can say so, here's a snippet of my transformations \"on the fly\" at the time of writing this question:  </p>\n\n<p>For training and cv respectively:   </p>\n\n<p>```\n    # taken textbook from <a href=\"https://arxiv.org/pdf/1812.01187.pdf\">https://arxiv.org/pdf/1812.01187.pdf</a>\n    transform_train = transforms.Compose([\n        transforms.RandomResizedCrop(224),\n        transforms.ColorJitter(brightness=jitter, contrast=jitter, saturation=jitter, hue=.1),\n        transforms.RandomHorizontalFlip(p=0.5),\n        # PCA Noise should go here,\n        transforms.ToTensor(),\n        transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n    ])</p>\n\n<pre><code>transform_validation = transforms.Compose([\n    transforms.CenterCrop(224),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n])  \n</code></pre>\n\n<p>```</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 604114,
          "author_name": "lorenzofabbri92",
          "author_url": "",
          "post_date": "08/21/2019 03:47:34",
          "content": "<p>I see you're using the mean and std from the paper. Is this giving you better performances?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 604687,
          "author_name": "michelml",
          "author_url": "",
          "post_date": "08/21/2019 16:49:51",
          "content": "<p>I haven't put much thought into that yet to be honest, I really applied the baseline augmentation of the paper as-is without the PCA noise part because for now it is beyond my comprehension. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 604698,
          "author_name": "lorenzofabbri92",
          "author_url": "",
          "post_date": "08/21/2019 16:59:43",
          "content": "<p>Thanks. Was just asking since I'm stuck at a low score so I'm trying to gather as much info as possible!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 604111,
      "author_name": "lorenzofabbri92",
      "author_url": "",
      "post_date": "08/21/2019 03:37:38",
      "content": "<p>Are you still using ResNet18? Thanks.</p>",
      "votes": null,
      "replies": [
        {
          "id": 604695,
          "author_name": "michelml",
          "author_url": "",
          "post_date": "08/21/2019 16:56:26",
          "content": "<p>I moved to efficientnet(s), my reasoning was that 1) <a href=\"https://arxiv.org/pdf/1905.11946.pdf\">they are faster to train for the accuracy you get</a> and 2) it seemed easier to scale to more complex models (efficientnet-0 to efficientnet-[1,2,3,4,5,6,7]) granularly as they have pretty much the same architecture except minor tweaks. I would test all models architecture if I'd have unlimited resources though - some leaders seem to use densenet(s).</p>\n\n<p>I have to tell you - I consider myself still a noob, so you take this information at your own risk :D . Nevertheless, I can tell you I was able to reach a LB score of .313 with an efficientnet-3 working with 448 crops on 512px RGB images of both sites - and this score went up to LB .423 when I applied the <a href=\"https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak\">plates leak</a> .</p>\n\n<p>P.S. I'd loved to be challenged on the model architecture choice reasoning if someone would have interesting insights on that end.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 606282,
      "author_name": "zaharch",
      "author_url": "",
      "post_date": "08/23/2019 11:43:50",
      "content": "<p>Hi <a href=\"/michelml\">@michelml</a> , how many images per second do you train with your setup? I am getting 110 now, see <a href=\"https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&amp;sk=695f9605874c7306837645ff814e7fc9\">my setup description here</a>.</p>",
      "votes": null,
      "replies": [
        {
          "id": 606452,
          "author_name": "darraghdog",
          "author_url": "",
          "post_date": "08/23/2019 15:41:18",
          "content": "<p>I presume that is on full sized images ? Around 10 minutes per epoch I guess - that is great. Thanks for sharing the details</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "600036": "I am training a model with the current setup:   \n\n-----------------\n\nmodel:resnet18\n\ninputs: 2*256x256 RGB w/ [baseline bag of tricks] (https://arxiv.org/pdf/1812.01187.pdf) for train and validation sets\n\nlr: Cosine Annealing Scheduler from 3e-4 to 1e-7 (based on previous experiments and lr finder)\noptimizer: Adam   \n\nloss: CrossEntropy   \n\nepochs: 120 w/ early stopping after 50 epochs   \n\nresources: Tesla K80 (12 GB Memory | 61 GB RAM | 100 GB SSD)\n\n**estimate training time: ~8 hours**  \n\n----------------\n\n8 hours, in my view, seems a bit long to iterate fast. Since I'm trying to learn from the experts in this competition, I wonder what resources current medalists are using to arrive [faster] at such high LB scores? How do you iterate fast and to what extent the resources you use help you iterate that fast?",
    "600041": "Thanks for sharing, Michel. I run my experiments (ResNet18) on GCP instances with Tesla P4. I notice that K80 is slower than P4. Did you sign up for GCP credits? May I also ask what do you mean by \"2-head resnet\"?  Thanks!",
    "600069": "How long does it take you to train such a model on a P4 @wjshenggggg ? And, over how many epochs?   \n\nTwo-head is probably the wrong terminology, I meant that my model takes two inputs (one for each site of a well) that are passed separately to a resnet18 \"layer\" without its final linear layer, these two independent resnet outputs are then concatenated and passed to a final linear layer afterward.",
    "600101": "With an image size of 320, it takes ~10 mins per epoch excluding validation time. I have been running 30 epochs and it is still improving. I am currently trying to overfit to it to find a cutoff point for the epoch. But it is hard to quantify and compare my case as apples to apples to yours.",
    "600134": "Sure it's hard to compare apple to apple, but it helps! 10 minutes per epoch * 30 = 300 minutes = 300 minutes / 60 = 5 hours ... seems like a training process as slow as mine (no offense :) ). I'd be curious to hear other people on this one.",
    "603635": "Are you resizing on the fly?",
    "604007": "No I'm not @christofhenkel  ! I'm data-augmenting on the fly if we can say so, here's a snippet of my transformations \"on the fly\" at the time of writing this question:  \n\nFor training and cv respectively:   \n\n```\n    # taken textbook from https://arxiv.org/pdf/1812.01187.pdf\n    transform_train = transforms.Compose([\n        transforms.RandomResizedCrop(224),\n        transforms.ColorJitter(brightness=jitter, contrast=jitter, saturation=jitter, hue=.1),\n        transforms.RandomHorizontalFlip(p=0.5),\n        # PCA Noise should go here,\n        transforms.ToTensor(),\n        transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n    ])\n    \n    transform_validation = transforms.Compose([\n        transforms.CenterCrop(224),\n        transforms.ToTensor(),\n        transforms.Normalize(mean=(123.68, 116.779, 103.939), std=(58.393, 57.12, 57.375))\n    ])  \n```",
    "604111": "Are you still using ResNet18? Thanks.",
    "604114": "I see you're using the mean and std from the paper. Is this giving you better performances?",
    "604687": "I haven't put much thought into that yet to be honest, I really applied the baseline augmentation of the paper as-is without the PCA noise part because for now it is beyond my comprehension.",
    "604695": "I moved to efficientnet(s), my reasoning was that 1) [they are faster to train for the accuracy you get](https://arxiv.org/pdf/1905.11946.pdf) and 2) it seemed easier to scale to more complex models (efficientnet-0 to efficientnet-[1,2,3,4,5,6,7]) granularly as they have pretty much the same architecture except minor tweaks. I would test all models architecture if I'd have unlimited resources though - some leaders seem to use densenet(s).\n\nI have to tell you - I consider myself still a noob, so you take this information at your own risk :D . Nevertheless, I can tell you I was able to reach a LB score of .313 with an efficientnet-3 working with 448 crops on 512px RGB images of both sites - and this score went up to LB .423 when I applied the [plates leak](https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak) .\n\nP.S. I'd loved to be challenged on the model architecture choice reasoning if someone would have interesting insights on that end.",
    "604698": "Thanks. Was just asking since I'm stuck at a low score so I'm trying to gather as much info as possible!",
    "606282": "Hi @michelml , how many images per second do you train with your setup? I am getting 110 now, see [my setup description here](https://medium.com/@zaharchikishev/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4?source=friends_link&amp;sk=695f9605874c7306837645ff814e7fc9).",
    "606452": "I presume that is on full sized images ? Around 10 minutes per epoch I guess - that is great. Thanks for sharing the details"
  },
  "source": "meta"
}