{
  "id": 58942,
  "title": "Any method to accelerate image feature extraction",
  "url": "/competitions/avito-demand-prediction/discussion/58942",
  "author_name": "",
  "post_date": "2018-06-15T15:22:14.259729700Z",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>We basically follow the kernel here <a href=\"https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality\">https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality</a> to extract features but seems it is time-consuming. I tried 48 hour with 10 core CPU and it not finished. Wondering if there is some tricks to accelerate the process</p>",
  "messages": [
    {
      "id": "343564",
      "postDate": "06/15/2018 15:22:14",
      "content": "<p>We basically follow the kernel here <a href=\"https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality\">https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality</a> to extract features but seems it is time-consuming. I tried 48 hour with 10 core CPU and it not finished. Wondering if there is some tricks to accelerate the process</p>",
      "rawMarkdown": "We basically follow the kernel here https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality to extract features but seems it is time-consuming. I tried 48 hour with 10 core CPU and it not finished. Wondering if there is some tricks to accelerate the process",
      "votes": null
    },
    {
      "id": "343572",
      "postDate": "06/15/2018 15:40:44",
      "content": "<p>The job can be divided into two parts: 1) load figures fomr zip and 2) use opencv to get image information. The first one is a CPU extensive work, so try to parallelize it. Another point is that do not dump too many figures simultaneously, you may reach your RAM limit which will reduces the speed significantly. I tried to make sure the RAM usage is around 80% and it finished in less than 10 hours. </p>",
      "rawMarkdown": "The job can be divided into two parts: 1) load figures fomr zip and 2) use opencv to get image information. The first one is a CPU extensive work, so try to parallelize it. Another point is that do not dump too many figures simultaneously, you may reach your RAM limit which will reduces the speed significantly. I tried to make sure the RAM usage is around 80% and it finished in less than 10 hours.",
      "votes": null
    },
    {
      "id": "345553",
      "postDate": "06/20/2018 03:13:13",
      "content": "<p>Finished?</p>",
      "rawMarkdown": "Finished?",
      "votes": null
    },
    {
      "id": "345581",
      "postDate": "06/20/2018 04:35:31",
      "content": "<p>Not start yet= = </p>",
      "rawMarkdown": "Not start yet= =",
      "votes": null
    },
    {
      "id": "345610",
      "postDate": "06/20/2018 05:43:39",
      "content": "<p>GCP please. Hahah</p>",
      "rawMarkdown": "GCP please. Hahah",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 343572,
      "author_name": "naivelamb",
      "author_url": "",
      "post_date": "06/15/2018 15:40:44",
      "content": "<p>The job can be divided into two parts: 1) load figures fomr zip and 2) use opencv to get image information. The first one is a CPU extensive work, so try to parallelize it. Another point is that do not dump too many figures simultaneously, you may reach your RAM limit which will reduces the speed significantly. I tried to make sure the RAM usage is around 80% and it finished in less than 10 hours. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 345553,
      "author_name": "shujian",
      "author_url": "",
      "post_date": "06/20/2018 03:13:13",
      "content": "<p>Finished?</p>",
      "votes": null,
      "replies": [
        {
          "id": 345581,
          "author_name": "strideradu",
          "author_url": "",
          "post_date": "06/20/2018 04:35:31",
          "content": "<p>Not start yet= = </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 345610,
          "author_name": "shujian",
          "author_url": "",
          "post_date": "06/20/2018 05:43:39",
          "content": "<p>GCP please. Hahah</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "343564": "We basically follow the kernel here https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality to extract features but seems it is time-consuming. I tried 48 hour with 10 core CPU and it not finished. Wondering if there is some tricks to accelerate the process",
    "343572": "The job can be divided into two parts: 1) load figures fomr zip and 2) use opencv to get image information. The first one is a CPU extensive work, so try to parallelize it. Another point is that do not dump too many figures simultaneously, you may reach your RAM limit which will reduces the speed significantly. I tried to make sure the RAM usage is around 80% and it finished in less than 10 hours.",
    "345553": "Finished?",
    "345581": "Not start yet= =",
    "345610": "GCP please. Hahah"
  },
  "source": "meta"
}