{
  "id": 69192,
  "title": "Getting started. With NGC DIGITS at GCP ",
  "url": "/competitions/inclusive-images-challenge/discussion/69192",
  "author_name": "",
  "post_date": "2018-10-21T14:25:52.622210400Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Presuming the dataset is downloaded to GCP based ngc instance that has DIGITS server,  what will be the next step forimporting the dataset to digits?\nPresuming the below steps  have been completed:\n1) instance with <a href=\"https://console.cloud.google.com/marketplace/partners/nvidia-ngc-public\">https://console.cloud.google.com/marketplace/partners/nvidia-ngc-public</a> started\n2) dataset downloaded to <a href=\"https://console.cloud.google.com/storage/browser/inclusive-images_challenge\">gs</a> <a href=\"https://console.cloud.google.com/storage/browser/inclusive-images_challenge\">https://console.cloud.google.com/storage/browser/inclusive-images_challenge</a> and to the the instance\n3) digits server started with the sequence below</p>\n\n<pre><code>$    docker pull nvcr.io/nvidia/digits:18.09\n$    nvidia-docker run --name digits -d -p 8888:5000 \\\n -v /home/username/data:/data:ro\n -v /home/username/digits-jobs:/workspace/jobs nvcr.io/nvidia/digits:18.09\n</code></pre>\n\n<p>4) creating dataset</p>\n\n<p>Update:\nNow it appears to me that images should be sorted into folders somehow. I can see there only one option of involving manual expert's labour, though that can be implemented via Amazon mTurk. Otherwise DIGITS will not process the dataset, as it seems to me.</p>",
  "messages": [
    {
      "id": "407613",
      "postDate": "10/21/2018 14:25:52",
      "content": "<p>Presuming the dataset is downloaded to GCP based ngc instance that has DIGITS server,  what will be the next step forimporting the dataset to digits?\nPresuming the below steps  have been completed:\n1) instance with <a href=\"https://console.cloud.google.com/marketplace/partners/nvidia-ngc-public\">https://console.cloud.google.com/marketplace/partners/nvidia-ngc-public</a> started\n2) dataset downloaded to <a href=\"https://console.cloud.google.com/storage/browser/inclusive-images_challenge\">gs</a> <a href=\"https://console.cloud.google.com/storage/browser/inclusive-images_challenge\">https://console.cloud.google.com/storage/browser/inclusive-images_challenge</a> and to the the instance\n3) digits server started with the sequence below</p>\n\n<pre><code>$    docker pull nvcr.io/nvidia/digits:18.09\n$    nvidia-docker run --name digits -d -p 8888:5000 \\\n -v /home/username/data:/data:ro\n -v /home/username/digits-jobs:/workspace/jobs nvcr.io/nvidia/digits:18.09\n</code></pre>\n\n<p>4) creating dataset</p>\n\n<p>Update:\nNow it appears to me that images should be sorted into folders somehow. I can see there only one option of involving manual expert's labour, though that can be implemented via Amazon mTurk. Otherwise DIGITS will not process the dataset, as it seems to me.</p>",
      "rawMarkdown": "Presuming the dataset is downloaded to GCP based ngc instance that has DIGITS server,  what will be the next step forimporting the dataset to digits?\nPresuming the below steps  have been completed:\n1) instance with https://console.cloud.google.com/marketplace/partners/nvidia-ngc-public started\n2) dataset downloaded to [gs][1] https://console.cloud.google.com/storage/browser/inclusive-images_challenge and to the the instance\n3) digits server started with the sequence below\n\n    $    docker pull nvcr.io/nvidia/digits:18.09\n    $    nvidia-docker run --name digits -d -p 8888:5000 \\\n     -v /home/username/data:/data:ro\n     -v /home/username/digits-jobs:/workspace/jobs nvcr.io/nvidia/digits:18.09\n\n4) creating dataset\n\n\n  [1]: https://console.cloud.google.com/storage/browser/inclusive-images_challenge\n\nUpdate:\nNow it appears to me that images should be sorted into folders somehow. I can see there only one option of involving manual expert's labour, though that can be implemented via Amazon mTurk. Otherwise DIGITS will not process the dataset, as it seems to me.",
      "votes": null
    },
    {
      "id": "407868",
      "postDate": "10/21/2018 22:55:28",
      "content": "<p>I shall change the direction, perhaps, Download an imagenet dataset like </p>\n\n<pre><code>$ wget --no-check-certificate https://nvidia.box.com/shared/static/gzr5iewf5aouhc5exhp3higw6lzhcysj.gz -O \n\nilsvrc12_urls.tar.gz\n$ tar -xzvf ilsvrc12_urls.tar.gz\n$ wget https://rawgit.com/dusty-nv/jetson-inference/master/tools/imagenet-download.py\n$ python imagenet-download.py ilsvrc12_urls.txt . --jobs 100 --retry 3 --sleep 0\n</code></pre>\n\n<p>And then use it for mass classification of objects with DIGITS.</p>",
      "rawMarkdown": "I shall change the direction, perhaps, Download an imagenet dataset like \n\n    $ wget --no-check-certificate https://nvidia.box.com/shared/static/gzr5iewf5aouhc5exhp3higw6lzhcysj.gz -O \n\n    ilsvrc12_urls.tar.gz\n    $ tar -xzvf ilsvrc12_urls.tar.gz\n    $ wget https://rawgit.com/dusty-nv/jetson-inference/master/tools/imagenet-download.py\n    $ python imagenet-download.py ilsvrc12_urls.txt . --jobs 100 --retry 3 --sleep 0\n\nAnd then use it for mass classification of objects with DIGITS.",
      "votes": null
    },
    {
      "id": "408139",
      "postDate": "10/22/2018 11:14:59",
      "content": "<p>Refined steps:</p>\n\n<pre><code>1) start DIGITS server and import to it the Dataset ImageNet-ILSVRC12-subset\n2) train the model based on the dataset\n3) use the model to classify given objects [512gb]\n4) save the results [ how? and in a what form?]\n5) format the  results to comply with the specifications for the submission file [how?]\n</code></pre>",
      "rawMarkdown": "Refined steps:\n\n    1) start DIGITS server and import to it the Dataset ImageNet-ILSVRC12-subset\n    2) train the model based on the dataset\n    3) use the model to classify given objects [512gb]\n    4) save the results [ how? and in a what form?]\n    5) format the  results to comply with the specifications for the submission file [how?]",
      "votes": null
    },
    {
      "id": "410961",
      "postDate": "10/27/2018 03:02:50",
      "content": "<p>generating list of images to direct the list of gs located  images to DIGITS server:</p>\n\n<pre><code>gsutil ls gs://inclusive-images_challenge/train  &amp;gt; val.txt\n</code></pre>\n\n<p>though the local ssd turned out to be faster:</p>\n\n<pre><code>nvidia@nvidia-ngc-image-9-vm:~/data$ ls train/ &amp;gt;txt.xtx\n</code></pre>\n\n<p>the latter creates 35mb txt file that seems to take a while while importing to DIGITS\n<img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/2058723/dba04bc8b171e5cb40e48b9b28aebb41/mz3n6j1%20-%20Imgur.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "generating list of images to direct the list of gs located  images to DIGITS server:\n\n    gsutil ls gs://inclusive-images_challenge/train  &gt; val.txt\n\nthough the local ssd turned out to be faster:\n\n    nvidia@nvidia-ngc-image-9-vm:~/data$ ls train/ &gt;txt.xtx\nthe latter creates 35mb txt file that seems to take a while while importing to DIGITS\n![enter image description here][1]\n\n\n  [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/2058723/dba04bc8b171e5cb40e48b9b28aebb41/mz3n6j1%20-%20Imgur.png",
      "votes": null
    },
    {
      "id": "414249",
      "postDate": "11/02/2018 12:10:10",
      "content": "<p>It seems that I have finally sorted out how to output a single result.csv file from 'classify many' operation in DIGITS:</p>\n\n<pre><code>  dump_csv = open('/home/ubuntu/results.csv', 'w')\n    dump_csv.write('Path\\tProbabilities')\n    for i,j in zip(paths, scores):\n        newline = i + '\\t' + str(j) + '\\n'\n        dump_csv.write(newline)\n    dump_csv.close()\n</code></pre>\n\n<p>source: <a href=\"https://github.com/NVIDIA/DIGITS/issues/61\">https://github.com/NVIDIA/DIGITS/issues/61</a></p>",
      "rawMarkdown": "It seems that I have finally sorted out how to output a single result.csv file from 'classify many' operation in DIGITS:\n  \n\n      dump_csv = open('/home/ubuntu/results.csv', 'w')\n        dump_csv.write('Path\\tProbabilities')\n        for i,j in zip(paths, scores):\n            newline = i + '\\t' + str(j) + '\\n'\n            dump_csv.write(newline)\n        dump_csv.close()\n\nsource: https://github.com/NVIDIA/DIGITS/issues/61",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 407868,
      "author_name": "avolod",
      "author_url": "",
      "post_date": "10/21/2018 22:55:28",
      "content": "<p>I shall change the direction, perhaps, Download an imagenet dataset like </p>\n\n<pre><code>$ wget --no-check-certificate https://nvidia.box.com/shared/static/gzr5iewf5aouhc5exhp3higw6lzhcysj.gz -O \n\nilsvrc12_urls.tar.gz\n$ tar -xzvf ilsvrc12_urls.tar.gz\n$ wget https://rawgit.com/dusty-nv/jetson-inference/master/tools/imagenet-download.py\n$ python imagenet-download.py ilsvrc12_urls.txt . --jobs 100 --retry 3 --sleep 0\n</code></pre>\n\n<p>And then use it for mass classification of objects with DIGITS.</p>",
      "votes": null,
      "replies": [
        {
          "id": 408139,
          "author_name": "avolod",
          "author_url": "",
          "post_date": "10/22/2018 11:14:59",
          "content": "<p>Refined steps:</p>\n\n<pre><code>1) start DIGITS server and import to it the Dataset ImageNet-ILSVRC12-subset\n2) train the model based on the dataset\n3) use the model to classify given objects [512gb]\n4) save the results [ how? and in a what form?]\n5) format the  results to comply with the specifications for the submission file [how?]\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 410961,
          "author_name": "avolod",
          "author_url": "",
          "post_date": "10/27/2018 03:02:50",
          "content": "<p>generating list of images to direct the list of gs located  images to DIGITS server:</p>\n\n<pre><code>gsutil ls gs://inclusive-images_challenge/train  &amp;gt; val.txt\n</code></pre>\n\n<p>though the local ssd turned out to be faster:</p>\n\n<pre><code>nvidia@nvidia-ngc-image-9-vm:~/data$ ls train/ &amp;gt;txt.xtx\n</code></pre>\n\n<p>the latter creates 35mb txt file that seems to take a while while importing to DIGITS\n<img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/2058723/dba04bc8b171e5cb40e48b9b28aebb41/mz3n6j1%20-%20Imgur.png\" alt=\"enter image description here\"></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 414249,
          "author_name": "avolod",
          "author_url": "",
          "post_date": "11/02/2018 12:10:10",
          "content": "<p>It seems that I have finally sorted out how to output a single result.csv file from 'classify many' operation in DIGITS:</p>\n\n<pre><code>  dump_csv = open('/home/ubuntu/results.csv', 'w')\n    dump_csv.write('Path\\tProbabilities')\n    for i,j in zip(paths, scores):\n        newline = i + '\\t' + str(j) + '\\n'\n        dump_csv.write(newline)\n    dump_csv.close()\n</code></pre>\n\n<p>source: <a href=\"https://github.com/NVIDIA/DIGITS/issues/61\">https://github.com/NVIDIA/DIGITS/issues/61</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "407613": "Presuming the dataset is downloaded to GCP based ngc instance that has DIGITS server,  what will be the next step forimporting the dataset to digits?\nPresuming the below steps  have been completed:\n1) instance with https://console.cloud.google.com/marketplace/partners/nvidia-ngc-public started\n2) dataset downloaded to [gs][1] https://console.cloud.google.com/storage/browser/inclusive-images_challenge and to the the instance\n3) digits server started with the sequence below\n\n    $    docker pull nvcr.io/nvidia/digits:18.09\n    $    nvidia-docker run --name digits -d -p 8888:5000 \\\n     -v /home/username/data:/data:ro\n     -v /home/username/digits-jobs:/workspace/jobs nvcr.io/nvidia/digits:18.09\n\n4) creating dataset\n\n\n  [1]: https://console.cloud.google.com/storage/browser/inclusive-images_challenge\n\nUpdate:\nNow it appears to me that images should be sorted into folders somehow. I can see there only one option of involving manual expert's labour, though that can be implemented via Amazon mTurk. Otherwise DIGITS will not process the dataset, as it seems to me.",
    "407868": "I shall change the direction, perhaps, Download an imagenet dataset like \n\n    $ wget --no-check-certificate https://nvidia.box.com/shared/static/gzr5iewf5aouhc5exhp3higw6lzhcysj.gz -O \n\n    ilsvrc12_urls.tar.gz\n    $ tar -xzvf ilsvrc12_urls.tar.gz\n    $ wget https://rawgit.com/dusty-nv/jetson-inference/master/tools/imagenet-download.py\n    $ python imagenet-download.py ilsvrc12_urls.txt . --jobs 100 --retry 3 --sleep 0\n\nAnd then use it for mass classification of objects with DIGITS.",
    "408139": "Refined steps:\n\n    1) start DIGITS server and import to it the Dataset ImageNet-ILSVRC12-subset\n    2) train the model based on the dataset\n    3) use the model to classify given objects [512gb]\n    4) save the results [ how? and in a what form?]\n    5) format the  results to comply with the specifications for the submission file [how?]",
    "410961": "generating list of images to direct the list of gs located  images to DIGITS server:\n\n    gsutil ls gs://inclusive-images_challenge/train  &gt; val.txt\n\nthough the local ssd turned out to be faster:\n\n    nvidia@nvidia-ngc-image-9-vm:~/data$ ls train/ &gt;txt.xtx\nthe latter creates 35mb txt file that seems to take a while while importing to DIGITS\n![enter image description here][1]\n\n\n  [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/inbox/2058723/dba04bc8b171e5cb40e48b9b28aebb41/mz3n6j1%20-%20Imgur.png",
    "414249": "It seems that I have finally sorted out how to output a single result.csv file from 'classify many' operation in DIGITS:\n  \n\n      dump_csv = open('/home/ubuntu/results.csv', 'w')\n        dump_csv.write('Path\\tProbabilities')\n        for i,j in zip(paths, scores):\n            newline = i + '\\t' + str(j) + '\\n'\n            dump_csv.write(newline)\n        dump_csv.close()\n\nsource: https://github.com/NVIDIA/DIGITS/issues/61"
  },
  "source": "meta"
}