{
  "id": 30314,
  "title": "Keras+TensorFlow killed by colfax server",
  "url": "/competitions/intel-mobileodt-cervical-cancer-screening/discussion/30314",
  "author_name": "Spencer",
  "post_date": "2017-03-18T11:24:42.936000",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I am trying to run keras with TensorFlow backend on the colfax server. </p>\n\n<p>However, my process was always killed by the server, even the official mnist example.</p>\n\n<p>Below are the error messages I got when I tested the example</p>\n\n<p>```\nUsing TensorFlow backend.</p>\n\n<p>Downloading data from <a href=\"https://s3.amazonaws.com/img-datasets/mnist.npz\">https://s3.amazonaws.com/img-datasets/mnist.npz</a></p>\n\n<p>X_train shape: (60000, 28, 28, 1)</p>\n\n<p>60000 train samples</p>\n\n<p>10000 test samples</p>\n\n<p>/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your <code>Conv2D</code> call to the Keras 2 API: <code>Conv2D(32, (3, 3), padding=\"valid\", input_shape=(28, 28, 1...)</code>\n  '` call to the Keras 2 API: ' + signature)</p>\n\n<p>/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your <code>Conv2D</code> call to the Keras 2 API: <code>Conv2D(32, (3, 3))</code>\n  '` call to the Keras 2 API: ' + signature)</p>\n\n<p>/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/models.py:826: UserWarning: The <code>nb_epoch</code> argument in <code>fit</code> has been renamed <code>epochs</code>.\n  warnings.warn('The <code>nb_epoch</code> argument in <code>fit</code> '</p>\n\n<p>Train on 60000 samples, validate on 10000 samples</p>\n\n<p>Epoch 1/12\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>18304/60000 [========&gt;.....................] - ETA: 43s - loss: 0.7248 - acc: 0.7711Killed\n```</p>\n\n<p>Is Keras+TensorFlow banned by colfax? Or is there any workaround to run Keras on the colfax server?</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": 168939,
      "postDate": "2017-03-18T18:21:18.120Z",
      "content": "<p>To those users not familiar with the cluster (like me):</p>\n\n<p>As the <a href=\"https://access.colfaxresearch.com/?p=compute\">document</a> says, we have to use the scheduler to run long programs on the computing node.\nWe use <code>qsub</code> to submit our jobs, which could be wrapped in a shell script.</p>\n\n<p>One thing I did not notice is that we need to change the path to absolute path instead of relative path.\nTake my example, I use conda to isolate my packages, and my python script is named <code>main.py</code></p>\n\n<p>Then I needed to write a shell script, let's call it <code>main.sh</code>, and write the following lines</p>\n\n<p><code>source activate py35env</code></p>\n\n<p><code>python [the absolute path]/main.py</code></p>\n\n<p>Then send the task (the shell script) to the scheduler by typing </p>\n\n<p><code>qsub main.sh</code></p>\n\n<p>One should remember that in the Python script you also need to use absolute pathes.</p>\n\n<p>-Spencer</p>",
      "rawMarkdown": "To those users not familiar with the cluster (like me):\n\nAs the [document][1] says, we have to use the scheduler to run long programs on the computing node.\nWe use ```qsub``` to submit our jobs, which could be wrapped in a shell script.\n\nOne thing I did not notice is that we need to change the path to absolute path instead of relative path.\nTake my example, I use conda to isolate my packages, and my python script is named ```main.py```\n\nThen I needed to write a shell script, let's call it ```main.sh```, and write the following lines\n\n\n```source activate py35env```\n\n```python [the absolute path]/main.py```\n\n\nThen send the task (the shell script) to the scheduler by typing \n\n```qsub main.sh```\n\nOne should remember that in the Python script you also need to use absolute pathes.\n\n-Spencer\n\n\n  [1]: https://access.colfaxresearch.com/?p=compute",
      "votes": 3
    },
    {
      "id": 168851,
      "postDate": "2017-03-18T11:24:42.937Z",
      "content": "<p>I am trying to run keras with TensorFlow backend on the colfax server. </p>\n\n<p>However, my process was always killed by the server, even the official mnist example.</p>\n\n<p>Below are the error messages I got when I tested the example</p>\n\n<p>```\nUsing TensorFlow backend.</p>\n\n<p>Downloading data from <a href=\"https://s3.amazonaws.com/img-datasets/mnist.npz\">https://s3.amazonaws.com/img-datasets/mnist.npz</a></p>\n\n<p>X_train shape: (60000, 28, 28, 1)</p>\n\n<p>60000 train samples</p>\n\n<p>10000 test samples</p>\n\n<p>/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your <code>Conv2D</code> call to the Keras 2 API: <code>Conv2D(32, (3, 3), padding=\"valid\", input_shape=(28, 28, 1...)</code>\n  '` call to the Keras 2 API: ' + signature)</p>\n\n<p>/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your <code>Conv2D</code> call to the Keras 2 API: <code>Conv2D(32, (3, 3))</code>\n  '` call to the Keras 2 API: ' + signature)</p>\n\n<p>/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/models.py:826: UserWarning: The <code>nb_epoch</code> argument in <code>fit</code> has been renamed <code>epochs</code>.\n  warnings.warn('The <code>nb_epoch</code> argument in <code>fit</code> '</p>\n\n<p>Train on 60000 samples, validate on 10000 samples</p>\n\n<p>Epoch 1/12\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.</p>\n\n<p>18304/60000 [========&gt;.....................] - ETA: 43s - loss: 0.7248 - acc: 0.7711Killed\n```</p>\n\n<p>Is Keras+TensorFlow banned by colfax? Or is there any workaround to run Keras on the colfax server?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "I am trying to run keras with TensorFlow backend on the colfax server. \n\nHowever, my process was always killed by the server, even the official mnist example.\n\nBelow are the error messages I got when I tested the example\n\n```\nUsing TensorFlow backend.\n\nDownloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n\nX_train shape: (60000, 28, 28, 1)\n\n60000 train samples\n\n10000 test samples\n\n/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3), padding=\"valid\", input_shape=(28, 28, 1...)`\n  '` call to the Keras 2 API: ' + signature)\n\n/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3))`\n  '` call to the Keras 2 API: ' + signature)\n\n/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/models.py:826: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n  warnings.warn('The `nb_epoch` argument in `fit` '\n\nTrain on 60000 samples, validate on 10000 samples\n\nEpoch 1/12\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n\n18304/60000 [========>.....................] - ETA: 43s - loss: 0.7248 - acc: 0.7711Killed\n```\n\nIs Keras+TensorFlow banned by colfax? Or is there any workaround to run Keras on the colfax server?\n\nThanks!",
      "votes": 3
    },
    {
      "id": 168859,
      "postDate": "2017-03-18T11:38:55.870Z",
      "content": "<p>Do you use <code>qsub</code> command? The <code>qsub</code> command requests job queues. <br>\nRef: <a href=\"https://access.colfaxresearch.com/?p=compute\">https://access.colfaxresearch.com/?p=compute</a> <br>\nThe python process which is spending a long time, will be terminated (maybe?), if is not by <code>qsub</code> command. <br>\nI was also troubled by this. The usage of <code>qsub</code> command is described in <a href=\"https://www.kaggle.com/kambarakun/intel-mobileodt-cervical-cancer-screening/how-to-start-with-python-on-colfax-cluster\">my kernel</a> too:)</p>",
      "rawMarkdown": "Do you use `qsub` command? The `qsub` command requests job queues.  \nRef: [https://access.colfaxresearch.com/?p=compute](https://access.colfaxresearch.com/?p=compute)  \nThe python process which is spending a long time, will be terminated (maybe?), if is not by `qsub` command.  \nI was also troubled by this. The usage of `qsub` command is described in [my kernel](https://www.kaggle.com/kambarakun/intel-mobileodt-cervical-cancer-screening/how-to-start-with-python-on-colfax-cluster) too:)",
      "replies": [
        {
          "id": 168863,
          "postDate": "2017-03-18T11:59:55.600Z",
          "content": "<p>Hi,</p>\n\n<p>Thanks for pointing out qsub.</p>\n\n<p>I just tried by creating a <code>mnist_cnn.sh</code> which contains only one line \n<code>\npython mnist_cnn.py\n</code></p>\n\n<p>However, the task only stayed in the queue for less than 20 seconds (which means failed I guess). I will update if I come up some workaround in this thread.</p>",
          "rawMarkdown": "Hi,\n\nThanks for pointing out qsub.\n\n I just tried by creating a ```mnist_cnn.sh``` which contains only one line \n```\npython mnist_cnn.py\n```\n\nHowever, the task only stayed in the queue for less than 20 seconds (which means failed I guess). I will update if I come up some workaround in this thread.",
          "votes": 1
        }
      ]
    },
    {
      "id": 168922,
      "postDate": "2017-03-18T17:14:45.827Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 168939,
      "author_name": "Spencer",
      "author_url": "",
      "post_date": "2017-03-18T18:21:18.120000",
      "content": "<p>To those users not familiar with the cluster (like me):</p>\n\n<p>As the <a href=\"https://access.colfaxresearch.com/?p=compute\">document</a> says, we have to use the scheduler to run long programs on the computing node.\nWe use <code>qsub</code> to submit our jobs, which could be wrapped in a shell script.</p>\n\n<p>One thing I did not notice is that we need to change the path to absolute path instead of relative path.\nTake my example, I use conda to isolate my packages, and my python script is named <code>main.py</code></p>\n\n<p>Then I needed to write a shell script, let's call it <code>main.sh</code>, and write the following lines</p>\n\n<p><code>source activate py35env</code></p>\n\n<p><code>python [the absolute path]/main.py</code></p>\n\n<p>Then send the task (the shell script) to the scheduler by typing </p>\n\n<p><code>qsub main.sh</code></p>\n\n<p>One should remember that in the Python script you also need to use absolute pathes.</p>\n\n<p>-Spencer</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 168859,
      "author_name": "kambarakun",
      "author_url": "",
      "post_date": "2017-03-18T11:38:55.870000",
      "content": "<p>Do you use <code>qsub</code> command? The <code>qsub</code> command requests job queues. <br>\nRef: <a href=\"https://access.colfaxresearch.com/?p=compute\">https://access.colfaxresearch.com/?p=compute</a> <br>\nThe python process which is spending a long time, will be terminated (maybe?), if is not by <code>qsub</code> command. <br>\nI was also troubled by this. The usage of <code>qsub</code> command is described in <a href=\"https://www.kaggle.com/kambarakun/intel-mobileodt-cervical-cancer-screening/how-to-start-with-python-on-colfax-cluster\">my kernel</a> too:)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 168863,
          "author_name": "Spencer",
          "author_url": "",
          "post_date": "2017-03-18T11:59:55.600000",
          "content": "<p>Hi,</p>\n\n<p>Thanks for pointing out qsub.</p>\n\n<p>I just tried by creating a <code>mnist_cnn.sh</code> which contains only one line \n<code>\npython mnist_cnn.py\n</code></p>\n\n<p>However, the task only stayed in the queue for less than 20 seconds (which means failed I guess). I will update if I come up some workaround in this thread.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 168922,
      "author_name": "",
      "author_url": "",
      "post_date": "2017-03-18T17:14:45.827000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "168939": "To those users not familiar with the cluster (like me):\n\nAs the [document][1] says, we have to use the scheduler to run long programs on the computing node.\nWe use ```qsub``` to submit our jobs, which could be wrapped in a shell script.\n\nOne thing I did not notice is that we need to change the path to absolute path instead of relative path.\nTake my example, I use conda to isolate my packages, and my python script is named ```main.py```\n\nThen I needed to write a shell script, let's call it ```main.sh```, and write the following lines\n\n\n```source activate py35env```\n\n```python [the absolute path]/main.py```\n\n\nThen send the task (the shell script) to the scheduler by typing \n\n```qsub main.sh```\n\nOne should remember that in the Python script you also need to use absolute pathes.\n\n-Spencer\n\n\n  [1]: https://access.colfaxresearch.com/?p=compute",
    "168851": "I am trying to run keras with TensorFlow backend on the colfax server. \n\nHowever, my process was always killed by the server, even the official mnist example.\n\nBelow are the error messages I got when I tested the example\n\n```\nUsing TensorFlow backend.\n\nDownloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n\nX_train shape: (60000, 28, 28, 1)\n\n60000 train samples\n\n10000 test samples\n\n/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3), padding=\"valid\", input_shape=(28, 28, 1...)`\n  '` call to the Keras 2 API: ' + signature)\n\n/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/legacy/interfaces.py:86: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3))`\n  '` call to the Keras 2 API: ' + signature)\n\n/home/u2601/.conda/envs/py35env/lib/python3.5/site-packages/keras/models.py:826: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n  warnings.warn('The `nb_epoch` argument in `fit` '\n\nTrain on 60000 samples, validate on 10000 samples\n\nEpoch 1/12\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n\nW tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n\n18304/60000 [========>.....................] - ETA: 43s - loss: 0.7248 - acc: 0.7711Killed\n```\n\nIs Keras+TensorFlow banned by colfax? Or is there any workaround to run Keras on the colfax server?\n\nThanks!",
    "168859": "Do you use `qsub` command? The `qsub` command requests job queues.  \nRef: [https://access.colfaxresearch.com/?p=compute](https://access.colfaxresearch.com/?p=compute)  \nThe python process which is spending a long time, will be terminated (maybe?), if is not by `qsub` command.  \nI was also troubled by this. The usage of `qsub` command is described in [my kernel](https://www.kaggle.com/kambarakun/intel-mobileodt-cervical-cancer-screening/how-to-start-with-python-on-colfax-cluster) too:)",
    "168922": ""
  }
}