If you are using Colfax Cluster provided, probably you use qsub tool to submit your huge job to the cluster. Then you wait until the task is terminated and you can look to STDIN.oXXXX or STDIN.eXXXX (or your customized output name).
However, there is an option to launch qsub in the interactive mode and execute the code on node directly.
Terminal
\n\nHere is the magic command that you can run once connected to the cluster :
\n\n\nqsub -V -I interactive_qsub\n
where interactive_qsub file is
#PBS -l nodes=1:knl7210:ram96gb -N interactive_qsub -d /path/to/your/home
Jupyter-Notebook
\n\nAnother option is to run a code from Jupyter-Notebook once you are running jupyter on the cluster without browser and visualize the interface on your local machine. Follow this for more details.
\n\nThus, you can, for example run caffe training on a dataset and follow the progress. It can be done like this. Write the following in a jupyter cell and execute:
\n\nargs_str = \" \".join(['train', '-solver', solver_filename, '-weights', weights_filename])
qsub_script = \"\\\"#PBS -V -I -x -N caffe_train -l nodes=1:knl7210:ram96gb\\ncaffe %s\\\"\" % args_str
!echo {qsub_script} > qsub_script
!chmod +x qsub_script
!qsub $PWD/qsub_script
This script submits a job to the cluster and runs created script file qsub_script in the interactive mode.
See here for more details and have fun !
\n\nPS. Thank you Intel for providing freely such a thing
", "messages": [ { "id": "169587", "postDate": "03/21/2017 16:44:49", "content": "If you are using Colfax Cluster provided, probably you use qsub tool to submit your huge job to the cluster. Then you wait until the task is terminated and you can look to STDIN.oXXXX or STDIN.eXXXX (or your customized output name).
However, there is an option to launch qsub in the interactive mode and execute the code on node directly.
Terminal
\n\nHere is the magic command that you can run once connected to the cluster :
\n\n\nqsub -V -I interactive_qsub\n
where interactive_qsub file is
#PBS -l nodes=1:knl7210:ram96gb -N interactive_qsub -d /path/to/your/home
Jupyter-Notebook
\n\nAnother option is to run a code from Jupyter-Notebook once you are running jupyter on the cluster without browser and visualize the interface on your local machine. Follow this for more details.
\n\nThus, you can, for example run caffe training on a dataset and follow the progress. It can be done like this. Write the following in a jupyter cell and execute:
\n\nargs_str = \" \".join(['train', '-solver', solver_filename, '-weights', weights_filename])
qsub_script = \"\\\"#PBS -V -I -x -N caffe_train -l nodes=1:knl7210:ram96gb\\ncaffe %s\\\"\" % args_str
!echo {qsub_script} > qsub_script
!chmod +x qsub_script
!qsub $PWD/qsub_script
This script submits a job to the cluster and runs created script file qsub_script in the interactive mode.
See here for more details and have fun !
\n\nPS. Thank you Intel for providing freely such a thing
", "rawMarkdown": "If you are using Colfax Cluster provided, probably you use `qsub` tool to submit your *huge* job to the cluster. Then you wait until the task is terminated and you can look to STDIN.oXXXX or STDIN.eXXXX (or your customized output name). \n\nHowever, there is an option to launch `qsub` in the interactive mode and execute the code on node directly. \n\n\n**Terminal** \n\nHere is the magic command that you can run once connected to the cluster : \n\n```\nqsub -V -I interactive_qsub\n```\n\nwhere `interactive_qsub` file is\n\n```#PBS -l nodes=1:knl7210:ram96gb -N interactive_qsub -d /path/to/your/home```\n\n**Jupyter-Notebook** \n\nAnother option is to run a code from *Jupyter-Notebook* once you are running jupyter on the cluster without browser and visualize the interface on your local machine. Follow [this][2] for more details. \n\nThus, you can, for example run caffe training on a dataset and follow the progress. It can be done like this. Write the following in a jupyter cell and execute:\n\n```args_str = \" \".join(['train', '-solver', solver_filename, '-weights', weights_filename])```\n\n```qsub_script = \"\\\"#PBS -V -I -x -N caffe_train -l nodes=1:knl7210:ram96gb\\ncaffe %s\\\"\" % args_str```\n\n```!echo {qsub_script} > qsub_script```\n\n```!chmod +x qsub_script```\n\n```!qsub $PWD/qsub_script```\n\nThis script submits a job to the cluster and runs created script file `qsub_script` in the interactive mode. \n\n\nSee [here][1] for more details and have fun !\n\nPS. Thank you Intel for providing freely such a thing\n\n\n\n [1]: http://docs.adaptivecomputing.com/torque/5-1-3/Content/topics/torque/commands/qsub.htm\n\n [2]: https://www.kaggle.com/kambarakun/intel-mobileodt-cervical-cancer-screening/how-to-start-with-python-on-colfax-cluster", "votes": null }, { "id": "169590", "postDate": "03/21/2017 16:53:18", "content": "Each node has 64*4 cores. If you request 4 nodes you're requesting a total of 1024 cores distributed through 4 worker nodes. These resources are all dedicated exclusively to you which is probably not adequate for interactive use through Jupyter, etc.
", "rawMarkdown": "Each node has 64*4 cores. If you request 4 nodes you're requesting a total of 1024 cores distributed through 4 worker nodes. These resources are all dedicated exclusively to you which is probably not adequate for interactive use through Jupyter, etc.", "votes": null }, { "id": "169591", "postDate": "03/21/2017 16:58:36", "content": "I thought about directly launching scripts and observing the stdout without waiting.
\n\nI'm not sure about the possibility of launching Jupyter inside the job and port forwarding up to ones local machine.
", "rawMarkdown": "I thought about directly launching scripts and observing the stdout without waiting. \n\nI'm not sure about the possibility of launching Jupyter inside the job and port forwarding up to ones local machine.", "votes": null }, { "id": "169594", "postDate": "03/21/2017 17:02:40", "content": "If you request multiple nodes you need to make your job use all nodes (that doesn't work transparently). For example, by accessing $PBSNODES and making your code distributed. If you don't do that you're running on a single node and having 3 idle nodes. If your code is not prepared to run on multiple nodes use #PBS -l nodes=1. If everyone runs the command above without knowing what's going on then I'm afraid there won't be resources for everybody.
", "rawMarkdown": "If you request multiple nodes you need to make your job use all nodes (that doesn't work transparently). For example, by accessing $PBSNODES and making your code distributed. If you don't do that you're running on a single node and having 3 idle nodes. If your code is not prepared to run on multiple nodes use **#PBS -l nodes=1**. If everyone runs the command above without knowing what's going on then I'm afraid there won't be resources for everybody.", "votes": null }, { "id": "169596", "postDate": "03/21/2017 17:07:57", "content": "Right, it is not evident. I changed it in the text to prevent this overload. But anyway it is up to all of us to use properly the resources. And hopefully, they provide Intel flavoured Caffe which should be adapted to the cluster and the nodes.
", "rawMarkdown": "Right, it is not evident. I changed it in the text to prevent this overload. But anyway it is up to all of us to use properly the resources. And hopefully, they provide Intel flavoured Caffe which should be adapted to the cluster and the nodes.", "votes": null }, { "id": "169891", "postDate": "03/23/2017 06:16:45", "content": "\n\n\nI think, we can use only 1 node on Colfax.
\n
\n If you request 4 node, you can use only 1 node.
I'm very sorry, I was wrong!
\nIt was something wrong that I did try before:(
However, so that, it may be a problem of nothing of the guidelines to use nodes.
\n\nAnyways, thank you so much to make this discussion!!
", "rawMarkdown": "> I think, we can use only 1 node on Colfax. \n> If you request 4 node, you can use only 1 node.\n\nI'm very sorry, I was wrong! \nIt was something wrong that I did try before:( \n\nHowever, so that, it may be a problem of nothing of the guidelines to use nodes. \n\nAnyways, thank you so much to make this discussion!!", "votes": null }, { "id": "169928", "postDate": "03/23/2017 09:13:53", "content": "Again, it depends on the code you executing. Basic example of Colfax documentation shows that you can use all requested nodes:
\n\n$ qsub -V -I -x ~/tmp/distrJob \nqsub: waiting for job 4171.c001 to start\nqsub: job 4171.c001 ready\n###################################################################\n# Colfax Cluster - https://colfaxresearch.com/\n# Date: Thu Mar 23 02:00:20 PDT 2017\n# Job ID: 4171.c001\n# User: USER\n# Resources: neednodes=4:knl,nodes=4:knl,walltime=24:00:00\n###################################################################\nLaunching the parallel job from mother superior c001-n038...\n[1] DAPL startup: RLIMIT_MEMLOCK too small\n[2] DAPL startup: RLIMIT_MEMLOCK too small\n[3] DAPL startup: RLIMIT_MEMLOCK too small\nHello world from host c001-n040 (rank 2)!\nHello world from host c001-n038 (rank 0)!\nHello world from host c001-n041 (rank 3)!\nHello world from host c001-n039 (rank 1)!\nqsub: job 4171.c001 completed\nThey provide also caffe with mlsl options at /opt/caffe-mlsl, so caffe funs have a huge advantage :)
Each node has 64*4 cores. If you request 4 nodes you're requesting a total of 1024 cores distributed through 4 worker nodes. These resources are all dedicated exclusively to you which is probably not adequate for interactive use through Jupyter, etc.
", "votes": null, "replies": [ { "id": 169591, "author_name": "vfdev5", "author_url": "", "post_date": "03/21/2017 16:58:36", "content": "I thought about directly launching scripts and observing the stdout without waiting.
\n\nI'm not sure about the possibility of launching Jupyter inside the job and port forwarding up to ones local machine.
", "votes": null, "replies": [] }, { "id": 169594, "author_name": "aamaia", "author_url": "", "post_date": "03/21/2017 17:02:40", "content": "If you request multiple nodes you need to make your job use all nodes (that doesn't work transparently). For example, by accessing $PBSNODES and making your code distributed. If you don't do that you're running on a single node and having 3 idle nodes. If your code is not prepared to run on multiple nodes use #PBS -l nodes=1. If everyone runs the command above without knowing what's going on then I'm afraid there won't be resources for everybody.
", "votes": null, "replies": [] }, { "id": 169596, "author_name": "vfdev5", "author_url": "", "post_date": "03/21/2017 17:07:57", "content": "Right, it is not evident. I changed it in the text to prevent this overload. But anyway it is up to all of us to use properly the resources. And hopefully, they provide Intel flavoured Caffe which should be adapted to the cluster and the nodes.
", "votes": null, "replies": [] }, { "id": 169891, "author_name": "kambarakun", "author_url": "", "post_date": "03/23/2017 06:16:45", "content": "\n\n\nI think, we can use only 1 node on Colfax.
\n
\n If you request 4 node, you can use only 1 node.
I'm very sorry, I was wrong!
\nIt was something wrong that I did try before:(
However, so that, it may be a problem of nothing of the guidelines to use nodes.
\n\nAnyways, thank you so much to make this discussion!!
", "votes": null, "replies": [] }, { "id": 169928, "author_name": "vfdev5", "author_url": "", "post_date": "03/23/2017 09:13:53", "content": "Again, it depends on the code you executing. Basic example of Colfax documentation shows that you can use all requested nodes:
\n\n$ qsub -V -I -x ~/tmp/distrJob \nqsub: waiting for job 4171.c001 to start\nqsub: job 4171.c001 ready\n###################################################################\n# Colfax Cluster - https://colfaxresearch.com/\n# Date: Thu Mar 23 02:00:20 PDT 2017\n# Job ID: 4171.c001\n# User: USER\n# Resources: neednodes=4:knl,nodes=4:knl,walltime=24:00:00\n###################################################################\nLaunching the parallel job from mother superior c001-n038...\n[1] DAPL startup: RLIMIT_MEMLOCK too small\n[2] DAPL startup: RLIMIT_MEMLOCK too small\n[3] DAPL startup: RLIMIT_MEMLOCK too small\nHello world from host c001-n040 (rank 2)!\nHello world from host c001-n038 (rank 0)!\nHello world from host c001-n041 (rank 3)!\nHello world from host c001-n039 (rank 1)!\nqsub: job 4171.c001 completed\nThey provide also caffe with mlsl options at /opt/caffe-mlsl, so caffe funs have a huge advantage :)