I started creating a small distributed computing framework in Python to use some of the extra clock-cycles laying around my house. I decided to open-source it so anyone could use it. The link is below, i'll give a short synopsis of PyCompute below the link.
\r\nhttp://hesamrabeti.github.com/PyCompute/
\r\nI’ve architected PyCompute to be a light-weight centralized task scheduling system. PyCompute has two components: PyComputeServer and PyComputeClient
\r\nPyComputeServer
\r\nThis component is used to schedule tasks and provide for a checkpointing/bottleneck condition. PyComputeServer.addTask() is used to schedule tasks to be dispatched to PyComputeClients. PyComputeServer.finish() blocks execution\r\n until all tasks scheduled have been completed.
\r\nPyComputeClient
\r\nThis component is the workhorse of PyCompute. It connects to a PyComputeServer using good old sockets, allowing the use of geographically dispersed computers. PyComputeClient will connect to the server address specified and proceed to request\r\n tasks, perform the tasks, and confirm the completion of said tasks.
\r\nAs an example, the way I will be using PyCompute for the flight quest will be to design a Python program which will instantiate a PyComputeServer on one computer. My data will be stored on a separate computer on an SQL server. My other\r\n computers and my friends and family’s computers will have PyComputeClient installed on them and they will point to my\r\nPyComputeServer. The PyComputeClients will request tasks from the server. For example, a task will be to calculate the average gate delay of a certain airport.\r\nPyComputeClients will have read-only access to the SQL server containing the data and will form queries to obtain the information necessary to complete the task. After each task is finished, the client will output their results to a different schema\r\n on the SQL server. This can also be used on AWS or any other cloud provider.
\r\nThe use of SQL is not necessary. The only thing set in stone is the fact that tasks are scheduled through\r\nPyComputeServer and a True or a False must be returned by the task. After the client has received the task, there are no limitations set on the actual execution of the task.
\r\nOne convenient feature of PyCompute is that the clients stay active even after the\r\nPyComputeServer goes down. In addition, PyComputeClients will reset themselves, request new code, and execute the new code each time they connect to the server. This way, one can modify the client’s task code and have the code propagate to\r\n clients automatically.
\r\n\r\n
Please note this is very much a beta version. I've only been working on this for a day with minimal validation and lots of hacks. So please let me know if you find any bugs or have any feedback. A word of caution: I did not pay much attention to security,\r\n so take the neccesary cautions and make sure you understand the implications of using this software.
\r\n\r\n
\r\n
", "messages": [ { "id": "17534", "postDate": "12/08/2012 09:56:34", "content": "
I started creating a small distributed computing framework in Python to use some of the extra clock-cycles laying around my house. I decided to open-source it so anyone could use it. The link is below, i'll give a short synopsis of PyCompute below the link.
\r\nhttp://hesamrabeti.github.com/PyCompute/
\r\nI’ve architected PyCompute to be a light-weight centralized task scheduling system. PyCompute has two components: PyComputeServer and PyComputeClient
\r\nPyComputeServer
\r\nThis component is used to schedule tasks and provide for a checkpointing/bottleneck condition. PyComputeServer.addTask() is used to schedule tasks to be dispatched to PyComputeClients. PyComputeServer.finish() blocks execution\r\n until all tasks scheduled have been completed.
\r\nPyComputeClient
\r\nThis component is the workhorse of PyCompute. It connects to a PyComputeServer using good old sockets, allowing the use of geographically dispersed computers. PyComputeClient will connect to the server address specified and proceed to request\r\n tasks, perform the tasks, and confirm the completion of said tasks.
\r\nAs an example, the way I will be using PyCompute for the flight quest will be to design a Python program which will instantiate a PyComputeServer on one computer. My data will be stored on a separate computer on an SQL server. My other\r\n computers and my friends and family’s computers will have PyComputeClient installed on them and they will point to my\r\nPyComputeServer. The PyComputeClients will request tasks from the server. For example, a task will be to calculate the average gate delay of a certain airport.\r\nPyComputeClients will have read-only access to the SQL server containing the data and will form queries to obtain the information necessary to complete the task. After each task is finished, the client will output their results to a different schema\r\n on the SQL server. This can also be used on AWS or any other cloud provider.
\r\nThe use of SQL is not necessary. The only thing set in stone is the fact that tasks are scheduled through\r\nPyComputeServer and a True or a False must be returned by the task. After the client has received the task, there are no limitations set on the actual execution of the task.
\r\nOne convenient feature of PyCompute is that the clients stay active even after the\r\nPyComputeServer goes down. In addition, PyComputeClients will reset themselves, request new code, and execute the new code each time they connect to the server. This way, one can modify the client’s task code and have the code propagate to\r\n clients automatically.
\r\n\r\n
Please note this is very much a beta version. I've only been working on this for a day with minimal validation and lots of hacks. So please let me know if you find any bugs or have any feedback. A word of caution: I did not pay much attention to security,\r\n so take the neccesary cautions and make sure you understand the implications of using this software.
\r\n\r\n
\r\n
", "rawMarkdown": "", "votes": null }, { "id": "17544", "postDate": "12/09/2012 01:52:48", "content": "
I added exampleApplication.py which creates a server and dispatches some very simple tasks. Also, added a guide.txt for a quick how-to of running the example.
", "rawMarkdown": "", "votes": null }, { "id": "17565", "postDate": "12/09/2012 17:56:48", "content": "Very cool. Ill give this a try as soon as my exams are over. Thanks!
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 17544, "author_name": "rabeti", "author_url": "", "post_date": "12/09/2012 01:52:48", "content": "I added exampleApplication.py which creates a server and dispatches some very simple tasks. Also, added a guide.txt for a quick how-to of running the example.
", "votes": null, "replies": [] }, { "id": 17565, "author_name": "miketery", "author_url": "", "post_date": "12/09/2012 17:56:48", "content": "Very cool. Ill give this a try as soon as my exams are over. Thanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "17534": "", "17544": "", "17565": "" }, "source": "meta" }