Launch a Work Queue factory.
The command line arguments for `work_queue_factory` can be set for a
factory object (with dashes replaced with underscores). Creating a factory
object does not immediately launch it, so this is a good time to configure
the resources, number of workers, etc. Factory objects function as Python
context managers, so to indicate that a set of commands should be run with
a factory running, wrap them in a `with` statement. The factory will be
cleaned up automtically at the end of the block. You can also make
config changes to the factory while it is running. As an example,
# normal WQ setup stuff
workers = work_queue.Factory("sge", "myproject")
workers.cores = 4
with workers:
# submit some tasks
workers.max_workers = 300
# got a pile of tasks, allow more workers
# any additional cleanup steps on the manager
def work_queue.Factory.__init__ |
( |
|
self, |
|
|
|
batch_type, |
|
|
|
manager_name = None , |
|
|
|
manager_host_port = None , |
|
|
|
factory_binary = None , |
|
|
|
worker_binary = None , |
|
|
|
log_file = os.devnull |
|
) |
| |
Create a factory for the given batch_type and manager name.
manager_name or, manager_host_port should be specified.
If factory_binary or worker_binary is not
specified, $PATH will be searched.
References work_queue.Factory._config_file, work_queue.Factory._construct_command_line(), work_queue.Factory._factory_binary, work_queue.Factory._factory_proc, work_queue.Factory._find_exe(), work_queue.Factory._log_file, work_queue.Factory._opts, work_queue.Factory._set_manager(), work_queue.Factory._write_config(), and rmsummary.snapshots_count.