Dynamically Control Consumed Resources

Problem

You want to dynamically change the consumed resources of the task.

Solution

Use the task’s get_property() method with JOB_SCHEDULER_CONTEXT input. You can call yield_resource() to decrease the resources consumed by the task and reserve_resource() to increase the resources consumed by the task.

 1from agiparallel.client import ClusterJobScheduler
 2from agiparallel.constants import ConsumableResources, TaskProperties
 3
 4
 5class ResourcesTask():
 6
 7    def execute(self):
 8
 9        self.scheduler_context = self.get_property(TaskProperties.JOB_SCHEDULER_CONTEXT)
10
11        if not self.scheduler_context:
12            raise Exception('Could not get job scheduler context!')
13
14        # You can pass 0 to query the amount that the task is currently consuming
15        current_consumption = self.scheduler_context.yield_resource(ConsumableResources.CORES, 0)
16        print("Initial: Consuming " + str(current_consumption) + " Cores")
17
18        # Let's yield the resource to the global resource poll
19        current_consumption = self.scheduler_context.yield_resource(ConsumableResources.CORES, 1)
20        print("After YieldResource: Consuming " + str(current_consumption) + " Cores")
21
22        # We can also dynamically increase our resource consumption
23        current_consumption = self.scheduler_context.reserve_resource(ConsumableResources.CORES, 1)
24        print("After ReserveResource: Consuming " + str(current_consumption) + " Cores")
25
26        # Output should resemble:
27        # Initial: Consuming 1 Cores
28        # After YieldResource: Consuming 0 Cores
29        # After ReserveResource: Consuming 1 Cores
30
31
32def resource_manager_example():
33    with ClusterJobScheduler("localhost") as client:
34        client.connect()
35        job = client.create_job()
36        task = ResourcesTask()
37        job.add_task(task)
38        job.submit()
39        job.wait_until_done()
40
41        print(job.tasks[0].standard_output)
42
43
44if __name__ == "__main__":
45    resource_manager_example()

Discussion

Accurate resource tracking is important for efficiently using all available resources. A high degree of flexibility is allowed for more advanced cases where more control is needed for specifying which task is consuming which resource. Resource yielding comes in handy when a known task is not doing any real work and the system should consider its resources available to the system. The most common use case for this is when the task is waiting for work to be done. Resource reserving is much less common. Reserve a resource when more work is about to be done and the system should be notified.

Resources also have a concept of private and global availability. A global resource is the most common and the default. Global resources are available to all tasks – there are no restrictions. A private resource is a special kind of resource that restricts who can consume it, most commonly children of the task. Make a resource private by specifying True to the restrict_to_children parameter of yield_resource(). Resources from private resources are consumed first before consuming global resources. The most common reason for using private resources over global resources is to ensure child tasks will have a pool of resources available only to them. That is, using private resources can guarantee that a child task will not have to wait in the queue because it does not have the necessary resources.

Note

When a task submits a child job all of its resources are yielded by default. You will only have to yield resources if you choose to manually do so.

See Also

Other Resources