Explicitly Manage Dependencies

Problem

You want to exclude or add dependencies to the list of streamed dependencies.

Solution

To add dependencies, call the add_dependency() method on the job.

In this example, the name of the module is “MyLibrary” and the name of the file is “MyClass”. Additionally, add the path to the directory where the dependency is located to the PYTHONPATH environment variable.

 1from agiparallel.client import ClusterJobScheduler
 2import importlib
 3
 4
 5class NeedsAdditionalFilesTask:
 6    def execute(self):
 7        MyClass = importlib.import_module("MyLibrary.MyClass")
 8
 9
10def add_dependency_example():
11    with ClusterJobScheduler("localhost") as scheduler:
12        scheduler.connect()
13        job = scheduler.create_job()
14
15        # add the dependency by specifying the module
16        job.add_dependency("MyLibrary")
17
18        job.add_task(NeedsAdditionalFilesTask())
19
20        job.submit()
21        job.wait_until_done()
22
23
24if __name__ == "__main__":
25    add_dependency_example()

Discussion

Internally, the dependencies needed to run a task are automatically detected and the dependencies needed to run a task are automatically streamed. At times, additional control over this automatic behavior may be needed. For instance, client applications may not have a dependency on some required dependencies if they are using dependency injection or reflection to get types, so it will be necessary to send the dependencies manually.

See Also

Reference