Explicitly Manage Dependencies¶
Problem¶
You want to exclude or add dependencies to the list of streamed dependencies.
Solution¶
To add dependencies, call the add_dependency()
method on
the job.
In this example, the name of the module is “MyLibrary” and the name of the file is “MyClass”. Additionally, add the path to the directory where the dependency is located to the PYTHONPATH environment variable.
1from agiparallel.client import ClusterJobScheduler
2import importlib
3
4
5class NeedsAdditionalFilesTask:
6 def execute(self):
7 MyClass = importlib.import_module("MyLibrary.MyClass")
8
9
10def add_dependency_example():
11 with ClusterJobScheduler("localhost") as scheduler:
12 scheduler.connect()
13 job = scheduler.create_job()
14
15 # add the dependency by specifying the module
16 job.add_dependency("MyLibrary")
17
18 job.add_task(NeedsAdditionalFilesTask())
19
20 job.submit()
21 job.wait_until_done()
22
23
24if __name__ == "__main__":
25 add_dependency_example()
Discussion¶
Internally, the dependencies needed to run a task are automatically detected and the dependencies needed to run a task are automatically streamed. At times, additional control over this automatic behavior may be needed. For instance, client applications may not have a dependency on some required dependencies if they are using dependency injection or reflection to get types, so it will be necessary to send the dependencies manually.