Include or remove additional third party libraries

You want to include or exclude jars to the list of jars sent in a job.

By default, any jars that might be needed for the job will be sent. In some cases, this may not work or become inefficient. There are a few ways to add or remove third party jars that are sent in a job. There are three categories of when the list of streamed jars need to be modified:

Add or remove libraries individually using the additional / excluded list

Add or remove jars explicitly by adding them to the additional jar collection returned from Job.getAdditionalJars() and Job.getExcludedJars() respectively. The strings that the collection expects are the full or relative paths to the location of the jar files.

Tip
The values in the additional jar collection can be separated by the value of Java's path.separator property. For instance, the value "lib\JarA.jar;lib\JarB.jar" will add both JarA.jar and JarB.jar to the list of jars on Windows. This is useful when getting the path from another Java property.

Modify the list of streamed Java libraries by using the java.class.path

The value of Java's java.class.path property determines the list of sent jars. For instance, adding a new jar to the -classpath option when starting the JVM will add the jar to the list of those sent. The java.class.path property can also be dynamically set and modified. The system will read the java.class.path property and send the specified jars.

Tip
To read more about how to set the Java class path, see the documentation here.

Disable sending Java jars automatically and install them manually on each agent.

Jobs are executed in separate JVMs and third-party libraries are often needed in tasks. Java jar dependencies are automatically sent by default. Although this works in many cases, there are times when jars need to be manually added or removed. For instance, in many java application servers the value of the java.class.path property does not contain the jars that are needed for the task. Instead, the jars are added dynamically by a class loader. This requires user intervention. In these cases, it is important to understand the process behind deciding which jars to send.

Here are the steps used to build the list of jars to send with the job:

1. The jar that the Task and TaskEnvironment is run from (if any) are added.
2. All jars that are currently in Java's java.class.path property are added.
3. Any jars that are in the excluded jar collection are removed.
4. Any jars that are in the additional jar collection are added.

A file will only be added if it ends with a "jar" suffix. Duplicate jar names will be ignored. That is a path such as java -classpath C:\Jars\foo.jar;foo.jar will only send the jar at C:\Jars\foo.jar.

The only exception to this rule is if the special "*" value is added to the excluded jar collection. In this case, no jars are added except the jars in the additional jar collection.

Native libraries can also be added to a job. However unlike jars, native libraries are not sent automatically. Fortunately, it is easy to add them if needed. Add the native library to the additional native libries collection that is returned from Job.getAdditionalNativeLibraries(). A very common pattern developers use is to add the value of java.library.path. This will have all the native libraries the client has. This is the manual process of doing what is done automatically with jars and java.class.path.

Note
A common mistake that some developers make is adding the native library to the additional jar collection. A native library added to the additional jar collection will be ignored. Make sure to add the native library to the correct collection.

Putting it all together

Now that the different rules for the list of jars have been reviewed, try to work through a few examples.

Here is the code for an example:

package stkparallelcomputingserversdk.howto;

import java.util.List;

import agi.parallel.client.*;
import agi.parallel.infrastructure.*;

public class AddingThirdPartyLibraries {
    public static void main(String[] args) {
        try (IJobScheduler scheduler = new ClusterJobScheduler("localhost")) {
            scheduler.connect();

            Job job = scheduler.createJob();
            job.addTask(new SimpleTask());

            List<String> additionalJarList = job.getAdditionalJars();
            additionalJarList.add("C:\\Jars\\CompanyA.jar;CompanyC.jar");

            List<String> excludedJarList = job.getExcludedJars();
            excludedJarList.add("CompanyB.jar");

            List<String> additionalNativeLibrariesList = job.getAdditionalNativeLibraries();
            additionalNativeLibrariesList.add("CompanyD.dll");
            additionalNativeLibrariesList.add(System.getProperty("java.library.path"));

            job.submit();
            job.waitUntilDone();
        }
    }

    public static class SimpleTask extends Task {
        @Override
        public void execute() {
        }
    }
}

Assuming that code is in a jar called AddingThirdPartyLibraries.jar, let's go through a few scenarios

For this command:

java -classpath AddingThirdPartyLibraries.jar;agi.parallel.client-2.9.0.jar AddingThirdPartyLibraries

What are the jars that are sent with the job?

1. AddingThirdPartyLibraries.jar
2. C:\\Jars\\CompanyA.jar
3. CompanyC.jar
4. CompanyD.dll

Why?

AddingThirdPartyLibraries.jar is sent because it is the jar that contains the task. CompanyA.jar and CompanyC.jar are sent because those were added manually. CompanyD.dll (a native library) is also sent because it was added manually.

For this command:

java -classpath AddingThirdPartyLibraries.jar;agi.parallel.client-2.9.0.jar;Company1.jar;Company2.jar;CompanyB.jar AddingThirdPartyLibraries

What are the jars that are sent with the job?

The same jars as the example above, with the addition of Company1.jar and Company2.jar.

Why not CompanyB?

Because it was explicitly excluded when it was added to the excluded jar list.

For this command:

set CLASSPATH=AddingThirdPartyLibraries.jar;agi.parallel.client-2.9.0.jar;Company1.jar;Company2.jar;CompanyB.jar
java AddingThirdPartyLibraries

What are the jars sent with the job?

Exactly the same as above. Again, it does not matter how the java.class.path is specified.

Other Resources