Key concepts

There are many options on a job. All of the properties are optional. Below is a table of all the options and their default values.

• There is a performance overhead when submitting jobs. When jobs are submitted to the job scheduler, the jobs are sent with their jar files. If possible, packing more tasks per job is a way to decrease this overhead.

Here is an annotated example:

try(IJobScheduler scheduler = new ClusterJobScheduler("localhost")) {
    scheduler.connect();

    // Create job using IJobScheduler.createJob
    Job job = scheduler.createJob();

    // Add one or more tasks
    job.addTask(new SimpleTask(1, 1));

    // Set the task environment
    job.setTaskEnvironment(new MyEnvironment());

    // Set the various job options
    job.setName("My job name");
    job.setDescription("My job description");
    job.setMaxTaskInterruptedRetryAttempts((short) 0);
    job.setPriority(JobPriority.LOW);
    job.getTaskPreconditions().add(CommonResources.AVAILABLE_CORES, Operator.GREATER_THAN, 2);

    // Submit task to job scheduler
    job.submit();

    // Wait for the results
    job.waitUntilDone();
}

Properties that can be set either before a task is submitted or when it is running:

Read-only properties set by the infrastructure:

A task can transition through several different states during its lifetime. Task states are represented by the TaskStatus enum.

All tasks start at the NotSubmitted state. After the task is submitted, the state changes to the Submitted state while the task sits in the coordinator queue. When a task is assigned to an agent for execution, its status changes to Assigned and then to Running once the Agent starts running the task. If no errors occur during execution of the task, the task status changes to Completed. If an uncaught exception is encountered, it goes into the Failed state. Below is a state transition diagram and a full table of task statuses.

• How should tasks be split? It has been found that splitting the workload into more units can yield better performance. A common problem in parallel applications is that the workload is split more coarsely than is optimal. In some cases, the performance of the task becomes equal to the "time it takes for the slowest task to complete". As with any performance tip, measure, measure, measure.

• Design tasks so they are decoupled from the application code as much as possible. Try not to store application logic in the Task class. This will prove helpful when refactoring and testing code.

• If certain fields on the task instance are not used, ignore or remove the fields to reduce the size of the object being sent. In Java, mark unused fields with the transient keyword to avoid serialization.

Here is an annotated example:

public static class SimpleTask extends Task {
    private int a;
    private int b;

    public SimpleTask(int first, int second) {
        this.a = first;
        this.b = second;
    }

    @Override
    public void execute() {
        this.setResult(this.a + this.b);
    }
}

A task environment is set up only once per host process. After a process sets up a task environment, the process remains idle until a task specifying the task environment is assigned to the process for execution. In other words, a host runs one task environment and multiple tasks. After each task is executed, the host checks whether any of the task environment's RecycleSettings conditions have been satisfied. If any one of the conditions have been met, the process is recycled. The Agent Tray Application exposes a user interface for configuring an agent's host recycle settings. The values set in the UI will be overridden if a submitted job's TaskEnvironment's RecycleSettings have been set programmatically. A task environment can also be recycled if the agent needs to free up a process to run an incoming task that has a different task environment. For more information, read Host Assignment and Recycling.

To identify whether two task environment references are equal, a few properties of the task environment are checked. A task environment is uniquely identified via a combination of its Id and the AdditionalId properties. Two task environment identifications are equal if and only if both Id and AdditionalId properties are equal.

Here are a few examples to demonstrate. Below is code for a number of task environments:

public static class TaskEnvironmentA extends TaskEnvironment {
    public TaskEnvironmentA() {
        this.setId(UUID.fromString("94CFE45C-5EA9-4592-BB64-B83A5E72DB77"));
    }

    @Override
    public void setup() {}

    @Override
    public void teardown() {}
}

public static class TaskEnvironmentB extends TaskEnvironment {
    public TaskEnvironmentB() {
        this.setId(UUID.fromString("94CFE45C-5EA9-4592-BB64-B83A5E72DB77"));
    }

    @Override
    public void setup() {}

    @Override
    public void teardown() {}
}

public static class TaskEnvironmentC extends TaskEnvironment {
    public TaskEnvironmentC() {
        this.setId(UUID.fromString("94CFE45C-5EA9-4592-BB64-B83A5E72DB77"));
        this.setAdditionalId("ham");
    }

    @Override
    public void setup() {}

    @Override
    public void teardown() {}
}

public static class TaskEnvironmentD extends TaskEnvironment {
    public TaskEnvironmentD() {
        this.setAdditionalId("ham");
    }

    @Override
    public void setup() {}

    @Override
    public void teardown() {}
}

public static class TaskEnvironmentE extends TaskEnvironment {
    public TaskEnvironmentE() {
        this.setAdditionalId("ham");
    }

    @Override
    public void setup() {}

    @Override
    public void teardown() {}
}

public static class TaskEnvironmentF extends TaskEnvironment {
    public TaskEnvironmentF() {
        this.setId(UUID.fromString("B766B6B0-232A-4878-A7CF-82F354E117D4"));
    }

    @Override
    public void setup() {}

    @Override
    public void teardown() {}
}

Imagine if there was a hypothetical method called TaskEnvironmmentIsEqual, which compares whether two task environments have the same identification reference. The results are below with the explanations of their results in the comments.

// IsEqual: true
// Reason:  Both Id and AdditionalId properties are the same.
taskEnvironmentIsEqual(new TaskEnvironmentA(), new TaskEnvironmentB());
// IsEqual: false
// Reason:  Id properties are not the same.
taskEnvironmentIsEqual(new TaskEnvironmentA(), new TaskEnvironmentF());
// IsEqual: false
// Reason:  Id properties are not the same, even though AdditionalId properties are the same.
taskEnvironmentIsEqual(new TaskEnvironmentA(), new TaskEnvironmentC());
// IsEqual: false
// Reason:  Additional properties are the same. But the Id is by default random so they are not equal.
taskEnvironmentIsEqual(new TaskEnvironmentD(), new TaskEnvironmentE());

Here is an annotated example:

public static class MyEnvironment extends TaskEnvironment {
    @Override
    public void setup() {
        // Put your common setup logic in the setup method.
    }

    @Override
    public void teardown() {
        // Put your common teardown logic in the teardown method.
    }
}