Key concepts

There are many options on a job. All of the properties are optional. Below is a table of all the options and their default values.

• There is a performance overhead when submitting jobs. When jobs are submitted to the job scheduler, the jobs are sent with their .NET assemblies. If possible, packing more tasks per job is a way to decrease this overhead.

Here is an annotated example:

using (IJobScheduler scheduler = new ClusterJobScheduler("localhost"))
{
    scheduler.Connect();

    // Create job using IJobScheduler.CreateJob
    Job job = scheduler.CreateJob();

    // Add one or more tasks
    job.AddTask(new SimpleTask(1, 1));

    // Set the task environment
    job.TaskEnvironment = new MyEnvironment();

    // Set the various job options
    job.Name = "My job name";
    job.Description = "My job description";
    job.MaxTaskInterruptedRetryAttempts = 0;
    job.Priority = JobPriority.Low;
    job.TaskPreconditions.Add(CommonResources.AvailableCores, Operator.GreaterThan, 2);

    // Submit task to job scheduler
    job.Submit();

    // Wait for the results
    job.WaitUntilDone();
}

Tasks are serialized as .NET object when submitted to a job scheduler. When a task is executed, a copy of the .NET object is deserialized and it's Execute method is called.

For the task to be sent to the host, the object must be serializable. Currently, only the BinaryFormatter is supported but additional formatters will be supported in the future. The developer must ensure the object can be serialized by the .NET Binary Serializer. For more information on .NET Binary Serialization, see the MSDN documentation on binary serialization.

Note
Note when the task is deserialized by .NET, the task's constructor will not be called.

Properties that can be set either before a task is submitted or when it is running:

Read-only properties set by the infrastructure:

A task can transition through several different states during its lifetime. Task states are represented by the TaskStatus enum.

All tasks start at the NotSubmitted state. After the task is submitted, the state changes to the Submitted state while the task sits in the coordinator queue. When a task is assigned to an agent for execution, its status changes to Assigned and then to Running once the Agent starts running the task. If no errors occur during execution of the task, the task status changes to Completed. If an uncaught exception is encountered, it goes into the Failed state. Below is a state transition diagram and a full table of task statuses.

• How should tasks be split? It has been found that splitting the workload into more units can yield better performance. A common problem in parallel applications is that the workload is split more coarsely than is optimal. In some cases, the performance of the task becomes equal to the "time it takes for the slowest task to complete". As with any performance tip, measure, measure, measure.

• Design tasks so they are decoupled from the application code as much as possible. Try not to store application logic in the Task class. This will prove helpful when refactoring and testing code.

• If certain fields on the task instance are not used, ignore or remove the fields to reduce the size of the object being sent. Otherwise, mark unused fields with the NonSerializedAttribute keyword to avoid serialization.

Here is an annotated example:

// You must always make the Task class serializable
[Serializable]
public class SimpleTask : Task
{
    // The instance of the class is automatically serialized with the task
    private int a;
    private int b;

    public SimpleTask(int first, int second)
    {
        a = first;
        b = second;
    }

    public override void Execute()
    {
        // Put your application logic in the Execute method
        Result = a + b;
    }
}

A task environment is set up only once per host process. After a process sets up a task environment, the process remains idle until a task specifying the task environment is assigned to the process for execution. In other words, a host runs one task environment and multiple tasks. After each task is executed, the host checks whether any of the task environment's RecycleSettings conditions have been satisfied. If any one of the conditions have been met, the process is recycled. The Agent Tray Application exposes a user interface for configuring an agent's host recycle settings. The values set in the UI will be overridden if a submitted job's TaskEnvironment's RecycleSettings have been set programmatically. A task environment can also be recycled if the agent needs to free up a process to run an incoming task that has a different task environment. For more information, read Host Assignment and Recycling.

To identify whether two task environment references are equal, a few properties of the task environment are checked. A task environment is uniquely identified via a combination of its Id and the AdditionalId properties. Two task environment identifications are equal if and only if both Id and AdditionalId properties are equal.

Here are a few examples to demonstrate. Below is code for a number of task environments:

[Serializable]
public class TaskEnvironmentA : TaskEnvironment
{
    public TaskEnvironmentA()
    {
        this.Id = new Guid("94CFE45C-5EA9-4592-BB64-B83A5E72DB77");
    }

    public override void Setup() { }
    public override void Teardown() { }
}

[Serializable]
public class TaskEnvironmentB : TaskEnvironment
{
    public TaskEnvironmentB()
    {
        this.Id = new Guid("94CFE45C-5EA9-4592-BB64-B83A5E72DB77");
    }

    public override void Setup() { }
    public override void Teardown() { }
}

[Serializable]
public class TaskEnvironmentC : TaskEnvironment
{
    public TaskEnvironmentC()
    {
        this.Id = new Guid("94CFE45C-5EA9-4592-BB64-B83A5E72DB77");
        this.AdditionalId = "ham";
    }

    public override void Setup() { }
    public override void Teardown() { }
}

[Serializable]
public class TaskEnvironmentD : TaskEnvironment
{
    public TaskEnvironmentD()
    {
        this.AdditionalId = "ham";
    }

    public override void Setup() { }
    public override void Teardown() { }
}

[Serializable]
public class TaskEnvironmentE : TaskEnvironment
{
    public TaskEnvironmentE()
    {
        this.AdditionalId = "ham";
    }

    public override void Setup() { }
    public override void Teardown() { }
}

[Serializable]
public class TaskEnvironmentF : TaskEnvironment
{
    public TaskEnvironmentF()
    {
        this.Id = new Guid("B766B6B0-232A-4878-A7CF-82F354E117D4");
    }

    public override void Setup() { }
    public override void Teardown() { }
}

Imagine if there was a hypothetical method called TaskEnvironmmentIsEqual, which compares whether two task environments have the same identification reference. The results are below with the explanations of their results in the comments.

// IsEqual: true
// Reason:  Both Id and AdditionalId properties are the same.
TaskEnvironmentIsEqual(new TaskEnvironmentA(), new TaskEnvironmentB());

// IsEqual: false
// Reason:  Id properties are not the same.
TaskEnvironmentIsEqual(new TaskEnvironmentA(), new TaskEnvironmentF());

// IsEqual: false
// Reason:  Id properties are not the same, even though AdditionalId properties are the same.
TaskEnvironmentIsEqual(new TaskEnvironmentA(), new TaskEnvironmentC());

// IsEqual: false
// Reason:  Additional properties are the same. But the Id is by default random so they are not equal.
TaskEnvironmentIsEqual(new TaskEnvironmentD(), new TaskEnvironmentE());

As with Tasks, the .NET binary serializer must be able to serialize user defined TaskEnvironments. Read this topic.

Here is an annotated example:

// You must always make the TaskEnvironment class serializable
[Serializable]
public class MyEnvironment : TaskEnvironment
{
    public override void Setup()
    {
        // Put your common setup logic in the Setup method
    }

    public override void Teardown()
    {
        // Put your common teardown logic in the Teardown method
    }
}