Internals

Understanding the task, host, and agent life cycle can be crucial to effectively writing applications. Most of these details are very technical and are not needed to successfully write an application. The information in this section can prove useful in helping to understand why the system behaves in certain ways.

In this section the following is explained:

Agent selection
Process isolation
Task workflow
Host assignment and recycling

Agent selection

The coordinator internally has a task queue. When a task is submitted to the coordinator, the task gets put in the queue. The task's order in the queue is based on the job's priority (JobPriority). The coordinator takes the first task from the queue whose preconditions (JobTaskPreconditions) are satisfied.

Once a task is taken from the queue, the coordinator assigns the task to an available agent based on the job's specified scheduling algorithm (JobAgentSelectionPreference). The default agent selection algorithm first sort agents based on priority, then it cycles through each agent until the agent's maximum task capacity is reached.

Process isolation

Tasks are run in hosts, which are isolated processes. If a host process crashes, it will not affect the other host processes or the agent. For the .NET host process, a separate AppDomain is created to run the task's code. The assemblies needed to run the task's code are streamed to the host and resolved in the task's AppDomain when needed.

Task workflow

When a host starts up, the host calls TaskEnvironmentSetup on the task environment it receives. Then the host loops for two states:

If the agent sends a task to the host, call TaskExecute, and then send the result back to the client. If the task fails, then the host is shutdown.
If the agent recycles the host, then call TaskEnvironmentTeardown and shut down the host process.

The chart below shows this state diagram:

Host assignment and recycling

Most of the workflow associated with the agent centers around comparing task environment identifications and checking whether a host has already loaded the environment. As a refresher, this topic explains how task environment identifications are compared.

When an agent is assigned a task, its workflow is as follows. First, the agent checks the task environment identification of the task. If an idle host process with that task environment loaded already exists, the agent sends the task to that host. If no hosts have the specified task environment loaded, the agent checks if it can start another host. If the agent cannot start another host because an empty host slot is not available, the agent will cause one of the idle host processes to exit and start a new host process in its place. This procedure is called recycling a host. The new host is then given the new task environment.

Note
Here are a few scenarios and a description of what happens internally in each situation. For simplicity, the task environment identification GUIDS have been replaced with letters. To learn more about the task environment identification comparison mechanism, read this. Situation: One idle host (task environment id "A"), One empty host slot. Task is received with environment id "A". Agent will... assign the task to the idle host. Situation: One idle host (task environment id "A"), One empty host slot. Task is received with environment id "B". Agent will... create a new host in the empty slot. New host will be sent environment id "B". Situation: Two idle hosts (task environment id "A" and "B" respectively). Task is received with environment id "C". Agent will... recycle one of the idle hosts and start a new host with environment id "C". Situation: Task completes on a host (host completed 1 task so far). Task Environment's HostRecycleSettingsFixedNumberOfTasks option is set to 32. Agent will... do nothing. Host task count is still below HostRecycleSettingsFixedNumberOfTasks. Host process remains idle. Situation: Task completes on a host (host completed 32 tasks so far). Task Environment's HostRecycleSettingsFixedNumberOfTasks option is set to 32. Agent will... recycle the host.

Note

Here are a few scenarios and a description of what happens internally in each situation. For simplicity, the task environment identification GUIDS have been replaced with letters. To learn more about the task environment identification comparison mechanism, read this.

Situation: One idle host (task environment id "A"), One empty host slot. Task is received with environment id "A".
Agent will... assign the task to the idle host.

Situation: One idle host (task environment id "A"), One empty host slot. Task is received with environment id "B".
Agent will... create a new host in the empty slot. New host will be sent environment id "B".

Situation: Two idle hosts (task environment id "A" and "B" respectively). Task is received with environment id "C".
Agent will... recycle one of the idle hosts and start a new host with environment id "C".

Situation: Task completes on a host (host completed 1 task so far). Task Environment's HostRecycleSettingsFixedNumberOfTasks option is set to 32.
Agent will... do nothing. Host task count is still below HostRecycleSettingsFixedNumberOfTasks. Host process remains idle.

Situation: Task completes on a host (host completed 32 tasks so far). Task Environment's HostRecycleSettingsFixedNumberOfTasks option is set to 32.
Agent will... recycle the host.

After a host completes a task, the agent will check whether the host has met any of the requirements of its task environment's HostRecycleSettings. Various recycling strategies are set through the TaskEnvironmentRecycleSettings's properties. Recycling strategies fall into three categories: time, memory, and job based.

Time Based

HostRecycleSettingsRegularInterval: Continually recycles hosts after a set number of milliseconds, measured from the time the host started.
Above, one job with five tasks is submitted to run on two hosts. The task environment’s HostRecycleSettingsRegularInterval property is set to a designated period. The hosts will recycle after the specified period of time, as measured from when the host process first started. If a host is recycled, the regular interval is measured from the time the new host process starts.

HostRecycleSettingsIdleTimeout: Recycles a host after it has spent a set number of milliseconds not executing any tasks.
Above, a job with two tasks is submitted to run on a host. The task environment’s HostRecycleSettingsIdleTimeout property is set to a period of time. The host will recycle after the designated period, measured from the time the last task finished executing.

HostRecycleSettingsSpecificTimes: Recycles hosts at specified times of day (input as strings, times separated by commas ',').
Above, one job with seven tasks is submitted to run on two hosts. The task environment’s HostRecycleSettingsSpecificTimes property is set to a list of three times. Each started host is recycled following the specified times after it has finished executing its most recent task.

Memory Based

HostRecycleSettingsPrivateMemoryUsageMaximum: Recycles a host after the process reaches the specified amount of private memory (specified in bytes, corresponding to the Commit Size column in the Windows 10 Task Manager).

Job Based

HostRecycleSettingsFixedNumberOfTasks: Recycles after the host completes a set number of tasks.
Above, a job with seven tasks is submitted to run on two hosts. The task environment’s HostRecycleSettingsFixedNumberOfTasks property is set to three. Each host will recycle after every three tasks it executes. If a host executes less than three tasks, the host process remains idle.

HostRecycleSettingsJobCompletion: Recycles the hosts used for a given job once the job completes.
Above, two jobs, each with three tasks and identical task environments, are submitted to the coordinator and ran on two available hosts. The task environment’s HostRecycleSettingsJobCompletion property is set to true. Each host exits after all of its Job A tasks are completed. New host processes are started to execute Job B’s tasks, after which the hosts exit.

HostRecycleSettingsTaskEnvironmentNoLongerReferenced: Recycles when the host's task environment is no longer needed. If two jobs have the same task environment, the hosts will not recycle when moving on to the second job since the task environment is still in use. The host will recycle once the environment is no longer used (i.e. both jobs complete).
Above, two jobs, each with three tasks and identical task environments, are submitted to the coordinator and ran on two available hosts. The task environment's HostRecycleSettingsTaskEnvironmentNoLongerReferenced property is set to true. The two host processes stay alive until all of Job A and Job B's tasks have been completed, after which no more tasks require the shared task environment and the host processes exit.

The code sample below highlights the differences between the JobCompletion and TaskEnvironmentNoLongerReferenced recycling strategies:

Copy

namespace CodeSamples
{
    using System;

    using AGI.Parallel.Client;
    using AGI.Parallel.Infrastructure;

    public class Program
    {
        public static void Main(string[] args)
        {
            using (IJobScheduler scheduler = new ClusterJobScheduler("localhost"))
            {
                scheduler.Connect();

                // Create HostRecycleSetting to use in TaskEnvironment
                HostRecycleSettings settingsTaskEnvironmentNoLongerReferenced = new HostRecycleSettings { TaskEnvironmentNoLongerReferenced = true };
                HostRecycleSettings settingsJobCompletion = new HostRecycleSettings { JobCompletion = true };

                // Create jobs and set TaskEnvironments
                Job job1 = scheduler.CreateJob();
                Job job2 = scheduler.CreateJob();
                Job job3 = scheduler.CreateJob();
                Job job4 = scheduler.CreateJob();
                Job job5 = scheduler.CreateJob();
                Job job6 = scheduler.CreateJob();
                job1.TaskEnvironment.RecycleSettings = settingsTaskEnvironmentNoLongerReferenced;
                job2.TaskEnvironment.RecycleSettings = settingsTaskEnvironmentNoLongerReferenced;
                job3.TaskEnvironment.RecycleSettings = settingsTaskEnvironmentNoLongerReferenced;
                job4.TaskEnvironment.RecycleSettings = settingsTaskEnvironmentNoLongerReferenced;
                job5.TaskEnvironment.RecycleSettings = settingsJobCompletion;
                job6.TaskEnvironment.RecycleSettings = settingsJobCompletion;

                // Jobs 1 and 2 have identical IDs
                job1.TaskEnvironment.Id = new Guid("A457B1FA-BE37-49b4-8B95-A53A16E2AA5F");
                job2.TaskEnvironment.Id = new Guid("A457B1FA-BE37-49b4-8B95-A53A16E2AA5F");
                // Jobs 3 and 4 have different IDs
                job3.TaskEnvironment.Id = new Guid("B457B1FA-BE37-49b4-8B95-A53A16E2AA5F");
                job4.TaskEnvironment.Id = new Guid("C457B1FA-BE37-49b4-8B95-A53A16E2AA5F");
                // Jobs 5 and 6 have identical IDs 
                job5.TaskEnvironment.Id = new Guid("D457B1FA-BE37-49b4-8B95-A53A16E2AA5F");
                job6.TaskEnvironment.Id = new Guid("D457B1FA-BE37-49b4-8B95-A53A16E2AA5F");

                for (int i = 0; i < 100; i++)
                {
                    job1.AddTask(new BlankTask());
                    job2.AddTask(new BlankTask());
                    job3.AddTask(new BlankTask());
                    job4.AddTask(new BlankTask());
                    job5.AddTask(new BlankTask());
                    job6.AddTask(new BlankTask());
                }

                /*
                 * Hosts will be recycled after both jobs complete
                 * because the task environments are identical. 
                 */
                // Submit Jobs with identical environments (Recycling Strategy: TaskEnvironmentNoLongerReferenced)
                job1.Submit();
                job2.Submit();
                job1.WaitUntilDone();
                job2.WaitUntilDone();

                /*
                 * Hosts will be recycled after each individual job completes
                 * because the task environments are different.
                 */
                // Submit Jobs with different Environment (Recycling Strategy: TaskEnvironmentNoLongerReferenced)
                job3.Submit();
                job4.Submit();
                job3.WaitUntilDone();
                job4.WaitUntilDone();

                /*
                 * Hosts will be recycled after each individual job completes
                 * even though their task environments are identical.
                 */
                // Submit Jobs with identical Environment (JobCompletion)
                job5.Submit();
                job6.Submit();
                job5.WaitUntilDone();
                job6.WaitUntilDone();
            }
        }
    }

    // Create Blank Task
    [Serializable]
    public class BlankTask : Task
    {
        public override void Execute()
        {
        }
    }
}

Changes to hosts will be displayed in the AgentService.exe window. Shown below is the output after jobs 1 and 2 finish running (each job has 100 tasks).

Other Resources

Key concepts

Process Isolation