Jobs

Jobs make up the long running tasks in Vidispine. They are created in response to requests that would otherwise not be able to respond in time, such as import, export and transcode requests.

The actions performed by a job is determined by its type. Bound to the type are a number of steps, or tasks, defined by the task definitions. The tasks form a graph, and typically execute in sequence, but it is also possible for tasks to start in parallel. This happens for example when importing and transcoding a growing file. The transfer step will initiate the transfer and then trigger the transcode step to start once enough data (the header) from the file has been transferred.

digraph finite_state_machine {
    rankdir=LR;
    size="4,3";
    node [shape = point] start;
    node [fontsize=12, fixedsize=true, width=1, height=1];
    FINISHED [shape = doublecircle, label = "FINISHED/\nFAILED"];
    ABORT_PENDING [shape = circle, label = "ABORT\nPENDING"];
    node [shape = doublecircle] FINISHED ABORTED;
    node [shape = circle];

    start -> READY;
    READY -> STARTED
    STARTED -> FINISHED;
    STARTED -> WAITING;
    STARTED -> ABORT_PENDING;
    ABORT_PENDING -> ABORTED;
    WAITING -> READY;
}

The states of a job are illustrated above. See below for a full description of the states and of the job step states.

Creating jobs

Create jobs by making requests to other RESTful resources:

Job type Relevant documentation
Import jobs Imports (Also Importing a file from a storage)
Export jobs Exports
Thumbnail jobs Thumbnail settings
Shape update/Essence version jobs Shapes
File actions Files
Sequence rendering Item sequences
Item list job Listing items in batch
Shape analyze Shape analysis

Concurrency

The number of jobs that execute in parallel is determined by the concurrentJobs configuration property.

Job pools

Using job pools it is possible to limit the number of concurrent low priority jobs, to make sure that higher priority jobs are able to start even if there are a large number of low priority jobs running. Job pools are configured using the job pool configuration resource.

PUT /configuration/job-pool
Content-Type: application/xml

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <pool>
    <priorityThreshold>HIGH</priorityThreshold>
    <size>2</size>
  </pool>
  <pool>
    <priorityThreshold>LOWEST</priorityThreshold>
    <size>3</size>
  </pool>
</JobPoolListDocument>

This configuration will allow at most 3 jobs with a priority of LOWEST to MEDIUM to execute at the same time. It will also allow up to 5 concurrent HIGH/HIGHEST priority jobs, as the second pool will contain jobs with a priority of LOWEST or higher (the priority threshold is the lower bound and pools have no upper priority bound.)

If there is no job pool with a priority threshold that matches low priority jobs then such jobs will not be started. For example, to only let jobs with a priority of MEDIUM or higher to execute:

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <pool>
    <priorityThreshold>MEDIUM</priorityThreshold>
    <size>3</size>
  </pool>
</JobPoolListDocument>

Note that the max concurrent job setting will only have an effect if it is lower then the size of all pools combined.

If no pools have been defined then <concurrentJobs> controls the number of concurrent jobs. This is the same setting as the concurrentJobs configuration property. So by default the job pool configuration will look like:

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <concurrentJobs>3</concurrentJobs>
</JobPoolListDocument>

Job problems

Jobs will enter the state WAITING if a recoverable problem has occurred. Depending on the problem the system might resolve itself or require manual assistance, for example if the system is out of storage space.

A system with no job problems will report:

GET /job/problem HTTP/1.1
Content-Type: application/xml

<JobProblemListDocument xmlns="http://xml.vidispine.com/schema/vidispine"/>

A system where the transcoder is unreachable for some reason may report:

GET /job/problem HTTP/1.1
Content-Type: application/xml

<JobProblemListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <problem>
    <id>31532534</id>
    <type>TranscoderOffline</type>
    <job>VX-172716</job>
  </problem>
</JobProblemListDocument>

There can be multiple jobs waiting for a problem to be resolved, for example, in case of transcoder or storage problems. For JavaScript problems there will however be one problem per job, as the problem condition is defined by a step specific for each job.

Job tasks

The action performed by a task can be implemented either as a method in a Java class or as a JavaScript. Using JavaScript is recommended for all new applications.

POST /task-definition/ HTTP/1.1
Content-Type: application/xml

<TaskDefinitionListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <task>
    <description>A custom JavaScript step</description>
    <script><![CDATA[
// This script does nothing but fail the job
job.fatalFail("Testing job failing");
]]></script>
    <step>10000</step>
    <dependency>
      <previous>false</previous>
      <allPrevious>true</allPrevious>
    </dependency>
    <jobType>PLACEHOLDER_IMPORT</jobType>
    <critical>false</critical>
  </task>
</TaskDefinitionListDocument>

Defining new tasks

See JavaScript tasks on how to create JavaScript tasks.

Task dependencies

The execution order is defined by the step numbers and dependencies of the steps. The dependency element defines which steps a specific step depend on. There is also the parallelDependency element that defines the dependencies that apply if the step is executing as a parallel step.

allPrevious = true The step requires all previous step to finish, before it can start.
previous = true The step requires the previous step to finish, before it can start
step = N The step requires step number N to finish, before it can start

Visualizing tasks

In order to easily see the dependencies between steps for a particular job type, there is functionality to render the job definition as a graph. In order to render the graph, the Graphviz package is required.

Custom job types

It is possible to define custom job types. Each custom job type is defined with a name and an integer identifier. Both of these must be unique. Task definitions can then be added to the job type, in the same way as described above.

Example

First we create the job type:

POST /task-definition/jobtype/MYCOMPANY_CUSTOM_JOB_TYPE?id=25000 HTTP/1.1
<TaskDefinitionListDocument xmlns="http://xml.vidispine.com/schema/vidispine"/>

Then we can add task definitions to our new job type:

POST /task-definition/ HTTP/1.1
Content-Type: application/xml

<TaskDefinitionListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <task>
    <description>A custom JavaScript step</description>
    <script><![CDATA[
    logger.log("My custom job is running!");
]]></script>
    <step>100</step>
    <dependency>
      <previous>false</previous>
      <allPrevious>true</allPrevious>
    </dependency>
    <jobType>MY_CUSTOM_JOB_TYPE</jobType>
    <critical>false</critical>
  </task>
</TaskDefinitionListDocument>

After this has been done, we can now run the job:

POST /job?type=MY_CUSTOM_JOB_TYPE
<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-29</jobId>
  <user>admin</user><started>2016-05-17T12:38:22.999Z</started>
  <type>MY_CUSTOM_JOB_TYPE</type>
  <status>READY</status>
  <priority>MEDIUM</priority>
</JobDocument>