Transcoders¶
When you import items the Vidispine transcoder will be used to detect the type of media that is being imported and, of course, to transcode the media to any formats that you have requested.
The common operations performed by the transcoder are:
- Media shape deduction
- Transcoding
- Sequence rendering
- Partial file extraction
- XMP extraction and rewrite
The Vidispine transcoder has a REST API that Vidispine uses to perform the above operations. This API is not described in this document, as it typically should not be accessed directly.
Adding a transcoder¶
Add a transcoder by creating a new transcoder
resource. The resource
document should contain information on how to reach the transcoder and what
storages the transcoder has direct access to.
POST /resource/
Content-Type: application/xml
<?xml version="1.0"?>
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<transcoder>
<url>http://transcoder.example.com:8888/</url>
<directAccess>
<filter>file:/srv/media/.*</filter>
</directAccess>
</transcoder>
</ResourceDocument>
Vidispine checks the status of transcoders continuously in the background. As such, if the configuration is correct you will see that the transcoder shows up as online in a few seconds.
GET /resource/VX-7
<?xml version="1.0"?>
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<id>VX-7</id>
<transcoder>
<url>http://transcoder.example.com:8888/</url>
<directAccess>
<filter>file:/srv/media/.*</filter>
</directAccess>
<state>ONLINE</state>
</transcoder>
</ResourceDocument>
The Vidispine installer will by default install and configure a transcoder in Vidispine for you, so this step is typically not needed.
Using multiple transcoders¶
Depending on your license, you may be allowed to use more than one transcoder. To do so, simply add additional transcoders as explained above. Vidispine will submit transcode jobs to the transcoder based on the current number of jobs being processed by the transcoder.
Vidispine will use the transcoder with the least amount of work. If a transcoder goes offline then any transcode job steps using that transcoder will fail and be retried using one of the online transcoders. If all transcoders are offline then jobs will wait for one to become available.
Changed in version 4.4: The clusterName
property must be set if multiple Vidispine
installations are to share a transcoder. Each installation must have a
unique cluster name. This applies regardless if the installations have the
same site name or not.
Warning
Do not connect multiple Vidispine installations to the same set of transcoders unless each installation has a distinct site name. If installations have the same site name then jobs from one installation may conflict with jobs from another.
Changed in version 4.4: This no longer applies; see above notice.
How transcoders perform jobs¶
A transcoder will perform a job as soon as it is received, and will not schedule jobs for later execution. Vidispine, that is, the user of the transcoder is responsible for scheduling which jobs a transcoder should execute and when they should be executed.
Transcoder job limit¶
New in version 4.12.
The maxJob
setting can be used to limit the number of Vidispine jobs that may use a specific transcoder
at the same time. If all transcoders are busy, jobs will be put on WAITING
state, with a TranscoderBusy
problem. The jobs will restart as soon as any qualified transcoder becomes available again.
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<id>VX-7</id>
<transcoder>
<url>http://transcoder.example.com:8888/</url>
<maxJob>5<maxJob>
</transcoder>
</ResourceDocument>
A Vidispine job typically triggers multiple transcoder jobs, e.g. shape deductions and transcodes, so in the above case there may still be more than 5 running jobs on a transcoder. They will however all belong to at most 5 jobs in Vidispine.
The above setting works for VSA transcoders as well. VSA users can also add transcoder.maxJob
to one of
the agent configuration files. For example:
$ cat /etc/vidispine/agent.conf.d/transcoder.conf
transcoder.maxJob=5
The transcoder’s configuration file¶
The transcoder configuration file config.xml
contains default settings
for the transcoder and need typically not be modified, as the settings
can instead by configured in Vidispine.
Modify the transcoder file¶
On a Linux system, copy the file /opt/vidispine/transcoder/config.xml
to
/etc/transcoder-config.xml
. Then edit /etc/transcoder-config.xml
. The
file in /etc
takes precedence over the file in /opt/vidispine
.
Modify the transcoder resource¶
New in version 4.2.3.
On all operating systems, the transcoder configuration can be changed by adding configuration to the resource definition of the transcoder (Adding a transcoder).
Note that port of the transcoder cannot be changed in this fashion.
Modifying the transcoder configuration is this fashion takes precedence over the local configuration file and the global transcoder configuration, see below.
Modify all transcoders¶
It is also possible to change the configuration of all transcoders,
by setting the configuration property transcoderDefaultConfiguration
to the XML representation of the transcoder configuration.
Thumbnail settings¶
Note
The preferred way of changing the thumbnail and poster settings is by
changing the appropriate values in the TranscodePresetDocument
in a shape tag. For example, by changing the
thumbnailResolution
and thumbnailPeriod
elements. The setting in
shape tag have priority over the transcoder setting.
The thumbnailResolution
element contains the default resolution of the
thumbnails produced by the transcoder.
<a:thumbnailResolution>
<a:width>320</a:width>
<a:height>240</a:height>
</a:thumbnailResolution>
You can also change the thumbnailing frequency by changing thumbnailPeriod
.
For example, to thumbnail every 3 seconds:
<a:thumbnailPeriod>
<a:samples>3</a:samples>
<a:timeBase>
<a:numerator>1</a:numerator>
<a:denominator>1</a:denominator>
</a:timeBase>
</a:thumbnailPeriod>
If the transcoder does not use a scene change detection plugin, the frequency defaults to once every 10 seconds.
StatsD settings¶
New in version 4.2.3.
To have the transcoder send metrics to a StatsD server you can either:
- Enable StatsD using the API, see StatsD
- Update the transcoder configuration with the address and port of the StatsD server:
<a:statsd>
<a:destination>
<a:address>127.0.0.1</a:address>
<a:port>8125</a:port>
</a:destination>
<a:prefix>t1</a:prefix>
</a:statsd>
The prefix
element configures the prefix to use for each metric.
By default this is the transcoder
.
Transcoder resources settings¶
Path to temporary storage¶
Since: | 4.2.5 |
---|
Controls where temporary files are stored. Defaults to /tmp
on UNIX-like systems, or %TEMP%
on Windows.
<a:tempPath>/mnt/largetemparea<a:tempPath>
Number of decoding threads¶
Controls the number of decoding threads. Defaults to 4 for I-frame-only formats. The actual number of threads used depends on codec. (New in 4.2.3.) Since version 4.2.3, this setting is used with more formats then before.
<a:decoderOfferThreads>8<a:decoderOfferThreads>
Number of encoding threads¶
Controls the number of encoding threads. Defaults to automatic setting. The actual number of threads used depends on codec.
<a:encoderThreads>8<a:encoderThreads>
HTTP buffer sizes¶
New in version 4.5.
Controls the size of HTTP reads and writes of the transcoder. dataBufferSize controls the maximum number of read bytes in memory. Default is 100 MB. dataBufferWriteSize controls the maximum number of write bytes in memory. Default is 100 MB. dataBufferFlushTime controls the number of seconds written bytes are stored in memory before it is flushed. Default is 4 seconds.
For system that uses segment files and where the transcoder has enough memory, it is recommended to increase these numbers, up to 10 times.
Image processing¶
To control the memory and disk usage used by the transcoder for image
processing, use the <imagemagick>
element in the transcoder configuration.
The most important settings are listed below, for a complete list,
see http://www.imagemagick.org/script/resources.php
(under environment variables, used without the MAGICK_
prefix
in the transcoder configuration).
Maximum heap usage¶
<a:imagemagick>
<a:key>MEMORY_LIMIT</a:key>
<a:value>1GB</a:value>
</a:imagemagick>
Temporary work area¶
<a:imagemagick>
<a:key>TEMPORARY_PATH</a:key>
<a:value>/var/tmp</a:value>
</a:imagemagick>
(New in 4.2.5.)
The default value is the value set by the general transcoder temporary path,
see above. It is recommended that the tempPath
setting is used, rather
than the imagemagick
one.
Operations overview¶
Zeroconf transcoders¶
The following is done to remove the need to configure the transcoders directly:
- Vidispine pushes its own license to the transcoder, so that each transcoder does not need a license file of their own.
- The transcoder returns the IP address from where the license was pushed, that is, the IP address of the application server, removing the need for explicitly configuring the reverse address, that is, where the transcoder can reach Vidispine, in most cases.
- In addition, Vidispine generates temporary pre-authorized URIs that are used by the transcoder. This removes the need for entering any application server information in the transcoder configuration file.
Reverse address and NAT¶
The reverse address does not work if there is NAT or other port forwarding
mechanisms between the application server and the transcoder.
If so, the address to VS-EA can be
overridden in the definition for the transcoder by setting the
<reverseAddress>
element.
<?xml version="1.0"?>
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<transcoder>
<url>http://transcoder.example.com:8888/</url>
<reverseAddress>vs.example.com</reverseAddress>
</transcoder>
</ResourceDocument>
For GlassFish, a new HTTP listener (noauth-listener
, default port is
the standard API port + 9) is installed on GlassFish for communication from the
transcoder to GlassFish. The rules for how the address to GlassFish is determined
are as follows:
- If the configuration property
apiNoauthUri
is set, it is used for all transcoders. - If the configuration property
apiNoauthPort
is set, it is used for together with the detected or manually set reverse address. - If the port for
noauth-listener
can be determined (GlassFish only), it is used with the reverse address. - If the port for
http-listener-1
can be determined (GlassFish only), it is used with the reverse address.
For JBoss, it is currently required that at least apiNoauthPort
is set.
Transcoder’s access to media¶
By default, the transcoder accesses non-file-schema media through the application server. This has several advantages:
- The same user is used for all file access.
- Possibility for support for extended file attributes and permissions.
- Support for other file systems (URI schemes).
Streaming the media puts some extra load on the application server. Some tuning might be necessary.
The transcoder resource in Vidispine can be set up to access files directly. By
adding a directAccess
element to the transcoder resource, Vidispine
will let the transcoder access the media directly. If no directAccess
elements are present, an implicit
<directAccess>
<filter>file:.*</filter>
</directAccess>
is added. In order to tell Vidispine that all files should go via the application server, add an
<directAccess>
<filter>NO_MATCH</filter> <!-- dummy regular expression that does not match anything -->
</directAccess>
Growing files¶
For all file systems that supports read-while-write, and for container
formats that are built for streaming (e.g. MXF), growing file is supported
when streamed through the application server. If growing files is required
to local files with the file
scheme, a directAccess
/NO_MATCH
element as per above must be added to the resource configuration.