Archive Integration

Vidispine has no built in integration with any archive vendors. It is however possible to write your own integration scripts which Vidispine will then invoke when a file is to be archived.

Creating an archive storage

In order to get a working integration with an external archive, a special storage must be created with type ARCHIVE. For this integration to work, a script must be associated with the storage (described below).

Example

Creating an ARCHIVE storage:

POST /storage
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>ARCHIVE</type>
  <capacity>1000000000</capacity>
  <archiveScript><![CDATA[
...
]]></archiveScript>
</StorageDocument>

Integrating with an archive using JavaScript

To enable integration, a JavaScript must be written which will perform the actual archive operation. In order to be as flexible as possible, this script can both make API calls to Vidispine (The api object), and invoke shell operations (The shell object).

The script also has access to a file object. This object has the following functions defined:

file.getMetadata(key)

If the specified metadata key is set on the file, the value is returned, otherwise null.

file.setMetadata(key, value)

Sets the specified key-value pair as metadata on the file.

file.getAllMetadata()

Returns a map of all file metadata.

The script must as its last assignment define an object with the following properties:

  • archive - A function invoked when an archive is to be performed.
  • restore - A function invoked when a restore is to be performed.
  • remove - A function invoked when a delete is to be performed.
  • restorePartial - A function invoked when a partial restore is to be performed. This function is optional. If it is missing and a partial restore is requested, the restore function will be invoked.

Example

A simple archive script. This script only performs file system copies and removes.

function getFilePath(url) {
  if (url.indexOf('file:///') === 0) {
    url = url.substring(7);
  }
  if (url.indexOf('file:/') === 0) {
      url = url.substring(5);
  }
  if (url.indexOf('/C:/') === 0) {
    url = url.substring(1);
  }
  return url;
}

o = {
  "archive": function(uri, id, data) {
    var archivePath = getFilePath(data.archiveDir);
    var path = getFilePath(uri);
    var filename = uri.substring(uri.lastIndexOf('/')+1);

    logger.log('Running command "cp '+path+' '+archivePath);

    var result = shell.exec('cp', path, archivePath);
    if (result.exitcode > 0) {
        throw "Failed to copy file to archive: "+result.err;
    } else {
        file.setMetadata('uri', archivePath+filename);
    }
  },
  "restore": function(uri, id, data) {
    var path = getFilePath(uri);
    var loc = getFilePath(file.getMetadata('uri'));

    logger.log('Running command "cp '+loc+' '+path);

    var result = shell.exec('cp', loc, path);
    if (result.exitcode > 0) {
        throw "Failed to copy file from archive: "+result.err;
    }
  },
  "remove": function(id, data) {
    var loc = getFilePath(file.getMetadata('uri'));

    logger.log('Running command "rm '+loc);

    var result = shell.exec('rm', loc);
    if (result.exitcode > 0) {
        throw "Failed to remove file: "+result.err;
    }
  }
};

Amazon Glacier

New in version 4.1.1.

Vidispine can archive files on Amazon Glacier. There are two different ways this can be achieved:

  • Creating a separate Glacier storage and move files from other storages there for archival.
  • Using an S3 storage and transition objects to the Glacier storage class.

Creating a dedicated Glacier storage

To create a storage used solely for Glacier archiving, you need to create a storage with an XML document like this:

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>ARCHIVE</type>
  <bean>GlacierBean</bean>
  <capacity>100000000000000</capacity>
  <metadata>
    <field>
      <key>glacierVaultName</key>
      <value>{vault name}</value>
    </field>
    <field>
      <key>glacierEndpoint</key>
      <value>https://glacier.us-east-1.amazonaws.com/</value>
    </field>
  </metadata>
</StorageDocument>

Files can then be moved to this storage either using storage rules or by initiating a copy job (Create a file move/copy job). Restore jobs must be initiated using storage rules.

Note that restore jobs typically take several hours, and the restore job will be put in the WAITING state while the restore initiation is in progress. This is to allow other jobs to run during this time.

Transitioning files From S3 to Glacier

There is no way, using the AWS SDK, to directly initiate a transition to the Glacier storage class for a single object. Instead, Object Lifecycle Management must be used. Vidispine will automatically detect when a transition to the Glacier class has happened, and put the file in the ARCHIVED state.

To restore a file so that it can be read directly, you can use the following request

PUT {file-resource}/restore
Query Parameters:
 
  • extraData

    Additional parameters relevant for the restore, in the form of key=value. Specify multiple parameters using multiple query parameters.

    expirationInDays={number-of-days}
    How long the restored files should be available.

    Deprecated since version 4.9: Use the expirationInDays query parameter instead.

  • expirationInDays

    Required parameter. How long the restored files should be available.

    New in version 4.9.

  • retrievalTier

    Optional parameter to set the Glacier retrieval tier to use when restoring the file. Can be set to either Expedited, Standard or Bulk. See Restoring Archived Objects for more information.

    New in version 4.9.

Produces:
  • text/plain – Informational text.
Status Codes:
  • 400 Invalid Input – Amazon rejects the restore request, a parameter value is invalid.
  • 404 Not Found – The file was not found.

This will cause Vidispine to ask Glacier to initiate a restore. Once the restore is complete, the file will be put in the CLOSED state, and it is available for direct access.

The expirationInDays parameter has to be set and specifies how long the restored files should be available. Once it has expired, it will be removed from direct access and once again end up in the ARCHIVED state.

Front Porch Diva Integration

New in version 4.1.

Requirements

  • StoragePlugin.jar

Installation

Deploy StoragePlugin.jar in GlassFish:

  • Open the web administration console.
  • Navigate to Applications.
  • Click Deploy... and select the StoragePlugin.jar file.
  • Make sure that Type is set to EJB Jar and that Compatibility is selected.
  • Click OK.

Configuration

Set up a shared folder which is accessible by both Vidispine and DIVarchive manager.

POST the following document to /API/storage

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
 <type>ARCHIVE</type>
 <capacity>1000000000000</capacity>
 <bean>DIVABean</bean>
 <metadata>
  <field>
   <!-- SSH host -->
   <!-- this is the hostname of the DIVA SSH service -->
   <key>hostname</key>
   <value>187.47.11.109</value>
  </field>
  <field>
   <!-- SSH username -->
   <!-- this is the username for the DIVA SSH service -->
   <key>username</key>
   <value>diva</value>
  </field>
  <field>
   <!-- SSH password -->
   <key>password</key>
   <value>diva</value>
  </field>
  <field>
   <!-- SSH port -->
   <key>port</key>
   <value>22</value>
  </field>

  <field>
   <!-- path to the shared folder on vidispine server -->
   <key>storagePath</key>
   <value>/shared/storage/</value>
  </field>
  <field>
   <!-- hostname or IP address for the DIVA manager -->
   <key>DIVAHostname</key>
   <value>187.47.11.109</value>
  </field>
  <field>
   <!-- TCP port for the DIVA manager -->
   <key>DIVAPort</key>
   <value>9065</value>
  </field>
  <field>
   <!-- Media name designates either a group of tape, or an array of disk
        declared in the configuration where the instance has to be created. -->
   <key>DIVAMediaName</key>
   <value>default</value>
  </field>
  <field>
   <!-- category -->
   <key>DIVACategory</key>
   <value>default</value>
  </field>
  <field>
   <!-- restore is not yet implemented -->
   <key>DIVARestoreDestination</key>
   <value></value>
  </field>
  <field>
   <!-- path to shared folder on DIVA server -->
   <key>DIVAFilePathRoot</key>
   <value>C:/shared/storage/</value>
  </field>
  <field>
   <!-- The value of this option is the name of the source/destination to be used
        by the specified command: archive, restore, copy... This server name
        must be a valid name as configured in the DIVA system. -->
   <key>DIVAServerName</key>
   <value>disk</value>
  </field>
 </metadata>

 <method>
  <uri>file:///shared/storage/</uri>
  <read>true</read>
  <write>true</write>
  <browse>true</browse>
 </method>
</StorageDocument>

Usage

To archive a file, copy it to the shared folder and wait for Vidispine to detect its presence. Once Vidispine has found the file, import it to trigger archiving.

Example using curl:

curl -X POST -uadmin:admin 'http://localhost:8080/API/storage/VX-4/file/VX-1/import' -Hcontent-type:application/xml -d '<MetadataDocument/>'

Archiving should begin shortly.