Storages

Storages are where Vidispine will store any files that are ingested/created in the system. All files on a storage location will get an entry in the Vidispine database, containing state, file size, hash etc. This is to keep track of any file changes.

For information about files in storage, see Files.

Storages

Storage types

A storage must be designated a type, based on what type of operations are to be performed on the contained files. Operations in this context are transcode, move, delete, and destination (that is, placing new files here).

LOCAL
A Vidispine specific storage, suitable for all operations. Note that LOCAL doesn’t necessarily imply that the storage is physically local. It should however be a dedicated Vidispine storage. That is, files on such storages should not be written to/deleted by any external application.
SHARED
A storage shared with another application, Vidispine will not create new files, nor perform any write operations here.
REMOTE
A storage on a remote computer, files should be copied to a local storage before used.
EXTERNAL
A storage placeholder.
ARCHIVE
A storage meant for archiving, needs a plugin bean or a JavaScript, described in more detail at Archive Integration.
EXPORT
Files are not monitored, but copy operations to here will create a file entry in the database.

Storage states

Storages will have one of the following states:

NONE
Not used.
READY
Operating normally.
OFFLINE
No available storage method could be reached.
FAILED
Currently not used in Vidispine.
DISABLED
Currently not used in Vidispine.
EVACUATING
Storage is being evacuated.
EVACUATED
Evacuating process finished.

For more information about storage evacuation, see section on Evacuating storages.

Storage priority

New in version 4.17.

Storage priority can be set when creating a storage. If a shape has duplicate files on different storages, the file on the highest priority storage will be selected as the source of transcoder or transfer jobs

Example:

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <priority>HIGH</priority>
  <method>
  ...
  </method>
</StorageDocument>

Available priority values are: HIGHEST, HIGH, MEDIUM, LOW, LOWEST. Default priority of a storage is MEDIUM.

Storage groups

Storages can be placed in named groups, called storage groups. These storage groups can then be used in Storage rules and Quota rules.

Storage capacity

When a storage is created a capacity can be specified. This is the total number of bytes that is freely available on the storage. The free capacity is calculated as total capacity - sum(file sizes in database list). Note that this means that the size of MISSING and LOST files are included in the used capacity. If you do not expect a file with these states to return, it is best to delete the file entity using the API.

Auto-detecting the storage capacity

By setting the element autoDetect in the StorageDocument you can make Vidispine read the capacity from the file system. This only works if the storage has a storage method that points to the local file system, that is, a file:// URI.

Warning

Do not enable auto-detection for multiple storages located on the same device, as each storage will then have the capacity of the device. This means that storages may appear to have free space in Vidispine, when there is actually no space left on the device.

Storage cleanup

If you have used storage rules to control the placement of files on storages then you may have noticed that files have been copied to the storages selected by the rules, but that files on the source storages have not been removed.

This is by design. Vidispine prefers to keep multiple copies of a file, and only remove the files when a storage is about to become full. The storage high and low watermarks control when files should start to be removed, and when enough files have been removed and storage cleanup should stop.

For example, for a 1 TB storage with a high watermark at 80% and a low watermark at 40%, Vidispine will keep adding files to the storage until the usage exceeds 800 GB. Once that happens cleanup would occur. Files that are deletable, that is, that have a copy on another storage and that is not required to exist according to the storage rules, will be deleted. Cleanup will stop once the usage has reached 400 GB or when there are no more deletable files.

If this behavior is not desirable, then there are two options.

  1. Update the storage rules to specify where files should not exist, using the not element. For example, using <not><any/></not>.

    <StorageRuleDocument xmlns="http://xml.vidispine.com/schema/vidispine">
      <storageCount>1</storageCount>
      <storage>VX-122</storage>
      <not><any/></not>
    </StorageRuleDocument>
    
  2. Set the high watermark on the storage to 0%. Updating the storage rules is preferred as storage cleanup will be triggered continuously if the high watermark is set at a low level.

Evacuating storages

If you would like to delete a storage, but you still have files there which are connected to items, you can first trigger an evacuation of the storage. This will cause Vidispine to attempt to delete redundant files, or move files to other storages. Once the evacuation is complete, the storage will get the state EVACUATED.

Storage methods

Methods are the way Vidispine talks to the storage. Every method has a base URL. See Storage method URIs for the list of supported schemes.

Retrieve a storage to check its status. The storage state shows if the storage is accessible to Vidispine. If a storage is not accessible, then its state will be OFFLINE. Check the failureMessage in the storage methods to find out why. The failure message will be the error from when the last attempt to connect to the storage was made, and will be available even when the storage comes back online again. Compare lastSuccess to lastFailure to determine if the error message is current or not.

If multiple methods are defined for one storage, it is important, in order to avoid inconsistencies, that they all point to the same physical location. E.g. a storage might have one file system method, and one HTTP method. The HTTP URL must point to the same physical location as the file system method.

Storage method examples

Here are some examples of valid storage methods:

  • file:///mnt/vidistorage/
  • ftp://vidispine:pA5sw0rd!?@10.85.0.10/storage/
  • azure://:%2ZmFuODl0MGg0MmJ5ZnZuczc5YmhndjkrZThodnV5Ymhqb2lwbW9lcmN4c2Rmc2Q0NThmdjQ0Mzc4cWF5NGcxNg0Kdjg0NyANCmw3csO2NWk%3D%3D@vsstorage/

Method types

Methods can also be of different type. By default, the type is empty. Only those methods (with empty types) are used by Vidispine when doing file operations, the other methods are ignored, but can be returned, for example when requesting URLs in search results.

Credentials are encrypted. This means that passwords cannot be viewed through the API/server logs.

Auto method types

One exception is method type AUTO, or any method type with prefix AUTO-. When a file URL is requested, with such method type, the a no-auth URL will be created (with the method URL as base).

If there is no AUTO method defined, but a file URL is requested with method type AUTO, an implicit one will be used automatically.

GET /item/VX-2406?content=uri&methodType=AUTO
Accept: application/xml
<ItemDocument xmlns="http://xml.vidispine.com/schema/vidispine" id="VX-2406">
  <files>
    <uri>http://vs.example.com:8089/APInoauth/storage/VX-1/file/VX-6537/0.7354486788234469/VX-6537.mp4</uri>
    <uri>http://vs.example.com:8089/APInoauth/storage/VX-1/file/VX-6536/0.7638025887084131/VX-6536.dv</uri>
  </files>
</ItemDocument>

The URL returned is only valid for the duration of fileTempKeyDuration minutes. The expiration timer is reset whenever the URL is used in a new operation (e.g. HEAD or GET).

AUTO-VSA method type

New in version 4.16.

When using URIs generated from the AUTO method type with a VSA storage, the files will be streamed from VSA through Vidispine server. Instead of that, the ÀUTO-VSA method type can be used to generate proxy URIs, which can later be used to generate noAuth URIs from the VSA on-demand.

The same Vidispine configuration property fileTempKeyDuration (default 10 minutes) is used to control the duration of both the proxy URI from the server and noAuth URI from the VSA.

Example:

First, generate a AUTO-VSA noauth URI:

GET /storage/file/VX-123?methodType=AUTO-VSA
Accept: application/xml

Response:

<FileDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-123</id>
  <path>demo.mov</path>
  <uri>
    http://localhost:8080/APInoauth/proxy/4e714b56-c3ab-49e9-b3f3-224aeaad7380?redirect=true
  </uri>
  <state>CLOSED</state>
  ...
</FileDocument>

And then, ask VSA to generate a noauth URI.

GET http://localhost:8080/APInoauth/proxy/4e714b56-c3ab-49e9-b3f3-224aeaad7380?redirect=true

Response:

HTTP/1.1 302 Found
Date: Thu, 20 Dec 2018 16:23:53 GMT
Accept-Ranges: bytes
Location: http://127.0.0.1:7090/4016eff6-5801-4ed2-a89d-518c9ee3b54a/demo.mov
Content-Length: 0

The URI in the Location header can be used to stream files from VSA directly.

The VSA noauth service will be running on port 7090 by default. And a noAuthUri property can be added to agent.conf to configure the noauth URI returned from the VSA.

For example:

noAuthUri=http://example.com:7090

Method metadata

In addition to select method types, method metadata can be given as instructions for the URI returned. Two metadata values are defined:

format

Specifies if any special format of the URI should be returned. By default, the normal URI is returned. Two values are defined:

SIGNED
Returns a http URI that points contains a signed URI directly to Azure or S3 storage. If a signed URI cannot be generated from the underlying (default) URI, no URI is returned.
SIGNED-AUTO
As above, but if no URI can be generated, an AUTO URI (see above) is returned.
expiration

Sets the expiration time of the signed URI, in minutes. To set a hard limit for the requested expiration time maxRequestedSignedURLTime should be set.

If not specified, the default expiration time is 60 minutes, unless azureSasValidTime or s3ProxyValidTime is set in which case these will act as default duration times.

contentDisposition
Sets the Content-Disposition header for the signed URI. If not specified, the Content-Disposition header will be set to null.
vsauri

Specifies if the VSA URI (schema vxa) should use UUID or name syntax. By default, UUID is used.

UUID
Return URI with hostname being the UUID of the VSA.
NAME
Return URI with hostname being the NAME of the VSA.
GET /item/VX-206?content=uri&methodMetadata=format=SIGNED-AUTO&methodMetadata=contentDisposition=attachment%3b+filename%3dmyfile.mov
Accept: application/xml
<ItemDocument xmlns="http://xml.vidispine.com/schema/vidispine" id="VX-206">
  <files>
    <uri>https://vstest.s3.amazonaws.com/VX-362.mp4?Expires=1439545041&amp;AWSAccessKeyId=AKIAJCCXQRY2MW4YQUVQ&amp;Signature=UcNdTIm1v1omM%2FaIGaYXf4QNfc%3D</uri>
    <uri>http://vs.example.com:8089/APInoauth/storage/VX-1/file/VX-336/0.7638025117084131/VX-336.dv</uri>
  </files>
</ItemDocument>

Parent directory management

For local file systems (method is using a file:// URI), Vidispine will by default remove empty parent directories when deleting the last file in the directory.

This can be controlled, either on system level or on storage level. If the storage metadata keepEmptyDirectories is set to true, empty directories are preserved in that storage. Likewise, if the configuration property keepEmptyDirectories is set to true, empty directories are preserved for all storages. Storage configuration overrules system configuration.

Storage scanning algorithm

By default, local file systems are scanned using what is called file visitors, which provides the best performance.

However, for some storages, especially mounted storages, ACLs on the file system may cause that algorithm to fail. By specifying the algorithm, if is possible to force VidiCore to use another algorithm.

This can be controlled, either on system level or on storage level, by the storage metadata scanMethodAlgorithm. Possible values are:

  • VISITOR - use file visitors if possible, otherwise iterator. This is the default.
  • ITERATE - use file iterators
  • LEGACY

Files

When are files scanned?

In order to discover changes made to files, or if any files have been removed/added, Vidispine will scan the storages periodically. It is possible to disable the scanning by not having any methods with browse=true on the storage. The scan interval is also configurable on a per storage basis by setting the scanInterval property. The value should be in seconds. Setting this to a higher value will lower the I/O load of the device, but any file changes will take longer to be discovered. This also means that file notifications for file changes or file creation will be triggered later for changes occurring outside of Vidispine’s control.

You can force a rescan of a storage by calling POST /storage/(storage-id)/rescan. This will trigger an immediate rescan of a storage if the supervisor is idle. If a supervisor is already busy processing the files then you may notice that the rescan happens some time later.

Avoiding frequent scan of S3 storages

Scanning a S3 storage can be expensive both in terms of time and money. To make it cheaper to access a S3 bucket, you can configure Vidispine to poll Amazon SQS for S3 events.

See S3 Event SQS Notifications for more information.

File States

Files can be in one of the following states:

NONE
Just created, not used.
OPEN
Discovered or created, not yet marked as finished.
CLOSED
File does no longer grow.
UNKNOWN
The current state is not known.
MISSING
File is missing from the file system/storage.
LOST
File has been missing for a longer period. Candidate for restoration from archive.
TO_APPEAR
File will appear on file system/storage, transfer subsystem or transcoder will create it.
TO_BE_DELETED
The file is no longer in use, and will be deleted at the next clean-up sweep.
BEING_READ
File is in use by transfer subsystem or transcoder.
ARCHIVED
File is archived.
AWAITING_SYNC
File will be synchronized by multi-site agent.

Vidispine will mark a file as MISSING when it is first detected that the file no longer exists on the storage. No action is taken for files that are missing. If the file does not appear within the time specified by lostLimit, then the file will be marked as LOST. Lost files will be restored from other copies if such exist.

Items and storages

By default, when creating a new file, Vidispine will choose the LOCAL storage with the highest free capacity. This can be changed in a few different ways:

File hashing

Vidispine will calculate a hash for all files in a storage. This is done by a background process, running continuously. Files are hashed one by one for performance reasons, so if a large number of files are added to the system in a short time span it might take some time for all hashes to be calculated. The default hashing algorithm is SHA-1. This can be changed by setting the configuration property fileHashAlgorithm. See below for a list of supported values.

Additional algorithms

Vidispine can be configured to calculate hashes using additional algorithms by setting the additionalHash metadata field on the storage. It should contain a comma separated list (no spaces) of algorithms. The supported algorithms are:

  • MD2
  • MD5
  • SHA-1
  • SHA-256
  • SHA-384
  • SHA-512

Manual hashing

Automatic background hashing can be disabled by setting the hashMode metadata field on the storage. A hash can then be set manually by calling PUT {file-resource}/hash/(hash).

Throttling storage I/O

Vidispine will retrieve information about files on a storage at the configured scan intervals. If you find that the I/O on your local disk drives is high, even when no transfers or transcodes are being performed, then you can try rate limiting the stat calls performed by Vidispine. Do this by setting statsPerSecond or the configuration property statsPerSecond to a suitable limit. During the file system scan, Vidispine will typically perform one stat per file.

An easy way to check if rate limiting the stat calls will have any effect is to disable the storage supervisors in Vidispine. This can be done using PUT /vidispine-service/service/StorageSupervisorServlet/disable. Remember to enable the service afterwards or you will find that Vidispine no longer detects new files on the storages, among other things.

It could also be that it’s the file hashing service that is the cause of the I/O. You should be able to tell which service is behind it by monitoring your disk devices. If there’s a high read activity/a large amount of data read from a device then it could be the file hashing that’s the cause. If the number of read operations per seconds is high then it’s more likely the storage supervisor.

Tip

Use tools such as htop, iotop, dstat and iostat to monitor your systems and devices.

Throttling transfer to and from a storage

It is possible to specify a bandwidth on a storage or a specific storage method. This causes any file transfers involving the specified storage or storage method to be throttled. If multiple transfers take place concurrently, the total bandwidth will be allocated between the transfers. If a bandwidth is set on both the storage and its storage methods, the lowest applicable bandwidth will be used.

To set a bandwidth you can set the bandwidth element in the StorageMethodDocument when creating or updating a storage or storage method. The bandwidth is set in bytes per second.

Example

Updating a storage to set a bandwidth of 50,000,000 bytes per second.

PUT /storage/VX-2
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>1000000000</capacity>
  <bandwidth>50000000</bandwidth>
</StorageDocument>

Example

Updating a storage method to set a bandwidth of 20,000,000 bytes per second.

PUT /storage/VX-2/method?uri=http://10.5.1.2/shared/&bandwidth=20000000

Temporary storages for transcoder output

The Vidispine transcoder requires that the destination (output) file can be partially updated. This is in order to be able to write header files after the essence has been written.

In previous versions, this is solved by the application server storing the intermediate result as a temporary file on the local file system (/tmp). This requires a lot of space on the application server.

With version 4.2.3, another strategy is available. Instead of storing the result as one file on the application server, several small files are stored directly on the destination file system as “segments”. After the transcode has finished, the segments are merged. On S3 storage, this merging can be done with S3 object(s)-to-object copy.

Control of the segment file strategy is via the useSegmentFiles configuration property.

Storage credentials

Storage credentials can be specified either directly in the storage URI or they can be stored in a credentials storage and referenced by an alias.

For example, a FTP storage could be configured either using ftp://testuser:testpassword@ftp.example.com/, or using ftp://exampleftp@ftp.example.com/; with exampleftp being an alias referencing the stored credentials.

Credentials can be stored in either:

  • A Java Keystore.
  • The local file system.
  • HashiCorp Vault.
  • In the VidiCore database, using the API secret endpoint.

Support and configuration for stored credentials depends on the URI schema, see Storage method URIs.

Java Keystore

A Java Keystore can be used to store private keys, for example, the private keys for a Google Cloud Platform service account. Configured in the server configuration file.

server.yaml
secrets:
  keyStore:
    path: /etc/vidispine/server.keystore
    password: changeit

Local file system

Credentials can be stored in files on the local file system. This could be one way to consume credentials from secrets in Kubernetes, or similar services that expose secrets via the local file system. Configured in the server configuration file.

A directory is configured as the root path and credentials are then stored within this directory. The file hierarchy and filenames to use within this root directory depends on the URI schema, see Storage method URIs.

Example:

server.yaml
secrets:
  file:
    path: /etc/secrets/

HashiCorp Vault

Using HashiCorp Vault, the alias used in storage URIs should match a secret in the Vault. The keys that will be read from that secret depends on the URI schema, see Storage method URIs. Configured in the server configuration file.

For example:

server.yaml
secrets:
  vault:
    address: http://vault.example.com:8200
    token: 2262e94c-39c3-b9a8-605d-f0450dfc558b
    keyPrefix: secret/

The keyPrefix setting can be used to for example select the backend to use. For example, with Vault configured with a “generic” backend mounted at secret/:

$ vault mounts
Path        Type       Default TTL  Max TTL  Description
secret/     generic    system       system   generic secret storage
sys/        system     n/a          n/a      system endpoints used for control, policy and debugging

In the following example the keys username and password are set, as required by the FTP URI schema.

$ vault write secret/exampleftp username=testuser password=testpassword
$ vault read secret/exampleftp
Key                  Value
---                  -----
refresh_interval     720h0m0s
password             testpassword
username             testuser

API secret endpoint

New in version 23.1.

Credentials are stored write-only in the VidiCore database. The endpoint is documented in API Reference Secret, and the values that will be read depends on the URI schema, see Storage method URIs.

Example: create a new secret and add username and password credentials to it:

PUT /secret/export-ftp-credentials
PUT /secret/export-ftp-credentials/username
PUT /secret/export-ftp-credentials/password

The credentials can then be used with an URI like ftp://export-ftp-credentials@ftp-server/export/.

Storage method URIs

Note

Storage method URIs require URI escaping for all characters that are reserved in URIs.

The following URI schemes are defined.

file

Syntax:file:///{path}
Example:file:///mnt/storage/, file:///C:/mystorage/
Note:The URI file://mnt/storage/ is not valid! (But file:/mnt/storage/ is.)

ftp

Syntax:ftp://{user}:{password}@{host}/{path}
Example:ftp://johndoe:secr3t@example.com/mystorage/

Stored credentials:

Syntax:ftp://{alias}@{host}/{path}
Local file system:Files <root>/<alias>/username and <root>/<alias>/password
HashiCorp Vault and API secret endpoint:
 Keys username and password

Add query parameter passive=false to force active mode. To set the client side ports used in active mode, set the configuration property ftpActiveModePortRange, the value should be a range, e.g. 42100-42200.

To set the client IP used in active mode, set the configuration property ftpActiveModeIp.

New in version 4.17: For some servers using a basic implementation of ftp and which does not support some of the commands often found, e.g. listing a directory without having to step into it first, the query parameter serverType=basic can be used if issues with connecting and listing files are experienced. This will in some cases provide a better compatibility.

sftp

Syntax:sftp://{user}:{password}@{host}/{path}
Example:sftp://johndoe:secr3t@example.com/mystorage/

Stored credentials:

Syntax:

sftp://{alias}@{host}/{path}

Note:

Currently only PKCS#1 keys are supported.

Local file system:

Files <root>/<alias>/username and <root>/<alias>/password, or

Files <root>/<alias>/username, <root>/<alias>/private_key and optionally <root>/<alias>/private_key_password

HashiCorp Vault and API secret endpoint:
 

Keys username and password, or

Keys username, private_key and optionally private_key_password

http

Syntax:http://{user}:{password}@{host}/{path}
Example:http://johndoe:secr3t@example.com/mystorage/
Note:Requires WebDAV support in host.

https

Syntax:https://{user}:{password}@{host}/{path}
Example:https://johndoe:secr3t@example.com/mystorage/
Note:Requires WebDAV support in host.

omms

Syntax:omms://{userId}:{userKey}@{hostList}/{clusterId}/{vaultId}/
Example:omms://c2f6a2f4-6927-11e1-cc94-ab94bd11183f:some%20secret@10.0.0.3,10.0.0.4/4255378f-dc73-fca3-e40d-5726008b3dac/0a49472d-15d4-12f1-862e-f9708d49267e/
Note:Object Matrix Matrix Store.

s3

Syntax:s3://{accessKey}:{secretKey}@{bucket}/{path}
Example:s3://KDASODSALSDI8U:RxZYlu23NDSIN293002WdlNyq@mystore/storage1/

Stored credentials:

Syntax:s3://{alias}@{bucket}/{path}
Local file system:Files <root>/<alias>/username (access key ID) and <root>/<alias>/password (secret access key)
HashiCorp Vault and API secret endpoint:
 Keys username (access key ID) and password (secret access key)

If no credentials are provided either directly in the URI or via stored credentials, then VidiCore will attempt to read credentials read from the AwsCredentials.properties file in the credentials directory, if one exists. Else, credentials will be read from the default locations used by the AWS SDK.

Valid S3 bucket names must agree with DNS requirements.

The following query parameters are supported:

region

The region parameter is used for two things:

  • To determine the endpoint, if no endpoint has been explicitly set
  • Used when signing requests with AWS Signature Version 4

This parameter is optional for AWS S3. VidiCore can automatically detect the bucket’s region.

See Regions and Endpoints in the Amazon documentation for more information.

endpoint

The endpoint that S3 requests will be sent to.

This parameter is optional for AWS S3. If not explicitly set, VidiCore will use an endpoint based on the bucket’s region, see above.

See Regions and Endpoints in the Amazon documentation for more information.

signerRegion

Deprecated since version 23.2.

This parameter is no longer available. It was implemented as a workaround as the old AWS SDK did not allow both endpoint and region to be set. With AWS SDK v2 this is now possible.

VidiCore will still read this parameter, to keep backwards compatibility. It will override and do the same thing as the region parameter. Having both these parameters set to different values are not supported.

signer

Deprecated since version 23.2.

With AWS SDK v2, only AWS Signature Version 4 is supported.

Storage method metadata keys can be used control the interaction with the storage.

storageClass

The default Amazon S3 storage class that will be used for new files created on an Amazon S3 storage. Can be either standard, infrequent or onezone-infrequent

Default:standard
sseAlgorithm

The encryption used to encrypt data on the server side. See Server-Side Encryption. By default, new objects will be encrypted with AES256 (SSE-S3).

This sets the x-amz-server-side-encryption header on PUT Object S3 requests.

Example:AES256
sseKeyId

The encryption used to encrypt data on the server side. See Server-Side Encryption. By default no encryption will be performed.

This sets the x-amz-server-side-encryption-aws-kms-key-id header on PUT Object S3 requests.

If the sseAlgorithm is present and has the value of aws:kms, this indicates the ID of the AWS Key Management Service (AWS KMS) master encryption key that was used for the object.

The KMS KEY you specify in the policy must use the arn:aws:kms:region:acct-id:key/key-id format.

Example:arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01
accelerate

Enable S3 Transfer Acceleration.

Default:false

Note

For S3 Transfer Acceleration to work, the endpoint or region parameter must be set. Also make sure that transfer acceleration is enabled on the bucket.

Other S3 compatible endpoints may not support transfer acceleration.

retrievalTier
The default Glacier retrieval tier to use when restoring the file. Can be set to either Expedited, Standard or Bulk. See Restoring Archived Objects for more information.

ssl

Deprecated since version 23.2.

VidiCore is by default using SSL when communicating with S3. To disable SSL, set the endpoint parameter to an http:// url.

roleArn

New in version 21.3.

The role ARN to try to assume to access the content of the bucket.

In order to be able to access buckets and content across accounts, it is now possible to supply a role ARN that VidiCore will try to assume to access the data.

roleExternalId

New in version 21.3.

The (optional) external id attached to the role specified as roleArn

stsRegion

(optional) The region to where calls to assume role are made (AWS STS). This should be set to something as close to your system as possible to reduce latency and get better reponse times (example: eu-west-1, us-east-2).

Note

  • When a role is being assumed VidiCore will need to contact AWS Security Token Service (STS) in order to complete the request. Unless the system is running on EC2/ECS the best practice when using role ARN for S3 storages would be to make sure the stsRegion parameter is being used. If this is not supplied, VidiCore will take more time trying to figure out which region to call (see below).
  • If no region is specified OR VidiCore is NOT running on EC2/ECS, VidiCore will fallback to the AWS default region which would be us-west-2. This is not recommended for optimal performance.
bucketOwnerFullControl

Support for controlling ownership of uploaded objects.

Set to true to have VidiCore attach the needed canned ACL to any uploads to this storage.

New in version 21.4.1.

Note

It is important to know that in order to have this feature working with VidiCore as a managed storage, you must only restrict s3:PutObject with the “s3:x-amz-acl”: “bucket-owner-full-control” in its own statement. Other actions such as s3:GetObject, s3:ListBucket etc must still be allowed without this restriction in order for VidiCore to manage the storage. Information about what actions VidiCore need to function properly can be found in the section Configuring and adding permissions IAM here.

ds3

Syntax:ds3://{accessKey}:{secretKey}@{bucket}/{path}
Example:ds3://KDASODSALSDI8U:RxZYlu23NDSIN2Nyq@bucketname/?endpoint=http://blackpearl-endpoint
Note:Spectra BlackPearl Deep Storage Gateway.

The following query parameters are supported:

endpoint
The endpoint of the BlackPearl service. This is mandatory.
chunkReadyTimeout

The maximum time (in seconds) of waiting for BlackPearl to prepare the target data chunk, or an EOF will be returned.

Default:1800
checksumType

If set, a client-side checksum will be computed and sent to BlackPearl gateway for data integrity verification. Supported checksum types are: md5, crc32 and crc32c.

Default:Empty, no checksum will be sent.

azure

Syntax:azure://:{accessKey}@{accountName}/{containerName}
Example:azure://:KLKau23dEE02WdlLiO@companyname/container1/

gs

Google Cloud Storage.

Using an OAuth2 access token:

Syntax:gs://:{accessToken}@{bucket}/?project={project}
Example:gs://:abc123@test-bucket/?project=12345

Using the credentials file specified in the GOOGLE_APPLICATION_CREDENTIALS environmental variable:

Syntax:gs://{bucket}/
Example:gs://test-bucket/

Stored credentials, using a PKCS#12 (.p12) private key:

Syntax:gs://{alias}@{bucket}/?project={project}&account={account}
Example:gs://test-key-p12@test-bucket/?project=12345&account=67890
Local file system:File <root>/<alias>, or file <root>/<alias>/private_key if <root>/<alias> is a directory.
HashiCorp Vault and API secret endpoint:
 Key private_key

Stored credentials, using a JSON private key:

Syntax:gs://{alias}@{bucket}/?project={project}
Example:gs://test-key-json@test-bucket/?project=12345
Local file system:File <root>/<alias>
HashiCorp Vault and API secret endpoint:
 Key private_key

universal

A universal URI is used to create a universal storage method. A universal storage method does not have a root URI, instead all files contain their own absolute URI.

See also

See Vidispine API URI encode for some notes on how to write URIs.

The universal storage method

A universal storage can be used to let Vidispine manage files which are not stored under a common root. Universal storages can be used like other storages, but there are certain differences. Before jumping to the differences, an example on how to use the storage:

Adding a universal storage

POST /storage
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <method>
    <uri>universal:/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
  </method>
  <showImportables>true</showImportables>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>STORAGE-VX-722</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>10000000000000</capacity>
  <freeCapacity>10000000000000</freeCapacity>
  <method>
    <id>STORAGEMETHOD-VX-728</id>
    <uri>universal:/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>10000000000000</lowWatermark>
  <highWatermark>10000000000000</highWatermark>
  <showImportables>true</showImportables>
</StorageDocument>

Adding a file

POST /storage/VX-722/file?path=file:///home/baz/vacation.mp4
<FileDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>FILE-VX-68264</id>
  <path>file:///home/baz/vacation.mp4</path>
  <uri>file:///home/baz/vacation.mp4</uri>
  <state>OPEN</state>
  <size>-1</size>
  <timestamp>2017-05-11T13:37:46.737+02:00</timestamp>
  <refreshFlag>1</refreshFlag>
  <storage>STORAGE-VX-722</storage>
  <metadata/>
</FileDocument>

After scanning, the metadata and hash checksum of the file will be updated.

Adding and importing a file

A file can be registered to a universal storage with its original URI, and imported at the same time:

POST /storage/VX-722/file/import?path=https://www.vidispine.com/wp-content/themes/vidispine/assets/image/vidispine-logo-small.png

The HTTPS URI in the request will be the actual source of the original shape of the item created.

Compared with a regular import request:

POST /import?uri=https://www.vidispine.com/wp-content/themes/vidispine/assets/image/vidispine-logo-small.png

The source file will be copied to a Vidispine managed storage. The newly copied file will be the file that makes up the original shape of the item. The HTTPS URI is then no longer used after the import.

Differences to regular methods

  • New files are not discovered by a universal method. For new files to be registered, an API call has to be done. However, Vidispine will detect when files have changed or been deleted.

  • Files can be written to a universal storage. However, it requires that either a full URI is given as API input, or returned by a file name script. Example for copying a file:

    POST /storage/file/VX-4448/storage/VX-722?filename=file:///tmp/somenewfile.txt
    
  • There is no capacity detection.

  • Scanning can be slower than for regular storages. The universal URI is not meant to be used to thousands of files in one file system. Then it is better to use the regular URI, and reference files by their relative paths.