Storages

Manage storages.

Managing storages

List all storages

GET /storage

Retrieves the storages that have been configured.

Query Parameters:
  • size (string) – Range of bytes, in format s1-s2. Returns storages with nominal size that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages.
  • freebytes (string) – Range of bytes, in format s1-s2. Returns storages with free space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages.
  • usedbytes (string) – Range of bytes, in format s1-s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages.
  • freeamount (string) – Range of percent as integers, in format s1-s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Default is -, all storages.
  • files (string) – Range of files as integers, in format s1-s2. Returns storages with number of files that is in that range. Either number can be omitted to not specify lower/upper limit. Default is -, all storages.
  • storagegroup (string[]) –

    List of storage groups.

    • storage-group - Returned storage is member of specified storage group.
    • -storage-group - Returned storage is not member of specified storage group.

    Default is to return all storages.

  • status (string[]) –

    List of storage status.

    • status - Returned storage has this status.
    • -status - Returned storage does not have this status.

    Default is to return all storages.

  • url (string) – Returns storages with a method matching this URL. May include wildcards * and ?.
Produces:
  • application/xml, application/jsonStorageListDocument
  • text/plain – CRLF-delimited list of storage ids
Role:

_storage_read

Example

GET /storage
<StorageListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <storage>
    <id>VX-1</id>
    <state>NONE</state>
    <type>LOCAL</type>
    <capacity>1600000000000</capacity>
    <freeCapacity>19993896424</freeCapacity>
    <method>
      <id>VX-6</id>
      <uri>file:///mnt/main/</uri>
      <bandwidth>0</bandwidth>
      <read>true</read>
      <write>true</write>
      <browse>true</browse>
      <lastSuccess>2014-07-04T08:39:00.739+02:00</lastSuccess>
      <type>NONE</type>
    </method>
    <metadata/>
    <lowWatermark>0</lowWatermark>
    <highWatermark>0</highWatermark>
    <showImportables>true</showImportables>
  </storage>
  <storage>
    <id>VX-2</id>
    <state>NONE</state>
    <type>LOCAL</type>
    <capacity>15000000000</capacity>
    <freeCapacity>19548305962</freeCapacity>
    <method>
      <id>VX-4</id>
      <uri>file:///mnt/ingest/</uri>
      <read>true</read>
      <write>true</write>
      <browse>true</browse>
      <lastSuccess>2014-07-04T08:38:56.773+02:00</lastSuccess>
      <type>NONE</type>
    </method>
    <metadata/>
    <lowWatermark>0</lowWatermark>
    <highWatermark>200000000000</highWatermark>
    <showImportables>true</showImportables>
  </storage>
</StorageListDocument>

Create a storage

POST /storage

Creates a new storage.

Accepts:
Produces:
  • application/xml, application/jsonStorageDocument
  • text/plain – Storage id
Role:

_storage_write

Note

If your storage is to be used for autoimport, the showImportables property in the StorageDocument must be set to true. The files on the storage will then be discovered and available for use with GET /storage/importable or GET /storage/auto-import/.

New in version 23.4: To configure a storage to be managed by a VSA:

  • create a vxa storage method containing the uuid of the VXA and the name of the share to be created within the VSA.
  • Set the key-value metadata vxaLocalPath on the storage method to the local path of the storage location as seen by the VSA.

For pre-configuring a storage for a VSA not existing yet, generate a uuid, configure the storage with this uuid, and use this uuid when deploying the VSA at a later time.

Example:

POST /storage
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
    <type>LOCAL</type>
    <method>
        <uri>vxa://b85c6f2c-49d6-4fee-bb57-7e556ebedaf5/media/</uri>
        <read>true</read>
        <write>true</write>
        <browse>true</browse>
        <type>NONE</type>
        <metadata>
            <field>
                <key>vxaLocalPath</key>
                <value>/mnt/assets/media</value>
            </field>
        </metadata>
    </method>
    <highWatermarkPercentage>100</highWatermarkPercentage>
    <lowWatermarkPercentage>100</lowWatermarkPercentage>
    <autoDetect>true</autoDetect>
    <showImportables>true</showImportables>
</StorageDocument>

Examples

POST /storage
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <method>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
  </method>
  <lowWatermarkPercentage>90</lowWatermarkPercentage>
  <highWatermarkPercentage>75</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>112500000000</lowWatermark>
  <highWatermark>135000000000</highWatermark>
  <lowWatermarkPercentage>75</lowWatermarkPercentage>
  <highWatermarkPercentage>90</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Retrieve a storage

GET /storage/(storage-id)

Retrieves a specific storage.

Produces:
Role:

_storage_read

Example

GET /storage/VX-2
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>112500000000</lowWatermark>
  <highWatermark>135000000000</highWatermark>
  <lowWatermarkPercentage>75</lowWatermarkPercentage>
  <highWatermarkPercentage>90</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Update a storage

PUT /storage/(storage-id)

Updates an existing storage.

New in version 22.2.

It is possible to add Resource tags to the storage and storage method.

Accepts:
Produces:
  • application/xml, application/jsonStorageDocument
  • text/plain – Storage id

Example

PUT /storage/VX-2
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <lowWatermarkPercentage>80</lowWatermarkPercentage>
  <highWatermarkPercentage>95</highWatermarkPercentage>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <resourceTag>
      <key>priority</key>
      <value>MEDIUM</value>
    </resourceTag>
  </method>
  <resourceTag>
    <key>storage-location</key>
    <value>london,berlin</value>
  </resourceTag>
  <resourceTag>
    <key>priority</key>
    <value>HIGH</value>
  </resourceTag>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
    <resourceTag>
      <key>priority</key>
      <value>MEDIUM</value>
    </resourceTag>
    <resourceTag sourceName="VX-4" sourceType="storage">
      <key>storage-location</key>
      <value>london,berlin</value>
    </resourceTag>
  </method>
  <metadata/>
  <resourceTag>
    <key>priority</key>
    <value>HIGH</value>
  </resourceTag>
  <resourceTag>
    <key>storage-location</key>
    <value>london,berlin</value>
  </resourceTag>
  <lowWatermark>120000000000</lowWatermark>
  <highWatermark>142500000000</highWatermark>
  <lowWatermarkPercentage>80</lowWatermarkPercentage>
  <highWatermarkPercentage>95</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Delete a storage

DELETE /storage/(storage-id)

Deletes the storage. All files in storage will remain after call, but the Vidispine system will no longer manage them.

Query Parameters:
  • safe (boolean) –
    • true - Storage will only be deleted if there are no files connected to items on the storage.
    • false (default) - Storage will be deleted, and any file entities will be disconnected from items and deleted.
Status Codes:
  • 409 Conflict – If safe parameter is true this error code will be given if there are files connected to items on the storage.
Role:

_storage_write

Storage information

Update the storage type

PUT /storage/(storage-id)/type/(type)

Sets the type of the storage.

Role:_storage_write

Retrieve the storage status

GET /storage/(storage-id)/status

Retrieves the status of the storage.

Produces:
  • text/plain – Status string
Role:

_storage_read

Retrieve the amount of free space on a storage

GET /storage/(storage-id)/freespace

Retrieves the amount of free space on the storage.

Produces:
  • text/plain – Amount of free space as decimal number, between 0 and 100, inclusive
Role:

_storage_read

Rescanning

Rescan a storage

POST /storage/(storage-id)/rescan

Triggers a rescan of a single storage.

The scanInterval property can be used to control how often (in seconds) a storage is scanned. Default is 60. By setting scanInterval to -1 storage scans will be disabled. By calling /rescan, the system is forced to rescan a storage without delay.

Query Parameters:
  • path (string) –

    Rescans a specific path on storage to find importable files. This will not update deleted files.

    New in version 5.4.

Storage methods

Storage methods specify the different ways of accessing files on a specific storage.

Credentials are encrypted. This means that passwords cannot be viewed through the API/server logs.

List storage methods

GET /storage/(storage-id)/method

Retrieves the access methods configured on a specific storage.

Query Parameters:
  • url (string) – Only return methods with this URL. Wildcards (?, *) can be used, for example http:*.
  • read (boolean) –
    • true - Only return methods which have file read capability.
    • false (default) - Return methods regardless of file read capability.
  • write (boolean) –
    • true - Only return methods which have file write capability.
    • false (default) - Return methods regardless of file write capability.
  • browse (boolean) –
    • true - Only return methods which have file browse capability.
    • false (default) - Return methods regardless of file browse capability.
Produces:
Role:

_storage_read

Example

GET /storage/VX-2/method
<StorageMethodListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <bandwidth>0</bandwidth>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <lastSuccess>2014-07-04T09:55:09.779+02:00</lastSuccess>
    <type>NONE</type>
  </method>
</StorageMethodListDocument>

Update or create a storage method

PUT /storage/(storage-id)/method

Adds a new access method to the storage. If URL matches an existing method, a new method is not created, instead the existing one is updated.

Query Parameters:
  • url (string) – Required. Method is accessed through this URL
  • read (boolean) –
    • true (default) - Method has file read capability
    • false - Method does not have file read capability
  • write (boolean) –
    • true (default) - Method has file write capability
    • false - Method does not have file write capability
  • browse (boolean) –
    • true (default) - Method has file browse capability
    • false - Method does not have file browse capability
  • bandwidth (integer) – The bandwidth of this method in bytes per second. Default 0.
  • type (string) – method type. Default is empty.
Produces:
  • text/plainTabbed tuples : id, URL, status string of method
Role:

_storage_write

Example

PUT /storage/VX-2/method?url=http://10.12.0.3/ingest/&read=true&write=false&browse=false
VX-5        http://10.12.0.3/ingest/        NONE

Retrieve a storage method

GET /storage/(storage-id)/method/(method-id)

Retrieves a specific access method to storage.

Produces:
Role:

_storage_read

Create/update a storage method

PUT /storage/(storage-id)/method/(method-id)

Updates access method to the storage.

Query Parameters:
  • url (string) – Required. Method is accessed through this URL
  • read (boolean) –
    • true - Method has file read capability.
    • false - Method does not have file read capability.
  • write (boolean) –
    • true - Method has file write capability.
    • false - Method does not have file write capability.
  • browse (boolean) –
    • true - Method has file browse capability.
    • false - Method does not have file browse capability.
  • bandwidth (integer) – The bandwidth of this method in bytes per second. Default 0.
  • type (string) – method type.
Produces:
  • text/plainTabbed tuples : id, URL, status string of method
Role:

_storage_write

Delete a storage method

DELETE /storage/(storage-id)/method/(method-id)
DELETE /storage/(storage-id)/method

Removes an access method from a storage.

Query Parameters:
  • url (string) – Method is accessed through this URL
Role:

_storage_write

Importing/exporting a storage definition

Vidispine can export a definition document describing all the files within a storage, which can later be imported to a new storage on a different Vidispine instance.

Export a storage definition

GET /storage/(storage-id)/export

Creates a StorageImportDocument that describes every file on the storage. This should be saved to a file which can later be used to import the storage definition.

Produces:
Role:

_storage_read

Import a storage definition

POST /storage/import

Creates a new storage based on the StorageImportDocument. A file entity will be created for each entry in the document, if a file with that ID does not already exist. Finally, a storage method will be added, with the path supplied in the call.

Query Parameters:
  • path (string) – The file system path to where the files are located.
Accepts:
Role:

_storage_write

Evacuating storages

If you would like to delete a storage, but you still have files there which are connected to items, you can first trigger an evacuation of the storage. This will cause Vidispine to attempt to delete redundant files, or move files to other storages. Once the evacuation is complete, the storage will get the state EVACUATED.

Evacuate a storage

PUT /storage/(storage-id)/evacuate

Trigger evacuation of a storage.

Role:_storage_write

Cancel evacuation of a storage

DELETE /storage/(storage-id)/evacuate

Cancel the evacuation process on a storage.

Role:_storage_write

Key-value metadata

Storages support key-value metadata.

Reserved keys

Certain keys are used to control the behavior of a storage, they must not be used to store generic metadata.

For systems with different stages the Environment Dependent flag shows whether a setting is considered to be identical in all stages (Environment Dependent: false) or whether it makes sense (or even is required) to configure it differently in each stages (Environment Dependent: true).

closeLimit

An integer specifying the number of minutes until the system automatically considers an open file to be closed.

Data Type:int
Default:5
Environment Dependent:false
lostLimit

An integer specifying the number of minutes until the system automatically considers a missing file to be lost.

Data Type:int
Default:10
Environment Dependent:false
toAppearLimit

An integer specifying the number of minutes the system waits for a file to appear before considering it to be missing.

Data Type:int
Default:10
Environment Dependent:false
refreshInterval

The interval (in seconds) that Vidispine will compare the list of file seen on disk with the list of files in the database. This to fix any possible mismatch.

Data Type:int
Default:900
Environment Dependent:false
hashMode

Set to false to disable hash computation for this storage.

Data Type:bool
Default:true
Environment Dependent:false
additionalHash

A comma-separated list of additional hashing algorithms that should be applied to files. E.g.: MD5,SHA-256.

The hash algorithm FILENAME is also supported, which takes the last part of the path concatenated with the file size, and calculates a SHA-1 checksum of that string.

Data Type:string
Default:(none)
Environment Dependent:false
fileListBatchSize

Maximum size of list of files that are updated in the database at the same time.

Data Type:int
Default:100000
Environment Dependent:false
keepMissingFiles

If set to false then missing files that do not belong to any items will be removed from the database instead of being marked as lost.

Data Type:bool
Default:To the value of the configuration property keepMissingFiles.
Environment Dependent:false
keepEmptyDirectories

Do not delete empty parent directories when deleting the last file in a directory, see Parent directory management.

Data Type:bool
Default:To the value of the configuration property keepEmptyDirectories.
Environment Dependent:false
statsPerSecond

An integer specifying the maximum number of stat system calls that are made to the storage (only file:// URIs). See also the configuration property statsPerSecond.

Data Type:int
Default:(none)
Environment Dependent:false
scanOnStart

By default, Vidispine will scan the file system on start-up. Storages that should never be scanned, that should solely use SQS or SNS for example, should have this parameter set to false.

Data Type:bool
Default:true
Environment Dependent:false
Since:5.0
refreshOnStart

By default, Vidispine synchronizes the database with the file system on start-up. On very large systems, this can take a very long time. Set this parameter to false, and use the rescan functionality (see below) instead.

Data Type:bool
Default:true
Environment Dependent:false
deleteFileIfNotFound

If set to true: if file that is marked for deletion cannot be deleted, but is not found on the file system, it is assumed to be deleted.

Data Type:bool
Default:false
Environment Dependent:false
deleteFileIfReadOnly

If a file that is marked for deletion cannot be deleted, and the file exists on a read-only storage, then:

  • If set to true: delete the file from the database, as if the file had been deleted.
  • If set to a file state: set the file to this state.
Data Type:string
Default:false
AllowedValues:[ true, false, ARCHIVED, LOST ]
Environment Dependent:false
excludeFilter

A regular expression. Any file with a path (relative to the storage’s root folder) that matches the expression will be ignored. Note that the expression must match the entire path, not only a part of the path.

Data Type:string
Default:(none)
Environment Dependent:false
detectRenamedFiles

If this metadata is set to true, whenever a file belonging to an item is marked as LOST, and there is another file on the same storage with the same hash value, the new file is associated with the item instead.

Other possible values are all, where the old (lost) file resided on any storage, or a comma-separated list of identifiers of storages on which the old (lost) file resided on.

Data Type:string
Default:(none)
AllowedValues:[ true, false, all ]
Environment Dependent:false
probeFileBeforeClosing

If this metadata is set to true, Vidispine will ensure that OPEN files are readable and are not being modified by any other process before marking them as CLOSED. Jobs with transfer steps will be set to WAITING with a FileUnavailable job problem until the file becomes available. This is useful for Windows storages where the size and metadata is constant while a file is copied to the storage, but the file cannot be read.

Data Type:bool
Default:false
Environment Dependent:false
probeFileByReading

When probeFileBeforeClosing is set to true, VidiCore’s primary method of probing a file before marking it as CLOSED is to set an exclusive lock on the file. If the storage does not support exclusive locks, a fallback method is used where VidiCore tries to read one byte of the file contents. probeFileByReading can be set to true to make VidiCore always use the fallback method of reading the file. This can be useful for network storages like SMB/CIFS that are mounted on the local filesystem, since setting an exclusive lock on a file may break external applications that are currently reading it.

Data Type:bool
Default:false
Environment Dependent:false
Since:22.4
vxaId

If this metadata field is set, the storage will be automatically deleted if the share is removed from the VidiCore Server Agent.

Data Type:string
Default:(none) for regular storages, automatically set by VSA when a share is added and a storage is created.
Environment Dependent:true
sqsName

The name of Amazon SQS queue to poll for S3 file events. See S3 Event SQS Notifications for more information.

This key is only read for S3 methods.

Data Type:string
Environment Dependent:true
sqsEndpoint

The endpoint of Amazon SQS queue.

This key is only read for S3 methods.

Data Type:string
Example:sqs.eu-west-1.amazonaws.com
Environment Dependent:true
snsTopic

The ARN of an SNS topic to receive S3 file events from. See S3 Event SNS Notifications for more information.

Data Type:string
Example:arn:aws:sns:eu-west-1:123456791011:topic_name
Environment Dependent:true
s3EventETag

When receiving file events from SNS this can be specified to set the ETag included in the message from the storage as metadata on the file.

Data Type:bool
Example:true
Default:false
Environment Dependent:false
s3EventTime

When receiving file events from SNS this can be specified to set the timestamp in the message received as metadata on the file.

Data Type:bool
Example:true
Default:false
Environment Dependent:false
hashingThreadCount

The number of hashing thread that should be run on the VSA side. Note: this configuration only work for VSA storage.

Data Type:int
Default:1
Environment Dependent:false
noDefaultHash

Do not calculate hash checksums for files on the storage.

Data Type:bool
Example:true
Default:false
Environment Dependent:false
verifyHashAfterTransfer

If a job is copying or moving a file to this storage the hash of the source file and the hash of the newly copied/moved file are compared to ensure that the new file is binary identical to the original file. If the hashes differ the job fails and the destination file is removed.

  • If set to true: hash verification is performed after the file has arrived on this storage.
  • If set to false: no hash verification is performed.

The hash verification procedure makes use of the hash calculation that is happening on all storages by default (see File hashing). If hash calculation is disabled on the source storage the hash is calculated before the copy/move operation is performed. If hash calculation is disabled on the destination storage the hash is calculated after the copy/move operation has completed. These hashes are stored in the Vidispine database for later reuse.

Note that hash calculation requires reading the whole file and thus is a lengthy process that can significantly increase execution time of copy/move jobs.

Data Type:bool
Default:false
Environment Dependent:false
storageActivationFile

Require a .storage file to be present in the storage method’s URI for storage to register as online. Takes precedence over the global setting.

Data Type:bool
Default:(none) - The global setting applies.
Environment Dependent:false
Since:4.17
scanMethodAlgorithm

Specify which scan algorithm to use, see Storage scanning algorithm.

Data Type:string
Default:To the value of the configuration property scanMethodAlgorithm.
Environment Dependent:true
Since:5.5.2
syncFileMetadataOnStart

When VidiCore does the initial storage scan after startup, the default behavior does not detect changed file metadata that might have occurred while VidiCore was offline or restarted. Setting this metadata to true will change this behavior so that VidiCore detects changed file metadata during startup.

Note that this setting is only applicable for storage methods that supports file metadata, and does currently not work together with storage sequence patterns.

Data Type:bool
Default:false
Environment Dependent:false
Since:22.2
archiveDirectlyFromStorages

Disables the usage of the staging area when archiving files that are located on these storages. Only applicable to archive integration using JavaScript.

Comma-separated list of storage identifiers, or * to disable it for all storages.

Data Type:string
Default:(none)
Environment Dependent:true
restoreDirectlyToStorages

Disables the usage of the staging area when restoring files to these storages. Only applicable to archive integration using JavaScript.

Comma-separated list of storage identifiers, or * to disable it for all storages.

Data Type:string
Default:(none)
Environment Dependent:true
storageRootIdentifier

For Vidispine storage resources mapping to different paths within the same physical storage like VSA storages, the metadata can be set on storage resources. The move operation uses native move ability to move files between these storage resources if the metadata values are the same.

Example:disk1, galaxy, file:///mnt/dev
Data Type:string
Default:(none)
Environment Dependent:true
Since:22.2.5
s3ObjectTagging

Enables synchronization between S3 Object Tagging and File metadata. See S3 Object Tagging for more information.

Data Type:bool
Default:false
Environment Dependent:false
Since:23.2

Size units

In the query parameters above, size can be specified in number of bytes as integer [ [ whitespace ] unit ] where the multiplier unit can be one of (case insensitive):

Unit Factor
KB 10001
K, KiB 210
MB 10002
M, MiB 220
GB 10003
G, GiB 230
TB 10004
T, TiB 240
PB 10005
P, PiB 250
EB 10006
E, EiB 260
ZB 10007
Z, ZiB 270
YB 10008
Y, YiB 280

It should be noted that currently, Vidispine has a 64-bit limit on the size of a storage, which means that multipliers larger than ZB are of limited use.