Storages

Managing storages

Retrieve list of storages

GET /storage

Retrieves the storages that have been configured.

Query Parameters:
 
  • size – Optional range of bytes, in format s1-s2. Returns storages with nominal size that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages.
  • freebytes – Optional range of bytes, in format s1-s2. Returns storages with free space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages.
  • usedbytes – Optional range of bytes, in format s1-s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages.
  • freeamount – Optional range of percent as integers, in format s1-s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Default is -, all storages.
  • files – Optional range of files as integers, in format s1-s2. Returns storages with number of files that is in that range. Either number can be omitted to not specify lower/upper limit. Default is -, all storages.
  • storagegroup

    Optional list of storage groups.

    • storage-group - Returned storage is member of specified storage group.
    • -storage-group Returned storage is not member of specified storage group.
    • default Return all storages.
  • status

    Optional list of storage status.

    • status - Returned storage has this status.
    • -status - Returned storage does not have this status.
    • default - Return all storages.
  • url – Returns storages with a method matching this URL. May include wildcards * and ?.
Produces:
  • application/xml, application/jsonStorageListDocument
  • text/plain – CRLF-delimited list of storage ids
Role:

_storage_read

Example

GET /storage
<StorageListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <storage>
    <id>VX-1</id>
    <state>NONE</state>
    <type>LOCAL</type>
    <capacity>1600000000000</capacity>
    <freeCapacity>19993896424</freeCapacity>
    <method>
      <id>VX-6</id>
      <uri>file:///mnt/main/</uri>
      <bandwidth>0</bandwidth>
      <read>true</read>
      <write>true</write>
      <browse>true</browse>
      <lastSuccess>2014-07-04T08:39:00.739+02:00</lastSuccess>
      <type>NONE</type>
    </method>
    <metadata/>
    <lowWatermark>0</lowWatermark>
    <highWatermark>0</highWatermark>
    <showImportables>true</showImportables>
  </storage>
  <storage>
    <id>VX-2</id>
    <state>NONE</state>
    <type>LOCAL</type>
    <capacity>15000000000</capacity>
    <freeCapacity>19548305962</freeCapacity>
    <method>
      <id>VX-4</id>
      <uri>file:///mnt/ingest/</uri>
      <read>true</read>
      <write>true</write>
      <browse>true</browse>
      <lastSuccess>2014-07-04T08:38:56.773+02:00</lastSuccess>
      <type>NONE</type>
    </method>
    <metadata/>
    <lowWatermark>0</lowWatermark>
    <highWatermark>200000000000</highWatermark>
    <showImportables>true</showImportables>
  </storage>
</StorageListDocument>

Create storage

POST /storage

Creates a new storage.

Accepts:
Produces:
  • application/xml, application/jsonStorageDocument
  • text/plain – Storage id
Role:

_storage_write

Note

If your storage is to be used for autoimport, the showImportables property in the StorageDocument must be set to true. The files on the storage will then be discovered and available for use with GET /storage/importable or GET /storage/auto-import/.

Example

POST /storage
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <method>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
  </method>
  <lowWatermarkPercent>90</lowWatermarkPercent>
  <highWatermarkPercent>75</highWatermarkPercent>
  <showImportables>true</showImportables>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>112500000000</lowWatermark>
  <highWatermark>135000000000</highWatermark>
  <lowWatermarkPercentage>75</lowWatermarkPercentage>
  <highWatermarkPercentage>90</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Get storage

GET /storage/(storage-id)

Retrieves a specific storage.

Produces:
Role:

_storage_read

Example

GET /storage/VX-2
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>112500000000</lowWatermark>
  <highWatermark>135000000000</highWatermark>
  <lowWatermarkPercentage>75</lowWatermarkPercentage>
  <highWatermarkPercentage>90</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Modify storage

PUT /storage/(storage-id)

Updates an existing storage.

Accepts:
Produces:
  • application/xml, application/jsonStorageDocument
  • text/plain – Storage id

Example

PUT /storage/VX-2
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <lowWatermarkPercentage>80</lowWatermarkPercentage>
  <highWatermarkPercentage>95</highWatermarkPercentage>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>120000000000</lowWatermark>
  <highWatermark>142500000000</highWatermark>
  <lowWatermarkPercentage>80</lowWatermarkPercentage>
  <highWatermarkPercentage>95</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Delete storage

DELETE /storage/(storage-id)

Deletes the storage. All files in storage will remain after call, but the Vidispine system will no longer manage them.

Query Parameters:
 
  • safe
    • true - Storage will only be deleted if there are no files connected to items on the storage.
    • false (default) - Storage will be deleted, and any file entities will be disconnected from items and deleted.
Status Codes:
  • 409 Conflict – If safe parameter is true this error code will be given if there are files connected to items on the storage.
Role:

_storage_write

Storage information

Set the storage type

PUT /storage/(storage-id)/type/(type)

Sets the type of the storage.

Role:_storage_write

Get storage status

GET /storage/(storage-id)/status

Retrieves the status of the storage.

Produces:
  • text/plain – Status string
Role:

_storage_read

Get amount of free space on storage

GET /storage/(storage-id)/freespace

Retrieves the amount of free space on the storage.

Produces:
  • text/plain – Amount of free space as decimal number, between 0 and 100, inclusive
Role:

_storage_read

Rescanning

The scanInterval property can be used to control how often (in seconds) a storage is scanned. Default is 60 seconds. By calling /rescan, the system is forced to rescan a storage without delay.

Rescan a storage

POST /storage/(storage-id)/rescan

Triggers a rescan of a single storage.

Storage methods

New in version 4.1: Credentials are encrypted. This means that passwords cannot be viewed through the API/server logs.

Get storage methods

GET /storage/(storage-id)/method

Retrieves the access methods configured on a specific storage.

Query Parameters:
 
  • url – Optional URL. Only return methods with this URL. Wildcards (?, *) can be used, for example http:*.
Matrix Parameters:
 
  • read
    • true - Only return methods which have file read capability.
    • false (default) - Return methods regardless of file read capability.
  • write
    • true - Only return methods which have file write capability.
    • false (default) - Return methods regardless of file write capability.
  • browse
    • true - Only return methods which have file browse capability.
    • false (default) - Return methods regardless of file browse capability.
Produces:
Role:

_storage_read

Example

GET /storage/VX-2/method
<StorageMethodListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <bandwidth>0</bandwidth>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <lastSuccess>2014-07-04T09:55:09.779+02:00</lastSuccess>
    <type>NONE</type>
  </method>
</StorageMethodListDocument>

Add new/update storage method

PUT /storage/(storage-id)/method

Adds a new access method to the storage. If URL matches an existing method, a new method is not created, instead the existing one is updated.

Query Parameters:
 
  • url – Mandatory URL, method is access through this URL
  • read
    • true (default) - Method has file read capability
    • false - Method does not have file read capability
  • write
    • true (default) - Method has file write capability
    • false - Method does not have file write capability
  • browse
    • true (default) - Method has file browse capability
    • false - Method does not have file browse capability
  • bandwidth – The bandwidth of this method in bytes per second. Default 0.
  • type – Optional method type. Default is empty.
Produces:
  • text/plainTabbed tuples : id, URL, status string of method
Role:

_storage_write

Example

PUT /storage/VX-2/method?url=http://10.12.0.3/ingest/&read=true&write=false&browse=false
VX-5        http://10.12.0.3/ingest/        NONE

Get storage method

GET /storage/(storage-id)/method/(method-id)

Retrieves a specific access method to storage.

Produces:
Role:

_storage_read

Add new/update storage method

PUT /storage/(storage-id)/method/(method-id)

Updates access method to the storage.

Query Parameters:
 
  • url – Mandatory URL, method is access through this URL
  • read
    • true - Method has file read capability.
    • false - Method does not have file read capability.
  • write
    • true - Method has file write capability.
    • false - Method does not have file write capability.
  • browse
    • true - Method has file browse capability.
    • false - Method does not have file browse capability.
  • bandwidth – The bandwidth of this method in bytes per second. Default 0.
  • type – Optional method type.
Produces:
  • text/plainTabbed tuples : id, URL, status string of method
Role:

_storage_write

Remove storage method

DELETE /storage/(storage-id)/method/(method-id)
DELETE /storage/(storage-id)/method;url=(url)

Removes an access method from a storage.

Role:_storage_write

Importing/exporting a storage definition

Vidispine can export a definition document describing all the files within a storage, which can later be imported to a new storage on a different Vidispine instance.

Export a storage definition

GET /storage/(storage-id)/export

Creates a StorageImportDocument that describes every file on the storage. This should be saved to a file which can later be used to import the storage definition.

Produces:
Role:

_storage_read

Import a storage definition

POST /storage/import

Creates a new storage based on the StorageImportDocument. A file entity will be created for each entry in the document, if a file with that ID does not already exist. Finally, a storage method will be added, with the path supplied in the call.

Query Parameters:
 
  • path – The file system path to where the files are located.
Accepts:
Role:

_storage_write

Evacuating storages

If you would like to delete a storage, but you still have files there which are connected to items, you can first trigger an evacuation of the storage. This will cause Vidispine to attempt to delete redundant files, or move files to other storages. Once the evacuation is complete, the storage will get the state EVACUATED.

Evacuate storage

PUT /storage/(storage-id)/evacuate

Trigger evacuation of a storage.

Role:_storage_write

Cancel evacuation of a storage

DELETE /storage/(storage-id)/evacuate

Cancel the evacuation process on a storage.

Role:_storage_write

Key-value metadata

Storages support key-value metadata.

Reserved keys

Certain keys are used to control the behavior of a storage, they must not be used to store generic metadata.

closeLimit

An integer specifying the number of minutes until the system automatically considers an open file to be closed.

Default:5
lostLimit

An integer specifying the number of minutes until the system automatically considers a missing file to be lost.

Default:10
toAppearLimit

An integer specifying the number of minutes the system waits for a file to appear before considering it to be missing.

Default:10
refreshInterval

The interval (in seconds) that Vidispine will compare the list of file seen on disk with the list of files in the database. This to fix any possible mismatch.

Default:900
hashMode

Set to false to disable hash computation for this storage.

Default:true
additionalHash

A comma-separated list of additional hashing algorithms that should be applied to files. E.g.: MD5,SHA-256

Default:(none)
fileListBatchSize

Maximum size of list of files that are updated in the database at the same time.

Default:100000
Since:4.0.3
keepMissingFiles

If set to false then missing files that do not belong to any items will be removed from the database instead of being marked as lost.

Default:To the value of the configuration property keepMissingFiles.
Since:4.1
keepEmptyDirectories

Do not delete empty parent directories when deleting the last file in a directory, see Parent directory management.

Default:To the value of the configuration property keepEmptyDirectories.
Since:4.2.5
statsPerSecond

An integer specifying the maximum number of stat system calls that are made to the storage (only file:// URIs). See also the configuration property statsPerSecond.

Default:(none)
Since:4.1.1
refreshOnStart

By default, Vidispine synchronizes the database with the file system on start-up. On very large systems, this can take a very long time. Set this parameter to false, and use the rescan functionality (see below) instead.

Default:true
Since:4.1.1
deleteFileIfNotFound

If set to true: if file that is marked for deletion cannot be deleted, but is not found on the file system, it is assumed to be deleted.

Default:false
Since:4.0.9
deleteFileIfReadOnly

If a file that is marked for deletion cannot be deleted, and the file exists on a readonly storage, then:

  • If set to true: delete the file from the database, as if the file had been deleted.
  • If set to a file state: set the file to this state.
Default:false
Example:true, ARCHIVED, LOST
Since:4.0.9
excludeFilter

A regular expression. Any file with a path (relative to the storage’s root folder) that matches the expression will be ignored. Note that the expression must match the entire path, not only a part of the path.

Default:(none)
Since:4.2
detectRenamedFiles

If this metadata is set to true, whenever a file belonging to an item is marked as LOST, and there is another file on the same storage with the same hash value, the new file is associated with the item instead.

Other possible values are all, where the old (lost) file resided on any storage, or a comma-separated list of identifiers of storages on which the old (lost) file resided on.

Default:(none)
Since:4.2
probeFileBeforeClosing

If this metadata is set to true, Vidispine will attempt to read OPEN files before marking them as CLOSED. This is useful for Windows storages where the metadata is constant while a file is copied to the storage, but the file cannot be read (file is busy).

Default:false
Since:4.2.1
vxaId

If this metadata field is set, the storage will be automatically deleted if the share is removed from the Vidispine Server Agent.

Default:(none) for regular storages, automatically set by VSA when a share is added and a storage is created.
Since:4.4
sqsName

The name of Amazon SQS queue to poll for S3 file events. See S3 Event Notifications for more information.

This key is only read for S3 methods.

sqsEndpoint

The endpoint of Amazon SQS queue.

This key is only read for S3 methods.

Example: sqs.eu-west-1.amazonaws.com

hashingThreadCount

The number of hashing thread that should be run on the VSA side. Note: this configuration only work for VSA storage.

Default:1
Since:4.11
verifyHashAfterTransfer

If a job is copying or moving a file to this storage the hash of the source file and the hash of the newly copied/moved file are compared to ensure that the new file is binary identical to the original file. If the hashes differ the job fails and the destination file is removed.

  • If set to true: hash verification is performed after the file has arrived on this storage.
  • If set to false: no hash verification is performed.

The hash verification procedure makes use of the hash calculation that is happening on all storages by default (see File hashing). If hash calculation is disabled on the source storage the hash is calculated before the copy/move operation is performed. If hash calculation is disabled on the destination storage the hash is calculated after the copy/move operation has completed. These hashes are stored in the Vidispine database for later reuse.

Note that hash calculation requires reading the whole file and thus is a lengthy process that can significantly increase execution time of copy/move jobs.

Default:false
Since:4.14

Size units

In the query parameters above, size can be specified in number of bytes as integer [ [ whitespace ] unit ] where the multiplier unit can be one of (case insensitive):

Unit Factor
KB 10001
K, KiB 210
MB 10002
M, MiB 220
GB 10003
G, GiB 230
TB 10004
T, TiB 240
PB 10005
P, PiB 250
EB 10006
E, EiB 260
ZB 10007
Z, ZiB 270
YB 10008
Y, YiB 280

It should be noted that currently, Vidispine has a 64-bit limit on the size of a storage, which means that multipliers larger than ZB are of limited use.