Storages¶
Managing storages¶
Retrieve list of storages¶
-
GET
/storage
¶ Retrieves the storages that have been configured.
Query Parameters: - size – Optional range of bytes, in format s1
-
s2. Returns storages with nominal size that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is-
, all storages. - freebytes – Optional range of bytes, in format s1
-
s2. Returns storages with free space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is-
, all storages. - usedbytes – Optional range of bytes, in format s1
-
s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is-
, all storages. - freeamount – Optional range of percent as integers, in format s1
-
s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Default is-
, all storages. - files – Optional range of files as integers, in format s1
-
s2. Returns storages with number of files that is in that range. Either number can be omitted to not specify lower/upper limit. Default is-
, all storages. - storagegroup –
Optional list of storage groups.
- storage-group - Returned storage is member of specified storage group.
-
storage-group Returned storage is not member of specified storage group.- default Return all storages.
- status –
Optional list of storage status.
- status - Returned storage has this status.
-
status - Returned storage does not have this status.- default - Return all storages.
- url – Returns storages with a method matching this URL. May include
wildcards
*
and?
.
Produces: - application/xml, application/json – StorageListDocument
- text/plain – CRLF-delimited list of storage ids
Role: _storage_read
- size – Optional range of bytes, in format s1
Example¶
GET /storage
<StorageListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<storage>
<id>VX-1</id>
<state>NONE</state>
<type>LOCAL</type>
<capacity>1600000000000</capacity>
<freeCapacity>19993896424</freeCapacity>
<method>
<id>VX-6</id>
<uri>file:///mnt/main/</uri>
<bandwidth>0</bandwidth>
<read>true</read>
<write>true</write>
<browse>true</browse>
<lastSuccess>2014-07-04T08:39:00.739+02:00</lastSuccess>
<type>NONE</type>
</method>
<metadata/>
<lowWatermark>0</lowWatermark>
<highWatermark>0</highWatermark>
<showImportables>true</showImportables>
</storage>
<storage>
<id>VX-2</id>
<state>NONE</state>
<type>LOCAL</type>
<capacity>15000000000</capacity>
<freeCapacity>19548305962</freeCapacity>
<method>
<id>VX-4</id>
<uri>file:///mnt/ingest/</uri>
<read>true</read>
<write>true</write>
<browse>true</browse>
<lastSuccess>2014-07-04T08:38:56.773+02:00</lastSuccess>
<type>NONE</type>
</method>
<metadata/>
<lowWatermark>0</lowWatermark>
<highWatermark>200000000000</highWatermark>
<showImportables>true</showImportables>
</storage>
</StorageListDocument>
Create storage¶
-
POST
/storage
¶ Creates a new storage.
Accepts: - application/xml, application/json – StorageDocument
Produces: - application/xml, application/json – StorageDocument
- text/plain – Storage id
Role: _storage_write
Note
If your storage is to be used for autoimport, the showImportables
property in the StorageDocument must be set to true
.
The files on the storage will then be discovered and available for use
with GET /storage/importable
or GET /storage/auto-import/
.
Example¶
POST /storage
Content-Type: application/xml
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<type>LOCAL</type>
<capacity>150000000000</capacity>
<method>
<uri>file:///mnt/ingest/</uri>
<read>true</read>
<write>true</write>
<browse>true</browse>
</method>
<lowWatermarkPercentage>90</lowWatermarkPercentage>
<highWatermarkPercentage>75</highWatermarkPercentage>
<showImportables>true</showImportables>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<id>VX-2</id>
<state>NONE</state>
<type>LOCAL</type>
<capacity>150000000000</capacity>
<freeCapacity>150000000000</freeCapacity>
<method>
<id>VX-4</id>
<uri>file:///mnt/ingest/</uri>
<read>true</read>
<write>true</write>
<browse>true</browse>
<type>NONE</type>
</method>
<metadata/>
<lowWatermark>112500000000</lowWatermark>
<highWatermark>135000000000</highWatermark>
<lowWatermarkPercentage>75</lowWatermarkPercentage>
<highWatermarkPercentage>90</highWatermarkPercentage>
<showImportables>true</showImportables>
</StorageDocument>
Get storage¶
-
GET
/storage/
(storage-id)¶ Retrieves a specific storage.
Produces: - application/xml, application/json – StorageDocument
Role: _storage_read
Example¶
GET /storage/VX-2
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<id>VX-2</id>
<state>NONE</state>
<type>LOCAL</type>
<capacity>150000000000</capacity>
<freeCapacity>150000000000</freeCapacity>
<method>
<id>VX-4</id>
<uri>file:///mnt/ingest/</uri>
<read>true</read>
<write>true</write>
<browse>true</browse>
<type>NONE</type>
</method>
<metadata/>
<lowWatermark>112500000000</lowWatermark>
<highWatermark>135000000000</highWatermark>
<lowWatermarkPercentage>75</lowWatermarkPercentage>
<highWatermarkPercentage>90</highWatermarkPercentage>
<showImportables>true</showImportables>
</StorageDocument>
Modify storage¶
-
PUT
/storage/
(storage-id)¶ Updates an existing storage.
Accepts: - application/xml, application/json – StorageDocument
Produces: - application/xml, application/json – StorageDocument
- text/plain – Storage id
Example¶
PUT /storage/VX-2
Content-Type: application/xml
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<type>LOCAL</type>
<capacity>150000000000</capacity>
<lowWatermarkPercentage>80</lowWatermarkPercentage>
<highWatermarkPercentage>95</highWatermarkPercentage>
</StorageDocument>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<id>VX-2</id>
<state>NONE</state>
<type>LOCAL</type>
<capacity>150000000000</capacity>
<freeCapacity>150000000000</freeCapacity>
<method>
<id>VX-4</id>
<uri>file:///mnt/ingest/</uri>
<read>true</read>
<write>true</write>
<browse>true</browse>
<type>NONE</type>
</method>
<metadata/>
<lowWatermark>120000000000</lowWatermark>
<highWatermark>142500000000</highWatermark>
<lowWatermarkPercentage>80</lowWatermarkPercentage>
<highWatermarkPercentage>95</highWatermarkPercentage>
<showImportables>true</showImportables>
</StorageDocument>
Delete storage¶
-
DELETE
/storage/
(storage-id)¶ Deletes the storage. All files in storage will remain after call, but the Vidispine system will no longer manage them.
Query Parameters: - safe –
true
- Storage will only be deleted if there are no files connected to items on the storage.false
(default) - Storage will be deleted, and any file entities will be disconnected from items and deleted.
Status Codes: - 409 Conflict – If
safe
parameter istrue
this error code will be given if there are files connected to items on the storage.
Role: _storage_write
- safe –
Storage information¶
Set the storage type¶
-
PUT
/storage/
(storage-id)/type/
(type)¶ Sets the type of the storage.
Role: _storage_write
Get storage status¶
-
GET
/storage/
(storage-id)/status
¶ Retrieves the status of the storage.
Produces: - text/plain – Status string
Role: _storage_read
Get amount of free space on storage¶
-
GET
/storage/
(storage-id)/freespace
¶ Retrieves the amount of free space on the storage.
Produces: - text/plain – Amount of free space as decimal number, between 0 and 100, inclusive
Role: _storage_read
Rescanning¶
The scanInterval
property can be used to control how often (in seconds)
a storage is scanned. Default is 60 seconds. By calling /rescan
,
the system is forced to rescan a storage without delay.
Rescan a storage¶
-
POST
/storage/
(storage-id)/rescan
¶ Triggers a rescan of a single storage.
Storage methods¶
New in version 4.1: Credentials are encrypted. This means that passwords cannot be viewed through the API/server logs.
Get storage methods¶
-
GET
/storage/
(storage-id)/method
¶ Retrieves the access methods configured on a specific storage.
Query Parameters: - url – Optional URL. Only return methods with this URL.
Wildcards (
?
,*
) can be used, for examplehttp:*
.
Matrix Parameters: - read –
true
- Only return methods which have file read capability.false
(default) - Return methods regardless of file read capability.
- write –
true
- Only return methods which have file write capability.false
(default) - Return methods regardless of file write capability.
- browse –
true
- Only return methods which have file browse capability.false
(default) - Return methods regardless of file browse capability.
Produces: - application/xml, application/json – StorageMethodListDocument
Role: _storage_read
- url – Optional URL. Only return methods with this URL.
Wildcards (
Example¶
GET /storage/VX-2/method
<StorageMethodListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
<method>
<id>VX-4</id>
<uri>file:///mnt/ingest/</uri>
<bandwidth>0</bandwidth>
<read>true</read>
<write>true</write>
<browse>true</browse>
<lastSuccess>2014-07-04T09:55:09.779+02:00</lastSuccess>
<type>NONE</type>
</method>
</StorageMethodListDocument>
Add new/update storage method¶
-
PUT
/storage/
(storage-id)/method
¶ Adds a new access method to the storage. If URL matches an existing method, a new method is not created, instead the existing one is updated.
Query Parameters: - url – Mandatory URL, method is access through this URL
- read –
true
(default) - Method has file read capabilityfalse
- Method does not have file read capability
- write –
true
(default) - Method has file write capabilityfalse
- Method does not have file write capability
- browse –
true
(default) - Method has file browse capabilityfalse
- Method does not have file browse capability
- bandwidth – The bandwidth of this
method in bytes per second. Default
0
. - type – Optional method type. Default is empty.
Produces: - text/plain – Tabbed tuples : id, URL, status string of method
Role: _storage_write
Example¶
PUT /storage/VX-2/method?url=http://10.12.0.3/ingest/&read=true&write=false&browse=false
VX-5 http://10.12.0.3/ingest/ NONE
Get storage method¶
-
GET
/storage/
(storage-id)/method/
(method-id)¶ Retrieves a specific access method to storage.
Produces: - application/xml, application/json – StorageMethodListDocument
Role: _storage_read
Add new/update storage method¶
-
PUT
/storage/
(storage-id)/method/
(method-id)¶ Updates access method to the storage.
Query Parameters: - url – Mandatory URL, method is access through this URL
- read –
true
- Method has file read capability.false
- Method does not have file read capability.
- write –
true
- Method has file write capability.false
- Method does not have file write capability.
- browse –
true
- Method has file browse capability.false
- Method does not have file browse capability.
- bandwidth – The bandwidth of this
method in bytes per second. Default
0
. - type – Optional method type.
Produces: - text/plain – Tabbed tuples : id, URL, status string of method
Role: _storage_write
Importing/exporting a storage definition¶
Vidispine can export a definition document describing all the files within a storage, which can later be imported to a new storage on a different Vidispine instance.
Export a storage definition¶
-
GET
/storage/
(storage-id)/export
¶ Creates a StorageImportDocument that describes every file on the storage. This should be saved to a file which can later be used to import the storage definition.
Produces: - application/xml, application/json – StorageImportDocument
Role: _storage_read
Import a storage definition¶
-
POST
/storage/import
¶ Creates a new storage based on the StorageImportDocument. A file entity will be created for each entry in the document, if a file with that ID does not already exist. Finally, a storage method will be added, with the path supplied in the call.
Query Parameters: - path – The file system path to where the files are located.
Accepts: - application/xml, application/json – A StorageImportDocument
Role: _storage_write
Evacuating storages¶
If you would like to delete a storage, but you still have files there which are
connected to items, you can first trigger an evacuation of the storage. This
will cause Vidispine to attempt to delete redundant files, or move files to
other storages. Once the evacuation is complete, the storage will get the state
EVACUATED
.
Evacuate storage¶
-
PUT
/storage/
(storage-id)/evacuate
¶ Trigger evacuation of a storage.
Role: _storage_write
Cancel evacuation of a storage¶
-
DELETE
/storage/
(storage-id)/evacuate
¶ Cancel the evacuation process on a storage.
Role: _storage_write
Key-value metadata¶
Storages support key-value metadata.
Reserved keys¶
Certain keys are used to control the behavior of a storage, they must not be used to store generic metadata.
-
closeLimit
¶ An integer specifying the number of minutes until the system automatically considers an open file to be closed.
Default: 5
-
lostLimit
¶ An integer specifying the number of minutes until the system automatically considers a missing file to be lost.
Default: 10
-
toAppearLimit
¶ An integer specifying the number of minutes the system waits for a file to appear before considering it to be missing.
Default: 10
-
refreshInterval
¶ The interval (in seconds) that Vidispine will compare the list of file seen on disk with the list of files in the database. This to fix any possible mismatch.
Default: 900
-
hashMode
¶ Set to
false
to disable hash computation for this storage.Default: true
-
additionalHash
¶ A comma-separated list of additional hashing algorithms that should be applied to files. E.g.:
MD5,SHA-256
.Since 4.14, the hash algorithm
FILENAME
is also supported, which takes the last part of the path concatenated with the file size, and calculates a SHA-1 checksum of that string.Default: (none)
-
fileListBatchSize
¶ Maximum size of list of files that are updated in the database at the same time.
Default: 100000 Since: 4.0.3
-
keepMissingFiles
¶ If set to
false
then missing files that do not belong to any items will be removed from the database instead of being marked as lost.Default: To the value of the configuration property keepMissingFiles
.Since: 4.1
-
keepEmptyDirectories
¶ Do not delete empty parent directories when deleting the last file in a directory, see Parent directory management.
Default: To the value of the configuration property keepEmptyDirectories
.Since: 4.2.5
-
statsPerSecond
¶ An integer specifying the maximum number of
stat
system calls that are made to the storage (onlyfile://
URIs). See also the configuration propertystatsPerSecond
.Default: (none) Since: 4.1.1
-
refreshOnStart
¶ By default, Vidispine synchronizes the database with the file system on start-up. On very large systems, this can take a very long time. Set this parameter to
false
, and use the rescan functionality (see below) instead.Default: true Since: 4.1.1
-
deleteFileIfNotFound
¶ If set to
true
: if file that is marked for deletion cannot be deleted, but is not found on the file system, it is assumed to be deleted.Default: false Since: 4.0.9
-
deleteFileIfReadOnly
¶ If a file that is marked for deletion cannot be deleted, and the file exists on a readonly storage, then:
- If set to
true
: delete the file from the database, as if the file had been deleted. - If set to a file state: set the file to this state.
Default: false Example: true
,ARCHIVED
,LOST
Since: 4.0.9 - If set to
-
excludeFilter
¶ A regular expression. Any file with a path (relative to the storage’s root folder) that matches the expression will be ignored. Note that the expression must match the entire path, not only a part of the path.
Default: (none) Since: 4.2
-
detectRenamedFiles
¶ If this metadata is set to
true
, whenever a file belonging to an item is marked asLOST
, and there is another file on the same storage with the same hash value, the new file is associated with the item instead.Other possible values are
all
, where the old (lost) file resided on any storage, or a comma-separated list of identifiers of storages on which the old (lost) file resided on.Default: (none) Since: 4.2
-
probeFileBeforeClosing
¶ If this metadata is set to
true
, Vidispine will attempt to readOPEN
files before marking them asCLOSED
. This is useful for Windows storages where the metadata is constant while a file is copied to the storage, but the file cannot be read (file is busy).Default: false Since: 4.2.1
-
vxaId
¶ If this metadata field is set, the storage will be automatically deleted if the share is removed from the Vidispine Server Agent.
Default: (none) for regular storages, automatically set by VSA when a share is added and a storage is created. Since: 4.4
-
sqsName
¶ The name of Amazon SQS queue to poll for S3 file events. See S3 Event Notifications for more information.
This key is only read for S3 methods.
-
sqsEndpoint
¶ The endpoint of Amazon SQS queue.
This key is only read for S3 methods.
Example:
sqs.eu-west-1.amazonaws.com
-
hashingThreadCount
¶ The number of hashing thread that should be run on the VSA side. Note: this configuration only work for VSA storage.
Default: 1 Since: 4.11
-
noDefaultHash
¶ Do not calculate hash checksums for files on the storage.
Example:
true
Default: false
Since: 4.14
-
verifyHashAfterTransfer
¶ If a job is copying or moving a file to this storage the hash of the source file and the hash of the newly copied/moved file are compared to ensure that the new file is binary identical to the original file. If the hashes differ the job fails and the destination file is removed.
- If set to
true
: hash verification is performed after the file has arrived on this storage. - If set to
false
: no hash verification is performed.
The hash verification procedure makes use of the hash calculation that is happening on all storages by default (see File hashing). If hash calculation is disabled on the source storage the hash is calculated before the copy/move operation is performed. If hash calculation is disabled on the destination storage the hash is calculated after the copy/move operation has completed. These hashes are stored in the Vidispine database for later reuse.
Note that hash calculation requires reading the whole file and thus is a lengthy process that can significantly increase execution time of copy/move jobs.
Default: false Since: 4.14 - If set to
Size units¶
In the query parameters above, size can be specified in number of bytes as
integer [ [ whitespace ] unit ]
where the multiplier unit can be one of
(case insensitive):
Unit | Factor |
---|---|
KB |
10001 |
K , KiB |
210 |
MB |
10002 |
M , MiB |
220 |
GB |
10003 |
G , GiB |
230 |
TB |
10004 |
T , TiB |
240 |
PB |
10005 |
P , PiB |
250 |
EB |
10006 |
E , EiB |
260 |
ZB |
10007 |
Z , ZiB |
270 |
YB |
10008 |
Y , YiB |
280 |
It should be noted that currently, Vidispine has a 64-bit limit on the size of
a storage, which means that multipliers larger than ZB
are of limited use.