Version 5.3

The release notes will tell you what’s new in each version, and any changes that you must be aware of when upgrading. For reference, Vidispine ticket numbers are printed as (#1234).

5.3.6

2021-03-26

Bug fixes

  • “db check” command should fail if the database is empty (#4668).
  • Incorrect facet results for Elasticsearch, if the SearchDocument contains a “filter” (#4666).

Transcoder fixes

  • Audio/video(frozen video) issues after a conform job for .mxf files (#4735).
  • Audio/video(unsynced audio) issues after a conform job for .mov (#4733).
  • Transcode job no failing after a file reading error (#4707).
  • Incorrect samples count and duration for aac (#4681).
  • Transcoder license endpoint race condition (#4618).

5.3.5

2021-01-22

Bug fixes

  • Effective lock change notification is not triggered when item is added into a collection (#4436).
  • Prevent multiple services from starting if cluster is in a “split-brain” status (#4653).
  • Removed metadata field shows up again after trimming (#4646).

Transcoder fixes

  • Slow shape deduction of large MXFs due to large block size (#4675).
  • Transcoder does not allocate large enough moov header (#4674).
  • New mutex is created instead of locking existing mutex (#4679).

Agent fixes

  • Failed to rename files in a VSA share whose URI contains query parameters (#4609).

5.3.4

2020-12-11

Bug fixes

  • Item metadata cannot be returned as expected if queries contain group (#4637).
  • Restoring large file from Glacier vault results in IOException (#4638).
  • Audit trails body field is truncated after few hundred chars (between 700-1100) (#4636).
  • Image import fails due to ArrayIndexOutOfBoundsException (#4605).

Transcoder fixes

  • Output frame calculation does not respect FPS change (#4593).
  • Sample format AV_SAMPLE_FMT_S32 not supported error when encoding AAC (#4587).

5.3.3

2020-11-05

Bug fixes

  • Time out error in “dbstats” selftest (#4186).
  • JobCruncherWorker flooding logs if ActiveMQ is not ready (#4020).
  • Item order on collection search not same as order in collection (#4611).
  • Incorrect indexing after metadata field creation/update (#4612).
  • Incorrect link to support portal on Welcome page (#4610).
  • Swedish characters not displayed correctly when transferred via FTP (#4477).
  • JavaScript objects have too strict argument type checks with GraalJS (#4214).
  • Missing field default values while accessing collection metadata (#4461).
  • Filter on groups does not work for the /search endpoint (#4562).
  • Deleted metadata field still appears in metadata (#4411).
  • Item not re-indexed after being associated with a file (#4449).
  • NPE when restoring file from Glacier to S3 (#4187).

Transcoder fixes

  • H264_FIELDBASED InterlaceMode setting not applied for Nablet (#4632).

5.3.2

2020-10-21

Improvements

  • Support TLS protocol and certificate setting for LDAP connections (#4551).
  • Include fields with default values in the search index (#4464).
  • Storage rules should allow for new storage actions while processing the current batch (#4533).
  • Expose storage class for files on S3 Glacier Deep Archive (#4201).

Bug fixes

  • FTP control connection not kept alive while file transfer is in progress (#4388).
  • Preset with both mix and stream elements do not render properly (#4606).
  • Slow collection update of large collections due to synchronous indexing (#4520).
  • Unnecessary reindexing of files on VSA if scan fails and resumes (#4567).
  • Possible ACL reindex failure after transaction failure (#4563).
  • Metadata dataset validation fails for values containing slashes (#4522).
  • Glacier archival fails due to timeout exception (#4594).
  • Glacier vault storage doesn’t handle the retrievalTier from storage metadata (#4552).
  • NullPointerException in GlacierBean (#4542).
  • Collection not reindexed after being renamed (#4525).
  • Collection metadata not validated for POST API/collection (#4493).
  • Incorrect transient metadata for item if file copy job fails (#4174).
  • Incorrect deletion lock metadata for copied file (#3825).
  • User can assign himself to a group without _group_write permission (#4546).
  • Imports failing due to Infinispan lock errors for XMP fields (#4487).
  • Bulk collection delete fails if collection has ancestor relationship (#4405).
  • Direct S3 to S3 transfers not used when useSegmentFiles is true (#4383).
  • Metadata modified notification triggered on empty changes (#4465).
  • Renamed file not re-detected until after 5 hours (#4579).
  • No write from transcoder in 1 hour causes mutable range write to fail (#4575).
  • Error retrieving thumbnails from VSA (#4165).

Transcoder fixes

  • Audio segments does not start at zero PTS (causing blanks in video) (#4607).
  • Audio is broken for mov input files (#4536).
  • Transcode of MP3 files fail with a license error (#4527).
  • Distorted audio for Nablet AAC if sample format is planar (#4526).
  • Transcoder does not allow multiple segments that are equal in timeline/CONFORM job (#4446).
  • Missing timeCodeTimeBase element on container component (#4159).

5.3.1

2020-08-30

New features

  • Support returning additional user/group information in ACL (#4505).
  • Allow core metadata fields to be modified (#4482).

Improvements

  • Send file delete notification if a lowres file is removed due to transcoding failure (#4499).
  • Be able to configure the global spring async pool (#4495).

Bug fixes

  • Change the owner of an entity will generate a duplicate “indexLog” entry (#4450).

    Note: db migrate is needed to remove the already existing duplicate entries in t_indexlog. The migration is considered as optional since this issue doesn’t cause any problem functionally. It only affects the performance of ACL reindex. You can choose when to run db migrate, and the duplicate entries won’t be removed until then.

  • Prevent possible JGroups “split-brain” issue during VidiCore startup (#4518).

  • Mimetype missing on new shape versions (#4514).

  • The output file of a “m4a” shape tag should have “m4a” as file extension (#4504).

  • Reindex/index progress stuck on large collections (#4498).

  • VidiCore fails to update VidiCoder License in some cases (#4494).

  • Incorrect search hit on the /search endpoint in some cases (#4488).

  • Possible OOM when updating metadata referenced in many places (#4484).

  • Glacier file with special character in name stops storage scan (#4472).

  • Export between VSA shares takes long time and results in incorrect file size (#4460).

  • Too many parameter error in SQL query from DB cleanup (#4438).

  • Placeholder/Raw import jobs missing the configured CC extraction (#4421).

  • Incorrect timecode if fetching metadata containing “inherited timespan” using the “interval” query parameter (#4368).

  • Export with both tag and interval ignores the tag (#4156).

  • Removed child collection is still available from parent collection search (#4404)

  • Glacier Vault Archive restore job created by storage rule is aborted due to timeout (#4523)

Transcoder fixes

  • Creating video in 60 fps produces invalid length (#4481).
  • Incorrect system metadata fields on item level for NTSC drop-frame videos (#4480).
  • Fix invalid audio in MXF created by AVID (#4418).

5.3

2020-07-15

New names

VidiCore

Vidispine Server, Vidispine API, VaaS are now all called VidiCore. The packages and names in documentation will change during the second half of 2020. Consequently, VSA and VMA now stands for VidiCore Server Agent and VidiCore Management Agent.

VidiCoder

The transcoder’s new name is VidiCoder. Packages and documentation will change to reflect the new name during 2020.

Breaking changes

  • The priority attribute in the MergedAccessDocument has been changed to rank (#4430). Check doc for more information.
  • If you’re using external identifiers, database migration may require manual intervention. Check the “Upgrading from 5.2” section below for details.

Features and improvements

New file analyze endpoints

Shape deduction can now be performed on non-imported files or IMF packages in VidiCore (#4039).

POST /storage/{storage-id}/file/{file-id}/analyze
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-10525</jobId>
  <user>admin</user>
  <started>2020-07-14T09:04:51.768Z</started>
  <status>READY</status>
  <type>FILE_ANALYZE</type>
  <priority>MEDIUM</priority>
</JobDocument>

The resulting shape can be found in the job metadata and retrieved using the new shape endpoint on a file.

Bulky metadata storage

By default, the values of bulky metadata are stored in the database. In a large system, this can occupy a large portion of the database. Now, it is possible to store bulky metadata on the file system (or cloud storage) (#4440). See bulky metadata storage.

Configurable site prefix, identifier format, and logging level via the API

Traditionally, the VidiCore site prefix and identifier format can be configured using JVM properties. And the log levels are configured in the server.yaml file.

In 5.3, they can be configured using the API as well (#4417). See doc. This is especial useful to VidiCore users on VidiNet.

Entity ownership transfer

The ownership of an entity can be transferred to a different user when the original owner is being deleted. Check the usage of the transferAccess parameter (#4327).

VSA Thread configuration

It is now possible to configure the number of worker threads and selector runner threads in agent properties (#4226). This could improve VSA performance when under high load.

VidiCoder (Transcoder) library update

The transcoder uses ffmpeg for some codecs. The ffmpeg libraries are bundled with the transcoder. In version 5.3, a major upgrade of the ffmpeg library is done. This means some ffmpeg-specific parameters may have be renamed. If you are using these parameters, let Vidispine now. We have also taken the opportunity to clean up some of the edge-case handling of slightly incorrect media files in VidiCoder, as the new ffmpeg library is better at handling these. Hence, you should test that your type of media works with the new VidiCoder version. However, in the meantime, rest assured that you can still use VidiCoder 5.2 with VidiCore 5.3, they are compatible.

Other improvements

  • Add support for Bearer token validation using a public key (#4478).
  • Implement support for import of MPEG-DASH package (#4379).
  • Implement support for export of MPEG-DASH package (#4381).

Bug fixes

  • There can be duplicate external identifiers for the same entity type (#4350).
  • Sidecar file cannot be imported for the second time (#4235).
  • Glacier Vault storage can only get credentials from AwsCredentials.properties (#4219).

Upgrading from 5.2

External identifier migration

External identifiers in VidiCore should be unique among the same entity type. But if there are multiple “overlapping” external identifier namespaces defined, duplicate identifiers could exist.

This has been fixed in 5.3 (#4350). A stronger constraint has been added to the table to prevent duplicates. However, if there are already duplicate entries in the table, manual intervention is required.

Below are the steps:

  1. Use this SQL statement to identify duplicated entries:
SELECT i2.c_entity_type, i2.c_entity_id, i2.c_identifier_value, i2.c_identifier_name
FROM
(SELECT c_entity_type, c_identifier_value FROM t_externalidentry
    GROUP BY c_entity_type, c_identifier_value HAVING COUNT(*) > 1) i1
INNER JOIN
(SELECT * from t_externalidentry ) i2
ON i2.c_entity_type = i1.c_entity_type AND i2.c_identifier_value = i1.c_identifier_value
ORDER BY c_identifier_value;
  1. If the above query doesn’t return anything, meaning no duplicates, you can skip the rest and processed with automatic database migration.

  2. If there are duplicate entries, you need to decide which one(s) to remove.

    Below is a sample data produced by the SQL query. The result should help you determine which entities have duplicate external-ids, and their external-id values and namespaces.

    And the external identifier can be removed using: DELETE {entity-type}/{entity-id}/{identifier-value}.

     c_entity_type     | c_entity_id | c_identifier_value | c_identifier_name
-----------------------+-------------+--------------------+-------------------
 com.vidispine.db.Item | VX-11       | my_test_id         | namespace_1
 com.vidispine.db.Item | VX-2704     | my_test_id         | namespace_2
(2 rows)

Automatic database migration can be performed after all duplicates have been removed.