S3 Object Tagging

S3 storages in VidiCore can be configured to synchronize and represent tags on S3 objects as metadata on the corresponding File entities.

VidiCore will automatically add a prefix (s3_) to keys of metadata entries that are read from S3. If the S3 object has a tag like Project=Alpha, the corresponding metadata entry on the File entity will be s3_Project=Alpha.

Users can add, modify or delete tags by updating the File entity metadata, see Key-value metadata. Note that only metadata entries with keys that has the prefix s3_ will be synchronized to S3.

Configuration

  1. The user or role being used to access the S3 storage must have permissions to read and update tags on S3 objects. The required actions are s3:GetObjectTagging and s3:PutObjectTagging. This has to be done in the respective S3 buckets.
  2. S3 object tagging must be enabled on the VidiCore storage. This is done by setting storage metadata s3ObjectTagging to true, see Key-value metadata.
  3. It is recommended that the storage metadata syncFileMetadataOnStart is set to true, see Key-value metadata, otherwise VidiCore will not detect tags that were added, changed or deleted while VidiCore is offline.

Additional notes

Due to the way this functionality is implemented, inconsistencies between tags on S3 objects and metadata on File entities can occur if a value is updated multiple times both by VidiCore and externally (i.e. directly via the S3 API) in quick succession.

One scenario that will cause inconsistencies is:

  1. S3 object has tag Project=Alpha and corresponding File entity in VidiCore has metadata s3_Project=Alpha.
  2. Metadata is updated via VidiCore API to s3_Project=Beta.
  3. Object tag is updated via S3 API to s3_Project=Alpha.

If step 2 and 3 happens in quick succession, VidiCore might not be updated correctly and still have File metadata s3_Project=Beta. VidiCore will not detect the correct tag until the whole system is restarted.

To avoid this issue it is recommended that S3 Event SQS Notifications is used.