S3 Event SQS Notifications

Vidispine supports polling Amazon SQS for S3 events, to detected changes on the Amazon bucket. This makes it less expensive to detect file changes on S3 buckets.

Configure S3 notifications

  1. Create an Amazon SQS queue. You could create the queue from either the AWS Management Console or using the SQS SDK. Make sure that the S3 bucket is allowed to send messages to the queue, for example, using a policy like:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "sqs:*"
            ],
            "Resource": [
                "arn:aws:sqs:eu-west-1:12121221212:exampleQueue"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": [
                        "arn:aws:s3:*:*:bucketname"
                    ]
                }
            }
        }
    ]
}

Note

  • Use one queue per bucket. Don’t send events from multiple buckets to the same queue, as this is not supported by Vidispine.
  • Vidispine will connect to the queue using the credentials from the S3 method URI, so that user must have access to both the bucket and SQS queue.
  1. Configure the S3 bucket to send ObjectCreated:* and ObjectRemoved:* events to SQS. The notifications can be set up from the S3 bucket properties in the AWS Management Console, or using the S3 SDK.

Configure the storage

  1. To have Vidispine poll a SQS queue instead of scanning a S3 bucket, set the storage method metadata sqsName and sqsEndpoint to enable this feature:

    PUT /storage/VX-1/method/VX-2/metadata/sqsName
    Content-Type: text/plain
    
    s3-event-queue
    
    PUT /storage/VX-1/method/VX-2/metadata/sqsEndpoint
    Content-Type: text/plain
    
    sqs.eu-west-1.amazonaws.com
    
    GET /storage/VX-1
    
    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
      ...
      <method>
        <uri>s3://bucketname/</uri>
        ...
        <metadata>
          <field>
            <key>sqsName</key>
            <value>s3-event-queue</value>
          </field>
          <field>
            <key>sqsEndpoint</key>
            <value>sqs.eu-west-1.amazonaws.com</value>
          </field>
        </metadata>
      </method>
      ...
    </StorageDocument>
    
  2. Then make sure that the storage metadata scanOnStart is true (this is the default).

    Due to the distributed nature of Amazon SQS, the messages come unordered. On every start up, Vidispine will need to purge the queue and do a full scan of the storage, to sync the file list with database.

  3. You can then set the storage to be scanned less often by setting scanInterval to a larger number than default, for example 1 hour.

    Every scanInterval, Vidispine will perform a storage scan, as a precaution of any possible inconsistency.

You can check the storage method status (lastSuccess, lastFailure, failureMessage) to determine if the configuration is correct or not. For example, if a non-existing queue is specified:

<failureMessage>
   Error polling SQS: The specified queue does not exist for this wsdl version. (...)
</failureMessage>