S3 Event SNS Notifications

New in version 5.0.

Vidispine supports receiving S3 events from an SNS topic. By using this, periodic scanning of the storage can be avoided. The events types handled by Vidispine is ObjectCreated and ObjectRemoved and for these messages either a file is considered present in the bucket or removed. No difference is made between for instance a ObjectCreated:Put or ObjectCreated:Post.

Configure SNS S3 notifications

  1. Create an Amazon SNS topic. You could create the queue from either the AWS Management Console or using the SNS SDK. Make sure that the S3 bucket is allowed to send messages to the topic, for example, using a policy on the topic like:
{
  "Version": "2008-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:eu-west-1:123456791011:topic_name",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:bucket_name"
        }
      }
    }
  ]
}

Note

  • With a Amazon SNS topic HTTPS needs to be used for the Vidispine instance since sending credentials needed in the URL to subscribe is not allowed.
  • Use one topic per bucket. Don’t send events from multiple buckets to the same topic, as this is not supported by Vidispine.
  1. Configure the S3 bucket to send ObjectCreated:* and ObjectRemoved:* events to SNS. The notifications can be set up from the S3 bucket properties in the AWS Management Console, or using the S3 SDK.

  2. When the storage has been configured in Vidispine it can be subscribed to the topic from either AWS Management Console, or using the S3 SDK. The endpoint to subscribe to is:

    API/sns-endpoint

Configure the storage

  1. To have Vidispine receive file events from an SNS topic instead of scanning a S3 bucket, set the storage method metadata snsTopic to the ARN of the topic:

    PUT /storage/VX-1/method/VX-2/metadata/snsTopic
    Content-Type: text/plain
    
    arn:aws:sns:eu-west-1:123456791011:topic_name
    
    GET /storage/VX-1
    
    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
      ...
      <method>
        <uri>s3://bucketname/</uri>
        ...
        <metadata>
          <field>
            <key>snsTopic</key>
            <value>arn:aws:sns:eu-west-1:123456791011:topic_name</value>
          </field>
        </metadata>
      </method>
      ...
    </StorageDocument>
    
  2. With SNS active, regular scanning of the storage will still be performed. However, storage scanning can be disabled by setting scanInterval to -1. A manual scan can also be triggered with:

    POST /storage/VX-1/rescan
    

Note

  • Vidispine will automatically confirm the subscription confirmation sent out by Amazon if the topic ARN matches the one specified in the method.