SNS Subscriptions Tutorial

In a Hybrik workflow, you can have Notification Tasks that report when the workflow has reached a certain step. But, how do you receive information about the overall progress of a job? A task at the end of a workflow will tell you that the workflow has completed, but it won’t tell you when the workflow is 30% complete. You could poll the Hybrik REST API to get the status of a particular job, but it is far simpler to simply get notifications. In order to get these system-level notifications, Hybrik uses the Simple Notification Service (SNS), a component of Amazon Web Services. SNS notifications can be used with any job to trigger messages about the progress of the job and its various tasks.

Amazon’s SNS is a managed push-based notification service that coordinates the delivery of messages to various endpoints. Messages in SNS are handled by a publish/subscribe model, so you will need to set up SNS publishing within Hybrik and then choose how the message should be handled by SNS. Each message is grouped under a “topic”, so when you publish and subscribe, you must do so for a particular topic. As an example, you could create two topics on SNS called “testing” and “production”. Then, when you are submitting jobs to Hybrik, those jobs could trigger messages to one of those topics. The “testing” topic could send messages to your email, whereas the “production” topic could alert you on your mobile phone.

Amazon SNS supports the following types of notifications

HTTP, HTTPS
- Subscribers specify a URL as part of the subscription registration. Notifications will be delivered through an HTTP POST to the specified URL
Email, Email-JSON
- Messages are sent to registered addresses as email. Email-JSON sends notifications as a JSON object, while Email sends text-based email
SQS
- Users can specify an SQS standard queue as the endpoint. Amazon SNS will enqueue a notification message to the specified queue (which subscribers can then process using SQS APIs such as ReceiveMessage, DeleteMessage, etc.). Note that FIFO queues are not currently supported
SMS
- Messages are sent to registered phone numbers as SMS text messages

There is a very small charge for notifications, which depends on the type of notification. For example, 100,000 notifications over HTTP would cost $0.06. SMS message charges vary by destination country.

To start, we need to create an SNS topic in AWS. In your AWS dashboard is a section called Messaging, and the Simple Notification Service is located there. Go to the SNS dashboard, and under “Common Actions”, choose the “Create Topic” choice.

You will be given a dialog to create a new topic. The topic name is required, as this will be used to create your custom topic. The Display name is only required if you are creating an SMS subscription.

Now that you have created a topic, AWS will assign it a Topic ARN (Amazon Resource Name). This ARN is unique across all of Amazon. In this example, the topic ARN is arn:aws:sns:us-east-1:01234567890:my_test_sns.

You will use the ARN for any external services (like Hybrik) that want to publish to that SNS topic.

Now that you have your topic created, you need to create a Subscription to that topic. Subscriptions define how messages published to your SNS topic are handled, and you can set up multiple subscriptions for any given topic. For example, you could have messages sent to different people by different mechanisms like SMS or Email. Or, you could trigger different external processes using HTTP or SQS messaging.

You will get a confirmation that the subscription was created.

When you create a subscription, AWS will send a verification message to that endpoint and confirm that the endpoint accepts the subscription. So, for example, you can’t send an email via this method unless the recipient verifies that they want to accept emails from this topic.

Now that you have your SNS notifications setup in AWS, you can configure Hybrik to publish notifications to your topic. Hybrik can publish status change events for both jobs and tasks. These status changes include:

Job Status Change
- Queued
- Active
- Completed
- Failed
- Progress
Task Status Change
- Queued
- Active
- Completed
- Failed
- Progress

You can subscribe to one or more of these events, and Hybrik will send SNS notifications when the events occur. Please note that Task Status Changes can generate large numbers of notifications. For example, if you have a job with 10 tasks, you could easily generate hundreds of notifications for the job, as each task would be reporting its own progress. Most Hybrik users only request Job Status Changes.

To create a subscription in Hybrik:

Log in to the Hybrik web interface and navigate to the Account > Notifications page
Click the “New Subscription” button. The fields are described below
- Name
  - A name for your subscription.
- Key
  - The Key matches a subscription_key that you can (optionally) specify in your job. Specify a key such as “production_jobs”, or any string you like. Events from jobs with a matching subscription_key specified will be published to this SNS Topic. You can also leave this field empty to receive events from all jobs. Place subscription_key at the root level of your job JSON.
- Type: SNS
- SNS Topic ARN: The AWS ARN that was generated when you created your SNS Topic (e.g., “arn:aws:sns:us-east-1:1234567890:my-sns-topic”). Hybrik will publish events to this SNS topic.
- User Data
  - You can specify any custom data here that will be included with all events we publish to this subscription. You can set job-specific user_data in your job JSON.
- Credentials
  - Select the computing group whose credentials Hybrik should use to publish the SNS. This computing group’s credentials must have the AWS SNS:Publish permission for the specified SNS Topic ARN resource.
- Notification
  - Select the events you would like to subscribe to.
- Include Metadata toggle
  - If this is flipped to Yes, job provided user_data will be included along with other details such as job name, source, start_time, and end_time.
Press the Test button. This will check to ensure that Hybrik is able to publish to your SNS topic using the selected credentials.
Press the Save button to save the subscription.

Example Responses

When Hybrik publishes a notification to the SNS Topic, it may include one or more events inside. Hybrik will combine multiple events in order to reduce the number of SNS notifications being issued. Below is an example response from Hybrik which includes both Job and Task data

{
  "user_data": "my subscription user data",
  "events": [
    "event_name": [
      "array of individual event payloads for this event type"
    ],
    "job_status_change": [
      {
        "id": 100,
        "status": "queued"
      },
      {
        "id": 120,
        "status": "active"
      },
      {
        "id": 125,
        "progress": 55
      }],
    "task_status_change": [
      {
        "id": 394,
        "job_id": 120,
        "status": "active"
      }
    ],
  ]
}

The status of any job or task can be in one of four states:

queued
active
completed
failed

Hybrik Jobs

Subscription Key

When submitting a job, you will generally want to specify a subscription_key. This allows you to assign the job to a particular SNS category. So, for example, you could have two SNS subscriptions configured, one with a subscription_key of test_jobs and one with a subscription_key of production_jobs. Therefore, different types of jobs would send messages to different end points. The subscription_key is included in the Job JSON at the root level, like below:

{
  "name": "Hybrik API Example#1 - simple transcode",
  "subscription_key": "production_jobs",
  "payload": { 
    ... 
  }
}

The subscription_key is in the job must match one set in the SNS Subscription in the Hybrik Web Console.

Events related to this job will only be published to subscriptions that have a matching key. If you set the job’s subscription_key to null or an empty string "", then no events will be published for this job.

User Data

If you want to have your own job user_data returned in the SNS notification, you will want to toggle Include Metadata to Yes. User_data provided in the job JSON will be included along with other details such as job name, source, start_time, and end_time.

Given this Hybrik job:

{
  "name": "Hybrik API Example#1 - simple transcode",
  "subscription_key": "production_jobs",
  "user_data": {
    "my_identifier": 12345
  },
  "payload": { 
    ... 
  }
}

Here is an example SNS notification with Include Metadata set to Yes:

"SingleEvent": {
    "job_status_change": [
        {
            "id": "3804555",
            "status": "completed",
            "metadata": {
                "name": "My SNS example Job",
                "user_data": {
                    "my_identifier": 12345
                },
                "source": "s3://foo/bar/my_file.mov",
                "start_time": "XXXX-04-26T17:49:58.000Z",
                "end_time": "XXXX-04-26T18:09:31.000Z"
            }
        }
    ]
}

Comprehensive Sample Response

This example covers all possible properties/events that can be included in a notification response.

{
    // Subscription user_data is included if subscription user_data is defined, otherwise this is omitted
    "user_data": "{\"my_subscription_user_data\": 1, \"jobs\": \"dev\"}",

    // Events are organized by event type
    "events": {
        // Task Status Change Events
        // status can be "queued", "active", "completed", "failed"
        // status and/or progress are only included if the corresponding event type is enabled in subscription
        "task_status_change": [
            {
                "id": "23934784", // Task ID
                "job_id": "9521921",
                "status": "active",
                "progress": 50
            },
            {
                "id": "23935000",
                "job_id": "9521950",
                "status": "queued"
            },
            {
                "id": "23935010",
                "job_id": "9521951",
                "progress": 34
            },
            {
                "id": "23934797",
                "job_id": "9521922",
                "status": "completed",
                "progress": 100
            },
            
        ],
        
        // Job Status Change Events
        // status can be "queued", "active", "completed", "failed"
        "job_status_change": [
            {
                "id": "9521921", // Job ID
                "progress": 95,
                // if Metadata is enabled, available metadata properties name (job name), source, start_time, end_time, error, user_data (job user data) are included if available
                "metadata": { 
                    "name": "Test Job",
                    "source": "https://hybrik-examples.s3.amazonaws.com/public/sources/tears_of_steel_720p.mov",
                    "start_time": "2023-11-16T23:35:59.000Z"
                }
            },
            {
                "id": "9521922",
                "status": "completed",
                "progress": 100,
                "metadata": {
                    "name": "Test Job",
                    "user_data": {
                        "my_job_user_data": 1,
                        "client_id": "customer123",
                        "type": "dev"
                    },
                    "source": "s3://hybrik-examples/public/sources/tears_of_steel_720p.mov",
                    "start_time": "2023-11-17T00:08:06.000Z",
                    "end_time": "2023-11-17T00:11:58.000Z"
                }
            },
            {
                "id": "9521925",
                "status": "failed",
                "metadata": {
                    "name": "Test Job",
                    "user_data": {
                        "my_job_user_data": 1,
                        "client_id": "customer123",
                        "type": "dev"
                    },
                    "source": "https://hybrik-examples.s3.amazonaws.com/public/sources/tears_of_steel_720p.mov",
                    "start_time": "2023-11-17T00:20:50.000Z",
                    "end_time": "2023-11-17T00:20:55.000Z",
                    "error": "Job cancelled by user"
                }
            }
        ]
    }
}

Amason SNS Overview

Configuring SNS Topics

Hybrik SNS Notifications

Example Responses

Hybrik Jobs

Subscription Key

User Data

Comprehensive Sample Response