Getting Started Tutorial

Welcome to Dolby Hybrik! Hybrik allows you to easily manage large-scale media processing operations in the cloud. Hybrik provides transcoding, media analysis, and quality control services running across a very large number of machines. All storage and computing takes place in your own VPC, resulting in the highest security and lowest possible cost.

Once you have your AWS Credentials ready, we’ll walk you through the following setup steps to get you up and running with Hybrik:

• Setting Up A New Computing Group
• Your First Transcode
• Creating Your Own Job
• Editing Job JSON

And if you run into an issue at any point, be sure to check our Troubleshooting Tips below for some helpful solutions.

AWS Credentials

To begin working with Hybrik, you will need a set of AWS credentials for the machines that will be launched in your account. It is good practice to use a new Hybrik-specific IAM (Identity and Access Management) user for Hybrik activity, rather than using your master credentials. If you are not familiar with IAM on AWS, please read our Hybrik IAM Tutorial.

You should never share your credentials with anyone outside your organization. Hybrik will never ask you for your credentials. When you create a Hybrik Computing Group, you will use the credentials you have created. Different Computing Groups can use different IAM credentials if you so desire. This can be useful for tracking expenses for each computing group’s expenses separately within your AWS account.

Jobs and Tasks

All operations in Hybrik are defined by jobs. A job defines the source file(s) along with all of the associated operations that you wish to perform on that source. Each of these operations is known as a task. Available tasks include transcode, package, analyze, quality control, copy, and notify. Tasks may execute in parallel or in series, depending on the requirements of your workflow. For example, when creating an HLS output, it is possible to create all of the layers in parallel on different machines. In order to create the master HLS manifest, however, all the layers have to be completed first. Therefore a job specifies not only the tasks, but also the sequence of execution of the tasks.

All jobs within Hybrik are defined by a JSON (JavaScript Object Notation) file. JSON is an open-standard, human-readable mechanism for defining structured data. The Hybrik JSON file describes all of the tasks to be executed, as well as the specific order for each task. This includes error path operations. For example, the job may specify that a source file gets analyzed before transcoding to ensure that it meets the submission standards for size and data rate. If a source file fails, then it could trigger a series of additional tasks, such as copying the file to a new location and notifying the submitter that their file is incorrect. Please see the Hybrik JSON Tutorial for more information about the structure of Hybrik JSON.

Computing Groups

Your Hybrik account can be configured to support multiple Computing Groups. Each Computing Group specifies an AWS Region, a Machine Type, and a Launch Type. There are many reasons why you might want to configure multiple Computing Groups. For example, you might have one computing group in Europe to handle S3 assets stored there, while having a different group in the US to transcode US-based assets. You always want to perform your transcoding in the same region that you are storing your data. If you have your data in one region and your processing in another, you will be charged by Amazon for moving the data between the regions.

Another reason you might want different Computing Groups is to have different types of machines performing different tasks. Each job can have user-specified tags that direct the job to a machine with a matching tag. For example, suppose that you have two types of jobs – one type that is a high-priority job and another type that doesn’t have a deadline but needs to be done at the lowest possible price. You could set up two Computing Groups; one could be configured to launch very high performance machines whenever a high-priority job arrives, and another group could be configured with a fewer number of cheaper, lower-performance machines to limit costs. Large media operations will often be configured with many different Computing Groups in order to precisely control the types, quantity, and cost of machines being launched.

Interface Overview

When you are given a Hybrik account, you will receive an email instructing you to set your password. Once you set your password, you will be able to log into the Hybrik interface. On the left side of the interface is a series of main menu items. Click on a main menu item to see the sub-items for that category.

• Dashboard
  • Gives you an overview of all running jobs, machines, and costs
• Storage
  • Browse
    • Browse your cloud storage, select files to preview, analyze, or transcode
• Jobs
  • Create
    • Create a Job by selecting a source file, transcoding template, and destination
  • Edit
    • Edit a Job JSON file
  • Active Jobs
    • A list of all currently running Jobs, showing progress on each task
  • Queued Jobs
    • A list of all Jobs that are currently waiting for a machine on which to execute
  • Completed Jobs
    • A list of all Jobs that have completed successfully
  • Failed Jobs
    • A list of all Jobs that have one or more failed tasks
• Machines
  • Activity
    • A list of all currently active machines in your Computing Groups
  • Configuration
    • Where you configure Computing Groups
• Account
  • Info
    • Information about your account, including API keys
  • Users
    • Manage sub-users and their permissions
  • Notifications
    • Create and manage SNS notifications for your jobs
  • Credentials Vault
    • Manage your cloud accounts, including AWS IAM credentials

Setting Up A New Computing Group

1. On the left side of the screen, select the menu item “Machines” and then the sub-menu item “Configuration”. This will show you a list of all the current Computing Groups. If you haven’t created one yet, this will be blank.
2. Select the “New Computing Group” button at the top of the page, then choose AWS or GCP, depending on your cloud service platform. This will bring up a dialog that allows you to create a new Computing Group. First, give this Computing Group a name. It can be anything you wish, but it is helpful to be descriptive for differentiating groups. For example, “US-East-1 Urgent” might be a good name. Next, choose the Product Version from the drop-down options. For now, choose the highest version, but not the “Always use latest…” option. The Product Version allows you to configure your Computing Group to use specific versions of the Hybrik product. This can be important when you have received approval for an output created by a specific product version, and you want to guarantee that changes made in later product versions will not apply to your outputs. Next, give a Priority to this Computing Group. Priority goes from 1 to 255, with 255 being the highest priority level. If there are two Computing Groups, the Computing Group with the higher priority will launch machines first. It’s fine to use the default of 100.
3. Next you will need to add your AWS credentials that you want to use with this computing group. Choose the “Add New Credentials” button. These credentials are encrypted and stored in your Credential Vault. The credentials are used to launch EC2 machines in your AWS account and to allow those machines to access your data on S3. This will bring up a dialog to add a new set of AWS credentials to your account.
   • Name
     • Give a name to the credentials. This is just to help you distinguish between sets of credentials. Example: “Dave’s AWS Credentials for Hybrik”
   • API Key
     • This is the name that can be referenced in API -based jobs that wish to use this credential set. Example: “dt_hybrik”.
   • Description
     • A short description of the credentials. (Optional)
   • Type
     • The type of credentials used. Choices are:
       • AWS
       • Google Cloud Platform
       • Google Storage Interop
       • Azure Storage
       • Cloudian S3
       • SwiftStack
       • Wasabi
       • Username/Password
   • Access Key ID
     • The public part of the access key pair
   • Secret Access Key
     • The secret part of the access key pair

   We’ll use AWS for this tutorial. Click “Save” to complete the credential creation process.

4. In the next section you need to specify the type and location of the machines that will be launched as part of this computing group. First you will need to specify the AWS Region where the machines for this group will be launched. The region you select should be the same one where your data is located. If you have source material in different regions, then you will want different computing groups for each region. If your data and your machines are not in the same region, AWS will charge you to move the data between the regions.
   
   • AWS Region
     • The AWS Region where machines will be launched

   There are currently 14 different AWS regions -- Here is a list of the regions and their locations.

   Code	Name
   us-east-1	US East (N. Virginia)
   us-east-2	US East (Ohio)
   us-west-1	US West (N. California)
   us-west-2	US West (Oregon)
   ca-central-1	Canada (Central)
   eu-central-1	EU (Frankfurt)
   eu-west-1	EU (Ireland)
   eu-west-2	EU (London)
   eu-west-3	EU (Paris)
   eu-north-1	EU (Stockholm)
   ap-east-1	Asia Pacific (Hong Kong)
   ap-northeast-1	Asia Pacific (Tokyo)
   ap-northeast-2	Asia Pacific (Seoul)
   ap-northeast-3	Asia Pacific (Osaka-Local)
   ap-southeast-1	Asia Pacific (Singapore)
   ap-southeast-2	Asia Pacific (Sydney)
   ap-south-1	Asia Pacific (Mumbai)
   sa-east-1	South America (São Paulo)

   • Group Type
     • The group type of machine that will be launched as part of this group
       • Hot
         • These machines run 24/7, regardless of job load. They cost the “on-demand” price from AWS.
         • Hot machines are usually used in situations where the latency of starting On-Demand or Spot machines (which can take several minutes) would be too much.
       • On-Demand
         • These machines only launch when there are jobs to be run, and cost the “on-demand” price from AWS.
         • On-Demand machines might be chosen for extremely long jobs (>6 hours), since there is always a risk of losing a Spot machine.
       • Spot
         • These machines only launch when there are jobs to be run, but they will be bid for on the AWS Spot Market. Savings on the spot market can be over 80% compared to the On-Demand price.
         • The majority of Hybrik customers do 100% of their processing on Spot machines in order to maximize their savings.
   • Instance Type
     • The AWS EC2 instance type that will be used in this configuration
   • Maximum Instances
     • The maximum number of machines that this group will ever launch.
   • Minimum Instances
     • If you set this to more than zero when using Spot instances, then Hybrik will always keep that number of Spot machines running. This is like having a Hot machine, but instead of paying the On-Demand price you are paying the Spot Market price for the machine.
   • Pricing
     • This button will let you see the Spot Market price graph of your chosen machine type.
   • Maximum Bid
     • If you are using Spot machines, you can specify the maximum amount that you are willing to bid for a machine.
   • On-Demand Failover
     • Checking this box allows Hybrik to automatically bid up to maximum of the on-demand price, and then switch over to On-Demand machines at that point.
     • Hybrik recommends that you check this box whenever using Spot machines.
   • Max Idle Time (min.)
     • This defines how long after finishing its tasks an instance should wait for new tasks before shutting down. The default is 5 minutes.
5. In the last section are Task Tags. It is optional whether you use Tags with your Computing Groups. The purpose of Tags is to allow the routing of specific jobs to specific machines. For example, you might want to route high-priority jobs to high-performance machines. There are two types of Tags that you can specify for a Computing Group.
   • Mandatory Tags
     • If you set a Mandatory Tag, then only tasks with that Tag will be executed in this Computing Group. For example, if you a set a Mandatory Tag of “super-urgent” for this Group, then only tasks with a matching “super-urgent” tag will be run on this Group of machines.
   • Provided Tags
     • If you set a Provided Tag for this Computing Group, then this Group will execute any task that has a matching tag. In addition, this Computing Group will execute any non-tagged tasks. In other words, a Provided Tag says that a Computing Group is “available” to execute Tasks with that tag, but it can do other things as well.
6. Select the Save button at the bottom of the dialog and you are ready to go!

Your First Transcode

Jobs in Hybrik are defined by JSON files. To get started, please download some of our sample JSON files and decompress the ZIP file. Next you can can submit the jobs to Hybrik from the web interface.

1. Go to the Jobs section, and you will see a Queued Jobs menu item. There are four different states that a job can be in:
   • Active
   • Queued
   • Completed
   • Failed

   

   You can submit a new job from any of these locations. The actions that you can take on a particular screen will always be in top section above the list items. If a selection is greyed out, that means that you need to select a particular item (or items) in the list first.

2. At the top left you will see a “Submit Job JSON” button. Choose this and then select the job JSON files that you downloaded.
3. You will see your selections show up in the list of queued jobs. Once these jobs have been queued, Hybrik will launch machines in your AWS account to process the jobs.
4. Go to the Machine Activity page to see machines being launched in your account. It can take a few minutes for the machines to be acquired on the spot market and then have the Hybrik software loaded and begin processing. When you move to a production environment, you will make decisions on the amount of latency that is acceptable for your processing. You can may choose to have some machines running constantly, or only have them launch as needed.
5. As machines start to run, the jobs will move from the queued state to the active state. Check the Active Jobs page to see the jobs as they are running. On the upper right of the screen, you will see several icons. If you hover your mouse over the icons they will tell you their operation. The gear icon will allow you to change the columns of data that are displayed in each window. The details icon will show (or hide) a details view at the bottom of the page. Selecting a job in the upper window will then show you the details of that job in the lower pane. Congratulations, you have completed your first transcodes! Once the jobs have finished, they will show up in the Completed Jobs section.

Creating Your Own Job

Your next step will be to create a job using your own assets. First, make sure that you have some video files in an Amazon S3 bucket. Then follow these steps to create your own transcode.

1. Go to the Jobs menu and select Create. This will bring you to the Create Job page. Choose a Template Category, then choose one of the available Template options.

   

2. Next, you will select the source file that you would like to process. Under File Locations, and to the right of the Source field is a Browse button. Click this to display a browser that allows you to see the contents of your storage. Navigate to the file you would like to transcode, select it and click the Select button at the bottom of the dialog.

3. Next you can adjust the Video & Audio Parameters to your requirements by modifying the available fields.

   

4. Next, choose the Destination Path (storage location) for your output file. Use the Browse button to bring up a file selector just like you did when you were selecting the source file.

   

   Navigate into the desired folder for your output, then click the Select Folder button at the bottom of the dialog.

   

5. Finally, you can select Submit Job to submit it for processing. There are also the choices of Edit JSON and Export JSON. If you choose Edit JSON you will be presented with Hybrik’s JSON editor which will allow you to hand-modify your job JSON. If you choose Export JSON, the JSON file will be downloaded to your local computer for later submission or editing. After submitting the job, you can go to the Active Jobs or the All Jobs page to view your job’s progress.

   

Editing Job JSON

As mentioned previously, all jobs in Hybrik are defined by a JSON structure. In order to get full utilization out of the Hybrik system, you will need to know how to edit JSON. This will allow you to make any change you need in your Hybrik jobs and fully customize your workflows. Please see the Hybrik JSON Tutorial for more information on the Hybrik JSON structure. Let’s create and edit a sample JSON file.

1. Perform steps 1 - 4 from Creating Your Own Job above, only this time choose Edit JSON as the last step. This will present you with a JSON text editor.

2. In the JSON, you’ll see at line 7 and 8 there are definitions present that define the height and width of the target output. Modify those values to be 720 and 1280, respectively.

   Also change the bitrate_kb to a different value. Remember that it is in kilobits, so 2400 is equal to 2400 kilobits or 2.4 Mbits. Also change the job name (line 16 in the example), and then click Save Job JSON. This can be your practice file while you test out different parameters. You can submit the JSON you created either from the Job Editor or from any of the Job pages (All, Queued, Active, Completed, or Failed). You can edit your JSON files from within Hybrik or by using the text editor of your choice. We highly recommend using a JSON editor (rather than a plain text editor), since it gives you many tools to help you during the editing process. This will help with producing correctly formatted JSON. JSON is very specific in its requirements, and a missing bracket or an extra comma will result in invalid JSON.