Machine Performance Analysis

In the Hybrik service, all machine processing happens in the customer’s own Amazon AWS account, and the customer pays AWS directly for that machine time. Hybrik allows for the selection of many different types of AWS machine instances for processing, and so a common question is “which machine should I use?”. The answer is… “it depends”. Why? Remember that Hybrik can support multiple computing groups that can be configured to run different types of machines. In your particular workflow, you could have two types of jobs: the jobs where only speed matters and you don’t care about price, and the jobs where price matters much more than performance. The answer to “which machine” for these two scenarios will be different. This tutorial will give you some insight into how to correctly configure machines for your media workflow.

The first thing to note is that Hybrik supports both on-demand and spot market machines on AWS. With spot market machines, there is a savings of about 75% compared to on-demand pricing. But, there is a chance of losing the spot machine depending on user demand. Hybrik manages this situation by providing automatic failover. Because of the cost savings and the failover protection, many of our customers choose to use spot machines for all of their encoding.

The next important factor in choosing machines is to understand that Hybrik uses standard CPU-based machines. There is no benefit to selecting machines with GPU or FPGA capabilities. Another factor to consider is that no media processing is perfectly threaded. For example, a 16 core machine may have 95% CPU utilization during an encode, but a 32 core machine may only have 60%. This utilization can vary depending on the input codec, output codec, and intermediate processing steps. Thus a machine that costs twice as much may only give 1.5X in performance increase. This is why it is important to run some of your own tests. We recommend people start their evaluation of Hybrik using a standard machine type like a c4.2xlarge or c5.2xlarge, and then test other configurations to optimize for their particular workflow.

AWS instance processing time is priced per hour, and is relative to the configuration of the instance. The configuration includes such properties as the number of processors, amount of memory installed, and maximum network bandwidth. The more of each property is available to an instance, the higher the price to use that type of instance. In the section below, we give you some example cost/performance numbers for transcoding on various machine types.

Example Cost/Performance Tests

In the table below, we have compared various common AWS machine types. The test performed was for 1 hour of 1080p source being encoded to a 1080p output H264 at 5Mbps using the medium x264 preset. In the table, we include the following items:

• Machine Type
  • The name of the AWS instance
• vCPUs
  • The number of virtual CPUs in the instance
• RT Factor
  • How fast was the transcode compared to real time. A RT Factor of 1.0 would mean that transcoding the 1 hour file took 1 hour. A RT Factor of 0.5 would mean that transcoding a 1 hour file took 2 hours. A RT Factor of 2.0 would mean that transcoding a 1 hour file took 30 minutes.
• Cost/Hour of Source
  • What was the absolute cost to transcode an hour of source

Running Your Own Performance Test

Should you want to run your own performance test, with your type of sources and with your list of preferred instance types, below is our process for setting up and executing such a test. Please make sure you’ve reviewed the Computing Groups section in the Getting Started tutorial; you should also review the Tagging Tutorial for information on how tags work in conjunction with computing groups as we make use of the tagging mechanism for these tests.

Hybrik Machine Configuration

1. Configure a set of Hybrik Computing Groups, calling out each instance type, one per machine configuration

   Setting	Value
   Computing Group Name	Instance type it will launch, for example c3.xlarge
   Instance Type	Select to match the computing group name, c3.xlarge
   AWS Region	Whichever region you use most
   Group Type	Spot
   Minimum Instances	0
   Maximum Instances	1
   on-demand failover	Check the box
   Max Idle Time	1
   Mandatory Tags	Set to match the computing group name, c3.xlarge
   Provided Tags	Leave empty

2. Save this machine configuration, and repeat (clone) the above steps for each of the instance types you want to test with. You should end up with a list of computing groups something like this (make sure that each row has the exact same value in the Group Name, Machine Type, and Mandatory Tags column)

{  
  "definitions": {
    "profile_name": "c3.xlarge",
    "source": "s3://hybrik-dt-east/Customer_Testing/Test_assets/Long Form Master.mp4",
    "destination": "s3://hybrik-dt-east/Internal_Testing/Performance_Testing/Version_{hybrik_version}"
  },
  "name": "Performance Testing - {{profile_name}}",
  "task_tags": [
    "{{profile_name}}"
  ],
  "payload": {
    "elements": [
      {
        "uid": "source_file",
        "kind": "source",
        "payload": {
          "kind": "asset_url",
          "payload": {
            "storage_provider": "s3",
            "url": "{{source}}"
          }
        }
      },
      {
        "uid": "transcode_task",
        "kind": "transcode",
        "payload": {
          "targets": [
            {
              "file_pattern": "{{profile_name}}{default_extension}",
              "container": {
                "kind": "mp4"
              },
              "video": {
                "width": 1920,
                "height": 1080,
                "bitrate_mode": "vbr",
                "bitrate_kb": 5000,
                "max_bitrate_kb": 5750,
                "vbv_buffer_size_kb": 5750,
                "frame_rate": 25,
                "codec": "h264",
                "profile": "high",
                "level": "4.0",
                "preset": "medium"
              },
              "audio": [
                {
                  "codec": "aac",
                  "channels": 2,
                  "sample_rate": 48000,
                  "sample_size": 16,
                  "bitrate_kb": 128,
                  "bitrate_mode": "vbr"
                }
              ],
              "existing_files": "replace"
            }
          ],
          "location": {
            "storage_provider": "s3",
            "path": "{{destination}}"
          }
        }
      }
    ],
    "connections": [
      {
        "from": [
          {
            "element": "source_file"
          }
        ],
        "to": {
          "success": [
            {
              "element": "transcode_task"
            }
          ]
        }
      }
    ]
  },
  "priority": "100"
}

Collating and Comparing Results

When each job is finished, you can view the transcode duration in the Jobs > Completed Jobs menu in the Hybrik web UI, in the Machine Duration column. The number there, for example 9,616 sec, is within a second or two of the total transcode time (this number also contains a very small API call overhead, which can be ignored for the purpose of our comparison).

Here is an example of what the duration results look like in the web interface.

• Machine Duration
  • Actual computer time (what AWS bills you for)
• Total Duration
  • Wall clock time for the job