Definitions and Placeholders Tutorial

The Hybrik job JSON allows you to construct complex media processing workflows. In the standard JSON file, each task has its own section that completely defines all the components of that task. There are times where it can be convenient to create something like a variable in your JSON, and that is where Definitions and Globals can assist you. Let’s take an example of a workflow where several tasks all need to refer to the same folder location. You could, of course, explicitly state the exact folder location in each task. Or, you could define the folder location once, and then have every task refer to the definition. This becomes particularly useful when you are modifying JSON files to submit via the API. Rather than having to replace a dozen different locations in your JSON, you could simply replace one. Globals are similar, but rather than being defined by the user, they are defined by Hybrik.

Definition Example

At the start of your JSON file there can be a definitions object. Inside this definitions object are all the items that you would like to replace in your file. When Hybrik encounters text bracketed in double curly-brackets, then it will look for a matching Definition and replace the contents of the label with the contents of the Definition. Let’s look at a simple example:

{     
  "name": "Test Job",
  "definitions": {
    "source_file": "s3://hybrik-examples/public/sources/sample1.mp4",
    "destination_folder": "s3://hybrik-examples/public/output/"
  },
  "payload": {
    "elements": [
      {
        "uid": "source_file",
        "kind": "source",
        "payload": {
          "kind": "asset_url",
          "payload": {
            "storage_provider": "s3",
            "url": "{{source_file}}"
          }
        }
      },
      {
        "uid": "transcode_task",
        "kind": "transcode",
        "task": {
          "retry_method": "fail"
        },
        "payload": {
          "location": {
            "storage_provider": "s3",
            "path": "{{destination_folder}}"
          },
          "targets": [
                                  ...

In this example, we have defined something called the source_file and the destination_folder. Whenever there is a {{source_file}} or {{destination_folder}} text anywhere in the rest of the JSON file, it will replace these placeholders with what has been defined in the definitions section.

The Definitions section is not restricted to just simple text. You can include object definitions as well. For example, suppose you were creating an HLS file where you wanted every layer to have the same audio properties. You could define an entire audio_track object in the definitions. This could include information about codec, channels, filters, etc. It would look like this:

{  
  "definitions": {
    "source_file": "s3://hybrik-examples/public/sources/sample1.mp4",
    "destination_folder": "s3://hybrik-examples/public/output/",
    "audio_track": {
      "codec": "heaac_v2",
      "channels": 2,
      "sample_rate": 44100,
      "bitrate_kb": 128,
      "filters": [
        {
          "kind": "normalize",
          "payload": {
            "kind": "ebur128",
            "payload": {
              "allow_unprecise_mode": false,
              "integrated_lufs": -16,
              "true_peak_dbfs": -2
            }
          }
        }
      ]
    }
  },
  ...
}

Then, in each of your layers, you would only need to reference the {{audio_track}} definition. This way, if you ever need to change, for example, the audio filtering, you only need to change it in the Definition section. Your HLS target would look like:

...
"targets": [
  {
    "file_pattern": "Layer1.m3u8",
    "container": {
      "kind": "hls",
      "segment_duration": 10,
      "segment_file_pattern": "Layer1_%05d.ts"
    },
    "video": {
      "codec": "h264",
      "width": 256,
      "height": 144,
      "frame_rate": 23.976,
      "bitrate_kb": 180
    },
    "audio": [
      "{{audio_track}}"
    ]
  }
]
...

JSON Inheritance

In previous example, the entire audio_track object defined in the Definitions is substituted everywhere that the {{audio_track}} tag appears in the JSON file. But what about when we need to replace part of an object, rather than the whole object? Let’s take the following example:

{
  "definitions": {
    "source_file": "s3://hybrik-examples/public/sources/sample1.mp4",
    "destination_folder": "s3://hybrik-examples/public/output/",
    "video_basics": {
      "codec": "h264",
      "profile": "high",
      "level": "4.0",
      "bitrate_mode": "vbr",
      "preset": "slow",
      "frame_rate": 23.976,
      "height": 1920,
      "width": 1080
    }
  },
  ...
}

We would like all of our targets to use the same basic video parameters defined above, but we want each target to have it’s own bitrate. So, the {{video_basics}} tag is going to replace part of our video transcoding description but not all of it. We can’t do the following:

...
"targets": [
  {
    "file_pattern": "Target1.mp4",
    "container": {
      "kind": "mp4",
    },
    "video": {
      "{{video_basics}}", <-- no!
      "bitrate_kb": 4000,
      "vbv_buffer_size_kb": 6000
    },
    ...

This would be invalid JSON. Instead, we can use the $inherits property of JSON. It looks like this:

"targets": [
  {
    "file_pattern": "Target1.mp4",
    "container": {
      "kind": "mp4",
    },
    "video": {
      "$inherits": "{{video_basics}}",
      "bitrate_kb": 4000,
      "vbv_buffer_size_kb": 6000
    },
    ...

When Hybrik evaluates this JSON, the $inherits property gets expanded to this:

"targets": [
  {
    "file_pattern": "Target1.mp4",
    "container": {
      "kind": "mp4",
    },
    "video": {
      "codec": "h264",
      "profile": "high",
      "level": "4.0",
      "bitrate_mode": "vbr",
      "preset": "slow",
      "frame_rate": 23.976,
      "height": 1920,
      "width": 1080,
      "bitrate_kb": 4000,
      "vbv_buffer_size_kb": 6000
    },
    ...

Just remember to use the $inherits tag when you are replacing part of an object rather than the entire object.

Hybrik Placeholders

There are a few special variables in Hybrik that can be used to refer to commonly needed values. These are referred to as “Globals” or Placeholders, and are defined by Hybrik rather than being defined by the user. Placeholders are referenced with a single pair of curly brackets. Here is a simple example:

"targets": [
  {
    "file_pattern": "{source_basename}.mp4",
    "existing_files": "replace",
    "container": {
      "kind": "mp4"
    },
    ...
  }
]

The value {source_basename} is not defined anywhere by the user. Instead, it is created by Hybrik by taking the original source name and removing the file extension. Therefore my_source_file.mov becomes my_source_file. This is a convenient way of referring to commonly needed items in your files.

Here is a table of the Hybrik Placeholders along with their properties:

Note that not all Placeholders are available in all Tasks. For example, the {default_extension} is only valid in the transcode Task in which the container is being specified. The reason for this is that you can have many different tasks in your workflow, and a particular Placeholder may not determinable from the location you are at. For example, if you have 5 different parallel Transcode Tasks in your workflow followed by a notification, the Placeholder {default_extension} could not be used in the Notification since there would be no way to determine which task this was referring to.