Analyze and Quality Control Tutorials

Hybrik supports both analysis and quality control of media content. Automated analysis and QC can be executed prior to transcoding to ensure that the incoming content meets the required standards, as well as after transcoding to ensure that the results meet the expected output criteria. You can also execute jobs that only do analysis without doing any subsequent transcoding.

Hybrik separates analysis and quality control into separate Analyze and QC Tasks. The analyze task will report the requested data about the file, while the qc task makes a pass/fail decision for your content by comparing the metadata reported by the Analyze Task with criteria you supply. For example, the Analyze Task could report that there are five seconds of black at the beginning of a video. This might be correct for your requirements, or it might be incorrect. Using the QC Task, you would specify what are the allowable values for each type of analysis.

There are a variety of different types of analysis that Hybrik can perform on your media, and some of them may be inappropriate (or unnecessary) for your media type. Some analysis tasks can take significant time to run, so Hybrik allows you to specify what types of analysis you would like to perform on your content.

In this tutorial, we’ll create a job that runs an Analysis Task on the source file, a QC Task that ensures the source file meets a set of acceptance criteria that we’ll define, and then performs a Transcode Task if the criteria have been met. This tutorial assumes you are already familiar with the basic Hybrik JSON; if you have not already done so, please take a moment to review the Hybrik JSON Tutorial.

General Properties

There are two categories of properties that can be specified in an analyze task: general_properties and deep_properties. The general properties are those that can be extracted from the header content of the file. Deep properties require that the individual frames of the media be decompressed and analyzed, which takes more time.

The available general properties of a media file include:

Container
- Kind
- Faststart
- Duration
- Size
- Overall bitrate
Video
- PID
- Codec
- Profile
- Level
- Bitrate mode (CBR, VBR, etc.)
- Video stream bitrate
- Frame rate
- Frame dimensions
- Interlacing
- Display Aspect Ratio
- Pixel Aspect Ratio
- Chroma and color space info
- Duration
- etc
Audio (one for each audio track)
- PID
- Codec
- Bitrate info
- Sample Rate
- Channel count
- Duration
- Profile
- etc

When you specify that you would like analysis of the general_properties of a file, Hybrik will return all of these parameters.

Deep Properties

The deep_properties of a media file require examining each frame (both audio and video) of the file to determine the specific property. This often requires both analyzing the frame and comparing it with other frames in the file. The deep properties are generally configurable, allowing you to set specific thresholds for detection and control the sensitivity of the analysis.

For example, there is a deep property to identify black frames within the video. In the qc task definition, you may select both what percentage of pixels in the frame have to be black in order for the whole frame to be considered black, as well as the brightness value that registers that a pixel is black.

In the case of black detection, you must determine the boundary conditions for that particular analysis – e.g., if a frame is almost entirely black, but has a few pixels that are dark gray, does it count as a black frame?

The available deep properties include:

Video
- Black
  - detect frames that are black within the video
- Black borders
  - detect letter- or pillar-boxing
- Interlace mode
  - analyze frame-by-frame for interlacing artifacts
- Levels
  - analyze the video for min/max, Y, Cb, Cr, etc.
- Blockiness
  - detect compression block artifacts
- HDR Levels
  - calculate the MaxFALL and MaxCLL for HDR encoding
- Source complexity
  - measure for how complex the content is over time
- Content variance
  - analyze how much the content changes over time
- Scene change detection
  - detect scene changes probabilities
- PSE
  - detect Photo-Sensitive Epilepsy (PSE) artifacts
- Compressed Statistics
  - determine compressed frame sizes etc.
- Compressed Quality
  - determine quality of the underlying bitstream like PQ values etc.
- SSIM
  - determine the SSIM value between the media and a reference
- PSNR
  - determine the PSNR value between the media and a reference
- VMAF
  - use the Netflix VMAF method to compare the media and a reference
Audio
- Levels
  - DC offset, RMS peak, etc.
- EBU R.128
  - loudness analysis
- Dolby Professional Loudness
  - Perform Dolby loudness analysis on the audio
- Volume
  - simple volume measurement
- Silence
  - detect silent segments in the audio tracks
- PSNR
  - compare the asset to a reference audio track
- Emergency Alert
  - detect emergency alert signals in the audio
- Channel Compositions
  - determine channel composition in the audio
- Test Tone
  - find a test tone in the audio

Analyze Task JSON

When creating an analyze Task, you must specify which properties are going to be analyzed. In the example below, we are specifying that we want the general properties to be analyzed, as well as the following deep properties: audio levels, silence, black video, and black borders. More specifically, we are also saying that in order for a segment to be considered “silent”, it must be silent, under -60 dB, for more than 1 second. For a segment to be considered “black”, less than it must be black for more than 1 second. This allows us, for example, to skip over false positive results for brief moments of silence or black in our media.

{
  "uid": "analyze_task",
  "kind": "analyze",
  "payload": {
    "general_properties": {
      "enabled": true
    },
    "deep_properties": {
      "audio": {
        "levels": {
          "enabled": true,
          "is_optional": true
        },
        "silence": {
          "enabled": true,
          "is_optional": true,
          "noise_db": -60,
          "duration_sec": 1
        }
      },
      "video": {
        "black": {
          "enabled": true,
          "is_optional": true,
          "black_level": 0.03,
          "duration_sec": 1
        },
        "black_borders": {
          "enabled": true,
          "is_optional": true
        }
      }
    }
  }
}

Any time we do not specify a property in the analysis, Hybrik will use the default values. In the case of the black detection, our default value for black pixel detection is 0.1 (i.e., 10% of white). In terms of the number of pixels on a screen that must be black (black_pixel_ratio), we default to 98%. You could change both of these values in the JSON by setting them explicitly, as with this example:

{
  "black": {
    "enabled": true,
    "black_level": 0.2,
    "black_pixel_ratio": 0.99,
    "duration_sec": 2.0
  }
}

Please see the Hybrik API documentation for a complete list of the available settings for each analysis type.

Analyze Task Results

When Hybrik completes a job, it returns a JSON structure with all of the data that resulted from the job. Each task returns its own section of JSON. You can access this data either programmatically using the Hybrik API or as a file via the Hybrik UI. If you are using the UI, go to the Completed Jobs view, select a specific job, and then choose “Export Job Summary”. This will export a JSON file which contains the summary of the job. This file may contain a significant amount of information, including:

Any errors
Job statistics
- Execution time
- Task count
- Retries
- etc
For each task
- Machine info
- CPU utilization
- Memory usage
- etc
Task statistics
- Execution Time
- Start
- Stop
- Retries
- etc
Task Results

You can search in the job summary for the “Analyzer” section, which is where Analyzer results are returned. If you have multiple Analyze tasks in your workflow (for example, for both the input and output video), then your Job Summary will have multiple Analyze results. The results are broken into the following sections:

probes
- this lists all the analysis components that were run
general_properties
- the general properties of the file
deep_properties
- the deep properties of the file – each sub property is in its own object

The “General Properties” object looks like the following

{
 "general_properties": {
   "container": {
     "duration_sec": 60.022,
     "bitrate_kb": 1312.112,
     "size_kb": 9614,
     "kind": "mp4",
     "faststart": false
   },
   "audio": [
     {
       "pid": 2,
       "duration_sec": 60.022,
       "codec": "aac_lc",
       "bitrate_mode": "cbr",
       "bitrate_kb": 127.628,
       "sample_rate": 48000,
       "channels": 2,
       "language": "en"
     }
   ],
   "video": {
     "pid": 1,
     "duration_sec": 60.019,
     "codec": "h264",
     "profile": "High",
     "level": "L3",
     "frame_rate": 23.976023976023978,
     "bitrate_kb": 1180.352,
     "height": 480,
     "width": 848,
     "frame_rate_mode": "constant",
     "interlace_mode": "progressive",
     "dar": 1.767,
     "par": 1,
     "clean_aperture_height": 480,
     "clean_aperture_width": 848,
     "chroma_format": "cs_420",
     "bit_resolution": 8
   }
 }
}

general_properties has container, audio, and video sections. The container section contains general information about the media container. You will notice that the audio section is an array, and there will be one array object for each audio track. The video section gives you basic information about the video track.

deep_properties has an object for each type of audio or video analysis performed. Remember that each type of desired analysis must be specified. Deep property analysis tasks can be quite CPU intensive, so Hybrik only performs the deep analysis tasks specified in the job.

Each analysis will return a set of values, often including statistics like the mean value, standard deviation, minimum value, maximum value, etc. These values can be helpful in your QC process. Some analysis types also return charting data, which is a sampling of the specified value over time. Charting data is returned as a set of 100 values that are evenly spaced out over the duration of the file.

{
  "deep_properties": {
    "video": {
      "levels": {
        "frames": 1439,
        "bpp": 8,
        "y": {
          "broadcast_safe": false,
          "mean": 121.2,
          "stddev": 12.58,
          "mean_frame_max": 154.6,
          "min": 37,
          "min_pos_frame": 901,
          "min_pos_sec": 37.6,
          "max": 255,
          "max_pos_frame": 829,
          "max_pos_sec": 34.6
        },
        "charts": [
          {
            "datasets": [
              {
                "data": [
                  125.55,
                  126.613,
                  ...

Quality Control Task

The qc task allows you define a set of conditions that an asset needs to meet, and evaluates the results of an analyze task using those conditions. If all of the supplied conditions are not met, the QC Task will fail. QC Tasks can evaluate parameters associated with the file format/wrapper/container, the video stream, or the audio streams, depending on the type of criteria being tested.

Each QC test has a condition string, which defines the test using an expression compatible with the syntax used in the math.js JavaScript library. In addition the QC test can include pass and fail messages that are inserted into log files and reports.

For example, if we want to ensure that a file is not missing audio tracks, we could verify that the file contains at least one audio track by testing that the number of channels detected in the first audio track (audio[0]) during the analyze Task is greater than or equal to 1:

{
  "uid": "qc_task",
  "kind": "qc",
  "payload": {
    "conditions": {
      "pass": [
        {
          "condition": "general_properties.audio[0].channels >= 1",
          "message_pass": "PASS: Number of audio channels in first audio track is greater than or equal to 1, there are {general_properties.audio[0].channels} channels",
          "message_fail": "FAIL: Audio track not present or empty."
        }
      ]
    }
  }
}

Below is a complete qc task that performs the following evaluations against the source file:

File size is greater than 10 megabytes
Video width is at least 1920 pixels
Video height is at least 1080 pixels
Number of audio channels for the first audio track is at least 1
Audio sample rate is more than 44.1kHz
Any detected periods of black are less than 5 seconds long
Letterboxing (black borders at top and bottom) is not present
Average audio RMS level is at least -60db
Any detected periods of silent audio are less than 5 seconds long

Remember that whenever you create a qc task, you will also need to create a matching analyze task that generates all the values that the qc task will validate.

Regarding audio tracks and channels, the code below only evaluates the first track (the value 0 in general_properties.audio[0].channels), to evaluate other tracks replace the 0 with another integer.

{
  "uid": "qc_task",
  "kind": "qc",
  "payload": {
    "conditions": {
      "pass": [
        {
          "condition": "general_properties.container.size_kb > 10000",
          "message_pass": "PASS: Source file size is greater than 10 megabytes, size = {general_properties.container.size_kb} kb.",
          "message_fail": "FAIL: Source file size is less than or equal to 10 megabytes, size = {general_properties.container.size_kb} kb."
        },
        {
          "condition": "general_properties.video.width >= 1920",
          "message_pass": "PASS: Video frame width is greater than or equal to 1920, width is {general_properties.video.width} pixels.",
          "message_fail": "FAIL: Video frame width is less than 1920, width is {general_properties.video.width} pixels."
        },
        {
          "condition": "general_properties.video.height >= 1080",
          "message_pass": "PASS: Video frame height is greater than or equal to 1080, height is {general_properties.video.height} pixels.",
          "message_fail": "FAIL: Video frame height is less than 1080, height is {general_properties.video.height} pixels."
        },
        {
          "condition": "general_properties.audio.size() >= 1",
          "message_pass": "PASS: Number of audio tracks is greater than or equal to 1, there are {general_properties.audio.size()} tracks",
          "message_fail": "FAIL: There are no audio tracks."
        },
        {
          "condition": "general_properties.audio[0].sample_rate >= 44100",
          "message_pass": "PASS: Audio sample rate is greater than or equal to 44.1 kHz, sample rate is {general_properties.audio[0].sample_rate} Hz.",
          "message_fail": "FAIL: Audio sample rate is less than 44.1 kHz, sample rate is {general_properties.audio[0].sample_rate} Hz."
        },
        {
          "condition": "deep_properties.video.black.size() == 0 || deep_properties.video.black.max('duration') <= 5",
          "message_pass": "PASS: Longest black segment is shorter than 5 seconds.",
          "message_fail": "FAIL: At least one black segment is longer than or equal to 5 seconds."
        },
        {
          "condition": "deep_properties.video.black_borders.top == 0 && deep_properties.video.black_borders.bottom == 0",
          "message_pass": "PASS: Video letterboxing is 0 pixels.",
          "message_fail": "FAIL: Video letterboxing is more than 0 pixels, top = {deep_properties.video.black_borders.top} pixels, bottom = {deep_properties.video.black_borders.bottom} pixels."
        },
        {
          "condition": "deep_properties.video.black_borders.left == 0 && deep_properties.video.black_borders.right == 0",
          "message_pass": "PASS: Video pillarboxing is 0 pixels.",
          "message_fail": "FAIL: Video pillarboxing is more than 0 pixels, left = {deep_properties.video.black_borders.left} pixels, right = {deep_properties.video.black_borders.right} pixels."
        },
        {
          "condition": "deep_properties.audio[0].levels.rms_level_db >= -60",
          "message_pass": "PASS: Audio RMS level is higher than or equal to -60 dB, level is {deep_properties.audio[0].levels.rms_level_db} dB.",
          "message_fail": "FAIL: Audio level is lower than -60dB, level is {deep_properties.audio[0].levels.rms_level_db} dB."
        },
        {
          "condition": "deep_properties.audio[0].silence.size() == 0 || deep_properties.audio[0].silence.max('duration') <= 5",
          "message_pass": "PASS: Longest audio silence is shorter than 5 seconds.",
          "message_fail": "FAIL: At least one audio silence is longer than or equal to 5 seconds."
        }
      ]
    }
  }
}

Some of these tests are dependent on specific deep properties from the analyze Task. The video.black, video.black_borders, audio[].levels.rms_level, and audio[].silence[] tests would fail if the Analysis Task did not include the video.black, video.black_borders audio.levels, and audio.silence analyses in the deep_properties section.

In our connections array, we connect the transcode Task to the success state of the qc task, meaning that the transcode Task is triggered only if all of the tests within the QC Task have passed. We’ve connected an email notification to the error state of the QC Task, so that the transcode Task will not be triggered and someone can be notified to remedy the issue.

For more examples of conditions, see the Conditionals Tutorial.

Quality Control Task Results

Results of the comparisons made in a qc Task are available in the task result JSON. In the example below, the qc Task failed because one criterion wasn’t met - letterboxing bars measuring 76 pixels high were detected. The rest of the test passed: file size, video width/height, number of audio tracks, audio sample rate, black frame duration, pillarboxing bars, audio level, and audio silence were all within the limits we defined.

"hybrik_qc": {
  "errors": [
    "FAIL: Video letterboxing is more than 0 pixels, top = 76 pixels, bottom = 76 pixels."
  ],
  "passes": [
    "PASS: Source file size is greater than 10 megabytes, size = 296409639 kb.",
    "PASS: Video frame width is greater than or equal to 1920, width is 3840 pixels.",
    "PASS: Video frame height is greater than or equal to 1080, height is 2160 pixels.",
    "PASS: Number of audio tracks is greater than or equal to 1, there are 8 tracks",
    "PASS: Audio sample rate is greater than or equal to 44.1 kHz, sample rate is 48000 Hz.",
    "PASS: Longest black segment is shorter than 5 seconds.",
    "PASS: Video pillarboxing is 0 pixels.",
    "PASS: Audio RMS level is higher than or equal to -60 dB, level is -25.624174 dB.",
    "PASS: Longest audio silence is shorter than 5 seconds."
  ],
  "warnings": [
  ]
}

Examples

For more examples of conditions, see the Conditionals Tutorial