Tagging Tutorial

Hybrik allows you to create any number of Machine Groups. A Machine Group is a set of rules that determines the AWS region, machine type, number of machines, etc. that control the launching of machines in your AWS account. When a job is submitted, Hybrik will look at the Machine Groups to determine what type of machine to launch. Machine Groups can have different priorities, and Hybrik will launch machines in the Group that has a higher priority first. But how do you route a particular job or task over to a specific Machine Group? This is done with tagging. Simply put, if a job or task has a particular tag, then Hybrik will look for a Machine Group that has a matching tag. Tags can be any user-defined string. For example, if a job is given a tag of EUROPEAN_PRIORITY, then every task of that job will run on the Machine Group that has a tag of EUROPEAN_PRIORITY.

If you have a simple operation that only runs one type of machine in a single region, you may never use tagging. But tagging is a convenient way to manage multiple Machine Groups, allowing you to effectively route jobs depending on their region, priority, customer, etc. Since Machine Groups can have different AWS billing prefixes, some Hybrik customers use tagging to help provide billing information for different users.

Computing Group Tags

You associate a tag with a Computing Group in the Computing Group Editor dialog box. When you create or edit a Computing Group, this dialog allows you to choose all the settings associated with a Computing Group, such as AWS region, machine type, number of machines, etc. At the bottom of this dialog you will see the Task Tags section.

There are two types of tags that you can use here – Mandatory Tags and Provided Tags. In the example below, we have put the tag ACME_CUSTOMER in the Mandatory Tag section. This means that this Computing Group will only ever execute tasks that have the tag ACME_CUSTOMER. You can add multiple tags to a computing group by separating them with a comma. If, for example, we put the following - ACME_CUSTOMER, URGENT - into the dialog, then this Computing Group would only execute tasks that include BOTH of these tags. The Provided Tags allow you to specify additional tags that the Machine Group can accept. The Machine Group will execute tasks that have all the required Mandatory Tags AND zero or more of the Provided Tags. So, if we put ACME_CUSTOMER into the Mandatory Tags and put URGENT and PRIORITY into the Provided Tags, then the Machine Group would execute tasks that were either [ACME_CUSTOMER], [ACME_CUSTOMER, URGENT], [ACME_CUSTOMER, PRIORITY], or [ACME_CUSTOMER, PRIORITY, URGENT].

Job and Task Tags

Job Tags

It is important to remember that Hybrik jobs are composed of one or more tasks, and tasks get assigned to machines. So a job with 10 tasks may actually have those tasks executed on 10 different machines. You can assign tags to either a job or to a specific task. If you assign a tag to a job, then every task in that job is given the same tag and will only execute on Machine Groups that have a matching tag. You can also specify a tag for a specific task. Every task in a job can even be given a different tag. There is no requirement that every task be given a tag - it is perfectly acceptable to give one task a tag and leave the others untagged in which case the default computing group will be used.

A job tag looks like this:

{  
    "name": "Hybrik Tagging Example",
    "task_tags": [
        "ACME_CUSTOMER"
    ],
    "payload": {
        "elements": [
            {
                "uid": "source_file",
                "kind": "source",

This would assign the ACME_CUSTOMER tag to every task in this job, and therefore every task would execute on Machine Groups with the matching tag. This parameter is called the task_tags, and is an array because you can assign multiple tags to the tasks. There is another parameter called user_tags, which allows you to create any number of user-specified tags that can used for your internal tracking purposes. Hybrik will not take any action based on user_tags.

Read more on user_tags in our API Page under ‘create-job’. If you need to insert additional internal tracking data into a job, see our User Data Tutorial.

Task Tags

Suppose we only wanted to route the transcoding task to this Computing Group while letting all the other tasks run on any available machine? We could tag just the transcoding task like this:

{  
    "uid": "transcode_task",
    "kind": "transcode",
    "task": {
        "retry_method": "fail",
        "tags": [
            "ACME_CUSTOMER"
        ]
    }
 ...
}

Combining Job and Task Tags

If you include tags in both the job AND the task sections of your JSON, then you will create tasks that require both sets of tags as the two lists are concatenated together. For instance, if you assigned a tag of CUSTOMER_A at the job level, and a tag of PRIORITY in a transcode tasks, then all tasks in the job would require a Computing Group with the tag PRIORITY and additionally, the transcode task would need both a Computing Group with both CUSTOMER_A and PRIORITY in some combination of Mandatory and Provided tags.

Summarizing Tag behavior

• From a Task’s perspective
  • I will only run on a Computing Group that has all of my tags (from any combination mandatory and provided tags)
• From a Computing Group perspective
  • I will only execute Tasks that have all of my Mandatory tags and zero or more of my Provided tags