Hybrik Sources - asset_url

A Simple Source

Every Hybrik job starts from a source element. The asset_url is a simple way to load a single asset by URL. Let’s examine the structure:

{
    "uid": "source_file",
    "kind": "source",
    "payload": {
        "kind": "asset_url",
        "payload": {
            "storage_provider": "s3",
            "url": "s3://path/to/source.mov"
        }
    }
},

You will notice that the kind is asset_url. In this case we are using a single .mov file as our source element which would be passed to the next task in our connections_array. For a simple “convert A to B”, this is the source task type that you will want to use.

The source (source.mov), moves from storage and is passed into the Source Pipeline which the rest of the job uses. Here is a visual representation of what the source looks like and what it looks like when it’s passed into the Source Pipeline:

Specifying Access Credentials

By default, Hybrik will use the access credentials associated with the computing group used in your job. In a more complex account configuration, you might have one IAM user that can read from your storage and another to launch instances. You can specify a specific set of credentials for reading your source with the access parameter.

{
    "uid": "source_file",
    "kind": "source",
    "payload": {
        "kind": "asset_url",
        "payload": {
            "storage_provider": "s3",
            "url": "s3://path/to/source.mov",
            "access": {
              "credentials_key": "my_credential",
              "max_cross_region_mb": 100
            }
        }
    }
},

In the above example the access key has two parameters:

• credentials_key
  • Instead of using the default keys associated with the computing group, this will reference an “API Key” as defined in the “Credentials Vault” under the “Account” tab in the Hybrik web console. You do not actually paste the credentials in here, you use the API Key. In the image below, you would use my_aws_credentials. This key will reference the stored credentials.
• max_cross_region_mb
  • Some cloud storage providers might charge to transfer data across different regions. For example, if you have a 100 GB file stored in S3’s us-east-1 and your computing group uses instances in us-west-2, you will be charged per-GB by AWS for the cross-region transfer. If both your storage and compute happen in the same region, there is no additional charge. This parameter can limit the amount of data transferred.
  • This parameter can be set to -1 to allow for unlimited cross-region data transfer.
  • The default value if not specified is 100 MB.

Multiple Source Assets

If you wish to have multiple source elements or to combine elements, you will want to use the asset_complex source type.

Examples

• Example simple asset_url