Copy Task

The Copy Task can be used to copy files from one location in your bucket to another bucket or location within your bucket. You can also download from an http location to your bucket. You may wish to copy all of your output files to another location after the previous task completes.

Syntax for copy task

The copy task is fairly simple; any files written by the preceding task will be written to the destination specified in the task. Here is a basic example:

{
    "uid": "copy_task",
    "kind": "copy",
    "task": {
        "retry_method": "fail"
    },
    "payload": {
        "target": {
            "location": {
                "storage_provider": "s3",
                "path": ""
            },
            "existing_files": "replace",
            "file_pattern": "{source_name}"
        },
        "options": {
            "delete_source_on_completion": false
        }
    }
}

Options

There are many more options that you can specify with the copy task, including:

delete_sources - this deletes the task source once the copy has completed; effectively a move operation
restore_from_glacier - this allows files to be retrieved from S3 Glacier storage and copied to a new location, making them available for other operations

Example Jobs

Because it is the preceding task which tells the copy task which files to copy, you may find the Sources Tutorial useful.