Setting up a Dropbox Cloud Archive and Restoring Jobs with the Automation Engine

Since our inception in 2011, we have continuously enhanced the ELEMENTS platform to offer an unparalleled suite of tools tailored for modern media production workflows. In addition to a robust media asset manager, an integrated Rough-cut Editor, and a sophisticated Automation Engine, the ELEMENTS platform boasts cutting-edge cloud capabilities. By entering your cloud storage credentials into the Systems Integrations tab, you can seamlessly integrate and leverage your cloud storage in various ways. If your preferred cloud storage isn’t integrated yet, you can still utilise it by setting up specific Automation jobs. This blog provides a comprehensive guide on using the Automation Engine to effectively archive footage to Dropbox and harness the advanced features of the ELEMENTS Media Library for seamless restoration back to your on-premise storage.

What are the Benefits?

Integrating cloud storage into your ELEMENTS environment unlocks powerful new capabilities. For instance, it enables you to establish an advanced cloud archive system. When content is ingested onto your ELEMENTS system, the Media Library automatically creates and stores lightweight versions of your footage in selected quality settings while moving the original footage to the cloud. Despite the original footage residing off-site, users can still preview it, add metadata, utilize collaboration features, and even incorporate footage into rough-cut editing projects. Robust search functionality allows users to quickly locate footage based on metadata, creation time, annotations, user attribution, and more. Once identified, users can seamlessly retrieve assets from Dropbox back to ELEMENTS with a single click.

This feature can also be used outside the Media Library to archive files and folders straight from macOS Finder or Windows Explorer. Select and then right-click any number of files. In the open context menu, start the Archive to Dropbox automation job. Any number of pre-archive checks can be added to the job, and any user can be notified of the execution status via a Slack or Teams message, email, or system notification.

What is the Automation Engine?

Every ELEMENTS system incorporates our robust Automation Engine, designed to streamline post-production workflows with its intuitive user interface. This powerful tool comes equipped with pre-implemented steps that enable users to manage files, transcode footage, trigger notifications, execute custom Python and Bash scripts, and perform various other automated tasks seamlessly.

This time-saving tool gives users with little to no technical knowledge the power to reliably execute a whole list of complex tasks in seconds, straight from the Media Library, macOS Finder or Windows Explorer.

The Automation Engine also includes tasks specifically designed for seamless integration with connected cloud storage solutions. Supported platforms like Amazon AWS, Microsoft Azure, Google Cloud, or Backblaze B3 automatically extend the Automation Engine’s capabilities to utilize these cloud services. For other cloud providers, such as Dropbox, that are not yet integrated, users can employ Python scripts within the Automation Engine to achieve similar functionality.

Setting up Rclone

Sharing files between the ELEMENTS system and Dropbox will be facilitated by the Rclone tool. To set it up, you first need to create your own Dropbox App ID for Rclone by following these steps:

1. Log into the Dropbox app console with your Dropbox account.
2. Choose the type of access you want to use => Full Dropbox or app folder. If you want to use Team Folders, Full Dropbox is required.
3. Name your app.
4. Click Create App
5. Switch to the Permissions tab. Enable at least the following permissions: account_info.read, files.metadata.write, files.content.write, files.content.read, sharing.write. The files.metadata.read and sharing.read checkboxes will automatically be marked.
6. In the Settings tab, in the OAuth2 section, add the following redirect URI: http://localhost:53682/
7. Copy the app key and app secret values.

Provide the App key and App secret values to the ELEMENTS support and we will connect your Dropbox account.

Creating Automation Jobs

To create the jobs, open the ELEMENTS Web UI and head to the Jobs tab in the Automation section. Add a Python task to the newly created job and paste the code provided below. Any number of additional tasks can be added to the jobs, and these can be linked according to the execution status of the tasks.

Important job settings:

• For every job, turn on “Require user selection” in the Variable tab. This feature enables variable input selection.
• In the Advanced tab, configure whether the job should appear in the context menus of macOS Finder and Windows Explorer, and specify the Media Roots within the Media Library where it should be accessible. Additionally, in the same tab,  you can choose which users are allowed to initiate the selected job.

Archive to Dropbox

This Python task copies any file(s) from the Media Library to the identical path in Dropbox.

rclone copy {{ input | shell_escape }} my-drop-box:{{input | parent | shell_escape}}

You have the option to include a delete task with a variable input {{input}} to automatically execute after a successful upload, thereby freeing up storage capacity. Additionally, configure notification steps to receive alerts via Slack, Teams, or email upon task completion or in case of any encountered errors.

Archive to a Specific Location

This Python task copies any file(s) from the Media Library to a custom path in Dropbox. If the path doesn’t exist in your Dropbox storage, it will automatically be created.

rclone copy {{ input | shell_escape }} my-drop-box:your/custom/path

Similarly to the Archive to Dropbox automation, you can execute delete, notification, or any other tasks once the upload process is complete.

Restore from Dropbox

This Python task retrieves content that was previously archived to Dropbox.

rclone copy my-drop-box:/path/file(s) /data/beegfs1/.projects/path/file(s)

Even when the high-res content has been uploaded to Dropbox and deleted from the ELEMENTS system, the Media Library will still keep preview proxies on the system. You can preview the clips stored in your Dropbox cloud, complete with all their metadata. To restore the clip from Dropbox to its original location, simply select the clip in the Media Library and utilize the „Restore from Dropbox“ automation job.

Read more about the advanced cloud capabilities of the ELEMENTS platform.