Workflow Automation Use Case: Delete Large Frame Stacks in Seconds

Use the Automation Engine to delete thousands of files in just a few seconds

Date
Author
Filip Milovanovic
Post-production expert,
ELEMENTS
Category
Workflow

When it comes to high-end video production, more workflows than ever rely on frame stack formats such as DPX and EXR. These formats provide a number of benefits to VFX and color grading workflows, but they can also create technical challenges because of the way they are structured. Every frame is saved as a separate, usually large, file with a single frame of UHD 16 Bit footage at approximately 50MB in size. This means that just 10 minutes of 24 FPS footage amounts to 14400 files, and for one hour of footage you can expect around 86400 DPX files.

Working with vast quantities of exceptionally large files can complicate every step of the workflow, but in this blog, we are going to focus on one, file deletion. We will determine the challenges faced when deleting frame stacks and show you how the ELEMENTS Automation Engine can drastically improve this process.

While file deletion may sound like a trivial issue, it most certainly isn’t. Every experienced user that has ever tried deleting frame stacks via macOS Finder or Windows Explorer will know the pain of this long and drawn out process. For example, using the Explorer in Windows 10 to delete 5000 UHD DPX files (46MB each) through an otherwise unburdened 1Gbit connection took 44 seconds. Deleting 10000 files took 80 seconds and 20000 took 152 seconds. As expected, this situation is made drastically worse if the user is connected through a VPN connection. The slow file deletion stems from the fact that the application has to open almost every single file, run calculations and keep the user informed of the progress.

This is obviously not the best approach, and it is far better to complete this deletion task directly on the storage. This way, all the unnecessary metadata traffic and the increased waiting times are avoided. In addition to that, some of these solutions also bring added security features and may help you to prevent accidental deletions.

Using the Automation Engine

With our Automation Engine users have access to a powerful toolset that can help them to easily create automated workflows with any number of steps that can then be triggered in the most familiar way, through the context (right-click) menu of the macOS Finder or Windows Explorer. Users are also able to execute complex chains of actions right on the storage, effectively eliminating the possibility of human error. Starting a job is hard to mess up and requires basically no training. This is what our Automation Engine can offer.

This one-step automation has been created by one of our customers and is a perfect example of how a simple workflow in the ELEMENTS Automation Engine can save a lot of time. In its simplest form, only one Delete files task is executed. If required, the automation can be expanded by any number of different tasks, for instance, an e-mail notification that informs chosen users about the task and lists all deleted files.

Deleting frame stack sequences by using the Automation Engine is extremely fast, allowing you to delete practically any number of files in a matter of few seconds. But this approach also has other benefits. Add a step to check if the files have been backed up before deleting them, use a simple extension check script from our Secure Upload example to delete only files that have certain extensions or schedule the running to clean out selected folders on a regular basis. The Automation Engine also allows you to decide which user is allowed to trigger which automation job and improves accountability by keeping a detailed log.

Apart from being able to start the automation job from the macOS Finder or Windows Explorer, users can also do it from the Media Library. Pro tip, the automation can also be displayed as a button in the Media Library’s button bar, making the job easier to find. Individual automation Jobs can be enabled or disabled for every Media Root separately.

Even though the Automation Engine might be the most versatile and secure way to approach the task of deleting a large number of files, ELEMENTS systems offer two further possibilities.

Alternative: File Manager

Many storage vendors have solutions that allow you to manage the files on the storage. In the case of ELEMENTS systems, the browser-based UI allows you to do just that through the easy-to-use File Manager tab. Simply select all the files and folders and click the delete button. If multiple DPX sequences can be found in a folder, the File Manager will automatically recognise and group them together for easier handling.

Though very efficient, this approach is limited to those users who have administrative permissions. It would require that the user logs into the ELEMENTS Web UI, chooses the right files and deletes them. This approach is obviously susceptible to human error.

Alternative: Media Library

Another approach to the task is to delete the DPX sequence from the Media Library, our Media Asset Manager. The DPX sequence is recognised as such by the Media Library and is displayed as a clip. The clip can be played, shared and used for collaboration. Simply selecting any number of clips allows the user to delete the connected DPX files instantaneously. The user can even choose to delete or move only the DPX files while leaving the preview proxy in the Media Library. In that case, the clip will be marked with a trashcan symbol, but users will still be able to play it back and use all of the other functionalities of the Media Library. Should the user restore the DPX files onto the ELEMENTS system, the Media Library will recognise the files, reattach them to the proxy and remove the trashcan symbol, indicating that the files are once again located on the system.

Deleting the files in the Media Library is fast and more secure than when using the File Manager. This is due to the visual and intuitive nature of this tool and the integrated user permission structure.

Do you have a cool use case for the Automation Engine that you would like to share with us? We would love to hear about it.

Check out some of the other use cases, such as the secure media upload automation,  or an automation to easily consolidate media from an XML sequence into cloud.

Workflow

Everything You Need to Know About the Proxy Workflow in DaVinci Resolve

Workflow

Everything You Need to Know About The Proxy Workflow in Adobe Premiere Pro

Workflow

Setting up a Dropbox Cloud Archive and Restoring Jobs with the Automation Engine

Glossar

COBIT

COBIT ist ein international anerkanntes Rahmenwerk für das Management und die Governance von Informationstechnologie. Es bietet ein umfassendes Regelwerk von Prinzipien, Praktiken und analytischen Instrumenten und Modellen zur Steuerung der unternehmensweiten IT.