Storage Tiering in Post-Production

How to enjoy maximum performance from an efficient shared storage environment

Date
Author
Filip Milovanovic
Post-production expert,
ELEMENTS
Category
Workflow

Media workflows put some of the highest demands on just about every aspect of shared storage. Not only does footage with ever-increasing bitrates demand high storage capacity, but it also requires the highest performance to play back in real time. Simultaneously, the playback performance must be snappy and stable at all times.

Trying to fulfil all of these demands by using a single storage medium will have you stuck in a balancing act between performance, capacity and cost.

Instead, the principal approach to solving this challenge is to use a number of storage tiers, consisting of different storage mediums, with a powerful management layer on top. This will allow you to make the most of individual storage technologies and benefit from a cost-efficient environment at the same time.

Storage tiers

When it comes to shared media storage, we differentiate between three distinctive storage tiers:

Online storage – This tier provides the fastest access to the footage and is typically made up of flash-based storage mediums such as SSDs or NVMes. This kind of storage can provide a high throughput and support simultaneous access for a large number of users. This is down to the fact that flash storage, compared to hard drives, isn’t burdened by variable seek times, defragmentation or rotational latency – all of which have a negative effect on the data access time. Online storage is where the footage and the projects that are currently being worked on are stored.

Nearline storage – Easily accessible storage that offers somewhat slower access to the footage compared to online storage. Nearline storage provides the bulk of the storage capacity and is made up of a less cost intensive storage medium, typically hard drives. This is where footage might be stored when it is not being worked with – think of upcoming projects, next episodes or simply b-roll collections. When the time comes to use this footage, it can easily and quickly be moved to online storage or even used directly if the lower performance doesn’t bother you.

Offline storage – A pool of inexpensive storage that cannot be accessed as quickly as the other two storage tiers – think of an archive. Offline storage is often realised through usage of LTO tapes that offer high capacity at unparalleled prices and produce little to no maintenance cost when stored.

Cloud

Cloud storage enjoys an ever-increasing popularity within media workflows due to the benefits it provides such as: ultimate flexibility when it comes to capacity, data safety (your footage is safer in the cloud than on a shelf) and the elimination of a large, up-front investment in infrastructure associated with buying shared storage.

As internet connection speed has typically been a limiting factor for cloud utilisation, it was either used as offline storage to backup or archive the media or as nearline storage from which the required footage can be downloaded in a matter of hours. Today however, we are seeing a growing interest in using cloud as online storage. When working with low to medium bitrate footage, this can be achieved by implementing a high-bandwidth internet connection. When working with more demanding footage however, it makes more sense to “move” the applications to the cloud and thereby avoid the internet connection bottleneck. This area of the media industry is one that is currently being developed with a rapid pace, so we can certainly expect big innovations and new tools.

Media management

No matter how sensible it is to use multiple storage tiers, the efficiency of this storage environment for your workflows will ultimately be highly dependent on the media management mechanism. In many non-media workflows this data management would be automatised. A typical use case would see the data organised into tiers according to the date of last access and other clearly defined rules. This kind of multi-tiered, automated environment is typically referred to as Hybrid storage.

When it comes to media workflows, in most cases, this automated approach doesn’t necessarily work that well. This is due to the fact that the media’s access dates don’t have much significance when determining whether the footage will be needed soon. On top of that, video files are known to be notoriously large and may therefore require a long time to be moved back to the online storage. Instead, managing media files is best done by someone who is part of the project planning. This doesn’t mean that the process can’t be automated, but rather that the decision on how quickly you will be able to access your footage should be made intentionally.

Automation Engine

One of the main reasons why we have developed our ELEMENTS Automation Engine is to speed up and simplify the media management process. This flexible tool is included with every ELEMENTS system and allows you (among many other things) to move, copy, delete, sync etc. your media. The Automation Engine allows you to build complex workflows with steps such as transcoding, cloud operations, running python scripts and many more. Jobs can have as many steps as you wish and can be scheduled to run or executed through the Media Library, or simply from the context menu of the macOS Finder or Windows Explorer.

The Automation Engine has all the tools needed for the easy management of a tiered storage environment. It helps you to automate your media management and keeps every file exactly where it should be.

Media Library integrations

Using a Media Asset Manager to organise and distribute the footage across different storage tiers makes a great deal of sense. With the ELEMENTS Media Library, users have access to all the metadata and can always preview and play back the footage, no matter the file location – even when in deep cloud or LTO archive.

Users can quickly move selected assets to a different storage tier with a single push of a button. Uploading footage to the cloud or archiving it onto LTO tapes can be done just as easily. The fully indexed archive allows you to easily find the desired footage, preview, and restore it with a single click.

StorNext 7 File System Pooling

We have been enjoying a very productive relationship with Quantum for many years now and have implemented their high-performance file system StorNext into a large number of environments. This is why we were very excited to hear that the new StorNext 7 offers a number of new pooling features, so we simply had to try it out for ourselves. The interesting thing about these features is that they allow you to move the data without changing the metadata associated with the file or the file attributes. Pooling works on a block level and doesn’t actually move the files in the namespace – only the data of the file itself. This is a very efficient mechanism for managing a multi-tiered shared storage environment, and the integration of these tools into our Workflow Automation Engine allows for their intuitive and easy usage. To find out more about it head over to our blog post: Quantum StorNext 7 – Exploring the new tiering features

Vimeo

By loading this video, you accept Vimeo’s privacy policy.
Learn more

Load Video

Conclusion

Due to consistently high performance and capacity requirements, media workflows can benefit greatly from multi-tiered storage environments. In order to use the environment in the most efficient way however, a good media management strategy is needed. Thankfully, storage planning and management in media workflows is usually quite straight-forward. In most cases, the best approach is to simply keep the currently running projects and those that are planned to run soon on the fast storage layer, typically SSD- or NVMe-based. This layer can also be used for shared render and cache data.
The footage for the upcoming projects can be stored on the nearline, typically HDD- or Cloud-based storage which allows users to preview the footage and restore it in a relatively short timeframe. All footage that will not be required in the near future can be archived, while the ability to preview and restore it quickly remains possible. This is something that our Media Library can offer, no matter where your archived footage will reside – on the LTO tape, an off-site HDD system or in the cloud. To simplify, speed up and automate the periodic data distribution on the system, we recommend using our Workflow Automation Engine. This is by far the most efficient strategy for moving data between storage layers and it requires minimal interaction, thus greatly minimising the chances of human error.

When it comes to the optimal architecture of your next environment, you can contact us directly or speak with any of our partners worldwide. Together, we will find the right balance between the many different storage technologies available and build the perfect solution for your needs.

Workflow

Using the OpenAI Transcription Engine to Generate Subtitles

Workflow

Exploring the New Scene Cut Detection Features of DaVinci Resolve and Adobe Premiere Pro

Workflow

Automatic Transcription with DaVinci Resolve’s Speech-To-Text Function and Text-based Video Editing

Glossar

COBIT

COBIT ist ein international anerkanntes Rahmenwerk für das Management und die Governance von Informationstechnologie. Es bietet ein umfassendes Regelwerk von Prinzipien, Praktiken und analytischen Instrumenten und Modellen zur Steuerung der unternehmensweiten IT.