Without further ado
Quantums StorNext file system has enabled a large number of post-production workflows over the years, so it’s no surprise that the release of a new version has a lot of people excited, including us. The most notable changes and additions in the newest version include another (separate) UI, a switch to a capacity-and subscription-based licensing model and the addition of the File System Pooling feature, which is the main focus of this article.
What is File System Pooling?
Due to the intensely high demands of modern media workflows, post-production houses are always on the lookout for a storage solution that can consistently deliver ever-increasing performance. Consequently, the extremely fast NVMe drives have been receiving a lot of (well-deserved) attention in recent years. However, a Terabyte of storage on an NVMe comes at a hefty price-premium compared to the good-old Hard Drive. This is why media storage manufactures develop different strategies to tie these storage mediums into a single environment – one in which the user can profit from the performance of NVMe drives or even SSDs while at the same time relying on Hard Drives to provide the bulk of relatively low-cost storage. File System Pooling is Quantums solution to tackle this challenge. This feature allows users to group different types of storage mediums together inside one file system and define policies for moving the files between the pools.
How does File System Pooling work?
Pooling moves the data without changing the metadata associated with the file or the file attributes. Most importantly, pooling works on a block level and doesn’t actually move the files in the namespace – only the data of the file itself. Pooling can be initiated in two ways: Manually by a user or automatically by a preset pooling policy. Policies are scheduled tasks that use a set of criteria to identify the files to execute a pooling job with. Policy criteria relates to file size, files modification time, files location and more. It is important to mention that File System Pooling is handily integrated as part of StorNexts core filesystem components and not related to the tiering features of the separate StorNext Manager software suite, which are in our opinion historically complex and hard to use on a daily basis.
A pool is a set of stripe groups (or a single stripe group). Stripe groups are the StorNext terminology for an allocation group that consists of a set of disks or LUNs, which can come from different storage classes. Think one stripe group is entirely based on NVMe storage (e.g. our ELEMENTS BOLT) and another stripe group consists of LUNs from good-old HDD-based storage. These stripe groups could then be tagged “fast” and “slow” for example.
Pooling Jobs
Jobs define a set of instructions to be executed on specified content – either manually selected or identified by a policy. A pooling job can be either executed immediately or scheduled to run on a specific date and time. An internal locking mechanism of the StorNext filesystem prohibits changes to the file that is currently being used by a job. There are seven different job types offered and described in the StorNext documentation as follows:
move – This job moves the content to a target pool by making a new copy there and the old copy is discarded.
promote – This job moves the content to a target pool by making a new copy in the target pool and the old copy is retained.
demote – This job exchanges a new copy and an old copy, the new copy is discarded, unless the file changed, in which case a move is performed.
remove – This job removes the secondary copy and leaves the main copy intact.
inventory – This job counts the files and directories on each pool and generates a report.
checksum – This job reads and performs a checksum of the files using a specified algorithm and stores the information in an extended attribute on the file.
validate – This job reads and performs a checksum of the files using a specified algorithm, compare the checksum against the stored one, and reports the mismatches.
In our opinion, the most notable entries in the list of offered jobs are the promote and the demote jobs. Let’s consider them in an example:
A long stream of high bitrate footage with accordingly large file size is stored on the “slow” tier HDD pool. This clip is needed for editing and should be moved to the “fast” tier NVMe pool. Instead of simply moving the clip, it can be promoted to the fast tier pool. This job will effectively copy the footage and retain both copies. After the footage is no longer needed on the fast pool, it can be demoted. This job exchanges the two copies in a simple metadata-only operation and simply discards the copy on the fast tier pool. In case that the promoted content has been changed, these would be synced back obviously.
This promote/demote approach drastically reduces the overhead associated with copying the data back to the slow tier pool and is therefore far more efficient.
Quantum provides a web interface for the File System Pooling feature, which at the moment only runs on the 2nd node of an Xcellis cluster and is still somewhat limited in its functionality. But the whole feature set is available via the command “sntier” on all Linux based StorNext SAN clients that run at least version 7. Let´s look at a few commands to get you started. For a full command overview, run “man sntier” on a Linux SAN client.
Hint: You may want to check out our video at the end of this article which shows the ELEMENTS Automation Engine and its powerful client-side integration of these new StorNext 7 features.
Useful CLI commands
Before you can use “sntier”, you have to enable the service. This requires the StorNext filesystem services to be running and the SN Web API enabled (which has been the default setting for some time now). Run the following command to enable the service on the node:
# sntier service --enableNext you need to assign the stripe group(s) to the different pools, in our example we have one pool labeled “fast” and one pool labeled “slow”. The tags “nvme” and “qxs” refer to the stripe group names, which can be found in the file system configuration (e.g. via “cvadmin” => “show long”)
# sntier pool --add --sg fast nvme --sg slow qxs --mount /stornext/xcellis01/
After that you can review your pooling configuration:
# sntier pool
Pool status host localhost
Mount: /stornext/xcellis01
PoolName Capacity Used Exclusive Stripe Groups
fast 13.10 TiB 11.47% No nvme
slow 4.91 TiB 0.00% No qxs
Mount: /usr/adic/HAM/shared
Unassigned SGs: sg0
To create your first pooling job, run the following command. Specify the action (in this example “promote”) and the target pool (“fast”) plus the directory that you want to promote. Adding a trailing slash at the end will cause the operation to be recursive. The promotion will start immediately. Keep in mind that it will occupy a large amount of performance since the action happens in the storage backend at the highest possible performance. Simple promote job syntax without specified criteria:
# sntier submit --action promote --target fast --paths /stornext/pooling/dir1/Simple demote job syntax without specified criteria:
# sntier submit --action demote --target slow --paths /stornext/pooling/dir1/Depending on the amount of content, these operations can take quite some time. To review the activity, you can open the interactive jobs mode by using the following command:
# sntier jobsAlternatively, you can specify the number of the job and sntier will return a JSON output. This output can easily be parsed and used in tools and scripts.
# sntier jobs --show 1000
{
"content": {
"paths": [
"/stornext/xcellis01/test/"
],
"mount": "/stornext/xcellis01"
},
"originator": "admin",
"job": 1000,
"task": {
"action": "demote",
"target_pool": "slow",
"io_size": 1048576
},
"comp_report": {
"finishTime": "Tue Apr 6 13:48:26 2021",
"childjobs": 46,
"elapsedSec": 101.946394,
"filesSkipped": 0,
"filesMoved": 1000,
"bytesMoved": 50987008000,
"bytesPerSec": "477M",
"dirsChanged": 1,
"state": "completed",
"queueTime": "Tue Apr 6 13:46:44 2021",
"startTime": "Tue Apr 6 13:46:44 2021",
"filesScanned": 1000
}
}Expanding the functionality of File System Pooling
StorNext Pooling Jobs can easily be set up and triggered through the ELEMENTS Workflow Automation Engine. This integration allows you to execute Pooling Jobs together with a chain of other tasks, including: automatic transcoding, cloud synching, Slack notifications and even Media Library operations.
As an example, we have created a short video that shows how easy it is to integrate this feature flexibly into the ELEMENTS Automation Engine:
By loading this video, you accept Vimeo’s privacy policy.
Learn more
Conclusion
When it comes to media workflows, storage environments consisting of multiple types of storage mediums simply make a lot of sense. Using flash-based storage for extreme performance-demanding work and Hard Drives for basically everything else can drastically increase the efficiency of your workflows. However, this approach is only as effective as the data management mechanism itself. The new File System Pooling feature seems to offer all the features needed to profit from this kind of infrastructure and we are very excited to put it to work.
Read the full Quantum StorNext 7 File System Pooling documentation
