High-Speed Internet Transfer through Integration with ExpeDat

With blisteringly-fast internet speeds becoming the norm, internet transfers are playing an increasingly important role in post-production workflows. Just imagine a world in which all content is quickly delivered to the office over the internet – straight from the set, without the need to juggle SSDs and memory cards. Believe it or not, these kinds of workflows are possible even today. However, they do require more than just a faster internet connection.

The TCP bottleneck

Just like raising the ceiling doesn’t make you taller, simply having a faster internet connection doesn’t guarantee faster transfer speeds. The whole stack needs to be questioned.
TCP (Transmission Control Protocol), the internet’s main transmission protocol has been around since 1974. But fifty years ago, CPUs and storage weren’t nearly as fast as they are now, and network congestion was mostly a theoretical problem. Today, modern technologies can provide far higher amounts of data and put more pressure on the network. This is when TCP can start to break down. Network congestion may occur when a sender overflows the network with a large number of packets which leads to a degraded quality of service through packet delay and packet loss. When a packet doesn’t reach the receiver, TCP only has limited time to recover the data or else it is forced to stop the transmission entirely. During the data recovery, TCP slows down considerably only to quickly flood the network again. This feast-or-famine behaviour makes it hard for TCP to scale and can leave a lot of bandwidth unused.
There are many ways to tune TCP for better performance on a particular path, including more than a dozen algorithm variants in common use. But unless you are network engineer, and willing to constantly retune your network stack, you’re probably stuck with whatever defaults are in your operating system.

Fast transfers require faster protocols

The best way to achieve high-performance transfer speeds is to leave TCP behind and use a more modern protocol build. UDP (User Datagram Protocol) is a communications protocol primarily used to establish low-latency connections between applications over the internet.
Although UDP is often associated with connectionless services like DNS or video streaming protocols like RTP, a number of companies have used it as the basis for implementing modern algorithms that can fill up even the fastest internet paths. Some of the best-known players in this market are IBM Aspera and Signiant. However, one simpler and considerably less expensive solution is ExpeDat from Data Expedition, Inc.. Their MTP protocol that powers this solution is based on UDP and is able to adjust flow and correct transmission losses in real-time. The transfer is fully encrypted and integrates very well into the ELEMENTS platform.

ExpeDat setup

ExpeDat is a server component that works together with a client app (available for macOS, Windows, Linux) and allows end users to connect. The client app doesn’t need to be installed, but simply run. To trigger a connection to the share, a special expedat:// protocol is enabled on the system to launch the app when requested. The transfer runs over a single UDP port – the default port is “8080” but can easily be changed.
When installing ExpeDat, you will need to consider the following for performance and security reasons:

• Which node should run the ExpeDat server? For this, a web-facing gateway is best used on which the ExpeDat UDP-port can be forwarded from your internet WAN.
• Which users are allowed to run these sharing tasks? For security reasons, this should be limited to a few users.
• Sharing capabilities should be limited only to workspaces, specifically created for this purpose.

ExpeDat can be setup to use the system users, specific users from a list, or a combination of both. The “home folder” can also be jailed to restricted locations. Check out this documentation for all the specific configuration options and controls.

Integration into the Automation Engine – Example

For our sharing example with the Automation Engine, we are building a list of random users with random passwords and creating a temporary user for every share. User credentials are automatically used for the share. The script below adds the temporary ExpeDat users to the “svpasswd” file that holds the users. This file will be reloaded by ExpeDat server binary, making the share ready before the mail is sent out. It is recommended to create a dedicated „ExpeDat“ user on ELEMENTS, and add this user to any workspace you want to share. Then ExpeDat can run in the context of this user rather than root.

#!/bin/bash

expedat="/var/lib/syslink/expedat/svpasswd"
expedaturl="SYSTEM_IP_ADDRESS"
landingpageurl="https://examplelandingpage.com"
blacklist="/var/lib/syslink/expedat/blacklist"

if echo {{input|shell_escape}} | grep -f $blacklist
   then echo "Forbidden Folder"
   exit 1
fi

u=$(uuidgen | cut -c 25- )
p=$(uuidgen | cut -c 25- )
echo $u:$p:::{{input|shell_escape}}:" #timestamp:"$(date +"%s") >> $expedat
echo $landingpageurl"?d=$expedaturl&u=$u&p=$p"

With the script bellow, the Automation Engine also allows you to expire or remove temporary users after a set time period.

#!/bin/bash

expedat="/var/lib/syslink/expedat/svpasswd"

touch "$expedat"2
timenow=$(date +"%s") 
while IFS= read -r line
do
	timethen=$(echo ${line#*#timestamp:})
	if [[ -z "${timethen//[0-9]}" ]]; then
		if (( $timenow < $timethen + 2592000 )); then echo $line >> "$expedat"2; fi
	else
		echo $line >> "$expedat"2
	fi
done < "$expedat"
mv -f "$expedat"2 $expedat

Authorised users can easily create a read-only share (for file download), a write-only share (an upload drop zone), or a read-write share (for collaboration purposes). The code block below manually removes a share when no longer needed.

#!/bin/bash

expedat="/var/lib/syslink/expedat/svpasswd"

cat $expedat | grep -v ":{{input}}:" > "$expedat"2
mv -f "$expedat"2 $expedat

Since the recipient might not be familiar with ExpeDat, it’s a good idea to provide them with a landing page so that they can familiarise themselves with the tool, download the client app for their OS and launch the share. ExpeDat offers a simple PHP / js web-template for creating a simple landing page. Alternatively, the provided SDK can be used to rebuild or wrap their binaries into your own applications.

Conclusion

Fast data transmission over the internet is of great benefit to the post-production process. But in order to efficiently transfer large amounts of video files with already huge sizes, a better transmission protocol is needed, one that is faster than the traditional TCP. This is exactly where a tool like ExpeDat can help. Furthermore, the integration of ExpeDat capabilities into the ELEMENTS Automation Engine provides users with a quick and intuitive way to share footage without having a to deal with yet another tool in their day-to-day work.
Tools like ExpeDat however only make sense when large amounts of data need to be transferred regularly. A typical useful case would be a transfer between the set and the office or between facilities at different locations. If however, transferring large amounts of data is not what you require, but an intuitive and fast way to share individual clips, it is far more efficient to use an advanced, purpose-built collaboration tool like the ELEMENTS Media Library. This powerful MAM allows users to add frame-accurate comments on the footage, assign tasks to users, create editing timelines and share footage over the internet in a fast, intuitive and secure way.

Please keep in mind that the displayed code is simple mock-up code of minimum required functionality – would need to be modified for custom needs and enhanced for added security (e.g. mapping allowed workspaces for sharing to a whitelist or a blacklist document).

Find out more about ExpeDat.

Similar use-cases can also be covered through integration with Aspera and Signiant.