Research data no longer lives in one place. It sits on a parallel file system next to the compute cluster, on NAS shares in the lab, in S3-compatible object storage, on tape and increasingly in the cloud. Moving it between those places is still often done by hand or by a collection of scripts. A data mover replaces that with software that moves files between storage systems automatically, according to rules you define, and proves that every file arrived intact.
A data mover is software that copies or moves files between different storage systems based on policies, then verifies and records every transfer. It is not a backup tool, which keeps copies for recovery, and it is not a one-off migration script. It is a permanent service that keeps the right data on the right storage, at the right time.
Most HPC and research teams already move data with one of two tool families: rsync, or MPI-based parallel copy tools such as mpiFileUtils. Both are good at what they were built for. The question is what happens when data movement stops being an occasional job and becomes a daily, cross-storage operation.
rsync is the default tool for copying and synchronising directories, locally or over SSH. It only transfers what changed, which makes it efficient over slow links. It was not designed for parallel file systems with millions of files: one rsync process walks the tree and copies files one stream at a time, and building the file list alone can take hours on large directories. Teams work around this by splitting the tree and running many rsync processes in parallel with wrappers such as GNU parallel. By default rsync compares size and modification time; full checksums are possible but slow on large trees.
On HPC clusters, MPI-based tools such as mpiFileUtils (dcp to copy, dsync to synchronise, dcmp to compare) spread a copy across many processes on many nodes. For a one-off bulk copy between two parallel file systems mounted on the cluster, they are among the fastest options available. They run as batch jobs: someone with MPI and job-scheduler access has to size the job, launch it, check the output and compare the trees afterwards. They work on POSIX file systems mounted on the cluster nodes, not on object storage or cloud. They also consume the CPU, memory and network of the servers they run on, so a large copy competes with the computing jobs those nodes are there for. And they keep no log or history: once the job ends, there is no record of which files were copied, when, or by whom.
A data mover keeps the parallelism but adds what the other two leave to scripts: rules that decide what moves and when, support for file, object, tape and cloud storage, automatic verification of every transfer, and a permanent record of what moved where.
| Capability | rsync | MPI tools (mpiFileUtils) | NODEUM Data Mover |
|---|---|---|---|
| Parallel transfers built in | No | Yes | Yes |
| Moves data to object storage (S3) | No | No | Yes |
| Moves data to cloud | No | No | Yes |
| Verifies every transfer automatically | No | No | Yes |
| Selects data by policy (age, size, path, metadata) | No | No | Yes |
| Schedules and event triggers (API, webhooks) | No | No | Yes |
| Log and history of every file moved | No | No | Yes |
| Leaves compute nodes free for computing | Depends | No | Yes |
| Usable without scripting or MPI skills | No | No | Yes |
| Best for | Small syncs over slow links | One-off bulk copies inside a cluster | Continuous movement across storage tiers |
These tools are complementary. rsync and MPI tools remain useful for ad-hoc copies. A data mover is what you add when the same movements happen every day, across more than one kind of storage, and someone needs to prove they happened.
NODEUM Data Mover moves unstructured data between parallel file systems, NAS, object storage, tape and cloud based on policies. Every transfer is filtered, scheduled, verified and recorded, and it can be triggered by webhooks from your own workflows. It works alongside NODEUM Data Archiving when data needs to be preserved for the long term.