Research data no longer lives in one place. It sits on a parallel file system next to the compute cluster, on NAS shares in the lab, in S3-compatible object storage, on tape and increasingly in the cloud. Moving it between those places is still often done by hand or by a collection of scripts. A data mover replaces that with software that moves files between storage systems automatically, according to rules you define, and proves that every file arrived intact.
What is a data mover?
A data mover is software that copies or moves files between different storage systems based on policies, then verifies and records every transfer. It is not a backup tool, which keeps copies for recovery, and it is not a one-off migration script. It is a permanent service that keeps the right data on the right storage, at the right time.
What a data mover does
- Selects data by policy: files are picked by rules such as age, size, path, owner or metadata, instead of by someone running a command.
- Moves data between any storage: parallel file systems, NAS, object storage and cloud, in either direction.
- Verifies every transfer: checksums confirm that what arrived is identical to what was sent.
- Runs on a schedule or on events: transfers start at set times, or when another system triggers them through an API or webhook.
- Keeps an audit trail: every file moved is recorded, so you can show what moved, when and where.
rsync, MPI tools or a data mover?
Most HPC and research teams already move data with one of two tool families: rsync, or MPI-based parallel copy tools such as mpiFileUtils. Both are good at what they were built for. The question is what happens when data movement stops being an occasional job and becomes a daily, cross-storage operation.
rsync
rsync is the default tool for copying and synchronising directories, locally or over SSH. It only transfers what changed, which makes it efficient over slow links. It was not designed for parallel file systems with millions of files: one rsync process walks the tree and copies files one stream at a time, and building the file list alone can take hours on large directories. Teams work around this by splitting the tree and running many rsync processes in parallel with wrappers such as GNU parallel. By default rsync compares size and modification time; full checksums are possible but slow on large trees.
MPI-based parallel copy tools
On HPC clusters, MPI-based tools such as mpiFileUtils (dcp to copy, dsync to synchronise, dcmp to compare) spread a copy across many processes on many nodes. For a one-off bulk copy between two parallel file systems mounted on the cluster, they are among the fastest options available. They run as batch jobs: someone with MPI and job-scheduler access has to size the job, launch it, check the output and compare the trees afterwards. They work on POSIX file systems mounted on the cluster nodes, not on object storage or cloud. They also consume the CPU, memory and network of the servers they run on, so a large copy competes with the computing jobs those nodes are there for. And they keep no log or history: once the job ends, there is no record of which files were copied, when, or by whom.
Data mover
A data mover keeps the parallelism but adds what the other two leave to scripts: rules that decide what moves and when, support for file, object, tape and cloud storage, automatic verification of every transfer, and a permanent record of what moved where.
| Capability | rsync | MPI tools (mpiFileUtils) | NODEUM Data Mover |
|---|---|---|---|
| Parallel transfers built in | No | Yes | Yes |
| Moves data to object storage (S3) | No | No | Yes |
| Moves data to cloud | No | No | Yes |
| Verifies every transfer automatically | No | No | Yes |
| Selects data by policy (age, size, path, metadata) | No | No | Yes |
| Schedules and event triggers (API, webhooks) | No | No | Yes |
| Log and history of every file moved | No | No | Yes |
| Leaves compute nodes free for computing | Depends | No | Yes |
| Usable without scripting or MPI skills | No | No | Yes |
| Best for | Small syncs over slow links | One-off bulk copies inside a cluster | Continuous movement across storage tiers |
These tools are complementary. rsync and MPI tools remain useful for ad-hoc copies. A data mover is what you add when the same movements happen every day, across more than one kind of storage, and someone needs to prove they happened.
Three signs you need a data mover
- You are replacing a storage system. Migrations of hundreds of terabytes need verification and restart points, not a weekend of copying.
- Researchers keep asking to get archived and retrieved data back. Recall from object storage should be a request driven by users, not a ticket that waits for the one person who knows the system.
- Your GPUs are waiting for data. AI and analysis workloads need datasets staged onto fast storage before the job starts, and moved off when it ends.
What to look for when you choose one
- Storage coverage: the file systems, object stores, tape libraries and clouds you use today and plan to use.
- Scale: performance with millions or billions of files, not only total capacity.
- Verification and audit: checksums on every transfer and a searchable history.
- Policy engine: rules that storage administrators can define and change without writing code.
- Integration: an API and webhooks, so pipelines and other tools can trigger and follow transfers.
How NODEUM approaches data movement
NODEUM Data Mover moves unstructured data between parallel file systems, NAS, object storage, tape and cloud based on policies. Every transfer is filtered, scheduled, verified and recorded, and it can be triggered by webhooks from your own workflows. It works alongside NODEUM Data Archiving when data needs to be preserved for the long term.