Sooner or later every organization running HPC infrastructure has to place a lot of data somewhere else. Data that has cooled down should sit on object storage, a project's results need to be shared, or a new tier arrives and data must be placed on it.
This is not a migration. The systems stay; the data is placed where it belongs while researchers keep working. At petabyte scale, that HPC data movement is not a copy job either. This article walks through a phased plan for moving petabytes from a parallel file system such as Lustre or IBM Storage Scale (GPFS) to S3-compatible object storage without downtime.
Copying a few terabytes is a weekend task. Copying petabytes is not just the same task, longer.
Three things change at this scale.
First, the number of files matters more than the capacity. A petabyte made of large simulation outputs behaves very differently from a petabyte made of hundreds of millions of small instrument files. Scanning, comparing and verifying are all per-file operations, and on most systems the metadata work, not the bandwidth, sets the pace.
Second, the source stays in production. Researchers keep reading and writing while the copy runs, so the data you copied on day one is already out of date on day ten. The plan must assume a moving target.
Third, the target is not a file system. Object storage has keys instead of paths, no hard links, no POSIX permissions in the same form, and different behaviour for renames and small objects. Moving data from a parallel file system to object storage is a translation, not just a copy.
The classic advice is to spend weeks scanning the source before moving a single file. At petabyte scale that inventory is out of date before it is finished, and it delays the very work it was meant to plan. With NODEUM Data Mover you start from your decisions, not from a report: define the parameters once and let the first transfers produce the numbers.
Then start. Rules are re-applied on every pass, so the same parameters drive the bulk copy, the incremental syncs and the final pass. Each task reports what it processed, how much and what failed, so you learn the real file counts and throughput of your own data from the transfers themselves, not from a separate scanning project.
The first pass copies everything the rules select while users keep working. It is the longest phase and it should be designed to be interrupted.
Do not wait for a perfect estimate before you start. Run the first tasks, read the file counts and throughput Data Mover reports, and extend the plan from those real numbers rather than from the theoretical bandwidth of the network.
In practice: the Fenix research infrastructure. Fenix federates the supercomputing and data services of major European HPC centres, including JSC in Germany, BSC in Spain, CEA in France, CINECA in Italy and CSCS in Switzerland. Its researchers work across two storage worlds: fast parallel file systems based on GPFS or Lustre for computing, and federated object stores with a Swift interface for sharing data.
In 2021 the Jülich Supercomputing Centre, on behalf of the Fenix consortium, commissioned NODEUM and its partner Akquinet to extend Data Mover into a data mover service for the Fenix infrastructure. Researchers use it to stage data from the object stores onto a local parallel file system before processing, and to push results back to object storage so other researchers can use them, with the federation's authentication and authorisation rules applied to every transfer.
After the bulk copy, run repeated incremental passes. Each pass finds files that are new or changed since the last one and copies only those. Because each pass has less to do, each one is shorter than the previous one. Repeat until source and target match. There is no final freeze to plan around.
Track deletions as well as changes. If a user deletes a file on the source between passes, decide whether the target should delete it too or keep it, and apply that rule consistently.
A move is only finished when you can prove the data arrived intact. Size and modification time are not proof.
Verification also has a cost at petabyte scale, because every byte is read a second time. Build it into the transfer itself rather than running it as a separate project at the end.
This is where data movement differs from a migration. A migration ends in a cutover: a weekend when the source is frozen, a last sync runs and users switch over, followed by weeks of keeping the old system alive in case you need to roll back.
With NODEUM Data Mover there is no such moment. Data is not migrated, it is placed: each file goes to the location the rules define, and once it is transferred and its checksum verified, it lives there and the move is done for that file. Projects move one after another while everyone keeps working, so there is no big-bang date to negotiate with users and no old system to keep as a safety net.
What remains is the record. Data Mover logs what moved, when, from where, to where and with which checksum, so you can answer the questions that always come later: where did this dataset go, and was it complete. For data under retention rules, that record is not optional.
| Phase | Goal | Users affected? | Done when |
|---|---|---|---|
| 1. Define and go | Set source, target, rules, schedule and verification, then start | No | First tasks running and reporting real numbers |
| 2. Bulk copy | Copy selected data while the source stays live | No (throttled) | All tasks completed once |
| 3. Incremental syncs | Close the gap between source and target | No | Source and target match |
| 4. Verify | Prove every file arrived intact | No | Checksums match for every file |
NODEUM Data Mover was built for this kind of long-running, cross-storage operation. It moves unstructured data between parallel file systems, NAS, S3-compatible object storage, tape and cloud, based on policies you define by path, age, size, owner or metadata. Transfers run in parallel and are scheduled, so you can run the bulk copy at full speed at night and throttle it during the day. Every transfer is verified with checksums and recorded, which gives you the audit trail for each file. Webhooks and an API let your own workflows trigger transfers and follow their progress.
The same policies that place the data keep working once the move is done. Data that cools down keeps moving to object storage, and users can recall what they need. For data that must be kept for the long term, NODEUM Data Archiving adds containers with placement policies over secondary pools of type active archive, including tape.
If you need to place data from a parallel file system on object storage, we can define the parameters with you and show you how the phases above run in practice. Book a Data Mover demo with the NODEUM team.