Object Storage for HPC: When to Tier to S3, and When Not To

Every HPC centre, whether it is a research centre, an AI factory or a private supercomputing facility, hits the same wall sooner or later: the parallel file system fills up faster than the budget for the next expansion. Object storage looks like the obvious answer. It scales to many petabytes, it is cheaper per terabyte than a flash or disk tier tuned for bandwidth, and every vendor now speaks S3.

But HPC object storage is not a drop-in replacement for a parallel file system. Tier the right data to it and you free expensive capacity without anyone noticing. Tier the wrong data and jobs slow down, scripts break and users start copying files back by hand. The model that works gives each tier a clear role: the parallel file system for scratch and hot data, S3 object storage for long-term datasets, and data moved between the two when a project needs it.

This article explains what object storage is good and bad at in an HPC environment, which data belongs there, which data does not, and the patterns that make tiering work in practice.

What object storage does well for HPC

Object storage stores data as objects in a flat namespace of buckets, each object addressed by a key and read or written whole through an HTTP API such as S3. That design gives it a few strengths that matter to HPC teams.

• Capacity that scales out. You add nodes or disk enclosures and the system grows, without re-architecting a metadata service or rebalancing a file system by hand.

• Lower cost per terabyte. Object systems usually run on dense, capacity-optimised hardware and do not need the low-latency networks and metadata servers a parallel file system needs.

• Durability through erasure coding. Instead of keeping full replicas, most object stores split each object into data and parity fragments spread over many disks or sites. You can lose several drives, or a whole site in a geo-distributed setup, and still read the data, with far less raw capacity overhead than triple replication.

• High aggregate throughput for large objects. Many clients reading or writing large objects in parallel can drive a lot of bandwidth, which suits bulk ingest, bulk recall and streaming large datasets.

• A standard API. S3 is understood by analysis tools, cloud services, software and most modern data pipelines, which makes data on object storage easier to share beyond the cluster.

Where object storage falls short

The same design choices create limits that HPC applications run into quickly.

• Metadata operations are slow and expensive. Listing a directory with millions of entries, calling stat on every file, or walking a deep tree is what parallel file systems are built for. On object storage, a "directory" is only a key prefix, and listing it means paging through API calls.

• Small files hurt. Every object carries per-request overhead. Millions of small files mean millions of PUT and GET requests, each with HTTP latency. Throughput collapses long before bandwidth is reached.

• No POSIX semantics. There is no in-place update, no append, no byte-range write, no file locking and no hard links. An object is written whole and replaced whole. Applications that open a file, seek and write a few bytes will not work against S3 without a translation layer.

• Rename is a copy. Renaming a "directory" means copying and deleting every object under its prefix. A rename that takes milliseconds on Lustre or GPFS can take hours on a large prefix.

• Latency, not just throughput. Object storage is good at moving large amounts of data, not at answering quickly. First-byte latency is typically much higher than on a parallel file system, which matters for interactive work and for jobs that open many files.

• Permissions work differently. POSIX users, groups and ACLs do not map one to one onto bucket policies and access keys, so access control needs to be thought through before data moves.

None of this makes object storage a bad tier. It makes it a tier for a specific kind of data.

Which data to tier to object storage

The data that fits object storage is large, finished and rarely touched. In most HPC environments that means:

1. Completed project datasets. When a project or grant ends, its inputs and results usually need to be kept, but nobody reads them every day. They are the clearest candidate for tiering.
2. Cold results and outputs. Simulation outputs, processed datasets and instrument data that have not been accessed for weeks or months, and that are kept for reference, reproducibility or reuse.
3. Reference datasets shared beyond the cluster. Data that other teams, partners or cloud services need to read through S3 is often more useful on object storage than locked behind a cluster mount.
4. Large, write-once files. Big files written once and read occasionally move efficiently and land as a small number of large objects.

Checkpoints are a special case. Most checkpoints are written, kept briefly in case a job fails, then deleted, and they should stay on fast storage for that short life. Only the few you deliberately keep, for example the final state of a long run, are worth tiering, and only once they are no longer needed for a restart.

When not to tier to S3

Keep data on the parallel file system when any of the following is true.

• It is scratch. Scratch space exists to give running jobs the fastest possible I/O. Moving it to object storage defeats the purpose, and most scratch data is temporary anyway.

• It is small-file-heavy and still active. Directories with millions of small files that jobs still read, such as some genomics, imaging or machine learning datasets, will suffer from request overhead and listing costs.

• Applications need POSIX. Codes that update files in place, append to logs, rely on locks or rename directories as part of their workflow need a real file system.

• It is part of an active pipeline. Data that will be read again in the next days or weeks costs more to move out and back than it saves.

• Nobody has agreed how users will get it back. Tiering without a recall path just moves the problem from storage cost to support tickets.

A quick decision table

Moving data to HPC object storage without breaking how people work

Deciding which data belongs on object storage is half the job. The other half is moving it, and bringing it back, without disrupting users.

• Decide, then move with precise criteria. The storage team, a project owner or a researcher decides what should move and when, for example a completed project, or outputs older than six months in a given directory. The move itself should then select files by clear criteria such as path, age, last access, size or owner, rather than by hand-made file lists that go stale.

• Keep a namespace users recognise. Users think in paths, not buckets. Object keys that mirror the original paths keep the data readable with standard S3 tools, and let a researcher find a file where they expect it.

• Make recall a request, not a ticket. Users or pipelines should be able to ask for data back, and get it onto the parallel file system, without waiting for the one administrator who knows the system. An API or a webhook lets a workflow manager request the data before a job starts.

• Pack small files when you must move them. If the data contains many small files, packing them into larger objects before transfer avoids paying request overhead millions of times. The trade-off is that recalling a single file means reading the pack.

• Verify every transfer and keep a record. A move that deletes the source must be sure the copy is intact. Checksums on every object, and an audit trail of what moved where and when, turn data movement from a risk into a routine operation.

• Plan the network path. Moving data off the cluster should not compete with compute traffic. Running transfers on dedicated data mover nodes keeps the compute nodes free for jobs.

How NODEUM Data Mover moves data to object storage

NODEUM Data Mover executes the data movements you decide on. You choose what moves, from where and to where: between parallel file systems, NAS, S3-compatible object storage and cloud.

You select the data by storage source or dataset, then filter it by path, age, size, owner or metadata. Each move runs when you launch it, at a time you schedule, or when another system calls it through an API or webhook. Every transfer is verified with checksums and recorded, so you can show what moved, when and where. Transfers run on NODEUM's own nodes rather than on your compute nodes.

The same engine moves data back. When a dataset is needed again, a user or a pipeline asks for it and Data Mover returns it to fast storage, without someone rebuilding the original path by hand. If the data also needs long-term preservation, NODEUM Data Archiving adds containers with placement policies over secondary pools, including tape.

Supported object storage targets include OpenStack Swift, Ceph and MinIO over S3, and the object storage platforms of the main vendors, such as DDN, IBM and others.

The model that works is simple: the parallel file system for scratch and hot data, S3 object storage for long-term datasets, and data moved between the two whenever a project needs it. To get there, start by measuring what is actually cold, then decide what moves before you buy the capacity.

See it on your own data

Want to see data moved to object storage, and back, on a setup like yours? Book a Data Mover demo and we will walk through selecting the data, the verification and the recall path with your storage team.