AUTOMATED Data MOVEMENT BETWEEN COMPUTING NODES AND ON-PREMISE STORAGE

Today's Top Challenges

Today, supercomputing systems are so performant and scalable that the speed of data generation has never been so fast. Research centers have to store the generated content in “data repositories” that are located close to each other and that are well integrated.

Two different categories of data repositories are used as storage tiers:

• Active Data Repositories which provide the performance when data is written by supercomputing systems
• Archival Data Repositories with interfaces used in Cloud systems, which are more suitable for data sharing 

Research is becoming increasingly collaborative.  Sharing of data and FAIR data management is becoming mandatory. This adds to the requirements for modern data infrastructures.

The Ideal Solution

Research facilities need an automatic data mover engine which will manage the two different types of data repositories.

The ideal solution must provide the following features to users: 

• Organize the movement of the data from the Active to the Archival Data Repository
• Keep a direct access by the users to Active and Archival Data Repositories
• Integration with HPC workload managers like Slurm
• Provide a public API and SDK to facilitate integration with specific research applications

Federated cloud object stores are used at these locations ; in using standard Swift and S3 interfaces, they enable researchers to exchange their data. With the Data Mover, researchers can copy their data locally to the existing parallel file systems in order to process them on the fast HPC systems. 

In addition, researchers can copy data, which have been generated on these supercomputers to the cloud object storage in order to make them accessible to other researchers from all over the world.

SEE NODEUM IN ACTION

A platform which manages the storage of all files across all storage systems and where the date movement workflow is transparent for any users. These workflows allow data management between any type of storage systems and keep a direct control of the protected files.

It is the solution when your data volume and capacity is exploding, to unify capacity your management. The indexed central catalog provides any needed information on the files managed by the ecosystem :

1. Status : protected, archive, online, ...

2. Attribute : name, creation data, modification date, size, ....

3. Lifecycle in regards to its data mobility, and then the localization.

4. Customized Metadata

And the one-click movement capability allows any type of automations which ease the management and the decisions making.