POLICY-BASED DATA ORCHESTRATION

Data Movement Workflows

System allows the creation of different data movement tasks. This can be defined per data set and per storage but also with a certain level of policy : Migration, Archiving (Active, Offline), Protection.

Extended Data Set Selector

An unlimited data set selector brings granularity in the data you have to select. Movement can be executed between any type of storage from storage where hot data are stored to cold storage.

Scheduling

The internal scheduler can execute automatically, manual or scheduled task.

Filtering

Two different types of filtering are available : basic and advanced. Advanced provides embedded scripting editor. 

OPTIMIZE TO FILE MANAGEMENT PROCESSING

The solution is designed to optimize the file data management processing for different types of files. Here are three types of concepts that help to understand:

• The system is designed to automate the processing order,
• the solution is designed around multi-threaded file batching technology. Each task movement is defined by 3 stages: preparation, execution, and finalization. Tasks run in // with a batch system creating jobs, each job contain a list of files, the number of files in each job is determine by a maximum number of files or a when the capacity of all files is reached.
• includes different queuing systems which allow to execute in // the different type of actions which must be performed during a data movement processing.

The benefit is to not lose time in discovery phase, files start to be copied once the first batch of data has been received. The discovery process continues to run during the copy

PRIORITY MANAGEMENT

A priority level feature is available to each task execution; this feature is linked together with the tape drive allocation.

The solution allows a task which a higher priority to pause another task currently copying file to a tape. This will eject the tape, free up the drive remount the tape requested with the new task; and start the treatment.

RESOURCE MANAGEMENT

The workflow manager system provides natively mechanism to prevent task movement overload.

• By default, there is a memory threshold set at 80%. That’s mean, if the memory is over 80%, it is not possible to execute additional task movement.

• There is the possibility to execute task to clean the disk cache layer,
• There is a workflow for rehydration (which means for us the defragmentation of tapes).

The solution verifies before transferring any file the Status of any storage (available or not).

Nodeum always provides a clean stop of the processing if there is any problem in the source or destination storage.

 Even the source can be stopped – not reachable - …. Or the same for the destination. The behaviour is that the task is tagged as “stop by system”, logs task is updated with the root cause, error code, … And even the file logging is updated with status of each file not well processed and again why a display of the reason.

MD5

MD5 hash algorithm is a commonly used function for validating data integrity. An MD5 checksum is a 32 digit hexadecimal number that represents the hash of the contents of a file.

The calculation of an MD5 is an industry standard so the integrity can be checked on any system.

XXHASH

It is an Extremely fast Hash algorithm, the hash is faster to be generated. It is highly portable, and hashes are identical across all platforms (little / big endian). This is more and more popular in the video device, Nodeum uses the xxHash64be algorithm which is compatible with other product and software in the industries