DATA MOVER

Our robust task management capabilities empower users to have complete control and oversight over their data movement workflows. Users can effortlessly create and manage tasks for seamlessly migrating data between different storage tiers. They have the ability to efficiently handle large volumes of data and automate their workflows. By implementing filters, users can ensure that only the most relevant files are included in their data movement tasks, which greatly enhances the transfer process and minimizes the risk of errors or data loss. Our highly customizable platform allows users to create tailored filters that align with their specific needs. This includes setting up filters for specific file types, folders, or directories, as well as creating custom rules based on metadata or other criteria.

SCHEDULER

The workflow Manager incorporates a scheduler that enables the processing of various tasks initiated from different inputs. Execution can be automated, manual, or scheduled as needed.

FILTERING

Users can effortlessly manage and organize their data movement workflows with the filtering module, which allows for easy file filtering based on specific criteria. Choose between basic and advanced filtering options, with the advanced option providing an embedded scripting editor for added flexibility. Defining filters to automatically include or exclude files is a breeze, and users have a range of criteria to choose from, including file size, creation date, modification date, file type, and more. For more complex filtering needs, users can define filters that combine multiple criteria.

PRIORITY MANAGEMENT

The priority management features in Nodeum ensure that higher-priority tasks do not monopolize all available resources, leaving other workflows waiting indefinitely.

Nodeum utilizes Quality of Service (QoS) techniques to prioritize data movement operations based on their priority level. This approach guarantees that critical demands receive the necessary resources while still providing fair access to non-critical demands. By assigning different priority levels to various tasks, Nodeum ensures that resources are allocated accordingly.

Additionally, Nodeum employs Fair Queuing techniques to distribute resources among different requests, ensuring that each request receives an equal share of the remaining resources. This method prevents any single data movement workflow from monopolizing all available resources, allowing all remaining workflows to have fair access.

In summary, the priority management and QoS techniques integrated into Nodeum guarantee efficient resource allocation and workflow management, eliminating resource contention and ensuring fair access to resources for all workflows.

RESOURCE MANAGEMENT

This mechanism prevents overloading of task movement by monitoring the storage usage before transferring any files. Moreover, Nodeum ensures a smooth halt of the processing in case of any issues with the source or destination storage. Even if the source or destination becomes unreachable, the task is marked as "stop by system" and the task logs are updated with the root cause, error code, and the file logging is updated with the status of each file that was not processed correctly, along with an explanation for the reason.

The solution is designed to optimize the file data management processing for different types of files.

Here are three types of concepts that help to understand:

• The solution streamlines the processing order,
• It utilizes a multi-threaded file batching technology where each task movement consists of three stages: preparation, execution, and finalization. Tasks are executed in parallel using a batch system that creates jobs. Each job contains a list of files, with the number of files in each job determined by a maximum limit or when the capacity of all files is reached.
• It includes specific queuing systems that enable the execution of different types of actions required during data movement processing.

The advantage is that time is not wasted in the discovery phase; files begin to be copied as soon as the first batch of data is received. The discovery process continues to run while the copying takes place.

Nodeum's hook feature enables users to execute custom scripts or commands during specific events within each Data Movement process. These scripts or commands can automate additional tasks, integrate with other systems, or perform specific actions based on the event.

For instance, users can configure a hook to run a custom script after a data movement task is completed. This script can then perform additional actions like sending email notifications, updating metadata databases, or triggering events.

Hooks can be created through the Nodeum Console, the ND Client, or the RESTful API interface. The hook configuration allows users to define the triggering event, the script or command to run, and any parameters or variables to pass to it.

By leveraging Nodeum hooks, users can significantly enhance the automation and integration capabilities of the solution, enabling them to customize workflows and extend the platform's functionality to meet their specific requirements.