Skip to content
Dec 21, 2022 9:44:16 AM

Nodeum 2.0 - Release notes

About Nodeum 2.0

This new major version is a distributed systems architecture where each node will be a member of a Data Management cluster. The cluster will offer access to storage resources attached to a list of different storage nodes.

The solution will be seen as a single namespace on top for each deployed cluster. Multiple nodes are interconnected by a high-performance IP network. Even on long distances through reliable IP connections.

Each cluster node shares the global content catalog of the cluster to allow scalable data movement across each storage

This enhanced data management solution is designed to be faster, easier and more reliable with elastic scale-out architecture and integration to the most common end-user applications. This leads movement across posix file system and object storage (S3 and Swift). It ensures preservation of data accessibility between file level user management base and token based federation.

With this latest release, Nodeum end users have also more integration options in using hooks to execute some callback code before or after a data movement. Additionally, customers in HPC can now get benefit of Nodeum Data Movement with the SLURM workload manager.

In addition to its SDK and the Console interface, a new ND client is also available. This allows from a Linux, macOS, Windows operating system to interface with Nodeum, execute data movement tasks, monitor the status of the services and its running tasks.

 

Migration notes

 

Migration path

There is no migration path between Nodeum v1 and Nodeum v2.

 

How to upgrade ?

For automatic upgrade information, see How to deploy Nodeum Data Mover

Pay attention to the warnings described above.

Limitations and warning

  • Retrocompatibility to CENTOS 7 or RHEL 7 is not available

Support removal notice

N/A

Deprecation notice

N/A

 

Release 2.0.0 - December 16th, 2022

Nodeum 2.0.0 is a major release.

Workflow

  • ND-730 Scalable Workflow Manager in using gRPC and ProtoBuffer intercommunication
  • ND-729 Scalable Metadata Management centralizing File and Object
  • ND-303 Metadata inject in files
  • ND-1707 S3 metadata definition
  • ND-1156 Implement a filtering option in the Task execution Process Items
  • ND-1237 X-Delete-At, X-Delete-After and X-Object-Meta-[name] to the destination object
  • ND-1307 User Experience - Specify metadata for object storage (bucket - container - object)
  • ND-1478 Solved Issue - Browsing manually mounted Primary Storage
  • ND-1550 Implement the task throughput speed calculation
  • ND-1551 implement the scheduler
  • ND-1553 Solved Issue - Pending item even if the file has been well copied
  • ND-1554 Solved Issue - Pending issue status even if file has been copied
  • ND-1557 Reimplement the task metadata associated to a task
  • ND-1563 Solved Issue - Stopped mover if the task cannot be finished
  • ND-1568 Solved Issue - Pending issue status for folder , even if folder has been copied
  • ND-1591 Change the status of task to stopped by system
  • ND-1686 Add // processing in scaling # mover
  • ND-1552 | ND-1766 Priority Management
  • ND-1583 Filtering plugins in Nodejs
  • ND-1742 Filtering Option in Dispatcher
  • ND-1748 Solved Issue - Waiting in finalizing with high number of // mover
  • ND-1753 Solved Issue - Move Task are not seen in the task listing
  • ND-1775 Solved Issue - Task do not start - stay in queue
  • ND-1779 Solved Issue - no file listed
  • ND-1780 Solved Issue - callback error
  • ND-1662 Handle redundancy of scheduler service
  • ND-1752 Solved Issue - Review the move process - file erase at source
  • ND-1787 Solved Issue - Redis Error reading from Stream
  • ND-1771 Design a mechanism to assign specific port number to 3 main data mover services
  • ND-1782 Update the webservices to read tasks logs 
  • ND-1821 Solved Issue - Task metadata - missing the data migration task type
  • ND-1868 Solved Issue - Too many files opened
  • ND-1806 Add storage destination & writing time/date in file process list
  • ND-1824 Improve broker mechanism in the event flow management
  • ND-1829 Integrate Kafka as message broker across MicroServices
  • ND-1830 Solved Issue - Message Broker partition balancing
  • ND-1836 Solved Issue - failed are in pending even if task finished
  • ND-1839 Improve the movement order queueing consumption
  • ND-1843 Solved Issue - Container is not seen when copying to non existing container
  • ND-1861 Ignore directory in the listing - parsing
  • ND-1863 Improve task report with additional status
  • ND-1867 Solved Issue - Kafka run out of available brokers
  • ND-1876 Improve time to finalize a movement
  • ND-1877 Review the Kafka number of partition
  • ND-1891 Performance Improvement in reducing the call number
  • ND-1894 Split // option for finalizer and mover
  • ND-1874 Solved Issue - Data are copied twice in the data mover

Protocol & Storage Provider

  • ND-732 | ND-749 | ND-759 | ND-1273 | ND-1336 Design a Storage Manager Plugins Oriented
  • ND-1260 Redesign a new Posix File System Plugin handling NFS protocol
  • ND-1261 Redesign a new Posix File System Plugin handling SMB protocol
  • ND-1282 Resign a new Swift Plugin supporting Keystone authentication
  • ND-1527 Nas protocol in Storage Plugins
  • ND-1259 Design S3 Connector in using GO S3 API - standard command
  • ND-1521 Solved issue - Error when mounting s3fs in the data mover
  • ND-1542 Design a GRPC bridge between dispatcher - mover - finalizer
  • ND-1754 Add ability to change swift connection parameter with trigger metadata (eg. project_id)
  • ND-1730 Add "nodeum s3 - swift - rclone" in the list of standard backend
  • ND-1383 Rclone compatible with Azure storage 
  • ND-1845 Refresh Data Mover swift connection
  • ND-1940 | ND-1941 Plugin for Isilon Storage

Compatibility with Federated AAI Service

Ansible-Deployment

  • Ansible playbooks compatible with : RHEL8 - CentOS Stream 8 - Rocky 8.4 - 8.5 - 8.6 - 8.7
  • ND-1141 Review the template inventory
  • ND-1763 Solved Issue - add missing scheduler service in Ansible inventory
  • ND-1798 Ansible inventory update - microservice deployment configuration
  • ND-1797 Ansible inventory update - web authentication configuration (ldap, saml,/….)
  • ND-1799 Improve Ansible jwt configuration
  • ND-1825 Define Interface Binding Mapping in the inventory
  • ND-1882 Mechanism to encrypt password

ND Client

Hook plugin

  • ND-750 | ND-753 Design a plugin to handle Callback mechanism
  • ND-1776 Provide a python interpreter in the Callback feature

Web

  • ND-1774 Solved Issue -Task Interface - GoMicro Error message
  • ND-1158 Implement a sort option in the Task execution Process Items
  • ND-1805 Add search options in Task Log
  • ND-1773 Provide only swift native connector as backend for OpenStack Swift
  • ND-1953 Add overwrite option in the Task Option

Monitoring Micro Services

  • ND-992 | ND 1813  Monitor the services healthiness and get it ready to restart
  • ND-1055 Manage the mover load state
  • ND-1586 | ND-1814 Review List of Monitoring page
  • ND-1786 Implement central logger
  • ND-1811 Record information in redis to restore task execution

Slurm Workload Integration

  • ND-765 | ND-1802 | ND-1803 | ND-1828 - Design first integration to handle token management
  • ND-1917 Review LUA script to be compatible with defer task
  • ND-1915 Allow creation of a defer task
  • ND-1918 Enable autocleaning of defer task
  • ND-1924 Implement Status Slurm method in LUA script

Cluster Architecture

  • ND-1073 | ND-1128 | ND-1129 | ND-1838 Configure authentication in MongoDB
  • ND-394 Load Balancer Configuration on top of Nginx
  • ND-482 Scale RAILS Web Services across multiple servers
  • ND-1880 Scale ETCD across multiple servers
  • ND-1885 | ND-1886 Redis clustering configuration
  • ND-1887 | ND-1888 MariaDB clustering configuration
  • ND-1884 Support IPv6
  • ND-1912 Allow to disable http (port 80) and enable https (443) without modify standard configuration

 

Release 2.0.1 - December 16th, 2022

Nodeum 2.0.1 is a minor release.

  • ND-1989 Solved Issue - ngnix ansible ipv6 correction

 

Release 2.0.2 - February 22th, 2023

Nodeum 2.0.2 is a minor release.

Workflow

  • ND-2049 Improve the mount management of NFS -SMB Storage

Log system

  • ND-1585 |ND-1804 | ND-1975 | ND-2035 | ND-2038 | ND-2044 Log exportation Feature

Monitoring Micro Services

  • ND-2004 Update the system information page
  • ND-2022 Add host collections 

LDAP Plugin

  • ND-2050 Improvement to facilitate the dn retrieval

 

Release 2.0.3 - March 16th, 2023

Nodeum 2.0.3 is a minor release.

Console

  • ND-2092 | ND-2097 | ND-2014 | ND-2108 Improvement Console

Metadata

  • ND-2055 Review and organize DB collection in regards to file location management

Release 2.0.4 - July 12th, 2023

Nodeum 2.0.4 is a minor release.

Console

  • ND-2164 Solved Issue - Improve layout of long file/folder path and pool name
  • ND-2172 Solved Issue - report error message when license is wrong

Storage Provider

  • ND-1944 Support S3 Lyve Cloud
  • ND-2143 Solved Issue; ldap-user-mapper plugin doesn't decode properly the ldap information
  • ND-2173 Review the possibility to support multiple lua burst buffer plugins

Monitoring

  • Integration with Promotheus and Grafana

Log system

  • ND-1809 | ND-2127 Integration of Nodeum Event Logs to Grafana Loki
  • ND-2146 Retrieve the plugin logs and forward it back to related service
  • ND-2151 Solved Issue - problem of config - log level in the log page

Ansible-Deployment

  • ND-2133 Improve the LDAP definition into Ansible Group Vars - Options inventory
  • ND-2136 Add in the inventory to option to specify a nodename for webserver
  • ND-2162 vault encryption mechanism

Release 2.0.5 - July 17th, 2023

Nodeum 2.0.5 is a minor release.

Ansible-Deployment

  • ND-2198 | ND-2199 | ND-2200 | ND-2201 Improvement 

 

Release 2.0.6 - October 11th, 2023

Nodeum 2.0.6 is a minor release.

Console

  • ND-1538 Adapt Console to keep UID/GID context for source browsing
  • ND-1664 Add schedule information : missed_count, skipped_count, missed_last and missed_first

Workflow

  • #163 Checksum calculation during the copy process - CRC32
  • ND-1555 Checksum calculation during the copy process - md5
  • ND-1556 Checksum calculation during the copy process - xxhash
  • ND-2213 Solved Issue - problem to copy single file

Storage Provider

  • #4 Create a specific plugin to retrieve UID and all GID from user into a local LDAP
  • #156 Create a specific plugin to retrieve UID and all GID from user into a local LDAP

Release 2.0.7 - December 5th, 2023

Nodeum 2.0.7 is a minor release.

Docker-Deployment

  • #21 | #164 | #165 | #166 #170 | #171 | #181 Provide Docker images and Docker Compose file.

Hook plugin

  • ND-2119 Establish condition in hook to verify the task status before execution

Workflow

  • ND-2152 Solved Issue - run a previously executed task
  • #180 Remove running tasks from task history

Jessica Delage

RELATED ARTICLES