Skip to content
Feb 28, 2022 4:24:56 PM

Nodeum 2.0.0-beta Release notes

Migration notes

 

Migration path

There is no migration path between Nodeum v1 and Nodeum v2.

 

How to upgrade ?

For automatic upgrade information, see How to update Nodeum

Pay attention to the warnings described above.

Limitations and warning

  • Retrocompatibility to CENTOS 7 or RHEL 7 is not available

Support removal notice

N/A

Deprecation notice

N/A

 

Release 2.0.0-beta1 - January 5th, 2022

Nodeum 2.0.0-beta1 is a major release.

 

About Nodeum 2.0

This new major version is a distributed systems architecture where each node will be a member of a Data Management cluster. The cluster will offer access to storage resources attached to a list of different storage nodes.

The solution will be seen as a single namespace on top for each deployed cluster. Multiple nodes are interconnected by a high-performance IP network. Even on long distances through reliable IP connections.

Each cluster node shares the global content catalog of the cluster to allow scalable data movement across each node.

Workflow

  • ND-730 Scalable Workflow Manager in using gRPC and ProtoBuffer intercommunication
  • ND-729 Scalable Metadata Management centralizing File and Object

Protocol & Storage Provider

  • ND-732 | ND-749 | ND-759 | ND-1273 | ND-1336 Design a Storage Manager Plugins Oriented
  • ND-1260 Redesign a new Posix File System Plugin handling NFS protocol
  • ND-1261 Redesign a new Posix File System Plugin handling SMB protocol
  • ND-1282 Resign a new Swift Plugin supporting Keystone authentication

Compatibility with Federated AAI Service

  • ND-68 End-User Authentication - authorization management to allow Data Management and allow access to OpenStack Swift in using a Federated IDP

Ansible-Deployment

  • Ansible playbooks compatible with : RHEL8 - CentOS Stream 8 - Rocky 8.4 - 8.5

 

Release 2.0.0-beta2 - May 13th, 2022

Nodeum 2.0.0-beta2 is a major release.

 

About Nodeum 2.0

This new major version is a distributed systems architecture where each node will be a member of a Data Management cluster. The cluster will offer access to storage resources attached to a list of different storage nodes.

The solution will be seen as a single namespace on top for each deployed cluster. Multiple nodes are interconnected by a high-performance IP network. Even on long distances through reliable IP connections.

Each cluster node shares the global content catalog of the cluster to allow scalable data movement across each node.

Bash Client

Protocol & Storage Provider

  • ND-1527 Nas protocol in Storage Plugins
  • ND-1259 Design S3 Connector in using GO S3 API - standard command
  • ND-1521 Solved issue - Error when mounting s3fs in the data mover
  • ND-1542 Design a GRPC bridge between dispatcher - mover - finalizer

Compatibility with Federated AAI Service

Workflow

  • ND-303 Metadata inject in files
  • ND-1707 S3 metadata definition
  • ND-1156 Implement a filtering option in the Task execution Process Items
  • ND-1237 X-Delete-At, X-Delete-After and X-Object-Meta-[name] to the destination object
  • ND-1307 User Experience - Specify metadata for object storage (bucket - container - object)
  • ND-1478 Solved Issue - Browsing manually mounted Primary Storage
  • ND-1550 Implement the task throughput speed calculation
  • ND-1551 implement the scheduler
  • ND-1553 Solved Issue - Pending item even if the file has been well copied
  • ND-1554 Solved Issue - Pending issue status even if file has been copied
  • ND-1557 Reimplement the task metadata associated to a task
  • ND-1563 Solved Issue - Stopped mover if the task cannot be finished
  • ND-1568 Solved Issue - Pending issue status for folder , even if folder has been copied
  • ND-1591 Change the status of task to stopped by system
  • ND-1686 Add // processing in scaling # mover

Ansible-Deployment

  • ND-1141 Review the template inventory

 

Release 2.0.0-beta3 - June 21th, 2022

Nodeum 2.0.0-beta3 is a major release.

Protocol & Storage Provider

  • ND-1754 Add ability to change swift connection parameter with trigger metadata (eg. project_id)
  • ND-1730 Add "nodeum s3 - swift - rclone" in the list of standard backend

Workflow

Compatibility with Federated AAI Service

Ansible-Deployment

  • ND-1763 Solved Issue - add missing scheduler service in Ansible inventory

Release 2.0.0-beta4 - June 22th, 2022

Nodeum 2.0.0-beta4 is a minor release.

Workflow

Hook plugin

  • ND-750 | ND-753 Design a plugin to handle Callback mechanism

 

Release 2.0.0-beta5 - August 4th, 2022

Nodeum 2.0.0-beta5 is a minor release.

Web

  • ND-1774 Solved Issue -Task Interface - GoMicro Error message
  • ND-1158 Implement a sort option in the Task execution Process Items
  • ND-1805 Add search options in Task Log
  • ND-1773 Provide only swift native connector as backend for OpenStack Swift

Workflow

  • ND-1662 Handle redundancy of scheduler service
  • ND-1752 Solved Issue - Review the move process - file erase at source
  • ND-1787 Solved Issue - Redis Error reading from Stream
  • ND-1771 Design a mechanism to assign specific port number to 3 main data mover services
  • ND-1782 Update the webservices to read tasks logs 
  • ND-1821 Solved Issue - Task metadata - missing the data migration task type

Hook plugin

  • ND-1776 Provide a python interpreter in the Callback feature

Monitoring Micro Services

  • ND-992 | ND 1813  Monitor the services heathiness and get it ready to restart
  • ND-1055 Manage the mover load state
  • ND-1586 | ND-1814 Review List of Monitoring page
  • ND-1786 Implement central logger
  • ND-1811 Record information in redis to restore task execution

Protocol & Storage Provider

  • ND-1383 Rclone compatible with Azure storage 

Ansible-Deployment

  • ND-1798 Ansible inventory update - microservice deployment configuration
  • ND-1797 Ansible inventory update - web authentication configuration (ldap, saml,/….)
  • ND-1799 Improve Ansible jwt configuration

 

Release 2.0.0-beta6 - September 30th, 2022

Nodeum 2.0.0-beta6 is a minor release.

Slurm Plugin

  • ND-765 | ND-1802 | ND-1803 | ND-1828 - Design first integration to handle token management

Workflow

  • ND-1868 Solved Issue - Too many files opened
  • ND-1806 Add storage destination & writing time/date in file process list
  • ND-1824 Improve broker mechanism in the event flow management
  • ND-1829 Integrate Kafka as message broker across MicroServices
  • ND-1830 Solved Issue - Message Broker partition balancing
  • ND-1836 Solved Issue - failed are in pending even if task finished
  • ND-1839 Improve the movement order queueing consumption
  • ND-1843 Solved Issue - Container is not seen when copying to non existing container
  • ND-1861 Ignore directory in the listing - parsing
  • ND-1863 Improve task report with additional status
  • ND-1867 Solved Issue - Kafka run out of available brokers
  • ND-1876 Improve time to finalize a movement

Ansible-Deployment

  • ND-1825 Define Interface Binding Mapping in the inventory

Compatibility with Federated AAI Service

  • Handle Token Refresh in LUA

Protocol & Storage Provider

  • ND-1845 Refresh Data Mover swift connection

     

Release 2.0.0-beta7 - October 28th, 2022

Nodeum 2.0.0-beta7 is a minor release.

Cluster Architecture

  • ND-1073 | ND-1128 | ND-1129 | ND-1838 Configure authentication in MongoDB
  • ND-394 Load Balancer Configuration on top of Nginx
  • ND-482 Scale RAILS WebServices across multiple servers
  • ND-1880 Scale ETCD across multiple servers
  • ND-1885 | ND-1886 Redis clustering configuration
  • ND-1887 | ND-1888 MariaDB clustering configuration

 

Workflow

  • ND-1877 Review the Kafka number of partition
  • ND-1891 Performance Improvement in reducing the call number
  • ND-1894 Split // option for finalizer and mover

 Ansible-Deployment

  • ND-1882 Mechanism to encrypt password

Release 2.0.0-beta8 - November 30th, 2022

Nodeum 2.0.0-beta8 is a minor release.

Slurm Workload Integration

  • ND-1917 Review LUA script to be compatible with defer task
  • ND-1915 Allow creation of a defer task
  • ND-1918 Enable autocleaning of defer task

Web

  • ND-1953 Add overwrite option in the Task Option

Workflow

  • ND-1874 Solved Issue - Data are copied twice in the data mover

Cluster Architecture

  • ND-1884 Support IPv6
  • ND-1912 Allow to disable http (port 80) and enable https (443) without modify standard confIguration

Protocol & Storage Provider

  • ND-1940 | ND-1941 Plugin for Isilon Storage

 


 

Jessica Delage

RELATED ARTICLES