Nodeum Client

The Nodeum Client nd command line tool provides a modern set of commands to execute data movement operations with Nodeum. The nd commandline tool is built for compatibility with the Nodeum v2 for expected functionality and behavior.

nd has the following syntax:

nd [GLOBALFLAGS] COMMAND --help

See Command Quick Reference for a list of supported commands.

Copyright

nd is a property of Nodeum and its subsidaries, if any.
The intellectual and technical concepts contained herein are proprietary to Nodeum
and its subsidaries and may be covered by Belgium and Foreign Patents, patents in process,
and are protected by trade secret or copyright law.
Dissemination of this information or reproduction of this material is strictly forbidden unless
prior written permission is obtained from Nodeum.

Related Version: 2.0.6

Quickstart

Install nd

Install the nd command line tool onto the host machine. Go in the section that corresponds to the host machine operating system or environment:

Linux

The following commands add a temporary extension to your system PATH for running the nd bash client.
Defer to your operating system instructions for making permanent modifications to your system PATH.

Alternatively, execute nd by navigating to the parent folder and running ./nd --help

curl https://get.nodeum.io/public/nd-2.0.6-linux-amd64 \
  --create-dirs \
  -o $HOME/nodeum-binaries/nd

chmod +x $HOME/nodeum-binaries/nd
export PATH=$PATH:$HOME/nodeum-binaries/

nd --help

macOS

curl https://get.nodeum.io/public/nd-2.0.6-darwin-amd64 \
  --create-dirs \
  -o $HOME/nodeum-binaries/nd

chmod +x $HOME/nodeum-binaries/nd
export PATH=$PATH:$HOME/nodeum-binaries/

nd --help

Windows

Open the following file in a browser:

https://get.nodeum.io/public/nd-2.0.6-windows-amd64

Once download, rename the file to nd.exe and execute the file by double clicking on it, or by running the following in the command prompt or powershell:

\path\to\nd.exe --help

ND client installation (RPM example)

# wget https://get.nodeum.io/public/nd-2.0.6-x86_64.rpm
# dnf install ./nd-2.0.6-x86_64.rpm

1. Access to the interface

[[email protected] ~]$ nd
NAME:
   nd - Nodeum CLI

USAGE:
   nd [global options] command [command options] [arguments...]

VERSION:
   2.0.6

COMMANDS:
   admin
   config    configure the Nodeum Client
   copy, cp  create copy task
   move, mv  create move task
   task
   help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --json                          output as JSON (default: false)
   --config value                  path to configuration file (default: <config-dir>/config.json) [$ND_CONFIG]
   --config-dir value, -C value    path to configuration folder (default: "/home/nodeum01/.config/.nd") [$ND_CONFIG_DIR]
   --alias value                   alias in configuration file for authentication (default: "default") [$ND_ALIAS]
   --url value                     URL of Nodeum [$ND_URL]
   --access-token value            for API authentication (1st authentication method) [$ND_ACCESS_TOKEN]
   --refresh-token value           for API authentication (1st authentication method, not saved in config) [$ND_REFRESH_TOKEN]
   --authorization-endpoint value  for Device Authorization Flow (2nd authentication method)
   --token-endpoint value          for Device Authorization Flow (2nd authentication method)
   --client-id value               for Device Authorization Flow (2nd authentication method)
   --scopes value                  for Device Authorization Flow (2nd authentication method)
   --persist-session               persist Device Authorization session on disk for 1 hour (default: true)
   --persist-session-renew         if persist session is enabled, renew the token (default: false)
   --username value                for API authentication (3rd authentication method) [$ND_USERNAME]
   --password value                for API authentication (3rd authentication method) [$ND_PASSWORD]
   --anonymous                     no login (default: false)
   --help, -h                      show help (default: false)
   --version, -v                   print the version (default: false)

2. Command Quick Reference

The following table lists nd commands:

Name Shortcut Description
admin   Access the administration commands
config   Configure the Nodeum client
copy cp Create a task to copy data between two storages
move mv Create a task to move data between two storages
task   List detailed information about created task
help h Displays a summary of command usage and parameters on the terminal

Note: the nd client provides a Bash completion mechanism to facilitate the search of commands.

Parameters

Global Parameters

Name Shortcut Description Default
–json   Output as JSON false
–config value -C value Path to configuration file /home/nodeum01/.config/.nd/config.json
–config-dir value -C value Path to configuration folder (default: “/home/nodeum01/.config/.nd”)  
–alias value   Alias in configuration file for authentication default
–url value   URL of Nodeum  
–access-token value   for API authentication (1st authentication method)  
–refresh-token value   for API authentication (1st authentication method, not saved in config)  
–authorization-endpoint value   for Device Authorization Flow (2nd authentication method)  
–token-endpoint value   for Device Authorization Flow (2nd authentication method)  
–client-id value   for Device Authorization Flow (2nd authentication method)  
–scopes value   for Device Authorization Flow (2nd authentication method)  
–persist-session   persist Device Authorization session on disk for 1 hour true
–persist-session-renew   if persist session is enabled, renew the token false
–username value   for API authentication (3rd authentication method)  
–password value   for API authentication (3rd authentication method)  
–anonymous   no login false
–help -h show help false
–version -v print the version false

Mandatory Parameters for OpenID integration

Parameters are available for each data movement task.

Name Description Value
–md project_name=<value> Name of project defined in OpenStack string
–md project_domain_name=<value> Name of the project’s domain defined in OpenStack string
–md user_domain_name=<value> Name of the project’s user defined in OpenStack string
–md region_name=<value> Name of the region’s project defined in OpenStack string

Standard Parameters

Parameters are available for each data movement task.

Name Shortcut Description Value Default value
–help -h Show help   false
–no-run   Create a task and don’t launch the task directly   false
–name value -n Name of task string automatically generated
–comment value   Comment of task string empty
–overwrite value   Overwrite all identical files already stored at destination true - false false
–priority value   Priority of the task, between 0 and 9 (0 is the highest priority) 0 - 9 0
–recursive -R Execute a recursive copy of the folder. If subfolders are present, the service will also copy the contents of each subfolder   false
–working-dir value –wd Defines the workpath to be kept at destination . - … - path 0
–ignore-hidden value   Task will not handle hidden file(s) true - false false
–progress value   Display live progress when running the task true - false true
–processed-nodes value   Display the processed nodes when running a task when the flag–progress is set none, error, all error

Advanced Parameters

Parameters are available for each data movement task.

Name Shortcut Description Value Default value
–context-uid value –uid Define the User ID which will handle the movement integer unset
–context-gid value –gid Define the Group ID which will handle the movement integer unset
–defer   when requesting the run of the task, will defer it for later with an unique ID   false
–parallel value   Define the number of mover which will handle the movement. Maximum value is determined by the deployed implementation. 1-20 1
–callback type   Add callback. Format is type:./path/to/file ./path/to/file  
–trigger-md key=value –md key=value Set metadata on the trigger. Format is key=value. Accepts multiple inputs key=value  
–task-md key=value   Set metadata on the task. Format is key=value. Accepts multiple inputs key=value  
–files-md key=value   Set metadata on the files. Format is key=value. Accepts multiple inputs key=value  

3. Configuration

nd uses a JSON formatted configuration file used for storing certain kinds of information, such as the authentication and authorization options.

For Linux and macOS, the default configuration file location is .config/.nd/config.json which is store in the $HOME. For Windows, the configuration file is stored in $AppData$.

You can display the configuration file location in using the command `nd --help´:

[[email protected] ~]$ nd --help
NAME:
   nd - Nodeum CLI
....
   --config value                  path to configuration file (default: <config-dir>/config.json) [$ND_CONFIG]
   --config-dir value, -C value    path to configuration folder (default: "/home/nodeum01/.config/.nd") [$ND_CONFIG_DIR]

You can use the --config value where value is the path to a JSON formatted configuration file that nd uses for storing data.
The ND_CONFIG environment variable can be used to set the value.

SSL

nd client allows SSL configuration to communicate with the Data Mover service which listen in HTTPS.
It is required to add the public certificate generated with the server.

The public certificate file has to be stored in this config-dir folder: .config/.nd/certs/CAs/.

The certificates on the server side have been generated following this command:

openssl11 req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
 -keyout ./nodeum_certs/private.key \
 -out ./nodeum_certs/public.crt \
 -subj "/CN=nodeum-qualif.mt-c.local" \
 -addext "subjectAltName=DNS:nodeum.domain.local,DNS:localhost,IP:127.0.0.1,IP:1.1.1.1"

Where 1.1.1.1 is the ip of the nginx interface and nodeum.domain.local is its hostname including domain name.

Authentication and Authorization

Description

Two authentifcation options are available:

  • Username / Password
  • IDP with OpenID

Command

Command Description
nd config save this command save the nd configuration for authentication
nd config review this command display the information related to the credentials

Username / Password configuration

The nd client provides a basic method of authentication in using Username / Password credentials.

[[email protected]]$ nd config save \
  --url=http://<url> \
  --username=<username> \
  --password=<password>
Option Description
--url this refer to the node which hosts the following service DATA MANAGEMENT WEB SERVICES
--username this is the username to grant authorization to the service
--password this is the password associated to the username that will grant the access to the service

OpenID configuration

The nd client provides also an OpenID authentication mechanism. In this case, nd Client has to be configured with the approriated IDP to handle proper token management.

The --persist-session-renew option can be defined to true to force a request token process any time the user has to interfact with the nd client.

[[email protected]]$ nd config save \
  --url=http://<url> \
  --authorization-endpoint=<authorization endpoint> \
  --token-endpoint=<token endpoint> \
  --client-id=<client id> \
  --scopes="openid,profile,offline_access,email"
Option Description
--url this refer to the node which hosts the following service DATA MANAGEMENT WEB SERVICES
--authorization-endpoint this is the endpoint url to grant authorization to the service
--token-endpoint this url is used to programmatically request tokens
--client-id this refer to the client identifier which is provided by the OpenID provider

Alias & default flags

Description

Alias & default flags allow structuration of different group of settings. Different alias can be defined in the configuration file.

Definition of alias & default flags

Alias and flags are declared in the configuration file ~/.config/.nd/config.json".
Default flags are defined for each available parameter command.
Flags can be overwritted in the nd command.

Example in config file:

{
  // ...
  // default: name of alias used by default
  "default": {          
   // ...
   "default_flags": {
    // this is the name of the command, example: nd copy
    "copy": {
       // this is the flag, example: nd copy --md project_domain_name=MYPROJECTDOMAIN --md user_domain_name=MYUSERDOMAIN
      "md": ["project_domain_name=MYPROJECTDOMAIN", "user_domain_name=MYUSERDOMAIN"]
    }
   }
  }
  // myorganisation: name of another alias
  "myorganisation": {  
     // ...
  }
}

Alias usage

The nd command allows the usage of alias, example : nd --alias myorganisation copy

4. Data Mover Service Status

Command

Command Description
nd admin status retrieve the status and the health of each services part of the cluster
nd admin logs retrieve all Logs and return their contents

Data Management Monitoring Services

nd admin status command requests the Data Management Monitoring service to retrieve the status and health of each services part of the cluster.
This command returns list of services. The following information are displayed:

  • Service Status
  • Service version
  • Host where the service is deployed
  • Its uptime
  • Consumed memory

Output where all services are reachable

╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│   │ NAME              │ VERSION                    │ ADDRESS         │ UPTIME     │ MEMORY    │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast          │ 10.2.2.83:50050 │ N/A        │ N/A       │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s  │ 34.92 MB  │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB  │
│ ● │ nodeum.finalizer  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50030 │ 171h6m21s  │ 25.39 MB  │
│ ● │ nodeum.finalizer  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50030 │ 171h8m40s  │ 24.34 MB  │
│ ● │ nodeum.front      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A        │ N/A       │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s  │ 17.20 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s  │ 25.23 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s  │ 25.63 MB  │
│ ● │ nodeum.scheduler  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s  │ 6.91 MB   │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.83:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.84:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.85:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mysql             │ 10.6.11-MariaDB            │ 10.2.2.84:3306  │ 150h21m0s  │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.83:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.84:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.85:26379 │ 150h16m52s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.83:6379  │ 285h37m50s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.84:6379  │ 169h59m59s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.85:6379  │ 169h59m59s │ N/A       │
│ ● │ kafka             │ 3.2.0                      │ 10.2.2.84:9092  │ N/A        │ N/A       │
│ ● │ kafka             │ 3.2.0                      │ 10.2.2.83:9092  │ N/A        │ N/A       │
│ ● │ kafka             │ 3.2.0                      │ 10.2.2.85:9092  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.83:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.84:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.85:2379  │ N/A        │ N/A       │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯

Output where some services are not available

╭───┬───────────────────┬────────────────────────────┬─────────────────┬────────────┬───────────╮
│   │ NAME              │ VERSION                    │ ADDRESS         │ UPTIME     │ MEMORY    │
├───┼───────────────────┼────────────────────────────┼─────────────────┼────────────┼───────────┤
│ ● │ micro.http.broker │ ff.http.broadcast          │ 10.2.2.83:50050 │ N/A        │ N/A       │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50010 │ 172h11m8s  │ 34.92 MB  │
│ ● │ nodeum.dispatcher │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50010 │ 172h13m32s │ 22.61 MB  │
│ ● │ nodeum.finalizer  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50030 │ 171h6m21s  │ 25.39 MB  │
│ ● │ nodeum.finalizer  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50030 │ 171h8m40s  │ 24.34 MB  │
│ ● │ nodeum.front      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50000 │ N/A        │ N/A       │
│ ● │ nodeum.monitoring │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50060 │ 171h2m47s  │ 17.20 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.84:50020 │ 172h4m48s  │ 25.23 MB  │
│ ● │ nodeum.mover      │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.85:50020 │ 172h4m48s  │ 25.63 MB  │
│ ● │ nodeum.scheduler  │ v2.0.0-beta7-22-g9ffff06ba │ 10.2.2.83:50040 │ 172h18m6s  │ 6.91 MB   │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.83:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.84:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mongo             │ 4.4.17                     │ 10.2.2.85:27017 │ 268h0m44s  │ 212.86 MB │
│ ● │ mysql             │ 10.6.11-MariaDB            │ 10.2.2.84:3306  │ 150h21m0s  │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.83:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.84:26379 │ 150h16m52s │ N/A       │
│ ● │ redis-sentinel    │ 6.2.6                      │ 10.2.2.85:26379 │ 150h16m52s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.83:6379  │ 285h37m50s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.84:6379  │ 169h59m59s │ N/A       │
│ ● │ redis             │ 6.2.6                      │ 10.2.2.85:6379  │ 169h59m59s │ N/A       │
│ ● │ kafka             │ 3.2.0                      │ 10.2.2.84:9092  │ N/A        │ N/A       │
│ ● │ kafka             │ 3.2.0                      │ 10.2.2.83:9092  │ N/A        │ N/A       │
│ ● │ kafka             │ 3.2.0                      │ 10.2.2.85:9092  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.83:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.84:2379  │ N/A        │ N/A       │
│ ● │ etcd              │ 3.4.13                     │ 10.2.2.85:2379  │ N/A        │ N/A       │
│ ○ │ nodeum.mover      │                            │                 │            │           │
│ ○ │ nodeum.finalizer  │                            │                 │            │           │
╰───┴───────────────────┴────────────────────────────┴─────────────────┴────────────┴───────────╯

Data Management Log Management Services

nd admin logs command requests the Data Management Log Management Services to retrieve all Logs and return their contents.

[[email protected] ~]$ nd admin logs --help
NAME:
   nd admin logs - display application logs

USAGE:
   nd admin logs [command options] [arguments...]

OPTIONS:
   --since value, -S value                          show logs not older than the specified date
   --until value, -U value                          show logs not newer than the specified date
   --tag value, -u value [ --tag value, -u value ]  filter by tags
   --help, -h                                       show help (default: false)

This command allows different parameters:

Option Description
since to only show logs not older than a specified date
until to only show logs not newer than a specified dat
tag to filter the logs per type of services. Example nodeum.monitoring

Export

The Logs can be exported in using the standard OS mechanism of exportation

./nd admin logs > 'nodeum_site-name_log.txt'

Where nodeum_site-name_log.txt is the file name and site represents the name of the site.

5. Data Mover Task Management

Follow these steps to create a task:

  1. Use Mandatory Parameters
  2. Define the data source
  3. Define the destination
  4. Apply others parameters if needed

Working Directory Explanation

The definition of a working directory allows to define where the files will be stored at the destination. Different options are available, they are described in the following definition.

With --wd=.

Source Destination Result
nod://source/folder/FILE.txt nod://dest/directory/ nod://dest/directory/FILE.txt
nod://source/folder/FILE.txt nod://dest/RENAMED.txt nod://dest/RENAMED.txt
nod://source/folder/ nod://dest/directory/ nod://dest/directory/FILE.txt
nod://source/folder/ nod://dest/directory nod://dest/directory/FILE.txt
nod://source/folder nod://dest/directory/ nod://dest/directory/folder/FILE.txt
nod://source/folder nod://dest/directory nod://dest/directory/FILE.txt

With --wd=..

Source Destination Result
nod://source/folder/FILE.txt nod://dest/directory/ nod://dest/directory/folder/FILE.txt
nod://source/folder/FILE.txt nod://dest/RENAMED.txt nod://dest/RENAMED.txt
nod://source/folder/ nod://dest/directory/ nod://dest/directory/folder/FILE.txt
nod://source/folder/ nod://dest/directory nod://dest/directory/FILE.txt
nod://source/folder nod://dest/directory/ nod://dest/directory/source/folder/FILE.txt
nod://source/folder nod://dest/directory nod://dest/directory/FILE.txt

Task Creation

Description

This command send a copy request to the data mover service from a storage A (nod://posix_storage/) to a storage B (nod-cloud://cloud_storage/)

Command with minimal parameters

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my projectdomain name> \ --md user_domain_name=<my user domain name> \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax
nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my projectdomain name> \
--md user_domain_name=<my user domain name> \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Example of creation task with additional parameters

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://largedata2_pool/storagetestdata/ \ --recursive \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax
nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Examples of Tasks Creation

Execute a task copy from Posix to Swift

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax
nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Execute a task copy from Swift to Posix

nd copy \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/

Detailed syntax
nd copy \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod-cloud://cloud_storage/container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/

Execute a task move from Posix to Swift

nd move \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod://posix_storage/path/ \ --recursive \ --ignore-hidden \ nod://posix_storage/path/subpath/ \ nod-cloud://cloud_storage/container

Detailed syntax
nd move \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod://posix_storage/path/ \
--recursive \
--ignore-hidden \
nod://posix_storage/path/subpath/ \
nod-cloud://cloud_storage/container

Execute a task move from Swift to Posix

nd move \ --md project_name=<my project name> \ --md project_domain_name=<my project domain name> \ --md user_domain_name=<my user domain name> \ --working-dir nod-cloud://cloud_storage/container/ \ --recursive \ --ignore-hidden \ nod-cloud://cloud_storage/container/path/ \ nod://posix_storage/path/

Detailed syntax
nd move \
--md project_name=<my project name> \
--md project_domain_name=<my project domain name> \
--md user_domain_name=<my user domain name> \
--working-dir nod-cloud://swift_connector/my_new_container/ \
--recursive \
--ignore-hidden \
nod-cloud://cloud_storage/container/path/ \
nod://posix_storage/path/

Command to list created tasks

Command

nd task list

Description

This command list all tasks created by the user in the data mover service.

The columns describe:

Output Description
TASK ID ID of the task
TASK NAME Name of the task defined during the creation
COMMENT Associated comment
CREATED BY User who has created the task

Output

+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| TASK ID                  | TASK NAME                                                  | COMMENT | CREATED BY        | LAST EXECUTION STATUS |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| 6287a774a91db0b194e97d8d | From /largedata2_pool/storagetestdata/data5 to my_pool1    |         | Valery Guilleaume | done                  |
| 628b45f5a91db04211739dc6 | From /largedata2_pool/storagetestdata/data5 to my_pool1    |         | Valery Guilleaume | done                  |
| 628b468ea91db04211739dc9 | From /largedata2_pool/storagetestdata/data5 to my_pool1    |         | Valery Guilleaume | done                  |
| 6331d52ba91db02d6797e6ae | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6331d5aca91db02d6797e6b1 | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6331d5c5a91db02d6797e6b4 | From nod://largedata2_pool/storagetestdata/ to vg--1590    |         | Valery Guilleaume | stopped_by_user       |
| 6331d677a91db02d6797e6b7 | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | done                  |
| 6331d692a91db02d6797e6ba | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6331d948a91db02d6797e6c6 | From nod://largedata2_pool/storagetestdata/ to vg--1598    |         | Valery Guilleaume | stopped_by_user       |
| 6333ffdda91db091264b68a5 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | finished with warning |
| 6333fff6a91db092d94b68a2 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | finished with warning |
| 63355214a91db0397f128dbf | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 63355218a91db0397f128dc2 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 63355220a91db0a6ec22073e | From nod://largedata2_pool/storagetestdata/ to vg--1501    |         | Valery Guilleaume | done                  |
| 6335549aa91db0397f128dc5 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 6335549da91db0397f128dc8 | From nod://largedata2_pool/storagetestdata/ to vg--1502    |         | Valery Guilleaume | done                  |
| 633554a1a91db0a6ec220741 | From nod://largedata2_pool/storagetestdata/ to vg--1506    |         | Valery Guilleaume | done                  |
| 63358216a91db0397f128dcb | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | finished with warning |
| 6335822da91db0397f128dce | From nod://largedata2_pool/storagetestdata/ to vg--1502    |         | Valery Guilleaume | done                  |
| 633584d5a91db0397f128dd1 | From nod://largedata2_pool/storagetestdata/ to vg--1502    |         | Valery Guilleaume | finished with warning |
| 6336b341a91db0397f128dd4 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
| 6336b35da91db0c2985ff420 | From nod://largedata2_pool/storagetestdata/ to vg--1500    |         | Valery Guilleaume | done                  |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+
| NUMBER OF TASK(S)       22                                                            |         |                   |                       |
+--------------------------+------------------------------------------------------------+---------+-------------------+-----------------------+

Display tasks status

Description

At the end of each task execution, the task result is displayed.

INFO Connecting with device flow...
INFO Connected with user Valery Guilleaume
Processed size ... done! [8.17MB in 39s]
Processed items ... done! [471 in 39s]
          ID: 633ecc74975854bbc799a424
     Task ID: 633ecc74a91db0f38f7abc2e
        Name: From nod://largedata2_pool/storagetestdata/ to vguilleaume
     Comment:
  Created by: Valery Guilleaume
       Nodes: 471 / 471
        Size: 8.17 MB / 8.17 MB
      Status: done

Get the status of a task

Description

Anytime the status of task can be displayed.

Command

# nd task status 633ecc74a91db0f38f7abc2e

where 633ecc74a91db0f38f7abc2e is the id of the task

Output
INFO Connecting with device flow...
INFO Connected with user Valery Guilleaume
          ID: 633ecc74975854bbc799a424
     Task ID: 633ecc74a91db0f38f7abc2e
        Name: From nod://largedata2_pool/storagetestdata/ to vguilleaume
     Comment:
  Created by: Valery Guilleaume
       Nodes: 471 / 471
        Size: 8.17 MB / 8.17 MB
      Status: done

Command to list the status of all executed tasks

Command

nd task list-exec 6389c04605e7b8ff6df35cc4

Description

This command list all tasks executed by the user in the data mover service.
The columns describe:

Output Description
ID ID of the executed task
STARTED AT Date when the task has been started
FINISHED AT Date when the task has been finished
NODES Number of files copied / Total number of files to be copied
SIZE Size of files copied / Total size of files to be copied
STATUS Status of the executed task

Output

+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
| ID                       | STARTED AT          | FINISHED AT         | NODES   | SIZE                | STATUS          |
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+
| 6389c105be32a3507d90a6ea | 02 Dec 22 10:10 CET | 02 Dec 22 10:27 CET | 86 / 86 | 5.89 GB / 5.89 GB   | done            |
| 6389e5e191c3044a236f7baf | 02 Dec 22 12:47 CET | 02 Dec 22 12:49 CET | 3 / 86  | 108.38 MB / 5.89 GB | stopped_by_user |
| 6389e64391c3044a236f7c09 | 02 Dec 22 12:49 CET | 02 Dec 22 12:50 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 6389f33d91c3044a236f7c63 | 02 Dec 22 13:44 CET | 02 Dec 22 14:00 CET | 22 / 86 | 1.76 GB / 5.89 GB   | stopped_by_user |
| 6389f704c4df1940f3253444 | 02 Dec 22 14:00 CET | 02 Dec 22 14:03 CET | 5 / 86  | 108.41 MB / 5.89 GB | stopped_by_user |
| 6389f7bac4df1940f325349e | 02 Dec 22 14:03 CET | 02 Dec 22 14:04 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 6389f812c4df1940f32534f8 | 02 Dec 22 14:05 CET | 02 Dec 22 14:06 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 6389fc3cc4df1940f3253552 | 02 Dec 22 14:23 CET | 02 Dec 22 16:11 CET | 0 / 0   | 0.00 B / 0.00 B     | stopped_by_user |
| 638a188cfbd1fa439d567ae2 | 02 Dec 22 16:23 CET | 02 Dec 22 16:25 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 638a1916fbd1fa439d567b3c | 02 Dec 22 16:26 CET | 02 Dec 22 16:27 CET | 2 / 86  | 19.52 kB / 5.89 GB  | stopped_by_user |
| 638a1986fbd1fa439d567b96 | 02 Dec 22 16:28 CET | 02 Dec 22 16:30 CET | 2 / 86  | 22.02 kB / 5.89 GB  | stopped_by_user |
| 638a21c5fbd1fa439d567bf0 | 02 Dec 22 17:03 CET | 05 Dec 22 11:18 CET | 86 / 86 | 5.89 GB / 5.89 GB   | done            |
+--------------------------+---------------------+---------------------+---------+---------------------+-----------------+