Versions Compared

Old Version 2

changes.mady.by.user Zane England

Saved on

compared with

New Version 3

changes.mady.by.user Zane England

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Note

All yarnman service commands need to be run with sudo

ym-set-static-ip.sh

Info

This command encrypts the local keys and configuration using clevis/tang

ym-generate-certs.sh

Info

This command encrypts the local keys and configuration using clevis/tang

ym-install.sh

Info

This command encrypts the local keys and configuration using clevis/tang

ym-encrypt-at-rest.sh

Info

This command encrypts the local keys and configuration using clevis/tang

Code Block
yarnman@ym-ph-test [ ~ ]$ sudo ym-encrypt-at-rest.sh
Database key found proceeding
Number of pins required for decryption :1
Number of pins this must be equal or greater than the number of pins required for decryption :3
Enter URL for tang server 1 :http://10.101.10.10:6655
Enter THP for tang server 1 :DwLco7FJtXWxFTprQ5M3cojJsZo
Connection successful to : http://10.101.10.10:6655
Enter URL for tang server 2 :http://10.101.10.11:6655
Enter THP for tang server 2 :0Lqk7DroJ0g3patTCgTweMUAHPc
Connection successful to : http://10.101.10.11:6655
Enter URL for tang server 3 :http://10.101.10.12:6655
Enter THP for tang server 3 :GEpmSTQfz8ctVxdgQEp_rnS3za
Connection successful to : http://10.101.10.12:6655

{
  "t": 1,
  "pins": {
    "tang": [
      {
        "url": "http://10.101.10.10:6655",
        "thp": "DwLco7FJtXWxFTprQ5M3cojJsZo"
      },
      {
        "url": "http://10.101.10.11:6655",
        "thp": "0Lqk7DroJ0g3patTCgTweMUAHPc"
      },
      {
        "url": "http://10.101.10.12:6655",
        "thp": "GEpmSTQfz8ctVxdgQEp_rnS3za"
      }
    ]
  }
}
Do you want to encrypt configuration? Y or Ny
encrypt configuration
Encrypting keys
1668397245104 INFO  Encrypting private and SSL keys using settings:
1668397245106 INFO    - not overwriting existing encrypted files and not deleting any original files after encryption
1668397245106 INFO  --------------------------------
1668397245106 INFO  Encrypting...
1668397245308 INFO    - 'private-encryption-key.pem' encrypted successfully
1668397245543 INFO    - 'ssl-key.pem' encrypted successfully
1668397245543 INFO  --------------------------------
1668397245543 INFO  Finished encrypting the files
Encrypting config
1668397245643 INFO  Starting the encryption of 1 local configuration fields through Clevis Shamir Secret Sharing
1668397245743 INFO  Attempting to encrypt the following local config fields: couchdb.password
1668397245843 INFO  Local key 'couchdb.password' encrypted successfully
1668397245943 INFO  1 local config fields encrypted, 0 fields omitted
Do you want to take a backup of database key this will be shown on console? Y orNy
Echo private key to console
-----BEGIN RSA PRIVATE KEY-----
REMOVED
-----END RSA PRIVATE KEY-----
Encrypted private key is 8129 bytes
restarting services
Config encryption is complete

ym-upgrade.sh

Info

This command upgrades yarnman

...

Note

A reboot may be required to apply OS patches if they are bundled into the update.

ym-backup-setup.sh

Sets up the local backup service account on the yarnman node, and the passphrase used on the backup
Code Block
yarnman@node1 [ ~ ]$ sudo ym-backup-setup.sh 

Starting yarnman ph4 backup
Backup password not set 
Set Backup password: 
Backup password (again): 
Clevis not setup
using local backup password
no backup configuration file found creating
yarnman@node1 [ ~ ]$
Note

No login access is available to the backup service account

ym-backup-actions.sh

all the backup commands are done via the script above

Setup sftp as the backup method and ssh public keys
Code Block
yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a sftp-user-setup

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = sftp-user-setup
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
creating keys for ym-backup-user
public key for ssh/sftp
ssh-rsa ****LongStringForPubKey****
yarnman@node1 [ ~ ]$
Copy ssh pub key to sftp server

if ssh access is available to the SFTP server you can copy the ssh public key for login, otherwise provide the key to your SFTP Administrator.

Code Block
yarnman@node1 [ ~ ]$ su
Password: 
yarnman@node1 [ /var/home/yarnman ]# sudo -u ym-backup-user ssh-copy-id -i /home/ym-backup-user/.ssh/id_rsa.pub sftpbackup@10.101.10.86

/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ym-backup-user/.ssh/id_rsa.pub"
The authenticity of host '10.101.10.86 (10.101.10.86)' can't be established.
ED25519 key fingerprint is SHA256:****j7t+o1aQu5FoWlxS0uhKzCe414jt3****
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Authorized uses only. All activity may be monitored and reported.
sftpbackup@10.101.10.86's password: 

Number of key(s) added: 1
Setup SFTP destination for backup

the script will prompt for backup path, ip address and userid to the SFTP server

...

Info

you may be prompted for username/password if the SSH pub key hasn’t been added to the SFTP server, this is OK for the initial setup, however scheduled/automated backups will fail

Check if backups exist at location

for first time configuration no backups will be available, nor a backup repository which will be setup in the next section.

Code Block
yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a snapshots

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = snapshots
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
Checking snapshots for profile :yarnman-sftp
2023/08/11 04:41:34 profile 'yarnman-sftp': starting 'snapshots'
2023/08/11 04:41:34 unfiltered extra flags: 
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
Fatal: unable to open config file: Lstat: file does not exist
Is there a repository at the following location?
sftp:sftpbackup@10.101.10.86:/home/sftpbackup/yarnman
2023/08/11 04:41:34 snapshots on profile 'yarnman-sftp': exit status 1
Initialise the repository

the password used from the initial ym-backup-setup.sh will automatically be used

...

Info

Initialising can only be preformed once to a repository, an error will occur if it exists already.

List backups (snapshots)

list all backups available , on a new repository this will be blank

...

Info

repository 7180598c opened (version 2, compression level auto) indicating a valid backup location

Manual Backup

preform a manual backup

Code Block
yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a backup

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = backup
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
Running backup for profile :yarnman-sftp
2023/08/11 04:46:11 profile 'yarnman-sftp': starting 'backup'
2023/08/11 04:46:11 unfiltered extra flags: 
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
repository 7180598c opened (version 2, compression level auto)
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/var/opt/yarnlab/yarnman/config /var/opt/yarnlab/couchdb/config /var/opt/yarnlab/couchdb/data /var/opt/yarnlab/couchdb/certs /var/opt/yarnlab/tang/db /var/opt/yarnlab/certs /var/opt/yarnlab/registry]
start backup on [/var/opt/yarnlab/yarnman/config /var/opt/yarnlab/couchdb/config /var/opt/yarnlab/couchdb/data /var/opt/yarnlab/couchdb/certs /var/opt/yarnlab/tang/db /var/opt/yarnlab/certs /var/opt/yarnlab/registry]
scan finished in 0.233s: 564 files, 5.211 MiB

Files:         564 new,     0 changed,     0 unmodified
Dirs:          348 new,     0 changed,     0 unmodified
Data Blobs:    404 new
Tree Blobs:    349 new
Added to the repository: 5.479 MiB (736.577 KiB stored)

processed 564 files, 5.211 MiB in 0:00
snapshot fa50ff98 saved
2023/08/11 04:46:12 profile 'yarnman-sftp': finished 'backup'
2023/08/11 04:46:12 profile 'yarnman-sftp': cleaning up repository using retention information
2023/08/11 04:46:12 unfiltered extra flags: 
repository 7180598c opened (version 2, compression level auto)
Applying Policy: keep 3 daily, 1 weekly, 1 monthly snapshots and all snapshots with tags [[manual]] and all snapshots within 3m of the newest
snapshots for (host [node76-restore4], paths [/var/opt/yarnlab/certs, /var/opt/yarnlab/couchdb/certs, /var/opt/yarnlab/couchdb/config, /var/opt/yarnlab/couchdb/data, /var/opt/yarnlab/registry, /var/opt/yarnlab/tang/db, /var/opt/yarnlab/yarnman/config]):
keep 1 snapshots:
ID        Time                 Host             Tags            Reasons           Paths
-----------------------------------------------------------------------------------------------------------------
fa50ff98  2023-08-11 04:46:11           node1  ym-backup-sftp  within 3m         /var/opt/yarnlab/certs
                                                                daily snapshot    /var/opt/yarnlab/couchdb/certs
                                                                weekly snapshot   /var/opt/yarnlab/couchdb/config
                                                                monthly snapshot  /var/opt/yarnlab/couchdb/data
                                                                                  /var/opt/yarnlab/registry
                                                                                  /var/opt/yarnlab/tang/db
                                                                                  /var/opt/yarnlab/yarnman/config
-----------------------------------------------------------------------------------------------------------------
1 snapshots

yarnman@node1 [ ~ ]$

Schedule

By default the schedule is setup to backup at 1am UTC every day, This can be modified in the config file with as the root user

...

Code Block
PENDING

Enable Schedule

sudo ym-backup-actions.sh -p sftp -a schedule

Disable Schedule

sudo ym-backup-actions.sh -p sftp -a unschedule

Check status of schedule

sudo ym-backup-actions.sh -p sftp -a status

Restore backup

To restore a snapshot to an existing node.

...

This is to allow replication to all nodes, to prevent any schedule jobs/ reports from rerunning from the last backup

Rebuild Disaster recovery

Pre-Req

  • Deploy new OVA with same version as the backup

  • Setup as a new install (eg Configure with ip, user/pass, generate certificates if prompted)

  • install yarnman

  • confirm can reach appadmin webpage, Do not Login or Accept the EULA as we will restore over this.

  • Setup backup to same repo for the node to be restored, Do Not initiate the repo or preform a backup

...

Follow instructions after the restore completes.

Alternate Manual Method (not recommended)

*** snapshot command doesnt work in manual mode yet, also requires sudo ym-backup-setup.sh to be run ?

...

Code Block
sudo ym-backup-actions.sh -p manual -a manual-sftp-restore -i 10.101.10.86 -k /home/sftpbackup/path/ -r xxxxx

ym-service-commands.sh start

Info

This command starts the yarnman services

Code Block
yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh start
starting yarnman.service
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 08:24:21 UTC; 5ms ago
    Process: 56027 ExecStartPre=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 56037 (docker-compose)
      Tasks: 5 (limit: 4694)
     Memory: 5.0M
     CGroup: /system.slice/yarnman.service
             └─56037 /usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans

ym-service-commands.sh stop

Info

This command stops the yarnman services

Code Block
yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh stop
stopping yarnman.service
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2022-08-17 08:24:16 UTC; 6ms ago
    Process: 4221 ExecStart=/usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans (code=exited, status=0/SUCCESS)
    Process: 55552 ExecStop=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 4221 (code=exited, status=0/SUCCESS)

Aug 17 08:24:14 yarnman-test docker-compose[4221]: ym-redis exited with code 0
Aug 17 08:24:14 yarnman-test docker-compose[55552]: Container ym-redis  Removed
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Stopped
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Removing
Aug 17 08:24:15 yarnman-test docker-compose[4221]: ym-couchdb exited with code 0
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Removed
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Network yarnman_yl-yarnman  Removing
Aug 17 08:24:16 yarnman-test docker-compose[55552]: Network yarnman_yl-yarnman  Removed
Aug 17 08:24:16 yarnman-test systemd[1]: yarnman.service: Succeeded.
Aug 17 08:24:16 yarnman-test systemd[1]: Stopped yarnman.

ym-service-commands.sh restart

Info

this command restarts the yarnman services

Code Block
yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh restart
restarting yarnman.service
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 08:27:36 UTC; 6ms ago
    Process: 63277 ExecStartPre=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 63287 (docker-compose)
      Tasks: 6 (limit: 4694)
     Memory: 4.9M
     CGroup: /system.slice/yarnman.service
             └─63287 /usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans

Aug 17 08:27:36 yarnman-test systemd[1]: Starting yarnman...
Aug 17 08:27:36 yarnman-test docker-compose[63277]: yarnman  Warning: No resource found to remove
Aug 17 08:27:36 yarnman-test systemd[1]: Started yarnman.

ym-service-commands.sh status

Info

this command shows the systemd service status

Code Block
yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh status
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 08:29:13 UTC; 4s ago
    Process: 67157 ExecStartPre=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 67167 (docker-compose)
      Tasks: 9 (limit: 4694)
     Memory: 15.7M
     CGroup: /system.slice/yarnman.service
             └─67167 /usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans

Aug 17 08:29:14 yarnman-test docker-compose[67167]: ym-couchdb  | [info] 2022-08-17T08:29:14.759420Z nonode@nohost <0.11.0> -------- Application ddoc_cache started on node nonode@nohost
Aug 17 08:29:14 yarnman-test docker-compose[67167]: ym-couchdb  | [info] 2022-08-17T08:29:14.769878Z nonode@nohost <0.11.0> -------- Application global_changes started on node nonode@nohost
Aug 17 08:29:14 yarnman-test docker-compose[67167]: ym-couchdb  | [info] 2022-08-17T08:29:14.769962Z nonode@nohost <0.11.0> -------- Application jiffy started on node nonode@nohost
Aug 17 08:29:14 yarnman-test docker-compose[67167]: ym-couchdb  | [info] 2022-08-17T08:29:14.774590Z nonode@nohost <0.11.0> -------- Application mango started on node nonode@nohost
Aug 17 08:29:14 yarnman-test docker-compose[67167]: ym-couchdb  | [info] 2022-08-17T08:29:14.779025Z nonode@nohost <0.11.0> -------- Application setup started on node nonode@nohost
Aug 17 08:29:14 yarnman-test docker-compose[67167]: ym-couchdb  | [info] 2022-08-17T08:29:14.779045Z nonode@nohost <0.11.0> -------- Application snappy started on node nonode@nohost
Aug 17 08:29:15 yarnman-test docker-compose[67167]: ym-yarnman  | 1660724955149 WARN  Setting Default startup.
Aug 17 08:29:15 yarnman-test docker-compose[67167]: ym-couchdb  | [notice] 2022-08-17T08:29:15.166800Z nonode@nohost <0.334.0> 144d89930f localhost:5984 127.0.0.1 undefined GET / 200 ok 70
Aug 17 08:29:16 yarnman-test docker-compose[67167]: ym-couchdb  | [notice] 2022-08-17T08:29:16.252345Z nonode@nohost <0.335.0> 23ea8ef0ca localhost:5984 127.0.0.1 undefined GET / 200 ok 1
Aug 17 08:29:17 yarnman-test docker-compose[67167]: ym-couchdb  | [notice] 2022-08-17T08:29:17.323062Z nonode@nohost <0.465.0> a377eb4c4c localhost:5984 127.0.0.1 undefined GET / 200 ok 0

ym-service-commands.sh status-pm2

Info

this command shows the internal processes of yarnman

...

Note that the status-pm2 options will change based on the terminal/console width/resolution

ym-service-commands.sh yarnman-logs

Info

This command shows the scrolling output of yarnman services press CTRL+c to exit

ym-service-commands.sh couchdb-logs

Info

This command shows the scrolling output of dabase logs press CTRL+c to exit

ym-service-commands.sh redis-logs

Info

This command shows the scrolling output of message bus logs press CTRL+c to exit

ym-service-commands.sh tang-logs

Info

This command shows the scrolling output of NBE logs press CTRL+c to exit

ym-service-commands.sh tang-thp

Note

Note that this command was previously ym-service-commands.sh tang-adv

...

Code Block
yarnman@ym-ph-test [ ~ ]$ sudo ym-service-commands.sh tang-adv
9_CZiwV9PKBlQfehPKZO7cd5ZpM

ym-service-commands.sh update-jtapi

Info

This command updates jtapi for test_mate

Code Block
PENDING

ym-edit-config.sh enable-local-admin-access

Info

This command enables local admin access on port 3999

Code Block
PENDING

ym-edit-config.sh disable-local-admin-access

Info

This command disables local admin access on port 3999

Code Block
PENDING

ym-edit-config.sh enable-local-couchdb-access

Info

This command enables couchdb access

Code Block
PENDING

ym-edit-config.sh disable-local-couchdb-access

Info

This command disables couchdb access

Code Block
PENDING

ym-edit-config.sh set-local-yarnman-container-name

Info

This command sets the container hostname for clustered systems

Code Block
PENDING

ym-edit-config.sh unset-local-yarnman-container-name

Info

This command unsets the container hostname for clustered systems

Code Block
PENDING

ym-edit-config.sh enable-yarnman-logs

Info

This command enables yarnman trace logs

Code Block
PENDING

ym-edit-config.sh disable-yarnman-logs

Info

This command enables yarnman debug logs (default)

...