sudo ym-set-static-ip.sh

This command encrypts the local keys and configuration using clevis/tang

This command encrypts the local keys and configuration using clevis/tang

This command encrypts the local keys and configuration using clevis/tang

yarnman@ym-ph-test [ ~ ]$ sudo ym-encrypt-at-rest.sh
Database key found proceeding
Number of pins required for decryption :1
Number of pins this must be equal or greater than the number of pins required for decryption :3
Enter URL for tang server 1 :http://10.101.10.10:6655
Enter THP for tang server 1 :DwLco7FJtXWxFTprQ5M3cojJsZo
Connection successful to : http://10.101.10.10:6655
Enter URL for tang server 2 :http://10.101.10.11:6655
Enter THP for tang server 2 :0Lqk7DroJ0g3patTCgTweMUAHPc
Connection successful to : http://10.101.10.11:6655
Enter URL for tang server 3 :http://10.101.10.12:6655
Enter THP for tang server 3 :GEpmSTQfz8ctVxdgQEp_rnS3za
Connection successful to : http://10.101.10.12:6655

{
  "t": 1,
  "pins": {
    "tang": [
      {
        "url": "http://10.101.10.10:6655",
        "thp": "DwLco7FJtXWxFTprQ5M3cojJsZo"
      },
      {
        "url": "http://10.101.10.11:6655",
        "thp": "0Lqk7DroJ0g3patTCgTweMUAHPc"
      },
      {
        "url": "http://10.101.10.12:6655",
        "thp": "GEpmSTQfz8ctVxdgQEp_rnS3za"
      }
    ]
  }
}
Do you want to encrypt configuration? Y or Ny
encrypt configuration
Encrypting keys
1668397245104 INFO  Encrypting private and SSL keys using settings:
1668397245106 INFO    - not overwriting existing encrypted files and not deleting any original files after encryption
1668397245106 INFO  --------------------------------
1668397245106 INFO  Encrypting...
1668397245308 INFO    - 'private-encryption-key.pem' encrypted successfully
1668397245543 INFO    - 'ssl-key.pem' encrypted successfully
1668397245543 INFO  --------------------------------
1668397245543 INFO  Finished encrypting the files
Encrypting config
1668397245643 INFO  Starting the encryption of 1 local configuration fields through Clevis Shamir Secret Sharing
1668397245743 INFO  Attempting to encrypt the following local config fields: couchdb.password
1668397245843 INFO  Local key 'couchdb.password' encrypted successfully
1668397245943 INFO  1 local config fields encrypted, 0 fields omitted
Do you want to take a backup of database key this will be shown on console? Y orNy
Echo private key to console
-----BEGIN RSA PRIVATE KEY-----
REMOVED
-----END RSA PRIVATE KEY-----
Encrypted private key is 8129 bytes
restarting services
Config encryption is complete

This command upgrades yarnman

sudo ym-generate-certs.sh

A reboot may be required to apply OS patches if they are bundled into the update.

yarnman@node1yarnman-test docker-compose[55552]: Container ym-redis  Removed
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Stopped
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Removing
Aug 17 08:24:15 yarnman-test docker-compose[4221]: ym-couchdb exited with code 0
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Removed
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Network yarnman_yl-yarnman  Removing
Aug 17 08:24:16 yarnman-test docker-compose[55552]: Network yarnman_yl-yarnman  Removed
Aug 17 08:24:16 yarnman-test systemd[1]: yarnman.service: Succeeded.
Aug 17 08:24:16 yarnman-test systemd[1]: Stopped yarnman.

No login access is available to the backup service account

yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a sftp-user-setup

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = sftp-user-setup
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
creating keys for ym-backup-user
public key for ssh/sftp
ssh-rsa ****LongStringForPubKey****
yarnman@node1 [ ~ ]$

yarnman@node1 [ ~ ]$ su
Password: 
yarnman@node1 [ /var/home/yarnman ]# sudo -u ym-backup-user ssh-copy-id -i /home/ym-backup-user/.ssh/id_rsa.pub sftpbackup@10.101.10.86

/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ym-backup-user/.ssh/id_rsa.pub"
The authenticity of host '10.101.10.86 (10.101.10.86)' can't be established.
ED25519 key fingerprint is SHA256:****j7t+o1aQu5FoWlxS0uhKzCe414jt3****
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Authorized uses only. All activity may be monitored and reported.
sftpbackup@10.101.10.86's password: 

Number of key(s) added: 1

yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a sftp-setup-connection

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = sftp-setup-connection
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
SFTP config is /var/opt/yarnlab/backup/sftp
enter sftp infomation
SFTP Username: sftpbackup
SFTP Host: 10.101.10.86
SFTP backup directory Path i.e /srv/yarnman/backup: /home/sftpbackup/yarnman
sftp:yarnman@10.101.10.86:/home/sftpbackup/yarnman
yarnman@node1 [ ~ ]$

you may be prompted for username/password if the SSH pub key hasn’t been added to the SFTP server, this is OK for the initial setup, however scheduled/automated backups will fail

yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a snapshots

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = snapshots
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
Checking snapshots for profile :yarnman-sftp
2023/08/11 04:41:34 profile 'yarnman-sftp': starting 'snapshots'
2023/08/11 04:41:34 unfiltered extra flags: 
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
Fatal: unable to open config file: Lstat: file does not exist
Is there a repository at the following location?
sftp:sftpbackup@10.101.10.86:/home/sftpbackup/yarnman
2023/08/11 04:41:34 snapshots on profile 'yarnman-sftp': exit status 1

yarnman@node1loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 08:27:36 UTC; 6ms ago
    Process: 63277 ExecStartPre=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 63287 (docker-compose)
      Tasks: 6 (limit: 4694)
     Memory: 4.9M
     CGroup: /system.slice/yarnman.service
             └─63287 /usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans

Aug 17 08:27:36 yarnman-test systemd[1]: Starting yarnman...
Aug 17 08:27:36 yarnman-test docker-compose[63277]: yarnman  Warning: No resource found to remove
Aug 17 08:27:36 yarnman-test systemd[1]: Started yarnman.

Initialising can only be preformed once to a repository, an error will occur if it exists already.

yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a snapshots

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = snapshots
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
Checking snapshots for profile :yarnman-sftp
2023/08/11 04:44:19 profile 'yarnman-sftp': starting 'snapshots'
2023/08/11 04:44:19 unfiltered extra flags: 
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
repository 7180598c opened (version 2, compression level auto)
2023/08/11 04:44:20 profile 'yarnman-sftp': finished 'snapshots'
yarnman@node1 [ ~ ]$

repository 7180598c opened (version 2, compression level auto) indicating a valid backup location

yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a backup

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = backup
RESTORECOMMIT     = 
RESTORE_IP        = 
RESTORE_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
Running backup for profile :yarnman-sftp
2023/08/11 04:46:11 profile 'yarnman-sftp': starting 'backup'
2023/08/11 04:46:11 unfiltered extra flags: 
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
repository 7180598c opened (version 2, compression level auto)
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/var/opt/yarnlab/yarnman/config /var/opt/yarnlab/couchdb/config /var/opt/yarnlab/couchdb/data /var/opt/yarnlab/couchdb/certs /var/opt/yarnlab/tang/db /var/opt/yarnlab/certs /var/opt/yarnlab/registry]
start backup on [/var/opt/yarnlab/yarnman/config /var/opt/yarnlab/couchdb/config /var/opt/yarnlab/couchdb/data /var/opt/yarnlab/couchdb/certs /var/opt/yarnlab/tang/db /var/opt/yarnlab/certs /var/opt/yarnlab/registry]
scan finished in 0.233s: 564 files, 5.211 MiB

Files:         564 new,     0 changed,     0 unmodified
Dirs:          348 new,     0 changed,     0 unmodified
Data Blobs:    404 new
Tree Blobs:    349 new
Added to the repository: 5.479 MiB (736.577 KiB stored)

processed 564 files, 5.211 MiB in 0:00
snapshot fa50ff98 saved
2023/08/11 04:46:12 profile 'yarnman-sftp': finished 'backup'
2023/08/11 04:46:12 profile 'yarnman-sftp': cleaning up repository using retention information
2023/08/11 04:46:12 unfiltered extra flags: 
repository 7180598c opened (version 2, compression level auto)
Applying Policy: keep 3 daily, 1 weekly, 1 monthly snapshots and all snapshots with tags [[manual]] and all snapshots within 3m of the newest
snapshots for (host [node76-restore4], paths [/var/opt/yarnlab/certs, /var/opt/yarnlab/couchdb/certs, /var/opt/yarnlab/couchdb/config, /var/opt/yarnlab/couchdb/data, /var/opt/yarnlab/registry, /var/opt/yarnlab/tang/db, /var/opt/yarnlab/yarnman/config]):
keep 1 snapshots:
ID        Time                 Host             Tags │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────────────────────────────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 2   │ administration-app-0ca298ae6a834cf29c661930c58cb621      │ default     │ 2.5.18  │ fork    │ 236      │ 10s    │ 0    │ online    │ 0%       │ 137.8mb  │ ym-… │ enabled  │
│ 0   │ arm_fc30b4f5d59f4275829ff8b65d02914b                     │ default     │ 2.5.18  │ fork    │ 121      │ 19s    │ 5    │ online    │ 0%       │ 65.1mb   │ ym-… │ enabled  │
│ 3   │ interconnect-service-49ab91419f064823b8ab85806b3b4ce1    │ default     │ 2.5.18  │ fork    │ 260      │ 8s     │ 0    │ online    │ 0%       │ 138.8mb  │ ym-… │ enabled  │
│ 1   │ jadeberlin_arm_fc30b4f5d59f4275829ff8b65d02914b          │ default     │ N/A     │ fork    │ 0        │ 0      │ 4    │ errored   │ 0%       │ 0b       │ ym-… │ disabled │
│ 4   │ proxy-service-a4500ec67fcc491399dc395e12c1bbe1           │ default     │ 2.5.18  │ fork    │ 271      │ 6s     │ 0    │ online    │ 0%       │ 105.3mb  │ ym-… │ enabled  │
│ 5   │ workflow-service-8b4edbbb287c468cae0f023dd7e0cf44        │ default     │ 2.5.18  │ fork    │ 282      │ 5s     │ 0    │ online    │ 0%       │ 175.4mb  │ ym-… │ enabled  │
└─────┴──────────────────────────────────────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
[PM2][WARN] Current process list is not synchronized with saved list. Type 'pm2 save' to synchronize.

nano /var/opt/yarnlab/yarnman/config/ym-backup-config.yml

PENDING

Enable Schedule

sudo ym-backup-actions.sh -p sftp -a schedule

Disable Schedule

sudo ym-backup-actions.sh -p sftp -a unschedule

Check status of schedule

sudo ym-backup-actions.sh -p sftp -a status

yarnman@node1 [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a restore -r fa50ff98

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = restore
RESTORECOMMIT     = latest
BACKUP_IP         = 
BACKUP_PATH      = 
settting sftp mode
profile mode :yarnman-sftp
Restore backup for profile :yarnman-sftp
starting restore
Restore backup for profile :yarnman-sftp commit :latest
Are you sure you want to restore backup? Y or Ny
Restore Backup
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
Backup nodeId's match

Stopping yarnman services
Removing exising configuration to prevent duplicates
Starting restic restore

2023/08/16 08:08:33 profile 'yarnman-sftp': finished 'restore'
Resetting permissions
Starting Database and Encryption services
[+] Creating 5/5
 ✔ Network yarnman_yl-yarnman  Created                                                                                                                                                        0.0s 
 ✔ Container ym-redis          Created                                                                                                                                                        0.1s 
 ✔ Container ym-couchdb        Created                                                                                                                                                        0.1s 
 ✔ Container ym-tang           Created                                                                                                                                                        0.1s 
 ✔ Container ym-yarnman        Created                                                                                                                                                        0.1s 
[+] Running 1/1
 ✔ Container ym-couchdb  Started                                                                                                                                                              0.3s 
[+] Running 1/1
 ✔ Container ym-tang  Started

Checking number of admin nodes
number of admin nodes :x
Yarnman is in distributed mode
Check couchdb replication on other nodes is healthy and after 5 minutes reboot yarnman or run systemctl stop yarnman.service and systemctl start yarnman.service

A new SFTP/SSH key will be created, this will need to be added to the backups server for future automated backups to function again. interactive (user/pass) can be used for a restore , if the new ssh key can’t be added to the backup server at time of restore.

the Hostname doesn’t need to be the same as the restore backup, however any new backups will backup with the new hostname.

If building with a different IP address, Replication will need to be adjusted to the new IP address as well as Clevin/Tang if setup.

sudo ym-backup-setup.sh 
sudo ym-backup-actions.sh -p sftp -a sftp-user-setup
as root user ;  sudo -u ym-backup-user ssh-copy-id -i /home/ym-backup-user/.ssh/id_rsa.pub sftpbackup@10.101.10.86
sudo ym-backup-actions.sh -p sftp -a sftp-setup-connection
sudo ym-backup-actions.sh -p sftp -a snapshots
sudo ym-backup-actions.sh -p sftp -a restore -r xxxxx

yarnman@node79-restore [ ~ ]$ sudo ym-backup-actions.sh -p sftp -a restore -r 5f13f62b

backup config found
PROFILE_NAME_VAR  = sftp
ACTION_VAR        = restore
RESTORECOMMIT     = 5f13f62b BACKUP_IP      COMMAND   =  BACKUP_PATH      =  settting sftp mode profile mode :yarnman-sftpCREATED Restore backup for profile :yarnman-sftp starting restore
Restore backup for profile :yarnman-sftp commit :5f13f62b
Are you sure you want to restore backup? Y or Ny
Restore Backup
subprocess ssh: Authorized uses only. All activity may be monitored and reported.
Current Node Id is :arm_46b194ad3d374b7397fa14b1a3136d56
Backup Node Id is :arm_3110b0b79eb84bd899291d5e0d231009
Do you want to apply this backup that has different nodeId? Y or N

sudo ym-backup-actions.sh -p manual -a manual-sftp-snapshots -i 10.101.10.86 -k /home/sftpbackup/path/ 

sudo ym-backup-actions.sh -p manual -a manual-sftp-restore -i 10.101.10.86 -k /home/sftpbackup/path/ -r xxxxx

This command starts the yarnman services

yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh start
starting yarnman.service
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 08:24:21 UTC; 5ms ago
    Process: 56027 ExecStartPre=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 56037 (docker-compose) STATUS                 PORTS                                                                                                          Tasks: 5 (limit: 4694)      Memory: 5.0M      CGroup: /system.slice/yarnman.service              └─56037 /usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans

This command stops the yarnman services

yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh stop
stopping yarnman.service
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2022-08-17 08:24:16 UTC; 6ms ago
    Process: 4221 ExecStart=/usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans (code=exited, status=0/SUCCESS)
    Process: 55552 ExecStop=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 4221 (code=exited, status=0/SUCCESS)

Aug 17 08:24:14 yarnman-test docker-compose[4221]: ym-redis exited with code 0
Aug 17 08:24:14 yarnman-test docker-compose[55552]: Container ym-redis  Removed
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Stopped
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Removing
Aug 17 08:24:15 yarnman-test docker-compose[4221]: ym-couchdb exited with code 0
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Container ym-couchdb  Removed
Aug 17 08:24:15 yarnman-test docker-compose[55552]: Network yarnman_yl-yarnman  Removing
Aug 17 08:24:16 yarnman-test docker-compose[55552]: Network yarnman_yl-yarnman  Removed
Aug 17 08:24:16 yarnman-test systemd[1]: yarnman.service: Succeeded.
Aug 17 08:24:16 yarnman-test systemd[1]: Stopped yarnman.

this command restarts the yarnman services

yarnman@yarnman-test [ ~ ]$ sudo ym-service-commands.sh restart
restarting yarnman.service
● yarnman.service - yarnman
     Loaded: loaded (/usr/lib/systemd/system/yarnman.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 08:27:36 UTC; 6ms ago
    Process: 63277 ExecStartPre=/usr/bin/docker-compose -f docker-compose.yml down (code=exited, status=0/SUCCESS)
   Main PID: 63287 (docker-compose)
      Tasks: 6 (limit: 4694)
     Memory: 4.9M
     CGroup: /system.slice/yarnman.service NAMES
203da0dedf3b   ym-local/ym-yarnman:ym-bbpatch-fcab0f3         "/bin/bash -c 'node …"   7 hours ago   Up 7 hours (healthy)   0.0.0.0:80->80/tcp, :::80->80/tcp, 0/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:6700-6710->6700-6710/udp, :::6700-6710->6700-6710/udp   ym-yarnman
edcca82fbcb8   localhost:5000/ym-couchdb:yl-ph-n18-cb516f8c   "tini -- /docker-ent…"   7 hours ago   Up 7 hours (healthy)   4369/tcp, 5984/tcp, 9100/tcp, 0.0.0.0:6984->6984/tcp, :::6984->6984/tcp                                                                          ym-couchdb
8aee8553dec6   localhost:5000/ym-tang:yl-ph-n18-cb516f8c      "/usr/local/bin/star…"   7 hours ago   Up 7 hours (healthy)   6655/tcp, 0.0.0.0:6655->8080/tcp, :::6655->8080/tcp                                                                                └─63287 /usr/bin/docker-compose -f docker-compose.yml -f docker-compose-override.yml up --remove-orphans  Aug 17 08:27:36 yarnman-test systemd[1]: Starting yarnman...
Aug 17 08:27:36 yarnman-test docker-compose[63277]: yarnman  Warning: No resource found to remove
Aug 17 08:27:36 yarnman-test systemd[1]: Started yarnman. ym-tang
63b6e664ab62   localhost:5000/ym-redis:yl-ph-n18-cb516f8c     "docker-entrypoint.s…"   7 hours ago   Up 7 hours (healthy)   6379/tcp   

this command shows the internal processes of yarnman

yarnman@yarnman-test [ ~ ]$ 4.70%     194.9MiB / 6GiB     3.17%     7.87MB / 29.2MB   17.5MB / 6.46MB   61
8aee8553dec6   ym-tang      3.31%     804KiB / 128MiB     0.61%     2.15kB / 0B       0B / 0B           2
63b6e664ab62   ym-redis     0.56%     11.04MiB / 512MiB   2.16%     157MB / 70.8MB    0B / 0B           5

This command shows the scrolling output of yarnman services press CTRL+c to exit

This command shows the scrolling output of dabase logs press CTRL+c to exit

This command shows the scrolling output of message bus logs press CTRL+c to exit

This command shows the scrolling output of NBE logs press CTRL+c to exit

Note that this command was previously ym-service-commands.sh tang-adv

This command shows the tag thp used for setting up configuration encryption

yarnman@ym-ph-test [ ~ ]$ sudo ym-service-commands.sh tang-adv
9_CZiwV9PKBlQfehPKZO7cd5ZpM

This command updates jtapi for test_mate

PENDING

This command enables local admin access on port 3999

PENDING

This command disables local admin access on port 3999

PENDING

This command enables couchdb access

PENDING

This command disables couchdb access

PENDING

This command sets the container hostname for clustered systems

PENDING

This command unsets the container hostname for clustered systems

PENDING

This command enables yarnman trace logs

PENDING

This command enables yarnman debug logs (default)

PENDING --------------------------------
1668397245106 INFO  Encrypting...
1668397245308 INFO    - 'private-encryption-key.pem' encrypted successfully
1668397245543 INFO    - 'ssl-key.pem' encrypted successfully
1668397245543 INFO  --------------------------------
1668397245543 INFO  Finished encrypting the files
Encrypting config
1668397245643 INFO  Starting the encryption of 1 local configuration fields through Clevis Shamir Secret Sharing
1668397245743 INFO  Attempting to encrypt the following local config fields: couchdb.password
1668397245843 INFO  Local key 'couchdb.password' encrypted successfully
1668397245943 INFO  1 local config fields encrypted, 0 fields omitted
Do you want to take a backup of database key this will be shown on console? Y orNy
Echo private key to console
-----BEGIN RSA PRIVATE KEY-----
REMOVED
-----END RSA PRIVATE KEY-----
Encrypted private key is 8129 bytes
restarting services
Config encryption is complete