Key and Configuration Encryption

...

Customisation

Yarnman Application Additional Ports

With yarn_man Photon additional steps are required for Adding Secondary Local Auth Administration Access

1. This step requires root access

   • to switch to root access run the following command “su root” and enter the root password set during installation

2. Manually edit the following file

   • nano /var/opt/yarnlab/yarnman/docker-compose-override.yml

   • Code Block
     version: '3.7' services: yarnman: ports: - "3999:3999" expose: - "3999"

   • Ensure that the top row show version: '3.7'

3. Create the 2nd Administration application and ensure the port select matches what is set for ports and expose in docker-compose-override.yml

   • 

4. Restart yarnman services

   • sudo ym-service-commands.sh restart

5. You will now be able to access the second administration application on port 3999 using https://<IP address>:3999/

   • NOTE that http to https redirect will not work on this port and https:// must be entered

   • It is suggested to use in private browser or similar as the authentication sessions will conflict with LDAP users and the older session will close

   • 

Enable database access for Replication

...

Changing private key default passphrase

1. This step requires root access

   • to switch to root access run the following command “su root” and enter the root password set during installation

   • If the encryption at rest process has been run previously the private key must be decrypted

     • If the key is not encrypted skip the decryption step

     • to verify run the following command If no file is found that means the key is encrypted

     • Code Block
       ls -la /var/opt/yarnlab/yarnman/config/private-encryption-key.pem ls: cannot access '/var/opt/yarnlab/yarnman/config/private-encryption-key.pem': No such file or directory

     • to verify the key is encrypted

     • Code Block
       ls -la /var/opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe -rw-r--r-- 1 ym-yarnman-app ym-yarnman-app-gp 8129 Nov 14 03:40 /var/opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe

2. Switch into docker container by running the following command - Note that the prompt changes from the root to container shell

   • Code Block
     docker exec -it ym-yarnman /bin/bash ym-yarnman-app@yl-ym-yarnman:/opt/yarnlab/yarnman$

3. to decrypt the key run the following command

   • Code Block
     clevis decrypt < /opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe > /opt/yarnlab/yarnman/config/private-encryption-key.pem

4. reset permissions

   • chmod 600 /opt/yarnlab/yarnman/config/private-encryption-key.pem

5. change passphrase from default “yarnman”

   • Code Block
     ssh-keygen -p -f /opt/yarnlab/yarnman/config/private-encryption-key.pem Enter old passphrase: Enter new passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved with the new passphrase.

6. backup old key

   • Code Block
     mv /opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe /opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe.bk

7. exit the container shell

   • Code Block
     exit exit root@ym-ph-test [ /var/opt/yarnlab/ ]#

   • verify the key is decrypted and ensure the that

   • Code Block
     ls -la /var/opt/yarnlab/yarnman/config/private-encryption-key.pem -rw-r--r-- 1 ym-yarnman-app ym-yarnman-app-gp 3326 Nov 20 20:21 /var/opt/yarnlab/yarnman/config/private-encryption-key.pem

8. add new passphrase to /var/opt/yarnlab/yarnman/config/local.yaml

   • Code Block
     encryption: dbPassphrase: 'Clouduc123'

   • TO update yq -i '.encryption.dbPassphrase = "Clouduc123"' /var/opt/yarnlab/yarnman/config/local.yaml

9. encrypt passphrase

   • Code Block

     docker exec ym-yarnman node ./scripts/encrypt-local-config.js -k encryption.dbPassphrase 1668977064139 INFO Starting the encryption of 1 local configuration fields through Clevis Shamir Secret Sharing 1668977064142 INFO Attempting to encrypt the following local config fields: encryption.dbPassphrase 1668977064371 INFO Local key 'encryption.dbPassphrase' encrypted successfully 1668977064371 INFO 1 local config fields encrypted, 0 fields omitted

   • verify

     • Code Block
       cat cat /var/opt/yarnlab/yarnman/config/local.yaml

10. re encrypt keys

    • Code Block

      docker exec ym-yarnman node ./scripts/encrypt-keys.js 1668977138519 INFO Encrypting private and SSL keys using settings: 1668977138521 INFO - not overwriting existing encrypted files and not deleting any original files after encryption 1668977138522 INFO -------------------------------- 1668977138522 INFO Encrypting... 1668977138768 INFO - 'private-encryption-key.pem' encrypted successfully 1668977138768 INFO - 'ssl-key.pem' already encrypted, not overwriting 1668977138768 INFO -------------------------------- 1668977138768 INFO Finished encrypting the files

11. restart services

    • Code Block
      systemctl restart yarnman

    • verify while services are restarting look for

      • Code Block

        docker logs ym-yarnman -f 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206414 INFO Decrypting 1 encrypted configuration keys 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206415 INFO Decrypting configuration key 'encryption.dbPassphrase' 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206500 INFO Configuration key 'encryption.dbPassphrase' decrypted successfully 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206500 INFO Finished decrypting 1 configuration keys

Setup Couchdb Replication

1. login to yarnman administration application web interface

2. Navigate to Authentication database

   • Rename the Default Authentication Database name from “Central DB” to “<Node Name> Central DB” or other suitably unique name

   • 

3. Navigate to Authentication Policies

   • Rename the Default Authentication Policy name from “Central DB-Only Policy” to “<Node Name> Central DB-Only Policy” or other suitably unique name

   • 

4. Navigate to Nodes and select the Standalone node

   • 

5. Update the yarnman node name

   • 

6. Navigate to Nodes , select the node you wanted to setup and click on the Replication tab

   • 

7. Click on Add Replication

   • 

   • Enter the source and target connection strings

     • Source Address Syntax https://10.222.1.4:6984 - NOTE that the default source address is 10.222.1.4

       • Source address: https://10.222.1.4:6984

       • Source username: architect

       • source password: yarnman

     • Remote address Syntax https://<TargetIP>:6984

       • Target address https://10.101.10.10:6984

       • Target username: architect

       • Target password: yarnman

8. Once replication is setup status can be reviewed by clicking on the replication address, eg https://10.101.10.10:6984 . If the replication shows blank, the Sync button can be pressed to kick off replication again.

   1. 
   2. 

9. Repeat for each pair of nodes to achieve a full mesh

   • If there are 2 datacenters repeat for each primary node in each data centre -

     • 2 node - 2 replications

       • n1->n2

       • n2->n1

     • 3 node - 6 replications

       • n1->n2

       • n1->n3

       • n2->n1

       • n2->n3

       • n3->n1

       • n3->n2

     • 4 node - 12 replications

       • n1->n2

       • n1->n3

       • n1->n4

       • n2->n1

       • n2->n3

       • n2->n4

       • n3->n1

       • n3->n2

       • n3->n4

       • n4->n1

       • n4->n2

       • n4->n3

10. if you have any issues with replications in state failing run the following command and review the log messages

    • https://yarnlab.atlassian.net/wiki/spaces/YMN/pages/2822111233/Yarnman+PH4+Deployment+and+Installation#ym-service-commands.sh-yarnman-logs

Yarnman HTTP Certificate Notes

This is a manual proces until

Generate CSR

1. Switch user to root

   • Code Block
     su root

2. Run the following command to create the CSR request config file

   • Code Block
     nano /var/opt/yarnlab/yarnman/config/yarnman-ssl.cnf

   • add copy the following contenst and replace <FQDN>with the Fully Quailifed Domain Name of the server

   • Code Block
     [req] distinguished_name = req_distinguished_name req_extensions = v3_req [ req_distinguished_name ] emailAddress = Email Address (emailAddress_max = 64) [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = <FQDN>

3. Run the following command to generate the CSR

   • Command Syntax

     • Code Block
       openssl req -config /var/opt/yarnlab/yarnman/config/yarnman-ssl.cnf -new -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCATION}/O=${ORGANIZATION}/OU=${FUNCTION}/CN=${FQDN}" \ -out /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr -key /var/opt/yarnlab/yarnman/config/ssl-key.pem -passin pass:yarnman -sha512 -newkey rsa:4096

   • All of the following need to be replaced

     • ${COUNTRY}

     • ${STATE}

     • ${LOCATION}

     • ${ORGANIZATION}

     • ${FUNCTION}

     • ${FQDN

   • Example

     • Code Block
       openssl req -config /var/opt/yarnlab/yarnman/config/yarnman-ssl.cnf -new -subj "/C=AU/ST=NSW/L=SYDNEY/O=yarnlab/OU=lab/CN=yarnman.test.yarnlab.io" \ -out /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr -key /var/opt/yarnlab/yarnman/config/ssl-key.pem -passin pass:yarnman -sha512 -newkey rsa:4096

4. Collect CSR for signing

   1. Option 1- SFTP download from /var/opt/yarnlab/upgrade/

      1. cp /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr /var/opt/yarnlab/yarnman/upgrade/yarnman-ssl.csr

   2. Option 2 - copy content to new file yarnman-ssl.cnf

      1. cat /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr

5. Once signed certificate has been received from CA

   1. Review if certificate has intermediate CA siging and follow process below

6. Backup existing SSL public certificate

   • Code Block
     cp /var/opt/yarnlab/yarnman/config/ssl-cert.cert /var/opt/yarnlab/yarnman/config/ssl-cert.cert.bk

   • Code Block
     cat /var/opt/yarnlab/yarnman/config/ssl-cert.cert

7. Update public certificatge

   1. Option 1

      • upload to /var/opt/yarnlab/yarnman/upgrade/ssl-cert.cert

      • Code Block
        rm /var/opt/yarnlab/yarnman/config/ssl-cert.cert mv /var/opt/yarnlab/yarnman/upgrade/ssl-cert.cert /var/opt/yarnlab/yarnman/config/ssl-cert.cert systemctl restart yarnman

   2. nano /var/opt/yarnlab/yarnman/config/ssl-cert.cert

      • Code Block
        systemctl restart yarnman

Verification

PENDING openssl verification commands 

Configuring Intermediate CA Certificates

Typical format for standard SSL.

...

chmod 755 /var/opt/yarnlab/yarnman/config/ca
chmod 644 /var/opt/yarnlab/yarnman/config/ca/*.crt

Photon iptables

In order to have perstitant firewall rules for docker containers, we need to populate the DOCKER-USER table, this is processed by iptables before traffic hits the host, hence we can’t apply the firewall rules directly on INPUT table (used by eth0)

...

[ /var/home/yarnman ]# iptables -t filter -vL --line-numbers
Chain INPUT (policy DROP 161 packets, 9805 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1       70  5662 ACCEPT     all  --  lo     any     anywhere             anywhere            
2      321 24962 ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
3        1    64 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:22

  

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 RETURN     tcp  --  eth0    any     10.202.30.10         anywhere             tcp dpt:6984 /* Allow CouchDB Traffic -  */
2        9   732 RETURN     tcp  --  eth0    any     10.202.30.11         anywhere             tcp dpt:6984 /* Allow CouchDB Traffic -  */
3        0     0 RETURN     tcp  --  eth0    any     10.101.10.36         anywhere             tcp dpt:6984 /* Allow CouchDB Traffic -  */
4       55  3300 DROP       tcp  --  eth0    any     anywhere             anywhere             tcp dpt:6984 /* block non replication Coucdb nodes -  */
5     3591  264K RETURN     all  --  any    any     anywhere             anywhere  

default ip4save

This is the default of /etc/systemd/scripts/ip4save

...

# init
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:DOCKER-USER - [0:0]
# Allow local-only connections
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#keep commented till upgrade issues are sorted
#-A INPUT -j LOG --log-prefix "FIREWALL:INPUT "
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A OUTPUT -j ACCEPT
-A DOCKER-USER -i eth0 -p tcp -s 10.202.30.10,10.202.30.11,10.101.10.36 --dport 6984 -m comment --comment "Allow CouchDB Traffic - " -j RETURN
-A DOCKER-USER -i eth0 -p tcp -s 0.0.0.0/0 --dport 6984 -m comment --comment "block non replication Coucdb nodes - " -j DROP
-A DOCKER-USER -i eth0 -p tcp -s 10.202.30.10,10.202.30.11,10.101.10.36 --dport 6655 -m comment --comment "Allow ClevisTang Traffic - " -j RETURN
-A DOCKER-USER -i eth0 -p tcp -s 0.0.0.0/0 --dport 6655 -m comment --comment "block non ClevisTang nodes - " -j DROP
COMMIT

Logging

...

-A DOCKER-USER -i eth0 -p tcp -s 0.0.0.0/0 --dport 6984 -m comment --comment "block non replication Coucdb nodes - " -m limit --limit 5/min -j LOG --log-prefix "couchdb drop -"
-A DOCKER-USER -i eth0 -p tcp -s 0.0.0.0/0 --dport 6984 -m comment --comment "block non replication Coucdb nodes - " -j DROP

you can view these in dmesg as root

root@yarnman-1 [ /var/home/yarnman ]# dmesg
[   34.799506] couchdb drop -IN=eth0 OUT=br-7cee03840940 MAC=00:50:56:9f:04:4f:02:50:56:56:44:52:08:00 SRC=10.101.10.86 DST=10.222.1.4 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=61886 DF PROTO=TCP SPT=59210 DPT=6984 WINDOW=42340 RES=0x00 SYN URGP=0 

Monitoring

watch can be used to repeat the same command, to watch the counters increase

...