Key and Configuration Encryption

1. Collect the thp (thumbprint for each tang servers individually this is a unique identifier for each nodes tang server

   1. yarnman@ym-ph-test [ ~ ]$ sudo ym-service-commands.sh tang-thp E7SN3eGxrnyoGiHGJBt4GDU8MRw OR if older versions this command was previously yarnman@ym-ph-test [ ~ ]$ sudo ym-service-commands.sh tang-adv E7SN3eGxrnyoGiHGJBt4GDU8MRw

2. Run the key and config encryption

   • Node1 sudo ym-encrypt-at-rest.sh Database key found proceeding Number of pins required for decryption :1 Number of pins this must be equal or greater than the number of pins required for decryption :3 Enter URL for tang server 1 :http://10.101.10.13:6655 Enter THP for tang server 1 :o38piqOs5UwunlcUmqMVwulml34 Connection successful to : http://10.101.10.10:6655 Enter URL for tang server 2 :http://10.101.10.11:6655 Enter THP for tang server 2 :0Lqk7DroJ0g3patTCgTweMUAHPc Connection successful to : http://10.101.10.11:6655 Enter URL for tang server 3 :http://10.101.10.12:6655 Enter THP for tang server 3 :GEpmSTQfz8ctVxdgQEp_rnS3za Connection successful to : http://10.101.10.12:6655 ... Node4 sudo ym-encrypt-at-rest.sh Database key found proceeding Number of pins required for decryption :1 Number of pins this must be equal or greater than the number of pins required for decryption :3 Enter URL for tang server 1 :http://10.101.10.10:6655 Enter THP for tang server 1 :DwLco7FJtXWxFTprQ5M3cojJsZo Connection successful to : http://10.101.10.10:6655 Enter URL for tang server 2 :http://10.101.10.11:6655 Enter THP for tang server 2 :0Lqk7DroJ0g3patTCgTweMUAHPc Connection successful to : http://10.101.10.11:6655 Enter URL for tang server 3 :http://10.101.10.12:6655 Enter THP for tang server 3 :GEpmSTQfz8ctVxdgQEp_rnS3za Connection successful to : http://10.101.10.12:6655

      

Customisation

Yarnman Application Additional Ports

With yarn_man Photon additional steps are required for Adding Secondary Local Auth Administration Access

1. Enable the Additional listening port with the ym-edit-config command

   • yarnman@yarnman-1 [ ~ ]$ sudo ym-edit-config.sh enable-local-admin-access

   • You will be prompted to restart the Yarnman service

   • Port 3999 is the default alternative port to be used

2. The following file is modified by the script. Snippet shown for reference

   • /var/opt/yarnlab/yarnman/docker-compose-override.yml

   • version: '3.7' services: yarnman: ports: - "3999:3999"

3. Create the 2nd Administration application and ensure the port is 3999 and binding address is 0.0.0.0

   • Open

     

4. You will now be able to access the second administration application on port 3999 using https://<IP address>:3999/

   • NOTE that http to https redirect will not work on this port and https:// must be entered

   • It is suggested to use in private browser or similar as the authentication sessions will conflict with LDAP users and the older session will close

   • Open

     

Enable database access for Replication

1. Enable the listening port for couchdb for replication

   • sudo ym-edit-config.sh enable-local-couchdb-access

   • You will be prompted to restart the yarnman service

   • Port 6984 will be used

2. The following file is modified by the script. Snippet shown for reference

   • /var/opt/yarnlab/yarnman/docker-compose-override.yml

   • version: '3.7' services: couchdb: ports: - "6984:6984"

   • Optionally access to couchdb can be restricted to IP addresses using iptables

Changing private key default passphrase

1. This step requires root access

   • to switch to root access run the following command “su root” and enter the root password set during installation

   • If the encryption at rest process has been run previously the private key must be decrypted

     • If the key is not encrypted skip the decryption step

     • to verify run the following command If no file is found that means the key is encrypted

     • ls -la /var/opt/yarnlab/yarnman/config/private-encryption-key.pem ls: cannot access '/var/opt/yarnlab/yarnman/config/private-encryption-key.pem': No such file or directory

        

     • to verify the key is encrypted

     • ls -la /var/opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe -rw-r--r-- 1 ym-yarnman-app ym-yarnman-app-gp 8129 Nov 14 03:40 /var/opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe

2. Switch into docker container by running the following command - Note that the prompt changes from the root to container shell

   • docker exec -it ym-yarnman /bin/bash ym-yarnman-app@yl-ym-yarnman:/opt/yarnlab/yarnman$

3. to decrypt the key run the following command

   • clevis decrypt < /opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe > /opt/yarnlab/yarnman/config/private-encryption-key.pem

4. reset permissions

   • chmod 600 /opt/yarnlab/yarnman/config/private-encryption-key.pem

5. change passphrase from default “yarnman”

   • ssh-keygen -p -f /opt/yarnlab/yarnman/config/private-encryption-key.pem Enter old passphrase: Enter new passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved with the new passphrase.

6. backup old key

   • mv /opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe /opt/yarnlab/yarnman/config/private-encryption-key.pem.jwe.bk

7. exit the container shell

   • exit exit root@ym-ph-test [ /var/opt/yarnlab/ ]#

   • verify the key is decrypted and ensure the that

   • ls -la /var/opt/yarnlab/yarnman/config/private-encryption-key.pem -rw-r--r-- 1 ym-yarnman-app ym-yarnman-app-gp 3326 Nov 20 20:21 /var/opt/yarnlab/yarnman/config/private-encryption-key.pem

8. add new passphrase to /var/opt/yarnlab/yarnman/config/local.yaml

   • encryption: dbPassphrase: 'somepassword'

   • TO update yq -i '.encryption.dbPassphrase = "somepassword"' /var/opt/yarnlab/yarnman/config/local.yaml

9. encrypt passphrase

   • docker exec ym-yarnman node ./scripts/encrypt-local-config.js -k encryption.dbPassphrase 1668977064139 INFO Starting the encryption of 1 local configuration fields through Clevis Shamir Secret Sharing 1668977064142 INFO Attempting to encrypt the following local config fields: encryption.dbPassphrase 1668977064371 INFO Local key 'encryption.dbPassphrase' encrypted successfully 1668977064371 INFO 1 local config fields encrypted, 0 fields omitted

   • verify

     • cat cat /var/opt/yarnlab/yarnman/config/local.yaml

10. re encrypt keys

    • docker exec ym-yarnman node ./scripts/encrypt-keys.js 1668977138519 INFO Encrypting private and SSL keys using settings: 1668977138521 INFO - not overwriting existing encrypted files and not deleting any original files after encryption 1668977138522 INFO -------------------------------- 1668977138522 INFO Encrypting... 1668977138768 INFO - 'private-encryption-key.pem' encrypted successfully 1668977138768 INFO - 'ssl-key.pem' already encrypted, not overwriting 1668977138768 INFO -------------------------------- 1668977138768 INFO Finished encrypting the files

11. restart services

    • systemctl restart yarnman

    • verify while services are restarting look for

      • docker logs ym-yarnman -f 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206414 INFO Decrypting 1 encrypted configuration keys 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206415 INFO Decrypting configuration key 'encryption.dbPassphrase' 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206500 INFO Configuration key 'encryption.dbPassphrase' decrypted successfully 2|administration-app-b6925c3239fc4c878ff6888ce5cb2b51 | 1668977206500 INFO Finished decrypting 1 configuration keys

Setup Couchdb Replication

1. login to yarnman administration application web interface

2. Navigate to Authentication database

   • Rename the Default Authentication Database name from “Central DB” to “<Node Name> Central DB” or other suitably unique name

   • Open

     

3. Navigate to Authentication Policies

   • Rename the Default Authentication Policy name from “Central DB-Only Policy” to “<Node Name> Central DB-Only Policy” or other suitably unique name

   • Open

     

      

4. Navigate to Nodes and select the Standalone node

   • Open

     

5. Update the yarnman node name

   • Open

     

      

6. Navigate to Nodes , select the node you wanted to setup and click on the Replication tab

   • Open

     

7. Click on Add Replication

   • Open

     

   • Enter the source and target connection strings

     • Source Address Syntax https://10.222.1.4:6984 - NOTE that the default source address is 10.222.1.4

       • Source address: https://10.222.1.4:6984

       • Source username: architect

       • source password: somepassword

     • Remote address Syntax https://<TargetIP>:6984

       • Target address https://10.101.10.10:6984

       • Target username: architect

       • Target password: somepassword

8. Once replication is setup status can be reviewed by clicking on the replication address, eg https://10.101.10.10:6984 . If the replication shows blank, the Sync button can be pressed to kick off replication again.

   1. Open

      
   2. Open

      

    

9. Repeat for each pair of nodes to achieve a full mesh

   • If there are 2 datacenters repeat for each primary node in each data centre -

     • 2 node - 2 replications

       • n1->n2

       • n2->n1

     • 3 node - 6 replications

       • n1->n2

       • n1->n3

       • n2->n1

       • n2->n3

       • n3->n1

       • n3->n2

     • 4 node - 12 replications

       • n1->n2

       • n1->n3

       • n1->n4

       • n2->n1

       • n2->n3

       • n2->n4

       • n3->n1

       • n3->n2

       • n3->n4

       • n4->n1

       • n4->n2

       • n4->n3

Yarnman HTTP Certificate Notes

This is a manual proces until https://yarnlab.atlassian.net/browse/YMN-4962

Generate CSR

1. Switch user to root

   • su root

2. Run the following command to create the CSR request config file

   • nano /var/opt/yarnlab/yarnman/config/yarnman-ssl.cnf

   • copy the following content and replace servername.example.com with the Fully Qualified Domain Name of the server and sample@sample.com with an email address.

   • [req] distinguished_name = req_distinguished_name req_extensions = v3_req [ req_distinguished_name ] emailAddress = sample@sample.com [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = servername.example.com

3. Run the following command to generate the CSR

   • Command Syntax

     • openssl req -config /var/opt/yarnlab/yarnman/config/yarnman-ssl.cnf -new -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCATION}/O=${ORGANIZATION}/OU=${FUNCTION}/CN=${FQDN}" \ -out /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr -key /var/opt/yarnlab/yarnman/config/ssl-key.pem -passin pass:somepassword -sha512 -newkey rsa:4096

   • All of the following need to be replaced

     • ${COUNTRY}

     • ${STATE}

     • ${LOCATION}

     • ${ORGANIZATION}

     • ${FUNCTION}

     • ${FQDN}

   • Example

     • openssl req -config /var/opt/yarnlab/yarnman/config/yarnman-ssl.cnf -new -subj "/C=AU/ST=NSW/L=SYDNEY/O=yarnlab/OU=lab/CN=yarnman.test.yarnlab.io" \ -out /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr -key /var/opt/yarnlab/yarnman/config/ssl-key.pem -passin pass:yarnman -sha512 -newkey rsa:4096

4. Collect CSR for signing

   1. Option 1- SFTP download from /var/opt/yarnlab/upgrade/

      1. cp /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr /var/opt/yarnlab/yarnman/upgrade/yarnman-ssl.csr

   2. Option 2 - copy content from your ssh terminal to obtain the base64 text

      1. cat /var/opt/yarnlab/yarnman/config/yarnman-ssl.csr

5. Once signed certificate has been received from CA

   1. Note if the certificate has intermediate CA, Extra steps will be required detailed in the step Configuring Intermediate CA Certificates

6. rename/move existing SSL public certificate

   • mv /var/opt/yarnlab/yarnman/config/ssl-cert.cert /var/opt/yarnlab/yarnman/config/ssl-cert.cert.bk

7. Update public certificate

   • Option 1
     upload to /tmp from your sftp program, set/validate correct permission and restart yarnman

     • mv /tmp/certname.something /var/opt/yarnlab/yarnman/config/ssl-cert.cert chown ym-yarnman-app:ym-yarnman-app-gp /var/opt/yarnlab/yarnman/config/ssl-cert.cert chmod 644 /var/opt/yarnlab/yarnman/config/ssl-cert.cert systemctl restart yarnman

   • Option 2
     use nano to paste the contents of the base64 certificate

     • nano /var/opt/yarnlab/yarnman/config/ssl-cert.cert chown ym-yarnman-app:ym-yarnman-app-gp /var/opt/yarnlab/yarnman/config/ssl-cert.cert chmod 644 /var/opt/yarnlab/yarnman/config/ssl-cert.cert systemctl restart yarnman

Verification

Verify contents of CSR

Verify the certificate is valid with openssl command

Configuring Intermediate CA Certificates

Typical format for standard SSL.

/var/opt/yarnlab/yarnman/config/

ssl-cert.cert - Standard certificate sent to clients

ssl-key.pem - Private key file for checking response

In order to enable intermediate certificates we must create new directory of ca in in /var/opt/yarnlab/yarnman/config/

add files in order of the cert chain so the files in the directory look like

The /ca folder contains the intermediate certificates that will be loaded in order. The easiest way to achieve this is to use the naming conventions 1-, 2- etc. Each certificate must end in .crt in order to be loaded.

Once the folder is created and at least one certificate is added in the format indicated the services on the node must be restarted.

File permissions should be as follow

If required to change the owner and group

If required to change the permissions

restart yarnman

Verification

If you have multiple intermediates

Photon iptables

In order to have perstitant firewall rules for docker containers, we need to populate the DOCKER-USER table, this is processed by iptables before traffic hits the host, hence we can’t apply the firewall rules directly on INPUT table (used by eth0)

In this example we will allow traffic to couchdb from ip address

• 10.202.30.10

• 10.202.30.11

• 10.101.10.36

You will need to su as root to modify this file.

Modify the Existing ruleset applied on startup /etc/systemd/scripts/ip4save

We need to add the table/filter :DOCKER-USER - [0:0] under the existing filter/table list and the required firewall rules at the bottom before the COMMIT

The File will look similar to

Reload the server for the firewall rules to take affect

You can verify the ruleset with the command in a root prompt

some output removed from the other tables and docker internal but this shows 10.202.30.10 , 10.202.30.11, 10.101.10.36 can communicate with TCP/6984 and everything else is dropped . You can see 55 packets have been blocked, and 9 packets have been allowed from 10.202.30.11

default ip4save

This is the default of /etc/systemd/scripts/ip4save

on the off-chance you need to rollback to it

To add other firewall rules eg allow traffic only between 3 ips to Clevins-Tang insert them above the COMMIT line eg

Final file will look like /etc/systemd/scripts/ip4save

Logging

With IPtables you need to LOG before dropping the packet, the simplest way is to duplicate the rule with the LOG jump

Monitoring

watch can be used to repeat the same command, to watch the counters increase

Highlight keywords from Log and current packet counts

Open