On sie 25, Damiano Venturin wrote: That's correct. An application role does not need to concern itself with X.509 certificates that are used by the frontend webserver. If you haven't noticed, DebOps roles are designed a bit differently than "normal" Ansible roles on Ansible Galaxy. Various roles in the project expose a "public API" in the form of dependent variables and custom Ansible local facts. Other roles can then use these mechanisms to interface with DebOps roles. In this specific case, the 'owncloud' role uses the 'nginx' role to configure the frontend nginx webserver (this is readily apparent in the 'service/owncloud.yml' playbook). In turn, the 'nginx' role configures the nginx server to support ACME authentication challenges issued by Let's Encrypt. Then the 'pki' role, which maintains the X.509 certificate infrastructure in DebOps, can use the configured nginx instance to request Let's Encrypt certificates, which then can be picked up by the 'nginx' role which configures the nginx webserver to use them. You can find more details about this process in the pki[1] and nginx[2] role documentation. As you can see, the whole process does not involve the 'owncloud' role at all. This also means, that other application roles like 'netbox', 'phpipam', 'librenms', 'gitlab', etc. can also benefit from the same underlying mechanism, and do not require custom repeated solutions to handle Let's Encrypt certificates themselves. Let me know if you have any more questions. Maciej [1]: https://docs.debops.org/en/master/ansible/roles/pki/acme-integration.html [2]: https://docs.debops.org/en/master/ansible/roles/nginx/acme-support.html

On sie 25, Damiano Venturin wrote: If you write custom playbooks that use these roles, then sure, that could be a solution. In DebOps however, the 'pki' role is part of the 'common.yml' playbook executed on all hosts. The role is designed in such a way that it's not needed to include it in other playbooks - roles that want to use the PKI infrasteructure can do so just by getting information from the 'ansible_local.pki.*' local facts. The initialization of ACME certificates is a bit involved process, because for 'pki' role to correctly get Let's Encrypt certs, it requires a working webserver. So the usual procedure is: - run common playbook that includes 'pki' - run nginx role - run 'service/pki' playbook that re-runs the 'pki' role which detects configured 'nginx' server and requests ACME certs where apropriate. The 'pki' role configures the 'pki-realm' script to renew the certificates preiodically, so you don't need to do that yourself. -- Maciej

On 8/25/20 2:22 PM, Maciej Delmanowski wrote: The logic is clear and helped me a lot! The actuation did present some difficulties (mostly because I was overthinking) so I'm reporting here the steps I've followed. If they are incorrect please correct me otherwise they'll be a reference for someone else. Say that: * your domain is example.com * the subdomain for nextcloud is cloud.example.com * the host running nextcloud is nc.example.xyz These are the var files I've set in nc's inventory folder (ansible/inventory/host_vars/nc/) pki.yml --- pki_enabled: True pki_acme: True pki_acme_home_create: False pki_realms: - name: "cloud.{{ ansible_domain }}" acme: True #acme_ca: "le-staging-v2" #for testing acme_ca: "le-live-v2"#for production Basically this file says: use Nginx acme abilities to produce Letsencrypt certificates for cloud.example.com acme_ca simply uses Letsencrypt staging or production API. Keep using le-staging-v2 until the playbooks work as expected then switch to production (le-live-v2) nginx.yml --- nginx_pki: True nginx_acme: True nginx_acme_server: False This enables Letsencrypt acme capabilities in Nginx owncloud.yml --- owncloud__deploy_state: "present" owncloud__webserver: "nginx" owncloud__variant: "nextcloud" owncloud__release: "16" owncloud__deploy_path_mode: "0755" owncloud__data_path: "/mnt/data/nextcloud" owncloud__admin_username: "admin" owncloud__domain: "{{ ansible_domain }}" owncloud__fqdn: "cloud.{{ ansible_domain }}" # Drops db in case of new installation owncloud__do_drop_db: False owncloud__recommended_php_packages: - "curl" - "bz2" - "imagick" - "intl" - "mcrypt" owncloud__smb_support: False owncloud__ldap_enabled: False owncloud__apcu_enabled: False owncloud__redis_enabled: True owncloud__redis_host: "127.0.0.1" owncloud__redis_password: '{{ lookup("password", secret + "/redis/clusters/" + ansible_domain + "/password") }}' owncloud__database: "mariadb" owncloud__database_server: "localhost" owncloud__database_name: "{{ owncloud__variant }}" owncloud__database_user: "{{ owncloud__variant }}" owncloud__database_password: '{{ lookup("password", secret + "/mariadb/nc/credentials/nextcloud/password") }}' owncloud__password_length: 18 #whatever This simply configures Nextcloud accordingly to my needs Once the files are done, I make sure that there the nginx servers in /etc/nginx/sites-*/* are well defined so that nginx won't crash at reload. I often delete them all. Then I run: debops common -l nc debops service/owncloud debops service/pki and everything works fine. Thanks Maciej for the usual impeccable support!

On sie 25, Damiano Venturin wrote: It's tricky. The script checks the validity of the certificate and only requests a new one when the current is near the end of the valid period, to avoid hitting Let's Encrypt rate limits. However, if you got the certificate just fine the first time, the renewal should work. If it doesn't, there will be an error.log file present in the realm directory with details about the issue. You can schedule the next run from the root account by executing: /usr/local/lib/pki/pki-realm schedule The script will check all the realms and any ACME certs will be renewed if they are near the end of their vailidity period. I suppose that if you want to really test the mechanism, you could modify the 'pki-realm' script to check validity and renew when there are 90+ days left, but I would do that only with acme-staging CA to be sure. The whole 'pki' role was designed in 2015 so it's pretty old and needs to be refreshed at some point. The scripts could also be rewritten in Python to be more manageable, but that's a bit more involved. I'm currently working on rewritten 'debops' scripts, which could incorporate the support for PKI on the Ansible Controller side; when that's done, the 'pki' role can then be redesigned to improve the PKI management on the remote side as well. It will take some time, hopefully in October the new scripts will be ready. -- Maciej

On Tue, 2020-08-25 at 16:54 +0200, Damiano Venturin wrote: Probably not what you wanted to hear, but I'm happily using the DebOps Icinga roles for this ;) I'm rigorously monitoring all services including all TLS certificates that are in use. Icinga has check_ssl_cert for this. I use it to monitor: - Certificate chain validity - Certificate expiration - OCSP status - Common name of certificate issuer (in case the certificate gets reissued by the wrong CA) - DANE validity - All of the above with StartTLS-enabled services as well Icinga, as well as the Icinga web interface, has a steep initial learning curve. For me it was absolutely worthwhile. At work we're also monitoring certificate transparency logs (basically everything that you can find in https://crt.sh/), but that's more for security and audit reasons and not availability per se. But yeah, it would be nice to have some kind of failure notification mechanism in the pki role. Imre

On sie 25, Damiano Venturin wrote: There's currently no mechanism. However, I think that the best solution would be to add a mechanism of script hooks, similar to how external certificates can now be handled via a custom script. The additional hooks could execute scripts with a specific environment variables. Then you can add whatever notification method you like - e-mail messages, SNMP traps, icinga notifications, MQTT messages, and so on. Perhaps hooking the PKI to something like a RabbitMQ event bus could help you maintain ACME certs in a distributed environment - you use a shared private key, one host takes the role of "renewal node" and other hosts listen to the reneval event and get the new certificates that way. But that should probably stay optional for now, not everybody wants to set up a RabbitMQ cluster. If you want to, and you like bash, you can try and add something like that in the 'pki-realm' script. Looking forward for the pull request. And if not, perhaps this could be added when the 'pki' role is rewritten. -- Maciej