1:34 p.m.
Hello everyone
I just spent a couple of hours on the owncloud role. My understanding is
that this role does not support acme to automatically produce
letsencrypt certificates. Is this correct?
Dam
1:47 p.m.
On sie 25, Damiano Venturin wrote:
I just spent a couple of hours on the owncloud role. My
understanding
is that this role does not support acme to automatically produce
letsencrypt certificates. Is this correct?
That's correct. An application role does not need to concern itself with X.509
certificates that are used by the frontend webserver.
If you haven't noticed, DebOps roles are designed a bit differently than
"normal" Ansible roles on Ansible Galaxy. Various roles in the project expose
a "public API" in the form of dependent variables and custom Ansible local
facts. Other roles can then use these mechanisms to interface with DebOps
roles.
In this specific case, the 'owncloud' role uses the 'nginx' role to
configure
the frontend nginx webserver (this is readily apparent in the
'service/owncloud.yml' playbook). In turn, the 'nginx' role configures
the
nginx server to support ACME authentication challenges issued by Let's
Encrypt. Then the 'pki' role, which maintains the X.509 certificate
infrastructure in DebOps, can use the configured nginx instance to request
Let's Encrypt certificates, which then can be picked up by the 'nginx' role
which configures the nginx webserver to use them. You can find more details
about this process in the pki[1] and nginx[2] role documentation.
As you can see, the whole process does not involve the 'owncloud' role at all.
This also means, that other application roles like 'netbox', 'phpipam',
'librenms', 'gitlab', etc. can also benefit from the same underlying
mechanism, and do not require custom repeated solutions to handle Let's
Encrypt certificates themselves.
Let me know if you have any more questions.
Maciej
[1]: https://docs.debops.org/en/master/ansible/roles/pki/acme-integration.html
[2]: https://docs.debops.org/en/master/ansible/roles/nginx/acme-support.html
2 p.m.
On 8/25/20 1:47 PM, Maciej Delmanowski wrote:
On sie 25, Damiano Venturin wrote:
> I just spent a couple of hours on the owncloud role. My understanding
> is that this role does not support acme to automatically produce
> letsencrypt certificates. Is this correct?
That's correct. An application role does not need to concern itself with X.509
certificates that are used by the frontend webserver.
If you haven't noticed, DebOps roles are designed a bit differently than
"normal" Ansible roles on Ansible Galaxy. Various roles in the project expose
a "public API" in the form of dependent variables and custom Ansible local
facts. Other roles can then use these mechanisms to interface with DebOps
roles.
In this specific case, the 'owncloud' role uses the 'nginx' role to
configure
the frontend nginx webserver (this is readily apparent in the
'service/owncloud.yml' playbook). In turn, the 'nginx' role configures
the
nginx server to support ACME authentication challenges issued by Let's
Encrypt. Then the 'pki' role, which maintains the X.509 certificate
infrastructure in DebOps, can use the configured nginx instance to request
Let's Encrypt certificates, which then can be picked up by the 'nginx' role
which configures the nginx webserver to use them. You can find more details
about this process in the pki[1] and nginx[2] role documentation.
As you can see, the whole process does not involve the 'owncloud' role at all.
makes sense
This also means, that other application roles like 'netbox',
'phpipam',
'librenms', 'gitlab', etc. can also benefit from the same underlying
mechanism, and do not require custom repeated solutions to handle Let's
Encrypt certificates themselves.
Let me know if you have any more questions.
Maciej
[1]:https://docs.debops.org/en/master/ansible/roles/pki/acme-integration.html
[2]:https://docs.debops.org/en/master/ansible/roles/nginx/acme-support.html
so infact to make it work I added to owncloud-nginx.yml
- role: pki
tags: ["role::pki", "skip::pki"]
pki_dependent_realms: "{{ owncloud__pki_dependent_realms }}"
when: (owncloud__pki_dependent_realms|length>0)
and then I added this in the owncloud.yml variables
owncloud__pki_dependent_realms:
- name: "cloud.example.com"
enabled: True
acme: True
#acme_ca: "le-staging-v2" #for testing
acme_ca: "le-live-v2"#for production
acme_default_subdomains: []
acme_subdomains: []
but now that you've explained I see how this is unnecessary.
Perhaps all I shall do is to add in nginx.yml variables file
nginx__pki_dependent_realms:
- name: "cloud.example.com"
enabled: True
acme: True
#acme_ca: "le-staging-v2" #for testing
acme_ca: "le-live-v2"#for production
acme_default_subdomains: []
acme_subdomains: []
and this should do the trick, right?
I'll go through the doc again anyway.
2:22 p.m.
On sie 25, Damiano Venturin wrote:
so infact to make it work I added to owncloud-nginx.yml
- role: pki
tags: ["role::pki", "skip::pki"]
pki_dependent_realms: "{{ owncloud__pki_dependent_realms }}"
when: (owncloud__pki_dependent_realms|length>0)
and then I added this in the owncloud.yml variables
owncloud__pki_dependent_realms:
- name: "cloud.example.com"
enabled: True
acme: True
#acme_ca: "le-staging-v2" #for testing
acme_ca: "le-live-v2"#for production
acme_default_subdomains: []
acme_subdomains: []
but now that you've explained I see how this is unnecessary.
Perhaps all I shall do is to add in nginx.yml variables file
nginx__pki_dependent_realms:
- name: "cloud.example.com"
enabled: True
acme: True
#acme_ca: "le-staging-v2" #for testing
acme_ca: "le-live-v2"#for production
acme_default_subdomains: []
acme_subdomains: []
and this should do the trick, right?
If you write custom playbooks that use these roles, then sure, that could be
a solution. In DebOps however, the 'pki' role is part of the 'common.yml'
playbook executed on all hosts. The role is designed in such a way that it's
not needed to include it in other playbooks - roles that want to use the PKI
infrasteructure can do so just by getting information from the
'ansible_local.pki.*' local facts.
The initialization of ACME certificates is a bit involved process, because for
'pki' role to correctly get Let's Encrypt certs, it requires a working
webserver. So the usual procedure is:
- run common playbook that includes 'pki'
- run nginx role
- run 'service/pki' playbook that re-runs the 'pki' role which detects
configured 'nginx' server and requests ACME certs where apropriate.
The 'pki' role configures the 'pki-realm' script to renew the
certificates
preiodically, so you don't need to do that yourself.
-- Maciej
2:35 p.m.
If you write custom playbooks that use these roles, then sure, that
could be
a solution. In DebOps however, the 'pki' role is part of the
'common.yml'
playbook executed on all hosts. The role is designed in such a way that it's
not needed to include it in other playbooks - roles that want to use the PKI
infrasteructure can do so just by getting information from the
'ansible_local.pki.*' local facts.
I do use custom playbooks but I'm
striving to keep them at minimum.
Ideally I'd keep none. I see no point in keeping a custom role for
something that can work out of the box
The initialization of ACME certificates is a bit involved process,
yes. The PKI role is rather tough. I already went through the docs
several times and I remember reading the role itself a few times but it
still is opaque to me and I always make some mistakes. Perhaps I need to
spend more time on it and go deeper.
because for
'pki' role to correctly get Let's Encrypt certs, it requires a working
webserver. So the usual procedure is:
- run common playbook that includes 'pki'
- run nginx role
- run 'service/pki' playbook that re-runs the 'pki' role which detects
configured 'nginx' server and requests ACME certs where apropriate.
I'm on it. I'll let you know how it goes.
The 'pki' role configures the 'pki-realm' script to renew the
certificates
preiodically, so you don't need to do that yourself.
I see you have developed
clairvoyance: you already answered my next
question :-)
>
> -- Maciej
4:04 p.m.
On 8/25/20 2:22 PM, Maciej Delmanowski wrote:
If you write custom playbooks that use these roles, then sure, that
could be
a solution. In DebOps however, the 'pki' role is part of the
'common.yml'
playbook executed on all hosts. The role is designed in such a way that it's
not needed to include it in other playbooks - roles that want to use the PKI
infrasteructure can do so just by getting information from the
'ansible_local.pki.*' local facts.
The initialization of ACME certificates is a bit involved process, because for
'pki' role to correctly get Let's Encrypt certs, it requires a working
webserver. So the usual procedure is:
- run common playbook that includes 'pki'
- run nginx role
- run 'service/pki' playbook that re-runs the 'pki' role which detects
configured 'nginx' server and requests ACME certs where apropriate.
The 'pki' role configures the 'pki-realm' script to renew the
certificates
preiodically, so you don't need to do that yourself.
The logic is clear and helped me a lot!
The actuation did present some difficulties (mostly because I was
overthinking) so I'm reporting here the steps I've followed.
If they are incorrect please correct me otherwise they'll be a reference
for someone else.
Say that:
* your domain is example.com
* the subdomain for nextcloud is cloud.example.com
* the host running nextcloud is nc.example.xyz
These are the var files I've set in nc's inventory folder
(ansible/inventory/host_vars/nc/)
pki.yml
---
pki_enabled: True
pki_acme: True
pki_acme_home_create: False
pki_realms:
- name: "cloud.{{ ansible_domain }}"
acme: True
#acme_ca: "le-staging-v2" #for testing
acme_ca: "le-live-v2"#for production
Basically this file says: use Nginx acme abilities to produce
Letsencrypt certificates for cloud.example.com
acme_ca simply uses Letsencrypt staging or production API. Keep using
le-staging-v2 until the playbooks work as expected then switch to
production (le-live-v2)
nginx.yml
---
nginx_pki: True
nginx_acme: True
nginx_acme_server: False
This enables Letsencrypt acme capabilities in Nginx
owncloud.yml
---
owncloud__deploy_state: "present"
owncloud__webserver: "nginx"
owncloud__variant: "nextcloud"
owncloud__release: "16"
owncloud__deploy_path_mode: "0755"
owncloud__data_path: "/mnt/data/nextcloud"
owncloud__admin_username: "admin"
owncloud__domain: "{{ ansible_domain }}"
owncloud__fqdn: "cloud.{{ ansible_domain }}"
# Drops db in case of new installation
owncloud__do_drop_db: False
owncloud__recommended_php_packages:
- "curl"
- "bz2"
- "imagick"
- "intl"
- "mcrypt"
owncloud__smb_support: False
owncloud__ldap_enabled: False
owncloud__apcu_enabled: False
owncloud__redis_enabled: True
owncloud__redis_host: "127.0.0.1"
owncloud__redis_password: '{{ lookup("password", secret +
"/redis/clusters/" + ansible_domain + "/password") }}'
owncloud__database: "mariadb"
owncloud__database_server: "localhost"
owncloud__database_name: "{{ owncloud__variant }}"
owncloud__database_user: "{{ owncloud__variant }}"
owncloud__database_password: '{{ lookup("password", secret +
"/mariadb/nc/credentials/nextcloud/password") }}'
owncloud__password_length: 18 #whatever
This simply configures Nextcloud accordingly to my needs
Once the files are done, I make sure that there the nginx servers in
/etc/nginx/sites-*/* are well defined so that nginx won't crash at
reload. I often delete them all.
Then I run:
debops common -l nc
debops service/owncloud
debops service/pki
and everything works fine.
Thanks Maciej for the usual impeccable support!
4:10 p.m.
New subject: how to test pki-realm script? Was [Re: owncloud and letsencrypt]
On 8/25/20 2:22 PM, Maciej Delmanowski wrote:
The 'pki' role configures the 'pki-realm' script to
renew the
certificates
preiodically, so you don't need to do that yourself.
I see the cron job in /etc/cron.d which runs weekly
"/usr/local/lib/pki/pki-realm schedule"
which is a 2500 lines bash script accepting init, new-realm, schedule as
options
Now how can I test that the script is working as expected (beside
waiting 90 days) ?
4:22 p.m.
New subject: how to test pki-realm script? Was [Re: owncloud and letsencrypt]
On sie 25, Damiano Venturin wrote:
On 8/25/20 2:22 PM, Maciej Delmanowski wrote:
> The 'pki' role configures the 'pki-realm' script to renew the
> certificates
> preiodically, so you don't need to do that yourself.
I see the cron job in /etc/cron.d which runs weekly
"/usr/local/lib/pki/pki-realm schedule"
which is a 2500 lines bash script accepting init, new-realm, schedule
as options
Now how can I test that the script is working as expected (beside
waiting 90 days) ?
It's tricky. The script checks the validity of the certificate and only
requests a new one when the current is near the end of the valid period, to
avoid hitting Let's Encrypt rate limits. However, if you got the certificate
just fine the first time, the renewal should work. If it doesn't, there will
be an error.log file present in the realm directory with details about the
issue.
You can schedule the next run from the root account by executing:
/usr/local/lib/pki/pki-realm schedule
The script will check all the realms and any ACME certs will be renewed if
they are near the end of their vailidity period. I suppose that if you want to
really test the mechanism, you could modify the 'pki-realm' script to check
validity and renew when there are 90+ days left, but I would do that only with
acme-staging CA to be sure.
The whole 'pki' role was designed in 2015 so it's pretty old and needs to be
refreshed at some point. The scripts could also be rewritten in Python to be
more manageable, but that's a bit more involved. I'm currently working on
rewritten 'debops' scripts, which could incorporate the support for PKI on the
Ansible Controller side; when that's done, the 'pki' role can then be
redesigned to improve the PKI management on the remote side as well. It will
take some time, hopefully in October the new scripts will be ready.
-- Maciej
4:54 p.m.
New subject: how to test pki-realm script? Was [Re: owncloud and letsencrypt]
On 8/25/20 4:22 PM, Maciej Delmanowski wrote:
The script will check all the realms and any ACME certs will be renewed if
they are near the end of their vailidity period.
I think that the key here is
"near"
I suppose that if you want to
really test the mechanism, you could modify the 'pki-realm' script to check
validity and renew when there are 90+ days left, but I would do that only with
acme-staging CA to be sure.
this one is a good solution.
Otherwise another acceptable solution would be to receive a notification
the first time that the scripts fails as there should still be some days
left before the certificates expire (eventually the cron job could be
daily to narrow down the time frame). I need to check Letsencrypt doc to
see how early they accept renewal.
Is there in place any notification mechanism or shall I look into the
script and eventually fire an email?
The whole 'pki' role was designed in 2015 so it's pretty
old and needs to be
refreshed at some point. The scripts could also be rewritten in Python to be
more manageable, but that's a bit more involved.
I personally prefer bash
because my python is limited but ... whatever works
I'm currently working on
rewritten 'debops' scripts, which could incorporate the support for PKI on the
Ansible Controller side; when that's done, the 'pki' role can then be
redesigned to improve the PKI management on the remote side as well.
sexy! :-)
> It will
> take some time, hopefully in October the new scripts will be ready.
8:38 p.m.
New subject: how to test pki-realm script? Was [Re: owncloud and letsencrypt]
On Tue, 2020-08-25 at 16:54 +0200, Damiano Venturin wrote:
Otherwise another acceptable solution would be to receive a
notification
the first time that the scripts fails as there should still be some days
left before the certificates expire (eventually the cron job could be
daily to narrow down the time frame). I need to check Letsencrypt doc to
see how early they accept renewal.
Is there in place any notification mechanism or shall I look into the
script and eventually fire an email?
Probably not what you wanted to hear, but I'm happily using the DebOps Icinga
roles for this ;)
I'm rigorously monitoring all services including all TLS certificates that are
in use. Icinga has check_ssl_cert for this. I use it to monitor:
- Certificate chain validity
- Certificate expiration
- OCSP status
- Common name of certificate issuer (in case the certificate gets reissued by
the wrong CA)
- DANE validity
- All of the above with StartTLS-enabled services as well
Icinga, as well as the Icinga web interface, has a steep initial learning
curve. For me it was absolutely worthwhile.
At work we're also monitoring certificate transparency logs (basically
everything that you can find in https://crt.sh/), but that's more for security
and audit reasons and not availability per se.
But yeah, it would be nice to have some kind of failure notification mechanism
in the pki role.
Imre
8:57 p.m.
New subject: how to test pki-realm script? Was [Re: owncloud and letsencrypt]
On sie 25, Damiano Venturin wrote:
Otherwise another acceptable solution would be to receive a
notification the first time that the scripts fails as there should
still be some days left before the certificates expire (eventually the
cron job could be daily to narrow down the time frame). I need to check
Letsencrypt doc to see how early they accept renewal.
Is there in place any notification mechanism or shall I look into the
script and eventually fire an email?
There's currently no mechanism. However, I think that the best solution would
be to add a mechanism of script hooks, similar to how external certificates
can now be handled via a custom script. The additional hooks could execute
scripts with a specific environment variables. Then you can add whatever
notification method you like - e-mail messages, SNMP traps, icinga
notifications, MQTT messages, and so on.
Perhaps hooking the PKI to something like a RabbitMQ event bus could help you
maintain ACME certs in a distributed environment - you use a shared private
key, one host takes the role of "renewal node" and other hosts listen to the
reneval event and get the new certificates that way. But that should probably
stay optional for now, not everybody wants to set up a RabbitMQ cluster.
If you want to, and you like bash, you can try and add something like that in
the 'pki-realm' script. Looking forward for the pull request. And if not,
perhaps this could be added when the 'pki' role is rewritten.
-- Maciej
4:53 p.m.
New subject: how to test pki-realm script? Was [Re: owncloud and letsencrypt]
On 8/25/20 8:38 PM, Imre Jonk wrote:
Probably not what you wanted to hear, but I'm happily using the
DebOps Icinga
roles for this ;)
on the contrary this is really triggering (masochism?) :-D
I'm rigorously monitoring all services including all TLS
certificates that are
in use. Icinga has check_ssl_cert for this. I use it to monitor:
- Certificate chain validity
- Certificate expiration
- OCSP status
- Common name of certificate issuer (in case the certificate gets reissued by
the wrong CA)
- DANE validity
- All of the above with StartTLS-enabled services as well>
Icinga, as well as the Icinga web interface, has a steep initial learning
curve. For me it was absolutely worthwhile.
so far, every single heavily committed choice I have made has paid back
(Even DVORAK!) and I'm dreaming of a centralized and extensive
monitoring service.
Thanks a lot for this precious insight.
Dam
1337
days inactive
1339
days old
11 comments
3 participants
participants (3)
-
Damiano Venturin -
Imre Jonk -
Maciej Delmanowski