[debops-users] First DebOps stable release - v1.0.0
by Maciej Delmanowski
Hello everyone,
DebOps v1.0.0 is here, let's release the HypnoDrones.
New DebOps release, v1.0.0
--------------------------
You can find the new version of DebOps on:
GitHub: https://github.com/debops/debops/releases/tag/v1.0.0
PyPI: https://pypi.python.org/pypi/debops/1.0.0
Galaxy: https://galaxy.ansible.com/debops/debops/ (but see below)
You can upgrade the Python-based installation by running the command:
pip install --upgrade debops
The experimental support for multi-role repositories will be/has been removed
from Ansible Galaxy and Mazer. The Ansible team is instead introducing Ansible
Collections, which use tarballs to distribute Ansible content instead of git
repositories. The Mazer client knows how to use it, but Ansible Galaxy doesn't
yet, therefore new DebOps release will not show up there until June 2019 when
new version of Ansible Galaxy service is released. Until then, you can build
your own tarball with DebOps roles using the 'make collection' command, or
install DebOps via PyPI or directly from the GitHub repository. Installation
instructions can be found here:
https://docs.debops.org/en/v1.0.0/user-guide/install.html
The brief Changelog can also be found on the documentation page:
https://docs.debops.org/en/v1.0.0/news/changelog.html
Complete, detailed changelog can be viewed using the 'git log' command.
The DebOps documentation has a separate page which details important changes
from previous release in the Ansible inventory or on the remote hosts which
you might need to perform manually:
https://docs.debops.org/en/v1.0.0/news/upgrades.html
The Python packages available on PyPI, as well as the tarballs available on
GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver
network using the command:
gpg --keyserver hkp://pool.sks-keyservers.net \
--recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC
The v1.0.0 release contains ~370 commits since the last release in February
2019, not counting merges. Here are the usual suspects:
237 Maciej Delmanowski
32 Alin Alexandru
29 Leonardo Bechea
21 Robin Schneider
10 Imre Jonk
8 Jérémy Gardais
5 Thomas Danielsson
4 Hartmut Goebel
4 Reto Gantenbein
3 Rainer 'rei' Schuth
2 Damiano Venturin
2 Evilham
2 Serge Victor
1 André Jucovsky Bianchi
1 Dionysis Grigoropoulos
1 Jonathan Struebel
1 Julien Palard
1 Mathieu
1 Mike Mindel
1 Nicolas Quiniou-Briand
1 Stefan G. Weichinger
1 Tim Freund
1 monsterry
Thanks to everyone involved in this release, and I see you in the commits.
Goals reached since previous release
------------------------------------
The LDAP support has been completely redesigned from the ground up, both on the
server-side ('debops.slapd' role), and in the new client-side ('debops.ldap'
role). At the moment LDAP can be enabled for basic system services - UNIX
accounts and groups, 'sudo' configuration, SSH public key lookup. There are
still many more DebOps roles which have to be updated to use the new
infrastructure, but that will come in later releases.
Support for Docker Registry comes to DebOps as well, via the
'debops.docker_registry' Ansible role. This role integrates with the
'debops.gitlab' role which you can use to manage the Registry contents.
The rest of the goals outlined in the previous release were not met due to
focus on other things. But there are many new features and bugfixes present, I
suggest you check the Changelog for details.
The first stable release of DebOps is here
------------------------------------------
I decided to make the first stable release of DebOps early due to the fact that
the project is used more and more in production environments, where large
changes of the various roles might not be possible. A few times users of the
v0.8.1 release found bugs in some roles that were fixed in the 'master' branch,
but were unable to switch due to large amount of changes since the previous
release. Because of that, I think that a change of the release process is
needed. There are still many roles that need to be cleaned up and rewritten,
but hopefully a stable branch that will get bugfixes over time will be useful
to the users that require a bit more predictability.
The stable branch, first one named 'stable-1.0', will get bugfixes and security
fixes as long as the code in the 'master' branch is compatible, but there will
be no new features or major changes in the existing roles. If a role in the
'master' branch has been modified too much and direct cherry-picking of the
changes is impossible, the bugfixes might go directly to the stable branch, but
at this point a given release will probably become unsupported when a few more
recent releases are available.
DebOps uses the Semantic Versioning for its releases, with major.minor.patch
numbers. Depending on the changes, major release bump might happen on the next
release, but it's hard to tell what changes might require this. Most likely
large modifications to the Ansible inventory or changes on the remote hosts
that require a host rebuild from scratch will result in major version bump,
less severe changes will have a minor version bump.
The release schedule from the 'master' branch will probably stay the same - a
few months between each release, so that features can be polished and any bugs
fixed beforehand. Updates to the stable branches will be quickier though,
depending on bugfixes.
Plans for the next release
--------------------------
The release of Debian Buster draws closer, and I would like to prepare DebOps
for it beforehand. A migration or possible rewrite of the AppArmor role will
need to happen soon, also the 'debops.ipxe' and 'debops.preseed' roles need to
be updated to support installation of Buster via preseeding.
Various application roles will be updated to use the new 'debops.ldap' role to
manage LDAP entries and access to the directory. I would also like to include
support for Samba and Heimdal Kerberos to reap the benefits of the
'slapd-smbk5pwd' module that allows password synchronization between various
LDAP objects. This should also benefit NFS support which could finally be
properly secured via Kerberos.
Support for installing Golang applications from source will be implemented as
well, compilation will be done on a separate unprivileged account. And as
usual, new applications, rewrites of the old roles and bugfixes here and there.
Thanks for reading and see you out there.
Maciej Delmanowski
5 years, 5 months
[debops-users] Plans for testing DebOps roles
by Nicolas Quiniou-Briand
Hello DebOps community,
I start using DebOps project since few days and I would like to get
involved in DebOps project. I'm reading the mailing-list since 4 months.
My idea is to start by improving DebOps documentation because I'm a
documentation fanatic :-)
Also, I would like to know if plans to using a tool like Testinfra are
still here. I ask this question because I'm currently working on this
subject and I would be interested to contribute on that part.
--
Nicolas Quiniou-Briand
Jabber/XMPP : nqb(a)azyx.fr
5 years, 6 months
[debops-users] icinga_web error with no Endpoint object called
by Jan Kowalsky
Hi Maciej,
hi all,
I get reproducible the following error on service/icinga_web
TASK [debops.icinga_web : Kickstart Icinga Director configuration]
***********************************************************************************************************************************************
fatal: [erdmaennchen.datenkollektiv.net]: FAILED! => {"changed": true,
"cmd": ["icingacli", "director", "kickstart", "run"], "delta":
"0:00:00.260849", "end": "2019-05-23 14:46:38.305353", "msg": "non-zero
return code", "rc": 1, "start": "2019-05-23 14:46:38.044504", "stderr":
"", "stderr_lines": [], "stdout": "ERROR:
Icinga\\Exception\\ConfigurationError in
/usr/local/src/icinga_web/github.com/Icinga/icingaweb2-module-director/library/Director/KickstartHelper.php:348
with message: I found no Endpoint object called \"erdmaennchen\" on
erdmaennchen:5665", "stdout_lines": ["ERROR:
Icinga\\Exception\\ConfigurationError in
/usr/local/src/icinga_web/github.com/Icinga/icingaweb2-module-director/library/Director/KickstartHelper.php:348
with message: I found no Endpoint object called \"erdmaennchen\" on
erdmaennchen:5665"]}
The same thing I got if I run icingacli director kickstart run from
command line. So probably it's not an icinga problem.
But maybe someone can point me to the right direction? I didn't found
anything usable in relation with this error message.
I did until now:
setup a new host with the roles
icinga
icinga_db
icinga_web
one after one.
icinga web is up and running. I can login, I see the localhost - but in
the Icinga-Directory I can't add any hosts. And Icinga doesn't know any
check commands.
On -> Icinga-Director -> Deployments -> create config I get the error:
Unable to detect your deployment endpoint. I was looking for the first
endpoint configured with an assigned API user in the "master" zone.
Migrating from nagios I was hoping not to have to go to deep into icinga...
I checkt already fqdn. It's all fine.
Kind regard
Jan
5 years, 6 months
[debops-users] Removal of old DebOps repositories
by Maciej Delmanowski
Hello everyone,
At the moment the DebOps organization on GitHub[1] has 150 repositories, most
of which are old Ansible roles in separate repo each. This seems to cause
confusion among new and experienced DebOps users regarding where exactly the
project is developed and how to use it effectively. Even the archived
repositories can cause such issues.
After talking about it for a bit on the #debops IRC channel, we came to the
conclusion that the old role repositories, no longer available via Ansible
Galaxy, should be removed from GitHub as well to focus attention on the
DebOps monorepo.
That unfortunately means loss of easy access to the git history, issues and
pull requests. The git history of each role is included in the DebOps
monorepo, although in a rather hard to access state due to the fact that the
files of each role had to be moved to keep the commit signatures intact.
However, the migration to the monorepo happened almost 2 years ago, and
separate role repositories haven't had much changes since, so I think removing
them will be beneficial.
During the weekend, I'll move any open issues from the separate repositories
to the monorepo so that they can be dealt with later on. This will generate
notofication mails about each move, which I cannot disable from the GitHub
interface, so it will be a bit spammy. It looks like GitHub does not support
migration of the pull requests, therefore any open pull requests to the old
roles will be lost.
This also raises the question of using a centralized platform like GitHub or
GitLab for project development vs a dedicated mailing list which could
maintain an archive of normal discussions, discussions about new patches and
bug reports in a consistent and portable format.[2][3] I wonder how many
existing and potential DebOps users would be interested in such a change?
A GitHub and GitLab mirror of course would still be maintained, but perhaps
focus on development using a mailing list would work out better in the long
run? This could also focus discussion about DebOps development and usage on
the mailing list, move to a Mailman3 with newer web interface could also
help.
Don't worry, I don't plan to stop using GitHub issues and pull requests just
yet, just thinking aloud. :)
Cheers,
Maciej
[1]: https://github.com/debops/
[2]: https://git-send-email.io/
[3]: https://drewdevault.com/2018/07/02/Email-driven-git.html
5 years, 7 months
[debops-users] Contributing new roles to DebOps
by Imre Jonk
Hi all,
Quick introduction: I am the system administrator of CipherMail B.V., a
company which specializes in email security, and Stichting Bits of
Freedom, the digital rights movement in the Netherlands. I also
volunteer for a community wireless network in Amsterdam called Amsterdam
Wireless. I've started using DebOps in Februari this year when we were
provisioning a new datacenter at CipherMail. So far it's been really
great and I'm introducing DebOps at Bits of Freedom now as well. Last
year I graduated with a bachelor's thesis about deploying DNSSEC at a
large Dutch hosting company. They went from zero to full DNSSEC with
PowerDNS and some Python scripts I wrote. You can find the thesis
(Dutch) here:
https://www.imrejonk.nl/deploying-dnssec-at-a-web-hosting-company/.
Hobbies and sports include computer tinkering, running, shooting sports,
biking, amateur radio and scuba diving.
To business. I wrote some DebOps-integrated roles at CipherMail which
might be useful to others. My employer has agreed to release the roles
under the GPL version 3 (or later). Some roles lack documentation, tests
and functionality to be useful to anyone, so I'll have to do some work
to improve the overall role quality. I guess I'll do that and create PRs
for new debops-contrib roles soon.
These are the roles I'm talking about:
- amsw.dnsui: provides complete functionality for integrating Opera's
DNS-UI, an LDAP-authenticated web frontend for PowerDNS authoritative
server. The role uses the nginx PAM module for LDAP authentication.
Requires PowerDNS 4.1+ for DNSSEC functionality. Depends on
debops.secret, debops.users, debops.postgresql, debops.php, debops.nginx
and debops.cron.
- amsw.powerdns_auth: a simple role for PowerDNS authoritative server
management. Right now you can only use this role with the Postgresql
backend and the upstream packages (I needed v4.1 for DNSSEC capabilities
on the API). Nonetheless it should be very easy to integrate this with
debops.mariadb and the PowerDNS BIND backend.
- ciphermail.nut: almost complete role for managing a Network UPS Tools
(NUT) server. Needs documentation and tests.
- ciphermail.nut_client: NUT client management.
- ciphermail.cups: manages our CUPS print server. Integrates with
debops.pki and applies some hardening tricks to ensure our print jobs
are always TLS encrypted with verified certificates.
- ciphermail.simplesamlphp: manages a SAML 2.0 iDP-only SimpleSAMLphp
installation, integrated with debops.pki. We use it for Single Sign-On.
- ciphermail.docker_container: simple role for managing Docker containers.
- ciphermail.docker_network: additional Docker role for managing networks.
We've also modified some existing DebOps roles because they weren't
entirely satisfactory for us. We'll contribute these changes back as
well. We're using a lot of DebOps roles :)
Any suggestions (on where I should start, for example) are most welcome!
Cheers,
Imre
5 years, 7 months
[debops-users] Installing package from strech backports
by Hartmut Goebel
Hi,
I try to install a backport from strech-backport, but this does not work
as I expected. While the preferences are set, I did not find an "apt
update" (or such) task to be performed. What am I missing?
role defaults:
ejabberd__apt_preferences__dependent_list:
- packages: ['ejabberd', "erlang-*"]
backports: [ 'stretch' ]
reason: 'Version parity with Debian Buster'
by_role: 'debops.ejabberd'
playbook:
- role: debops.apt_preferences
tags: [ 'role::apt_preferences', 'skip::apt_preferences' ]
apt_preferences__dependent_list:
- '{{ ejabberd__apt_preferences__dependent_list }}'
- role: debops.ejabberd
tags: [ 'role::ejabberd', 'skip::ejabberd' ]
tasks:
- name: Install required packages
package:
name: '{{ item }}'
state: 'present'
with_flattened:
- '{{ ejabberd__base_packages }}'
- '{{ ejabberd__packages }}'
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel(a)crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
5 years, 7 months
Re: [debops-users] Status of debops.prosody and the acme integration
by Maciej Delmanowski
On May 13, Hartmut Goebel wrote:
>> If you still have trouble understanding the DebOps PKI/ACME setup after
>> reading the documentation, then I suppose that the docs need some
>> improvements. Any idea about what could be added to make this setup easier to
>> understand is welcome. :-)
> Indeed I still do not really get this realm thing.
I suppose the explanations below could be included in the docs at some point,
or improve existing documentation.
A "PKI realm" is just a placeholder name for a bunle of the private key, X.509
certificate and Root CA certificate. This bundle has a certain directory
structure, rules for naming files and what symlinks are present. It's designed
so that from the outside of the 'debops.pki', other Ansible roles or services
they manage have a standardized, uniform location where they can find X.509
certificates and private keys.
In different guides that describe setting up TLS for different services like
webservers, mail servers, databases, etc. you can find that people put the
private keys and X.509 certificates in different directories - for example
/etc/nginx/ssl/, /etc/postfix/certs/, /etc/ssl/certs/, and so on. The
'debops.pki' role turns this around by setting up an uniform set of
directories split into "PKI realms", so that a host can have multiple sets of
certificates, each for different purposes. Then, various services can be
configured to get the private key and certificate files from those specific
directories, including privileged access to the private keys when needed.
> I suggest to extend pki-realms.rst to explain how to choose realm-names,
> including some examples. Questions this should answer are e.g.:
>
> * Why is the default realm "domain" and what is the idea behind?
Usually when you set up a cluster of DebOps hosts, they share a DNS domain
together, like '*.example.org'. Thats essentially the "domain" the default PKI
realm refers to. The "domain" PKI realm is supposed to be used for internal
communication between cluster nodes, where you don't need to worry about
services exposed to the public which would probably rather use publicly-signed
certificates, from Let's Encrypt or other Certificate Authorities.
You have full control over the internal CA which signes the "domain"
certificates, they also have multiple levels of wildcards to make internal
service deployment easier. The "domain" PKI realm has the ACME support
explicitly disabled in case you are deploying the hosts on a private network
to which Let's Encrypt might not have access.
The 'debops.pki' is designed in such a way that the whole PKI realm is either
generated dynamically by the role via scripts, or all of its important details
like private keys, externally-signed X.509 certificates, etc. come from the
Ansible Controller. To recreate a PKI realm all you need to do is to remove it
from the remote host (rm -rf /etc/pki/realms/<realm>) and re-run the
'debops.pki' role - it should recreate a given realm automatically based on
its Ansible inventory configuration and contents of the 'secret/' directory.
> * Does the realm-name just a name or does it have another meaning?
> (Could the realm be named e.g. "xxxxaaaayyyy"?)
Originally the 'pki-realm' script supports different naming schemes for PKI
realms:
- the single string name, like "domain", "xxxxaaaayyyy" or whatever you want.
These PKI realm names can be thought of as "handles" and they don't have any
impact on the domains stored in the X.509 certificates. IIRC, the role will
by default use the host's FQDN and DNS domain name to generate such realms,
but you can override that. So, you could easily setup multiple "domain",
"domain2", "domainX" PKI realms that way, for testing or otherwise.
- the DNS-based name, which contains dots, like "example.com",
"host.example.com" and the like. These PKI realms base their X.509
certificates after the realm name by default. This is also the reason the
default PKI realm is named "domain" and not "{{ ansible_domain }}" - you can
create a new PKI realm for your DNS domain on hosts that are reachable
publicly and they will automatically get the Let's Encrypt certificates when
possible, or will let you easily use external certificates grabbed from some
other CA.
Some DebOps roles like 'debops.nginx' can check the list of available PKI
realms via the local facts and use some other PKI realm rather than the
default one automatically. For example, if you create an "example.com" PKI
realm and then use the "example.com" DNS domain, standalone or with
a subdomain like "sub.example.com", the 'debops.nginx' will check if a PKI
realm named after a given FQDN or DNS domain exist and will use it instead
of the "domain" PKI realm used by default. You can think of it as a shortcut
to easily manage X.509 certificates for multiple websites, each one with its
own FQDN domain name.
- the mail-based name, like "user(a)example.org" - any PKI realm name which
contains the '@' character qualifies as one. These PKI realms were meant to
host the client certificates used to authenticate to services, but this idea
was not developed further, so far.
> o If this is different for ACME realms, this should be stated, too.
PKI realms have a concept of multiple certificate authorities - there's one
set of private keys which can be signed by different CAs - internal CA,
external CA, ACME CA and self-signed when everything else is disabled. You can
have an "example.org" PKI realm which has certificates signed by both internal
CA and the Let's Encrypt CA (via ACME), and the 'pki-realm' script will
automatically switch between them after checking the validity of their X.509
certificates.
So, you can name your PKI realms that are meant to be used to hold ACME
certificates however you want, but naming them after a DNS domain name could
make things a bit easier, both when creating the certificate requests, and
when interfacing with other roles like 'debops.nginx'.
> * Is is a good idea to name the realm just "host", and if not, why?
You can name a PKI realm "host", without any further configuration it should
use the DNS domain of the host as a base of its X.509 certificate requests. If
you plan to set up an nginx webserver that uses this PKI realm, you will have
to point to it specifically via the 'item.pki_realm' parameter, because the
'debops.nginx' role will not detect it as a valid server name.
> * The remaining questions might only be relevant if the realm-name has
> some meaning:
> o How shall I name the realm when
> o a) hosting a web-server with virtual domains from other apex
> domains (www.myngo.org, www.mycorp.com),
I would go with:
pki_realms:
- name: 'myngo.org'
acme_subdomains: [ 'www' ]
- name: 'mycorp.com'
acme_subdomains: [ 'www' ]
Assuming that you want to rely on Let's Encrypt to get the certificates for
these domains. If you plan to get them elsewhere, for example from a different
public CA, I would go with:
pki_realms:
- name: 'myngo.org'
acme: False
- name: 'mycorp.com'
acme: False
And put the external private key and certificate files in the 'secret/'
directory (see the documentation about external certificates for details).
> o b) hosting a nginx-server and e.g. a postfix-server for
> different domains.
Here I assume that you mean the nginx server hosts a single service with
multiple separate DNS domains, with Postfix doing the same, which requires
that multiple domains are embedded in a single X.509 certificate. In that
case I would name the PKI realm with a single string, and point the respective
nginx server and Postfix to that particular PKI realm, for example:
pki_realms:
- name: 'multiple-certs'
subject: [ 'o=My Org', 'CN=myorg.com' ]
domains:
- 'myorg.org'
- 'myorg.net'
You could test the resulting X.509 certificates a few times to come up with
the best version, then send the generated certificate request to a CA (or
eanble ACME support).
> * Best-practice for configuring realms if different servers (e.g.
> nginx, postfix) server different domains, esp. how and in which
> yml-file to define the pki realms.
I would name the PKI realms based on the DNS domains you use for these
services, optionally including additional subdomains like 'www' if you don't
use wildcard certificates. This again, assuming that you want either Let's
Encrypt certificates, or external certificates from some other CA. You can
define PKI realms in the inventory of a given host, for example
'ansible/inventory/host_vars/<hostname>/pki.yml', or put the common ones in
'ansible/inventory/group_vars/all/pki.yml'.
> If this sounds like a "big" setup, IMHO it is not. For example, on my
> personal server I'm serving one of my web-sites, the web-sited for some
> friends and not want to set up a prosody server for some club I'm a
> member of.
Sure, the number of different services doesn't really matter here, all that
matters is how many DNS domains you want to use to point to these services. At
the end of the day services don't care what set of private key and X.509
certificate they use, usually it's only important that the service has access
to some kind of certs, otherwise TLS doesn't work at all, or a service refuses
to start.
>>> * Changing the pki_system_realm has no effect on the prosody
>>> configuration. I assume since the debops.pki role is not used.
>> First check if the /etc/ansible/facts.d/pki.fact has the correct realm you
>> specified as the default one. Next, er-run the debops.prosody role, it should
>> pick the new realm up from the local facts and modify the Prosody
>> configuration.
>
> What I did:
>
> * run services/prosody
> * found the wrong real is in the prosody config-file
> * changed pki_system_realm the host
> * run services/prosody again
>
> Now since debops.prosody does not run debops.pki, pki.fact was not
> updated. Thus the prosody config-file was still wrong. Worth a bug-report?
Did you change the 'pki_system_realm' in the inventory? You would have to
apply the 'debops.pki' role on the host before running the 'debops.prosody'
role, so that it updated its own local facts.
I dodn't want to create a precedent where 'debops.pki' is used as a dependency
of another role in a playbook. It's used in the common playbook, and at least
in my case certificates are not changed all that often. Otherwise you would
have to add it as a dependency of 'debops.nginx', and this quickly snowballs
to all of the different applications that use debops.nginx as a dependency,
etc.
Cheers,
Maciej
5 years, 7 months
[debops-users] Status of debops.prosody and the acme integration
by Hartmut Goebel
Hi,
I' want to setup a secure, state-of-the-art XMPP server. Since debops
provides a prosody role, I want to use this, but it seams to be a bit
incomplete. (I don't care about the ldap-integration being a to-do.)
- What is the general state of this debops.prosody role?
- Is debops.prosody integrated with debops.pki and esp. the ACME support?
Thanks in advance for any answers,
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel(a)crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
5 years, 7 months
