DNS & DHCP
by hvjunk
Evening,
Some notes/requests:
- Consider including Kea (also from ISC) for DHCP
- Knot/NSD as authoritative Domain Name Servers. BIND has it issues or sometimes too big, when you need something less complex than BIND ;)
4 years, 4 months
Wrapper roles and interchangeable services
by Maciej Delmanowski
Hello everyone,
Due to the discussion in another thread about switching from nginx to Caddy as
the webserver[1], I'd like to pick your brains about dealing with shared but
interchangeable resources in DebOps.
Currently, infrastructure managed by DebOps is defined in separate Ansible
playbooks, each playbook uses one or multiple Ansible roles. Some of these
roles manage resources (services, files, configuration, etc.) that are
singular in nature - there's only one set of APT preferences available to be
configured, or one PostgreSQL database service, and so on. There's no need to
think aboout alternatives for these.
One the other end, there are usually application roles like gitlab, netbox,
basically any higher-level application that rely on the lower level Ansible
roles to configure other services for them, like firewall, web server, and so
on. In most cases these application roles/playbooks are not enabled by default
so users can mix and match them in their infrastructure at will, and there's
no need to worry about shared interface for them.
But in the middle between these two extremes, there's a set of services that
can be shared between different applications and are provided in Debian with
alternatives which influence the user preference for them. Some examples
include:
- webserver (nginx, apache2, caddy)
- firewall (ferm, firewalld, ufw)
- SMTP server (nullmailer, postfix, dma)
- NTP server (openntpd, chrony, timesyncd)
Due to how DebOps codebase is designed and restrictions around it (read only
roles and playbooks), it is hard to come up with a good method of handling
these services via Ansible. Over the years, we found different strategies of
dealing with them in the project:
1. Create a role that manages a service and can support alternative
applications to do so. This is most prominently used in the 'ntp' role,
which supports installation of different NTP servers chosen by the user.
It's a reasonable method of encapsulation for this kind of problem, but it
makes the role much more complicated that it needs to be, and is not
sustainable with more complex services like webservers.
2. Create two or more alternative playbooks which use alternative services,
each playbook is activated by its own Ansible inventory group. This
happened with the 'owncloud' role and its possibility to use either 'nginx'
or 'apache2' as the webserver frontend. This method moves the problem from
the role level all the way to the actual hosts themselves - if somebody
wants to install Nextcloud with Apache2 as the frontend for some reason,
they cannot use the same host to configure another application that only
knows how to integrate with 'nginx', say NetBox, because the Nextcloud
installation will be broken in the process.
This solution is also not really scalable - if we go one level down, we get
the firewall role (ferm) which is used by dozens of other roles. If we add
an alternatve 'firewalld' role (which I would like to do at some point),
each service that relies on 'ferm' would have two or more alternative
playbooks. In an extreme scenario, DebOps could provide users with multiple
sets of 'site.yml' playbook trees to choose from, but with each choice the
number of those sets would increase exponentially (firewall * webserver,
for example).
These are two extremes we came up so far, so I suspect that the answer might
be somewhere in the middle. A possible solution I came up with is a concept of
a "wrapper role" which moves the complexity for managing multiple services in
a single role onto itself by importing selected service role during Ansible
execution. This solution has interesting properties and drawbacks, but let's
start with an example.
Below you can find a description of the 'firewall' role. It lets us switch
between 'ferm' and 'firewalld' services on the playbook level without the need
to modify existing playbooks:
# ansible/roles/firewall/defaults/main.yml
firewall__enabled: True
firewall__service: 'ferm' # alt: firewalld
firewall__ferm__dependent_rules: []
firewall__firewalld__dependent_rules: []
# ansible/roles/firewall/tasks/main.yml
- name: Manage the firewall service
import_role:
name: '{{ firewall__service }}'
tags:
- 'role::{{ firewall__service }}'
- 'skip::{{ firewall__service }}'
vars:
ferm__dependent_rules: '{{ firewall__ferm__dependent_rules }}'
firewalld__dependent_rules: '{{ firewall__firewalld__dependent_rules }}'
when: firewall__enabled|bool
# ansible/playbooks/service/firewall.yml
- name: Manage the host firewall
hosts: [ 'debops_all_hosts', 'debops_service_firewall' ]
become: True
roles:
- role: firewall
tags: [ 'role::firewall', 'skip::firewall' ]
With this setup, the complexity is moved "in the middle", into a separate role
which then imports the desired final service role. The problem comes with
selecting the actual service role - this cannot be done on the Ansible
inventory level. The 'import_role' tasks are processed by Ansible before the
inventory is even touched, so only the default value ('ferm') would have any
effect on the 'import_role' task. But even if we could define that on the
inventory level, Ansible would want to run two competing roles on different
hosts at the same time. It cannot do that though, and will select the first
service role that it knows about (ferm) and run it on all hosts.
However, this can be solved by the use of the '--extra-vars' Ansible option.
If we specify the 'firewall__service' variable on the command line, we can
override it during Ansible execution and change the default 'ferm' role to
'firewalld'. With this in mind, I added the support for global variables in
previous release of DebOps[2].
Users can deal with use of different firewall services on different hosts by
creating separate inventories for different sets of hosts and selecting
different firewall services per inventory. With that in mind, I want to add
support for multiple Ansible inventories in a DebOps project directory with
shared resources like PKI infrastructure to faciliate easier environment
integration.
Other roles that depend on the hypothetical 'firewall' role would just add it
in their playbooks as normal and pass the '*__dependent_rules' variables to it
with the desired configuration. As long as the dependent configuration does
not involve variables from the role defaults and/or is properly wrapped in
default([]) constructs, there shouldn't be any issues with this usage.
This becomes a bit more complex when we consider other such "wrapper role"
that might require dependencies on our "firewall" role. For example, the
'ntp' role would become a wrapper role for 'openntpd', 'chrony', 'ntpdate',
'timesyncd' roles, some of which require firewall configuration, and some not.
How can we deal with this? By multiple role imports, of course. Below is an
example rewritten 'ntp' role that supports different NTP servers, some
defining their own firewall rules:
# ansible/roles/ntp/defaults/main.yml
ntp__service: 'openntpd' # alt: chrony, timesyncd, ntpdate
ntp__deploy_state: 'present'
ntp__ferm__dependent_rules:
- '{{ openntpd__ferm__dependent_rules | d([]) }}'
- '{{ chrony__ferm__dependent_rules | d([]) }}'
ntp__firewalld__dependent_rules:
- '{{ openntpd__firewalld__dependent_rules | d([]) }}'
- '{{ chrony__firewalld__dependent_rules | d([]) }}'
# ansible/roles/ntp/tasks/main.yml
- name: Run the firewall role
import_role:
name: 'firewall'
vars:
firewall__ferm__dependent_rules:
- '{{ ntp__ferm__dependent_rules }}'
firewall__firewalld__dependent_rules:
- '{{ ntp__firewalld__dependent_rules }}'
tags:
- 'role::firewall'
- 'skip::firewall'
when: ntp__service in [ 'openntpd', 'chrony' ]
- name: Run the service role
import_role:
name: '{{ ntp__service }}'
tags:
- 'role::{{ ntp__service }}'
- 'skip::{{ ntp__service }}'
vars:
openntpd__deploy_state: '{{ ntp__deploy_state }}'
chrony__deploy_state: '{{ ntp__deploy_state }}'
timesyncd__deploy_state: '{{ ntp__deploy_state }}'
ntpdate__deploy_state: '{{ ntp__deploy_state }}'
# ansible/playbooks/service/ntp.yml
- name: Manage the NTP service
hosts: [ 'debops_all_hosts', 'debops_service_ntp' ]
become: True
roles:
- role: ntp
tags: [ 'role::ntp', 'skip::ntp' ]
As you can see, the complexity is removed from the playbook level and
encapsulated in the intermediate "wrapper" role. The final service roles can
focus on their own services without the need to concern themselves with the
alternatives, as long as they provide the required variables for 'ferm' and
'firewalld' roles to consume. Initial deployment doesn't change from the
current DebOps standards, but switching between different alternatives
involves 3-step process. For example to switch from the default 'openntpd'
service after it has been deployed, users have to:
- run the playbook with 'ntp__deploy_state=absent' to remove openntpd and any
dependent configuration such as that in the firewall
- set the 'ntp__service=chrony' in the 'global-vars.yml' file.
- run the playbook again, to apply the new service configuration
Now, the thought in everyone's minds probably is "this is too complex". Yes,
I agree with you all. But in actuality, the complexity stays the same, we are
just moving it around and concentrating everything from "both sides": we merge
the two playbooks together, and on the other side, we move the selection of
what service to use from the service role into a separate 'wrapper role" which
needs to be aware of all the possible choices to work correctly. Since all
that complexity is now concentrated in one spot, that's why it looks to be
bigger than it actually is.
In fact, the final service roles, and the firewall roles, can be used just
fine without the wrapper roles when we define a playbook for each combination
separately. For example, 'ferm' firewall with 'chrony' NTP server:
# ansible/playbooks/service/ferm-chrony.yml
- name: Manage the chrony service with ferm firewall
hosts: [ 'debops_service_chrony_ferm' ]
become: True
roles:
- role: ferm
tags: [ 'role::ferm', 'skip::ferm' ]
ferm__dependent_rules:
- '{{ chrony__ferm__dependent_rules }}'
- role: chrony
tags: [ 'role::chrony', 'skip::chrony' ]
And we're back to the current state of the art.
The actual implementation of all this would take some time. The 'ntp' role
would have to be split, but we have most of the code written already so
that's an easy step to take and could be done first as a proof of concept.
A 'firewall' role could be easily created and only support 'ferm' for now, so
that we can update all playbooks in anticipation for the new 'firewalld'
and/or 'ufw' roles. For the webserver we could do the same, but the 'apache'
role would have to be upgraded to support similar functionality to the
'nginx' role, not to mention that both probably should be rewritten from
scratch with current configuration models in mind.
But before I start messing around in the actual code, a question: is this
overengineering? Am I going too far with this? All the trouble comes from the
fact that DebOps roles are written with separation of concerns in mind, and
pass the configuration for dependent roles via variables instead of directly
messing with the services. It seems that this approach is not really common
in the Ansible world outside of DebOps. Most of the time role creators write
example configuration for a given role in the documentation and let the users
deal with it on their own... But usually these roles focus on working with
multiple Linux distributions. DebOps went essentially in an opposite
direction, with focus on a single Linux distribution (Debian), but with the
intention to provide as much support in that space as possible.
An alternative approach would be to rip out the "glue" from the roles and
playbooks and let the users deal with combining separate services together
themselves. That would move the problem outside of Ansible in a way.
Eventually we could deal with this via the rewritten 'debops' scripts which
could manage the inventory for the users and combine different setups to
generate playbooks for Ansible to use. This would let us deal with the
complexity using a normal programming language (Python) instead of going with
it through Ansible and its DSL. But we are a long way from that.
Let me know what your thoughts are on the subject.
-- Maciej
[1]: https://lists.debops.org/hyperkitty/list/debops-devel@lists.debops.org/th...
[2]: https://docs.debops.org/en/master/user-guide/project-directories.html#the...
4 years, 4 months
Project development and governance model
by Maciej Delmanowski
Hello everyone,
During today's online meeting via Jitsi, we talked a bit about how DebOps is
currently developed, how this might change in the future and what are possible
scenarios for the project. Two models were discussed, one where there's much
more maintainers and contributors involved in the development with much larger
scope, and a second one where project stays tightly focused on a small set of
features with fewer contributors. I think that this warrants further
discussion on the mailing list, since many people are using DebOps in
production environments and I imagine that they wouldn't want to see suprises
there.
Going a bit backwards, current release model and schedule is described
here[1], and if you haven't seen that yet, the development worklow which I'm
using at the moment is described in the documentation as well[2]. What is left
out (I think) is how the actual code is developed and where does it come from.
I usually include a report of the number of contributions with each new
release, and you can see a monthly progress on GitHub insights page[3]. At
present the project is developed mostly by me and a few other people active in
any given month, and there's usually a long tail of one or two changes per
person. The changes get into the project via GitHub pull requests, and after
a review usually by me, I pull them to my local repository and merge to the
'master' branch when they are accepted. At the current size of the project it
seems that this is enough.
Is that a good governance model for DebOps? I suppose, it seems that it works
out well so far. In this model I get the last word about what gets in and what
doesn't, but on the other hand there's a backlog of proposed changes that need
to be reviewed and accepted which takes time, so it's currently slow. DebOps
is at the moment developed by volunteers in their free time - some of that
time is of course during their normal work hours if DebOps is used at work.
With more and more contributors providing patches that will probably have to
change, otherwise the pace of development will grind to a halt.
There are two solutions we discussed a bit during the meeting. When the
current codebase and its scope is too large, we could essentially split the
project into smaller subprojects with each one being maintained by a person or
a group of people who then try and combine everything into a whole. This is
a model used by OpenStack and can probably be used effectively when each
subproject is backed by a company and has dedicated team behind it. On the
other hand we had a similar situation in the past, where each role was
developed in a separate 'git' repository which increased the maintenance
burden when changes had to be synchronized everywhere. That's why switch to
the monorepo happened.
Another way is to use the model used by the Linux kernel developers, where
there are essentially "layers" of contributors that create changes and push
them to maintainers of subsystems which after review push them to Linus to
merge in his official kernel. If you're interested, the whole process was
nicely explained by Greg Kroah-Hartman in his 2016 presentation about the
topic[4].
Both models assume that a large number of people participate in the project,
so at the moment we don't have to worry about which path to take. But I think
that the model used by the kernel developers fits better for a project such as
DebOps, which is developed in the open, shared space by a dedicated community.
Use of 'git' as the version control system also gives us an advantage here
- since the project is maintained in a monorepo, we can leverage the mechanism
of codeowners[5] to give interested people "ownership' of parts of the code.
In such case they would take over reviews and maintenance of selected parts of
the codebase. That file is currently present in the DebOps monorepo[6] but is
not really used for anything - I'm keeping track on all changes and nobody so
far volunteered to help with reviews of a specific role or part of the
codebase, at least since that mechanism was added. If you want to be included
to easily keep track of parts of the repository that interest you, let me
know. In such case I'll wait with merging a given pull request for your
review.
Another closely related topic are the git commits themselves. At the moment,
with relatively small number of contributions, I accept the git commits as-is,
with close examination for any security issues. But if number of contributors
increases ten-fold, this will become an attack vector - remember that DebOps
code is executed with root privileges on a large and distributed
infrastructure - very interesting target indeed... In such case, the commits
will have to be cleaner to meet approval.
At the moment in our contributor workflow[7] we mandate use of GPG signatures
in git commits and advocate for clean and concise git commit messages[8], but
that's rarely enfoorced. The output of 'git log -p --no-merges' leaves a lot
to be desired in the context of easily readable and clean commits, and I'm
guilty of it as well - partially because I'm trying to record changes in the
official Changelog file and I don't want to simply repeat the note from the
Changelog in the commit message. So now I wonder, if a better focus on cleaner
git commits in lieu of less verbose Changelog file would work better in the
long run? How many of you rely on the Changelog to keep track of important
changes (the upgrade notes are of course another matter and should stay as
they are)?
As for refactoring badly done commits, this can be done but refactoring
overrides the GPG signatures... So maybe we should only care about them on the
"intake" instead, for example on the GitHub pull request page, and when the
code is pulled and being prepared for merge to the 'master' branch, I could
freely refactor and fix up the commits when necessary? The original GPG
signatures will be goned by this point, but authorship of the code will
remain. In such case we coould also stop using non-ff merges which create
a separate signed merge commit to mark acceptance of code, and the resulting
commit history would look a bit better without the need for '--no-merges'.
Or alternatively, should I point out all the issues in the pull request before
the pull, then the original authors could rework the patches, rebase where
necessary and provide a set of clean commits? To be honest that seems a bit
imposing to me, but if more and more people are relying on my judgement about
what gets into the codebase, perhaps my standards need to be higher?
Let me know what you think,
Maciej
[1]: https://docs.debops.org/en/master/news/releases.html
[2]: https://docs.debops.org/en/master/developer-guide/development-model.html
[3]: https://github.com/debops/debops/pulse/monthly
[4]: https://www.youtube.com/watch?v=vyenmLqJQjs
[5]: https://docs.github.com/en/github/creating-cloning-and-archiving-reposito...
[6]: https://github.com/debops/debops/blob/master/CODEOWNERS
[7]: https://docs.debops.org/en/master/developer-guide/contribution-workflow.html
[8]: https://chris.beams.io/posts/git-commit/
4 years, 4 months
caddy: "easier" web server + HTTPS
by hvjunk
While setting up this DHCP/DNS server, I got reminded of Caddy
I’m busy replacing nginx with Caddy, for the simple reasons:
- simpl(er) configs
- API
- Automatic LE HTTPS certificate fetching
Also consider this as a possiblity/etc. especially when not needing all the WAF from nginx
4 years, 5 months
Documentation team, assemble!
by Tasos Alvas
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hey,
So, I have also been thinking about a few things @nqb mentioned on the meet.
I'm happy we care about a lot of the same stuff, so I thought I'd invite
some
structured discussion on a few issues.
Also this touches on what @drybjed just posted on the project development
model,
but I'll try to keep this about doc and post that stuff there.
# Documented means explicit
Maybe I'm hammering on this one too often, but `parse_kv_config` has unit
tests,
excellent coverage, is totally required to understand debops, and it's only
presented as an implementation detail on each role.
It has all of the overhead of a promise to the end user, only missing the
part
where we make that promise.
No action required, I'm handling this one, I'm just mentioning it as an
example.
As the project is facing the challenge of decentralizing responsibility, a
whole set of guidelines need to be made explicit so they can be discussed
and enforced without necessarily involving Maciej.
It might be tempting to have him document everything every time there's an
issue
but there must be a better way!
# Doc team. STYLE FORCE?
Keeping the top-level doc consistent is not a task that will occur
naturally as
devs write roles.
It involves keeping track of the onboarding path of all (or some) of the
supported user types (say admin/user/dev), possibly presenting simplified
versions to some.
This mindset can be incompatible with someone who just explained a subject
fully to the computer.
I propose a documentation team, as I think it's not only a good
responsibility
to decentralize, but also a domain that could benefit from a perspective
that's
not the code author's.
In broad strokes, the process I have in mind is like this:
* Doc team gets `/docs` in `CODEOWNERS`, either wholesale or in chunks
* Require a doc team member to close issues that add global functionality
(No need to block the PRs themselves)
* Leave the issue open and tag someone responsible if replies to an issue
should result in documentation
This doesn't apply equally to roles, since their actual codeowners will know
the internal workings better, but the doc team will still be in the position
to define and advocate for best practices. Obviously I'm not suggesting we
write people's role doc. :D
So the doc team turns answers into guidelines, helps people adhere to those
guidelines, identifies unclear points and iterates.
It might make sense to generalize the group's function to coding style as
well,
since those issues are likely to benefit from the same process.
I'm not happy about bloating the group's responsibilities, but then it can
be
called `STYLE FORCE!`. xD
I think that following such a convention would allow people to contribute to
the project earlier and stay involved until they're ready to handle harder
stuff. Even going from "I fixed some typos" to "I wrote an extra page" can
be
intimidating.
Anyway, that's my RFC. I'd rather not be the team alone;
I'm not even remotely financially secure to be able to solo such a
commitment.
If we're 2 or 3 people I can see it working, though.
Tasos
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEXbaOHtREtFSBLXcdOXguVye2c7EFAl8ZbEQACgkQOXguVye2
c7G38hAAhPFStDkNfTf57q8Ta4EEjEIXlyR8gcZTCr4RG3sCW3Lvy+EpHVwTh7YP
YqBrHKS83yRFld0xLEjRN0d7SiUlJCMWG93zY7qCRysU/CvFQYQftFsbHlekWRqk
tECFHadaRQ4/GJr1BSHkgrjRBqvVIsg0JBh9JqyJC0Nso+Z4CqAvknRSXqgS6ytb
FeWrqMSy8Q8eCN5csjvs31+TVf6Vv+9+MffSuFNv2yDs0pwyOopNRhNAtwirOqdk
ys08orPbGCfugZLdnMBIARG3ROpnPSPIEoK9Wkm0XVskxZI5YByVg4rtEL6zYWBp
+Erzjgxc/1ryTh23q6gVZv8LAvxd01Z5zIcJg9snEEJFhLKd5E2GEB6lmH1D15nX
izSw7XDguDWNon7eWikakOs0+Y0BaBORilgmJgMisZOzgjcKU/htdtc8oQYTLaqy
mYQPOlXcH/R6HsdmNFficejZzH5oeqnYhNvN7qBoONCvTQDBX1+1CP5PQUbLUJ9U
ngg3M3cVvEhVPXp3AVLw6Gp9SJijpLK47JrZYOpCk9j5FSEpjO6J9E99WX4pQVI1
fvw2aCWmxwRxxgNqEZzmGCVYxj82kz5VnLWIEU8rc8fK1OudhQvZsS1yhENUKgZo
mbwuapqdjj3WAwJNvdXDGZhZiEyEN0oWUz6aMDeycjWEY7P2uyA=
=EY7U
-----END PGP SIGNATURE-----
4 years, 5 months
New project mailing list: debops-devel
by Maciej Delmanowski
Hello everyone,
Yesterday we had our first online meeting via Jitsi which went pretty well.
I think that Imre Jonk will write more about it soon, so I won't spoil too
much here.
Recently there was an idea of creating a separate mailing list focused on
DebOps development instead of its general usage. This comes from the fact
that there are two sets of users of this project: some people just want to use
it to manage their infrastructure and focus on their own work, and some are
interested in contributing ideas or code to DebOps itself.
Due to that, I went ahead and created a new mailing list,
'debops-devel(a)lists.debops.org', meant to be used for discussions about DebOps
development. You can view both lists (debops-devel and debops-users) similarly
to how debian-devel and debian-users lists are used in the Debian project.
Interested users can subscribe to the new mailing list via the Postorius web
interface[1]. I plan to use this list to discuss large changes in the
codebase, explain new ideas planned for implementation in the project, discuss
the problems found in IT infrastructure and how we can solve them via DebOps,
and similar topics.
Cheers,
Maciej
[1]: https://lists.debops.org/postorius/lists/debops-devel.lists.debops.org/
4 years, 5 months
