[debops-users] News from the DebOps project
by Maciej Delmanowski
Hello everyone,
Welcome from the rainy Gdańsk, Poland, to another irregular status update
about the DebOps project. This time I want to talk about where the project
stands after the migration to the monorepo and what I want to work on next.
Untangling the mess
-------------------
Since the latest release, I have been working on untangling the mess that are
the 'debops.auth' and 'debops.console' roles. They were one of the first roles
in DebOps, thought out to be aggregates of sorts for different areas of host
management - authentication and authorization for the first one, and "Linux
console stuff" for the second. Over time I realized that doing aggregate roles
like these doesn't make much sense, when you can achieve the same by utilizing
Ansible playbooks - technically both types of aggregation result in the same
set of tasks, and having more granular roles lets you mix and match different
contexts as you see fit.
Therefore for the last few days I have been working on splitting the above
roles apart. You can see the results in the new set of roles in the 'master'
branch.
The 'debops.netbase' role, extracted from the 'debops.console' role lets you
configure '/etc/hosts' and '/etc/networks'. In the future, it could also take
over '/etc/services' to handle all the configuration managed by the 'netbase'
Debian package, however I didn't want to distrupt everything just yet. Since
'/etc/services' file needs to be mangled somewhat to properly mesh the DebOps
changes with Debian base, that's still something to think about. Perhaps
utilizing 'libnss-db' as a database of DebOps-managed services could be
a solution, although I'm not sure. An alternative in a larger cluster would be
providing additional service information via LDAP.
The 'debops.sudo' role, extracted from the 'debops.auth' role lets you
configure 'sudo' service. Since the '/etc/sudoers.d/' directory exists,
I don't think that other roles will use 'debops.sudo' role as a dependency
that much - creating a configuration file directly is much easier. However
having one place to configure global 'sudo' defaults seems like a decent idea.
The 'debops.system_groups' role, extracted from the 'debops.auth' role,
finally somewhat formalizes various UNIX system groups used by DebOps, like
'admins', 'sshusers' and 'sftponly'. They are properly documented, and role
allows for easy addition of new ones, as well as creation of simple ACL for
UNIX groups that can be used by other DebOps roles.
Having the above two roles as separate from 'debops.auth' allowed me to
reorganize the contents of the 'debops.bootstrap' role and replace parts of it
with the new roles. This allows for better idempotency within the project and
keeps configuration of different services in their respective place.
The things left in 'debops.auth' are a few services focused on password
management, like libpam-cracklib, which will need to be replaced with
something better suited for LDAP environment. There's also a few different
things related to LDAP authentication and authorization. These will need to be
split into smaller roles in the future. I would like to refresh the
'debops.slapd' role a bit and add replication setup before I'll mess with LDAP
stuff though.
The 'debops.console' role contains a few tasks related to filesystem mounts,
which should be moved to a separate role. I'm not sure where
fstab-related role could be placed in the 'site.yml' - having it before
network configuration will make NFS and other network filesystems broken
during initial configuration, having the filesystem mounts too late in the
playbook might hinder configuration of shared filesystems for different
applications. Perhaps two or three roles would be needed for this.
The other thing in 'debops.console' left to handle is the "fsckfix"
configuration, which requires modification of kernel boot parameters via
'debops.grub' role to properly configure fsckfix under systemd. However
'debops.grub' role is currently a bit ill-designed, and needs to be cleaned
up[1]. I'll get to it in due time.
[1]: https://github.com/debops/ansible-grub/issues/8
Python 3 compatibility
----------------------
The DebOps scripts should be now mostly compatible with Python 3. There are
now source and wheel Python packages available on PyPI built to work with both
Python 2.7 and Python 3.5+, so the project should be covered for full
transition to Python 3 installations only. There are slight issues here and
there, but thanks to the issue reports and PRs from the community they are
slowly weeded out. Thanks everyone for helping along. :-)
This unfortunately is not yet the case with the environments managed by
DebOps. Almost all DebOps roles use 'python-*' APT packages when needed, which
result in Py2 compatibility only; Ansible local facts that are written in
Python use '#!/usr/bin/env python' which results in the facts not working
correctly when Python 2 environment is not available.
To remedy that in the long term, I finally created the 'debops.python' role
which manages multiple Python versions at once. In the future I plan to
install any Python APT packages via this role as a dependency, which will
allow to selectively disable Python 2/Python 3 support as needed. I'm not sure
if Debian hosts without Python 2.7 installed are entirely possible, but soon
we will be able to find out.
The Ansible local facts and some other scripts used by roles internally have
a slightly different issue. Due to the use of 'pycodestyle' for PEP8 linting
of the scripts, shebang lines cannot be templated by Jinja. The solution I'm
considering is to provide two versions of the same script, one for the Python
3 environment, and another for Python 2 environment. You can see an example in
the 'debops.python' role. I suspect that most of the scripts would be the same
between two versions, therefore keeping both instances updated should be
relatively easy to do.
Unprivileged LXC containers
---------------------------
The 'debops.lxc' role hasn't been refreshed for a while, apart from the
changes in the shell scripts. Since Debian Wheezy will be EOL soon[2], it's
time to drop the cruft that was needed to setup LXC on Wheezy via
'jessie-backports'.
One other thing that I would like to implement is unprivileged LXC containers.
I've found a guide which implements them on Debian Stretch[3] which seems
pretty easy to implement in DebOps. One obstacle is to find a way to sanely
handle subuids and subgids. The 'root' account would need to have one set of
subuids/subgids so that unprivileged LXC containers created by root and
started at boot time would be possible. I wonder if one set of subuids/subgids
shared among different LXC containers which would use essentially the same
users/groups on the host would be a good way to handle this, security wise.
Alternatively, the root account would have to have multiple sets of
subuids/subgids for each unprivileged LXC container which quickly becomes an
implementation problem. Perhaps there is some project already that can help us
handle this?
[2]: https://wiki.debian.org/LTS
[3]: https://github.com/BenSartor/unprivileged-lxc-containers
Reorganization of DebOps Policy
-------------------------------
The old project documentation still needs to be moved into the new
documentation structure. One particular section is the DebOps Policy and
Guidelines[4] which formally defines how DebOps project works, how Ansible
roles are written, etc. I think that restructuring it into one unified section
of the documentation with numbered and easily referenced sub-sections might be
useful. This could then be used by 'ansible-lint' tests to reference relevant
parts of the DebOps Policy.
[4]: https://docs.debops.org/en/master/debops-policy/index.html
Public "work in progress' branches
----------------------------------
I recently published some "work in progress" branches in my fork of the DebOps
project on GitHub[4], named as 'wip-*'. If you like the idea, let me know
- I can publish more stuff, after cleaning up the private data used for
testing. The published code won't necessarily reflect the final roles and
playbooks, but I'm trying to have working solutions where possible.
[4]: https://github.com/drybjed/debops/branches
Ongoing development
-------------------
There are of course things that need work which I mentioned previously
- Travis-CI test suite needs to be refreshed, GitLab CI tests need to be
expanded to actually verify that the roles configure the applications
correctly, there are some monitoring roles like Icinga setup that is really
overdue, etc. This will be done as I go. DebOps has multiple areas of focus,
some of them are properly fleshed out already, others not so much. If you feel
that you have a good idea which could be implemented in the project, either
for the DebOps environments via Ansible roles, or in the monorepo itself as
the stuff that supports the project itself, let me know.
Thanks for your time and remember to have fun.
Maciej
6 years, 7 months
[debops-users] question regarding ifupdown role
by Jan Kowalsky
Hi all,
I try my first steps with ifupdown role and I wondering why there is a a
bridge "br0" attached without configuring it.
My config:
'brnc':
inet: 'static'
bridge_ports: 'none'
The result:
cat interfaces.config.d/*
# This file is managed remotely, all changes will be lost
allow-hotplug enp2s0
iface enp2s0 inet manual
# This file is managed remotely, all changes will be lost
allow-boot br0
iface br0 inet dhcp
bridge_fd 0
bridge_maxwait 0
bridge_ports enp2s0
iface br0 inet6 auto
# This file is managed remotely, all changes will be lost
allow-boot brnc
iface brnc inet manual
bridge_fd 0
bridge_maxwait 0
bridge_ports none
Maybe it's related with the lxc-role. But also there is no br0
configured at all.
Any idea?
Thanks and regards
Jan
6 years, 8 months
[debops-users] sysctr problem
by Jan Kowalsky
Hi all,
anybody has an hint, why this does not work:
sysctl__host_parameters:
- name: 'network'
options:
- 'net.ipv6.conf.all.accept_ra': 1
- 'net.ipv6.conf.default.accept_ra': 1
Error message is:
fatal: [host.example.org]: FAILED! => {"changed": false, "msg":
"AnsibleFilterError: {{ sysctl__shared_memory_map |
combine(sysctl__paging_map) | combine(sysctl__hardening_map) |
combine(sysctl__parameters) | combine(sysctl__group_parameters) |
combine(sysctl__host_parameters) }}: |combine expects dictionaries, got
[{u'name': u'network', u'options': [{u'net.ipv6.conf.all.accept_ra': 1},
{u'net.ipv6.conf.default.accept_ra': 1}]}]"}
same whith:
# enable router advertisement on host for ipv6
sysctl__host_parameters:
- name: 'network'
options:
- name: 'net.ipv6.conf.all.accept_ra'
value: 1
comment: 'Accept IPv6 RAs.'
- name: 'net.ipv6.conf.default.accept_ra'
value: 1
comment: 'Accept IPv6 RAs.'
Thanks and best regards
Jan
6 years, 8 months
[debops-users] New DebOps release: v0.7.2
by Maciej Delmanowski
Hello everyone,
This mail is kind of late, due to the recent Easter holidays, sorry about
that.
New DebOps release
------------------
About a week ago I released a new version of DebOps, v0.7.1 and immediately
v0.7.2 to fix an issue in the Docker image build. This slip up was due to how
the code is currently tested on Travis, and the test suite will need to be
overhauled during the next release cycle to avoid similar issues in the future.
You can find the new version of DebOps on:
GitHub: https://github.com/debops/debops/releases/tag/v0.7.2
PyPI: https://pypi.python.org/pypi/debops/0.7.2
You can also install the new scripts by running the command:
pip install --upgrade debops
You can find the brief Changelog in the DebOps documentation page:
https://docs.debops.org/en/latest/news/changelog.html
Complete, detailed changelog can be viewed using the 'git log' command.
The DebOps documentation has a separate page which details important changes
from previous release in the Ansible inventory or on the remote hosts which
you might need to perform manually:
https://docs.debops.org/en/latest/news/upgrades.html
The Python packages available on PyPI, as well as the tarballs available on
GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver
network using the command:
gpg --keyserver hkp://pool.sks-keyservers.net \
--recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC
The new release has a relatively small number of commits since the previous
one, 167. List of commit authors without merges:
122 Maciej Delmanowski
15 Robin Schneider
6 Alin Alexandru
4 Jérémy Gardais
3 Stefan G. Weichinger
2 Gaudenz Steinlin
2 Jan Katins
2 Jason Goldfine-Middleton
2 Peter Niederlag
1 Joseph Rawson
1 Konstantinos Koukopoulos
1 Markus Wigge
1 Oussema Cherni
1 Pedro Luis López Sánchez
1 Russell T. Sherman
1 Scott Saunders
1 Serge Victor
1 Werner M. Krauß
Big thanks to everyone who participated in this release!
Code linting and cleanup
------------------------
Most of the changes were related to linting existing code by various tools.
Right now, yamllint, pycodestyle and shellcheck are active during Travis CI
tests and any errors detected by these tools result in a failed test. To avoid
that, I suggest that you check the code beforehand locally before commiting.
There are sill some '/bin/sh' scripts which were omitted due to insufficient
detection, they will be fixed during the next release cycle.
This release comes close after the Ansible 2.5 stable release, which is now the
default target for DebOps. Due to that I cleaned up most of the deprecation
warnings in various roles, which should help with finding new ones in the
future. The missing inventory groups still emit warnings during execution,
however that has to be dealt with on Ansible side.
Role directories were cleaned up and the unnecessary '.travis.yml', 'README',
etc. files were removed. I'm still not sure how Ansible Galaxy team plans to
design the support for multiple roles in a single git repository, therefore
they might come back later.
Installation improvements
-------------------------
The 'debops' script can now correctly point Ansible to roles and playbooks
included with the Python package installed from PyPI. This allows for simple
stable releases which you can contain in their specific Python virtual
environments. You can still download the latest DebOps monorepo using the
'debops-update' command, which will take precedence over the packaged version.
DebOps now depends on the 'python-ldap' library to allow correct usage of the
'ldap_attr' and 'ldap_entry' Ansible modules. Unfortunately this changes the
installation procedure a bit since 'python-ldap' Python package does not
provide a "wheel" (binary package) and needs to be compiled from source,
therefore external dependencies are also required. The exact details will be
explained in the installation instructions, for now installing 'python-ldap'
APT package should be enough if you install DebOps from PyPI system-wide. User
or virtualenv installation requires a development environment.
DebOps should be now mostly compatible with Python 3 environment. If you find
any issues while using it with Python 3, let me know. Currently various DebOps
roles install Python 2 packages by default, however I plan to introduce a way
for the roles to selectively install Python 2 or Python 3 packages. I hope
that this allows Py3-only Debian installs in the future.
New Ansible roles
-----------------
Some of the 'debops-contrib' roles were finally integrated with the main
DebOps playbooks. The 'debops.etckeeper' role was slightly redesigned and
integrated with Ansible local facts to track any changes between Ansible runs
- the modifications will be committed automatically when Ansible gathers host
facts. Etckeeper is now enabled by default on all DebOps hosts, but that can
be disabled if needed.
The 'kernel_module' contrib role was redesigned from scratch and is now
included as 'debops.kmod'. It can be used by other roles to manage kernel
module configuration and load modules on demand if the host supports it.
The 'debops.ansible' role is back, and can now install Ansible in multiple
ways - either from Debian Archive via the backports repository, from the
Ansible upstream APT repository, or via a locally built '.deb' package.
The new 'debops.apt_mark' role can be used to change the APT package
installation status (automatic, manual) which is useful if your fresh
Debian/Ubuntu installs end up with many packages set for automatic removal.
The role is included in the common playbook by default, however there are no
specific APT packages marked, you should prepare a list yourself in the
Ansible inventory.
Changes in existing roles
-------------------------
A significant change in the PKI environment managed by DebOps is addition of
wildcard host subdomains. Previously, the generated X.509 certificates had only
'*.example.org' wildcard which allowed configuration of services for the entire
domain. The new X.509 certificates, in addition to that, will also have the
'*.<host>.example.org' wildcards which should cover services specific to
a particular host in the cluster. This should allow easy integration of
per-host monitoring services like Prometheus, Cockpit and the like. If you want
to enable the new functionality in existing PKI environments, recreating the
"end-entity" certificates should be enough, there's no need to recreate the
entire CA infrastructure.
ACME v2 is now supported via the 'acme-tiny' Python script - existing ACME PKI
realms will need to be recreated to support new Let's Encrypt API endpoints,
see the upgrade notes for details.
There were significant changes related to the '/proc' filesystem in the
'debops.proc_hidepid' role. DebOps now configures the 'procadmins' group with
a static GID 70, which allows synchronization of the '/proc' access between
a LXC host and LXC containers. The role will also ensure that the
/proc/sched_debug' file is not world-readable anymore to disabllow information
leakage to unprivileged users or unprivileged LXC containers. The
'debops.proc_hidepid' role will also check if '/proc' filesystem changes are
allowed, which should improve support for DebOps in unprivileged LXC
containers.
The bug in 'debops.nginx' related to change in Jinja2 library behaviour has
been fixed, you can now use Jinja2 2.9+ versions without issues.
The 'debops.sysctl' role was refreshed and now uses custom filter plugins
provided by DebOps to manage configuration. It will also check which kernel
parameters can be modified in the current context (host, VM, LXC container)
and only configure the parameters that are allowed.
A custom hook system has been added to the 'debops.ifupdown' role. The first
hook, 'filter-dhcp-options', can be used to selectively apply DHCP options
received from the DHCP server per network interface - useful with multiple
networks with DHCP servers.
Plans for the next release
--------------------------
The test suite used on Travis-CI needs to be revamped because the test
installs an old version of the 'debops' package instead of the "current" one.
Due to this the above mentioned quick fix was needed. Additionally, at the
moment the whole test is repeated three times, however that doesn't have to be
the case - linting, Docker image and documentation could be tested just once,
in paralell with different Python version tests.
Since Python, shell and YAML code is covered in the test suite, I would like
to add 'ansible-lint' with a set of custom rules that describe the current
DebOps practices. Additionally, roles could be verified with 'testinfra' and
additional set of tests after the playbook execution is done in GitLab CI.
There's still lots of documentation to be written, especially that the
installation instructions need to be updated. In this release cycle I'll try to
move the DebOps Guidelines and Policies into their proper place in the docs, so
that they can be properly referenced. Some of them will most likely receive
their own 'ansible-lint' tests as well.
Recent changes at work prompted me to rethink now DebOps manages networks and
IP forwarding. Right now in the 'debops.ifupdown', 'debops.ferm' and some other
roles there exist a concept of "public" ie. default, and "private" ie.
additional network, but this is counter-productive in more complicated
environments where, for example, the default network is internal and therefore
not "public". I'll try to revisit these roles and see if they can be changed to
manage networking more effectively with the new concepts in mind.
IP forwarding is currently enabled globally in 'debops.ferm' role, however this
seems to be a bad fit since the role needs additional variables to track the
state of forwarding between runs. Moving the forward configuration entirely
into 'debops.ifupdown' role and making it configurable per-interface seems to
be a better solution than the current one. The improved 'debops.sysctl' role
can then be used to configure IP forwarding in the kernel per interface, via
generated dependent configuration similarly how 'debops.ifupdown' is currently
configuring 'iptables' via generated configuration for 'debops.ferm'.
See you on the next release,
Maciej
6 years, 8 months
