[debops-users] New DebOps release - v0.8.1

Maciej Delmanowski drybjed at drybjed.net
Sat Feb 2 23:57:31 CET 2019


Hello everyone,

The next release of DebOps is here, after almost 6 months. The previous
release was done after 4 months of work, I wonder if a few months between
releases is a good trend.

Also, Groundhog Day!


New DebOps release, v0.8.1
--------------------------

You can find the new version of DebOps on:

GitHub: https://github.com/debops/debops/releases/tag/v0.8.1
PyPI:   https://pypi.python.org/pypi/debops/0.8.1
Galaxy: https://galaxy.ansible.com/debops/debops/

You can upgrade the Python-based installation by running the command:

    pip install --upgrade debops

DebOps is again oficially available on the Ansible Galaxy website, and the
multi-role repository can be downloaded using the experimental 'mazer' content
manager. You can find more details about installing DebOps in different ways
in the rewritten installation documentation:

    https://docs.debops.org/en/v0.8.1/user-guide/install.html

The brief Changelog can also be found on the documentation page:

    https://docs.debops.org/en/v0.8.1/news/changelog.html

Complete, detailed changelog can be viewed using the 'git log' command.

The DebOps documentation has a separate page which details important changes
from previous release in the Ansible inventory or on the remote hosts which
you might need to perform manually:

    https://docs.debops.org/en/v0.8.1/news/upgrades.html

The Python packages available on PyPI, as well as the tarballs available on
GitHub are signed with my GPG key. You can get it  from the OpenPGP keyserver
network using the command:

    gpg --keyserver hkp://pool.sks-keyservers.net \
        --recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC

The v0.8.1 release ended up with ~400 commits. Here's the list of people
involved in creating it, apart from merge commits:

   244  Maciej Delmanowski
    24  Rainer 'rei' Schuth
    16  Markus Wigge
    14  Robin Schneider
    12  Evilham
    12  Thomas Danielsson
    11  Jérémy Gardais
    10  Mike Nowak
     8  Martin Kimmerle
     8  Patryk Ściborek
     8  Serge Victor
     4  Nicolas Quiniou-Briand
     3  Norbert Summer
     3  Pedro Luis López Sánchez
     2  André Jucovsky Bianchi
     2  Eric Renfro
     2  Jan Kowalsky
     2  Stuart Mumford
     1  Alex Schmitz
     1  Aloïs Oddon
     1  Christian Fischer
     1  Fabio Bonelli
     1  Gaudenz Steinlin
     1  Jack Hill
     1  Jonathan Struebel
     1  Kamila Součková
     1  Konstantinos Koukopoulos
     1  Korbinian Preisler
     1  Mariska Hoogenboom
     1  Nick Janetakis
     1  Petr Gajdůšek
     1  Russell T. Sherman
     1  Stuart Read

Big thanks to everyone involved in this release, for finding out and fixing
bugs, and adding new features to the roles.


Goals reached since latest release
----------------------------------

The Redis support has been redesigned - the 'debops.redis' role has been split
into 'debops.redis_server' and 'debops.redis_sentinel' roles. Both roles
support multi-instance installations.

The IP forwarding configuration was rethought, the 'debops.ifupdown' role now
uses the 'debops.sysctl' role to configure kernel parameters independently for
each network interface. Now the packet forwarding is not configured globally
any more, but independently per interface. Unfortunately some software like
'lxc-net' or 'libvirt' can still enable IP forwarding globally, I haven't
found a workaround for that yet.

The 'debops.dnsmasq' role has been rewritten and now uses the custom DebOps
filter plugins to merge configuration from different sources together. The
role can now synergize with the 'lxc-net' service managed by the 'debops.lxc'
role to allow easy use of the internal LXC containers, which can be reached by
their hostname instead of an IP address.

Unfortunately, nothing happend with the LDAP support in DebOps, but I hope to
improve it in the next release.


Supercharged LXC support
------------------------

During last few months I focused on improving the LXC container support in
DebOps. The 'debops.lxc' role gained support for managing unprivileged LXC
containers, lots of configuration tweaks and a few custom scripts to support
easier container management from the command line.

The 'lxc-net' service is now configured by default to support internal LXC
containers, with their own DNS subdomain. If the 'debops.dnsmasq' role is
configured on the LXC host, containers can be reached using their hostnames
instead of IP addresses - this can be used to give unprivileged users access
to their own LXC containers without giving them access to the 'lxc-*' commands
to not let them mess with each other's containers.

The 'lxc_ssh.py' custom Ansible connection plugin has been included in DebOps.
It can be used to manage LXC containers on the remote host with direct SSH
access. You can find the details in the 'debops.lxc' role documentation.

On hosts with 'systemd', the LXC containers will be started using the
'lxc at .service' instances, with custom overrides. This helps the service
manager correctly shut down the LXC containers at reboot and avoids issues
with forcibly killing them after a timeout.


Reorganization of the bootstrap playbook
----------------------------------------

The initial configuration of the '/etc/hosts' file done by the
'debops.bootstrap' playbook has been ripped from it, redesigned, and
incorporated into the 'debops.netbase' role, which is now included in the
'bootstrap.yml' playbook. This should ensure that the '/etc/hosts'
configuration is idempotent between the bootstrap and common playbooks, and
the domain configuration is set correctly the first time. The 'debops.netbase'
role supports different domain configurations (via external DNS, directly in
'/etc/hosts') and should correclty update the host database in most cases.

The 'debops.apt' role has been added to the 'bootstrap.yml' playbook to allow
early APT configuration. This should ensure that the support for HTTPS APT
repositories, as well as release backports is present when needed by other
roles.

The last thing left in the 'debops.bootstrap' role is management of the System
Administrator accounts. I'm thinking about creating a new role for that
purpose, which would allow configuration of multiple, separate, sysadmin UNIX
accounts as well as support LDAP configuration if present. After that the
'debops.bootstrap' role will probably be removed and replaced completely by
the playbook itself.


Improvements in the DebOps repository
-------------------------------------

The Vagrant environment has been updated - Vagrant hosts now use Avahi to
discover other hosts in the same cluster and configure a custom
'*.vagrant.test' DNS domain in their '/etc/hosts' files. This drops the need
for an external DNS/DHCP server for the test environment, however the setup
requires use of a bridged network. Best results can be achieved under Linux,
with LXC or libvirt environments.

The DebOps roles and playbooks have been cleaned up and are now tested using
the 'ansible-lint' command. This results in a perfect 5/5 score on the Ansible
Galaxy, woohoo! ;-)

The 'make versions' command can be used to check currently pinned versions of
third-party software used in DebOps roles, and compare them to the upstream
versions of the released software. Since the project gains support for more
third-party applications installed via 'git' repositories, this should help
the project track the updated software better. The roles that use the software
from the Debian Archive, or third-party APT repositories, are not included in
these checks. The check has been implemented using the 'uscan' command
included in the Debian 'devscripts' APT package.

The 'debops.ansible_plugins' role now contains all of the custom Ansible
plugins used in DebOps roles (apart from the connection and callback plugins
which are not useful on the role level). This should fix issues with the
custom lookup plugins not being available when DebOps roles are used
independently from the whole DebOps project. The lookup plugins don't require
the 'debops' Python package to be installed, and will work just fine without
it,


Plans for the next release
--------------------------

At the moment, Debian Testing release is beginning to freeze, with the Debian
Buster release aimed some time around August 2019. If we keep with the 6-month
release cycle, the next DebOps release could be done somewhere around the same
time. So let's aim for that and see what happens. Some of the new Debian
Buster features like AppArmor enabled by default will have to be implemented
in DebOps roles, that probably will end up as one of the main goals for the
next DebOps version.

In the next release cycle I would like to focus on the clustered environment
managed by DebOps - DNS (bind9), LDAP (slapd, sssd, some kind of LDAP
management interface like LAM, better support for LDAP in various
applications), remote storage (NFSv4, block storage provided by Minio, Docker
Registry for container storage), centralization of various services
(centralized logs via logstash, filebeat, load balancing via haproxy and/or
keepalived). I'm interested in further exploring the way storage and compute
nodes can be managed together, to allow easy reinstallation or scaling of
compute nodes as well as data replication and security on the storage nodes.

I'd like to explore the "unprivileged" side of DebOps more, as explained in my
previous e-mail. At present it seems that the roles could be designed with
usage via an admin account by default, but allow for direct unprivileged
operation in some cases. I'm not sure how the main playbook will have to
change to accomodate more unprivileged operation yet.


Once again, thanks to everyone involved in this project, and remember to have
fun with it. See you next time.

Maciej Delmanowski
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 585 bytes
Desc: not available
URL: <https://lists.debops.org/pipermail/debops-users/attachments/20190202/e799b642/attachment.sig>


More information about the debops-users mailing list