[debops-users] New DebOps release - v0.8.1

Hello everyone, The next release of DebOps is here, after almost 6 months. The previous release was done after 4 months of work, I wonder if a few months between releases is a good trend. Also, Groundhog Day! New DebOps release, v0.8.1 -------------------------- You can find the new version of DebOps on: GitHub: https://github.com/debops/debops/releases/tag/v0.8.1 PyPI: https://pypi.python.org/pypi/debops/0.8.1 Galaxy: https://galaxy.ansible.com/debops/debops/ You can upgrade the Python-based installation by running the command: pip install --upgrade debops DebOps is again oficially available on the Ansible Galaxy website, and the multi-role repository can be downloaded using the experimental 'mazer' content manager. You can find more details about installing DebOps in different ways in the rewritten installation documentation: https://docs.debops.org/en/v0.8.1/user-guide/install.html The brief Changelog can also be found on the documentation page: https://docs.debops.org/en/v0.8.1/news/changelog.html Complete, detailed changelog can be viewed using the 'git log' command. The DebOps documentation has a separate page which details important changes from previous release in the Ansible inventory or on the remote hosts which you might need to perform manually: https://docs.debops.org/en/v0.8.1/news/upgrades.html The Python packages available on PyPI, as well as the tarballs available on GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver network using the command: gpg --keyserver hkp://pool.sks-keyservers.net \ --recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC The v0.8.1 release ended up with ~400 commits. Here's the list of people involved in creating it, apart from merge commits: 244 Maciej Delmanowski 24 Rainer 'rei' Schuth 16 Markus Wigge 14 Robin Schneider 12 Evilham 12 Thomas Danielsson 11 Jérémy Gardais 10 Mike Nowak 8 Martin Kimmerle 8 Patryk Ściborek 8 Serge Victor 4 Nicolas Quiniou-Briand 3 Norbert Summer 3 Pedro Luis López Sánchez 2 André Jucovsky Bianchi 2 Eric Renfro 2 Jan Kowalsky 2 Stuart Mumford 1 Alex Schmitz 1 Aloïs Oddon 1 Christian Fischer 1 Fabio Bonelli 1 Gaudenz Steinlin 1 Jack Hill 1 Jonathan Struebel 1 Kamila Součková 1 Konstantinos Koukopoulos 1 Korbinian Preisler 1 Mariska Hoogenboom 1 Nick Janetakis 1 Petr Gajdůšek 1 Russell T. Sherman 1 Stuart Read Big thanks to everyone involved in this release, for finding out and fixing bugs, and adding new features to the roles. Goals reached since latest release ---------------------------------- The Redis support has been redesigned - the 'debops.redis' role has been split into 'debops.redis_server' and 'debops.redis_sentinel' roles. Both roles support multi-instance installations. The IP forwarding configuration was rethought, the 'debops.ifupdown' role now uses the 'debops.sysctl' role to configure kernel parameters independently for each network interface. Now the packet forwarding is not configured globally any more, but independently per interface. Unfortunately some software like 'lxc-net' or 'libvirt' can still enable IP forwarding globally, I haven't found a workaround for that yet. The 'debops.dnsmasq' role has been rewritten and now uses the custom DebOps filter plugins to merge configuration from different sources together. The role can now synergize with the 'lxc-net' service managed by the 'debops.lxc' role to allow easy use of the internal LXC containers, which can be reached by their hostname instead of an IP address. Unfortunately, nothing happend with the LDAP support in DebOps, but I hope to improve it in the next release. Supercharged LXC support ------------------------ During last few months I focused on improving the LXC container support in DebOps. The 'debops.lxc' role gained support for managing unprivileged LXC containers, lots of configuration tweaks and a few custom scripts to support easier container management from the command line. The 'lxc-net' service is now configured by default to support internal LXC containers, with their own DNS subdomain. If the 'debops.dnsmasq' role is configured on the LXC host, containers can be reached using their hostnames instead of IP addresses - this can be used to give unprivileged users access to their own LXC containers without giving them access to the 'lxc-*' commands to not let them mess with each other's containers. The 'lxc_ssh.py' custom Ansible connection plugin has been included in DebOps. It can be used to manage LXC containers on the remote host with direct SSH access. You can find the details in the 'debops.lxc' role documentation. On hosts with 'systemd', the LXC containers will be started using the 'lxc@.service' instances, with custom overrides. This helps the service manager correctly shut down the LXC containers at reboot and avoids issues with forcibly killing them after a timeout. Reorganization of the bootstrap playbook ---------------------------------------- The initial configuration of the '/etc/hosts' file done by the 'debops.bootstrap' playbook has been ripped from it, redesigned, and incorporated into the 'debops.netbase' role, which is now included in the 'bootstrap.yml' playbook. This should ensure that the '/etc/hosts' configuration is idempotent between the bootstrap and common playbooks, and the domain configuration is set correctly the first time. The 'debops.netbase' role supports different domain configurations (via external DNS, directly in '/etc/hosts') and should correclty update the host database in most cases. The 'debops.apt' role has been added to the 'bootstrap.yml' playbook to allow early APT configuration. This should ensure that the support for HTTPS APT repositories, as well as release backports is present when needed by other roles. The last thing left in the 'debops.bootstrap' role is management of the System Administrator accounts. I'm thinking about creating a new role for that purpose, which would allow configuration of multiple, separate, sysadmin UNIX accounts as well as support LDAP configuration if present. After that the 'debops.bootstrap' role will probably be removed and replaced completely by the playbook itself. Improvements in the DebOps repository ------------------------------------- The Vagrant environment has been updated - Vagrant hosts now use Avahi to discover other hosts in the same cluster and configure a custom '*.vagrant.test' DNS domain in their '/etc/hosts' files. This drops the need for an external DNS/DHCP server for the test environment, however the setup requires use of a bridged network. Best results can be achieved under Linux, with LXC or libvirt environments. The DebOps roles and playbooks have been cleaned up and are now tested using the 'ansible-lint' command. This results in a perfect 5/5 score on the Ansible Galaxy, woohoo! ;-) The 'make versions' command can be used to check currently pinned versions of third-party software used in DebOps roles, and compare them to the upstream versions of the released software. Since the project gains support for more third-party applications installed via 'git' repositories, this should help the project track the updated software better. The roles that use the software from the Debian Archive, or third-party APT repositories, are not included in these checks. The check has been implemented using the 'uscan' command included in the Debian 'devscripts' APT package. The 'debops.ansible_plugins' role now contains all of the custom Ansible plugins used in DebOps roles (apart from the connection and callback plugins which are not useful on the role level). This should fix issues with the custom lookup plugins not being available when DebOps roles are used independently from the whole DebOps project. The lookup plugins don't require the 'debops' Python package to be installed, and will work just fine without it, Plans for the next release -------------------------- At the moment, Debian Testing release is beginning to freeze, with the Debian Buster release aimed some time around August 2019. If we keep with the 6-month release cycle, the next DebOps release could be done somewhere around the same time. So let's aim for that and see what happens. Some of the new Debian Buster features like AppArmor enabled by default will have to be implemented in DebOps roles, that probably will end up as one of the main goals for the next DebOps version. In the next release cycle I would like to focus on the clustered environment managed by DebOps - DNS (bind9), LDAP (slapd, sssd, some kind of LDAP management interface like LAM, better support for LDAP in various applications), remote storage (NFSv4, block storage provided by Minio, Docker Registry for container storage), centralization of various services (centralized logs via logstash, filebeat, load balancing via haproxy and/or keepalived). I'm interested in further exploring the way storage and compute nodes can be managed together, to allow easy reinstallation or scaling of compute nodes as well as data replication and security on the storage nodes. I'd like to explore the "unprivileged" side of DebOps more, as explained in my previous e-mail. At present it seems that the roles could be designed with usage via an admin account by default, but allow for direct unprivileged operation in some cases. I'm not sure how the main playbook will have to change to accomodate more unprivileged operation yet. Once again, thanks to everyone involved in this project, and remember to have fun with it. See you next time. Maciej Delmanowski